Non-invasive methods for heart rate measurement in fish based on photoplethysmography

Heart rate is an important physiological indicator for understanding the metabolic state of fish and the effects of environmental changes on fish (Lefrançois and Claireaux, 2003; Campbell et al., 2007; Campbell and Egginton, 2007; Anttila et al., 2014; Joyce et al., 2016; Lonthair et al., 2017). Nevertheless, existing methods for measuring heart rate in fish are often invasive, causing injury, or requiring delicate manipulation, making the measurements not efficient enough. For electrocardiogram (ECG) in fish, the conventional methods include invasive, implantable and non-invasive approaches. However, both invasive and implantable ECG could cause damage to the fish, because the invasive method involves inserting electrodes near the fish's heart (Shelton and Randall, 1962; Zhao et al., 2019) and the implantable method requires surgical implantation of a bio-logger (Arvén Norling, 2017; Hvas et al., 2020). Although the non-invasive method only involves placing the fish on an electrode grid without electrode insertion (Altimiras and Larsen, 2000; Casselman et al., 2012), the devices used for non-invasive ECG are more complicated compared with the other two methods. Overall, all three ECG methods require expertise and are not simplified enough in their operation. Alternative methods, either invasive or requiring delicate manipulation, have also been used to measure fish heart rate. For instance, Brodeur et al. (2001) employed a Doppler flow probe placed around the ventral aorta of the fish, and this approach necessitates surgery. Hafner and Lubecke (2010) utilized a Doppler radar sensor to measure the heart rate of fish. Benslimane et al. (2019) and Muir et al. (2021) adapted the mice Doppler echocardiography platform to assess the cardiac function of fish. Furthermore, researchers have employed microscopic video analysis to study embryos or small-sized fish, but this method typically required the subject to be transparent, as demonstrated in studies by Nepstad et al. (2017), Puybareau et al. (2017) and Mousavi and Patil (2020).

Here, we introduce two non-invasive and simple methods for measuring fish heart rate based on traditional photoplethysmography (PPG) and remote photoplethysmography (rPPG). PPG involves measuring heart rate by detecting blood volume changes in the heart or blood vessels (Ismail et al., 2021). In fish, PPG can be performed by both non-invasive and implantable methods. The implantable method requires implanting the device into the fish, which can cause injury (Svendsen et al., 2021a,b). The non-invasive method of PPG is of either transmittance or reflectance type according to the relative position between the light source and the photodetector. In the transmittance type, the light source is placed opposite to the photodetector, allowing the emitted light to pass through the tissue and be detected by the photodetector. In the reflectance type, the light source and the photodetector are placed on the same side, with the light being reflected by the tissue before being captured by the photodetector (Ismail et al., 2021). Most of current non-invasive PPG methods for fish are of the transmittance type. Yoshida et al. (2009) and Naderi et al. (A. M. Naderi, R. S. T. Torres, Z. Zhang and H. Cao, unpublished data) adopted the transmittance-type PPG to measure the heart rate of zebrafish (Brachydanio rerio), medaka (Oryzias latipes), goldfish (Carassius auratus) and largemouth bass (Micropterus salmoides). When the experimental fish were small in size, superior heart rate signals were obtained, but when the experimental fish were large in size, the quality of their heart rate signals diminished. Moreover, the transmittance-type PPG device requires the light source and photodetector to be separated, which is inconvenient. In contrast, the reflectance-type PPG integrates the light source and photodetector into a single sensor, making it simpler and more efficient. For example, in shellfish studies, a single sensor placed close to the heart region on the shell was used for heart rate measurements (Chen et al., 2016; Dong et al., 2021). Therefore, in this study, we introduced the reflectance-type PPG for measuring the heart rate of fish.

rPPG involves measuring heart rate by detecting the subtle color variation of the human skin tissue from videos (Verkruysse et al., 2008; Lin and Lin, 2017; Jaiswal and Meenpal, 2020). Studies of rPPG in fish have not been reported, and only related studies have measured heart rate in transparent zebrafish using imaging PPG (Pylatiuk et al., 2014; De Luca et al., 2015; Martin et al., 2019; Machikhin et al., 2020; Volkov et al., 2022). The application of rPPG for fish heart rate measurements offers improved convenience, as it requires only a camera and an appropriate light source for non-invasive measurements, without the need for special equipment. Relevant studies have been conducted to improve the accuracy of rPPG in humans, which can benefit our research. For example, Anderson et al. (2010) demonstrated that the joint blind source separation (JBSS) outperformed blind source separation (BSS) in exploiting the dependencies of sources across datasets. A JBSS method called independent vector analysis–gaussian (IVA-G) was employed for rPPG (Zhang et al., 2021; Cheng et al., 2021). Independent component analysis (ICA) is a method utilized for feature extraction and signal separation (Hyvärinen and Oja, 2000). A computationally efficient algorithm of ICA called fast fixed-point independent component analysis (FastICA), developed by Hyvarinen (1999), was applied for rPPG (Karmuse et al., 2022). Additionally, empirical mode decomposition (EMD) is an adaptive method introduced for analyzing non-linear and non-stationary signals (Torres et al., 2011), serving as an advanced signal processing method for rPPG (Al-Naji et al., 2017; Al-Naji and Chahl, 2018).

In this study, the large yellow croaker (Larimichthys crocea) was used in our experiments because of its prominent economic significance and highest yield among mariculture fishes in China (Chen et al., 2020). Therefore, it served as a representative fish for our research. The development of these two PPG heart rate measurement methods will bring several benefits as listed below. Firstly, the non-invasive methods mitigate the stress response of the fish during measurements, leading to a more accurate reflection of their physiological condition. Secondly, it eliminates the need for surgical implantation of devices in fish, minimizing the harm to animals during the research process and complying with animal welfare and ethical principles. Thirdly, the simplicity of the PPG method enhances measurement throughput, which is particularly advantageous for genetic studies on fish heart rate and physiology. Our original aim in developing PPG methods was to investigate the genetics of cardiac function in the large yellow croaker.

All experiment procedures were approved by the Animal Care and Use Committee at the College of Ocean and Earth Sciences, Xiamen University, in accordance with the Guidelines for the Care and Use of Animals for Scientific Purposes set by the Ministry of Science and Technology, Beijing, China (no. 398, 2006), the National Standards for Biosafety Laboratories (GB19489–2008), and the Care and Use of Laboratory Animals published by the US National Institutes of Health (NIH publication no.85–23, revised 1996).

Large yellow croakers were obtained from Fufa Aquatic Products Co., Ltd (Ningde, Fujian, China) and cultured in seawater (Fufa Aquatic Products Co., Ltd). The fish were held in a tank (length: 5 m, width: 2 m, height: 1.4 m) with the water level maintained at 0.5 m. They were fed once a day in the morning and the seawater was changed daily with filtered seawater. The water temperature was maintained at 13±0.2°C, following the natural light cycle.

A group of fish (n=6, mass: 13.9–54.2 g, body length: 9.2–14.4 cm) was randomly selected for the PPG experiment. Each fish was anesthetized (130 ppm MS222 buffered with NaHCO₃) until the equilibrium was completely lost and the opercular movement was reduced. Subsequently, the fish was placed in the custom-made apparatus, ventral side up, with continuous gill irrigation with aerated seawater containing a maintenance anesthetic concentration (110 ppm MS222 buffered with NaHCO₃), by inserting a tube into the fish's mouth at a water flow rate of 200 ml min⁻¹. The 110 ppm MS222 was the minimum concentration for maintained anesthesia as determined by us in preliminary experiments. During the experiment, the fish remained fully submerged in the water. To capture the PPG signal, a reflectance-type infrared sensor with both an infrared emitter and receiver was positioned in the heart area on the ventral surface of the fish, as shown in Fig. 1A. The infrared emitter has a wavelength of 940 nm and a power of 10 mW. The infrared sensor was connected to an amplifier (Amp 03, Newshift), which, in turn, was connected to a data acquisition system (Powerlab, ADInstruments). Software (Labchart8, ADInstruments) was used to set the filtering parameters, with a low-pass filter set at 5 Hz, and the sampling rate was set at 1000 Hz. Additionally, ECG measurement was conducted simultaneously to evaluate the accuracy of PPG. Three electrodes were used for ECG signal capture. The positive electrode was inserted near the heart, the negative electrode was inserted into the body or placed into the seawater, and the reference electrode was placed into the seawater, as depicted in Fig. 1A. The electrodes were connected to an amplifier (Bio Amp, ADInstruments) which was connected to the data acquisition system (Powerlab, ADInstruments). The filtering parameters were set using software (Labchart8, ADInstruments) with the low-pass filter set at 100 Hz and the high-pass filter set at 10 Hz, maintaining the sampling rate at 1000 Hz. Each fish underwent a 5 min measurement period. After the measurement, the fish was placed in non-anesthetic seawater for recovery before returning to another tank with the same conditions as described in ‘Experimental animals’. Upon completion of the experiment, the fish was returned to Fufa Aquatic Products Co., Ltd for further commercial cultivation.

Similarly, a group of fish (n=6, mass: 36.3–66.8 g, body length 12.5–15.7 cm) was also randomly selected for the rPPG experiment. The anesthesia induction and maintenance for each fish were identical to the procedures for the PPG experiment. The ventral side of each fish was recorded using a camera (Intel RealSense Depth Camera D435), with videos captured at 30 frames s⁻¹. The video recording software used was Intel RealSense Viewer (https://github.com/IntelRealSense/librealsense/releases/tag/v2.42.0). Subsequently, the rPPG signal was acquired using the analytical process described below (‘The analytical process of the proposed rPPG’). Before the analysis, the recorded videos were converted into image sequences using the conversion software rs_bag2image (https://github.com/UnaNancyOwen/rs_bag2image). To determine whether green light could enhance the accuracy of the rPPG method, videos were recorded under both natural light and green light conditions. Simultaneously, ECG measurements were conducted under both light conditions to evaluate the accuracy of the rPPG, as described above (‘PPG experiment’). Each fish was recorded for a duration of 5 min under each light condition. After completion of the measurement, the treatment of the fish was identical to that described for the PPG experiment.

The framework of the rPPG method is shown in Fig. 2. Initially, the regions of interest (ROI) for the ventral side of the fish and the background are selected, as illustrated in Fig. 1C. The signal matrix of two ROIs is presented by average pixel intensity of the RGB channels. Subsequently, ambient noise of the two ROIs is removed based on IVA-G (Anderson et al., 2011). According to the dichromatic model (Wang et al., 2017), the signal matrix of fish can be considered to contain specular reflection from skin surface, diffuse reflection from skin tissues, and illumination from the light source. Following this, illumination from the light source is removed by projecting the signal matrix to a specific orthogonal plane. After that, ICA is performed using Fast-ICA (Hyvarinen, 1999). The candidate independent component with the highest correlation with the green channel of the signal sequences is selected. Subsequently, the time–frequency decomposition of the candidate independent component is performed using EMD (Colominas et al., 2014), culminating in the obtention of the intrinsic mode function with the highest match to the heart rate range. Finally, the heart rate information is presented based on the intrinsic mode function. The algorithms for rPPG were executed in MATLAB. The details of the rPPG method are introduced below.

After obtaining k frames of the video image sequence of the ventral side of the fish, the signal matrix of the two selected ROIs was presented as follows. In each channel, the average pixel intensity of the ROI in each frame is calculated by summing all pixel values of the ROI and dividing by the number of pixels of the ROI. The average pixel intensities of the ROI for each frame are combined in time order to obtain the signal sequences of the ROI.

Next, to find the common ambient noise component in the SCVs, the linear correlation between the SCVs of the two ROIs is evaluated by calculating the 3×3 correlation matrix. The largest value in this matrix is found and used to zero out the two SCVs corresponding to it. This results in a new SCM, which is denoted as ⁠.

Assuming that the maximum value in the 3×3 correlation matrix is in the first row and column, it implies that SCV_hr1 and SCV_bg1 exhibit the strongest correlation among the SCVs. These two SCVs are treated as ambient noise components and set to zero to obtain SCM_new as shown on the right side of Eqn 4, where 0∈ℝ^1×k represents the zero vector.

The reconstructed SCM_new is subsequently inserted into Eqn 3 to obtain the updated X. The first three rows of X, representing the target ROI signal matrix, are taken as ⁠. X_new represents the signal matrix after the removal of the common ambient noise component.

According to the dichromatic model proposed by Wang et al. (2017), X_new, the signal matrix representing the ROI of the fish's ventral side after removing common ambient noise, can be regarded as a linear combination of specular reflection from the skin, diffuse reflection associated with the absorption and scattering of the light from the skin tissues, and illumination from the light source. To reduce signal dimensionality and eliminate noise from illumination of the light source, X_new is projected onto a plane orthogonal to the illumination of light source.

Next, the correlation coefficients between s₁, s₂ and G_hr are calculated separately. The independent component with the highest correlation with G_hr is selected because of the presence of hemoglobin in the blood of the large yellow croaker (Gu and Xu, 2011), which exhibits strong absorption of green light (Roggan et al., 1999). Lastly, given the known frame rate of the video and setting the signal's start time to 0, this independent component can be converted into the time series S_green(t).

The average heart rate within a 60 s period was used as a single record. Therefore, there were a total of 30 measurements for PPG, 30 measurements for rPPG under natural light conditions, and 30 measurements for rPPG under green light conditions. Bland–Altman plots (Bland and Altman, 1986) were adopted to assess agreement between PPG and ECG, as well as between rPPG and ECG, by plotting the differences between the methods against their average values, as depicted in Fig. 3D–F. Prior to conducting the Bland–Altman plots, the difference between PPG and ECG, as well as between rPPG and ECG, was assessed by using the Shapiro–Wilk test. In addition, the accuracy of PPG and rPPG was assessed through the Pearson's correlation coefficient (r), mean absolute error (MAE) and root mean square error (RMSE). All statistical analyses were performed in R.

As shown in Fig. 1B, the PPG signal exhibited complete synchronization with the ECG signal measured simultaneously. Moreover, the Bland–Altman plot demonstrated that all data points were within 95% limits of agreement (Fig. 3D), and the Pearson's correlation coefficient (r) between PPG and ECG reached 0.999999 (Fig. 3A). During the 5 min measurement, PPG data fluctuated in tandem with ECG, displaying a slight decreasing trend (Fig. S1). We attributed this to an accumulated effect of anesthesia, whereby, as the measurements progressed, the concentration of anesthetic within the fish's body exceeded the initial concentration, leading to a slight decrease in heart rate. Research conducted by Casselman et al. (2012) demonstrated a decrease in fish heart rate with an increase in anesthetic levels. These results corroborate the high accuracy of the reflectance-type PPG we developed. Yoshida et al. (2009) measured the heart rate of goldfish simultaneously with transmittance-type PPG and ECG, which also showed complete synchronization between the PPG and ECG signals. In fact, the waveforms of reflectance-type PPG and transmittance-type PPG are inverted relative to each other (Saritas et al., 2019). Therefore, if the PPG waveforms in the study of Yoshida et al. (2009) were inverted, the results would be almost identical to ours. Moreover, comparable results were observed in studies utilizing implantable PPG in fish (Svendsen et al., 2021a,b).

In this study, a 940 nm infrared emitter was utilized. Theoretically, light with a wavelength of 940 nm can penetrate human tissue to a depth of 5 mm or more (Finlayson et al., 2022). The PPG measurement position, as shown in Fig. 1A, and its surface is about 4–5 mm away from the heart. This distance allowed the infrared light to potentially reach the heart, leading us to presume that the PPG signal obtained was affected by the motion of the heart. The pulse transit time (PTT) was assessed by calculating the time interval between the R wave of the ECG and the P wave of the PPG here (Fig. 1B). The average PTT within a 60 s period for all individuals was a maximum of 100.13±7.67 ms (Table S2). There were no reported data on PTT associated with ECG and PPG in fish, making it difficult to evaluate whether the observed PTT duration was long or short. Consequently, relying solely on PTT duration made it difficult to speculate whether our PPG signal accurately captured the motion of the heart. Further studies are needed to confirm whether our PPG signal indeed reflects cardiac activity. Furthermore, in human studies, factors such as heart rate, blood pressure and anesthesia could affect the PTT of PPG signals (Drinnan et al., 2001; Ye and Jeong, 2010; Mukkamala et al., 2015). Moreover, in our research, PTT also exhibited variation over time, as indicated in Table S2.

For rPPG, as depicted in Fig. 1D, the waveform exhibited good synchronization with ECG. However, unlike the PPG waveform, the rPPG waveform appeared smoother and lacked the dicrotic notch (Fig. 1B,D). This observation aligned with the findings of Das et al. (2022), which also indicated that the rPPG waveform was smoother compared with the PPG waveform. This could be attributed to the potential filtering out of certain signals during the heart rate extraction process. The Bland–Altman plot results between rPPG and ECG showed that almost all data points were within 95% limits of agreement, regardless of natural light conditions or green light conditions (Fig. 3E,F). Furthermore, the Pearson's correlation coefficient (r) between rPPG and ECG was 0.946669 under natural light conditions and 0.965625 under green light conditions; MAE was 0.964 beats min⁻¹ under natural light conditions and 0.634 beats min⁻¹ under green light conditions; RMSE was 1.30 beats min⁻¹ under natural light conditions and 0.810 beats min⁻¹ under green light conditions (Table S1). These results indicate that rPPG exhibited high accuracy.

During the 5 min measurements, the rPPG data were identical to the PPG data in that they fluctuated with the ECG and exhibited a slight decreasing trend (Fig. S1). Currently, there are no reports on rPPG in fish, as the focus has primarily been on the imaging photoplethysmography (iPPG) method. In fact, the principles of rPPG and iPPG are similar, as both involve heart rate measurement using video or images (Verkruysse et al., 2008; Pylatiuk et al., 2014). iPPG research in fish has mainly focused on the embryonic stage (Pylatiuk et al., 2014; De Luca et al., 2015; Martin et al., 2019; Machikhin et al., 2020, 2022), achieving high accuracy with a Pearson's correlation coefficient between measured and true values of over 0.97 (Pylatiuk et al., 2014; Martin et al., 2019). This technique has been also applied to adult zebrafish but with relatively lower accuracy, as observed in the study by Mousavi and Patil (2020), where the Pearson's correlation coefficient ranged from 0.88 to 0.95. Therefore, compared with iPPG studies conducted during the embryonic stage, the accuracy of our proposed rPPG method was lower but still relatively high.

We also compared our findings with studies conducted in humans. Song et al. (2020) provided a summary of statistical parameters for various rPPG methods in human studies, reporting a range from 0.50 to 0.97 for Pearson's correlation coefficient, 1.53 to 14.86 beats min⁻¹ for MAE and 3.23 to 20.09 beats min⁻¹ for RMSE. In comparison, our results exhibited a relatively higher Pearson's correlation coefficient and lower MAE as well as RMSE. These differences may be attributed to the limited number of fish (only 6) used in our experiments, as opposed to the several hundred videos analyzed in human studies. Additionally, the more consistent conditions of our video recording may have contributed to these differences, in contrast to the varying recording conditions present in human video datasets.

Furthermore, as shown in Table S1, the Pearson's correlation coefficient (r) under green light conditions was higher compared with that under natural light conditions. Additionally, MAE and RMSE under green light conditions were lower than those observed under natural light conditions, indicating that the accuracy of the rPPG method could be improved by offering green light. This improvement may be attributed to the presence of hemoglobin in the blood of large yellow croaker (Gu and Xu, 2011), which strongly absorbs green light (Roggan et al., 1999). The green light-enhanced condition facilitated increased absorption of green light by the blood, thereby enhancing the rPPG signal. Studies conducted in humans have also shown that green light provides the most significant improvement compared with fluorescent light (Lin and Lin, 2017). Consequently, the active provision of green light can be employed to enhance rPPG signals as needed. Although our rPPG method demonstrated promising results through frequency domain analysis, noticeable noise persisted in the time domain results, highlighting a need for improving the signal-to-noise ratio. Future studies could benefit from incorporating emerging technologies such as deep learning and convolutional neural networks (Niu et al., 2018; Song et al., 2020).

Overall, PPG demonstrated higher accuracy than rPPG, as evidenced by its greater agreement (Fig. 3D–F), higher Pearson correlation coefficient (Fig. 3A–C) and lower MAE and RMSE values (Table S1). However, rPPG was capable of estimating heart rate from a greater distance away from the fish. In practice, the most challenging aspect of developing the methods was not improving the sensor or algorithm, but rather identifying the location with the strongest heart rate signal. Once the location was determined, obtaining the heart rate signal using either sensors or algorithms became relatively straightforward. Therefore, when applying this method, it is essential to first determine the measurement location and subsequently adjust the device or algorithm accordingly. In cases where fish with higher heart rates or darker ventral colors are being measured, increasing the light source intensity or raising the data sampling rate and video frame rate may be appropriate.

We thank Prof. Caihuan Ke for providing instrumentation assistance, which was the data acquisition system (Powerlab, ADInstruments).