Multichannel stroboscopic videography (MSV): a technique for visualizing multiple channels for behavioral measurements

The visualization of multiple channels of spatial information is common to numerous fields of biological study. Multichannel visualization is often associated with fluorescence microscopy, where distinct channels may be recorded by using different fluorophores (e.g. Parak et al., 2005; Miyawaki et al., 2003; Carlsson et al., 1994). Individual channels are visualized with a single camera by changing the filter sets in a light path that are specific to each fluorophore. Overlaying these channels into a single image provides the ability to map disparate types of information offered by each channel (e.g. gene expression, ion concentration, mechanical stresses) with respect to an organism's body. In this way, multichannel visualization provides a powerful means for experimental inquiry. However, the leverage gained by a multichannel approach has largely eluded studies of animal behavior owing to the relatively rapid motion of the subject. Here, we present a technique called multichannel stroboscopic videography (MSV) that permits multichannel visualization for behavioral experiments with a single camera. MSV operates through the use of strobe lights that are synchronized with respect to the exposures of a video camera. Each set of lights provides illumination that is specific to an individual channel in a recording. Channels are recorded during distinct video frame exposures. We illustrate the utility of this technique by measuring (1) two channels of kinematic data in a walking cockroach and (2) the flow field and kinematics in separate channels for a swimming zebrafish.

Our measurements of a walking cockroach were intended to demonstrate the general approach and value of MSV for automated kinematic analysis. Cockroaches are a model system for the neuromechanics of locomotion and studies on this insect can include simultaneous measurements of the animal's body and where its feet contact the ground (Kram et al., 1997; Schaefer and Ritzmann, 2001; Watson and Ritzmann, 1997; Full and Tu, 1990) that are acquired by established methods. As detailed in the Materials and Methods, we used MSV to independently optimize the illumination of the body in one channel and the feet in another. The contrast for both the body and feet was sufficient to automate the acquisition of coordinates for both features without the use of synthetic markers. This approach could be applied to other such situations where two or more sources of illumination provide high contrast for distinct features of an organism. Automated kinematic analysis allows for high-throughput behavioral experimentation.

Our experiment on a swimming zebrafish was designed to evaluate the use of MSV to generate two distinct types of data. One channel recorded the flow field around the fish using digital particle image velocimetry (DPIV) and the other tracked the peripheral shape of the body. In addition to the benefits of correlating measurements from the two channels, measurements of the fish's body were also incorporated in our analysis of DPIV data. In particular, automated tracking of the body allowed for the definition of a dynamic mask to reject erroneous velocity vectors. In addition, the position of the fluid–body interface was necessary to calculate the pressure field using a previously described method (Lucas et al., 2017; Dabiri et al., 2014). The ability to automate measurements of both the animal's body and flow field illustrates the powerful capacity of MSV for high-throughput experimentation with a complex hydrodynamic analysis.

MSV requires an ability to synchronize sources of stroboscopic illumination with respect to the frame exposures of a video camera. We used a high-speed video camera (FASTCAM Mini AX100, Photron, San Diego, CA, USA) with high spatial resolution (1024×1024 pixels) configured with a macro lens (Micro-Nikkor 105 mm f/2.8, Nikon Inc., Melville, NY, USA), appropriate for recording small animals (Fig. 1). This camera was configured to have the timing of exposures dictated by the rising edge of an external 5 V square-wave control signal of variable frequency. This signal was provided by a 2-channel arbitrary waveform function generator (DG1022Z, RIGOL Technologies, Beaverton, OR, USA), which also generated a second 5 V control signal for the lights at half the frequency of the camera's control signal. The signals used to operate the camera and light sources were synchronized using the ‘align-phase’ feature of the function generator and verified with a multi-channel digital oscilloscope (DS1054Z, RIGOL Technologies, Beaverton, OR, USA). When recording at high speed, MSV requires lights such as LEDs that have a nearly instantaneous response to a power control signal. A consequence of MSV is that the different channels generate measurements that are separated in time, which requires interpolating the data in post-processing (explained below).

We used MSV to measure the kinematics of the body and feet of a cockroach in separate channels. To visualize the body, we used an infrared (IR, 940 nm) LED panel placed above a translucent white acrylic diffuser, which was placed above the experimental tank (Fig. 1A). This generated a high-contrast image of the body with transmitted illumination. A circuit (Fig. S1) delivered power to this light when a control signal was set to 0 V and no power at 5 V. We visualized where the feet of the cockroach contacted the floor of the tank using a second light that consisted of a strip of white LEDs aligned with the edge of the floor. The floor was composed of a clear sheet of acrylic (30.5 cm×30.5 cm×1.3 cm) and the strip of LEDs generated an evanescent field by partial total internal reflection (Martin-Fernandez et al., 2013) above the acrylic surface. When the legs of the cockroach contacted the surface, light within the evanescent field was reflected and imaged from below. The strip of white LEDs was powered throughout the experiment, but their low brightness was barely visible when the IR light source was activated to visualize the cockroach's body. As a consequence, video recordings consisted of alternating high-contrast images suitable for automating the body and footfall kinematics of a walking cockroach in separate channels.

Our experimental setup for zebrafish allowed for simultaneous recordings of the fish's body and the surrounding flow field. The body was visualized with the same IR LED panel and diffuser (Fig. 1B) as used for the cockroach experiment. The IR LED panel was oriented below the experimental tank (7.5 cm×7.5 cm) and the camera was placed above to visualize the fish as a high-contrast silhouette from a dorsal perspective. We visualized flow by splitting a laser beam (2 W DPSS, 532 nm wavelength, Laser Quantum, San Jose, CA, USA) into two light paths, one of which was reflected upon two mirrors, and passing each path through sheet generator optics (Fig. 1B). This arrangement generated a plane of light parallel to the focal plane of the camera from two perpendicular sources. The laser sheet illuminated reflective particles (industrial diamond powder, 3–6 μm, Lasco Diamond Products, Chatsworth, CA, USA) mixed at a concentration of 0.0056% by weight in the water contained within the tank, which was filled to a depth of 2.5 cm.

The two different single-wavelength light sources used in our fish experiments created a chromatic aberration. Most lenses have different refractive indices for different wavelengths of light. As a consequence, a lens focused on a subject for one color will be out of focus for a different color. Owing to this effect, we optimized the setup for increased depth of field by increasing the intensity of the IR LED panel and the gain on the camera so that we could set the lens with a higher f number and hence smaller aperture. In addition, we focused our lens on the laser sheet because DPIV results were found to be more sensitive to focus.

The fish's body and flow field were independently recorded at high speed (Fig. 2). The IR LED panel and laser received a common 500 Hz control signal from the function generator. We modified the external control hardware of the laser to emit only when this signal reached 5 V. In contrast, the IR LED panel emitted light at 0 V and hence out of phase with the laser. The other channel from the function generator synchronized frame exposures with a 1000 Hz control signal. Thus two channels of alternating video frames were recorded for dynamic boundary tracking and flow visualization.

Experiments were performed on a single fish and one cockroach. The zebrafish [Danio rerio (Hamilton 1822); 120 days post-fertilization, 18.9 mm standard length] was maintained in a recirculating freshwater system at 27°C on a 14 h:10 h light:dark cycle. The cockroach [Gromphadorhina portentosa (Schaum 1853), 6.83 g] was obtained from a laboratory colony maintained at 24°C under a 12 h:12 h light:dark cycle. Both animals were transferred to their experimental tank and allowed to acclimate for at least 10 min prior to filming. Spontaneous behavior was recorded for both animals. For the fish, we stopped recording and saved the sequence to disk when the animal executed a turning maneuver within the horizontal plane of the laser sheet. All experiments on the fish were conducted in accordance with the University of California, Irvine's Institutional Animal Care and Use Committee (Protocol #AUP-17-012).

Our recordings of the cockroach were analyzed to automate tracking of both the body and feet. This procedure, and all data analyses, were performed by programming within MATLAB (v.2014b, MathWorks, Natick, MA, USA). Our program employed the image processing task generally known as blob analysis, which requires the conversion of grayscale images into binary images by defining an intensity value that separates dark and light pixels. This image segmentation technique, known as thresholding, generates ‘blobs’ of connected pixels from which features (e.g. centroid and area) may be calculated. The program first identified the frames for the body by the relatively high mean pixel intensity generated by the IR LED panel (Fig. 3A). This initial detection procedure allowed us to input the entire video without imposing a specific ordering to the frames and without manually separating frames into subdirectories. After thresholding, the cockroach's body was distinguished as the largest blob in each frame and its centroid position was recorded. We performed a similar operation to identify the illuminated points of contact for the feet from the darker video frames. The more subtle contrast of these images was improved by subtracting a time-averaged image to remove imperfections in the surface of the floor. We combined the coordinates from both channels by linear interpolation (‘interp1’ function in MATLAB) of the body position coordinates at the same time points for which we obtained measurements of the feet. The final result was a kinematic dataset for the body and feet that was obtained automatically through two channels.

One of the channels for our fish experiments similarly allowed for body tracking. Our program automatically tracked the boundary and midline of the dark fish body from the video frames illuminated by IR light (Fig. 3B). We applied a local thresholding technique (‘adaptthresh’ function in MATLAB) that is robust to nonuniform illumination and can be used with or, as implemented here, without an input reference image for background subtraction. The resulting binary image described the shape of the fish's body, which we refined with morphological operations to fill holes and connect any gaps with neighboring blobs. The fish blob was manually selected in the first video frame and subsequently identified by its area and proximity to the prior frame's blob. This blob was used as our dynamic mask for DPIV analysis and served as the basis for kinematic measurements. For each frame, we measured the blob's area and identified its center-of-area, boundary and midline. The midline was identified by distance mapping, which encodes a value for each pixel according to its closest proximity to the blob's edge. We applied distance mapping along the rows and columns of the binary image and the resulting maps were concatenated to produce the set of pixels that define the midline (Fig. 3B, right column). For kinematic analysis, the raw midline coordinates were smoothed with the ‘loess’ method, a locally weighted polynomial regression.

The DPIV flow fields were analyzed to estimate the pressure field around the swimming fish (Fig. 3C and Fig. S2). We analyzed our recordings of particle motion using an open-source MATLAB application PIVlab (Thielicke and Stamhuis, 2014). This software was configured for a direct Fourier transform correlation with three passes and 50% window overlap. We decreased the interrogation window sizes (64×64, 32×32 and 16×16) in each pass, which resulted in a 128×128–velocity vector field. We modified PIVlab to accept the dynamic mask that we identified from our blob analysis (Fig. 3B). A linear interpolation of the flow fields with respect to time was performed to estimate the flow field for the same instants of time for which we recorded body kinematics. Pressure calculations were performed with the queen2 algorithm (Dabiri et al., 2014) with default settings. This algorithm directly computed the pressure gradient term in the Navier–Stokes equations along several paths and performed a median-polling scheme to estimate the pressure at each point. These calculations required the coordinates of the mask boundary for the fish's body from each frame.

Our experiments illustrate how MSV allows multichannel visualization for behavioral experiments. Channels of data were obtained from images of alternating sources of illumination in a single-camera video recording. The two channels of kinematics in our cockroach experiment were acquired from images that individually visualized the footfall pattern and body position through time (Fig. 3A). In our fish experiment, flow was visualized with a DPIV channel and kinematics were obtained through another channel. This kinematics channel was then used in the post-processing of the DPIV channel (Fig. 3B,C). We see the potential for broad applications of MSV in the study of animal behavior and engineering research.

MSV can enhance the automated acquisition of kinematic measurements. Automated analyses allow high-throughput data acquisition that is useful for expanding the size of a dataset and may facilitate applications such as behavioral mutant screens (e.g. Brockerhoff et al., 1995; Mirat et al., 2013). Machine learning and other frontiers in image processing offer opportunities for the development of sophisticated software for automating kinematic measurements (Colyer et al., 2018; Robie et al., 2017). However, a more direct and robust approach to automation may be obtained from video recordings of subjects that are illuminated with high contrast. Under two or more sources of illumination, each source may be optimized to enhance the contrast of a particular feature. In the case of the cockroach, the lighting conditions for the body and feet were optimized independently (Fig. 3A). Recording over two channels allowed these features to be visualized with sufficient contrast for automated tracking. This example demonstrates the value of MSV in allowing landmark tracking under two or more sources of illumination, which may be required in animal behavior research for which automated tracking methods are not well established. MSV could be extended to tracking multiple individuals in an experiment. For example, individuals marked with a UV fluorescent tag (Delcourt et al., 2013, 2011) could be identified under UV illumination in one channel and body kinematics could be recorded via transmitted illumination in another channel.

Our fish experiment demonstrates the utility of MSV in measuring flow around a moving body. DPIV operates by identifying the displacement of particles within an Eulerian system grid (Stamhuis et al., 2002; Adrian, 1991). Excluding the cells within an interrogation window that contain a body may be accomplished by dynamic masking. Dynamic masks may be isolated with image processing from a single-channel recording when the body is uniformly illuminated (Gemmell et al., 2016), but the lighting conditions that are amenable to recording particles are generally unfavorable for visualizing the body. As a consequence, the body is often manually drawn for each frame of the video (Gemmell et al., 2016; Tytell and Lauder, 2004), which is labor-intensive. As demonstrated presently, MSV provides the opportunity to resolve this challenge by producing images of the moving body under transmitted illumination that may be easily converted into a dynamic binary mask. This automated process has the potential for utility beyond biology and into areas of fluid dynamics research. When imaging a moving subject in flow, MSV can be used in place of image processing techniques for masking that requires a priori descriptions of the object's geometry and motion (e.g. Nikoueeyan and Naughton, 2018; Dussol et al., 2016). As an alternative to MSV, it is possible to use two sources of illumination and multiple cameras to simultaneously acquire the different channels (Adhikari et al., 2015). However, such a system results in higher cost and greater computational time for image registration of the different camera views relative to MSV.

MSV offers additional benefits in the post-processing of flow measurements for near-field analysis. After acquiring a velocity field, investigators are generally interested in extracting derived features from those data. Such post-processing has included measurements of variables describing the circulation and spacing of vortices shed in the wake of an animal (Drucker and Lauder, 2001; Müller et al., 2000). When conducted in the far field, such calculations may not require articulating the location of the body's surface. However, the body's boundaries are essential for calculating near-field phenomena, such as boundary layers and flow separation (Anderson et al., 2001). Measurements of the surface in these cases offers the same challenge as the dynamic mask used in acquisition. Again, MSV resolves this task by allowing for the automated extraction of a body's boundary in the flow field, as demonstrated in our calculation of pressure (Fig. 3C). The algorithm that we used, developed previously (Lucas et al., 2017; Dabiri et al., 2014), required identification of the boundary of the fish for each video frame.

There are limitations to MSV that are a direct consequence of using a single camera to collect multi-channel data. Consecutive images from a single light source (Fig. 2C) are offset by an inter-frame interval determined by the camera's frame rate, which effectively reduces the frame rate by half. This drop in temporal resolution can be ameliorated, to some extent, by using a high-speed video camera and sufficient lighting to illuminate the object as exposure time decreases. Another potential drawback is that MSV does not generate simultaneous multi-channel data. Many biological investigations benefit from correlation of simultaneous data from multiple sources (e.g. Venkatraman et al., 2010; Mead et al., 2003). The high temporal resolution of our data allowed us to interpolate the data of one channel at corresponding time points of the second channel. Similar post-processing is required to synchronize measurements from different channels.

The ability of MSV to acquire multiple channels of image data has potential applications in diverse areas of biological research. Multichannel visualization is common to fluorescence microscopy, where distinct channels may be imaged with different fluorophores (e.g. Parak et al., 2005; Miyawaki et al., 2003; Carlsson et al., 1994). Each fluorophore, visualized with a distinct filter set, can offer a channel that maps a type of information (e.g. gene expression, ion concentration, mechanical stresses) to an organism's anatomy and these channels may be combined in a single image. The use of transgenic lines of animals for the visualization of neuronal activity has become a popular optical approach to neurophysiology but generally requires a stationary subject (e.g. Bruegmann et al., 2015; McLean and Fetcho, 2011). Using independent frame exposures, MSV may permit visualization of tissues in a moving subject by combining a fluorescence channel with another light source to visualize the body.

In summary, MSV offers the opportunity for multichannel visualization with a moving subject. By separating the channels in separate exposures of a video recording, the illumination for each channel may be optimized for a particular source of information. Such conditions offer the promise of automated analysis and sophisticated post-processing. We feel that the cockroach and fish experiments presented here illustrate just some of the possibilities for incorporating this approach in experimental methods for research in biology and engineering.

We thank T. Bradley for advice on patent protection and the two anonymous reviewers for their feedback on an earlier version of this manuscript. Thanks to A. Carrillo, A. McKee, and A. Peterson for helpful discussions.