Automated hull reconstruction motion tracking (HRMT) applied to sideways maneuvers of free-flying insects

The swimming of fish and the flight of insects are impressive feats of nature. These locomotor displays are the collective result of genetic,evolutionary, neurological, sensorial and biomechanical influences and, when quantified, provide a window into the inner workings of these animals. Behavioral studies generally follow one of two approaches(Stephens et al., 2008). In the first, organisms are put in artificial environments and given a small number of choices in response to a stimulus, sacrificing richness of behavior for added experimental control. In the second, the animal is observed `in the wild', in which case a broad spectrum of behavior can be observed but not easily characterized due to limited measurement capabilities. For investigations of insect flight, this dichotomy is exemplified by studies of wing motions in tethered and wild insects. Tethered flight allows detailed measurement of kinematics but does not allow the studying of maneuvers. Field studies investigate more complex behaviors but with limited measurement and control capabilities. What is missing is an approach that would marry the beneficial aspects of these methods and would allow recording of complex behaviors while giving full access to the locomotor metrics. Such an approach would significantly increase the range of behaviors that can be investigated quantitatively.

State-of-the-art approaches to capturing the motion of locomoting animals involve high-speed, 3D videography combined with digitization of the captured sequences (Lauder and Madden,2008). Most current techniques for extracting 3D body and wing kinematics of flying animals rely on manual motion tracking. One approach involves positioning a computer model of an organism so that it overlays the image of the filmed organism (Fry et al.,2003; Fry et al.,2005; Liu and Sun,2008). Another method requires tracking the position of representative marker features on the organism through time(Jensen, 1956; Nachtigall, 1966; Zanker, 1990; Hedrick et al., 2002; Wang et al., 2003; Combes and Daniel, 2003; Russell, 2004; Hedrick and Daniel, 2006). Unfortunately, these techniques demand significant human input, resulting in poorly characterized or uncharacterized errors, limited throughput and red-eyed researchers. More automated methods, similar to those developed for motion tracking of cockroaches and fish(Revzen et al., 2005; Fontaine et al., 2007), require the development of morphologically appropriate wing and body models when applied to flying animals(Fontaine, 2008; Fontaine et al., 2009). Thus,there remains a need for accurate, automated and versatile methods that do not require morphological inputs.

The study of insect flight in particular stands to benefit from high throughput and accurate tracking techniques. Asymmetries in flight kinematics appear to be quite subtle, even for wing motions that bring about extreme maneuvers. For instance, it has been reported that fruit flies execute rapid changes in yaw, or saccades, by inducing differences between the amplitude of the left and right wings of about 5 deg. and shifting the stroke plane by about 2 deg. (Fry et al.,2003). Further exploration of the myriad maneuvers performed by insects will require large data sets that allow for identification of slight kinematic manipulations. In addition to addressing maneuverability, such data would offer insight into the roles of aerodynamics, efficiency, control and stability in insect flight (Wang,2005).

Here, we outline a novel approach to the motion capture of flying insects. Rather than restricting the flight behavior, we film the rich free-flight repertoire of insects. This, by necessity, sacrifices much of our control over the maneuvers the insects perform. However, by automating our apparatus and recording many such events, we can identify common strategies used in similar maneuvers. Most importantly, our experimental arrangement is designed to yield films that contain time-resolved, 3D information about the motion of the insect body and wings during flight. In this work, we focus on a novel motion tracking technique we term Hull Reconstruction Motion Tracking (HRMT). We demonstrate that subtle, yet statistically significant, differences in flight modes can be clearly discerned using this method. More specifically, we examine sideways flight of fruit flies, and show that the generation of lateral acceleration is associated with changes in the timing of the rapid flipping of the wings. Overall, this approach is a key step toward a quantitative description of the rich flight behavior of insects.

Generally, an analysis of insect flight requires a method for recording the flight events and a method for recovering the flight kinematics. Here, we describe techniques that allow for the automation of both high-speed videography and motion tracking.

We have assembled an automated, versatile system for capturing many video sequences of flying insects. The apparatus is composed of three high-speed cameras focused on a cubical filming volume contained within a large Plexiglas flight chamber (Fig. 1A). The cameras are orthogonally arranged using precision rails mounted on an optical table. We use Phantom v7.1 CMOS digital cameras (Vision Research, Wayne, NJ,USA) that are sensitive to visible light. We find that filming at 8000 frames s^–1 at a resolution of 512 pixels×512 pixels is a suitable compromise in temporal and spatial resolution. At this rate, we capture about 30–35 wing orientations per wing stroke of the fruit fly(Drosophila melanogaster), which beats its wings approximately 250 times per second. The cameras are event triggered, as described below, and are synchronized electronically. In our experiments, the cameras automatically save the images on an internal memory buffer. Once a recording sequence is finished, the cameras dump the images onto an external computer hard drive and then become available for recording more flight events.

Because fruit flies are small, measuring about 3 mm in body length, we magnify with an optical bellows (Nikon PB-5, Nikon USA, Melville, NY, USA) and a zoom lens (Nikon Macro-Nikkor, 28–105 mm) attached to each camera, as shown in Fig. 1A. The bellows can be expanded or contracted to achieve varying magnification and thus accommodate different-sized filming volumes. For D. melanogaster,typical cubical volumes described in this paper measure 1.5 cm in side length. This filming arrangement insures that perspective distortion between the near and far portions of the chamber is less than 5%.

Achieving crisp images of fruit flies in flight requires short exposure times (<30 μs), high magnification and large depth-of-field (high f-stop) values. These requirements all reduce the light available for filming. To avoid heating the filming volume, we use three slide projectors (Kodak Ektagraphic series, Kodak, Rochester, NY, USA) that provide infrared-filtered intense white light. Each lamp is directed toward its opposing camera, as in Fig. 1A. Thus, our films consist of silhouettes or shadows of the flying insect(Fig. 2A).

When released in the flight chamber, flies rarely enter the filming volume,which corresponds to only 0.05% of the chamber. To capture films only when a fly is in the volume, we assembled an optical detection system(Newman, 1982; Ellington, 1984; Ennos, 1989). A schematic diagram of this system is shown in Fig. 1B. A laser (red HeNe, Thorlabs, Newton, NJ, USA) emits a 2 mm diameter beam that is split and re-routed to intersect the filming volume through the sides of the flight chamber. Each beam passes through a Galilean expander, crosses the filming volume, and impinges on a photodiode. The photodiodes are connected to a custom-made switching circuit that signals to the cameras when the two beams are simultaneously intercepted. This triggers the cameras to initiate recording.

The beam expanders in our assembly allow us to match the triggering volume to the filming volume, thereby maximizing the number of captured flight sequences. This versatility also accommodates the filming of insects of varying sizes. We generally expand the beam to 1–2 cm in diameter. Since the fly body area is of the order of 2 mm², our circuit is designed to reliably trigger on beam intensity disturbances of only a few per cent.

In a typical experiment, we release between one and 20 flies in the filming chamber. When interested in flight statistics, we release up to 20 flies and film for up to 3 h. In these experiments we obtain up to 10 events per hour. The flight chamber measures 13 cm on each side, so the flies are more than 20 body lengths from the nearest wall, indicating that the walls have negligible influence on the aerodynamics.

In order to analyze the vast amount of data collected with our apparatus,we have developed a method for automatically extracting the wing and body positions from flight films. This method is accurate, fast, model independent and broadly applicable. Our tracking algorithm neatly divides into four steps:image processing and registration, hull reconstruction, `dissection' of the hull reconstruction into a body and two wings, and extraction of positions and orientations. We implement all stages using custom-written MATLAB code(available at http://cohengroup.ccmr.cornell.edu/).

Our hull reconstruction method requires crisp silhouettes of the flies and accurate registration of the pixels in the images. To achieve registration, we first precisely align the cameras by fine adjustment of translation stage mounts. This procedure positions the center of each camera view to within a few pixels of a common point in space and also establishes the global,orthogonal coordinate system employed throughout this work. The procedure achieves equal magnification to better than 1%, as measured by imaging a ruled microscope slide that also determines the pixel-to-distance conversion. Next,we use the images from each flight movie itself to more precisely adjust the alignment and magnification. To obtain image silhouettes, we first subtract a background image from each picture. The resulting image is thresholded so that the insect shadow appears black on an otherwise white background(Fig. 2A, bottom). In order to calibrate the pixels so that their coordinates are aligned and properly scaled, we use a registration algorithm. We first enclose the silhouette from each view in the minimal bounding rectangle(Fig. 2B). We then scale and translate the images so that the pixel coordinates of the bounding rectangle corners match. For example, to register the pixels along the vertical direction, we shift and scale images from one of the horizontal cameras such that its vertical coordinate is consistent with images from the second,reference horizontal camera. We vertically shift the image from the first view such that the top of its bounding rectangle has the same vertical coordinate value as that of the reference view. Then, we scale the image from the first view such that the bottom of its bounding rectangle has the same vertical coordinate as that of the reference view. The same procedure is used to register the other image coordinates. Typically we find that the images need to be scaled by less than 1% and shifted by about 5 pixels to achieve registration. To insure consistent registration for each movie, we find the average shift and scale values for the entire image sequence and apply these values to all images. The resulting thresholded and registered image sequences are fed into the hull reconstruction algorithm.

In the context of our experiments, the method of visual hull reconstruction(Baumgart, 1974) entails using the three sets of 2D silhouettes to construct a 3D shape. Specifically, our algorithm identifies volume pixels, or voxels, in 3D whose 2D projections map onto black pixels in all three images. More intuitively, this procedure is equivalent to the geometric exercise of placing the images on three adjacent sides of a rectangular prism and extending each shadow in a direction perpendicular to the image (Fig. 3A). Here, simple extension of each shadow is justified by the rather small perspective distortion. In this scenario, the hull volume corresponds to the intersection of the 3D extended shadows. An example of the resulting shape is shown Fig. 3B. This collection of voxels forms a convex volume that envelops the 3D shape corresponding to the real insect. We show that, by using three cameras to image the insect, we obtain a visual hull that is sufficiently close in shape to the real insect that wing and body positions and orientations can be extracted.

To identify the hull, the reconstruction algorithm must systematically scan through and analyze voxels in the filming volume. For typical images, the bounding rectangle side length is only one-fifth of the image side length. Consequently, this procedure is sped up 100-fold by only considering voxels corresponding to pixels located within the bounding rectangles(Cheung, 2003). In addition, we find that a coarse-graining optimization leads to an additional factor of four reduction in the run time (Cheung,2003) while maintaining accurate coordinate extraction, as assessed in the next section. This procedure entails grouping sets of 8 unit voxels into coarse-grained voxels each of size 2 pixels×2 pixels×2 pixels. A subsampling routine is used to determine whether the coarse-grained voxel should be included in the hull. Two of the eight voxels are randomly picked and analyzed to determine whether their projections are contained within all three shadows. If both sampled voxels correspond to shadows, the coarse-grained voxel is included in the hull. The final output of these procedures is a collection of coordinates specifying the coarse-grained voxels that are part of the visual hull (Fig. 3B). We find that, using MATLAB on a desktop computer, our implementation of this algorithm rapidly constructs a 3D hull (see supplementary code at http://cohengroup.ccmr.cornell.edu/).

Portions of the hull that correspond to the body, right wing and left wing form well-defined groups of voxels. To collect voxels that are near one another, we use a k-means clustering algorithm with a Euclidean distance metric (MATLAB, 2004; The Math Works Inc., Natick, MA, USA). We find that identifying four clusters (k=4) neatly isolates two separate groups of voxels corresponding to the left and right wings and two additional larger groups of voxels that correspond to the anterior and posterior of the insect body. These two larger clusters are merged to identify all the voxels corresponding to the body, and the smaller clusters correspond to the wings. In Fig. 4, the voxels corresponding to each of the body, right wing and left wing are shown in different colors in order to illustrate how well these groupings are identified.

From these voxel groupings, we recover the positions and orientations of the body and wings using a combination of centroid determination, principal components analysis (PCA) and geometric information about the insects. The centroids of the body, right wing and left wing correspond to the mean of the voxel coordinates in each grouping. PCA finds each voxel grouping orientation by determining the principal axis of the moment of inertia (MATLAB, 2004). Performing PCA on the body voxels, we extract the principal body axis vector, Â, that identifies the Euler angles for the body pitch, β,and yaw, ψ (Fig. 5A). To determine the third Euler angle for the body roll, ρ, we perform a second round of clustering on the body voxels with k=3 and find three clusters, corresponding to the head, thorax and abdomen(Fig. 5B). The centroids for these clusters constitute three points that define the plane of bilateral symmetry for the body. We take the roll ρ to be the angle between the normal vector to this plane, L̂, and

Because each wing is thin, rigid and often occluded in one camera view by the insect body, its visual hull resembles a parallelepiped whose long axis is parallel to the wing span vector, Ŝ. To determine Ŝ, we apply PCA to the wing hull voxels. This vector allows determination of the Euler angles for the stroke, ϕ, and deviation,θ (Fig. 5D). The hull cross-sections perpendicular to Ŝ form parallelograms(Fig. 5E, right). The wing chord vector, Ĉ, is parallel to the longer diagonal of the parallelograms. The third Euler angle for each wing is the pitch angle, η,and is defined to be the angle between Ĉ and the unit stroke vector, ϕ. To determine Ĉ, hull voxels near the mid-span(within 2 voxel side lengths) are projected onto a plane normal to Ŝ, and the chord is the vector connecting the two voxel projections having the greatest separation. The definitions of all wing orientation angles are detailed in Fig. 5F.

These procedures lead to a full kinematic description consisting of 18 coordinates: three centroid coordinates and three Euler angles each for the body, right wing and left wing. These coordinates are computed independently for each time step in the movie and checked visually for mistakes. Together,these techniques constitute a HRMT method for extracting 3D kinematics from several 2D views of a flying insect. While the data in this paper pertain to insect flight, this method is applicable with suitable modifications to a variety of 3D motion studies of other complex-shaped, moving objects in space.

Discerning subtle differences in flight modes requires clear knowledge of errors in the data recovery method. Such errors cannot be determined from the movies of insects alone, as the actual kinematics are not known beforehand. Instead, we estimate the measurement error by running HRMT on a computer-generated model insect and comparing the extracted positions and orientations of the body with those we impose. The model insect consists of five ellipsoids: three for the head, thorax and abdomen, and one for each wing(Fig. 6A). We orient the ellipsoids in a given configuration, use a ray-tracing algorithm to determine the three orthogonal shadows, and run our analysis routine to extract the positions and orientations of the body and wings. Compared with the model insect volume, the hull volume is larger and contains extra protrusions that vary in size and location for different insect orientations. These protrusions arise because of occluded regions that are blocked from the view of all three cameras. The protrusions ultimately cause errors in the recovered coordinates,and these errors depend on the orientation of the body and the positions and orientations of the wings. Thus, though validating HRMT using such simulated data does not account for image registration errors, it does account for occlusion defects, which appear to be the primary source of error for the method. However, determining the error dependence on all relevant coordinate variables is not feasible. Because fruit flies typically assume a limited set of orientations and use a typical wing stroke pattern for flight, we perform an analysis that determines errors for realistic insect configurations. Our synthetic data correspond to a fixed body and 34 wing configurations obtained by applying a manual tracking program to a single stroke from a movie of a hovering fruit fly. Our manual tracking software relies on overlaying images of a virtual fly and is similar to other implementations(Fry et al., 2003; Liu and Sun, 2008). We estimate errors for all wing positions within this stroke as well as the errors associated with viewing this stroke from different angles.

To obtain measurement errors for a typical viewing configuration, we fix the virtual fly body in an orientation of (ψ, β, ρ)=(0, 59, 0)deg. and plot the imposed time series data (open circles) and measured values(filled circles) for body and wing positions and orientation angles(Fig. 6C–F). The errors for each variable are concisely displayed as a histogram of the residual,defined as the difference between the measured value and the imposed value. For both the body and wing centroids, errors are within the coarse-grained voxel size of 2 pixels. The body orientation is also accurately recovered,generally to within a few degrees. The wing orientation angles and associated residuals for the right wing are shown in Fig. 6F, and the errors for the left wing have similar statistics. Errors for the wing orientations are typically under 5 deg.

The time series and residual data of Fig. 6 reveal several features of the hull reconstruction method. Most of our measurements average over the voxels in the hull and thus result in subvoxel resolution. Also, the residuals are nearly always centered on zero, indicating that there are only small systematic deviations. Further, the residuals have standard deviations of less than 2 pixels in the positions and 4 deg. in the orientations.

Furthermore, we find that in nearly all the cases we have examined, the mean residuals remain under 3 pixels and under 5 deg., regardless of both wing position during the stroke and viewing configuration. To summarize the dependence on wing position during the stroke, we plot the residuals forϕ, θ and η as a function of stroke angle in the body frame of reference, ϕ_b, for 16 different viewing configurations(Fig. 7). The configurations range in ψ from 0 to 45 deg., in β from 45 to 90 deg., and in ρfrom 0 to 60 deg. In total, this analysis comprises 544 different postures of the insect, and the use of a single wing stroke in the analysis is justified by the fact that the basic wing motion varies in subtle ways even during extreme maneuvers (Fry et al.,2003). The residuals show no obvious trend with ϕ_band all have standard deviations of less than 5 deg.

To summarize the dependence on viewing configuration, we plot the residuals averaged over an entire stroke as a function of body orientation (ψ,β, ρ) relative to the viewing configuration(Fig. 8). The residuals for the body and wing positions are all centered within two pixels of zero(Fig. 8A,C). With the exception of highly pitched (β ∼90 deg.) or highly rolled (ρ>15 deg.)body orientations, the residuals for body and wing orientation angles are also centered within 2 deg. of zero. The increased errors at high β and ρare not expected to affect most aerodynamic analyses as the fluid force is generated almost entirely by the wings, whose motions are accurately resolved. Thus, because HRMT tracks the body, right wing and left wing independently in the lab frame of reference, the accuracy of coordinate extraction in any one component is independent of any other. Collectively, these results indicate that HRMT is an accurate method for the motion capture of flying insects.

In order to show the utility of the HRMT method, we apply it to recorded maneuvers that exemplify how insects use lateral forces in flight. In particular, we emphasize aspects of these maneuvers performed by insects that differ from similar maneuvers performed by fixed-wing aircraft. In fixed-wing flight, lateral forces are usually generated by rolling or banking the aircraft and inducing a horizontal component to the lift force on the wings. This force enables an airplane or a helicopter to make a turn(Federal Aviation Administration,2001). Insects, on the other hand, could take advantage of the unique features of flapping flight to generate lateral motions. Here, we present an analysis that shows fruit flies can indeed manipulate their wing strokes to generate lateral forces in a manner that is different from simple banking. In addition, we propose a simplified mechanism describing how these kinematic manipulations contribute to the lateral force production. This mechanism takes advantage of several features of flapping flight, including the large arc-like trajectory of the wings and the independent control of the left and right wing.

In Fig. 9A,C, we show top views of trajectories of two maneuvers. In the first, a fly performs a `dodge'maneuver in which its yaw orientation remains nearly constant while it moves from one straight trajectory to another parallel trajectory(Fig. 9A). In the second trajectory, a fly performs a `sashay' maneuver in which it continuously reorients to face the inside of a turn(Fig. 9C). For this sashay, the body velocity is nearly perpendicular to the yaw direction of the insect. In both of these recorded maneuvers, we see that the insects undergo significant lateral acceleration; that is, the insects produce forces perpendicular to the body orientation in the xy-plane(Fig. 9B,D). See supplementary material for movies of these two maneuvers (Movies 1 and 2).

These forces originate from the detailed wing motions. In Fig. 10 we plot the stroke angle, ϕ, the deviation angle, θ, and the wing pitch angle, η, versus time for the left (blue) and right (red) wings throughout the maneuvers. The flapping wing stroke consists of an upstroke and a downstroke,which are separated by rapid flipping of the wing at stroke reversal. During the downstroke, the wings move roughly horizontally in the lab frame and toward the head of the insect, and during the upstroke the wings move backward. Thus, the motion of the wings is primarily back and forth, so ϕis a nearly sinusoidal function with high amplitude(Fig. 10A,E). Deviation from the horizontal is captured in the angle θ. Because the wings tend to rise slightly at both stroke reversals, θ has two peaks per wing stroke(Fig. 10B,F). Throughout this motion the wings also rotate about the span axis. The wing pitch angle, η,captures this rotation. During the downstroke, the wing moves forward andη is ∼45 deg. At stroke reversal, η rapidly increases to nearly 180 deg. During the upstroke, the wing moves backwards and η is ∼135 deg. Finally, at the rear stroke reversal, η rapidly decreases to nearly 0 deg. before returning to the downstroke angle of 45 deg.(Fig. 10C,G).

This general flapping and flipping motion is maintained throughout the flight for both maneuvers. We observe symmetrical wing motion when the fly undergoes no lateral acceleration, near t ∼0.075 s for the dodge and t ∼0.033 s for the sashay(Fig. 9B,D). When the fly accelerates sideways, however, asymmetries appear between the motion of the left and right wings. Maximal sideways acceleration is about 15% gfor the dodge and 40% g for the sashay, and all orientation angles exhibit measurable differences between the wings during this lateral force generation. These asymmetries lead to differences in both the trajectory of the wing tips and the wing angles of attack, α, an important variable in determining aerodynamic forces. We define α as the angle between the chord of the wing and the instantaneous wing velocity and calculate it from the wing orientation angles, (ϕ, θ, η). We plot α versus time for the right (red) and left (blue) wings for each maneuver in Fig. 10D,H. In general, the time course of α is marked by periods of relatively constant values near 45 deg. at mid-stroke punctuated by rapid increases and decreases as the wing flips at each stroke reversal. Just as for the orientation angles, we observe asymmetries in α for the left and right wings when lateral accelerations are large.

The generation of sideways forces can be rationalized by considering how differences in the motions of the right and left wings lead to asymmetric fluid forces. For example, in both maneuvers, when the fly generates rightward force, the left wing stroke deviation angle, θ_L, is greater than the right wing stroke deviation angle, θ_R. Likewise, for leftward accelerating flight, θ_R>θ_L. These observations are consistent with the generation of lateral force by sideways tilting the wing stroke planes, in much the same way as a helicopter executes a banked turn. In essence, the lift force, which is normal to wing velocity, is redirected to have a horizontal component. To estimate the magnitude of the lateral acceleration from the redirected lift, we make the approximation that the vertical acceleration is about g and this is redirected by an angle|θ _R–θ_L|/2. For the dodge maneuver, this calculation reveals that the redirected lift force accounts for about 8% g, or about half of the lateral acceleration. For the sashay, a similar calculation shows that lift accounts for 30% g, or about 70% of the lateral acceleration. These estimates suggest that the mechanism of lateral force production is not entirely due to the redirected lift force on the wings. An additional mechanism for producing lateral forces may be associated with the consistent asymmetries in the wing angles of attack.

Generating lateral forces from asymmetries in α can be understood by considering the time-lapsed top view images in Fig. 11A,B. In these images,the angle of attack is related to the projected area of each wing. As the wings are primarily moving in the horizontal plane, a large projected area in the top view corresponds to a low angle of attack and a small projected area is associated with a large angle of attack. The nearly horizontal, arc-like wing motion suggests that the drag forces, which act anti-parallel to wing velocity, have a significant lateral component. This is consistent with the fact that the wings sweep out a large arc in ϕ, and thus have a lateral component to their trajectories near stroke reversals. When the wings move symmetrically, these drag forces cancel out(Fig. 11A). For motions with asymmetric angles of attack near stroke reversal, the wing with the largerα generates a larger drag force. This imbalance in drag forces induces a lateral acceleration (Fig. 11B).

A schematic representation of the asymmetric wing motion is shown in Fig. 11C. The bottom image shows the fly at the beginning of a downstroke. As the wings begin to move forward, the projected area of the right wing is smaller indicating thatα _R is greater than α_L. This asymmetry results in a net drag force that points to the left. Similarly, leftward drag forces are induced near the end of the downstroke whereα _L>α_R, at the start of the upstroke whereα _R>α_L, and at the end of the upstroke where α_L>α_R. Remarkably, as the schematic diagram in Fig. 11D shows,this seemingly complicated sequence of events can be generated simply by having identical curves for α_L and α_R that are shifted in time. In fact, we do observe timing differences in the measured curves for α_L and α_R for both the dodge and sashay maneuvers (Fig. 10D,H). These observations indicate that such time shifts in wing rotation are important for lateral force generation.

To quantify this idea, we determine the time shift by calculating the correlation integral I(Δt)=∫⁰_Tdtα_R(t)α_L(t–Δt)over a wing beat period, T, and choosing the Δt that maximizes I(Δt). We plot lateral acceleration a versus the normalized Δt/T in Fig. 12 and find that these variables are strongly correlated and that larger time shifts correspond to more extreme lateral accelerations. Included in the plot are individual wing strokes from the dodge and sashay maneuvers discussed above, as well as kinematic data from three additional captured sequences of sideways flight. In total, over 70 wing strokes and 45,000 individual kinematic measurements were extracted. Remarkably, we find a strong overlap in the data for these maneuvers. This indicates that the timing difference between right and left wing rotation may be a general feature in the mechanism of lateral force generation of fruit flies.

Finally, we note that that the steep functional form of α near stroke reversal allows slight timing differences to generate large differences in the angle of attack. For example, in the dodge maneuver, a time shift of 0.1 ms(2% T) is associated with an instantaneous angle of attack difference of up to 20 deg. In the sashay maneuver, a time shift of 0.5 ms (10%T) corresponds to an α difference of up to 60 deg. This suggests that lateral forces are particularly sensitive to slight manipulations of wing rotation timing.

We introduce here a new method called Hull Reconstruction Motion Tracking(HRMT) for automated, fast and accurate extraction of kinematic data from films of flying insects. In particular, we show that with appropriate morphological considerations, three camera views of each flight event are sufficient for extracting the full wing and body kinematics. Our implementation of HRMT is a unique form of motion tracking that combines and builds on image registration, hull reconstruction, clustering and several geometric and analytical techniques. The main source of error associated with the technique arises from regions outside the fly that are included in the hull because they are blocked from all camera views. Despite these occluded regions, we find that when we test the accuracy of this method by running the algorithm on synthetic data, errors are very small, of the order of 1–3 pixels for centroid positions and 1–5 deg. for the orientation angles.

The HRMT system has many directions for future improvements. For example,increasing the number of viewing directions will increase accuracy. Currently the analysis does not make use of intensity differences that can be used to differentiate between various components of an object. The analysis algorithms can be sped up through optimization, exporting portions of the code to programming languages which are faster than MATLAB, and making the code parallel. Also, our implementation does not resolve the roll of the body well,a notoriously difficult task due to the symmetry of the insect body. To better resolve roll, HRMT may be supplemented with marker-based feature tracking or with the imposition of a morphologically appropriate body model(Fontaine, 2008). Further, our current implementation of HRMT uses a simple image registration procedure that takes advantage of the orthogonal filming arrangement and low distortion due to perspective. Calibration of images from more general camera orientations and larger distortions due to perspective would require the use of more general photogrammetric techniques, such as the Camera Calibration Toolbox available for MATLAB (MATLAB, 2004). The small errors associated with HRMT for coarse-grained reconstruction also suggest that our method will remain accurate for arrangements that would require such modifications to the registration procedure. Finally, our current implementation does not quantify wing deformations. For D. melanogaster such deformations are small. We estimate that the wing camber is largest at stroke reversal and measures about 15%. Such deformations, however, are known to be significantly more prominent in larger insects and are important for understanding aeroelasticity(Combes and Daniel, 2003). In order to adapt HRMT to aeroelastic studies, the implementation described in this paper could be combined with other photogrammetric techniques in order to better resolve such deformations (Walker et al., 2008).

Overall, however, HRMT offers several improvements over present motion-capture techniques. Automation eliminates the need for a researcher to manually perform motion tracking. This allows errors to be characterized in a reliable way, and we show that these errors are small and generally have no systematic dependence on relevant variables. Also, our implementation is fast,easy to apply, and not memory intensive; it can be run on a commonly available personal computer. This allows for rapid extraction of flight data and determination of statistically significant trends. Because the kinematic data are measured entirely in the lab frame of reference, the recovered coordinates are also directly suited to aerodynamic analyses such as computational fluid flow solvers or numerical force models. Finally, HRMT is versatile and may be readily modified for other locomotion studies in which the motion of many components is important.

To illustrate the utility of this technique, we use the HRMT method to perform a comprehensive analysis of sideways flight maneuvers of fruit flies. Because our automated filming apparatus was used to capture hundreds of free-flight movies, we were able to then select five films showing unambiguous sideways flight. The HRMT method was used to automatically recover 45,000 kinematic measurements for over 70 wing strokes. By having access to all of these data, we show that flies are able to generate lateral forces in a manner that takes advantage of the unique features of flapping flight. In particular,we show that sideways-flying insects induce differences in the right and left wing angles of attack near stroke reversal. Based on these data, we propose a model for generating lateral forces by accounting for unbalanced drag due to the difference between the wing angles of attack. These differences lead to a`drag ratcheting' mechanism in which drag force asymmetries give directed sideways motion. Our simplified model predicts that asymmetries in the drag forces can be generated by having identical curves for α_L andα _R that are shifted in time relative to one another. This mechanism is consistent with measurements in dynamically scaled flapping wing experiments showing that drag is extremely sensitive to the timing of wing rotation at stroke reversal (Dickinson et al., 1999). To test this model, we use the HRMT method to analyze many fruit fly wing strokes associated with different values of lateral acceleration. We find that there is a strong correlation between the measured lateral acceleration and the measured time shift between the curves forα _R and α_L(Fig. 12). These observations indicate that free-flying fruit flies alter wing rotation timing during maneuvers. This manipulation may be actively controlled by steering muscles(Dickinson et al., 1993) or passively influenced by fluid, inertial or elastic forces(Bergou et al., 2007). Future studies may elucidate the fluid force generation mechanism in more detail,perhaps using dynamically scaled experiments (e.g. Dickinson et al., 1999), fluid force models (Berman and Wang,2007) or computational fluid dynamics algorithms(Xu and Wang, 2008). Irrespective of the detailed force mechanism, our free-flight data suggest manipulation of wing rotation timing is a robust way to control forces during flapping flight. Exotic aerial maneuvers might be implemented in flapping,flying robots using such simple actuation strategies.

Because we present HRMT as an automated alternative to manual tracking techniques, it is important to compare the two approaches. To make this comparison, we first designed a graphical user interface program in MATLAB that requires the user to position a model insect so that its shadows overlie the movie images of the actual insect. Our manual tracking program is similar to that of other groups (Fry et al.,2003; Liu and Sun,2008). We then performed manual tracking for over 200 frames of a flight sequence. The particular flight sequence captures the dodge maneuver that is discussed in detail above. We also ran HRMT on the same frames and computed the differences between the coordinates extracted by each method(Fig. A1). In general, we find a strong similarity in the two methods, with all mean differences in position coordinates being less than 8 pixels and all mean differences in angular coordinates being less than 5 deg. There are small, but systematic differences in the two methods. The differences are probably due in part to inaccuracy in the morphology and connectivity of the model insect needed for manual tracking. There may also be additional occlusion errors in the HRMT method due to morphological differences between the real and model flies. Nonetheless,the similarity of the results obtained by HRMT and by the manual tracking program suggests that both methods are capturing the key features of the wing and body motion.

LIST OF ABBREVIATIONS

 
• a

  lateral acceleration

 
• Â

  body axis unit vector

 
• Âxy

  body axis projection onto xy-plane

 
• Ĉ

  wing chord unit vector

 
• g

  gravitational acceleration

 
• HRMT

  hull reconstruction motion tracking

 
• L̂

  body lateral unit vector

 
• PCA

  principal components analysis

 
• Ŝ

  wing span unit vector

 
• Ŝxy

  wing span projection onto xy-plane

 
• t

  time

 
• Δt

  right/left wing rotation timing difference

 
• T

  wing beat period

 
• v̂

  wing velocity unit vector

 
• (x,

  y, z) position of body or wing centroid

 
• α

  wing aerodynamic angle of attack

 
• β

  body pitch angle

 
• \({\hat{{\beta}}}\)

  body pitch angle unit vector

 
• η

  wing pitch angle

 
• θ

  wing stroke deviation angle

 
• \({\hat{{\theta}}}\)

  wing stroke deviation unit vector

 
• ρ

  body roll angle

 
• ϕ

  wing stroke angle

 
• \({\hat{{\phi}}}\)

  wing stroke unit vector

 
• ϕb

  wing stroke angle in body frame

 
• ψ

  body yaw angle

 
• \({\hat{{\psi}}}\)

  body yaw unit vector