The influence of visual landscape on the free flight behavior of the fruit fly Drosophila melanogaster

An important goal of neuroethology is to determine how complex patterns of behavior emerge from the interactions between an animal and its environment. In the most general terms, what we recognize as behavior results from a continuous feedback loop in which an animal’s actions influence what it experiences, and the resulting change in sensory input modifies its motor output. Because of their small size and the wealth of studies on their sensory-motor physiology, flies are an excellent model system for studying the complex feedback between an animal’s motor behavior and its sensory world. Nearly anywhere in the world, without much effort, you can probably find a fly buzzing around in a seemingly random fashion. While appearing stochastic, the complex flight trajectory of the fly must ultimately emerge from the interactions among its sensory systems, its motor system and the local environment. The purpose of this study is to investigate how the visual patterns that a fly encounters as it moves through a complex landscape determine its flight behavior.

As a fly moves through its environment, images move across its retina and generate complex patterns of optic flow. A fly can use estimates of these flow patterns to provide information about its own motion, to discriminate objects from background and to determine the relative distance of objects (Collett and Land, 1975; Egelhaaf et al., 1988; Srinivasan, 1993; Srinivasan et al., 1999). Previous studies have demonstrated that the flight trajectories of many fly species consist of straight flight sequences interspersed with rapid changes in heading termed saccades (Collett and Land, 1975; Schilstra and van Hateren, 1999; Wagner, 1986). While several sensory control models have been proposed for the straight flight segments (Collett, 1980; Wolf and Heisenberg, 1990), the sensory stimuli responsible for initiating saccades are not known.

During straight flight, there is a focus of expansion within the fly’s visual field where image velocity is zero. Optic flow radiates from this point. Nearer objects move faster across a fly’s retina than those farther away. Simultaneous rotation and translation create optic flow fields that are more difficult to interpret. Thus, maintaining straight flight and minimizing rotation are important goals of the flight control system (Collett et al., 1993). Tethered flies turn reflexively in the same direction as any perceived large-field rotation of visual space in an attempt to reduce retinal slip (Götz, 1968, 1975). Flies are thought to rely upon this so-called optomotor response to correct for horizontal deviations from straight flight. A similar reflex, mediated by the detection of visual motion in the vertical direction, stabilizes altitude (David, 1979, 1984; Wehrhahn and Reichardt, 1975). Both responses are thought to operate via linear negative feedback systems in which motor output is inversely proportional to features of visual input, such as large-field image velocity.

In contrast, the saccades are rapid, intermittent events that presumably cannot be represented by a simple linear transformation of a sensory input. It is more likely that specific features within the fly’s visual world trigger the all-or-none events. One possibility is that saccades are triggered by looming objects, similar to the stimuli that evoke landing responses (Borst, 1990). By reconstructing the visual inputs and estimating the optic flow experienced by freely flying Drosophila melanogaster, we attempt to identify the visual computations that act to trigger saccades as the animal flies actively through its environment. The results show that classic linear models of flight control based on optomotor equilibrium cannot account for the behavior of freely flying flies under our experimental conditions.

Free flight behavior was video-tracked within an arena made from a translucent white acrylic cylinder 1 m in diameter and 0.6 m high (Fig. 1). Infrared illumination, to which the flies are insensitive (Hardie, 1985), was used to avoid interference with visually mediated aspects of the flies’ behavior. The floor of the arena was covered with infrared-absorbent flock paper (Edmonds Scientific, J54-853), and the top rim of the arena was lined with 200 infrared diodes. For trials with a textured visual background, we lined the arena with a black-and-white 72×14 random checkerboard pattern. Each square would subtend a 5°×5° portion of visual space from the center of the arena and was colored either black or white with a probability of 0.5. The arena was lit externally by a ring of eight incandescent lights controlled with a dimmer such that illumination within the arena ranged from 10 to 14 cd m^–2 for both the textured and the uniform backgrounds. A black curtain running from the upper rim of the cylinder to the ceiling prevented any view of the laboratory environment.

Within the curtain, two video cameras, separated by 57 cm, were suspended 130 cm above the arena, each at an angle of 15° from the vertical (30° relative to one another). The cameras were synchronized using pulses generated by a pulse generator (BSG-50, Horita). Flies were tracked at a rate of 30 frames s^–1. Digital signal processors (DSP-2000, Dage-MTI) subtracted, frame by frame, a static background image from the live video signal, causing the fly to appear as a bright spot against a black background. The images produced by the DSP were recorded using two video cassette recorders (VCRs). During acquisition, a time-code generator (TRG-50, Horita) stamped an identical time code on the audio track of each tape. When digitizing the images from the tapes, the software (Adobe Premiere) was able to read the time code from each of the tapes to ensure that the digitized frames from each camera were aligned.

The location of the centroid of the fly was determined in each frame. To convert from two two-dimensional images to three-dimensional spatial coordinates, a calibration was required. This calibration was performed using a 38 cm×38 cm×62 cm cube with 13 light-emitting diodes (LEDs) whose exact spatial positions were known. A transformation matrix based on the positions of the LEDs in each of the two-dimensional frames and their three-dimensional position was calculated using software written using Matlab (Mathworks). This transformation matrix was used to convert the centroid position in each of the two-dimensional frames to a three-dimensional spatial coordinate. The translational velocity of the fly was determined from the distance the fly covered in the x,y plane between samples. The change in the fly’s altitude between frames will be referred to as the vertical velocity. Because of the small size of Drosophila melanogaster and the large field of view monitored by the cameras, we were unable to determine the exact angular position of the fly’s body. Thus, to estimate the heading of the fly and its angular velocity about the yaw axis, we assume that the horizontal projection of the longitudinal axis of the fly is a tangent to the flight path at all times, which is equivalent to assuming no side-slip and no yaw relative to the flight path. While this assumption would be largely invalid under field conditions in the presence of wind, it may serve as a fair approximation under laboratory conditions in still air. Further, while the presence of moderate side-slip and yaw would quantitatively alter some of our analyses, they would not effect the primary conclusions.

All experiments were performed on 2- to 4-day-old fruit flies, Drosophila melanogaster Meigen, from a laboratory culture descended from 200 wild-caught females. Flies were starved (to motivate longer flight sequences) and adapted to the light level of the flight arena for 6 h prior to each experiment. Experiments with a textured background were performed using 22 females and 18 males. Experiments with a uniform background were performed with 25 females and 21 males. Some flies provided more than one trajectory. We filmed the flies in the flight arena one at a time, with data collection terminating after the fly had landed on either the floor or the walls of the arena.

The fly’s three-dimensional trajectory was smoothed to remove digitization errors using a fifth-order low-pass Butterworth filter with a cut-off set at 15 Hz, half the video sampling rate. The trajectory was then upsampled by a factor of 5 (to 150 Hz) using a cubic spline interpolation. The angle that the walls of the arena would subtend along the fly’s retina was then calculated assuming (i) that the fly’s body and head were yaw-stabilized with respect to its flight trajectory, and (ii) that the horizontal plane of the head was aligned parallel to the ground. These assumptions were necessary given the lack of information about the fly’s body position and the position of the fly’s head relative to its body. The angle the walls subtended across the fly’s retina was calculated by projecting the image of the walls onto a sphere, representing both the fly’s eyes, for each point along the fly’s trajectory. We generated a 1080×540 pixel image of the walls that represented all 360° of azimuth and 180° of elevation. Thus, each pixel represented the intensity of a square 0.33°×0.33° patch of visual space. The angular spacing (Δφ) and the angular sensitivity (Δρ) of the fly’s photoreceptors are both approximately 5° (Buchner et al., 1976; Götz, 1964; Heisenberg and Wolf, 1984). To blur the image such that it would appear as seen through the optics possessed by Drosophila melanogaster, each frame was then smoothed using a 30×30 pixel two-dimensional Gaussian filter with a standard deviation of 5 pixels and then downsampled by a factor of 15 along both dimensions. This created a 72×36 pixel matrix of intensities, with each pixel representing a 5°×5° square of visual space, for each point along the fly’s trajectory.

For each pixel, the intensity signal was converted to a contrast signal by subtracting the mean intensity of that pixel taken over the flight period. These contrast images were fed into horizontal and vertical motion-detector models. The direction of motion was analyzed using ‘delay and correlate’ Hassenstein–Reichardt elementary movement detectors (Borst and Egelhaaf, 1989; Reichardt, 1961). In this motion-detection model, the contrast signals from neighboring photoreceptors are compared by a multiplication after one of the signals has been delayed. A detailed description of the motion-detection algorithm that was used is included in the Appendix. The delay function within the detector model was performed by convolving the contrast signal with a first-order low-pass filter whose impulse response is an exponential decaying with a time constant of 40 ms. To ensure that our results were not affected by processing prior to the motion-detection phase, we also ran the motion-detection model by first processing the input intensities with a first-order high-pass filter (time constant 50 ms) instead of subtracting the mean intensity. The filtered intensities were then summed with the original intensities scaled by a factor of 0.15, in a manner identical to Kern et al. (2000). In examples using peripheral filtering, the time constant of the low-pass filter in the delay line of the motion detector was increased to 100 ms.

While flying within our flight arena, Drosophila melanogaster exhibited stereotyped flight trajectories consisting of straight flight segments interspersed with rapid saccades. During each saccade, the fly’s course heading changed by approximately 90° within 100 ms (Fig. 2A). The saccades are clearly visible as spikes occurring at 0.7 s intervals in the angular velocity recording (Fig. 2B). The kinematic changes associated with the saccades are not limited to changes in heading, but are accompanied by transient alterations in horizontal and vertical velocity. Event-triggered averages centered at each saccade indicate that horizontal velocity decreases slowly before and increases rapidly after each saccade, while vertical velocity increases slightly before and decreases after each saccade (Fig. 2C). In addition, a small rotation in the direction opposite to that of the saccade appears to precede the peak in angular velocity. The regularity of these kinematic changes suggests that the saccades represent a stereotyped motor program.

Avoiding a head-on collision against a wall requires a less-stereotyped avoidance maneuver than avoiding a glancing impact. For this reason, it is reasonable to suppose that flies might vary the magnitude of the saccade depending upon the pattern of visual motion. To test this hypothesis, we examined whether the total angular rotation of the saccades varied with the pattern of optic flow preceding each rapid turn. A saccade was defined as any turn in which the angular velocity exceeded 300° s^–1. We estimated the asymmetry in visual motion experienced by a fly prior to each saccade by calculating its approach angle towards the wall (Fig. 3A, right-hand panel). Approach angle was defined as the angle between the extension of the fly’s pre-saccade trajectory and the line normal to the tangent of the wall at the intersection point (Fig. 3A, left-hand panel).

When saccade angle is plotted against approach angle, two clusters of points are evident, one centered at +90° and the other at –90° (Fig. 3B), indicating that flies tend to turn sharply to either the left or right, with gentle changes in direction being less frequent. To test whether an asymmetry in visual motion experienced by the fly had any effect on the magnitude of the saccades, we regressed saccade angle on approach angle separately within both the leftward and rightward clusters of points in Fig. 3B. For both clusters of data, r² values were less than 0.01, indicating that no more than 1 % of the variance in saccade angle can be explained by variation in approach angle. The slopes for the upper and lower regression lines (0.05 and 0.14, respectively) were not statistically different from zero (P>0.5 for the upper, P>0.25 for the lower). The approach angle did, however, influence saccade direction because the probability of a left turn was greater for a negative approach angle and reduced for a positive approach angle (Fig. 3C). These results indicate that asymmetries in the output of the local motion detectors, which should be greatest when a fly passes close to one wall, affect the direction (left versus right) but not the amplitude of the subsequent saccade.

As flies move through their environment, they encounter visual backgrounds that differ in contrast and texture. To determine how a spatially structured background influences the basic flight pattern, we compared the behavior of flies within a textured background with that seen within a uniform background. In the uniform environment, the wall of the circular arena consisted of a homogeneous translucent white surface. This uniform environment was not, however, entirely devoid of visual features because of the presence of two contrast edges, a lower horizon between the dark floor and the white walls of the arena and an upper horizon where the top of the wall met the dark curtain above. For experiments in the textured environment, the wall of the arena was lined with a random checkerboard pattern to provide a richer visual background.

The flight trajectories generated with textured and uniform backgrounds were quite distinct (Fig. 4A). Histograms representing the time spent by the flies in different portions of the arena (Fig. 4B), as well as the spatial distribution of saccades (Fig. 4C), indicate that, when surrounded by a uniform background, flies approached the walls much more closely before saccading and, thus, explored a larger area of the arena. In contrast, the textured background constrained the flies to the center of the arena. Histograms of various flight parameters generated under the two visual conditions are shown in Fig. 5. The uniform background raised the flies’ horizontal velocity (P<0.0005, t-test) and increased the range of vertical velocities (P<0.01, F-test). The flies also flew at a higher altitude within the uniform background (P<0.0005, t-test), with the mean altitude lying almost exactly at the transition from the uniform white wall to the black curtain. Thus, the presence or absence of a richly textured visual background has a substantial impact on the motor output that emerges from the flight control system.

The regularity of the saccade interval, illustrated by the peaks in the angular velocity recording (Fig. 2), suggests that flies might rely upon an internal clock to control the timing of saccades. However, histograms of the interval between saccades demonstrate that the saccade rate was lower inside the uniform background (P<0.0005, Wilcoxon’s test) (Fig. 6), eliminating the possibility of a purely internal timing mechanism and suggesting that the saccade frequency results from an interaction between the fly’s control system and the visual environment. The decrease in saccade rate (Fig. 6) coupled with the increase in horizontal velocity (Fig. 5) within a uniform background corresponds to an increase in the distance covered between saccades (P<0.0005, Wilcoxon’s test) and the occurrence of saccades farther from the center of the arena. The uniform background did not, however, appear to affect the basic characteristics of the saccade behavior itself. The distributions of saccade amplitude (saccade angle) and direction (left versus right) were similar, and saccade amplitude was still independent of the fly’s position with respect to the walls of the arena (Fig. 7). For the uniform background, the slopes for the upper and lower regression lines, 0.35 and 0.43, respectively, were statistically different from zero (P<0.002 for both). However, these regression lines were not statistically different from their counterparts from the textured background. Further, the visual environment did not affect the distribution of angular velocity (Fig. 5). These observations provide further evidence for the stereotyped nature of saccades.

In addition to its influence on the occurrence of saccades, the visual environment might also affect the straight flight sequences between saccades. Such an influence would be expected if Drosophila melanogaster were to fly according to the optomotor equilibrium model, in which flies adjust their orientation to balance the pattern of visual motion in the left and right visual fields. To test for this influence, we identified inter-saccade flight sequences greater than 0.25 s in duration and then rotated and translated the segments such that the initial heading, defined by the first three points, was the same for each trace (Fig. 8). As indicated by the spread of superimposed trajectories, flies tended to deviate away from their original heading. To determine whether deviation from straight flight depended on the pattern of visual motion, we plotted this deviation angle (measured as the angle between the original heading and a regression through the subsequent flight path) against the approach angle (defined above). The slope of the regression line was significantly larger for the flights in the textured background (P<0.01), suggesting that the richer visual environment induced a greater deviation in the flight trajectory. However, the sign of the relationship indicates that flies deviate away from, not towards, the side experiencing greater visual motion, which is counter to a simple optomotor equilibrium model (Götz, 1968). Instead, this behavioral response is reminiscent of the centering response in freely flying bees (Srinivasan et al., 1991).

The results so far indicate that the visual world exerts a strong influence on the timing and spatial distribution of saccades. What are the specific features of the fly’s estimates of the optic flow patterns that mediate these stereotyped behaviors? One possibility is that the flies use a time-to-collision calculation (Wagner, 1982) to determine when they should initiate a saccade. However, the fact that the textured background reduces flight speed suggests that a time-to-collision model cannot provide the most parsimonious explanation for the timing of saccades. Calculating time to collision, slowly flying flies should approach the walls more closely than rapidly flying flies. The flies, however, do just the opposite, saccading at a greater distance from the walls within the textured background (Fig. 4). The failure of a time-to-collision model cannot be explained by the complete absence of visual cues in the uniform background because the two horizontal edges would provide adequate input for a time-to-collision calculation.

To gain some insight into the features of visual motion that might elicit the saccades, we used the kinematic data to reconstruct the visual world as seen from a fly’s perspective as it flew within the arena with both the textured (Fig. 9A, top) and the uniform (Fig. 9A, bottom) background. We divided the visual environment into quadrants measuring 90° (azimuth) by 180° (elevation). Fig. 9B represents a ‘snapshot’ of the fly’s visual world at the instant the fly was at the positions shown by the red circles in Fig. 9A. The lower panel of Fig. 9B emphasizes that, while visual cues in the uniform background were sparse, the two horizontal edges do provide input to the fly’s visual system. After reconstructing these images, we then estimated the fly’s perception of the magnitude and direction of optic flow using an array of vertical and horizontal Hassenstein–Reichardt elementary motion detectors (EMDs) (Reichardt, 1961; Borst and Egelhaaf, 1989) (for details, see Appendix). The properties of the EMDs were based on a standard model derived from behavioral and physiological experiments in flies (Egelhaaf and Borst, 1989; Reichardt and Poggio, 1976).

The vector fields representing the output of the local motion detectors are shown in Fig. 9C. In the case of the uniform background, only the horizons due to the top and bottom of the arena elicit responses from the EMDs. Because the flies generally flew in the forward direction, obstacles are more likely to appear in the frontal region of their visual field, so this portion of the visual field is most important for flight control. Thus, we concentrated on the 180° of azimuth that represent the frontal fields of view (Fig. 9C, regions 2 and 3). Each quadrant thus represented the frontal 90° of azimuth of each eye. Horizontal outputs of the EMDs were summed over the right and left halves of each of the frontal quadrants. Similarly, the vertical components were summed over the top and bottom halves of each quadrant (see Appendix for details). To search for features that might serve to initiate saccades, we constructed reverse correlations of the large-field motion signals using the initiation of each saccade as a trigger point.

According to the optomotor equilibrium model, a fly maintains a flight course by minimizing the net horizontal rotation of its visual surround (Götz, 1964, 1968). Thus, we first examined the pattern of large-field horizontal motion preceding each saccade. Large-field horizontal image motion results from rotation about the yaw axis as well as from sideways translation. Forward translation also generates image motion with a horizontal component, particularly about the equator of the eye. The large-field horizontal motion experienced in the frontal position of each eye was estimated by summing the output of the horizontal EMDs over each of the two front visual quadrants to generate a time course of the front-to-back motion the fly experiences (Fig. 10A; red arrows indicate the direction of large-field motion; see Appendix for details). The individual large-field horizontal motion traces are then overlaid and aligned such that the initiation of the saccade occurs at time zero (Fig. 10B). The average (shown by the red lines in Fig. 10B) provides a record of the typical large-field horizontal motion experienced on each half of the frontal field of view prior to, during and after each saccade. Ipsilateral refers to the side away from which the fly is turning (generally the side nearest the wall of the arena), and contralateral refers to the opposite side. As expected, the most prominent feature in these traces is a large horizontal motion signal produced when the animal rotates rapidly during the saccade (the peak within the gray shaded region). However, it is the features prior to the initiation of saccades that provide clues as to the stimulus trigger. Preceding each saccade, in the textured background, the fly experiences a steady front-to-back motion on the ipsilateral side that reaches a small peak and then decreases towards zero immediately preceding the saccade. On the contralateral side, the front-to-back motion is smaller, and immediately before the saccade there is a peak in front-to-back motion. Thus, certain features of the large-field horizontal flow might play a role in triggering the rapid turns. Large-field horizontal signals were entirely absent preceding saccades in the uniform background as a result of the absence of vertical edges. This indicated that unidirectional horizontal visual motion could not explain the occurrence of saccades under all visual conditions (Fig. 10B, lower traces).

Another feature that might serve as a saccade trigger is large-field vertical motion. Summing the output of the EMDs sensitive to vertical motion within the frontal quadrants provides a measure of the total vertical motion across each of the fly’s eyes (Fig. 10C). Large-field vertical motion can result from vertical translation or from rotation about the pitch or roll axis. Because of limitations due to the small size of Drosophila melanogaster, we are unable to determine the extent to which the fly was pitching and rolling over the course of its flight. Thus, under our modeling conditions, unidirectional large-field vertical motion can come only from translation upwards or downwards. When flying within a textured background, a slight downward image motion precedes each saccade on both sides, indicating upward translation of the fly before each saccade. However, the magnitude of these downward signals is much smaller for the uniform background, suggesting that the fly does not rely on these large-field vertical motion cues to trigger saccades.

A further cue that might initiate saccades is image expansion. To construct a rough measure of horizontal expansion seen by each eye, we summed the rightward horizontal motion over the right half of the frontal quadrant of the eye and the leftward motion within the left half of the frontal quadrant (Fig. 11A). Vertical expansion was determined by summing the upward motion components over the top half of the frontal quadrant and the downward components over the lower half of the frontal quadrant for each eye (see Appendix for exact details). Prior to each saccade, the fly experiences a slow horizontal expansion in its frontolateral visual field followed by an abrupt contraction caused by the rotation during the saccade itself (Fig. 11B, left). Focusing on the period preceding the contraction associated with the saccade, horizontal expansion on the ipsilateral side is relatively constant before each saccade within a textured background. On the contralateral side, the fly experiences much less horizontal expansion. Within the uniform background, the horizontal expansion component is undetectable because of the absence of vertical edges. Vertical expansion, however, is quite prominent because of the presence of the two horizontal edges (Fig. 11D). Prior to each saccade, there is a steady increase in vertical expansion on the ipsilateral side and a small increase on the contralateral side.

After adding the horizontal and vertical signals, the pattern of the reverse correlations was consistent within both the textured and uniform backgrounds, suggesting that the calculation of total large-field expansion is involved in the triggering of saccades under both visual conditions (Fig. 12). Thus, despite the differences in horizontal flight speed and the proximity to the walls preceding each saccade, the total expansion experienced prior to a saccade is independent of the background in which the fly was flying. This suggests that there is a threshold in large-field image expansion that triggers a saccade and that this threshold is defined by the amount of perceived visual motion, not the spatial structure of the environment. The substantial differences in flight behavior produced within the textured and uniform backgrounds emerge from the interaction between the fly’s flight control system and its visual world.

The results of these experiments indicate that the pattern of search behavior of freely flying Drosophila melanogaster emerges from interactions between the fly’s flight control system and its sensory environment. Flies explore sensory landscapes using a series of straight flight segments separated by rapid saccadic turns. As with primates, saccades confine the time in which the eye is rotating to brief bursts, an advantage to animals that rely upon translational optic flow to determine the range of objects in their environment (Land, 1999). While the direction of the saccade (left versus right) is influenced by visual input, the magnitude of the saccade is not (Figs 3, 7). Reconstructions of the fly’s visual input preceding each saccade indicate that image expansion may serve as a trigger for the rapid turns (Figs 9–12). Visual input also causes the fly to deviate from straight flight between saccades, but in a direction opposite to that predicted by a model based on optomotor equilibrium. Flies tend to turn away from the side experiencing a greater amount of visual motion (Fig. 8). As a result of these effects, the presence or absence of a textured visual background has substantial effects on free flight behavior (Figs 4–6). For a uniform background, translational velocity increases and this, with the accompanying decrease in saccade frequency, causes the fly to explore a greater proportion of the arena.

The main experimental limitation of this study was the low spatial and temporal resolution of our visualization system. This low resolution was the consequence of the deliberate choice to maximize the distance over which we could track flies. Current studies using high-speed video indicate that a visualization cube with sides no greater than 1 cm is required to capture wing and body kinematics accurately at 5000 frames s^–1 (S. Fry and M. H. Dickinson, in preparation). In the present study, flight trajectories were sampled at 30 frames s^–1 over a 1 m diameter arena, and we estimated saccade durations as brief as 100 ms, or approximately three sample points. This limited temporal resolution did not, however, prevent the detection of saccades, which appeared as clear spikes in recordings of angular velocity. In addition, we estimated saccade amplitude from the change in trajectory heading, a measurement that does not require a high temporal resolution of the saccade itself. The low spatial resolution prevented measurement of the fly’s body position and the position of the head relative to the body. To reconstruct the fly’s visual input, we assumed that the horizontal projection of its longitudinal body axis was aligned along the flight trajectory. These assumptions may be reasonable in still air because, unlike larger flies, Drosophila melanogaster display minimal side-slip under laboratory conditions (David, 1978). Drosophila melanogaster may, however, move their head relative to their body during flight, and the impact of such movements is absent from our estimation of optic flow. However, because our visual processing model included large-field spatial summation, its predictions should be robust with respect to the precise orientation of the fly’s head. Further, if the head stabilization reflex functions in flight to stabilize gaze in the face of body rotation (Land, 1999; van Hateren and Schilstra, 1999), this reflex would serve to dampen the motion of the eye and render the free flight conditions closer to those of our model.

A second critical set of assumptions relates to the filter values used in our modeling of the movement detector array. However, changing the filters upstream of the motion detector (from simple subtraction of the direct current signal to a high-pass filter with a time constant of 50 ms) and in the delay line of the detectors themselves (from 50 to 100 ms) did not alter the salient result of the analysis. Even with different filter settings, the output of total expansion followed a similar time course preceding saccades in both the uniform and textured backgrounds. Thus, our central conclusions that total expansion is currently the most parsimonious explanation for the saccade trigger is robust to the assumptions of our basic modeling methods.

The mean horizontal flight speed measured in this study (30 cm s^–1) is substantially slower than that reported in a recent study of the free flight of Drosophila melanogaster by Marden et al. (1997) (46–70 cm s^–1). These authors compared the free flight trajectories of two control lines of flies with those of two lines selected for their ability to fly through a baffled wind tunnel. While the maximum flight speed did not differ among the control and selected lines, the tunnel-selected flies were more likely to fly near peak performance. Given the results of the present study (Fig. 5), this large discrepancy in flight performance is probably due to differences in the visual environment. Marden et al. (1997) used a cubic arena with a side length of 0.5 m lined with white translucent Plexiglas which was back-illuminated with bright fluorescent lights. Thus, this environment would have provided little or no contrast input to expansion detectors and other motion-sensitive circuitry within the visual system. The absence of such input might explain the elevated flight velocity. Alternatively, the phototactic reflexes that were activated by the use of an ultraviolet light source might have elicited near-maximal flight speed. In either event, the differences in both mean and peak flight speed measured under different visual conditions suggest that the sensory environment exerts a strong influence on flight performance. It is even possible that the performance difference noted among control and tunnel-selected lines might result from a disparity in visual processing circuitry or some other sensory system involved in flight control.

These free flight experiments demonstrate that flies produce fixed-amplitude saccades of approximately ±90° within the horizontal plane (Fig. 7). Free flight experiments in other species have shown that the saccade angles in larger flies are typically smaller than those seen in Drosophila melanogaster, although the saccades exhibited by small houseflies (Fannia canicularis) are also approximately 90° (Zeil, 1986). One possible explanation for the constancy of saccade amplitude within each species is that the saccade motor program is terminated by visual feedback. However, our observation that saccades are of constant amplitude in a uniform visual background, from which no cues are available for orientation about the yaw axis, argues strongly against this possibility. Further, the additional visual information present during flight within the textured background does not alter the distribution of saccade amplitudes (Fig. 7). These data are consistent with tethered flight experiments in which the magnitude of torque spikes (thought to be the tethered flight equivalents of saccades) was unaffected by imposed motions of a stripe upon which the animal was fixating (Heisenberg and Wolf, 1979). Further, freely flying hoverflies, Syritta pipiens, make substantial errors when generating saccades towards targets and show no evidence of correcting the saccade once it has been initiated (Collett and Land, 1975).

Another possible explanation for the consistency of saccade amplitude is that saccades might represent the feedforward output of a stereotyped motor program performed without any sensory feedback. However, while the experiments described above appear to rule out a role for visual feedback, other modalities might still function to regulate saccade amplitude. For example, the observation that torque spikes are shorter when flies are tethered loosely, allowing them to rotate more freely about their yaw axis, suggests that haltere feedback may play a role in terminating the saccade motor program (Mayer et al., 1988).

Changing the fly’s visual environment altered its saccade rate, demonstrating that an internal clock mechanism is not responsible for the timing of saccades. Because the absence of large-field rotation signals during flight within a uniform background did not prevent the generation of saccades, it is unlikely that flies perform saccades in response to rotation cues. Similarly, the absence of large-field vertical motion before saccades eliminates the possibility that saccades are generated as a result of a fly’s perception that it is rising or falling. The noise inherent in our simple estimates of expansion preceding individual saccades (see Fig. 11) suggests that our model for calculating total expansion is a simplification of the calculations that might be performed by the fly. For example, the spatial integration performed on the dendrites of lobular plate cells functionally removes temporal noise, causing the cell’s membrane potential to vary smoothly with image velocity (Single and Borst, 1998). However, the fact that the average sum of horizontal and vertical expansion rises along a similar time course before saccades within both textured and uniform backgrounds (see Fig. 12), despite differences in flight speed and approach distance under these two visual conditions, suggests that image expansion plays a role in the initiation of saccades. The presence of vertical edges that can provide horizontal expansion cues, which are absent during flight within a uniform background, explains why the flies generate saccades more frequently within a textured background.

If flies use image expansion cues for their initiation, saccades may represent a reflexive response to avoid rapidly approaching objects. Neurons sensitive to small objects approaching from any orientation have been identified in locusts (Gabbiani et al., 1999, 2001; Rind and Simmons, 1992). These neurons fire at a peak rate when a small-field stimulus exceeds a threshold angle subtended on the locust’s eye (Gabbiani et al., 1999). Although it is possible that saccades occur as a result of the fly performing a similar calculation, it is more likely that neurons sensitive to large-field expansion stimuli are responsible. It has been suggested that neurons that spatially sum the output of multiple local motion detectors underlie the initiation of the expansion-dependent landing response (Borst, 1986; Borst and Bahde, 1986), and similar computations may underlie the triggering of saccades. In Calliphora erythrocephala, recordings from neurons descending through the central connective have detected descending cells that are sensitive to image expansion directly in front of the fly (Borst, 1991). Of the two classes of expansion-sensitive cells within the optic lobes of the hawkmoth Manduca sexta, the class 2 cells have properties that are consistent with our behavioral results (Wicklein and Strausfeld, 2000).

The optomotor response refers to a fly’s tendency to turn in the same direction as a large-field motion in order to minimize image motion across the retina (Götz, 1975; Heisenberg and Wolf, 1984). The fly is thought to use this response to correct deviations from straight flight that may arise from external disturbances, such as the presence of a strong crosswind, or internal asymmetries, such as damage to one wing. In our experiments, asymmetries in visual motion are generated whenever a fly moves along a path that does not intersect the center of the arena. Our data show that, when faced with such asymmetries, Drosophila melanogaster turn away from the side experiencing the greater amount of visual motion, a response opposite to that expected if the flies were to fly according to a simple optomotor equilibrium model. Thus, while we did not impose a perturbation in image rotation, our results suggest that freely flying flies move straight in the face of bilateral asymmetries in visual motion. These results are similar to those of experiments with monocularly blinded freely flying blowflies (Lucilia sp.), which show little difference in their free flight behavior compared with control animals (Kern and Egelhaaf, 2000). The same monocular flies did, however, show a tendency to turn in the direction of the non-occluded eye when walking (Kern and Egelhaaf, 2000; Kern et al., 2000). By rotating in this fashion, the walking flies might be shifting the focus of expansion laterally such that the sum of all the horizontal components of the optic flow would be zero, thereby restoring optomotor equilibrium.

Such an interpretation is unlikely in our experiments because the flies tended to deviate from a straight course by turning away from the nearer wall, the direction opposite to that which would restore optomotor equilibrium. Further, the slope relating approach angle and deviation angle was significantly larger during flight within a textured background, indicating that deviation increased with the amount of visual information. The direction of the deviations from straight flight in our experiments is reminiscent of the centering response seen in honeybees attempting to balance the image velocity on either side (Srinivasan et al., 1991).

During the straight flight segments between saccades, a fly could make use of both mechanosensory and visual cues to maintain a stable course. The fly’s haltere system is capable of sensing rotations about all three axes (Dickinson, 1999; Nalbach, 1993; Nalbach and Hengstenberg, 1994) and could use such information to correct course deviations. Our experiments also demonstrate that flies possess a visually mediated centering response that directs their flight path away from the side perceiving the greater amount of visual motion. Further, flies possess fixation behaviors in which they track small visual targets. Thus, within the flight control system, there are potential conflicts between a mechanosensory equilibrium system (the halteres) that attempts to maintain straight flight and a visual system that directs the fly away from obstacles and towards objects. Given that these two modalities may often act at crossed purposes, it is of interest to note that pathways exist through which each of these two modalities might alter the gain of the other. In Calliphora vicina, the muscles controlling the halteres receive input from the visual system (Chan et al., 1998). Thus, the visual system has the ability to either amplify or decrease the fly’s sensitivity to angular velocities. Evidence for the reciprocal pathway is also present. The haltere sensory cells can influence head position and, thus, visual motion sensitivity through their connections with neck muscle motor neurons (Gilbert and Bauer, 1998; Sandeman, 1980).

Previous models for flight control in large flies suggest that lobula plate tangential cells (LPTCs) sensitive to large-field horizontal motion (HS cells) are necessary for yaw stabilization and, thus, for straight flight (Hausen and Egelhaaf, 1989; Hausen and Wehrhahn, 1990). During the periods in which these horizontal cells would be active, the flies in our experiments do not respond in a way that would minimize asymmetries in optic flow by rotation, as would be predicted by the optomotor equilibrium model. Thus, the HS cells that respond strongly during tethered flight recordings do not appear to play an analogous role during free flight. If straight flight is maintained largely by feedback from the haltere system, the role of the HS cells in free flight must be re-evaluated. One hypothesis is that the HS cells correct low-frequency slow drifts that the halteres cannot detect. In addition, the LPTCs of the vertical system (VS cells) might approximate matched filters sensitive to optic flow patterns for various combinations of rotation about the pitch and roll axes (Franz and Krapp, 2000; Krapp and Hengstenberg, 1996). While the fly may rely more heavily upon halteres for flight stabilization about its yaw axis, visual feedback is important in stabilizing pitch and roll. Halteres do, however, provide feedback to stabilize pitch and roll as well (Dickinson, 1999).

Our data suggest a model of how free flight behavior might emerge from an interaction between a fly’s motor control system and its visual environment (Fig. 13). While flying along a trajectory, the fly uses an array of ‘delay and correlate’ motion detectors (Borst and Egelhaaf, 1989; Reichardt, 1961) to estimate optic flow. However, as it moves, the fly relies on its mechanosensory equilibrium system (halteres) to maintain a straight course. Over short distances, at least, the halteres alone appear sufficient to maintain straight flight. While dominated by feedback from the halteres, tonic feedback from the visual system directs the fly away from large obstacles via a centering response. In addition, the fly is continuously integrating the sum of the horizontal and vertical expansion, which has the effect of removing some of the noise in the expansion signal. If saccades are discrete ballistic events, they are likely to be triggered when some neural signal exceeds a threshold. The expansion signals rise gradually over the 700 ms preceding the saccade (Figs 11, 12) and are laden with rapid fluctuations as a result of the output of local elementary motion detectors. Thus, it is likely that the nervous system conditions the instantaneous signal prior to saccade initiation. For example, temporal integration performed on the expansion signal, in addition to removing noise, would also result in a signal that rises more rapidly, making a more precise trigger for each saccade. When the accumulated sum of horizontal and vertical expansion exceeds a threshold level on either side, the saccade causes the fly to rotate 90° away from the side on which expansion was greatest. Because of the variation in the estimate of total expansion preceding individual saccades, it is difficult to determine the latency of the collision avoidance response. Measurements of responses to visual stimuli indicate latencies of 50 ms during free flight (David, 1984) and 100 ms during tethered flight (Heisenberg and Wolf, 1988). Preliminary tethered flight experiments in which flies are stimulated with expanding squares suggest a collision avoidance latency of 50 ms (L. F. Tammero and M. H. Dickinson, unpublished results). Thus, the time-integrated function of total expansion is likely to exceed threshold within that time frame. As with many other escape responses, a saccade in one direction inhibits a saccade in the opposite direction, preventing the fly from attempting to turn in both directions at once. Following the saccade, the accumulated expansion will be cleared, and the fly will continue to fly along a straight trajectory, until total expansion increases again to a level at which another saccade is triggered. In this way, complex free flight patterns of behavior might emerge from a rather simple control algorithm.

The input to the elementary motion detector (EMD) model is a 36×72 matrix of time-varying contrasts that will be referred to as C_i,j(t). Each element of this matrix represents the contrast within a 5°×5° square of visual space at a given instant. Both the inputs and outputs of the EMD model are spaced at 5°. A delayed version of the contrast signal, D_i,j(t), within the motion detector is constructed by filtering the contrast signal via convolution with L(t), the impulse response of a first-order low-pass filter:

where

A time constant, τ, of 40 ms was selected on the basis of data from experiments involving larger flies (Borst and Bahde, 1986; Harris et al., 1999; O’Carroll et al., 1997). To ensure that our results are not dependent on the time constant in the delay line, simulations were repeated using different values of τ. The outputs of horizontal and vertical local motion detectors, h_i,j and v_i,j, are calculated as:

and

These equations show that the outputs of each of the two half-detectors are fed into the subtraction stage with equal weight. As defined, the horizontal local motion detector responds positively to rightward motion and the vertical motion detector responds positively to upward motion. The vector fields plotted in Fig. 9C represent the output of these local motion detectors.

To determine the large-field motion signals, the outputs of the local motion detectors are pooled spatially by linear summation. Large-field horizontal motion signals on the left and right, H_L(t) and H_R(t) are calculated as the sum taken over all rows for the columns that make up the frontal 180° of the fly’s field of the view:

and

The row and column indices (q₁, q₂, … q₄.; p₁, p₂,… p₈) represent the indices of the h_i,j matrix, as described by the following:

and

The locations of the edges of each 5° pixel can be calculated from the matrix indices according to:

and

with all values in degrees. Thus, the column with a j index of 54 would correspond to the area of visual space between 85° and 90° of azimuth.

To compute the reverse correlations, the points where the fly initiated each saccade (t₀) and the direction of the saccade (left or right) are first determined. For each saccade, a row vector representing the time course of the horizontal expansion from the 0.67 s before the initiation point to 0.33 s after the initiation point is formed.

H_L^rk is used to symbolize horizontal expansion on the left side preceding and following the kth saccade to the right:

If a fly were to saccade to the right m times and to the left n times, these row vectors would be assembled into ipsilateral and contralateral large-field horizontal motion matrices, H_Ips and H_Cont according to:

and

The individual traces shown in Fig. 10B are the rows of the H_Ips and the H_Cont matrices. This procedure is repeated for the large-field vertical motion signals, V_Ips and V_Cont, which are plotted in Fig. 10D. Measures of horizontal and vertical expansion are calculated according to the following:

and

H_Exp,Ips, H_Exp,Cont, V_Exp,Ips and V_Exp,Cont are assembled in the same manner as H_Ips and H_Cont. H_Exp,Ips and H_Exp,Cont are plotted in Fig. 11B, while V_Exp,Ips and V_Exp,R(t) are plotted in Fig. 11D.

The authors wish to thank Jocelyn Staunton for help in collecting the data presented and M. Frye and A. Borst for reading this manuscript. This work was supported by grants from the National Science Foundation (FD97-23424), ONR (FDN00014-99-1-0892) and DARPA (N00014-98-1-0855).