Processing of Artificial Visual Feedback in the Walking Fruit Fly Drosophila Melanogaster

Visual control of locomotion in the fruit fly Drosophila melanogaster has been studied previously under three basic conditions.

• In arena experiments, freely walking flies interact with their visual surroundings in a natural closed-loop situation (e.g. Bülthoff et al. 1982; Götz, 1980, 1994; Götz and Biesinger, 1985; Schuster, 1996; Strauss and Heisenberg, 1990, 1993; Wehner, 1979, 1981). By analogy with our everyday experience, the retinal images of objects seen by the test fly expand on approach, shrink on retreat and swerve in the direction opposite to the fly’s rotatory and translatory self-motion. Natural closed-loop conditions restricting motion to horizontal turns have been implemented in flies that were flying stationarily on a taut thread (Mayer et al. 1988) or walking on a tread compensator ball with detached orientation control (Götz and Wenking, 1973).

• In artificial closed-loop experiments, a test fly is usually mounted in a fixed position and orientation on a measuring device. Its intended steering actions are continuously recorded and used to control an artificial visual environment. With respect to the most frequently studied yaw turns, the natural situation is simulated by negative feedback but, depending on the task, all kinds of intended deviations can be used to modify the fly’s control over its visual environment (e.g. sign inversion, changes in gain). Flight simulators have been implemented with a torque meter or a wingbeat processor (e.g. Dill et al. 1993; Götz, 1994; Heisenberg and Wolf, 1984; Wolf and Heisenberg, 1990, 1991; Wolf et al. 1980, 1992) and a simulator for turning on the ground with an airborne styrofoam ball (Bülthoff, 1982). The term ‘artificial closed-loop’ may also apply to situations with partially motion-inhibited animals, such as flies turning on a thread. In such a case, the fly’s own actions are measured and used to manipulate its retinal image artificially.

• Real open-loop conditions are created when attempted steering actions of a fly in a fixed position and orientation do not interact with its visual input. Corresponding measurements were carried out during stationary flight on a torque meter, a thrust meter or a wingbeat processor (e.g. Buchner et al. 1978; Götz, 1964, 1968; Heide and Götz, 1996; Heisenberg and Wolf, 1984), or during stationary walking on a tread compensator (e.g. Götz, 1975; Götz and Wenking, 1973) or on an airborne styrofoam ball (e.g. Buchner, 1976, 1984; Bülthoff and Götz, 1979; Heisenberg and Buchner, 1977).

In the present paper, we introduce ‘virtual open-loop’ conditions as a novel paradigm for freely walking flies. In this paradigm, the test fly receives the proprioceptive and inertial input incidental to free walking, while visual feedback from its translatory and rotatory self-motion is largely suppressed. Differences between the reactions obtained in a similar visual environment under virtual open-loop or natural closed-loop conditions can be attributed predominantly to the visual feedback of self-motion. To suppress any kind of self-induced visual stimulation under virtual open-loop conditions, the actual panorama has to follow, without significant delay, the rotatory and translatory components of the fly’s locomotion in a horizontal plane. In technical terms, this requires continuous positive feedback of the visual stimulus with a coupling constant of +1. In an attempt to solve this problem, we continuously recorded the position and orientation of a fly walking freely on a disk in the centre of an arena. Using this information, we simulated, on a cylindrical wall of light-emitting diodes, a 360 ° panorama of selected virtual objects to appear in a fixed position on the retina of the mobile fly. Optomotor responses are elicited by moving these objects across the retina.

Here, we apply the new experimental paradigm to aspects of two-dimensional motion perception in freely walking flies. During flight, Drosophila melanogaster evaluates horizontal motion components to control its course and vertical motion components to control its altitude (Götz, 1968; Heide and Götz, 1996). During walking, the evaluation of vertical motion components seems to be absent; these components are relevant for controlling the force of flight, but not the walking speed (Götz, 1975; Götz and Wenking, 1973). This facilitates our present attempt to investigate the illusion of a horizontal angular displacement, which is elicited by vertical motion of objects with oblique edges.

What is the origin of this illusion? According to earlier results, the motion stimuli are processed by a two-dimensional array of correlation-type elementary motion detectors (EMDs; Egelhaaf and Borst, 1993; Hassenstein and Reichardt, 1956; Reichardt, 1961) predominantly formed by neuronal interactions between nearest-neighbour visual elements (Buchner, 1976; Buchner et al. 1978). Consequently, Drosophila melanogaster perceives motion through a mosaic of small windows, each the size of the receptive field of one EMD. Large-field neurones in the third visual ganglion collect the signals from motion detectors with common preferred directions, thus forming directionally selective neuronal channels for horizontal motion (Hausen and Egelhaaf, 1989; in Drosophila melanogaster: Pflugfelder and Heisenberg, 1995) and vertical motion (Hengstenberg, 1982; Hengstenberg et al. 1997; Krapp and Hengstenberg, 1996).

Uncertainty about the direction of pattern motion is expected if spatial summation of the signals of corresponding EMDs is the only mode of motion processing: local measurements of motion within receptive fields that are small compared with the whole extent of a moving contour yield the velocity component perpendicular to the ‘visible’ line segment of the contour. The velocity component parallel to this line segment is necessarily indeterminate without further processing. Both technical and biological systems are confronted with this basic ambiguity of motion detection, in text books referred to as the ‘aperture problem’. Perception of the true direction of object motion in flies may be limited by the comparatively small receptive fields of the individual EMDs. Indeed, in behavioural and electrophysiological experiments on both the housefly Musca domestica and the blowfly Calliphora erythrocephala, Borst et al. (1993) found no signs of motion processing other than the low-pass summation of the outputs of EMDs with common preferred directions. These authors presented to the flies a half-circle moving either vertically or horizontally behind the square-shaped aperture of a cathode ray tube (CRT) screen. The responses indicate that the flies and also their predominant pair of optomotor large-field neurones with horizontal preferred direction, the H1 cells on either side, mistook vertical for horizontal motion.

In these experiments, much of the half-circle figure was always hidden by the frame of the CRT. This might have prevented the flies from applying hypothetical second-stage algorithms to facilitate the identification of the figure and its direction of motion, e.g. by figure binding or contour assignment. Such algorithms have been proposed for both human and machine vision (e.g. Hummel and Biederman, 1992; Nakayama and Silverman, 1988). Here, we present to flies complete figures that can, in principle, be identified by such procedures. These figures are designed to stimulate, simultaneously and equally, horizontal and vertical channels when moving in either of the two orthogonal directions.

By comparing the responses obtained under real open-loop conditions, virtual open-loop conditions and natural closed-loop conditions, we tried (i) to assess the effect of locomotion-induced visual feedback on the motion illusion due to the aperture problem and (ii) to verify the reliability of the newly established virtual open-loop conditions.

One-to three-day-old female flies (Drosophila melanogaster, wild type ‘Berlin’) were used throughout this study. Their wings were shortened to approximately one-third of their normal length under cold anaesthesia (+4 °C) in a stream of dry air. For experiments with stationarily walking flies, the tip of a small V-shaped wire was glued under cold anaesthesia to the head and thorax (Loctite glass glue, hardened by 30 s of irradiation with ultraviolet light). This procedure immobilizes the head and body of the fly. Prior to testing, the flies were allowed at least 4 h of postoperative recovery time.

As in earlier experiments (Bülthoff et al. 1982), the test fly walked on a circular disk (radius r_d=43 mm) surrounded by an illuminated cylindrical screen. A water-filled moat around the disk prevented its escape. A red-sensitive electronic black-and-white camera monitored the movements of the fly from above (Valvo CCD design board with frame transfer chip NXA1101). The video information can be processed at a maximum time resolution of 50 Hz using an ATVista card (Truevision Inc.) in the non-interlaced mode in a PC. A frame rate of 20 Hz was sufficient for the present set of experiments. The translucent floor of the arena was weakly illuminated by red light from below. A red filter in front of the camera lens prevented the simultaneous processing of yellow-green stray light from the visual stimulus on the surrounding screen. The test fly could be recognized as a dark elongated spot on the homogeneously illuminated background of the video image. A computer program determined the position of the fly by frame-scanning and its main axis by fitting an ellipsoid to the outline of the spot. The angle of orientation was given by the main axis of the ellipsoid and by the actual direction of locomotion along this axis derived from the change in position between consecutive frames. During this period of 50 ms, the change in orientation is not expected to exceed ±40 °. Apparent turning on the spot still produced enough forward motion for the algorithm to indicate the orientation of the fly with respect to the main axis of the ellipsoid. Position and orientation were conveyed via a parallel port to the input/output (I/O) card (8255-port) of a second computer (PC Pentium 120 MHz) used to update the light-emitting diode (LED) panorama (see below) at a rate of 20, 10 or 5 Hz. In all of the following experiments on freely walking flies, the locomotor behaviour was reconstructed from the sequence of velocity vectors that represent direction and speed between consecutively recorded positions. It was not necessary to evaluate the orientation of the body axes for these purposes.

In a control experiment, the test fly was held in a fixed position and orientation in the centre of the arena, where it was allowed to walk on an airflow-supported 10 mg styrofoam ball (Buchner, 1976). The optically recorded motion of the dotted ball gave the intended turns in revolutions per metre path length towards the preferred side.

The newly developed light-emitting cylindrical screen (radius r_s=201 mm, height 224 mm) is subdivided into 90 imaging units, each comprising two columns of 32 LEDs. As shown in Fig. 1, the LEDs are mounted on an electronic driver board (A) held by radial slots on the lower side of an elevated, ring-shaped head plate (shown in cross section) and on the upper side of the mirror-inverted base plate (not shown). The LEDs (type RS585-589, maximum voltage 2.3 V, maximum current 30 mA, wavelength 565 nm) fitted into the 5 mm diameter holes shown on the left side of an aluminium block (B, axial section). The conically enlarged part of the holes on the right reflects off-axis radiation towards the translucent screen of the panorama. The right side of the block was covered with a 1 mm thick blackened aluminium diaphragm (C) consisting of 5 mm×5 mm perforations spaced 7 mm centre-to-centre from their horizontal or vertical neighbours. The light from an activated LED passes a 4 mm interspace and a 3 mm transparent acrylic plate (D). The right side of this plate is coated with the actual screen of the arena, a 40 μm thin Marata-diffuser (Spindler & Hoyer, 90012). Adjacent LEDs illuminate adjacent 7 mm×7 mm wide pixels on the screen. This corresponds to a resolving power of approximately 2 ° for a viewpoint at the centre of the arena. The comparatively high density of approximately 4 pixels per sampling element in the visual system of Drosophila excludes ‘aliasing’ of the received signals by geometric interference between the periodic structures of the arena and retina.

The presentation of the light-diffusing particles in a very thin layer prevents the blurring of the pixels and diminishes the probability of incident light being repeatedly scattered on its passage through the diffuser. The expected increase in transmission and decrease in reflection improve the contrast of the image. Bright pixels gain luminance because of increased transmission of light from the LEDs. Dark pixels lose luminance because of decreased reflection of ambient light coming from inside the arena – the transmitted portion of this light is effectively absorbed behind the screen by the blackened diaphragm and the mounting substratum of the LEDs. For a fly in the centre of the arena, a maximum luminance I_max of 60 cd m⁻² and a maximum contrast c_max=(I_max−I_min)/(I_max+I_min) of 0.94 was achieved by these measures. Fig. 7E,F shows examples of arbitrary patterns on the surface of the arena. The luminance of the 180×32=5760 pixels can be controlled independently of each other either by an existing program or by the reactions of the walking fly.

The block diagram in Fig. 2 illustrates the bus structure and the driver circuit for two of the 180 columns of 32 LEDs contained on a common printed board. Each LED is directly driven by a storing unit (flip-flop), the state of which can be changed via a 32-bit pattern bus and one of the 180 column-select lines. Four octal D-type edge-triggered flip-flops (74HC374) make up a column. The pattern bus interconnecting all the columns originates from four eight-bit ports (4/3 8255 ports) of an I/O card in the computer. The contents of this bus are stored in all of the columns of flip-flops which receive a column-select signal. The respective LEDs display the bit pattern until the next select pulse is given to this column. The column-select lines can be set, one at a time, by an eight-bit signal from the computer via an I/O port (one-third of an 8255 port). The signal acts on a cascade of 16-channel bi-FET analogue multiplexers (PMI MUX-16) arranged in two layers (labelled ‘multiplexer 8 into 180’ in Fig. 2; details not shown).

In order to save on additional driver units, the current through each of the LEDs is limited to 10 mA by a 300 Ω resistor. This limit fully exploits the maximum continuous current of 80 mA through the octal flip-flop circuit’s supply voltage connection. The limitation also seemed advisable with regard to the maximum total current through the 5760 LEDs in their all-active state. A 5 V, 60 A switching power supply (Kepco RAX 5-60K) is used to meet the requirements of maximum consumption.

The apparent brightness of activated LEDs within a particular column can be varied in nine steps by choosing the number of 0.5 ms light-on pulses (0–8) within subsequent time intervals of 4 ms. Since the clock frequency is at least ten times greater than the flicker-fusion frequency of the insects’ visual system, the flies perceive the time integral of the pulsed light. Technically, this is achieved by commonly grounding the cathodes of all LEDs belonging to a particular column through a power field-effect transistor (IRF 520, used in the switching mode). The serial output of a shift register (74HC165) determines its state. The shift register is closed into a ring and, in turn, is clocked at approximately 2 kHz using a timer circuit (555; each printed board contains its own clock generator to prevent synchronous modulation of the total power consumption). The apparent luminance of an activated LED is chosen by determining the ratio of high and low bits circling in the shift register. The bit pattern is delivered via a four-bit brightness bus to the parallel preset inputs of the register. Loading of the pattern is controlled by the same column-select lines as described above. The brightness information is held until the next select pulse is given to the column.

The configuration of the circular display is organized column-wise. The 32-bit pattern of a column and the four-bit brightness value are put out to the appropriate ports. The column-select line is then activated by writing the column number to the port which is connected to the multiplexer. The complete display can be written at 1000 frames s⁻¹. Any repetition in the displayed pattern can be used to enhance the frame rate further: the patterns of the 32-bit bus and the luminance bus can be left unchanged while column-select pulses are put out to all the columns that are to display this pattern. Because the actual pattern is held in the flip-flops, only the columns that display parts of the panorama subject to change need to be updated (e.g. virtual objects that are moving in front of a structured background). A shadow memory in the program keeps track of the displayed panorama and allows for the restoration of the background after a virtual object has passed.

The algorithm developed for the following experiments calculates the azimuth and elevation of the status source of the selected template for each of the LEDs according to which LEDs are switched on or off. The calculation takes into account the actual position and orientation of the fly, the selected apparent radius r_a/r_d and the viewpoint eccentricity factor f_e. The algorithm supports the evaluation of ‘kinetic’ templates to simulate horizontal or vertical motion within the visual field and the simultaneous calculation of more than one virtual panorama for experiments with independently moving fore-and background. Depending on the complexity of the task, the refresh time achieved on a 120 MHz PC ranged between 40 and 200 ms. Using the available information, the refresh cycle first updates a column of pixels either nearest to or in front of the walking fly. The update proceeds to the next columns on either side and continues outwards, always jumping from left to right and back again. This prevents the illusion of rotatory motion which is likely to occur if the update proceeds in a strictly clockwise or counterclockwise fashion around the fly.

Although, in principle, there is no limitation to the choice of parameters r_a/r_d and f_e, the condition |f_e|_max<r_a/r_d must be observed to prevent the fly from ‘touching’ a virtual panorama. Fig. 4 illustrates the influence of the actual position of a walking fly on the position of a virtual panorama with two identical landmarks. The example refers to a ratio r_a/r_d=2 and a viewpoint eccentricity factor of f_e=+2. These parameters simulate an increase in the actual walking speed by a factor of 2 and allow the fly to just ‘touch’ the virtual panorama.

The refresh rate of a virtual panorama in the present investigation is limited by the time required (i) to obtain an instantaneous template of a selected pattern that is supposed to move at a given speed in a given direction, (ii) to determine the actual position and orientation of the freely walking fly, and (iii) to convert the template into a virtual open-loop panorama simulating the suppression of visual feedback from the rotatory and translatory components of locomotion.

A compensation of the overall rotatory displacements due to changes in the orientation of the fly is a most essential first approximation of this goal. Local deviations from the overall rotatory displacements depend on both the position and the changes in the position of the walking fly. The deviations are expected to increase with the ratio r_f/r_s, where r_f is the distance of the fly and r_s the distance of the surrounding LED screen, both from the centre of the arena. In the present set-up, the mean (0.14) and the maximum (0.21) values of this ratio are small enough to keep deviations from the overall rotatory displacements within a moderate range. A complete compensation of the rotatory and translatory displacements due to changes in the orientation and position of the walking fly can be obtained at the expense of calculation time. The same holds for the compensation of the varying elevation of virtual objects due to changes in their distance to the fly. Only moderate changes can be expected for the reasons discussed above. Moreover, unlike rotatory displacements, the flies ignore elevation changes in some of the object-selection tests described by Schuster (1996). These arguments justify the low priority given to the compensation of the distance-related elevation effect in the present experiments. A compensation of locomotion-induced translation and elevation includes the control of expansion/contraction of visual objects in horizontal and vertical direction, respectively.

To obtain a sufficiently high refresh rate in the following experiments, the complete compensation of locomotion-induced rotation, translation and elevation has been occasionally reduced to the compensation of rotation and translation (e.g. in experiments with vertical stripes of maximum height, where the elevation effect is absent). A further reduction to the compensation of only the overall rotation was not necessary in the present investigation. The refresh rate and the extent of compensation will be specified in the descriptions of the different experiments. Unless otherwise stated, the results are presented as means ± S.E.M.

Optomotor reactions have been extensively studied under open-loop conditions where a walking test fly is held in a fixed position and orientation to the visual stimulus. Would freely walking flies follow optomotor stimuli under natural closed-loop conditions? Undisturbed proprioception indicating that the substratum is actually at rest might overrule the rotation signalled by the visual system. To address this question, the course-control response of single flies was recorded in a series of 5 min trials in a panorama where the vertical stripes of a sinusoidally or a rectangularly luminance-modulated grating with a 36 ° spatial wavelength rotated at 72 ° s⁻¹ (i.e. at a contrast frequency of 2 spatial periods s⁻¹) in a horizontal direction on the cylindrical screen of the LED arena. This corresponds to 60 revolutions of a grating in 5 min. The maximum luminance and contrast of the sinusoidal or rectangular modulation were I_max=60 cd m⁻² and c_max=0.94, respectively. The control of rotation, translation and elevation of the pattern according to the position and orientation of the fly was switched off in this experiment. Any eccentricity of the fly in the arena would account for an apparent modulation of the angular speed of the grating along the horizon of the visual field. Increasing with the fly’s distance from the centre of the arena, the modulation amplitude in the present set-up varies between 0 and 21 % of the overall angular speed of the grating. However, the optomotor course-control response of Drosophila depends mainly on the contrast frequency, the rate at which spatial periods of the grating pass the receptive field of an EMD (Götz, 1964). This parameter of the motion stimulus is, in general, independent of the distance of the receptive field from the grating.

In all flies, course control was strongly biased in the sense of rotation of the grating. The flies invariably walked in loops, sometimes with a radius close to zero (Fig. 5A). With either sinusoidal or rectangular luminance profiles of the grating, some individuals were able to compensate entirely, throughout the test, for the average perceived rotation. Path increments and angular displacements were sampled every 200 ms. An ‘overall gain’ is defined as the sum of the recorded angular displacements of a fly, divided by the 60 revolutions of the grating during a test period of 5 min. The means of the overall gain obtained with either sinusoidally or rectangularly modulated gratings in these and the following experiments (Fig. 6; walking, any speed) were not significantly different from each other and have been combined accordingly (paired t-test, two-tailed: P⪢0.05). The mean overall gain at c_max=0.94 amounted to 0.76±0.03 (N=16 flies, n=32 trials).

With all other parameters constant, the optomotor stimulation was gradually weakened by lowering the contrast. This was done by increasing the luminance of the dark elements of the patterns, while the maximum luminance was kept constant. The resulting overall gain is shown in Fig. 6. At very low contrast (c=0.014), the trajectories of the flies lost their rotatory bias (see Fig. 5B). For both sinusoidally and rectangularly modulated gratings, the contrast-dependence of the degree of overall compensatory gain of freely walking flies was compared with the contrast-dependence of the open-loop flight torque measured in earlier experiments (Götz, 1964) with female flies of the same wild-type (Berlin) strain and of a genetic eye colour variant, the mutant sepia, with almost identical optomotor behaviour. Comparison of the two sets of data in Fig. 6 shows the qualitative similarity of contrast-dependent reactions in fixed flying or freely walking fruit flies.

In another experiment, the flies were stimulated by pattern motion at full contrast (c_max=0.94) but low peak luminance I=0.008I_max, achieved using a cylindrical grey filter in front of the LEDs. The mean overall gain was 0.49±0.02 (N=16 flies, n=32 trials). The data on the right of Fig. 6 show the influence of luminance on the optomotor response.

In the present experiments, the flies seemed to modulate their optomotor reactions by leaving a variable slip speed between the rotation of the panorama and their turning response, and possibly also by varying the fraction of time spent walking. We therefore calculated an ‘overall gain while walking’ from all of the 200 ms path increments in which the actual walking speed exceeded 4.3 mm s⁻¹ (Fig. 6; walking, >4.3 mm s⁻¹), a lower limit corresponding to approximately 10 % of the maximum speed of straight walking in wild-type (Berlin) flies (Strauss and Heisenberg, 1990). For all contrasts where c>0.03 tested in Fig. 6, the ‘overall gain while walking’ was significantly higher than the corresponding ‘overall gain’ (Fig. 6, walking, any speed; c>0.03, P<0.0005; c⩽0.03, P>0.05). At contrasts higher than 0.3, the overall gain while walking indicates an almost full compensation of the average motion stimulus and does so even at the very low upper luminance level I_max of 0.5 cd m⁻² (Fig. 6, data on the right). Nevertheless, the overall gain while walking also decreases gradually with decreasing contrast of the rotating pattern. Hence, for sufficiently weak optomotor stimuli, freely walking flies have an overall retinal slip both while walking and during phases of locomotor inactivity. On average, walking speed exceeded 4.3 mm s⁻¹ for approximately one-third of the total test time. However, at the highest contrast and luminance, flies reached this criterion for approximately two-thirds of the total test time. The remainder of the time was characterized by 200 ms intervals spent walking at an average speed of ⩽4.3 mm s⁻¹, turning on the spot or resting. Resting is frequently observed when a fly has reached the outer rim of the elevated disk. The distribution of the angular displacements derived from the manifold of 200 ms time intervals can be described by the full width at half-maximum of a modified Gaussian function fitted to the data given in angular coordinates (not shown). The result, 36±6 ° (mean ± S.D.), seems to be largely independent of the contrast applied in the experiments of Fig. 6 (correlation, significance test: P>0.05).

In all of the following experiments, single flies were stimulated by the horizontal or vertical motion of eight dark parallelograms in different configurations on a bright background (Fig. 7). The background luminance was I_max=60 cd m⁻², the contrast of the dark figures was c_max=0.94. The parallelograms were oriented such that a pair of optomotorically passive edges, each extending over a viewing angle of approximately 30 °, was always oriented parallel to the direction of motion shown by the white arrow, while the other pair of optomotorically active edges (viewing angle approximately 42 °) was oriented obliquely at an angle of 45 ° to the direction of motion (viewing angles are given as they appear from the centre of the arena). The investigation comprises two experimental conditions and their mirror-symmetrical counterparts. In one experiment (Fig. 7A), the parallelograms rotated clockwise (cw) around the fly. As shown by the black arrows, the motion of the oblique edges consists of two equivalent orthogonal displacements: cw rotation and upward translation. The same holds for a complementary experiment with parallelograms moving upwards (Fig. 7B). Developed 360 ° aspects of the panorama show the distribution of the horizontally (Fig. 7E) and vertically (Fig. 7F) moving figures at an arbitrarily selected time and 0.25 s later. At any given time, the extent of the effective edge elements was constant and identical for rotational and translational stimulation. This also applies to the contrast frequency, the ratio of the absolute rotational speed to the spatial wavelength of the pattern.

The contrast frequency of a steadily moving periodical pattern determines the response of the correlation-type motion detectors (Hassenstein and Reichardt, 1956; Reichardt, 1961) which were found to be implemented in Drosophila (Bülthoff and Götz, 1979; Götz, 1964). A contrast frequency of 2 Hz was achieved at an absolute speed of 180 ° s⁻¹ for horizontal rotation and 128 ° s⁻¹ for vertical translation. (Raising the vertical translational speed to 180 ° s⁻¹ while keeping the contrast frequency of 2 Hz constant would require more spacing between the successive elements and consequently a variation in the total length of simultaneously visible oblique edges.) To exclude possible effects of asymmetries in the set-up or in the reactions of the fly, the experiments were repeated using mirror-symmetrical parallelograms which either rotated counterclockwise (ccw) around the fly (Fig. 7C) or moved upwards (Fig. 7D). Again, the corresponding motion of the oblique edges consists of two equivalent orthogonal displacements: ccw rotation and upward translation.

In the first set of natural closed-loop experiments, one fly at a time walked freely on the circular disk while a camera traced its position and orientation. Each fly was tested in a series of 5 min experiments under four different conditions selected in permutated order from Fig. 7A–D. Again, the control of rotation, translation and elevation of the patterns according to position and orientation of the fly was switched off in these experiments. A comparison of the results obtained with the cw-rotating (Fig. 7A) or upward-moving (Fig. 7B) parallelograms and the corresponding results obtained in the mirror-symmetrical ‘ccw set’ of experiments (Fig. 7C,D) gave no indication of systematic asymmetry in the responses of the 12 flies investigated. For each fly, the ratio of the reactions to upward translation and sideways rotation, Q=R_vertical/R_horizontal, was derived from the corresponding measurements of the ‘overall gain’ of the induced turning (see Recording devices in Materials and methods). On average, the rotatory course-control responses to the apparent stimulus elicited by the vertical upward translation of the objects were only approximately one-fifth of the response to true horizontal rotation (Q=0.19±0.03; Fig. 8, natural closed-loop).

The results of the above closed-loop experiments seemed to be incompatible with related observations on the housefly Musca domestica and blowfly Calliphora erythrocephala (Borst et al. 1993), according to which equal responses to upward motion and sideways rotation would have been expected. Does Drosophila melanogaster have at least a partial solution to the ‘aperture problem’ that the larger flies do not have? To answer this question, the previous set of experiments was repeated under real open-loop conditions, in which single flies were held with their head and body in a fixed position and orientation at the centre of the arena. The test fly was allowed to walk on an airflow-supported styrofoam ball devised to measure intended turning in revolutions per metre path length (Buchner, 1976). All four of the conditions described in Fig. 7A–D were presented to each fly in a permutated order. Each experiment ended after a fixed amount of forward motion on the ball. It seemed advisable to relate the rotational component to the walking activity of the fly rather than to the total measuring time since, in real open-loop experiments, tethered flies spend more time in a resting state. Again, all of the 10 flies investigated turned in the expected direction and no systematic asymmetries were detected in their responses to mirror-symmetrical pairs of stimuli. However, under these conditions, the course-control responses induced by vertical translation and the responses to true horizontal rotation were strikingly similar (Q=1.00±0.04; Fig. 8, real open-loop).

The marked differences between the results of open-loop and closed-loop experiments were an incentive to study, for the first time, the responses of single flies in a ‘virtual’ open-loop situation. In this novel paradigm, the test fly on the disk in the arena was allowed to receive all of the proprioceptive and inertial input incidental to free walking. However, the panorama shown on the surrounding LED screen was continuously modified such as to minimize, in the freely moving fly, the amount of visual feedback from its translatory and rotatory self-motion. This requires continuous adaptation of the angular position, magnitude and elevation of the constitutents of the panorama. To achieve this goal, the position and orientation of the test fly were continuously measured by the CCD camera mounted above the disk. The incoming signals were processed on-line and conveyed to the display-control program described in Materials and methods.

Virtual open-loop conditions, in the initial set of experiments, were established by complete compensation of locomotion-induced rotation, translation and elevation at a refresh rate of 5 Hz. The moving parallelograms were again displayed in one of the four variants shown in Fig. 7A–D. With the fly at rest in the centre of the arena, the events on the LED screen were identical to those in the previous sets of experiments. However, this basic ‘motion picture’ was now continuously re-established in an attempt to suppress the visual feedback received from the self-motion of the freely moving fly. These conditions combine a comparatively high spatial accuracy of feedback suppression with a comparatively low refresh rate. Nevertheless, all of the 10 flies investigated under virtual open-loop conditions turned in the expected direction. We found no systematic asymmetry in the responses to mirror-symmetrical pairs of stimuli. Surprisingly, with a refresh rate of 5 frames s⁻¹, the ratio Q of the reactions to upward translation and sideways rotation of the corresponding parallelograms (Q=0.81±0.15; Fig. 8, virtual open-loop, 5 Hz) is not significantly smaller than the ratio of 1 found in the ‘real open-loop’ experiments described above (one-tailed t-test: P>0.05).

Would an enhanced refresh rate further improve the simulation of open-loop conditions? While stored templates of a panorama can be displayed at 1000 frames s⁻¹, the refresh rate of ‘kinetic’ templates is limited by the time required to calculate the azimuth and elevation of every pixel according to the actual position and orientation of the fly. In order to increase the refresh rate further, we omitted the compensation for object elevation that includes the vertical component of object expansion and contraction. This allowed us to repeat the experiments with 10 flies at a refresh rate of 10 frames s⁻¹. The resulting ratio Q then became almost indistinguishable from the ‘real open-loop’ response ratio of the immobilized flies (Q=0.96±0.04; Fig. 8, virtual open-loop, 10 Hz; t-tests, differences to either 1 or 0.81±0.15: P>0.05).

The next set of experiments is part of a series in which we varied the positive visual feedback of self-rotation required to suppress, in a ‘virtual open-loop’ situation, the horizontal displacement of a retinal image when the test fly changes its orientation. Accordingly, the angular displacement of the retinal image induced by rotational motion of the fly was undercompensated by 50 % (coupling constant 0.5), appropriately compensated by 100 % (coupling constant 1.0; simulation of ‘real open-loop’ conditions), overcompensated by 150 % (coupling constant 1.5) or uncompensated (coupling constant 0; ‘closed-loop’ conditions). Displacements, in these experiments, were always appropriately compensated by 100 % for translation, whereas elevation was left uncompensated. The measurements were carried out with the same 10 flies in permutated order. The results (Fig. 9) represent the ratio Q of the reactions to upward translation and sideways rotation under these conditions. The result of the present closed-loop experiment (0.22±0.03) is not significantly different from a corresponding result in Fig. 8, which was obtained under ‘natural closed-loop’ conditions, i.e. without compensation of the translating displacements (0.19±0.03; two-tailed t-test: P⪢0.05). Possible reasons for the low impact of the translatory displacements in this experiment are outlined at the end of Materials and methods.

In the natural closed-loop experiments of Figs 8 and 9, the flies appear to be comparatively immune to the horizontal-motion illusion elicited by vertical motion of the oblique edges of the parallelograms B or D in Fig. 7. The flies seem to ‘realize’ that the vertical edges of these parallelograms do not follow the horizontal-motion illusion. To show that visual feedback from the vertical edges does indeed account for the suppression of the response to the motion illusion, we repeated the closed-loop experiments with the vertical motion of parallelograms A or C in mirror-image configurations resembling Fig. 7E, but with more evenly distributed parallelograms. The absence of vertical edges in these parallelograms (which in previous experiments were moving only horizontally) almost completely restored the strong responses to the horizontal-motion illusion found in the open-loop and virtual open-loop experiments: a direct comparison of the reactions to the upward translation and sideways rotation of corresponding configurations of parallelograms in 12 flies yielded a mean ratio of Q=0.96±0.11.

Vertical motion of the parallelograms in Fig. 7A–D was required to create a horizontal-motion illusion in the previous experiments. The closed-loop responses to this illusion reached the expected upper limit for the parallelograms without vertical edges (Fig. 7A,C), but only one-fifth of this upper limit for parallelograms with vertical edges (Fig. 7B,D). The observed immunity to the horizontal-motion illusion was almost completely absent under open-loop or virtual open-loop conditions.

Do vertical edges facilitate fixation of the otherwise identical parallelograms? To answer this question, we offered, to a fly on the disk in the arena, a choice between the four parallelograms in Fig. 7A–D, each of which was presented stationarily in one of the four quadrants of the cylindrical screen. The tracking device recorded the attempted approaches from the centre towards each of these targets. The preference −1⩽sP_v⩽s+1 for the parallelograms with vertical edges was derived from P_v=(n_v−n_h)/(n_v+n_h), where n_v and n_h are the numbers of approaches towards the parallelograms with vertical edges (Fig. 7B,D) or horizontal edges (Fig. 7A,C), respectively. To prevent a hypothetical choice reinforcement by olfactory traces, the two pairs of objects were exchanged for each of the 10 flies under investigation. The results obtained either with the centres of all parallelograms aligned at half-height of the LED screen (P_v=0.16±0.04, n=200) or with the parallelograms presented almost touching the floor (P_v=0.13±0.04, n=200) show a small but significant preference for the parallelograms with vertical edges (one-tailed t-test: P<0.005). This preference seems to be independent of the different elevations of the stationary objects in the two experiments (two-tailed t-test: P>0.05).

The results in Fig. 6 show that strong optomotor turning can be elicited in a freely walking Drosophila melanogaster, although the mechanosensory input received under closed-loop conditions is incompatible with the visual stimulus. The size, luminance and maximum contrast of the newly developed LED arena are sufficient to induce continuous turning up to a full compensation of the overall rotatory slip between stimulus and response. The contrast-dependence of the reaction matches the contrast-dependence of the torque recorded under open-loop conditions during tethered flight in a rotating striped drum (Götz, 1964). The sinusoidally modulated gratings used in some of the present experiments were achieved by adjusting the duty cycle of the 2 kHz light pulses delivered to the LEDs. The pulse rate exceeds the flicker-fusion frequency of Drosophila. This may explain the comparatively close correspondence between the reactions to the motion of steady or flickering stripes.

How does graded rotation with the surroundings come about? Variation in the ratio between the time spent compensating the overall rotatory stimulus and the time of optomotor inactivity did not explain the results. The contrast-dependent, gradual adaptation of the overall gain while walking (Fig. 6; walking, >4.3 mm s⁻¹) excludes this possibility: the actively compensating flies tolerate retinal slip increasingly as the contrast gets weaker. By definition, ‘overall gain while walking’ does not include the presumably graded contributions from occasional episodes of turning on the spot. The fraction of one-third of the total time spent in walking seems to be almost independent of the actual contrast. Approximately the same fraction was also found in arena experiments with a stationary visual environment (Strauss and Heisenberg, 1993). Only the strongest optomotor stimulus used in the present study (I_max=60 cd m⁻², c_max=0.94) increased the average fraction of walking activity to approximately two-thirds of the total test time.

Under virtual and under real open-loop conditions, the motion of objects with identical oblique edges either in a vertical direction (Fig. 7B,D) or in a horizontal direction (Fig. 7A,C) elicited approximately the same yaw responses. This is the expected outcome if no further visual processing beyond directional selective integration and temporal filtering is implemented in Drosophila. In the first stage, the fly perceives motion through a mosaic of small apertures, each of which is the size of the receptive field of one EMD. In principle, a single unit can only yield the motion component perpendicular to the ‘visible’ line segment of an extended contour, whereas the motion component parallel to this line segment is indeterminate. This limitation has been termed the ‘aperture problem’ (e.g. Wallach, 1976). Two classes of strategies are known to overcome it. (i) First-stage information about local velocities at each of several distant loci can be combined in certain ways to infer global motion in a second stage of processing. (ii) Global motion can be recovered from local measurements by the analysis of additional parameters from the same patch.

Parallelograms were chosen because they would facilitate second-order processing of visual information such as motion calculation along connected contours or figure binding in order to infer the global direction of motion (e.g. Hildreth, 1984; Horn and Schunk, 1981; Movshon, 1990; Ullman, 1983) or processes that specifically evaluate singular points in the image, such as the corners or endpoints of contours (e.g. Lorenceau et al. 1993; Nakayama and Silverman, 1988). Human observers unambiguously perceive the correct global motion direction of the parallelograms in Fig. 7. Vertical motion of these objects, however, was quantitatively misperceived by flies that were walking under virtual and under real open-loop conditions. The results shown in Fig. 8 exclude motion perception by higher-order visual processes such as motion perception according to the generalized gradient scheme (Srinivasan, 1990), a powerful local algorithm for overcoming the aperture problem by combining six spatio-temporal filters that compute different first-order partial derivatives from the same patch. Applied to the parallelograms used here, the algorithm would easily have provided the actual direction of motion.

The present study was not intended to deduce information about local algorithms involving second-order spatial derivatives (Reichardt et al. 1988; Reichardt and Schlögl, 1988; Uras et al. 1988). For a parallelogram, these terms vanish with the exception of the corner points. Borst et al. (1993) have shown experimentally that such algorithms are not implemented, at least in the two dipteran species Musca domestica and Calliphora erythrocephala.

In summary, open-loop responses in Drosophila melanogaster can be explained sufficiently by a summation of the output of the individual EMDs of the course-control system. The comparatively small optical aperture of an EMD quantitatively prohibits the unambiguous perception of the actual motion direction of an edge.

Our investigation of the course-control responses to the ‘horizontal-motion illusion’ elicited by vertically moving oblique edges has revealed two opposite aspects of the ‘aperture problem’ arising from the ambiguity of the direction of global motion that can be perceived within a limited receptive field.

• So far, analysis of the aperture problem has focused mainly on the case of a stationary observer in an open-loop situation. The present results from the corresponding open-loop or virtual open-loop experiments demonstrate quantitatively the inability of Drosophila to cope with this problem. The course-control response to vertically moving objects with oblique edges reaches an upper limit that is determined by the course-control response to a comparable horizontal motion of these objects. Although Drosophila is endowed with a dense net of EMDs for vertical motion components (Buchner and Buchner, 1984) that is used in visual flight control (Buchner et al. 1978; Götz and Wehrhahn, 1984), the freely walking flies do not seem to distinguish between the different directions of object motion in these experiments.

• Repetition of the previous open-loop experiments under closed-loop conditions drastically reduces the illusion-induced response to only one-fifth of its upper limit. This surprising result suggests that, with the visual feedback of self-motion received under natural conditions, the aperture problem cannot be a serious impediment to the fly.

The unexpectedly strong reduction in the illusion-induced response was achieved exclusively by the visual feedback from the vertical edges of the parallelograms. We now discuss three tentative explanations.

Steady-state optomotor course-control responses elicited by continuous motion of a periodic pattern in a flying or walking Drosophila melanogaster depend on the contrast frequency, a term describing the ratio of the angular velocity to the angular period of the pattern (in hertz) (Götz, 1964, 1975). The results of the open-loop and virtual open-loop experiments in Fig. 8 show the equivalence of the responses elicited at a contrast frequency of 2 Hz either to the illusion induced by vertical motion or directly to horizontal motion of comparable arrays of the same parallelograms. To a first approximation, the qualitatively different stimuli may be characterized exclusively by their associated contrast frequency. Under closed-loop conditions, vertical motion of the parallelograms with vertical edges (Fig. 7B,D) is likely to establish an equilibrium between two antagonistic stimuli: in one direction, the horizontal motion component of the oblique edges (2 Hz) diminished by a stimulus equivalent to the fly’s unknown rotatory motion with the illusion (xHz); in the other direction, the horizontal motion of the vertical edges (0 Hz) increased by the stimulus equivalent to the fly’s rotatory motion (xHz). The condition for the equilibrium of the two stimuli is 2−x=0+x or x=1. On average, the fly is expected to follow the horizontal-motion illusion at half angular speed. This simplifying consideration may explain a reduction of the illusion-induced response to half rather than to one-fifth of its upper limit.

Visual feedback of self-motion enables Drosophila to maintain a course towards a selected visual object (Heisenberg and Wolf, 1988). Horizontal fixation of a parallelogram in the frontal visual field requires relative motion (Reichardt, 1973) expected in the present set of experiments only under closed-loop conditions. Object fixation obtained under these conditions would counteract the horizontal-motion illusion – but does this effect really account for a substantial reduction of the response to the illusion? Two findings seem to exclude this possibility. (i) Missing reduction of the closed-loop response to upward motion of parallelograms without vertical edges (Fig. 7A,C). Fixation of these objects during vertical motion and tracking of the same objects during horizontal motion should have reduced the ratio Q of the rotatory reactions to below its actual value of 0.96 (Fig. 8). (ii) Excessive reduction of the closed-loop response to upward motion of parallelograms with vertical edges (Figs 7B,D, 8). As described above, a 15 % preference for these parallelograms has been observed in free choices between the four stationary objects shown in Fig. 7. This effect is most probably too small for a substantial reduction of the response to the illusion. However, upward motion according to Fig. 7F leaves vertical bars of continuous background illumination between adjacent parallelograms. A hypothetical fixation of these bars might have contributed to the reduction of the investigated response.

It is not easy to exclude this possibility. Shifting successive rows of parallelograms in a horizontal direction is likely to replace the hypothetical bar effect by a detrimental ‘barber pole’ motion illusion.

The surprisingly weak influence of object fixation on the response to the horizontal-motion illusion in the present experiments may be explained by the simultaneous presentation of eight parallelograms according to Fig. 7E,F.

Optomotor course-control and object fixation in a visual environment might not be the only options a fly can use to hold its course. Stabilization of the course with respect to a visual landmark might also be achieved by an operant trial-and-error procedure that is based on the ability of the fly to superimpose upon its trajectory minute voluntary test turns in alternating directions and to compare these turns with the accompanying retinal event: an oscillation of the retinal image of the landmark about its actual average position. The fly perceives a continuous reversal of the induced motion, provided that the trajectory does not significantly change the average position of the retinal image. Drift of the retinal image counteracts the reversal of the induced motion. To stabilize the average retinal image of a landmark, the fly has to select the trajectory that is associated with the highest possible rate of motion inversion.

This entirely hypothetical explanation of operant course-control is based on assumptions that have been partially verified in another context. Drosophila uses operant strategies if course-control manoeuvres are required to achieve an artificially associated goal (Wolf and Heisenberg, 1986, 1991; Wolf et al. 1992). The fly is particularly sensitive to a sudden inversion of the self-induced visual feedback (Wolf and Heisenberg, 1990) and is capable of focusing its attention on selected regions of its visual field (Wolf and Heisenberg, 1980; Götz, 1994). The attention may be focused preferentially on one of the vertical edges of the parallelograms, where the highest possible rate of motion inversion is achieved with a straight trajectory. To lock into one of the oblique edges would require continuous turning with approximately the angular speed of the horizontal-motion illusion. Drosophila possibly rejects this artificial situation.

Such hypothetical models might explain the almost complete suppression of the horizontal-motion illusion. So far, we have no convincing evidence for the application of operant strategies in the present paradigm.

Until now, the behaviour of freely walking flies has been studied in natural closed-loop experiments mainly by evaluating their trajectories on a disk surrounded by a comparatively simple stationary panorama consisting of two (‘Buridan’s paradigm’; Götz, 1980) or more inaccessible visual objects. Fig. 4 shows a virtual version of the Buridan experiment. Optomotor reactions, in contrast, have been analysed under real open-loop conditions in stationary walking flies. Until recently, panoramic visual motion stimulation could only be achieved using mechanical solutions, some in conjunction with wide-angle axial projection on a cylindrical screen. Cathode ray tubes (CRTs) are versatile in pattern generation, but only in a restricted area of the visual field (for references, see Introduction). The LED display described here permits a coherent 360 ° cylindrical stimulation at a maximum luminance of at least 60 cd m⁻², a contrast ranging from 0.014 to 0.94, and a maximum refresh rate of 1000 frames s⁻¹ at a pixel resolution of approximately 2 °×2 ° as seen from the centre of the arena. This facilitates the design of artificial closed-loop experiments with flies that are walking freely on a disk or flying stationarily on a torque meter or wingbeat processor.

The virtual open-loop situation described in the present paper is just one special case of artificial closed-loop experiments (Schuster, 1996; Schuster and Götz, 1996; Schuster et al. 1995). The fivefold difference between the responses to the horizontal-motion illusion obtained under natural closed-loop and real open-loop conditions is a sensitive indicator of the reliability of the virtual open-loop conditions achieved in this novel paradigm, namely the temporal and spatial resolution, the refresh rate and the delay time. As shown in Fig. 8, a refresh rate of 5 Hz for a complete compensation of the locomotion-induced rotation, translation and elevation of the surroundings already seems to come close to eliminating the visual information originating from the fly’s self-motion. At 10 Hz, the results become almost indistinguishable from those recorded under real open-loop conditions. To obtain this rate, we had to omit the analysis of vertical object elevation and expansion/contraction according to the nearness of the fly. However, this particular component of the refresh procedure appeared to be less important for perfect simulation than an increased refresh rate. Does the delayed compensation of visual displacements due to self-motion elicit spurious optomotor reactions? This is conceivable: whenever the fly turns to the left or right, the panorama follows the fly by means of a delayed positive feedback. An optomotor response to the delayed stimulus could, in principle, reinforce the previous turn. Does this putative reaction account for the response ratio Q=R_vertical/R_horizontal of approximately 1 obtained under virtual open-loop conditions? None of the present results supports this view. Prolongation of the panorama-refresh time from 100 ms to 200 ms should increase the response ratio Q if there were such a delay effect. However, a moderate decrease in Q was found in the experiments of Fig. 8. The same holds for the transition from compensation (coupling constant 1.0) to overcompensation (coupling constant 1.5) of the fly’s self-rotation in the experiments of Fig. 9. Instead of the increase in the response ratio Q expected to result if there were reinforcement of turning, a highly significant reduction in Q was recorded (two-tailed t-test: P<0.0005). Owing to their relative motion over the retina during both overcompensation (1.5) and undercompensation (0.5), the vertical edges became behaviourally relevant. Their perception slowed down the optomotor responses of the fly during vertical stimulation. Accordingly, real and virtual open-loop conditions at a refresh time of 100 ms and a coupling constant of 1.0 must have been almost equivalent in the present experiments, despite considerable differences in the proprioceptive and inertial inputs incidental to free or tethered walking. At least in the present experiments, the responses of the flies do not depend on their mobility.

It is tempting to apply virtual open-loop conditions to other problems, such as the reafference principle proposed by von Holst and Mittelstaedt (1950) to explain (i) the performance of voluntary movements in animals equipped with reflexes for the compensation of unintended displacements and (ii) the discrimination between sensory inputs originating from active or passive displacements (Heisenberg and Wolf, 1988; Mittelstaedt, 1971; Varjú, 1990;,von Holst and Mittelstaedt, 1950; Wolf and Heisenberg, 1990). The separation of visual input induced by active or passive displacements in a virtual open-loop situation may help to elucidate controversial details of this important principle.

The question remains of how flies can afford to confuse the principal directions of motion to the extreme degree found in the present open-loop and virtual open-loop experiments. Borst et al. (1993), who reported a similar degree of deception in their open-loop experiments on Musca domestica and Calliphora erythrocephala, pointed out that, under natural conditions, antagonistic illusionary portions of the visually perceived motion may cancel each other out if the direction-selective sum of the local contributions of the manifold of contours in the visual environment is considered. While this is likely to diminish the aperture problem of a moving or resting fly, it seems important to stress the fact that a natural condition is always a closed-loop situation. The present closed-loop results demonstrate that a freely moving fly, which is actively influencing its visual input, can further reduce its aperture problem in a very efficient way.

The present investigation stresses the significance of a dimensionless factor Q=R_vertical/R_horizontal, the ratio of the course-control responses to the vertical and horizontal motion of the leading and trailing oblique edges of appropriately oriented parallelograms. This factor varied over a surprisingly wide range between Q=1.0 (real open-loop experiments on stationarily walking flies) and Q=0.2 (natural closed-loop experiments on freely walking flies). Two results are of importance. (i) Factor Q can be interpreted as the ‘degree of deception’ about the actual direction of pattern motion, an illusion attributed to the limited ‘aperture’ of the motion-detecting subunits of a visual system. The factor Q=1.0 found in the open-loop experiments indicates the lack of an algorithm for solving the ‘aperture problem’ in Drosophila melanogaster. However, the solution of this problem by a sophisticated algorithm does not seem to be necessary: the reduction to Q=0.2 shows that the aperture problem tends to disappear under natural closed-loop conditions. (ii) In its function as a quality factor, Q assesses the simulation of real open-loop conditions in the newly introduced ‘virtual open-loop’ experiments on freely walking flies. The quality factors obtained at panorama refresh rates of 5 Hz (Q=0.81±0.15) and 10 Hz (Q=0.96±0.04) prove the reliability of virtual open-loop conditions obtained in a novel locomotion-controlled LED panorama.

We gratefully acknowledge the enthusiasm and precise work of Manfred Herre, Karl-Heinz Hofmann, Karl-Heinz Stahl, Dieter Wirth and their coworkers in the mechanical and electronic workshops of our institute in assembling the LED arena. We would like to thank our anonymous referees for quite a number of helpful comments and suggestions. We are indebted to Diane Blaurock for preparing several drawings and for polishing the manuscript. Alexander Borst kindly contributed valuable suggestions.