Honeybees learn the sign and magnitude of reward variations

Food is heterogeneously distributed in space and time, and the behavioural ecology of foraging aims to understand how foragers cope with such heterogeneity (e.g. Pyke, 1884; Kamil and Roitblat, 1985; Stephens and Krebs, 1986). For example, nectivorous insects can face unpredictable scenarios within a time scale of hours, for the amount of nectar found within flowers fluctuates continuously. This happens because of low and variable rates of floral nectar secretion, and because of the activity of various pollinators foraging simultaneously on the same flowers (e.g. Núñez, 1977; Vogel, 1983). Like many other animals, honeybees have evolved strategies to cope efficiently with variations in food availability (e.g. Núñez, 1966). In doing so, they rely on their memory store in order to decide when, where and how to forage. Honeybees learn the location of food and the time of day when such food is available, as well as the odours, colours and shapes of the flowers they visit (e.g. von Frisch,1967). Recently, we found that they also learn that reward level increases or decreases over time (Gil et al., 2007).

In our previous work (Gil et al.,2007), we trained honeybees to forage on a flower patch closely resembling a natural food source. The bees first had to associate two different colours with either the presence or the absence of a sugar reward,and then inspect the rewarding flowers in order to find either constant or variable volumes of sucrose solution throughout a series of visits to the patch. We then examined whether and how their experience with variable volumes of sugar solution changed their subsequent behaviour in the absence of food. We found that those bees that had experienced increasing volumes assigned more time to flower inspection when tested 24 and 48 h after training, and that they behaved differently neither because they were fed more or faster nor because they had more strongly associated the related predicting signals. Thus, we found that the food-gathering behaviour of honeybees can be subject to changes at a later time on the basis of memories about a specific property of reward; namely, that its level changes over time. These and other results suggest that honeybees have a built-in change detector that computes the sign of variations in the level of reward, a computation which is followed by estimates of expected rewards stored as reward memories(Gil et al., 2007; Gil et al., 2008; Gil et al., 2009).

Behavioural adjustments which depend upon the formation and subsequent activation of this type of memory are said to be the result of reward expectations (Tolman, 1959; Logan, 1960; Schultz, 2000). Expectations of reward are thought to denote complex cognitive abilities, and have been studied extensively in vertebrate species (e.g. Crespi, 1942; Peterson et al., 1978; Holland and Straub, 1979; Watanabe et al., 2001; O'Doherty et al., 2001). Such studies aim to understand the regulation of purposeful behaviour and the complexity of decision making and planning. However, because expectations of reward are frequently studied in the laboratory, little is know about how animals use them in the field. Honeybees provide a good opportunity to study the development of reward expectations under conditions closely mimicking natural foraging situations. It has been shown, for instance, that they make use of stored estimates of reward in order to control their recruitment behaviours (Raveret-Richter and Waddington, 1993; De Marco and Farina, 2001; De Marco et al.,2005), and that they are able to map both past and present rewards to actions so as to maximize their food intake rates(Greggers and Menzel, 1993; Bitterman, 1996; Waddington and Gottlieb, 1990; Wiegmann et al., 2003). In line with this, our previous results suggest that the ability to develop expectations of reward enables honeybees to regulate purposeful behaviour in the absence of reward, and even after relatively long periods of time(Gil et al., 2007). However,we do not know yet whether honeybees learn not only the sign but also the magnitude of reward variations. Such a capability would make it more likely for them to maximize their individual and collective rates of food collection;for instance, by increasing their chances of finding food when forage is heterogeneously distributed in space and time. Here, we present the results of an experiment addressing these issues.

In our experiment, bees had to forage individually on a flower patch offering low flow rates of sugar solution. They then experienced either a large or a small increase in reward level, or, instead, a decrease in reward level. Next, we examined how they searched for food at the site in the absence of reward and after a long foraging pause. In doing this, we pondered the effect of the bees' energy balance during foraging. We had two predictions. First, that the bees will search for food more persistently after having experienced increasing reward levels than after having experienced decreasing reward levels. And, second, that they will search for food more persistently after having experienced a large increase in reward level than after having experienced a small increase in reward level. We discuss our findings in the context of learning and foraging behaviour.

We placed a colony of Apis mellifera carnica Pollman 1879 bees in a two-frame observation hive, and trained marked bees to collect unscented 50%w/w sucrose solution at an artificial three-flower patch placed 80 m from the hive. These bees were not used as experimental subjects, but only to recruit nest-mates to the patch. The newcomers arriving at the patch were first trapped before having any contact with the offered solution, and then marked and immediately released. They thus became potential experimental subjects. Those that returned to the patch underwent a pre-training phase, and became the actual experimental bees.

The patch consisted of a single acrylic cylinder (4.5 cm diameter, 5 cm height) with three centred holes (flowers) placed 1 cm apart from each other. Each hole had a small container (40 mm diameter, 50 mm depth) connected to a specially designed feeder by means of a plastic cannula. Detailed descriptions of our feeder have been given elsewhere(Núñez, 1966; Núñez, 1970). Here, it will be sufficient to say that it delivered sugar solution to the bees at constant and adjustable flow rates, and that it was connected to the three flowers of the patch by means of three separate cannulas. Thus, at any given time during the experiment, the three separate flowers offered similar flow rates of sucrose solution. The overall flow rate offered by the patch always arose from the sum of the flow rates offered by each of these flowers. Each experimental bee had to introduce its head into each of the plastic containers in order to reach the sugar solution offered at the bottom of the flowers. Therefore, it had to learn how to handle the flowers in order to access the offered reward efficiently. Before training, each bee was allowed to forage on the patch once, in a so-called pre-training phase, in which it was fed ad libitum with unscented 50% w/w sucrose solution. As during training (see below), only one bee at a time underwent pre-training. Any other bee landing on the patch was first captured before having any contact with the offered reward, and then kept inside a small cage until the end of the experiment. Training began immediately after pre-training, as soon as the experimental bee returned to the patch. It involved four successive foraging excursions by that bee. During training, the patch offered unscented 20% w/w sucrose solution at two different flow rates. The experimental bee was first presented with one of these two different flow rates during its first two visits to the patch, and then with the other flow rate, either higher or lower(see below) than the preceding one, during its last two visits to the patch. We used three different flow rates of sucrose solution: high (15 μl min^–1, or 5 μl min^–1 per flower), medium(9 μl min^–1, or 3 μl min^–1 per flower)and low (3 μl min^–1, or 1 μl min^–1 per flower). These flow rates of sucrose solution lead to food-intake rates which are well within those that bees achieve while foraging on natural patches of flowers (Núñez,1977).

The experiment had four different experimental series: one decreasing series and three different increasing series. In the decreasing series(henceforth, S-15/3), we presented the bees first with the highest (visits 1-2) and then with the lowest flow rate of sugar solution (visits 3-4). In the first increasing series (henceforth, S-3/15), the bees were given first the lowest (visits 1-2) and then the highest flow rate (visits 3-4). In the second increasing series (henceforth, S-3/9), they were given first the lowest(visits 1-2) and then the medium flow rate (visits 3-4). And, finally, in the third increasing series (henceforth, S-9/15), they were given first the medium(visits 1-2) and then the highest flow rate (visits 3-4). Details of these treatments are shown in Table 1. During training, we recorded the foraging behaviour of the bees. Next, between training and testing, we removed the flower patch from the feeding site. Finally, testing began 24 h after training. During testing, the patch did not offer a sugar reward. Under these circumstances, we recorded the behaviour of the bees on their first two visits to the patch. We call these consecutive visits `first' and `second' test, or test 1 and 2.

Our analysis focused on the following variables: (1) the visit time during each training visit (in s), as the amount of time that each bee spent collecting food at the patch during each of the four successive training visits; (2) the total visit time during training (in min), as the sum of the visit times recorded in the four successive visits to the patch; (3) the volume of sugar solution collected during each training visit (in μl),computed as the product between the visit time and the flow rate of sucrose solution offered by the patch; (4) the total volume of sugar solution collected during training (in μl), as the sum of the volumes of sucrose solution that each bee collected in its four successive visits to the patch;(5) the visit time per test (in s), as the time that each bee spent searching for food at the patch per test; (6) the overall visit time during testing (in s), as the sum of the visit times recorded in both tests; (7) the cumulative inspection time per test (in s), as the amount of time that each bee spent searching for food inside the flowers per test; (8) the overall inspection time during testing (in s), as the sum of the single cumulative inspection times from both tests; (9) the number of flower inspections per test, as the number of times in which each bee introduced its head into any of the three flowers of the patch in each of the single tests; (10) the overall number of inspections during testing, as the sum of the number of flower inspections from both tests; and (11) the time elapsed between the two tests, or the inter-test time.

The data concerning the visit times and volumes of each training visit were analysed by means of one-way repeated-measures ANOVA, and LSD multiple comparisons. The variables concerning the tests, i.e. visit time, cumulative inspection time and number of inspections, were analysed by means of two-way repeated-measures ANOVA, with repeated measures on only one factor, the test;the other being the experimental series. If the interaction effect was significant, then we examined the single effects of the series and test by means of one-way ANOVA, with LSD multiple comparisons and paired t-tests, respectively. The remaining variables were analysed by means of one-way ANOVA with LSD multiple comparisons(Zar, 1996). All statistical analyses were done using the software Statistica(www.statsoft.com).

We compared the volume of sugar solution that the single bees collected during each of their four training visits, as well as the amount of time(visit time) that they spent foraging on the patch(Table 2). In S-15/3, the volume decreased significantly from visits 1-2 to 3-4, while the visit time changed in the opposite way (Table 2). In contrast, in both S-3/15 and S-3/9 the volume and visit time increased and decreased from visits 1-2 to 3-4, respectively(Table 2). The bees of S-9/15 collected similar volumes in their four visits to the patch, and their visit times decreased significantly from visits 1-2 to 3-4(Table 2). We also compared the total volume of sugar solution that the bees of the different series collected during training, and the total amount of time that they spent forging on the patch (Table 2). We found that the total volume of solution collected by the bees was maximal in S-9/15,intermediate in S-3/9 and minimal in both S-15/3 and S-3/15, and that the total visit time was maximal in S-3/9, intermediate in both S-15/3 and S-3/15,and minimal in S-9/15.

The day after training, we recorded the behaviour of the bees in the absence of reward during their first two visits to the patch. We shall refer to these visits as the `first' and `second' test, or test 1 and test 2. We first compared the results of the different series within each test. In test 1, we found that both the visit time and the cumulative inspection time were maximal in S-3/15 and S-9/15, intermediate in S-3/9, and minimal in S-15/3(Fig. 1A,C). We also found that the number of inspections was maximal in S-3/9 and S-9/15, intermediate in S-3/15, and minimal in S-15/3 (Fig. 1E). In test 2, the visit time was longer in S-3/15 than in the other series (Fig. 1B), and the cumulative inspection time was maximal in S-3/15, intermediate in S-3/9, and minimal in S-9/15 and S-15/3 (Fig. 1D). Also, the number of inspections was maximal in S-3/15,intermediate in S-3/9 and S-9/15, and minimal in S-15/3(Fig. 1F).

Next, we compared the results of the different tests within each series. We found that the visit time, the cumulative inspection time and the number of inspections for the bees of S-15/3 and S-3/15 did not change across tests(Fig. 1). The same happened in S-3/9, where neither the visit time nor the cumulative inspection time changed across tests (Fig. 1A–D). For S-3/9, however, the number of inspections was higher in test 1 than in test 2 (Fig. 1E,F). In S-9/15,both the visit time and the cumulative inspection time were longer in test 1 than in test 2 (Fig. 1A–D). The same was true for the number of inspections,which was higher in test 1 than in test 2(Fig. 1E,F).

Finally, we pooled data from the two tests and made an analysis of the overall test performance of the bees (Table 3). Thus, we found that the overall visit time was significantly longer in S-3/15 than in the other three series. Moreover, the overall cumulative inspection time was maximal in S-3/15, intermediate in both S-3/9 and S-9/15, and minimal in S-15/3. The overall number of inspections was maximal in both S-3/15 and S-3/9, intermediate in S-9/15, and minimal in S-15/3. The time elapsed between the first and the second test varied markedly across individuals, ranging from 8 to 144 min. Overall, it did not differ between series, although it was significantly shorter in S-3/15 than in S-3/9(planned comparison, P=0.03).

In this study, we asked whether honeybees learn the sign and magnitude of reward variations. In our experiment, bees first had to forage individually on a flower patch offering variable reward levels, and then search for food at the patch at a later time and in the absence of reward. We quantified three different but connected behavioural measures; namely, the duration of the visits that the bees made to the unrewarding patch, the number of unsuccessful flower inspections, and the cumulative duration of these inspections. These three variables co-vary, clearly, for a forager first has to find flowers,then land on them and, finally, inspect them in order to find food. But they also denote behaviours which are expected to be subject to specific feedback and context-dependent functions (Weiss,1971). Hence, we considered not only one but all of them as different manifestations of a honeybee's persistence to forage for food. Our conclusions are therefore based upon the joint outcome of these three measures.

In test 1, we found that the bees of all the increasing series searched for food more persistently than the bees of the decreasing series(Fig. 1A,C,E). They did not behave differently because they were fed more or faster(Table 2), as seen from the fact that the bees of the different increasing series behaved similarly after having collected different amounts of food for different amounts of time during training (Fig. 1A,C,E; Table 2). These results are in close agreement with earlier findings (Gil et al., 2007), and mean that an increase in reward level induces long-term behavioural changes by itself, and that this phenomenon can be largely invariant to the levels of reward and the final crop loads of the bees. In test 2, we found that the bees that had experienced the larger increase in reward level searched for food more persistently than the bees that had experienced the smaller increase in reward level(Fig. 1B,D, grey bars). As before, they did not behave differently because they were fed more or faster,as seen from the fact that the bees of S-9/15 collected more food, and faster,than the bees of S-3/15 (Table 2), but the bees of S-3/15 searched more persistently in test 2(Fig. 1B,D). Also, the bees of S-9/15 collected as much solution as the bees of S-3/9 in less than half the time (Table 2), but these two groups of bees, which had experienced a similar increase in reward level,behaved similarly in test 2 (Fig. 1). Taken together, our results support the view that honeybees have a built-in detector of variations in the level of reward(Gil et al., 2007; Gil et al., 2008; Gil et al., 2009). Theory poses that such a detector should compute not only the sign but also the magnitude of variations in the signal supplied by the corresponding sensor. Here, we show that honeybees can learn both the sign and magnitude of an increase in reward level. Interestingly, the bees that experienced the larger increase in reward level behaved similarly in the two tests, but the bees that experienced the small increase in reward level searched for food less persistently in test 2 than in test 1 (Fig. 1). Such a reduction was even more conspicuous in the bees that had experienced the highest reward levels(Fig. 1, S-9/15). Hence, an interaction must exist between the effect of an increase in reward level and that of the reward level itself.

One may wonder why the bees of the increasing series behaved differently in test 2, but not in test 1 (Fig. 1, grey bars, A,C,E vs B,D,F). We suggest that the strength of a forager's reliance upon previous experiences may become detectable only after one or more unsuccessful visits to the site. In line with this argument, bees with positive expectations will search for food for longer periods at an exhausted site (Fig. 1A,C,E, white vs grey bars). And because a honeybee's likelihood of searching for food at an empty site diminishes over time(Núñez, 1966),bees that experience a large increase in reward level would subsequently search for food more persistently than bees that experience a small increase in reward level (Fig. 1B,D,F,grey bars). Our results indeed support such a view, and thereby suggest that an increase in reward level leads to the formation of expectations enhancing a forager's reliance on a feeding site, and that the strength of such reliance increases with the magnitude of the previous increase in reward level.

Taking into account the results of both tests, one sees that the bees that experienced the largest increase in reward level not only searched for food more persistently but also spent less time in the hive between tests(Table 3). This is interesting because it is in the hive that foragers are exposed to cues and signals from other colony members, which they use to regulate their ongoing activities(e.g. von Frisch, 1946; von Frisch, 1967; Ribbands, 1954). Further, as food source profitability diminishes, foragers stay in the hive for longer periods of time between their foraging excursions(Núñez, 1966; Núñez, 1970; Grosclaude and Núñez,1998; De Marco and Farina,2001), which makes it more likely that they will be exposed to signals and cues from other colony members. Given that honeybees cooperate by sharing newly discovered food sources, it would be interesting to examine the relationship between a forager's past experience with variable rewards and its chances of being recruited to new food sources.

Expectations of reward are thought of as behavioural adjustments that depend upon the formation and subsequent activation of memories about specific properties of reward (Tolman,1959; Logan, 1960; Schultz, 2000). Studies on the reward expectations are important for understanding purposeful behaviour, and have been addressed in pigeons (e.g. Peterson et al., 1978),rodents (e.g. Crespi, 1942; Logan, 1960; Holland and Straub, 1979),non-human primates (e.g. Watanabe et al.,2001) and humans (e.g. O'Doherty et al., 2001). Here we show that honeybees develop long-term expectations that depend upon the sign and magnitude of past reward variations. These expectations help them to regulate their search behaviour in the absence of reward, and even after long periods of time. The rates of nectar secretion found in natural flowers are low and variable (Núñez,1977; Vogel, 1983; Baker and Baker, 1983). Nectar availability can either increase or decrease within hours depending on species-specific flowering patterns, weather conditions and the activity of various pollinators (Núñez,1977; Rathcke and Lacey,1985). A forager's ability to expect future rewards makes it more likely that it will compete better with other flower pollinators for limited resources. In this scheme, honeybees will make full use of past information about food so as to finally gain an advantage in cost effectiveness during flower inspection. Further, because each forager works in a way that optimizes the food collection of the whole colony, it would be interesting to investigate how the colony as a whole benefits from a forager's ability to develop reward expectations; for example, by studying the behaviour of honeybees foraging on multiple feeders offering increasing, decreasing and constant reward levels.

Optimal foraging theory poses that foragers assess patch quality using an optimization rule which tends to maximize their rates of energy gain(Charnov, 1976). First, each forager sets a threshold level of net energy gain. If it visits an above-level patch, then it forages until the level falls below expectation. By contrast,it abandons the patch when it is below level. Thus, food availability determines a forager's investment of time/energy during food collection(Charnov, 1976). However,although effort has been made to incorporate how learning and memory adapt to the problem of foraging (e.g. Kamil and Roitblat, 1985), optimal foraging theory does not capture how foragers control these investments in the absence of reward(Pyke, 1984). Furthermore, it has been reported that the behaviour of dogs(Devenport and Devenport,1993), least chipmunks and golden mantled ground squirrels(Devenport and Devenport,1994), pigeons (Mazur,1996) and rats (Devenport et al., 1997) can be predicted by the so-called temporal weighting rule (Devenport and Devenport,1994). According to this rule, the behaviour of animals dealing with variable rewards depends upon their memories about either the most recent reward level or the average reward level experienced at the feeding site,depending on the time elapsed between past and present experience. Our results do not match predictions from this model, however (see Results). Hence, a comprehensive model about how foraging decisions adapt to past reward variations is still lacking, and new observational and theoretical evidence is necessary to explain how honeybees and other animals adjust their behaviours in relation to the sign and magnitude of past variations in the level of reward.