The stalk-eyed fly as a model for aggression – is there a conserved role for 5-HT between vertebrates and invertebrates?

Aggressive behavior is ubiquitous for gaining access to desirable resources such as territory, food and mates (Edwards and Herberholz, 2005; Summers et al., 2005a,b), and hence aggression is critical for determining individual fitness. However, fighting is energetically costly and potentially injurious. As a consequence, diverse species have evolved signaling strategies during aggressive encounters with conspecifics to minimize physical engagement, often comprising elaborate displays incorporating various morphological ornaments and armaments. Across the majority of animal taxa, the ability to modulate aggressive responses appears to be governed by monoaminergic activity (Alekseyenko et al., 2013; Hoopfer, 2016; Rillich and Stevenson, 2014; Zhou et al., 2008), with serotonin (5-hydroxytryptamine, 5-HT) playing a key role (Bubak et al., 2015; Takahashi et al., 2012). In stalk-eyed flies, 5-HT appears to mediate appropriate behavioral responses upon perception of aggressive signals (Bubak et al., 2014a).

5-HT, 5-HT receptor structure and function, and the 5-HT transporter (SERT), which removes 5-HT from the synaptic cleft to terminate 5-HT signaling (Fig. 1), are phylogenetically conserved (Blenau and Baumann, 2001; Martin and Krantz, 2014). Despite this, 5-HT appears to play generally opposing roles in the generation of the complex behaviors associated with aggression in invertebrates and vertebrates (see Table S1). However, we propose that this seemingly contrasting role of 5-HT may be an overly simplistic generalization. In this Commentary, we will briefly outline the known functions of serotonergic signaling in aggression across invertebrates and vertebrates (for more comprehensive reviews, see Alekseyenko and Kravitz, 2015; de Boer et al., 2016; Takahashi et al., 2012), combined with findings from our stalk-eyed fly model (Box 1), to demonstrate that 5-HT plays a much more nuanced role when factors such as receptor subtype, other neuromodulators and specific phases within aggressive interactions are taken into consideration. The emerging picture suggests that the serotonergic mechanisms governing invertebrate aggression may be more reminiscent of those of vertebrates than previously thought.

In most vertebrates, 5-HT is largely viewed as an inhibitory neuromodulator of aggression (Carrillo et al., 2009; de Almeida et al., 2015; Nelson and Chiavegatto, 2001; Summers et al., 2005a; but see de Boer et al., 2015, 2016). This interpretation is principally based upon studies showing that reductions in 5-HT in vertebrates typically increase aggression (Table S1; Audero et al., 2013; Caramaschi et al., 2008; Cervantes and Delville, 2007; Mosienko et al., 2012; Perez-Rodriguez et al., 2010). Conversely, augmenting 5-HT availability, through either dietary supplementation or reducing SERT-mediated 5-HT clearance, suppresses aggression (Höglund et al., 2005; Holmes et al., 2002).

In contrast to the effect in vertebrates, most studies suggest 5-HT increases aggression in invertebrates (Table S1). Acute 5-HT injection into the hemolymph of crustaceans induces subordinate males to re-engage in confrontations with dominant opponents while decreasing their willingness to retreat (Antonsen and Paul, 1997; Huber et al., 1997; Livingstone et al., 1980; Panksepp et al., 2003), and, in some species, increases the probability of winning a fight (Momohara et al., 2013). Retention of dominant status in crayfish is also enhanced by increasing synaptic 5-HT through SERT blockade (Huber et al., 1997; Momohara et al., 2013; Panksepp and Huber, 2002). However, the role of 5-HT is likely to be more complex than a simple enhancement of aggressive behavior. In paired fights with a size discrepancy between opponents, injection of 5-HT increases aggression in smaller crayfish while decreasing aggression in larger animals, suggesting that, as in vertebrates (Blanchard and Meyza, 2019), 5-HT may alter risk assessment (Bacqué-Cazenave et al., 2018). Manipulations that elevate 5-HT tend to increase aggression in several insect species (Alekseyenko et al., 2010; Bubak et al., 2014b; Dierick and Greenspan, 2007; Dyakonova and Krushinsky, 2013; Kostowski and Tarchalska, 1972; Szczuka et al., 2013). These studies, in which 5-HT is experimentally increased just prior to a conflict, may be consistent with the rapid and transient increase in endogenous 5-HT observed in highly aggressive individuals during vertebrate interactions (de Boer et al., 2015; Matter et al., 1998; Summers et al., 2005a; Takahashi et al., 2012). In contrast to findings in vertebrates, decreasing 5-HT function prior to interaction does not appear to affect subsequent expression of aggressive behavior in either male Drosophila or male crickets (Dierick and Greenspan, 2007; Rillich and Stevenson, 2018; Stevenson et al., 2000), suggesting 5-HT is permissive but not essential for invertebrate aggression.

There are a number of caveats that are problematic with respect to our understanding of the role of 5-HT in invertebrate aggression. First, although the 5-HT–aggression relationship has been well studied in a range of vertebrate models (de Boer et al., 2015, 2016; Table S1), relatively little attention has been paid to the role of 5-HT in aggression in invertebrates other than arthropods (but see Edsinger and Dölen, 2018). A second issue involves assessing behavioral changes following systemic injection of 5-HT. Insects and crustaceans have a hemolymph–blood barrier (HBB) that is functionally analogous to the vertebrate blood–brain barrier (Otopalik et al., 2012; Schirmeier and Klämbt, 2015), which should prevent diffusion of 5-HT from the hemolymph to the brain. However, systemic administration of 5-HT clearly influences aggression in lobsters (Antonsen and Paul, 1997; Bacqué-Cazenave et al., 2018; Huber et al., 1997; Momohara et al., 2013; Peeke et al., 2000) and ants (Kostowski and Tarchalska, 1972; Szczuka et al., 2013), suggesting that effects are modulated by mechanisms outside the brain, or that the invertebrate HBB is permeable to monoamines. The latter possibility is suggested by data from our laboratory showing that, in mantis shrimp (Neogonodactylus oerstedii), systemic dopamine and 5-HT both cross the HBB (K.J.R., unpublished results), and from studies indicating that systemically administered dopamine can directly alter nervous system development and locomotion in Drosophila larvae (Budnik et al., 1989; Wakabayashi-Ito et al., 2011). Thus, results obtained by systemically injecting 5-HT do not rule out potential confounds from neurohormonal or negative feedback effects rather than direct effects on the brain. Third, some vertebrate studies suggest that the degree to which 5-HT affects aggression may depend on individual social status established after repeated interactions. For example, aggression-reducing effects of elevating 5-HT in male lizards are only seen in dominant males (Summers et al., 2005b), and in hamsters and some teleost fish, 5-HT is associated with the acquisition and maintenance of subordinate status (Harvey et al., 2012; Backström and Winberg, 2017). Although differential actions of 5-HT following repeated fights and social status have been noted in crayfish (Huber et al., 1997) and male crickets (Rillich and Stevenson, 2018), the majority of invertebrate studies only utilize single interactions between unfamiliar opponents. Finally, most work on vertebrate aggression has focused on males, although there is some evidence suggesting 5-HT may increase or have minimal effects on aggression in female rodents (see Table S1; de Boer and Newman-Tancredi, 2016; Joppa et al., 1997; Terranova et al., 2016; Villalba et al., 1997; but see Heiming et al., 2013; Kästner et al., 2019). Even less is known about the role of 5-HT in female invertebrate aggression. Thus, there is a clear need for further studies using multiple species before conclusions about the activational or sex-specific role of 5-HT in aggression across invertebrates can be drawn. Such knowledge is crucial for understanding not only how individual or sex-specific aggression can be discretely modulated by 5-HT activity but also why functional homologies or differences in such a conserved neurotransmitter system would have evolved across vertebrates and invertebrates.

In vertebrates, progress has been made in understanding how 5-HT modulates aggression through differential binding of specific 5-HT receptors, with 5-HT_1A, 5-HT_1B, 5-HT₂ and 5-HT₃ subtypes being involved (Box 2; Juárez et al., 2013; Morrison et al., 2015; Popova et al., 2010; Takahashi et al., 2012). In general, systemic activation of each subtype dampens vertebrate aggression, but the opposite effect can be induced in mammals when these receptors are activated either in specific brain regions or during certain contexts such as maternal aggression or self-defense (de Almeida and Lucion, 1997; Takahashi et al., 2012). In contrast, the contribution of 5-HT receptor subtypes to invertebrate aggression is not as well understood. Of the seven known 5-HT receptor families in mammals, three (5-HT₁, 5-HT₂ and 5-HT₇) have been described with notable sequence and functional homology in insects (Box 3; Tierney, 2018; Vleugels et al., 2013). As in vertebrates, adenylate cyclase activity and cAMP production are decreased by G_i-coupled 5-HT₁-like receptors but increased by G_s-coupled 5-HT₇-like receptors to exert inhibitory and excitatory effects, respectively, whereas excitatory 5-HT₂-like receptors function through Gq proteins to stimulate phospholipase C and subsequently increase Ca²⁺ (Tierney, 2018; Fig. 1).

Similar to findings in vertebrates, 5-HT₁-like and 5-HT₂-like receptor subtypes are implicated in insect aggression (see Table 1). In male Drosophila, aggression is reduced by 5-HT₂ receptors but enhanced by activation of 5-HT_1A receptors (Johnson et al., 2009). Further, the role of each subtype is specific to the type of aggressive behavior, with 5-HT_1A receptors predominantly affecting low-intensity aggression seen at contest initiation, such as threat displays, whereas 5-HT₂ receptors mediate high-intensity aggressive behaviors, such as lunging (Johnson et al., 2009). Our recent studies using the stalk-eyed fly (Teleopsis dalmanni) indicate a similar role for 5-HT_1A and 5-HT₂ receptors, respectively, in enhancing and reducing aggression (Bubak et al., 2019), suggesting receptor subtype activation as one mechanism to explain the generally opposing role of 5-HT in vertebrate versus invertebrate aggression. However, the story is not as simple as this; consideration of other factors points to more similarities than differences between invertebrates and vertebrates in how 5-HT can modulate aggression. In the following sections, we provide a summary of our work on aggression using the stalk-eyed fly. With this model, we hope to expand our knowledge of the role of 5-HT in altering aggressive behaviors with respect to discrete components of an aggressive interaction, such as contest initiation, intensity and termination, and we hope to determine how the actions of 5-HT may differ between the sexes (Box 1).

Pharmacologically increasing neural 5-HT using the precursor 5-hydroxytryptophan (5-HTP) increases high-intensity (defined by contact) behaviors in male stalk-eyed flies (Bubak et al., 2014b). This is consistent with studies demonstrating a positive relationship between increased 5-HT and aggression in other invertebrates (Table S1; Antonsen and Paul, 1997; Bubak et al., 2016b; Huber et al., 1997; Livingstone et al., 1980; Panksepp et al., 2003; Momohara et al., 2013; Dierick and Greenspan, 2007). However, females pretreated with 5-HTP exhibit no difference in either behavioral output or fight outcome (Bubak et al., 2019). Thus, as in some vertebrates (Joppa et al., 1997; Terranova et al., 2016; Villalba et al., 1997), there appears to be a sex difference in how 5-HT modulates aggression in stalk-eyed flies. This is supported by sex differences in components of 5-HT signaling, with males having higher 5-HT_1A but lower 5-HT₂ receptor expression, whereas 5-HT₇ receptor expression is equivalent between the sexes (Bubak et al., 2019). In addition, males show much lower expression of SERT, which would presumably result in reduced 5-HT clearance. Combined, this suggests the higher levels of aggression displayed by males are a result of elevated levels of 5-HT acting at 5-HT_1A receptors. Further, these findings indicate that 5-HT₂ activation may inhibit aggression in male T. dalmanni. The fact that administration of selective 5-HT_1A and 5-HT₂ agonists increases or decreases, respectively, inter-male aggression in Drosophila supports this hypothesis (Johnson et al., 2009). Therefore, the lower expression of 5-HT_1A and higher expression of 5-HT₂ in female stalk-eyed flies may account for the difference in aggressive behavior seen between sexes of this species.

Results available from insect systems suggest some similarities in the role(s) of 5-HT receptors in modulating aggression when compared with vertebrates. In both vertebrates and arthropods, 5-HT₂ receptors appear to dampen aggression (Bubak et al., 2019; Johnson et al., 2009; Muehlenkamp et al., 1995; Takahashi et al., 2011; Ten Eyck, 2008), but in contrast to the majority of rodent studies, 5-HT_1A activation appears to increase insect aggression (Bubak et al., 2019; Johnson et al., 2009). However, this discrepancy in 5-HT_1A modulation of aggression may owe more to whether the selected agonist is acting on somatodendritic 5-HT_1A autoreceptors located presynaptically or on postsynaptic 5-HT_1A heteroreceptors in terminal fields. For example, rodent aggression is reduced by 5-HT_1A agonists when they are either injected directly into the dorsal raphe or given systemically (de Boer et al., 2016; Van der Vegt et al., 2003a,b; Calcagnoli et al., 2015), which seems contradictory, as activating autoreceptors in the raphe causes a reduction in 5-HT neuron firing and thus a decrease in 5-HT availability (Fig. 1), whereas activating postsynaptic 5-HT_1A heteroreceptors mimics increased 5-HT signaling. Although the latter fits with an inhibitory role of 5-HT in vertebrate aggression, the anti-aggressive effects of autoreceptor activation suggest that 5-HT stimulates aggression. In support of this, social challenge in rodents and lizards is accompanied by rapid phasic increases in 5-HT in various brain regions (Watt et al., 2007; Nakazato, 2013; Takahashi et al., 2015). Combined, the overlap in functional effects of receptor subtypes again points to more similarities than differences between invertebrates and vertebrates in 5-HT-mediated modulation of aggression.

Social isolation increases aggressive behavior in both vertebrates and invertebrates (Twenge et al., 2001; Wongwitdecha and Marsden, 1996; Alexander, 1961; Johnson et al., 2009). Male Drosophila reared in isolation also show changes in 5-HT receptor expression compared with group-housed males (Johnson et al., 2009). Similarly, socially isolated male stalk-eyed flies are more aggressive, performing more high-intensity behaviors and contest initiations than their socially reared opponent (Bubak et al., 2019). Socially isolated males also have lower 5-HT₂ expression levels than their socially housed opponents, while expression of 5-HT_1A and 5-HT₇ receptors appears to be independent of rearing condition. This pattern of receptor expression changes is opposite to that seen in Drosophila, where isolation reduces expression of 5-HT_1A while 5-HT₂ expression increases (Johnson et al., 2009). In contrast, social isolation has no effect on the expression or intensity of aggressive behavior or the expression of any of the 5-HT receptor subtypes measured in female stalk-eyed flies (Bubak et al., 2019).

Social isolation has similar augmenting effects on aggression in rodents. For example, male rats reared in isolation exhibit abnormally high levels of unprovoked and contextually inappropriate violent aggression (Toth et al., 2011). Similarly, socially isolated arthropods express high levels of aggression (Johnson et al., 2009; Sibbald and Plowwright, 2014; Stevenson and Rillich, 2013) and abnormal high-intensity attacks (Bubak et al., 2019) relative to socially raised controls. Further, serotonergic modulation of aggressive responses through changes in the expression of 5-HT_1A and/or 5-HT₂ has been implicated for both socially isolated vertebrates (reviewed in Veneema, 2009) and invertebrates (Bubak et al., 2019; Johnson et al., 2009; Yeh et al., 1996). The role(s) of the 5-HT receptors appears to be species specific in insects (Johnson et al., 2009; Bubak et al., 2019). Similarly, in vertebrates, there are species differences in both 5-HT receptor subtype expression following social isolation as well as the brain region affected (Bibancos et al., 2007; Preece et al., 2004; Ross et al., 2019; Schiller et al., 2003).

Use of small interfering RNA (siRNA) to selectively knock down 5-HT₂ receptors in stalk-eyed flies decreases the receptor expression by approximately 30% in males, similar to that observed following social isolation (Bubak et al., 2019). Behaviorally, siRNA-treated males initiate more fights but perform the same amount of high-intensity aggression compared with their vehicle-treated opponents. Female aggression, as with both isolation and 5-HTP pretreatment, does not change following reduction of 5-HT₂ receptors (Bubak et al., 2019). These findings also suggest 5-HT₂ may modulate the willingness of males to engage in a fight, whereas escalations to potentially injurious levels are mediated by a separate mechanism. This differs from Drosophila, where stimulation of 5-HT₂ receptors in isolated males reduces high-intensity aggression but not fight initiation (Johnson et al., 2009). However, social isolation increases expression of 5-HT₂ receptors in male Drosophila (Johnson et al., 2009), whereas the opposite effect is seen in stalk-eyed flies (Bubak et al., 2019). In male crickets, activation of 5-HT₂ receptors inhibits aggression in subordinates, but only after they have fought (Rillich and Stevenson, 2018). In contrast, aggressiveness during honey bee colony defense appears to be increased by 5-HT₂ activation (Nouvian et al., 2018). These differences among insects in how 5-HT₂ receptors mediate the type of aggressive behavior expressed, along with social isolation effects on receptor expression, highlight the need for studying multiple species before making general conclusions about the role of 5-HT in invertebrate aggression.

The results discussed above suggest some similarity in the role of 5-HT₂ receptors in vertebrates and invertebrates. In crickets, stalk-eyed flies and Drosophila, 5-HT₂-like receptors appear to inhibit components of aggressive behavior such as fight initiation, intensity and reduction of aggression after defeat (Bubak et al., 2019; Johnson et al., 2009; Rillich et al., 2019). Similarly, selective 5-HT₂ agonists are effective in decreasing aggression in several vertebrate species (Muehlenkamp et al., 1995; Ten Eyck, 2008; Takahashi et al., 2012; but see Juárez et al., 2013).

Serotonin modulates a variety of other neurochemical systems, including the neuropeptides tachykinin (Tk; invertebrate equivalent to substance P) and neuropeptide F (NPF; invertebrate equivalent of neuropeptide Y), each of which has been linked to aggressive behavior in vertebrates and invertebrates (Katsouni et al., 2009; Takahashi et al., 2012). Other neuropeptides, such as oxytocin and vasopressin, have been shown to be important mediators of mammalian aggression (Caldwell, 2017); however, the role played by their functional orthologs (inotocin in insects; oxytocin/vasopressin-like peptide in crustaceans) in invertebrate aggression is largely unknown (Gruber, 2014; Liutkeviciute et al., 2016), although a recent study showed no relationship between the expression of inotocin receptors and aggression in mated ant queens (Chérasse and Aron, 2017). In contrast, Tk/substance P increases aggression across both taxa (Asahina et al., 2014; Halasz et al., 2009; Katsouni et al., 2009), whereas NPY/NPF suppresses aggression (Dierick and Greenspan, 2007; Karl et al., 2004). Both neuropeptides are influenced by 5-HT activity (Guiard et al., 2007; Hennessy et al., 2017; Karl et al., 2004; Sergeyev et al., 1999). In stalk-eyed flies, manipulation of serotonergic function alters the expression of Tk and NPF to modulate specific components of aggressive behavior, and effects differ as a function of sex and receptor subtype (Bubak et al., 2019).

Social isolation increases both contest initiation and escalation exclusively in male stalk-eyed flies. In addition, there is an increase in Tk expression in isolated males that is not evident in females (Bubak et al., 2019). Recent work in male mice also shows that social isolation increases the expression of a closely related neuropeptide, tachykinin 2, in portions of the limbic stress circuit, and increased fear and aggressive behaviors can be blocked in these animals by tachykinin 2 receptor antagonists (Zelikowsky et al., 2018). Pretreating socially raised male stalk-eyed flies with 5-HTP to increase 5-HT also increases Tk and high-intensity aggression, but does not affect contest initiation. Comparing the effects of isolation and 5-HTP treatment suggests that Tk may primarily control behaviors associated with fight escalation, but not necessarily affect other less-intense aggressive behaviors (Bubak et al., 2019). Similarly, reducing Tk signaling reduces high-intensity attacks but leaves milder aggressive behaviors unaffected in rats (Halasz et al., 2009). In Drosophila, sexually dimorphic Tk neurons also regulate male aggression, but this extends to both low- and high-intensity aggressive behaviors (Asahina et al., 2014), initially suggesting that Tk regulation of discrete types of aggression may represent an evolutionarily derived state possessed by vertebrates. However, the finding that Tk is most closely associated with high-intensity aggression in male T. dalmanni argues against this, and instead points to convergence in Tk function at the level of the species rather than phylum.

The association between Tk and 5-HT in mediating aggression in vertebrates versus invertebrates is less clear. Tachykinin receptors on 5-HT neurons in the mammalian hindbrain can directly modulate neuronal firing and release of 5-HT in terminal regions (Maejima et al., 2013), and there is evidence for 5-HT and Tk co-release from neurons in mammals (Chan-Palay et al., 1978). In contrast, Tk and 5-HT do not appear to be co-localized in neurons in the majority of invertebrates (Boyan et al., 2010; Boyer et al., 2007; Chamberlain et al., 1986; Langworthy et al., 1997). However, the finding that 5-HTP treatment elevates both Tk expression and high-intensity behaviors in male stalk-eyed flies (Bubak et al., 2019) implies an interaction between 5-HT, Tk and aggression. Whether this represents a direct functional interaction as opposed to an additive effect produced by independent actions of 5-HT and Tk is unknown. Serotonin neurons do appear to synapse on to Tk-immunoreactive terminals in desert locust brain (Ignell, 2001), suggesting a direct relationship via synaptic contact that may also be present in T. dalmanni.

Willingness to engage in a fight increases following administration of 5-HT₂ siRNA in male but not female stalk-eyed flies, despite similar reductions in 5-HT₂ expression levels in the two sexes. This may result, in part, from a sex-dependent interactive role between the 5-HT₂ receptor and the NPF system, as knockdown of the 5-HT₂ receptor only reduces NPF receptor expression in males (Bubak et al., 2019). Although this is consistent with an inhibitory role reported for NPF in modulating aggression in Drosophila and mice, decreases in NPF/NPY in these species specifically suppress high-intensity behaviors (Dierick and Greenspan, 2007; Karl et al., 2004). In contrast, reductions in NPF receptor expression following 5-HT₂ siRNA treatment have no effect on expression of high-intensity aggression in male stalk-eyed flies (Bubak et al., 2019). Further, 5-HT and NPF pathways appear to act independently in regulating aggression in male Drosophila (Dierick and Greenspan, 2007) and mice (Karl et al., 2004), whereas a direct positive relationship between 5-HT₂ receptors and NPF is indicated for male T. dalmanni.

Combined, the findings from the stalk-eyed fly system generate a complex picture of interplay among serotonergic and peptidergic pathways that may fine tune the expression of aggressive behavior as appropriate for that particular context. These studies suggest that although 5-HT has a critical role in male aggression, precisely how the confrontation proceeds is governed by selective activation of 5-HT receptor subtypes along with changes in activity of NPF and Tk. Reductions in 5-HT₂ activation seem to promote the motivation to engage with an opponent, which may be potentiated by reductions in NPF signaling. Once committed, the two opponents typically express equivalent amounts of low-intensity aggressive behaviors, but a sharp increase in expression of high-intensity behaviors in the last stages of the confrontation is shown by those that eventually win the fight (Bubak et al., 2016a). Thus, while a balance of signaling in favor of 5-HT_1A versus 5-HT₂ receptor activation may be sufficient to initiate a confrontation and maintain expression of low-intensity aggression, the shift to high-intensity aggressive behaviors required for winning depends upon an additional mechanism, such as increased Tk signaling. In contrast, the lower levels of aggression in female stalk-eyed flies appear to be maintained by heightened 5-HT₂ receptor activity, with NPF and Tk having no apparent function in this behavior. These findings suggest that 5-HT modulation of aggression in this species is permissive or inhibitory depending on receptor subtype, intensity of aggression, neuropeptide involvement and the sex of the individual.

Several studies show that, as in invertebrates, increases in 5-HT in vertebrates are associated with enhanced aggression. For example, administration of 5-HTP to mice increases 5-HT turnover and aggression intensity (Kulikov et al., 2012). Similar results are obtained in insects following 5-HTP pretreatment (Dierick and Greenspan, 2007; Bubak et al., 2013). Other studies have shown that 5-HT increases in specific brain regions in several vertebrate species just before or during aggression (Summers et al., 2003; van der Vegt, et al., 2003a,b; Watt et al., 2007). Because 5-HT is an evolutionarily ancient neurotransmitter present in all animal lineages (Moutkine et al., 2019), it is perhaps not surprising that a broadly shared role in aggression has been retained in both phyla, as seen for modulation of other behaviors critical for survival such as feeding, motor control and reproduction (Weiger, 1997).

The 5-HT receptors that modulate aggression appear to be similar in structure and function in invertebrates and vertebrates (Tierney, 2018; Vleugels et al., 2013, 2015), although experiments studying the effects of 5-HT receptor function in invertebrate aggression are limited. In crickets, stalk-eyed flies and Drosophila, 5-HT₂-like receptors appear to inhibit aggression, whereas 5-HT_1A-like receptor activation increases aggression (Table 1), and these effects may be exerted postsynaptically (Alekseyenko and Kravitz, 2015). In vertebrates, these two receptor subtypes also appear to strongly influence the expression of aggression, but to have a dampening effect. As discussed above, the anti-aggressive effects of 5-HT_1A agonists in rodents may result from a depression of serotonergic activity/release through actions at presynaptic somatodendritic autoreceptors (de Boer and Newman-Tracredi, 2016), implying that increased levels of 5-HT actually have a facilitatory role in vertebrate aggression similar to that demonstrated for arthropods. Further, it could be argued that what were presumed to be opposing effects of 5-HT_1A receptors on aggression between invertebrates and vertebrates are largely dependent on the balance of presynaptic versus postsynaptic activation, and that activation of postsynaptic 5-HT_1A receptors mimicking elevated 5-HT release should promote aggression in both groups. This premise is supported, in part, by the finding that infusion of 5-HT_1A agonists into some 5-HT terminal regions of the rodent brain can enhance aggression (Takahashi et al., 2012). However, this is only seen with specific types of aggression, such as maternal defense or alcohol-enhanced aggression (Takahashi et al., 2012).

This raises an important point, in that the degree to which behavioral outputs differ between species may depend on the context in which the aggressive confrontation is taking place (e.g. Ling et al., 2010; Harvey et al., 2012; Backström and Winberg, 2017; Rillich and Stevenson, 2018), making it difficult to parse out specific mechanisms. Therefore, aggression should be investigated under different contexts and at different stages of the interaction (e.g. fight initiation, escalation, termination), particularly those relevant to the life history of the species or sex. For example, pitting female stalk-eyed flies in a forced-fight paradigm with food being the incentive may be a less powerful stimulus to provoke aggressive confrontations than access to egg-laying sites. To obtain a more fundamental understanding of how aggression is differentially modulated between the sexes by receptor subtypes, different combinations of selective knockdown or conditional gene overexpression could be linked with delivery of specific pharmacological agents. Work with Drosophila shows that strains can be created in which expression of genes controlling key aspects of neural signaling are restricted to particular brain regions, allowing fine-tuned analysis of how transmitters such as 5-HT mediate specific behaviors (Alekseyenko et al., 2014; Alekseyenko and Kravitz, 2015). Application of similar techniques to other arthropods could provide powerful tools for elucidating how and why aggression regulation by 5-HT has either diverged or remained similar in response to diverse evolutionary pressures.

So, does 5-HT have divergent or similar functions in aggression between invertebrates and vertebrates? Based on our studies using stalk-eyed flies, along with the available literature, the answer seems to depend on exactly how the question is posed. Systemically induced increases in 5-HT in invertebrates largely enhance components of aggressive behaviors (Table S1; Antonsen and Paul, 1997; Bubak et al., 2014a,b; Dierick and Greenspan, 2007; Huber et al., 1997; Livingstone et al., 1980; Panksepp et al., 2003; but see Stevenson and Rillich, 2017). In contrast, 5-HT historically has been viewed as an inhibitory neuromodulator of aggression in vertebrates (Table S1; Nelson and Chiavegatto, 2001; Summers et al., 2005a; Summers and Winberg, 2006). However, several studies suggest that the role of 5-HT in modulating aggression is more complicated, and depends on the subtype of 5-HT receptor activated, effects of 5-HT within specific brain regions, the type of aggression studied and the use of animal models selected for high aggression (de Boer et al., 2015, 2016; Nelson and Trainor, 2007; Takahashi et al., 2012). At least, elements of serotonergic function appear to be conserved phylogenetically.

The major classes of serotonergic receptors (5-HT₁, 5-HT₂ and 5-HT₇) are estimated to have diverged over 800 million years ago (Peroutka and Howell, 1994), with a subsequent differentiation and appearance of new 5-HT receptors (including the 5-HT_1A receptor) when ancestral vertebrates appeared some 600–700 million years ago (Peroutka and Howell, 1994; Blair and Hedges, 2005). Thus, there may truly be a broad phylogenetic divergence in how 5-HT_1A receptors influence aggression, with the promoting effects in invertebrates representing a more ancestral state that has only been conserved in the vertebrate brain for mediating specific types of aggression. In contrast, the aggression-inhibiting role of the evolutionarily older 5-HT₂ receptors appears to have been conserved phylogenetically. However, these hypotheses will remain speculative until additional studies with different invertebrate species are conducted. Further, there is some debate as to whether pharmacological agents used to manipulate specific 5-HT receptor subtypes in vertebrate studies are equally efficacious in their invertebrate orthologs (Vleugels et al., 2015; Tierney, 2018), and sophisticated genetic manipulations to target specific brain regions have primarily been restricted to Drosophila (Alekseyenko et al., 2014; Alekseyenko and Kravitz, 2015). Despite this, considerable evidence is steadily accumulating to suggest that there is indeed a shared facilitatory role for 5-HT in vertebrate and invertebrate aggression.

The authors appreciate the valuable comments and critiques from three reviewers and the editor Charlotte Rutledge, and help generating the figures and tables from Gabrielle Welsh.