Many species from diverse and often distantly related animal groups (e.g. monkeys, crows, fish and bees) have a sense of number. This means that they can assess the number of items in a set – its ‘numerosity’. The brains of these phylogenetically distant species are markedly diverse. This Review examines the fundamentally different types of brains and neural mechanisms that give rise to numerical competence across the animal tree of life. Neural correlates of the number sense so far exist only for specific vertebrate species: the richest data concerning explicit and abstract number representations have been collected from the cerebral cortex of mammals, most notably human and nonhuman primates, but also from the pallium of corvid songbirds, which evolved independently of the mammalian cortex. In contrast, the neural data relating to implicit and reflexive numerical representations in amphibians and fish is limited. The neural basis of a number sense has not been explored in any protostome so far. However, promising candidate regions in the brains of insects, spiders and cephalopods – all of which are known to have number skills – are identified in this Review. A comparative neuroscientific approach will be indispensable for identifying evolutionarily stable neuronal circuits and deciphering codes that give rise to a sense of number across phylogeny.