Current approaches to fate mapping and lineage tracing using image data

Developmental biology, at its core, is concerned with one, fundamental question: how does a single cell give rise to the many different cell types, tissues and organs that comprise an adult organism? Although the simplest way to resolve this question would be to just follow that one cell along its journey to becoming an organism, this is quickly complicated by a host of technical issues that have stymied developmental biologists to this day. Direct observation of the embryo requires it to be at least somewhat transparent, able to survive artificial culture conditions and able to tolerate exposure to light. Methods such as interspecies transplantation, dye labelling (see Glossary, Box 1), electroporation (to introduce either dyes or genetic labels) or mosaic genetic labelling (see Glossary, Box 1) allow one to label a single cell or a small population of cells and visualize their location as the embryo develops (Vogt, 1929; Keller, 1976; Tam and Behringer, 1997; Lawson and Pedersen, 2007). However, these techniques are difficult to do on a large scale and generally only label small regions or populations of interest. More recent methods, such as DNA barcoding (see Glossary, Box 1), which labels cells with genetic tracers but can be technically challenging to obtain accurate cell lineages from (Kebschull and Zador, 2018; Masuyama et al., 2019; Salvador-Martínez et al., 2019), and single-cell RNA sequencing (see Glossary, Box 1) (Wagner et al., 2018; Farrell et al., 2018; Cao et al., 2019; Pijuan-Sala et al., 2019), from which lineage trajectories (see Glossary, Box 1) can be inferred with varying degrees of accuracy (Kester and van Oudenaarden, 2018; Baron and van Oudenaarden, 2019), provide information on large numbers of cells and their purported progeny. Lost with these methods, however, are the dynamics of cellular behaviour – how cells migrate, where and when they divide, how they interact with their neighbours, and largely everything in between the time when they were born to their final fate. To observe this, there is no substitute to being able to directly visualize and follow cells live in a developing embryo. Biologists have been attempting to do just this in one form or another for well over a century, yet it has only been in the last few years, with the advance of new light-microscopy methods, that we have been able to delve deeper and for longer periods of time into the developing embryo than ever before.

Beyond the introduction and advancement of new microscopes capable of imaging large and sensitive specimens (of which there has been an explosion in recent years; Lemon and McDole, 2020), biologists now have a wealth of new reporters, labels, sensors and probes with which to observe cells during development. Fluorophores now run the gamut of the visual spectrum, from the classical GFP to near-infrared proteins that excite in regions previously reserved for two-photon microscopy (Filonov et al., 2011; Shcherbakova et al., 2016; Matlashov et al., 2020). Optogenetics (see Glossary, Box 1) and photo-convertible proteins (see Glossary, Box 1) allow one to manipulate a system with light alone (Nowotschin and Hadjantonakis, 2009; Krueger et al., 2019), and live-cell sensors can report on everything from the dynamics of signalling pathways to rapid changes in voltage or calcium levels. ‘Visualizing development live’ is no longer restricted to merely watching blobs of nuclei as they wander about the embryo; we now have the ability to assess the complex behaviours of large numbers of cells all at once. We can visualize the temporal and spatial expression of genes, the complex behaviour of cells and tissues as they migrate, shape and fold. We can see cells as they transition from a naive, pluripotent state to a defined, functional cell, such as a twitching cardiomyocyte or an excitatory neuron. However, with this ability to visualize the dynamics of every cell in the embryo comes the even larger challenge of quantifying the dynamics of every cell in the embryo. With so much information now available from even a single time-lapse dataset, the human annotator cannot possibly cope alone. Fortunately, the generation of computational tools and methods needed to handle the deluge of ‘big data’ in imaging has advanced as rapidly and dizzyingly as the light microscopes that supply them.

The word ‘revolution’ is often thrown about when it comes to new techniques and advances in technology, but for the field of developmental biology these advances truly represent the start of a new renaissance era; we now have the ability to witness and examine embryonic development like never before. Combined with recent advances in genomics, such as single-cell RNA sequencing, the ability to couple the high spatial and temporal resolution of live imaging to precise and comprehensive information about a cell's transcriptional fate will enable researchers to examine in exquisite detail how a single cell becomes an embryo.

Some of the very first attempts to track cells in an embryo were carried out through observation with a simple compound microscope (Conklin, 1905) (Fig. 1). For simpler and very transparent organisms, such as Caenorhabditis elegans, this proved quite effective, if laborious (Sulston et al., 1983). With the explosion in new light microscopy methods such as light-sheet imaging (see Glossary, Box 1) (Huisken, 2004; Keller and Stelzer, 2008), not only have more traditional model organisms, such as Drosophila, mouse and zebrafish, been re-examined, but so too have more ‘exotic’ specimens, such as ascidians, Parahyle and pygmy squids (Wolff et al., 2017 preprint; Burnett et al., 2018; Guignard et al., 2020). For many applications, however, the use of these more ‘advanced’ and often experimental instruments is not required. More traditional methods such as point scanning (see Glossary, Box 1) or spinning disc confocal microscopy enable the tracking of 2D systems or ‘simpler’ 3D models, such as small organoids, thin tissues, cell monolayers or stem-cell clusters. These imaging methods are, however, unsuited to samples that are very large, very sensitive to light or, more commonly, a combination of both. In addition, imaging big samples or whole embryos requires a large field-of-view, as well as the ability to maintain the resolution needed to visualize single-cell behaviours, and do so rapidly and gently. It is for these reasons that the development of light-sheet microscopy has been such a boon to the field of developmental biology. Most embryos, whether they grow outside of the maternal environment or within, are extremely photosensitive and do not appreciate the extraneous illumination generated by confocal or wide-field microscopes (Icha et al., 2017). In confocal microscopy, although fluorescence emission is collected from the plane of focus, large parts of the specimen above and below are exposed to excitation light as it is swept across the specimen, irradiating regions that provide no useful information in return and leading to the accumulation of cellular damage in response to the absorption of additional photons. With light-sheet microscopy, optical sectioning is provided inherently by a very thin sheet of light, which provides high spatial resolution and only excites regions of the embryo that lie within the plane of focus. As such, no light is ‘wasted’ on regions where fluorescence emission is not actively being acquired, and the embryo is spared unnecessary exposure, reducing phototoxicity. Additionally, this thin sheet of light can be scanned very rapidly, and when combined with an opposing light sheet can cover even very large samples gently and with enough temporal resolution to follow rapid cellular behaviours. Many reviews have been written on the benefits and applications of light-sheet microscopy, which we will not go into detail for the purposes of this Review, but refer to the following for further reading (for example, see Weber and Huisken, 2011; Lim et al., 2014; Manderfield et al., 2015; Reynaud et al., 2015; Girkin and Carvalho, 2018; Wan et al., 2019a).

With the ability to image larger and larger samples for longer periods of time come significant computational and data challenges. Cinematic movies of embryonic development are always captivating, but unfortunately not particularly quantitative. To comprehensively track cells across embryo development, a number of strategies can be used, from clever fluorescent labelling to brute-force manual annotation to new and emerging machine-learning methods that strive to automatically segment and follow cells, reporting on their behaviours, shape changes and lineages with little human intervention.

Over the past 30 years, labelling strategies for lineage tracing have evolved together with imaging technologies in order to follow cells more comprehensively and to mark tissues with more flexibility. Fluorescent dyes and proteins have been engineered to be brighter, more photostable and enable deeper penetration into thick tissues (for comprehensive reviews and practical guides, see Yan and Bruchez, 2015; Cranfill et al., 2016; Jonkman et al., 2020). Recent advances in genomics, genome editing and optical techniques have made it even easier to tag cells in non-model organisms (Huang et al., 2016; Pomerantz et al., 2021). However, lineages can only be faithfully reconstructed from ubiquitously labelled samples if (1) cells can be unambiguously distinguished from their labelled neighbours; and (2) imaging is fast enough that the spatial context of a cell's surroundings is not dramatically different between time points (Meijering et al., 2009). These criteria can be challenging to guarantee in deep/light-scattering tissues where cells are densely packed or in systems sensitive to imaging with short time intervals. Mosaic genetic labelling (e.g. Brainbow; Weissman and Pan, 2015) can alleviate this problem by inducing random recombination of a multi-colour expression cassette, so that cells from different clonal progenies are permanently labelled with different colours. Lineage relationships can thus be recorded or inferred from the sparser labelling with less frequent imaging. Alternatively, cells or tissues at the intended location and stage can be selectively labelled using photo-activable (pa-) or photo-convertible fluorescent proteins (pcFPs) to visualise the targeted cell's progeny transiently before the induced FPs are diluted out during subsequent cell divisions. This strategy can also be coupled with targeted optogenetic manipulation to study mutant cells (He et al., 2020). Notably, it was recently demonstrated that many pcFPs can be engineered to be ‘primed-convertible’, i.e. converted under dual illumination of blue and red to near infrared (NIR) lasers, which allows for the confined targeting of small volumes by beam intersection (Dempsey et al., 2015; Klementieva et al., 2016; Mohr et al., 2017; 2016; Turkowyd et al., 2017; Welling et al., 2019). For a comprehensive review on the merits, limitations and scope of application of each cell labelling technique for cell tracking, we refer the reader to Buckingham and Meilhac (2011).

The capability to monitor and manipulate molecular processes during live imaging is a powerful tool to dissect the molecular and cellular mechanisms of development. In recent years, fluorescent labelling of DNA and RNA molecules in live cells has been deployed to study chromatin organization or transcriptional kinetics in multi-cellular developing systems (Berrocal et al., 2020; Bothma et al., 2014; Garcia et al., 2013; Liu et al., 2014). These methods often require subcellular or single-molecule resolution, which greatly benefit from the development of super-resolution imaging techniques (Chen et al., 2014; Li et al., 2015). Moreover, new biosensors are being actively developed to measure cell cycle (Zerjatke et al., 2017), apoptosis (Schott et al., 2017) or gene dynamics in general (Newman et al., 2011; Okumoto et al., 2012). Optogenetic tools of photo-sensitizer (e.g. KillerRed or SuperNova; Bulina et al., 2006; Takemoto et al., 2013) or photo-cleavable proteins (e.g. PhoCl; Zhang et al., 2017) can be used to precisely target cells for ablation or protein (in)activation, which offers deeper insight into the mechanisms of development.

Recent advances in high-throughput single-cell sequencing technologies have enabled the construction of lineage relationship and transcriptional trajectories of developing embryos from measurements of millions of individual cells with lineage barcodes (Wagner and Klein, 2020). However, because of the dramatic difference in the experimental modalities, lineages reconstructed by live imaging and by single-cell omics methods are usually placed on opposing sides in the minds of biologists. Contrary to popular belief, the two methods are actually highly complementary and can potentially form a powerful synergy to advance developmental systems biology (Liu and Keller, 2016). Spatial transcriptomics (see Glossary, Box 1) methods, e.g. sequential fluorescence in situ hybridsation (seqFISH) (Lubeck et al., 2014; Shah et al., 2016; Eng et al., 2019) and multiplexed error-robust FISH (MERFISH) (Chen et al., 2015; Moffitt et al., 2016; Xia et al., 2019), are especially attractive as they can achieve cellular-level gene profiling while preserving the spatial context of tissues. Conversely, a synthetic barcode recording system that denotes the lineage history of cells can also be read out in situ using MEMOIR (memory by engineered mutagenesis with optical in situ readout) (Frieda et al., 2017; Chow et al., 2021) or Zombie (Zombie is Optical Measurement of Barcodes by In situ Expression) (Askary et al., 2020). Although such methods go beyond the realm of live imaging, many existing microscopy and computational tools can be applied to analyse such data and to cross-validate lineage patterns and the underlying genetic and cellular mechanisms. The molecular trajectories predicted by in situ genomics and the cellular dynamics recorded by live imaging will greatly facilitate each other to discover new biology in the future.

Live imaging enables us to visualize dynamic developmental processes that could previously only be inferred from static snapshots. This provides a faithful record of highly dynamic cell behaviours, and being able to monitor a large number of cells simultaneously makes it possible to extract information that is both biologically and statistically meaningful. How do embryonic cells give rise to an animal with the correct shape and composition? Where do different tissues come from and how do they end up at the right location? ‘Seeing is believing’: visualizing developmental processes lays the foundation for formulating and testing hypotheses about morphogenesis and cellular dynamics.

The past decade has witnessed an explosion in not only the number of model systems that can be imaged live using fluorescence microscopy, but also the spatiotemporal resolution and the duration that development can be visualized with. The fast and gentle imaging capacity of light-sheet microscopy has enabled in toto reconstruction of embryogenesis at the single-cell level in many organisms, including C. elegans, Drosophila, zebrafish and mouse (Wu et al., 2013; Udan et al., 2014; Amat, 2014; Strnad et al., 2016; McDole et al., 2018; Shah et al., 2019; Welling et al., 2019). With the development of genomic and genetic techniques, fluorescent labelling of cell nuclei can now be achieved in embryos that were genetically less amenable in the past, and with great surgical precision (Huss et al., 2015; Benazeraf et al., 2017). This revives the classic work of embryologists and enables large populations of cells to be tracked simultaneously during prolonged embryonic development (Fig. 2). In vivo time-lapse imaging accompanied by cell tracking has provided a first glance of the overall cellular dynamics during the formation of many tissues, including the blood vessel (Arima et al., 2011), the zebrafish eye (Gordon et al., 2018; Azizi et al., 2020), the arthropod limb (Wolff et al., 2017 preprint), the heart (Ivanovitch et al., 2017; Yue et al., 2020) and many more. Live imaging of organoids (Held et al., 2018; Martyn et al., 2019; Benito-Kwiecinski et al., 2021) can provide unprecedented details of human cell behaviours that could be of clinical relevance.

Cell behaviours, such as proliferation, migration and shape change can be quantified from live-imaging data. At the single-cell level, this reveals crucial molecular and genetic mechanisms underlying the proliferation potential, motility and polarity of cells. Take live imaging of cancer cell progression as an example: computational tools (Kwak et al., 2010; Tsygankov et al., 2014; Barry et al., 2015; Tian et al., 2020) have been developed to track cells in a highly controllable 2D or 3D cultured environment to quantify their proliferation, morphology and migration dynamics. Clonal tracking in an induced breast tumour from epithelial acini revealed that tumours originated from clusters of cells, rather than isolated transformed cells (Alladin et al., 2020). Chemical or genetic manipulation of intercellular signalling and cell adhesion pathways have revealed molecular mechanisms underlying cancer cell migration (Biselli et al., 2017; Stallaert et al., 2018; Ilina et al., 2020), which are essential for understanding and controlling tumour metastasis. With the development of super-resolution and single-molecule imaging technologies, combining molecular dynamics and cell behaviours will provide deeper insight into the physiology of cancer cells in space and time.

At the tissue level, cell division, migration, rearrangement and shape change reflect how a developing system acquires its physical shape and form. Epithelial development is one of the best-studied examples of tissue morphogenesis, as many tissues originate from a 2D primordium. The flat sheet of progenitor cells is usually accessible to imaging, and its morphogenic features can be measured and modelled computationally (Khan et al., 2014; Reuille et al., 2015; Heller et al., 2016; Stegmaier et al., 2016; Etournay et al., 2016). By analysing orientated cell divisions, collective cell migration and cell shape change in the multicellular environment, biophysical models can be established to simulate epithelial spreading (Campinho et al., 2013; Lang et al., 2018), growth control (Puliafito et al., 2012), axis elongation (Wang et al., 2017) and folding (He et al., 2014; Monier et al., 2015). Notably, such models are often supported by measurement of the tissue's physical properties, such as stiffness and adhesion, as well as perturbation by genetic manipulation and/or laser ablation during development. The capacity for measuring and applying forces during live imaging is a powerful tool and a promising future direction for the study of tissue morphogenesis.

One important unsolved question in developmental biology is the reproducibility and variability of embryonic development. Do cells play dice? How different is embryogenesis from one individual to another? What constrains development so that individual animals are built with similar scale, shape and components? Key factors involved in governing this developmental robustness include intracellular gene regulatory networks and intercellular signalling (Naoki et al., 2019; Rohde et al., 2021 preprint). However, solving this problem relies crucially on our ability to image and analyse developing systems at the cellular level with high accuracy and in toto coverage, so that statistics from many individual embryos can be compared and assessed (Keller, 2013; Amat, 2014; Faure et al., 2016; Wan et al., 2019b; Hailstone et al., 2020). For instance, McDole et al. developed a multi-embryo registration framework termed TARDIS (time and relative dimension in space), whereby cell behaviours in space and time can be ‘averaged’ across different embryos to build a statistical fate map of post-implantation mouse development (McDole et al., 2018). Guignard et al. quantitatively measured cell lineage, cell geometry and cell fate of the highly invariant ascidian embryogenesis and found geometric control of cell-cell contacts to be the key factor ensuring reproducible fate specification (Guignard et al., 2020).

Live imaging is more than tracking cell movement. Cell lineages reconstructed from live imaging denote the complete developmental history from a progenitor cell, through rounds of division, relocation and differentiation, to specialized cell types that make up different tissues and organs. Hundreds of thousands of cells' lineages can be densely reconstructed from a single imaging session, which has significantly boosted the throughput and compensated for the spatial information missing from traditional clonal tracing methods. Empowered by live imaging, cell lineage reconstruction can now answer not only ‘which becomes what’, but also ‘what happens when, where and how’.

When are lineage identities specified, and how do they segregate spatially to different tissues during embryonic development? Whole-embryo imaging combined with cell fate identification offer tremendous information in global processes such as germ layer segregation and organogenesis (McDole et al., 2018; Shah et al., 2019). The reconstructed lineages, when carefully curated to guarantee accuracy, can be further utilized to answer questions at the single-cell level, for example, the (non-)existence of neuromesodermal progenitor cells in the zebrafish tailbud (Attardi et al., 2018) and the functional relationship between sibling cells in zebrafish spinal neurons (Wan et al., 2019b). From the observed lineage segregation events, we can infer the underlying molecular mechanisms that operate with the corresponding spatiotemporal patterns. Thus, it is essential to monitor or manipulate gene expression as we trace lineages. Delaune et al. found that mitotic events tend to happen at a certain phase of segmental clock gene oscillation (Delaune et al., 2012). Goolam et al. identified transcription factors that regulate differential fate bias in 4-cell mouse embryos and confirmed their roles through in vivo lineage tracing (Goolam et al., 2016). Live imaging can capture the instantaneous dynamics of gene expression and opens up unprecedented opportunities to uncover novel molecular and cellular mechanisms in cell cycle and cell fate determination (Plachta et al., 2011; White et al., 2016).

Cell lineages are of particular interest when it comes to tissue homeostasis and regeneration. Live imaging can reveal the location of the stem cells and characterize their behaviours in regenerating tissues. For example, in Parhyale limb regeneration, no specific stem cell population has been identified; instead, most epidermal cells are proliferative (Alwes et al., 2016). Live imaging of the regeneration of the Drosophila midgut (Martin et al., 2018) and axolotl spinal cord (Rost et al., 2016) revealed division orientation and division rate as essential factors in stem cell behaviour. Meanwhile, in self-renewing tissues, stem cell proliferation and differentiation need to be delicately balanced to maintain tissue homeostasis or continuous growth. When cultured ex vivo, neural stem cells follow a stereotypic lineage progression pattern from asymmetric to proliferative to terminal divisions, a programme that is largely cell-intrinsic (Costa et al., 2011). Conversely, nephron progenitor commitment was found to be a process mainly influenced by stochastic cell migration to different environments (Lawlor et al., 2019). Work by Rompolas et al. beautifully illustrated this balance by imaging epidermal tissue renewal in live mice, where they showed that stem cell commitment is delicately coordinated both temporally and spatially to achieve tissue homeostasis (Rompolas et al., 2016). Live imaging of stem cells can be used to reconstruct a large number of lineages, allowing niches to be identified and compared with each other, which would enable lineage patterns to be identified and the underlying mechanisms to be discovered.

These new advances in light microscopy that enable us to track cells and lineages as never before come, however, with their own new set of challenges. Not the least of which is the massive amount of data that is generated from acquiring time-lapse movies of embryonic development over long periods of time. Beyond the requirement of specialized tools merely to be able to visualize the data, producing quantitative results from these large and complex datasets is a challenge many biology labs struggle to overcome. Tracking cells in even a small mammalian embryo can produce millions of data points and require terabytes of storage and high-powered workstations or clusters to process. Custom software and algorithms laboriously generated for one model organism may not be applicable to another; cell size and shapes may be very different and time intervals and reporters vary, making the creation of one unified method extremely challenging. As a result, problems tend to be solved on an ‘as-needed’ basis, resulting in a patch-work of algorithms and methods that may work very well for the intended experiment, but are not broadly applicable. In addition to the computational expertise required to create these methods, there often needs be a certain degree of proficiency or familiarity to even use the method, provided it can be accessed and has been maintained to be compatible with current software environments, prohibiting its wide dissemination and use. This is not necessarily the fault of the creator, as making a method easy to use and pre-packaged or assembled in a friendly graphical user interface (GUI; see Glossary, Box 1) can often be as time consuming and require as much skill as developing the method itself. There is no guarantee that even when presented with big friendly buttons that the average user would find it compatible with or flexible to their needs.

In addition to needing the correct computational tools to analyse large datasets, those without access to these new, advanced light microscopes need access to the data itself. The hosting and dissemination of such large datasets remains a challenge, however. Even with the availability of cloud storage solutions, simply providing access to raw data requires continuous expense and expertise to set up and maintain. The menagerie of light microscopes available generate a wide variety of file-formats, metadata and annotations that can be difficult for the average user to parse, and ever-evolving software environments can lead to compatibility issues. Individual labs often do not have the resources to provide continuous access to their data, or the requisite software support. Community-wide initiatives have tackled such challenges in the past with databases and consortia for everything from genomes (NCBI Assembly, GenBank) to crystallography [Protein Data Bank (PDB)] to whole organisms [FlyBase, Zebrafish Information Network (ZFIN) and Mouse Genome Informatics (MGI)]. To ensure open access and the reproducibility of methods, similar initiatives are needed for the light microscopy field and the ever-increasing amount of imaging data. Fortunately, there are several attempts to do just this, such as the Image Data Resource (IDR; https://idr.openmicroscopy.org/) or the Euro Bioimaging consortium (https://www.eurobioimaging.eu/). There is also a large community-oriented effort to develop user-friendly tools to handle this new and burgeoning problem of big data, and with any luck new advances in machine learning (see Glossary, Box 1) will help these methods become more broadly applicable to everything from C. elegans to ascidians, Drosophila and mouse. We will discuss some of the various methods, applications and tools available to biologists to tackle their own tracking and lineage-tracing problems, however they get their data.

As we have discussed above, there is no one-size-fits-all cell analysis software solution available and choosing the right combination of software packages is an important step to analyse data efficiently. In Table 1, we provide an overview of some available cell-tracking packages that can be used for fate mapping and lineage tracing. Here, we show only software packages that model cell divisions and reconstruct a full cell-lineage tree. A complementary tracking software overview including non-dividing cell tracking and particle tracking can be found in a recent publication by Emami et al. (2020).

A key factor to consider when choosing a tracking software is the amount of data that needs to be analysed and the number of tracks necessary for the analysis. When the dataset or number of tracks is small (e.g. few/short movies with tens of cells), reconstructing the cell lineages manually is the most efficient and accurate analysis strategy. Manual labelling software is simple and quick to set up, as it only requires an image viewer and an annotation tool (e.g. the Fiji plugin MaMuT or TrackMate or CeLaVi) (Schindelin et al., 2012; Tinevez, 2017; Salvador-Martínez et al., 2021) and immediately yields highly accurate tracking results. MaMuT is especially designed for large 3D movies and can even accelerate the annotation by semi-automatically extending tracks (linking bright cells with similar radii) (Wolff et al., 2017 preprint). However, manual tracking is too time consuming to scale to datasets with hundreds or thousands of cells. In this case, automatic tracking software can be used to speed up the analysis. Every automatic tracking software has an internal model for lineage reconstruction that is used to detect and track cells. These models make implicit assumptions about the expected cell shapes and movement patterns. The key to selecting an appropriate software tool is to find the model for which assumptions best match the data at hand. In our overview, we highlight the three most important model aspects for lineage tracing: (1) cell detection in every video frame; (2) linking the detections between frames; and (3) detecting cell divisions.

The cell detection models of the discussed software packages can be broadly divided into point-detection and segmentation-based models. Point-detection models identify each cell by their centre and do not explicitly compute the cell outline or segmentation, e.g. MaMuT or its successor, Mastodon. Elephant and TGMM (Amat, 2014) additionally model cells as ellipsoids, but are still considered point-detection models. Alternatively, cells can be identified by segmentation, whereby the image is partitioned into multiple segments each containing one cell. The segmentation can be used to inform tracking and for downstream analyses that involve the whole cell area. Most recently, machine-learning algorithms are the state of the art in segmentation and ‘performed best in most segmentation scenarios’ and ‘exceptionally well’ on contrast enhancement microscopy images (Ulman, 2017). This shows the versatility of machine-learning approaches. Whereas rule-based (non-machine learning) algorithms are aimed at a particular image modality (e.g. fluorescence microscopy), machine-learning models learn from the data and thus the same model can be adjusted (trained) to fit different imaging conditions. However, this versatility comes at a cost and many challenges have to be overcome during the training process to obtain highly accurate models. Models, especially those with a large number of parameters, such as deep neuronal networks, require a large amount of training data (images paired with human annotations). Neural networks (see Glossary, Box 1) may start with random model parameters and are iteratively trained with examples from the training set. In each iteration, the prediction error of the model is calculated and the model parameters are adjusted to move the network output closer to the human annotation. In order to train neural networks that have many millions of parameters, large annotated training datasets are required. Once a training set is obtained, the training procedure itself needs to be carefully tuned (e.g. adjusting the learning rate) to obtain models that generalize well to new images. These non-trivial procedures make training neural networks in particular challenging for non-experts. For some types of data, training can be skipped, such as when pretrained models are available, eliminating the need to generate training data [e.g. btrack (https://bioimage.io); Ulicna et al. (2020 preprint) or for convex cells, see Schmidt et al. (2018 preprint)]. Even if one's data is not an exact match to a pretrained model, using one as a starting point for training can reduce the amount of one's own data required to get good performance. To make use of the growing number of available segmentation methods, ilastik (Berg et al., 2019), MorphoGraphX (Reuille et al., 2015), btrack (Ulicna et al., 2020 preprint) and Lineage Mapper (https://pages.nist.gov/Lineage-Mapper/) can ingest segmentations that were precomputed from these external models for lineage tracing.

The automatic trackers discussed here mostly fall under the category of tracking-by-assignment, whereby tracking is performed by linking detections between frames. For most ‘real world’ applications, considering all combinatorically possible links between detections is computationally infeasible. Trackers reduce the number of available links to a local neighbourhood and associate a ‘score/likelihood’ with each possible link, then select the links (and sometimes divisions) based on score. All trackers in Table 1 incorporate the spatial distance of the detections into the score. For videos with significant movement, Elephant incorporates deep learning (see Glossary, Box 1) of an optical flow to also contribute to the score. At high frame rates or for slow-moving cells, it can be beneficial to base the linking decision on the overlap of the segmentation (Lineage Tracker; Downey et al., 2011). For both slow and fast objects, cell features such as mean intensity, size or shape statistics are highly correlated between frames (Downey et al., 2011). This correlation informs the score function of Lineage Tracker and Lineage Mapper. Once each possible link has been scored, the tracking solution can be found as a set of high scoring links. Most of the trackers discussed here find the tracking solution sequentially, processing one frame at a time (e.g. by framewise Hungarian matching). Sequential trackers (Lineage Mapper, TGMM, MorphoGraphX, Lineage Tracker) often scale well to long videos as the computational costs increase linearly. Alternatively, global optimization that takes the full video into account can help to infer the cell positions from context even when single frames are uninformative. Such a global optimal linking model can be found in ilastik (Schiegg, 2013) and btrack (Ulicna et al., 2020 preprint).

Detecting cell division events is another crucial model aspect. In many cases, divisions can be identified with the help of the shape and position of the detected cells. Lineage Mapper, for example, uses a fixed formula based on roundness, cell size and daughter cell size to determine whether a division event has occurred. Beyond the correlation of cell shape and size (Downey et al., 2011), machine learning-based classifiers can be trained to identify dividing cells. TGMM uses a VGG classifier that is pretrained on cells in which a characteristic metaphase plate is visible during division (Amat, 2014). If the cell division characteristic does not match the training data, machine-learning models offer the ability to be retrained. Here, ilastik offers an interactive training interface to train a division classifier on object features (e.g. cell convexity, eccentricity). If pre-built classifiers or ‘off-the-shelf’ tools are insufficient for the type of data that needs to be analysed (too large, too variable or too complex), new models can be built and trained for specific purposes, such as finding cell division events in large mouse development time-lapses (McDole et al., 2018).

Obtaining a high-accuracy tracking result requires careful tuning and proofreading. All discussed tracking packages allow the user to adjust the behaviour of the internal detection/linking and division models. Some parameters, such as intensity thresholds, can be adjusted directly. However, finding the optimal values often requires prior expertise or running the model multiple times. An alternative is offered by the trackers with interactive learning models (Lineage Mapper, Elephant, ilastik). These offer a feedback loop, whereby the automated tracking solution can be corrected to update the tracking model parameters. This gives a more intuitive interface for adjusting the model parameters. However, even with interactive model training, it is nearly impossible to get perfect results. To achieve high tracking precision over long time spans, manual proofreading of results is necessary. This proofreading is directly supported by MorphographX and Elephant or by exporting the tracks into a manual tracking software (TrackMate; Tinevez, 2017). The lineage tree alone gives insight into a diverse set of collective cell behaviours (e.g. orientated cell divisions, cell migration) and can be analysed directly from the tracking solution. Other applications, such as analysing cell shape change in the multicellular environment, require additional cell characteristics to be measured. Although tracking tools measure some cell properties (e.g. shape, intensity distribution) automatically, measuring further properties requires an added layer of software. CellProfiler (Lamprecht et al., 2007) has a series of image-processing modules for measuring features that are commonly of interest (e.g. ‘Speckle Counting’ or colocalization). To measure more advanced (or less common) properties, a custom image analysis pipeline needs to be created. KNIME (Berthold, 2009) provides an easily accessible visual programming interface, in which a tracking and analysis pipeline can be constructed from preprogrammed building blocks (also known as nodes). These nodes also include ilastik and TrackMate, allowing functionality from those tools to be incorporated into a more advanced analysis pipeline.

Software packages are either sold as commercial software or are freely available as open-source software. Commercial solutions, such as Imaris or arivis Vision4D, provide support and are built to be easy to setup and easy to use. However, underlying tracking algorithms are confidential and therefore often ‘unsuitable for frontier research questions’ (Emami et al., 2020). Open-source software is transparent and free, but requires some expertise to set up and often needs significant effort to maintain compatibility with updates to the underlying framework. In Table 1, we present an overview of available lineage-tracking software solutions, their modelling choices, how they are distributed and for which platform they are available.

Once the realm of the classical embryologist, lineage tracing in modern developmental biology now requires the merger of advanced imaging methods, cutting-edge computer science and even the latest genomic technologies. With more data available than ever before, the question of how to manage and extract useful conclusions from the melee of results becomes even more important. Although some of this burden can be alleviated by carefully choosing labelling methods that can provide lineage information without complex computational requirements, the reality is that the vast majority of time-lapse image datasets over even short periods of development will require some heavy computational lifting, particularly if cellular behaviour is to be combined with object tracking/lineage tracing. Fortunately, even as light microscopes evolve to peer with even greater detail into the development of organisms and the lives of cells, so too do the computational tools needed to analyse them. Although generalizability and ease of access are as much of a problem for computational methods as they are for the microscopes and data themselves, the rush of new machine-learning frameworks to segment, track and quantify cell behaviours hopefully signal that the wait will not be long.

Among these new machine-learning frameworks, the advent of deep learning/neural networks has led to a revolutionary performance increase in a wide range of fields. Convolutional neural networks in particular yield exceptional (sometimes superhuman) accuracies, e.g. image recognition (Cireşan, 2011) and neuron segmentation (Lee, 2017), and have become a staple for object detection, segmentation and tracking. However, their performance crucially relies on the available training data. For supervised learning, these data need to be curated manually. For lineage tracing, this often cannot be out-/crowdsourced as it requires a high degree of familiarity with the data. Thus, human labour becomes the bottleneck. This problem can be somewhat alleviated by sharing training data and models (e.g. Model Zoo; https://modelzoo.co/). However, to truly solve this problem, new learning methods are needed that rely less on human annotations. Recent unsupervised learning techniques have shown to be highly data-efficient and even surpass fully supervised training in accuracy on natural images (Henaff, 2020). Combining these unsupervised learning techniques with tailor-made experiments could remove the need for human supervision all together, and make these deep-learning tools more easily accessible.

In this new era of developmental biology, data and results come thick and fast, making this an exhilarating time for the field. Visualizing, tracking and quantifying the movements and behaviours of every cell and lineage in a developing embryo is a key step towards the ultimate goal of understanding how an organism forms. The ability to compare development on a quantitative level, not only for a single animal but across multiple organisms and even species, will allow an evolutionary inspection into the myriad of ways nature derives complex form and function from simple starting materials.

We thank A. Schier for comments on the manuscript.