Tellu – an object-detector algorithm for automatic classification of intestinal organoids

The intestinal epithelium has been extensively studied in order to understand stem cell biology and several important diseases, including colorectal cancer. These studies have been accelerated by developing conditions that allow culturing of primary intestinal epithelium as 3D organoids ex vivo (Sato et al., 2009). The intestinal epithelial organoids are self-organising structures that, in many ways, recapitulate in vivo architecture of the intestinal epithelium, including crypt and villus units and growth factor requirements that resemble the in vivo niche-derived factors. Stem cells in these organoids recapitulate their in vivo behaviour, producing new stem cells as well as all differentiated cell types of the intestinal epithelium, thereby allowing for continuous passaging (Sato et al., 2011). During culture, organoids develop from cystic non-budding organoids enriched for stem and Paneth cells to early organoids with one to two crypt units and, further, to late organoids with three or more crypt units that surround the central part, representing the villus compartment that differentiates during culture. Under certain conditions, organoids form spheroids that represent a less differentiated state of the epithelium (Fordham et al., 2013).

Morphological classification and quantitative analysis of organoids have proved helpful in understanding stem cell biology and disease. For example, analysis of the organoid growth and differentiation stimulated by exogenous growth factors or other signalling molecules can be addressed by measuring the de novo crypt-budding reflecting stemness capacity (Almeqdadi et al., 2019). Analysis of spheroid formation has been useful in understanding differences between adult and foetal intestinal epithelium (Guiu et al., 2019), and in describing features of regenerating epithelium (Nusse et al., 2018). In addition, spheroid morphology is induced by WNT ligands (Farin et al., 2012), mutations in the tumour suppressor gene APC (Langlands et al., 2018) or co-culture with fibroblasts (see Fig. S4 in Farin et al., 2012), highlighting the impact of WNT signalling and stromal cells to epithelial differentiation. As the organoids grow in 3D, classification of their morphology requires time-consuming manual analysis, which is subjective, and prone to variations between experiments and bias. Artefacts absent in conventional 2D cell culture, such as collapsing, overlapping and out-of-focus organoids, complicate classic image processing and segmentation approaches, and make automated workflows difficult. More efficient quantification methods would expand the opportunities to use intestinal organoid classification in various research applications.

Although some tools and pipelines have been developed to quantitate organoids, they have certain limitations for classification. OrgaQuant, a deep convolutional neural network trained by Kassis et al. (2019), is limited to quantifying the size of intestinal spheroids and does not include morphological analyses or classification capabilities. Spiller and colleagues developed a machine learning-based tool to differentiate between live and dead organoids in order to facilitate drug screening in patient-derived organoids (Spiller et al., 2021). Although a potentially valuable tool for translational purposes, this method is limited to viability analyses and was not trained to classify organoids. In an interesting study on epigenetic control of intestinal epithelial differentiation, Ostrop and colleagues developed a pipeline to classify organoids (Ostrop et al., 2020). However, the requirement of a manual selection of features to train the classifier and the need to separately define classes for each experiment restricts the use of this pipeline as an easy-access method to enable automated classification of organoids.

In this work, we developed Tellu – an easy-to-use organoid classifier tool, i.e. a machine learning algorithm – and show that it enables fast, accurate and reproducible automated analysis of intestinal organoid morphology.

To develop a tool allowing automated classification of intestinal organoid morphology, we trained a real-time object detector algorithm. For this, we first created an annotated dataset of intestinal organoid images. Transmitted-light microscopy images of intestinal organoids derived from crypts isolated from the entire mouse small intestine at different stages of growth were manually divided into four classes of intestinal organoids based on their morphology. Non-budding, cystic organoids were labelled ‘organoid0’, early organoids with one or two crypt units ‘organoid1’, and late organoids with three or more buds ‘organoid3’. A separate class was created for ‘spheroids’ that lack crypt structures, and display a thin wall and larger diameter than cystic non-budding organoids (Fig. 1). The total dataset consisted of 840 images with a total of 23,066 annotations. Each image contained an average of 28.2 annotations (median 21; range 1-137). Since all organoids initially start from a cystic shape (organoid0), the four classes in the dataset were not equally represented: organoid0 represented 52% of the dataset with 11,922 instances, followed by organoid1 (24%; 5510 instances), organoid3 (14%; 3368 instances) and spheroid (10%; 2266 instances). We decided to split the annotated dataset into training (90%) versus model validation (10%) to maximize the size of the training dataset. Since this approach – instead of using the 80% versus 20% share that is frequently used – might render the validation set being too small to be representative, further validations were used to ensure that the model correctly generalises the organoid classes (see Figs 3–5).

To automate organoid classification, we chose the You Only Look Once (YOLO) object-detection algorithm that both predicts and classifies objects labelled as bounding boxes in an image (Redmon et al., 2016). We used a pre-trained YOLOv5 small model as the starting point and trained it by using a dataset of 756 images for 100 epochs (with one epoch going once through the whole dataset of images). Before reaching 100 epochs, the loss of function of the model for the training image set had converged and crossed the loss of function of the validation set, indicating that the learning rate slowed and that further iterations would likely lead the model to overfit, a process where the model keeps learning the training image set but fails to generalise the learning to data that has not seen before (Fig. 2A). To evaluate the model's performance, we assessed its precision (i.e. correctly detecting positive objects) and recall (i.e. not missing any objects) by using the mean average precision (mAP), a standard metric to evaluate overall model accuracy in the range of 0 to 1, with 1 representing an exact match to manually labelled detection. We used the mAP at a threshold of 50% overlap, i.e. at mAP=0.5, across the validation dataset, i.e. a set of 84 images, that was not used in training. The model achieved the best performance at epoch 60 (mAP:0.5=0.79) across classes (Fig. 2B); therefore, we chose it as the final learning point.

The classification performance of Tellu is shown as a confusion matrix displaying the proportion of the labelled dataset overlapping between ground truth – i.e. manual annotation by the trainer; x-axis – and the predicted classification – i.e. annotated by Tellu, y-axis (Fig. 2C). There, each column adds up to 1 and represents all annotations of that particular class. Performance in all classes was very good, with best performance for cysts (organoid0; 0.81) and late organoids (organoid3; 0.76). However, we noticed some differences in two situations: for early organoids, i.e. organoid1, Tellu labelled 0.15 and 0.11 of the annotations to organoid0 and organoid3, respectively. The difference reflects the non-binary continuum from cysts (organoid0) to late organoids (organoid3), an ambiguity that is also seen between manual counters. There was also some difference in the distinction between spheroids and early cysts (organoid0), where morphological changes between these two classes at very early stages of culture are minimal. Here, Tellu labelled 0.24 of spheroids to cysts (organoid0). Failures of Tellu due to overlap or out-of-focus organoids were modest: 0.05 for organoid3, 0.09 for organoid1, and 0.12 for organoid0 and spheroid (Fig. 2C).

Next, we assessed the intestinal organoid classification performance of Tellu by comparing its classification to that of two scientists experienced in manual intestinal organoid classification under variable conditions.

In the first experiment, classification performance was assessed at two time points (days 1 and 6) of normal intestinal organoid growth in the standard growth medium supplemented with EGF/Noggin/R-spondin1 (ENR) (Materials and Methods), with >200 organoids at either time point (Fig. 3). Remarkable alignment of the classifications for all classes was seen on day 6 [cyst (P1=0.28±0.02; P2=0.24±0.06; Tellu=0.21±0.04); early organoid (P1=0.24±0.04; P2=0.34±0.03; Tellu=0.33±0.06); late organoid (P1=0.47±0.05; P2=0.41±0.09; Tellu=0.43±0.06)]. Compared to both manual and automated analysis at day 1, reduction of cysts as well as induction of early and late organoids on day 6 demonstrates that Tellu faithfully captured the changes in organoid morphology. On day 1, when most organoids are at cyst stage (Fig. 2C, Cyst) and have a reduced ratio of spheroids (Fig. 2C, Spheroid), the Tellu classification was somewhat increased for cysts. However, this difference was only observed at very early time points. When we examined the detections made by Tellu, we noticed this disagreement was due to Tellu classifying single cells as cysts, a fact most probably derived from a harsh crypt dissociation.

In a second approach, we assessed the performance of Tellu under spheroid formation-inducing conditions. To this end, classification was done by using images of intestinal crypts grown for 24 h in primary intestinal fibroblast conditioned medium (FCM) or ENR medium as control. Again, we observed very good alignment of automatic and manual classification under spheroid-inducing conditions [cyst P1=0.38±0.04; P2=0.27±0.08; Tellu=0.23±0.06, spheroid P1=0.57±0.03; P2=0.68±0.07; Tellu=0.69±0.08] (Fig. 4, FCM). At this early time, abundance of early organoids (organoid1) was low and late organoids (organoid3) were not identified in manual or automated classification. Tellu also efficiently recognised pre-tumorigenic organoids isolated from the intestines of Apc^min mice or from chemically induced mouse colon tumours, which mainly grow as spheroids (Fig. S1). Thus, Tellu would also be useful when studying the growth of organoids that represent early intestinal tumorigenesis.

Since the detection process used by Tellu includes drawing a bounding box around each detected object in the image, we went on to assess whether the area of the bounding box can be used as a surrogate when manually calculating the size of organoids. To this end, quantification of organoid size by manual annotation of organoid boundaries with ImageJ was compared to the quantification based on the area of the bounding box around each organoid drawn by Tellu (Fig. 5A) and calculated as relative organoid growth between days 1 and 6 by using the images collected earlier (Fig. 3). Manual quantification (n=595) demonstrated a 7.7-fold increase in organoid size between days 1 and 6 (Fig. 5A). Quantification based on bounding boxes by Tellu showed a 6.7-fold increase (Fig. 5A), demonstrating the ability of Tellu to detect changes in organoid size. Whereas the resolution was slightly compromised due to the boxed versus accurate area approach, Tellu outperformed manual analysis with <5 min of hands-on time compared to a duration of >100 min for manual quantification.

In a second approach to evaluate the ability of Tellu to analyse organoid size in situations where differences are minor, we compared Tellu and manual quantification of organoid size in intestinal organoids that had been cultured for 5 days with epiregulin (EREG), with EGF or without any EGF-related factors (NR). This approach was based on identifying EREG as a stroma-derived growth factor potentially stimulating epithelial hyperproliferation (Ollila et al., 2018). In this experiment, alignment between automated Tellu and manual classification was robust. Both analyses showed enhanced growth in EGF-treated organoids, and further elevated growth in organoids that had been treated with EREG (Fig. 5B-D). Interestingly, combined treatment with EGF and EREG resulted in organoid size similar to that observed following treatment with EGF alone. This experiment confirms the ability of Tellu to quantitate organoid size and suggests a previously unidentified connection between signalling stimulated by different EGF receptor ligands.

Tellu represents a new open-access tool for the automated classification of intestinal organoids. It stimulates quantitative studies on intestinal organoids by providing a low-threshold tool to analyse large numbers of organoids currently requiring significant researcher hands-on time. Based on morphology Tellu uniquely enables the classification of intestinal organoids, including the stage of crypt budding, and also provides a proxy for organoid size.

Tellu is based on the YOLO object-detection algorithm. The YOLO algorithm was chosen owing to its speed and recently developed accuracy. It divides each image into a grid containing several cells, each represented as a vector that carries the cell coordinates relative to the whole image and a confidence value indicating the presence or absence of an object. Importantly, unlike other methods, such as Fast R-CNN (Ren et al., 2017), YOLO (Redmon et al., 2016) trains on whole images at once, including cells that do not contain any objects, making YOLO less likely to mistake background for objects. In addition, each cell predicts the class of the object, represented by another vector with a length corresponding to the number of classes. Although multiple bounding boxes can be predicted, the model gives only one class for each cell. Thus, YOLO first predicts the presence or absence of an object within a defined cell and provides a confidence value for the class of each predicted object.

As compared to previously published organoid analysis tools (Kassis et al., 2019; Ostrop et al., 2020; Spiller et al., 2021), Tellu provides significant added value. In addition to being the first easy-access tool to classify organoids, the YOLOv5 object-detection algorithm makes it compatible with video formats. It can be applied in real-time experimental setups. We trained Tellu and used it on brightfield still images. Still, real-time imaging with outputs in the format of videos is also used in the field (Yang et al., 2021), and Tellu is compatible with these (Movie 1). Quantifying organoid detections in videos by using Tellu may require object tracking algorithms to avoid counting the same organoid multiple times along the z-axis of a z-stack. This feature is planned to be implemented in a future release. Another benefit of the Yolov5 network is the easy implementation in a docker repository ensuring reproducibility and customizability.

During the revision process, another organoid classification tool, OrganoID, was released (Matthews et al., 2022). OrganoID is able to count and track single organoids, including small intestinal organoids, by using a neural network based on image segmentation, making it very useful for, e.g. time-lapse experiments, where the growth of individual organoids is monitored. However, OrganoID requires a basic level of python programming skills, whereas our workflow in Colaboratory makes sure programmable skills are not required. In addition, OrganoID is optimised for pancreatic organoids, whereas Tellu was trained exclusively for the classification of small intestinal organoids.

When considering the value of a tool for organoid classification, two parameters identified for other tools stand out – speed and accuracy – where speed should not compromise accurate detection. Notably, classification by Tellu was significantly superior regarding speed and not seriously compromised for accuracy when compared to manual quantification. With Tellu, analysis only takes a few seconds per image, with very little hands-on time. Output data contain the coordinates of the bounding box that can be used in addition to classification when deriving the relative area of each organoid. To some extent, all manual quantification is – especially in terms of classification – subjective and prone to bias as well as variation between experiments. Tellu performs consistently across large datasets without bias and, thus, is superior to manual analyses regarding reproducibility.

However, despite its numerous advantages, Tellu also has limitations. In dense organoid cultures where multiple organoids overlap, Tellu – as well as manual counters – struggles to distinguish organoids from each other. The organoid area calculation by Tellu is an approximation, as the bounding box does not account for the complex surface architecture of the organoids. However, these limitations are predictable and reproducible, and therefore do not compromise the value of Tellu in providing a quantitative, reproducible tool to classify intestinal organoids.

After removing the entire small intestine from the mouse, the mesentery was removed, and the intestine was flushed with cold PBS using a syringe and a 21-gauge needle. The intestine was cut open longitudinally and mucus removed by washing, i.e. immersing and shaking, the tissue three times in cold PBS. Thereafter, the tissue was minced into 3 mm fragments, placed horizontally in a 50 ml Falcon tube with 20-30 ml 10 mM EDTA in PBS, and left on ice for 1.45 h. During the first 45 min, the EDTA/PBS solution was changed every 15 min, after gently shaking the tube for 15 s and letting the tissue pieces sink before changing the solution. After incubation, crypts were detached by vigorous shaking for 20 s with two shakes per second. Crypts were isolated from villi and tissue pieces by filtering through a 70 µm strainer, pelleted by spinning the supernatant for 2 min at 200 rpm, washed with PBS, and collected as before. Crypts were resuspended in basal medium (DMEM:F12 medium (12634010, Gibco) supplemented with 1 M HEPES buffer pH 7,4 (H0887, Sigma), 1×Glutamax (35050061, Gibco), 1×Pen-strep (P4333, Sigma), 1×N-2 Supplement (17502001, Gibco) and 1× B-27 Supplement (17504044, Gibco) complemented with 50 ng ml⁻¹ of EGF (PMG8041, Gibco), 100 ng ml⁻¹ of Noggin (250-38, Peprotech), 250 ng ml⁻¹ of R-spondin 1 (3474-RS, R&D systems), 1 µM of N-acetylcysteine (A-5131, Sigma) and 10 µM of the ROCK1/2 inhibitor Y-27632 (Y0503, Sigma; used for first 2 days of culture), and mixed 1:3 ratio with Matrigel (356231, Corning) before plating 12 µl domes to 48-well plates. Matrigel domes were overlaid with ENR medium. Organoid medium was changed every two days. In experiments comparing the effects of the EGF receptor ligands EGF (R&D Systems 236-EG) and EREG (R&D Systems 968-EP), both ligands were used at 100 ng ml⁻¹.

Primary intestinal fibroblasts were isolated using an adapted version of a previously published protocol (Stzepourginski et al., 2015). Mouse small intestine was washed with PBS, ripped open and cut into 1 cm fragments. Tissue fragments were incubated in PBS containing EDTA (5 mM) and DTT (1 mM) for 20 min, and placed horizontally in a rotating (250 rpm) incubator at 37°C. EDTA solution was removed by washing tissue pieces three times with cold PBS by vigorously shaking for 15 s by hand, followed by foam removal and replacement of PBS between shakes. After the washes, the pieces were placed in PBS and minced thoroughly with a scalpel. Tissue pieces were then placed in a collagenase digestion solution [2 mg ml⁻¹ of collagenase type II and IV (Thermo Fisher Scientific) at a ratio of 1:1 dissolved in PBS containing 2% FBS] and horizontally rotated at 250 rpm for 20 min at 37°C. Cold fibroblast growth medium (DMEM+20% FBS+1% penicillin/streptomycin+1% L-glutamine) was added to stop collagenase activity. Single cells were released from the digestion suspension by trituration, and cells were pelleted by centrifugation (10 min, 1300 rpm) and resuspended in growth medium at 37°C. The suspension was plated on a Petri dish and cells were allowed to attach for 1 hour at 37°C. Thereafter, cells were washed three times with PBS to remove debris and unattached cells, and fresh medium was added. To collect fibroblast conditioned medium, 1×10⁵ intestinal fibroblasts were grown for 24 h in 6 cm plates in DMEM supplemented with 10% FBS. Medium was then changed to organoid basal medium for another 24 h, after which it was collected and centrifuged (5 min at 1100 rpm) to remove debris. Of this fibroblast conditioned medium, 350 µl were used per organoid-containing Matrigel dome. Organoids were imaged after 48 h.

To culture tumorigenic intestinal organoids, small intestinal crypts were collected from 15-17-week-old tumour-bearing Apc^min mice (JAX strain 002020), used to model the early stages of human colorectal cancer development (Su et al., 1992). The crypts were mixed with ENR medium and plated at 1:3 ratio with Matrigel as described before. After 2 days, the WNT agonist RSPO1 was removed from growth medium to select biallelic Apc mutant organoids that survive without exogenous WNT activation as described (Langlands et al., 2018). Apc mutant organoids were passaged and harvested using an ice-cold organoid harvesting solution (3700-100-01, R&D Systems) for at least 30 min. Matrigel domes were then scraped from the plate, pipetted 10× before collecting into a tube, and centrifuged at 300 g for 5 min. Then organoids were mechanically dissociated by pipetting up and down 40-50 times, and mixed with Matrigel as mentioned above.

By using two 7-day rounds of 2.5% DSS in drinking water, colonic tumours were induced following the azoxymethane–dextran sodium sulphate (AOM–DSS) protocol as described (Thaker et al., 2012). Used were male wild-type mice aged 10-12 weeks with a mixed genetic background. Animals were housed at the University of Helsinki Laboratory Animal Center following national and European legislation. Mice were euthanised 10 weeks after injection of AOM, the tumours of the distal colon cut into small pieces with a scalpel and placed in 5 ml of DMEM supplemented with 2.5 mg ml⁻¹ of collagenase 4 (17104019, Gibco) and 0.2 mg ml⁻¹ of DNase 1 (10104159001, Roche). Tumour pieces were shaken (130 rpm) at 37°C for 40 min and triturated 10× with a 5 ml pipet to dissociate the cells. Supernatants were collected and spun down at 1300 rpm for 5 min. Pellets were resuspended in ENR medium with 2.5 µM CHIR99021 (SML1046, Sigma) and mixed with Matrigel. After the first two passages, organoids were grown in EN medium, followed by culturing in Noggin only (N) medium after the second passage.

By using an EVOS FL microscope (Thermo Fisher) with a 4× objective, a total of 840 transmitted-light microscopy images were taken of intestinal organoids grown at different densities and obtained during different culture stages. Images were manually annotated using the python library labelImg and exported for the Yolo format as text files. The four categories of organoids in the text files were coded with integers, i.e. 0 for organoid0, 1 for organoid1, 2 for organoid3 and 3 for spheroid. (Fig. 1B). The dataset images were split into a training and a validation set consisting of 756 and 84 images, respectively, by using the python library module splitfolders. The annotated dataset has been registered at zenodo under doi:10.5281/zenodo.6768583.

The Yolov5 (DOI:10.5281/zenodo.3908559) state-of-the-art object detector was trained for 100 epochs with an image batch size (i.e. the number of images the algorithm works at the same time) of 28 and using the model yolov5 s.pt as pre-weights. The code was run in an Nvidia Tesla k8 graphics card from the Colaboratory research environment.

For automated quantification, images were analysed by Tellu using the code in the Jupyter notebook (https://colab.research.google.com/drive/1j-jutA52LlPuB4WGBDaoAhimqXC3xguX?usp=sharing) that is accessible through the GitHub repository (https://github.com/evdomene/tellervo). As an output, Tellu provides a text file for each analysed image, each row representing a detected object (organoid). The text file includes information on the organoid class, bounding box coordinates and confidence score. Files were tidied into a single file with all detections and metadata information, and further summarised into different condition groups by using R packages tidyverse and ggpubr for plotting.

Two independent researchers counted each organoid class in each image for manual quantification and summarised them by wells (four images per well). Further quantification and plots were done with R packages tidyverse and ggpubr (see above).

We thank Reeta Huhtala, Riikka Saikkonen and Myriam Sevigny for technical assistance, and the Laboratory Animal Center (LAC) of the University of Helsinki for their services.