During mammalian pre-implantation development, embryonic cells undergo two successive cell fate decisions. First, outer cells polarise to become the trophectoderm, which forms the placenta, or the inner cell mass (ICM). Second, ICM cells differentiate into the primitive endoderm (PrE), which forms the yolk sac, or epiblast, which gives rise to the foetus and is the source of embryonic stem cells (ESCs). Fibroblast growth factor 4 (FGF4) signalling promotes PrE specification in mice, playing a key role in determining the proportions of primitive endoderm and epiblast cells. However, the precise mechanisms governing lineage specification during mouse development remain unclear. Here, Strawbridge and colleagues Strawbridge and colleagues generate chimeras, in which ESCs are injected into mouse embryos. Intriguingly, they find that donor ESCs, expressing or lacking FGF4, displace host cells from the ICM into the trophectoderm during the first cell fate decision. The remaining unspecified host ICM cells can differentiate into PrE or epiblast in chimeras lacking FGF4, whereas in embryos injected with FGF4-expressing donor cells, host ICM cells are biased towards PrE. Using mathematical modelling, the authors further show that donor cells sequentially displace host cells into extra-embryonic lineages through physical crowding followed by FGF4 signalling. Overall, this work reveals how ESCs influence cell fate decisions in the early mammalian embryo.