Reaction-diffusion networks underlie pattern formation in a range of biological contexts, from morphogenesis of organisms to the polarisation of individual cells. One requirement for such molecular networks is that output patterns be scaled to system size. At the same time, kinetic properties of constituent molecules constrain the ability of networks to adapt to size changes. Here we explore these constraints and the consequences thereof within the conserved PAR cell polarity network. Using the stem cell-like germ lineage of the embryo as a model, we find that the behaviour of PAR proteins fails to scale with cell size. Theoretical analysis demonstrates that this lack of scaling results in a size threshold below which polarity is destabilized, yielding an unpolarized system. In empirically-constrained models, this threshold occurs near the size at which germ lineage cells normally switch between asymmetric and symmetric modes of division. Consistent with cell size limiting polarity and division asymmetry, genetic or physical reduction in germ lineage cell size is sufficient to trigger loss of polarity in normally polarizing cells at predicted size thresholds. Physical limits of polarity networks may be one mechanism by which cells read out geometrical features to inform cell fate decisions.