Deep Learning based segmentation models for medical imaging often fail under subtle distribution shifts calling into question the robustness of these models. Medical images however have the unique feature that there is limited structural variability between patients. We propose to exploit this notion and improve the robustness of deep learning based segmentation models by constraining the latent space to a learnt dictionary of base components. We incorporate a topological prior using persistent homology in the sampling of our dictionary to ensure topological accuracy after composition of the components. We further improve robustness by deep topological supervision applied in an hierarchical manner. We demonstrate the effectiveness of our method under various perturbations and in two single domain generalisation tasks.