Representation learning of whole-slide images (WSIs) has been mainly based on weak supervision with Multiple Instance Learning (MIL). However, the resulting slide embedding is highly tailored to the clinical task at hand, which limits model expressivity and generalization, especially in low-data regimes. Instead, we hypothesize that morphological redundancy can be leveraged to build a task-agnostic slide representation in an unsupervised fashion. To this end, we introduce PANTHER, a novel method based on the Gaussian mixture model that summarizes the set of WSI patches with morphological prototypes. Specifically, each patch is assumed to have been generated from a mixture distribution, where each mixture component represents a morphological exemplar. We then construct a compact slide representation from the estimated mixture parameters that can be readily used for a range of downstream tasks.By performing an extensive evaluation of PANTHER on survival and subtyping tasks using 13 datasets, we show that 1) PANTHER outperforms or is on par with supervised MIL and 2) the analysis of morphological prototypes brings new qualitative and quantitative insights into model interpretability. The code will be released upon acceptance.