Image-based crowd counting widely employs density map regression, which often suffers from severe performance degradation when tested on data from unseen scenarios. To address this so-called "domain shift" problem, we study single domain generalization (SDG) for crowd counting. Though SDG has been extensively explored, the existing approaches are mainly for classification and segmentation. They can hardly be extended to crowd counting due to its nature of density regression and label ambiguity (i.e., ambiguous pixel-level ground truths). We propose MPCount, a novel SDG approach effective even for narrow source distribution. Reconstructing diverse features for density map regression with a single memory bank, MPCount retains only domain-invariant representations using a content error mask and attention consistency loss. It further introduces the patch-wise classification as an auxiliary task to boost the robustness of density prediction with relatively accurate labels. Through extensive experiments on different datasets, MPCount is shown to significantly improve counting accuracy compared to the state-of-the-art approaches under diverse scenarios unobserved in the training data and narrow source distribution.