In this talk, we explore a holistic approach for building a natural language classification system tailored for content moderation in real-world scenarios. We discuss the importance of crafting well-defined content taxonomies and labeling guidelines to ensure data quality, and detail the active learning pipeline developed to handle rare events effectively. We also examine various techniques used to enhance the model's robustness and prevent overfitting. This approach generalizes to diverse content taxonomies and how the resulting classifiers can outperform standard off-the-shelf models in the context of content moderation.