In this paper, we study multi-label atomic activity recognition. Despite the notable progress in action recognition algorithms, they struggle to recognize atomic activities due to a deficiency in a holistic understanding of both multiple road users’ motions and their contextual information. We introduce Action-slot, a slot attention-based approach that learns visual action-centric representations, capturing both motion and contextual information. Our key idea is to design action slots that are capable of paying attention to regions where atomic activities occur, without the need for explicit perception guidance. To further enhance slot attention, we introduce a background slot that competes with action slots, aiding the training process in avoiding unnecessary focus on background regions devoid of activities. Yet, the imbalanced class distribution in the existing OATS dataset hampers the assessment of rare activities. To address the limitation, we collect a synthetic dataset called TACO, which is four times larger than OATS and features a balanced distribution of atomic activities. To validate the effectiveness of our method, we conduct comprehensive experiments and ablation studies against various action recognition baselines on OATS and TACO. We also show that the performance of multi-label atomic activity recognition on real-world datasets can be improved by pre-training representations on TACO. We will release our code and datasets upon acceptance.