In biological image recognition, different species might look similar resulting in a small margin, which causes errors in labeling images. Pollen grain image classification heavily suffers from both problems preventing from building well-calibrated recognition models. In this research, we aim to filter out aleatoric uncertainty caused by noisy labeling and similar shape of pollen species. To estimate aleatoric uncertainty, we propose a new Bayesian Focal Softmax loss function. It uses the softmax activation, which is more convenient for a single-label tasks compared to the original Focal loss based on the logistic function. The proposed loss function better estimates aleatoric uncertainty increasing the overall model performance. For evaluation, we used two datasets, POLLEN13L-det containing 13 classes of allergic pollen and POLLEN20L-det containing additional honey plant pollen species. We achieved the state-of-the-art results for both of them by applying the proposed loss function on RetinaNet. It improved the mAP and significantly reduced the variance compared to the regular Focal loss with softmax and provided much better aleatoric uncertainty estimate compared the Bayesian Focal loss with sigmoid activation