Beyond Accuracy: Generalisation, Bias, and Robustness in Deep Learning for Facial Emotion Recognition

Abstract

Facial emotion recognition has advanced rapidly through convolutional neural networks (CNNs), transfer learning, hybrid models, attention mechanisms, transformers, and multimodal systems. Yet much of the literature still evaluates models chiefly by their accuracy on a single dataset. This practice conceals failures of generalization, demographic bias, label noise, sensitivity to perturbations, and ethical constraints on deployment. This article presents a systematic review, structured using PICOC and an adapted PRISMA protocol, of peer-reviewed studies, reviews, public datasets, and deep learning models published between 2021 and 2026. It asks how recent machine learning (ML) and deep learning approaches to FER have navigated the trade-offs among benchmark accuracy, cross-dataset generalization, demographic bias, and robustness in real-world conditions. Rather than reproducing an accuracy-first ranking, the review advances a trustworthy FER framework that considers performance alongside domain transfer, fairness, explainability, robustness, and computational feasibility. The 38-source evidence matrix is coded across six reporting domains and summarized by a Trustworthiness Evidence Coverage (TEC) index, ranging from 0 to 100%, which measures the completeness of the evidence supporting a trustworthiness claim rather than the intrinsic trustworthiness of a model. The evidence indicates that CNNs and transfer learning remain strong baselines; CNN-ML hybrids may offer partial interpretability; attention and transformer models improve the selection of salient regions; and multimodality can increase robustness, albeit at the cost of greater complexity and privacy risk. The central conclusion is that aggregated accuracy is necessary but insufficient: FER studies should also report macro-F1, cross-dataset validation, group-disaggregated metrics, perturbation tests, explanations, and deployment conditions.