A Model-Agnostic Framework for Transparent, Fair, and Reproducible Automated Essay Scoring

Abstract

The most accurate automated essay scoring systems function as opaque black boxes. This lack of transparency presents a significant obstacle to trust among the students, teachers, and policymakers who rely on these tools, as fairness and accountability are non-negotiable. The design and evaluation of a model-agnostic framework for explainable AI in essay scoring are detailed, with a foundational focus on reproducibility and algorithmic fairness. By addressing these longstanding challenges, the proposed framework delivers a tool that is understandable and trustworthy for students, educators, and policymakers. The contributions of this work are threefold. First, a prompt-aware binning strategy is introduced to normalize essay scores into three proficiency tiers, which enhances model stability and enables a more meaningful fairness analysis. Second, both classic and state-of-the-art transformer models are benchmarked within a reproducible pipeline that utilizes deterministic controls and a manifest system to track every data artifact, ensuring that all findings are auditable and verifiable. Third, rather than focusing on fairness or interpretability in isolation, this work integrates both dimensions, using SHAP and LIME to dissect model logic and to identify and measure disparities across demographic groups. On the ASAP2.0 dataset, a fine-tuned DeBERTa-v3-large model achieves a macro-F1 of 0.898. An XGBoost model trained on a fused set of linguistic and transformer features achieves a competitive 0.890. This framework aims to set a new standard for transparent and trustworthy AI in educational assessment and offers a foundation for future research in explainable educational technology.