Employing a Hierarchical Rater Models for Automated Scoring: Scope Review on the Application in Educational Assessment

Abstract

This scope-review presents the milestones of how Hierarchical Rater Models (HRMs) become operable to used in automated essay scoring (AES) to improve instructional evaluation. Although essay evaluations—a useful instrument for evaluating higher-order cognitive abilities—have always depended on human raters, concerns regarding rater bias, inconsistency, and scalability have motivated the development of automated systems. By modeling rater behaviors, task complexity, and interaction effects, HRMs handle these issues and offer a strong base to minimize biases and increase dependability. Advances in machine learning and natural language processing (NLP) have helped HRMs to be included into AES systems. Leveraging HRMs to rectify rater biases and guarantee fairness, these systems include language analysis, semantic evaluation, and contextual understanding. This review also includes how Signal Detection Theory (SDT) is being included into HRMs to improve their capacity to assess rater sensitivity and bias and provide understandable results. By including HRMs into AES, feedback quality is improved as well as score accuracy, hence facilitating more exact formative evaluations become possible. Still present, though, are difficulties like computing complexity, dataset availability, and algorithmic bias. The paper emphasizes the possibilities of HRMs in creating fair, high-quality, scalable evaluation systems and supports ongoing research to improve these approaches for various educational environments.