TY - GEN
T1 - A method to predict interrater agreement in writing assessment based on raters' individual differences
AU - Jaunez, Roberto
AU - Villalon, Jorge
AU - Munoz, Gonzalo
AU - Baez, Gabriela
N1 - Publisher Copyright:
© 2019 IEEE.
PY - 2019/7
Y1 - 2019/7
N2 - Writing has been increasingly assessed within educational scenarios. However, assessing writing is a highly subjective task, making the grading of such tests a costly process that demands several graders for calculating interrater agreement to ensure reliability. A drawback of this approach is that interrater agreement does not explain the source of markers' differences; therefore, when agreement is low, the marking process must be repeated, and markers retrained. This article proposes a method to predict agreement, based on markers' individual differences using item response theory. Results showed that the method accurately predicts agreement even with partial evidence.
AB - Writing has been increasingly assessed within educational scenarios. However, assessing writing is a highly subjective task, making the grading of such tests a costly process that demands several graders for calculating interrater agreement to ensure reliability. A drawback of this approach is that interrater agreement does not explain the source of markers' differences; therefore, when agreement is low, the marking process must be repeated, and markers retrained. This article proposes a method to predict agreement, based on markers' individual differences using item response theory. Results showed that the method accurately predicts agreement even with partial evidence.
KW - Agreement prediction
KW - Interrater agreement
KW - Item response theory
KW - Writing assessment
UR - http://www.scopus.com/inward/record.url?scp=85072916551&partnerID=8YFLogxK
U2 - 10.1109/ICALT.2019.00083
DO - 10.1109/ICALT.2019.00083
M3 - Conference contribution
AN - SCOPUS:85072916551
T3 - Proceedings - IEEE 19th International Conference on Advanced Learning Technologies, ICALT 2019
SP - 255
EP - 257
BT - Proceedings - IEEE 19th International Conference on Advanced Learning Technologies, ICALT 2019
A2 - Chang, Maiga
A2 - Sampson, Demetrios G
A2 - Huang, Ronghuai
A2 - Gomes, Alex Sandro
A2 - Chen, Nian-Shing
A2 - Bittencourt, Ig Ibert
A2 - Kinshuk, Kinshuk
A2 - Dermeval, Diego
A2 - Bittencourt, Ibsen Mateus
PB - Institute of Electrical and Electronics Engineers Inc.
T2 - 19th IEEE International Conference on Advanced Learning Technologies, ICALT 2019
Y2 - 15 July 2019 through 18 July 2019
ER -