A method to predict interrater agreement in writing assessment based on raters' individual differences

Roberto Jaunez; Jorge Villalon; Gonzalo Munoz; Gabriela Baez

doi:10.1109/ICALT.2019.00083

A method to predict interrater agreement in writing assessment based on raters' individual differences

Roberto Jaunez, Jorge Villalon, Gonzalo Munoz, Gabriela Baez

School of Psychology

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

Abstract

Writing has been increasingly assessed within educational scenarios. However, assessing writing is a highly subjective task, making the grading of such tests a costly process that demands several graders for calculating interrater agreement to ensure reliability. A drawback of this approach is that interrater agreement does not explain the source of markers' differences; therefore, when agreement is low, the marking process must be repeated, and markers retrained. This article proposes a method to predict agreement, based on markers' individual differences using item response theory. Results showed that the method accurately predicts agreement even with partial evidence.

Original language	English
Title of host publication	Proceedings - IEEE 19th International Conference on Advanced Learning Technologies, ICALT 2019
Editors	Maiga Chang, Demetrios G Sampson, Ronghuai Huang, Alex Sandro Gomes, Nian-Shing Chen, Ig Ibert Bittencourt, Kinshuk Kinshuk, Diego Dermeval, Ibsen Mateus Bittencourt
Publisher	Institute of Electrical and Electronics Engineers Inc.
Pages	255-257
Number of pages	3
ISBN (Electronic)	9781728134857
DOIs	https://doi.org/10.1109/ICALT.2019.00083
State	Published - Jul 2019
Event	19th IEEE International Conference on Advanced Learning Technologies, ICALT 2019 - Maceio, Brazil Duration: 15 Jul 2019 → 18 Jul 2019

Publication series

Name	Proceedings - IEEE 19th International Conference on Advanced Learning Technologies, ICALT 2019

Conference

Conference	19th IEEE International Conference on Advanced Learning Technologies, ICALT 2019
Country/Territory	Brazil
City	Maceio
Period	15/07/19 → 18/07/19

Keywords

Agreement prediction
Interrater agreement
Item response theory
Writing assessment

Access to Document

10.1109/ICALT.2019.00083

Cite this

Jaunez, R., Villalon, J., Munoz, G., & Baez, G. (2019). A method to predict interrater agreement in writing assessment based on raters' individual differences. In M. Chang, D. G. Sampson, R. Huang, A. S. Gomes, N.-S. Chen, I. I. Bittencourt, K. Kinshuk, D. Dermeval, & I. M. Bittencourt (Eds.), Proceedings - IEEE 19th International Conference on Advanced Learning Technologies, ICALT 2019 (pp. 255-257). Article 8820915 (Proceedings - IEEE 19th International Conference on Advanced Learning Technologies, ICALT 2019). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/ICALT.2019.00083

Jaunez, Roberto ; Villalon, Jorge ; Munoz, Gonzalo et al. / A method to predict interrater agreement in writing assessment based on raters' individual differences. Proceedings - IEEE 19th International Conference on Advanced Learning Technologies, ICALT 2019. editor / Maiga Chang ; Demetrios G Sampson ; Ronghuai Huang ; Alex Sandro Gomes ; Nian-Shing Chen ; Ig Ibert Bittencourt ; Kinshuk Kinshuk ; Diego Dermeval ; Ibsen Mateus Bittencourt. Institute of Electrical and Electronics Engineers Inc., 2019. pp. 255-257 (Proceedings - IEEE 19th International Conference on Advanced Learning Technologies, ICALT 2019).

@inproceedings{562c9d2652c44527b594a8f890a054d7,

title = "A method to predict interrater agreement in writing assessment based on raters' individual differences",

abstract = "Writing has been increasingly assessed within educational scenarios. However, assessing writing is a highly subjective task, making the grading of such tests a costly process that demands several graders for calculating interrater agreement to ensure reliability. A drawback of this approach is that interrater agreement does not explain the source of markers' differences; therefore, when agreement is low, the marking process must be repeated, and markers retrained. This article proposes a method to predict agreement, based on markers' individual differences using item response theory. Results showed that the method accurately predicts agreement even with partial evidence.",

keywords = "Agreement prediction, Interrater agreement, Item response theory, Writing assessment",

author = "Roberto Jaunez and Jorge Villalon and Gonzalo Munoz and Gabriela Baez",

note = "Publisher Copyright: {\textcopyright} 2019 IEEE.; 19th IEEE International Conference on Advanced Learning Technologies, ICALT 2019 ; Conference date: 15-07-2019 Through 18-07-2019",

year = "2019",

month = jul,

doi = "10.1109/ICALT.2019.00083",

language = "English",

series = "Proceedings - IEEE 19th International Conference on Advanced Learning Technologies, ICALT 2019",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

pages = "255--257",

editor = "Maiga Chang and Sampson, {Demetrios G} and Ronghuai Huang and Gomes, {Alex Sandro} and Nian-Shing Chen and Bittencourt, {Ig Ibert} and Kinshuk Kinshuk and Diego Dermeval and Bittencourt, {Ibsen Mateus}",

booktitle = "Proceedings - IEEE 19th International Conference on Advanced Learning Technologies, ICALT 2019",

}

Jaunez, R, Villalon, J, Munoz, G & Baez, G 2019, A method to predict interrater agreement in writing assessment based on raters' individual differences. in M Chang, DG Sampson, R Huang, AS Gomes, N-S Chen, II Bittencourt, K Kinshuk, D Dermeval & IM Bittencourt (eds), Proceedings - IEEE 19th International Conference on Advanced Learning Technologies, ICALT 2019., 8820915, Proceedings - IEEE 19th International Conference on Advanced Learning Technologies, ICALT 2019, Institute of Electrical and Electronics Engineers Inc., pp. 255-257, 19th IEEE International Conference on Advanced Learning Technologies, ICALT 2019, Maceio, Brazil, 15/07/19. https://doi.org/10.1109/ICALT.2019.00083

A method to predict interrater agreement in writing assessment based on raters' individual differences. / Jaunez, Roberto; Villalon, Jorge; Munoz, Gonzalo et al.
Proceedings - IEEE 19th International Conference on Advanced Learning Technologies, ICALT 2019. ed. / Maiga Chang; Demetrios G Sampson; Ronghuai Huang; Alex Sandro Gomes; Nian-Shing Chen; Ig Ibert Bittencourt; Kinshuk Kinshuk; Diego Dermeval; Ibsen Mateus Bittencourt. Institute of Electrical and Electronics Engineers Inc., 2019. p. 255-257 8820915 (Proceedings - IEEE 19th International Conference on Advanced Learning Technologies, ICALT 2019).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - A method to predict interrater agreement in writing assessment based on raters' individual differences

AU - Jaunez, Roberto

AU - Villalon, Jorge

AU - Munoz, Gonzalo

AU - Baez, Gabriela

PY - 2019/7

Y1 - 2019/7

N2 - Writing has been increasingly assessed within educational scenarios. However, assessing writing is a highly subjective task, making the grading of such tests a costly process that demands several graders for calculating interrater agreement to ensure reliability. A drawback of this approach is that interrater agreement does not explain the source of markers' differences; therefore, when agreement is low, the marking process must be repeated, and markers retrained. This article proposes a method to predict agreement, based on markers' individual differences using item response theory. Results showed that the method accurately predicts agreement even with partial evidence.

AB - Writing has been increasingly assessed within educational scenarios. However, assessing writing is a highly subjective task, making the grading of such tests a costly process that demands several graders for calculating interrater agreement to ensure reliability. A drawback of this approach is that interrater agreement does not explain the source of markers' differences; therefore, when agreement is low, the marking process must be repeated, and markers retrained. This article proposes a method to predict agreement, based on markers' individual differences using item response theory. Results showed that the method accurately predicts agreement even with partial evidence.

KW - Agreement prediction

KW - Interrater agreement

KW - Item response theory

KW - Writing assessment

UR - http://www.scopus.com/inward/record.url?scp=85072916551&partnerID=8YFLogxK

U2 - 10.1109/ICALT.2019.00083

DO - 10.1109/ICALT.2019.00083

M3 - Conference contribution

AN - SCOPUS:85072916551

T3 - Proceedings - IEEE 19th International Conference on Advanced Learning Technologies, ICALT 2019

SP - 255

EP - 257

BT - Proceedings - IEEE 19th International Conference on Advanced Learning Technologies, ICALT 2019

A2 - Chang, Maiga

A2 - Sampson, Demetrios G

A2 - Huang, Ronghuai

A2 - Gomes, Alex Sandro

A2 - Chen, Nian-Shing

A2 - Bittencourt, Ig Ibert

A2 - Kinshuk, Kinshuk

A2 - Dermeval, Diego

A2 - Bittencourt, Ibsen Mateus

PB - Institute of Electrical and Electronics Engineers Inc.

T2 - 19th IEEE International Conference on Advanced Learning Technologies, ICALT 2019

Y2 - 15 July 2019 through 18 July 2019

ER -

Jaunez R, Villalon J, Munoz G, Baez G. A method to predict interrater agreement in writing assessment based on raters' individual differences. In Chang M, Sampson DG, Huang R, Gomes AS, Chen NS, Bittencourt II, Kinshuk K, Dermeval D, Bittencourt IM, editors, Proceedings - IEEE 19th International Conference on Advanced Learning Technologies, ICALT 2019. Institute of Electrical and Electronics Engineers Inc. 2019. p. 255-257. 8820915. (Proceedings - IEEE 19th International Conference on Advanced Learning Technologies, ICALT 2019). doi: 10.1109/ICALT.2019.00083

A method to predict interrater agreement in writing assessment based on raters' individual differences

Abstract

Publication series

Conference

Keywords

Access to Document

Fingerprint

Cite this