Document Type

Article

Publication Date

2023

Publication Title

Research & Practice in Assessment

Volume

18

Issue

1

Pages

19-32

Abstract

Integrative learning is an important outcome for graduates of higher education. Therefore, it should be well-defined and assessed reliably. The American Association of Colleges & Universities has developed a rubric to define and assess integrative learning, but it has low reliability. This pilot study examines whether this rubric's reliability can be improved by training users on how to use the rubric in a group setting rather than individually. Twelve faculty were trained to score undergraduate ePortfolios using the Integrative and Applied Learning VALUE Rubric. Half of the faculty were trained in an individual setting and half in a group setting using a popular norming protocol. Results indicate that group training does not improve interrater reliability, though it does improve rater confidence in their rubric scores. Implications include the need for more research comparing individual and group training as well as investigating the efficacy of current training protocols.

Rights

© 2023 Research & Practice Assessment, All Rights Reserved.

Included with the kind written permission of the copyright holder.

Original Publication Citation

Stafford, L., Cousins, E., Bol, L., & Mize, M. (2023). Improving reliability in assessing integrative learning using rubrics: Does group norming help? Research & Practice in Assessment, 18(1), 19-32. https://www.rpajournal.com/improving-reliability-in-assessing-integrative-learning-using-rubrics-does-group-norming-help/

ORCID

0009-0004-4105-5400 (Stafford)

Download

Share

COinS