Rubrics: (when) are they useful as an assessment tool?

Opportunity for teachers to do research

“I was working as a lecturer in the Psychobiology program. I really liked teaching at the UvA, but somewhere I had always missed doing research a little bit. When the Teaching & Learning Center gave lecturers the chance to do research on an educational topic, I immediately seized the opportunity.

I decided to focus on rubrics. In Psychobiology we work a lot with rubrics and in addition, I see a lot of potential in rubrics to achieve a good connection between assessment and learning objectives within a subject. However, I noticed that not much was known about the reliability and validity of rubrics. In addition, it appears that evaluators who work with rubrics often have varying experiences in practice. There are evaluators who find these very helpful tools. At the same time, there are also evaluators who would rather see these rubrics disappear and thus do not find the rubrics helpful. These two issues, namely the psychometric properties of rubrics and assessor satisfaction formed the core of my project.”

Assessor satisfaction with rubrics

“Rubrics can increase objectivity in reviewing. However, assessors often report that rubric descriptions deviate from their own judgments of student reports. Such deviations can reduce rubric quality and the satisfaction of teachers using rubrics. This study examined the quality of rubrics among evaluators with extensive experience in using rubrics. In addition, we examined which assessor characteristics are predictive of assessor satisfaction with rubrics.

Evaluator satisfaction with rubrics and evaluator characteristics such as experience and involvement in rubric development were determined with a questionnaire among evaluators of thesis reports of Psychobiology students. A prediction model was then developed to identify the reviewer characteristics that best predict assessor satisfaction.”

Development of a dashboard

“To assess the quality of rubrics, a computer program – a dashboard – was developed. With this program, the reliability, internal consistency, and constructs measured by the rubric can be examined. The analysis was conducted on the rubrics of all writing assignments within the bachelor of psychobiology in the 2020-2021 academic year but can easily be applied to other rubrics as well.”

Structure and level of flexibility play a major role in satisfaction with rubrics

“Meanwhile, the research fellowship has been completed and the project has produced two clear outcomes. First, I developed a dashboard that allows assessors to analyze psychometric properties, such as validity and reliability, of their rubrics. Second, the research on assessor satisfaction has been completed, resulting in a poster.

The main finding is that assessors with more experience are less satisfied with rubrics than assessors with less experience. Here, the level of flexibility offered by the rubric seems to play a major role. Beginning evaluators appreciate the clear structure that rubrics typically provide. Experienced graders seem to experience this structure as a hindrance and value more the freedom to make their own interpretation of the grading process. Incidentally, the questionnaire revealed that graders often received no introduction or training to the rubric. The literature indicates that such training is essential to the successful implementation of rubrics. Possibly introducing training in the use of rubrics could increase assessor satisfaction, especially among more experienced assessors.”

Two-year research project for teachers

“My project started in 2021. I was appointed as a research fellow at the Teaching & Learning Center for one day a week in the summer of that year. I did the implementation of the project within the Faculty of Science (FNWI). Currently, I am in a transition phase of this project. I transferred to the Faculty of Social and Behavioural Sciences (FMG), Department of Psychological Methodology, as of September 1. Here, I will also contribute to higher education research. More will become clear about the exact details of my research at FMG in the near future.”

Guidance from TLC

“Guidance on the research process is provided by TLC. In the beginning, for example, much attention is paid to a clear formulation of the research question(s). Also, the importance of a complete and concrete research proposal is always emphasized. I wholeheartedly support this approach. In particular, by writing a research proposal you set frameworks for your research and make clear what the desired outcomes of the research are. This ultimately increases efficiency in the execution of the project. So I can wholeheartedly recommend future fellow researchers to follow this working method as well!

The contact with the project leaders was perfect. There was plenty of room to meet and spar individually. It ensured that I could actually constantly maintain an overview without getting stuck.”

Professional development and implementation of research in teaching practice

“The added value of this track for me is to promote the professional development of teachers and teacher-researchers and to stimulate a research-based approach to educational innovations at the UvA. In this regard, I see the fellowship program primarily as a teaching innovation program. After all, concrete goals are formulated with all research fellows on which work is being done. What these goals are varies considerably between research fellows, but the condition is that it should concretely improve education at the UvA.

I also saw with my own eyes at the concluding symposium in July 2023 that this goal was achieved in all cases. With that, as far as I am concerned, this program is already a success. It is unique to work on the research project simultaneously with lecturers from very different programs within the UvA. I really appreciated the discussions and the substantive exchange of knowledge between the research fellows.

In addition to innovation for higher education, this research fellowship also provides professional development for the research fellows. As a research fellow specifically, the project ensures that you gain and add knowledge to the research topic. Also, the program ensures that you can make a profile on the topic. This then provides opportunities for involvement in the implementation of the research in educational practice. We are now looking with a group of teachers how to  start using the dashboard in practice.”

More about the Education Research Fellows Programme