Assessment & GenAI: from written assignment to on-site exam
Unsupervised writing assignments
The emergence of GenAI has made unsupervised written assignments more vulnerable. Based on the final product alone, as a lecturer, you cannot be certain whether the student has actually learned anything or that they have outsourced (parts of) the learning process to GenAI. Prohibiting the use of GenAI is impossible to enforce or monitor (detection programmes are unreliable and biased). Fortunately, there are several ways to adapt your written assignment to ensure that students go through the correct learning process. This requires a tailored approach.
Making written assignments safer
To make an unsupervised written assignment safer, you can monitor the writing process by setting interim deadlines and scheduling live meetings with the student. You can also add an oral defence/interview about the final product (for more information, see, for example, GenAI & assessment at course level – UvA Teaching and Learning Centres (TLC)). However, these types of measures are time-consuming and sometimes require additional training for assessors, which means that they are not feasible or desirable for every course.
Converting a written assignment into an on-site exam
For some courses, it is possible to convert a supervised written assignment into an on-site exam to make the assessment safer. This article offers guidance on determining whether your course is suitable for such an adjustment and what points are important to consider.
Is it possible to assess higher-order learning objectives in an exam setting?
Higher-order learning objectives that require students to analyse and evaluate in depth are not easy to assess in an on-site examination. If, for example, academic writing skills are explicitly included in the learning objectives in addition to analysis or evaluation, an on-site exam is not suitable. If writing is not the goal but a tool, then there are other options. The recommendations below may help:
- In some cases, the learning objectives can be covered by a number of open-ended or essay questions, for example with case studies, in which students have to analyse and evaluate within a specific context.
- It is sometimes even possible to assess higher-order learning objectives with closed questions. It should be noted, however, that this never requires the same level of thinking as an open-ended question, as the correct answer is already given. (For help with this, see, for example, Brame, C. (2022).)
- By allowing students to do preparatory work at home or during class (e.g. conducting literature research or developing a question) using GenAI, you can require more thinking in an on-site exam (although this is not the same as the thinking required for academic writing). For example, let students bring together the preparatory work in a different form during the exam (without the use of aids), such as policy advice, a newspaper article or a blog post for a popular science audience. For this type of supervised writing assignment, you need to give students more time, which means it does not combine well with other closed or open exam questions. Please note: shifting the thinking work to formative assignments requires serious student commitment and intensive supervision, which is not possible for every course.
Keep the basic principles of assessment design in mind
Don’t forget to review these points when converting a writing assignment into an on-site exam:
- If you request a digital exam hall (with computers and IT support) in good time via your OPC, students can type their answers without internet access and even use required readings. This will result in better thought-out (and legible) answers!
- Have you clearly indicated what kind of answer you expect? For example, in bullet points, in running sentences, a short paragraph, or in essay form? How long should the answer be?
- Keep an eye on the validity of the exam: ensure that all learning objectives and content topics are covered in the desired manner in terms of distribution and level, and avoid overlaps and gaps that may result from changing the exam format.
- Consider how many questions you can ask within the available time. Adding a few longer open questions should not compromise the validity of the rest of the exam, for example by removing too many knowledge or comprehension questions. Ensure that these learning objectives are also adequately covered in order to guarantee the validity of the assessment. At the bottom of the page on constructing assessments, you will find a useful overview of how long students need to answer each type of question.
- For closed questions, you need to ask enough questions to obtain a reliable result. If the exam consists entirely of closed questions, you need to come up with at least 40 four-choice questions. If the exam also contains open questions, 20-30 four-choice questions will suffice. Ensure a carefully considered distribution of question types across the material. Coming up with good questions (and especially good distractors, i.e. wrong answers) takes a lot of time: keep this in mind.
- Consider how you will grade the answers. For more essay-like questions where argumentation is more important than a right or wrong answer, it is often easier to work with a limited number of assessment criteria than to create an answer model with a scoring system. Align the assessment criteria with the learning objectives.
- Make sure you do not set overly high language proficiency standards for essay questions: in an exam setting with limited time, this is more difficult for students than in a take-home assignment.
- If you choose a different assessment format than the one specified in the course catalogue, you must obtain permission from the Exam Board for the change. If the assessment format remains the same, it is sufficient to inform students before the start of the course.
- Are students prepared for the change? Ensure that they have had sufficient opportunity to practice with new exam or question formats. Provide sample questions and discuss them. You can also organize an (online) question and answer session to alleviate stress.
- Keep the workload for assessors in mind as well. Can the grading still be completed within the hours available for the course?
Finally, remember that your course is part of a programme
If many written assignments in a programme are converted into on-site exams, the assessments across the programme become more monotonous. This can be detrimental for students who do not perform as well in an exam setting. Moreover, it can put pressure on the academic writing curriculum: without practice, a student cannot write a thesis. Coordinate with the rest of your team as much as possible to ensure that there is sufficient authenticity, variety and structure within the overall programme of assessment.
Help and advice
Would you like to discuss which measures are appropriate for your course? Make an appointment with the TLC Humanities assessment specialists.