Recap: Impact of GenAI on assessment

On 3 July 2024, the TLC Science organised a 2-hour hands-on workshop for Science lecturers in collaboration with TLC Central, TLC FGw and TLC FdR. The workshop focused on the impact of generative AI on assessment. How can you make sure your assessment is more AI-resilient? The participants listened to several student and lecturer examples, received didactical tips and tricks and, above all, got to work on their own assessment.

Primary concerns of the participants

The primary concerns of the participants regarding GenAI and assessment were that the learning objectives are no longer being assessed, that students’ critical thinking is not being assessed, that students don’t learn to write, that students no longer learn anything, and that students don’t use formative assessment effectively.

Students’ GenAI strategy

With unsupervised written assignments, students can easily misuse GenAI. For example by following these steps:

  1. Upload the context (assignment, syllabus, literature, rubric, etc.) and tell GenAI to do the assignment.
  2. Go step-by-step through the generated output, copy parts of the assignment that are not yet included in the previous response and tell GenAI to include it.
  3. Copy the course literature list and tell GenAI to add any references that are relevant.
  4. Copy the rubric and tell GenAI to adapt the output to score high on this rubric.

If students do this, it is possible to get a high grade for the assignment in 30 minutes without learning much about the topic. Especially the most competent students figure out how to work well with GenAI and engage less and less with the actual content of the assignments.

How to adapt your assignments?

Adapting your assignment to GenAI will require an investment in time and effort. Generally, no single solution will completely safeguard your assessment. It will always involve a mixed bag of solutions that depend on your specific situation.

Are you not sure whether your course assessment is vulnerable to unauthorised use of GenAI? Then you can first follow the steps from this checklist: Checklist for GenAI vulnerabilities in assessment at course level.

Broadly speaking, you have two options to make your assessment more AI-resilient:

  1. Adjust your current written assignment.
  2. Design an alternative assessment format.

See below for more information.

1. Adjust your current written assignment

Focus on student motivation

Students are generally not that interested in cheating, but in learning and preparing for their future. However, situations arise where students will look for shortcuts if they are not motivated to engage with an assessment, or they have other time pressures where they feel they have to prioritise one assessment over another.

Students are more inclined to do effortful work themselves if they are motivated and at ease. Below are some general tips to help you decide if any elements of your assessments should be adjusted to improve students motivation:

  • Engage personal interests or link them to students’ backgrounds.
  • Discuss with students the learning objectives and activities.
  • Shed light on the learning process and the importance of obtaining skills.
  • Include a Statement of Originality (for the thesis).
  • Consider how Al will affect professions, and include appropriate training, information or tasks for this.
Adapt the content of the assignment

We need to reconsider whether unsupervised written assignments still fit the learning outcomes at a time when many students are using GenAI to help them.

In some faculties and programmes, the learning outcomes assessed in written assignments are primarily content-based, with less focus on the actual academic writing skills. In such cases, it might not matter if students have used GenAI to help them correct structural or language errors, as long as the ideas, results and analysis are their own (although this is by no means a certainty in times of GenAI, even if you monitor the research and writing process). In fact, some argue that GenAI can even help make writing assignments more equitable for students, by helping to level the playing field for students who are not writing in their first language, for instance.

However, this must be reflected in how you grade the assignment.

Tips & tricks:

  • Consider making aspects like structure and language pass/fail.
  • Require students to deeply engage with discipline specific literature, have them reference specific arguments in papers and include page numbers.
  • Have assignments reflect high discipline-specific standards in reasoning and focus on the higher levels of Bloom’s Taxonomy, because GenAI responses often lack specific depth when higher order responses are required. For example: Let students reflect on argumentation in literature, compare sources or evaluate argumentation.
  • Require students to work with hypothetical, obscure or unique cases.
  • Refer to course-specific knowledge, e.g. “Use the concepts discussed in week 3 to …. ” or “Explain how materials of week 5 changed how you think about …”. However, a smart student is likely to upload all information you provide (including your lecture slides), so pair it with a validation of some kind.

However, be aware that future GenAI models are likely to also be able to do this. AI can understand context better and better, so don’t lean entirely on that.

Adjust the rubric

A short term change you can look at is to adjust the weighting of different aspects in the rubric to emphasize those skills that GenAI can’t do well. For example, to put more emphasis on the higher-order cognitive skills such as analysis, argumentation, and use of sources, and to put less emphasis on the lower-order cognitive skills such as language, style, and lay-out. So the actual assessment criteria don’t change, but the emphasis and weight put on them does.

If you do this, always go back to your learning objectives. What are you assessing? If it is writing skills, it’s unlikely that an unsupervised assessment is the right way to assess this, due to how proficient GenAI is in this area. A solution would be to put more emphasis on the whole writing process, rather than just the end product.

Focus on the process

Focusing on the process a student goes through for an assessment is a great way to safeguard your assessment. Here are some ways you can start to think about this:

  • Map out the process you think your students should go through as they work on their assignments, and create a process-path to visualise the milestones in their learning journey. These check-in points can be used in the evaluation of students’ work during the assessment.
  • Separate the learning process from the assessed ‘product’ by designing assessments that are continuous during the term or integrateĀ tasks into the assessment that require the documentation of the development or thinking process. For example, let students submit versions of the assignment at various stages in the process (e.g. a logbook, by turning on their track changes, or conducting a peer review).
Expand the process

At each step of the learning process, you can do one of two things:

  1. Keep the assignment unsupervised but validate it, so pair it with another assignment that cannot be done with GenAI. For example, adding a defense or oral exam or presenting an argument map before writing a paper.
  2. Move the assignment to a controlled environment. For example, showing an application of theory or use of models with a few broader essay-questions in an exam setting.

Some additional ideas for how you can separate the learning process from the end product and incorporate tasks along the process path:

  • Idea generation stage:
    • Students present an argument mindmap in class (show-and-tell).
    • Students are required to develop an idea based on a unique or personal case.
    • Have students do source search and evaluation in class.
  • Writing stage:
    • Students submit early/multiple drafts of the paper.
    • Students document their process with a research journal, screenshots of the process, self-reflection, a logbook, portfolio, etc.
    • Students write in a shared document so the lecturer can follow their progress.
    • Students can do multiple rounds of in-class writing as preparation for their paper.
  • Revision and editing stage:
    • Students are asked to use track changes so that the lecturer can see the revisions made.
    • Students do revision in class.
    • Students to a presentation in class for peer review.
  • Presentation stage:
    • A defense or some use of critical questions to justify/defend a position.
    • Assessed seminars, group discussions.
    • Randomly sampling students for an oral exam about their written work.
    • Students can do research on topics like theories and concepts at home, but then (with or without notes) respond to an application question in class.

2. Design an alternative assessment format

Use an alternative assessment format

While assessment focused on writing can be an excellent method of assessing your students, go back to your learning outcomes and have another look: Are there other ways of assessing whether students have met them?

Varying assessment formats is advisable for equity reasons, and to ensure a more complete sense of whether the student has met the learning outcomes.

Possible alternative assessment formats:

  • On-site exam (MC, short answer, long answer, essay questions, open book).
    • Note: Traditional exams (and oral exams to an extent) are not very authentic forms of assessment, in that they are not what students will have to do in the real world. This can impact motivation. They are also not always suitable for higher-order cognitive levels, unless you do writing on location. In other words, these are short term fixes. In the long term, we are not equipping students to survive in a workplace with genAI if all they know how to do is traditional exams.
  • Oral exam.
    • Note: If you are going to lean heavily on oral exams or presentations, then you also need to provide learning opportunities and materials for students to develop speaking and presentation skills.
  • On-site presentation.
  • Creative assignment (podcast, video, website, poster presentation, blog, infographic, animation, etc.).
    • Note: Make sure students aren’t (dis)advantaged by lacking/having particular skills, i.e. coding or video editing. You may need to adjust some learning outcomes, teach some additional skills or provide some extra support. Always go back to your learning outcomes and constructive alignment to make sure the emphasis of your assessment is appropriately aligned to ensure validity.

However, some alternative options can still be impacted by the use of GenAI, so make sure you consider the learning process when constructing these assessments.

Conclusion

See below for the solutions that the participants came up with to make their assessment more AI-resilient. They indicated that they found the workshop especially useful for thinking critically about what they actually want from their students. What is the learning objective and how can it best be assessed?

Solutions that the participants will apply in their course

Focus on the intrinsic motivation of students

Pay more attention to students’ intrinsic motivation. Because students feel more inclined to do effortful work themselves if they are motivated and at ease. For example, by giving them more freedom in the assignments.

Add a defense

Replace the final presentation with a defense to test their knowledge. In that case, students have to respond directly to questions from lecturers and/or students, which is more difficult to have generated by GenAI.

Adjust the rubric

In the rubric of your course, give more weight to the final presentation (in particular the answers they give to the questions afterwards).

Add a writing assignment in class

Replace the take-home writing assignment by a writing assignment in class. But be aware that in this case, you might (also) assess other skills of your students.

Offer more supervision

Offer more supervision during a long-term assignment. For example, twice a week instead of once a week.

Connect it to an internship

Connect the assignment to their internship.

Questions?

If you have any questions or are interested in this workshop, please send an email to tlc-science@uva.nl.