Unavoidably Exploitable

We’ve now seen that exams aren’t a very good form of assessment. Educators are aware of this, but there is little that can be done about it.

Why?

The answer is simple. Educational systems are under enormous pressure to operate within two key constraints:

1. Resources. Implementing and scaling a high-quality method of assessment to potentially millions of students requires huge amounts of money, time and staff. There are very rarely huge amounts of any of these things available.
2. Fairness (standardisation). No student should be advantaged or disadvantaged by any aspect of the process, since nobody will tolerate any compromise on fairness.

Let’s break it down a bit further with some examples:

• All students across the country need to be taught the same material. If students were taught different material to each other, then some would be put at an advantage over others during the exam. Effectively, we must tell all students in advance exactly what can and cannot show up in the exam. This means that the syllabus must be exam-centred, rather than being education-centred.
• The syllabus can only contain material that is easily examinable.
• Multiple-choice or clear right-or-wrong answers will usually be assessed by a computer. For everything else, the examiner must be human (and will remain so for the foreseeable future). Computers cannot be trusted to assess even a basic written question.
• Most of the time, there are too many exams for one person to assess. Multiple examiners must be recruited, which costs money. Since the examiners are people, they will vary in the way that they assess answers. One examiner might award 50% where another would award 70%. Clearly this isn’t fair. The solution is for examiners to follow a mark scheme. The mark scheme is a document that very clearly articulates the scoring criteria for each individual question.
• All students need to take the exam at the same time. Otherwise students and teachers could leak information to others who haven’t yet taken the exam. This requires a lot of coordination across the country.
• All exams must be collected and marked within a few weeks. There is a limited budget for this. There is also limited availability of capable examiners. To remain on time and within budget, the length of exams must be limited. Longer exams take longer to assess, and therefore cost more to assess. The result is that for many months of education, you are assessed for at most a few hours.

There is no easy way to comply with all of these requirements and still maintain a high-quality exam system.

We solve a lot of these problems by introducing a high degree of standardisation to exams, but this leads to other problems. Standardisation must remain from one exam to the next. Otherwise students from one year might be advantaged or disadvantaged compared to the next year. This creates consistency between past exams and future exams. Since past exams and mark schemes are visible to future students, this consistency leads to predictability. Predictability can be exploited.

In a standardised system, the syllabus cannot change too frequently. Neither can the style and format of questions. This would reduce the level of standardisation and make exams less fair. Together with the fact that the syllabus can only contain easily examinable material, it’s inevitable that you will see similar or even the same questions appearing from one year to the next. This can be exploited.

Clearly, the goal of education is to get an education. But in a standardised system, your education is not built around getting an education. Your education is built around your exams. As a society, we choose to sacrifice the quality of our education so that it can be easily quantified at reasonable cost. What a lovely thought.

Universities want to provide higher-quality education, but in doing so they have to decrease the level of standardisation in their exams. As a general rule, university exams are far less standardised than large-scale exams taken at high school. This closes some of the holes exposed by standardisation, but it also reveals another set of exploitable flaws. We’ll talk about these in Part 3.

Whether you are at school, university, or elsewhere, it doesn’t matter. Any form of assessment faces a fundamental tradeoff between its quality, fairness, and the need to operate within the boundaries of limited resources. Since there will exist flaws in any configuration of this tradeoff, there is no form of assessment that you cannot take advantage of.

Back to Articles

josh@examinationgame.com