IBDP results were released this week, and behind every celebratory social media post is a person or team knee-deep in the data: cohort averages, subject breakdowns, comparisons against last year and against the school down the road. Some will be slicing it further still, comparing genders, ethnicities, socioeconomic backgrounds, and whatever else their board or their local authority requires. Most will be lining up the grades against the “evidence-based” predictions a standardised test supplied months or even years earlier, to identify which students, departments, and even individual teachers have “underperformed”.
But the statistics themselves do no more than describe what’s happened. The value of the exercise sits in the interpretation, and in the strategic direction that follows from it. In too many schools, the individuals overseeing these things are expected to make statistical judgements that extend beyond their formal training. That’s fine if they recognise as much and bring in the right colleagues to ensure that the analysis is nuanced and robust. It is not if they overestimate their own ability and plough ahead, blissfully unaware.
“Quantitative assessment is out”
A teacher told me recently that her line manager had come back from a departmental PD session repeating something some schools have been saying for decades: quantitative assessment is “out”. She agreed, in principle. She knew she wasn’t allowed to give a straight five out of ten for five correct answers.¹ But she found something else perplexing: her purely qualitative expressions of her students’ achievements weren’t allowed to stand on their own either. What actually went into the spreadsheet, and onto every child’s report, was a grade on the same A-E scale that was used when every result was expressed as a numerical mark or a percentage from a test, only now it was tied to a content descriptor instead of a score. That grade was then the subject of quantitative analysis the moment it was mapped against standardised test results.
Quantitative methods haven’t gone away. They’ve simply moved to a higher level. Instead of a raw count of correct answers, or a weighted average of marks for a series of assessed tasks, schools’ spreadsheets are populated with judgement-based grades. But leaders, or whoever is considered the “data person” on the team, still invariably proceed to apply the same tried-and-tested, numerically expressed analytical techniques to surface the peaks, troughs, trends and outliers.
The teacher made a broader point too: schools in her part of the world routinely compare these grades with reports from standardised tests like NAPLAN and PAT, treating them as objective standards against which educators’ judgement should be checked, and concluding that judgement is “off” whenever the two don’t align. I’m sure the irony of a school espousing the virtues of qualitative assessment, and then testing teachers’ judgement using standardised data, isn’t lost on anyone reading this.
The pattern plays out in schools and systems across the globe. Same story, different acronym: SATs in English primary schools, GL Assessment’s PTE and PTM across the UK and international schools, Cambridge’s IBE Insight for IB cohorts, NWEA’s MAP Growth in the US. The list goes on. A school lines its cohort’s results up against the norm asserted by the test provider and treats any deviation as proof that a teacher did a better or worse job than the baseline, or that a student over- or under-performed relative to “expectation”.
What “standardised” actually means
Standardised scores are built on populations in the tens of thousands precisely because individual variation is expected, not exceptional. The people who build these tests know this, and say so in their documentation.
Take GL Assessment’s CAT4, a cognitive ability test widely used to benchmark and predict outcomes in UK and international schools, and commonly employed by the admissions departments of academically selective institutions. The company’s own technical report² gives the standard error of measurement for an overall CAT4 score as 3.4 (on a scale with an 82-point range) and states that the 90% confidence band for any single result is plus or minus six points. A 100 on the test report therefore doesn’t mean the student’s true score is 100. It means there’s a 90% chance it sits somewhere between 94 and 106. On individual test batteries, the band is even wider.
So, a reported CAT4 score is really the centre point of a surprisingly wide range of likely true scores. To put the numbers in context, in many grading systems a span of 12 points in an 82-point range (almost 15%) would represent more than a whole letter grade. That brings me to the test’s predictive power for certain qualifications: something many schools rely on to benchmark students’ and teachers’ performance.
According to the technical report, the correlation between CAT4 scores for the students in the sample and overall IB points is 0.35, which GL Assessment’s own commentary describes as “moderate”.³ Squared, that means only around 12% of the variation in the IB outcomes they analysed was explained by the student’s CAT4 score. For some subject areas, the correlation coefficient dipped below 0.25, dropping the explanatory power down to just 6%.
To be fair, that doesn’t necessarily mean that the proficiencies measured by CAT4 have little bearing on individual IB outcomes. The smaller and more narrowly self-selected sample described by the report would suppress the observed correlation even if the underlying relationship is strong. But one thing is clear: the published technical evidence doesn’t support treating CAT4 scores as a precise benchmark for individual IBDP performance.
Still, for admissions, individual progress and predictive purposes, many educators treat a single CAT4 score as gospel. A child sitting a few points below an arbitrary benchmark on any given sitting is well within the test’s margin of error, and yet in some schools it is the difference between being offered a place and not. In others, it lands them in a lower ability set or a remedial programme they don’t really need.
None of this is an argument against standardised assessments per se. Used for the purposes for which they were designed, they remain valuable tools. The problem arises when they are asked questions they were never built to answer with a precision that they will never be able to achieve.
The wrong problem
These challenges do not sit in isolation; they compound. Assessment data is increasingly based on descriptive criteria rather than raw numerical marks, and judgement-based data is inherently harder to standardise and moderate across teachers and schools. As a result, genuinely nuanced analysis is needed just to interpret it properly in the first place. That kind of analysis tends to produce more descriptive outputs and fewer hard numbers that are easily compared. But at institutional level, there is a natural tendency to sideline the equivocal parts, latch onto any hard facts that are presented in the executive summary, and put them up against standardised scores that carry far more uncertainty than most people realise.
Schools do need people who can read data well. Underlying patterns in a set of results get missed when nobody in the room can tell a meaningful gap from ordinary year-to-year variation, with costs for both current and future cohorts. That isn’t in dispute. But the conversation about data literacy in education almost always assumes the problem is scarcity. Not enough training, not enough expertise, not enough space given over to it in an already crowded CPD calendar.
In my experience, the more common problem is educators and administrators who have just enough knowledge to believe they understand what the data mean, but not enough to support genuine insight. I’m sure that readers with even a passing interest in cognitive science are already thinking “Dunning-Kruger.”
What Dunning & Kruger actually found
Most people, a past version of me included, picture the Dunning-Kruger effect as a graph⁴ of confidence rising, crashing, then slowly climbing back up as real competence develops. However, that isn’t quite what Kruger and Dunning found in 1999.
Their data doesn’t track anyone’s confidence over time. It compares different people at different levels of ability, measured once. Bottom-quartile performers rated their own ability around 30 percentile points higher than their actual score. Top performers slightly underestimated theirs.
The real finding turns out to be the better description of what’s actually happening in schools. After a single PD session, or a course run by a test provider as part of the commercial proposition, someone likely doesn’t yet have sufficient competence to see how much they don’t know. They have learned enough to get by in a leadership conversation that involves some stats, but in many cases this isn’t the start of a journey of self-discovery. A dozen other priorities mean they never develop their understanding significantly further.
Which brings me back to that line manager. Their PD session had given them enough insight to pick up “qualitative” and “quantitative” as a simple binary, and to understand that the former needed to take priority for individual student assessment. It hadn’t given them enough to engage with a discussion about the fact that A-E is a five-point sequential scale (fifteen, if you allow for pluses and minuses), and that this is bound to be analysed in a quantitative manner, whatever the criteria behind it. In truth, the only way to guarantee a purely qualitative assessment is to provide only written or verbal feedback that is in no way attached to a sequential scale. Everything else becomes a cell in a spreadsheet, probably attached to a formula conceived by whoever created the first version years ago.
An overconfidence problem, not a training problem
None of this is really about individual failure. Data literacy is one of the more consequential things school leaders are expected to have, and one of the least deliberately taught. It arrives ad hoc: whichever provider pitched most persuasively at group level, or whichever gap happened to be free in that term’s CPD calendar. It rarely runs longer than a day, and it is rarely revisited once delivered. But the confidence it produces lasts a good deal longer than the training itself, and it tends to be applied precisely at the level where the cost of being wrong compounds fastest.
Real data literacy sits closer to statistical reasoning than most CPD sessions currently allow for. It means understanding sampling, variance, and what a standardised score is actually built to answer, not simply which words and methods have fallen out of fashion. Until leaders are humble enough to recognise the limits of what a single PD session can teach them, and willing to lean on those who genuinely understand the statistics, the pattern will keep repeating.
¹ For what it’s worth, I see no problem whatsoever with giving a mark of five out of ten to someone who got five answers correct. The issue was never the number. It’s that the story can’t end there: that quantitative mark needs to be accompanied by the qualitative feedback a student actually needs to understand how to move forward and improve next time.
² GL Assessment, “CAT4 Technical Report — International Edition,” “Test reliability”: https://support.gl-education.com/media/2785/cat4-international-technical-report.pdf
³ GL Assessment, “CAT4 Technical Report — International Edition,” “International Baccalaureate (IB) indicators”: https://support.gl-education.com/media/2785/cat4-international-technical-report.pdf
⁴ The popular version of this graph shows confidence spiking early (sometimes labelled “Mount Stupid”), crashing into a “valley of despair” once the real difficulty of a subject becomes apparent, then climbing a “slope of enlightenment” towards genuine competence. It has no basis in Kruger and Dunning’s actual data: there’s no dip, no valley, no slope in what they measured. The curve appears to have been assembled sometime in the mid-2000s, in management training and internet culture, and has endured because it tells a better story than the finding it’s named after.