“Lies, damn lies and statistics”
attributed to Benjamin Disraeli
I can be a bit of smug sod at times. I often have those moments where in a conversation where I have to interject and say, “hang on a minute”, a la Marshall McLuhan in Annie Hall.
Quite often this happens in meetings where those present are poring through the statistics generated by RAISE Online. This post is dedicated to that situation.
Imagine that the national average level for a student leaving school is Level 4.
Now imagine there are 3 year groups, all with 6 students (incredible, I know). When they did their end of KS2 tests, they achieved the following outcomes:
|Year 7||Year 8||Year 9|
Which year group, on average, did better? Of course if you’re reading this and you’re any good at Maths, you’ll know that on average (i.e. the mean) all 3 years performed equally well. There are differences, but based on average entry point they’re all the same – in this scenario, they’re in line with the national average. Let’s look at how this translates to expectations at the end of KS4.
Now, secondary schools these days often use KS2 starting points to determine targets. How they do this varies from school to school, but basically the common ideology boils down to this: if a year group, on average, enters secondary school in line with national averages, then they should leave school, on average, in line with national.
Now, at KS4 nationally, the proportions of students who make 3 levels progress in Maths are:
- 15% of students at L2
- 41% of students at L3
- 69% of students at L4
- 77% of students at L5
L6 students don’t have a value on the 2014 figures because based on KS2 measures there was no such thing as a ‘Level 6 student’ when that cohort left primary school – so I’ll go with about 85% of students making 3LP. This is arbitrary, I know, but I think it’s a fair figure to use compared with L5 students. It’s more than likely going to be higher when they start coming through the GCSE system.
Now let’s look at what happens when we process these figures to see the number of students that should make 3LP based on present national averages. From which year group do you think would expect most students would make 3LP? Surely it’s the ones with the greater number of ‘higher attainers’, i.e. those students at L5 or better?
|Year 7||Year 8||Year 9|
|Students making 3LP||3.57||4.14||3.38|
Well as you can see – it’s Year 8, where all 6 students achieved level 4 coming out of KS2, that should expect to produce the greater number of students making 3LP.
Now don’t get me wrong, it’s only a small sample and the difference between each group is fractional. However this difference will only be exacerbated in ‘normal’ size schools.
So what does this actually mean then? Well, a couple of things:
- Averages don’t tell the whole story – well, duh – but…
- A higher proportion of lower attainers in a school has a greater negative effect on the number of students you would expect to make 3LP based on national figures, compared with the proportion of higher attainers and their potential positive effect.
In other words, two year groups might have, on average, achieved levels in line with national averages – but once you look deeper and consider the proportions of students making 3LP (and 4LP for that matter) at each level, it paints a very different picture.
In fact, we can go even simpler than this. It does not matter how many higher attainers are in your school – if there’s a significantly greater proportion of lower attainers (L3 or lower from KS2) compared with the previous year, even if the average progress/attainment is pretty much equivalent between the years, then your results are likely to be worse than last year.
Now I’m sure for the vast majority of readers you probably already know this. But if you’re having a battle with those monitoring your efforts in your department and need something to back up your arguments, it might be worthwhile sending them this way. This will probably need further explanation – and you want me help you in any way with this then let me know, as we’ve all got to support each other in this accountability culture.
I’m going to do a follow-up post on this in the near future, as this concept has big potential in terms of the idea of target setting, how they can be set relatively fairly, and how ‘aspirational targets’ need to take into account not the year group as a whole but the proportions of students that make them up.