“Lies, damn lies and statistics”

attributed to Benjamin Disraeli

I can be a bit of smug sod at times. I often have those moments where in a conversation where I have to interject and say, “hang on a minute”, a la Marshall McLuhan in Annie Hall.

Quite often this happens in meetings where those present are poring through the statistics generated by RAISE Online. **This post is dedicated to that situation.**

Imagine that the national average level for a student leaving school is Level 4.

Now imagine there are 3 year groups, all with 6 students (incredible, I know). When they did their end of KS2 tests, they achieved the following outcomes:

Year 7 | Year 8 | Year 9 | |

Student 1 | 2 | 4 | 2 |

Student 2 | 3 | 4 | 2 |

Student 3 | 4 | 4 | 4 |

Student 4 | 4 | 4 | 4 |

Student 5 | 5 | 4 | 6 |

Student 6 | 6 | 4 | 6 |

Which year group, on average, did better? Of course if you’re reading this and you’re any good at Maths, you’ll know that on average (i.e. the mean) all 3 years performed equally well. There are differences, but based on average entry point they’re all the same – in this scenario, they’re in line with the national average. Let’s look at how this translates to expectations at the end of KS4.

Now, secondary schools these days often use KS2 starting points to determine targets. How they do this varies from school to school, but basically the common ideology boils down to this: *if a year group, on average, enters secondary school in line with national averages, then they should leave school, on average, in line with national.*

Now, at KS4 nationally, the proportions of students who make 3 levels progress in Maths are:

- 15% of students at L2
- 41% of students at L3
- 69% of students at L4
- 77% of students at L5

L6 students don’t have a value on the 2014 figures because based on KS2 measures there was no such thing as a ‘Level 6 student’ when that cohort left primary school – so I’ll go with about 85% of students making 3LP. This is arbitrary, I know, but I think it’s a fair figure to use compared with L5 students. It’s more than likely going to be higher when they start coming through the GCSE system.

Now let’s look at what happens when we process these figures to see the number of students that should make 3LP based on present national averages. From which year group do you think would expect most students would make 3LP? Surely it’s the ones with the greater number of ‘higher attainers’, i.e. those students at L5 or better?

Year 7 | Year 8 | Year 9 | |

Student 1 | 2 | 4 | 2 |

Student 2 | 3 | 4 | 2 |

Student 3 | 4 | 4 | 4 |

Student 4 | 4 | 4 | 4 |

Student 5 | 5 | 4 | 6 |

Student 6 | 6 | 4 | 6 |

0.15 | 0.15 | 0 | 0.3 |

0.42 | 0.42 | 0 | 0 |

0.69 | 1.38 | 4.14 | 1.38 |

0.77 | 0.77 | 0 | 0 |

0.85 | 0.85 | 0 | 1.7 |

Students making 3LP |
3.57 |
4.14 |
3.38 |

Well as you can see – it’s Year 8, where all 6 students achieved level 4 coming out of KS2, that should expect to produce the greater number of students making 3LP.

Now don’t get me wrong, it’s only a small sample and the difference between each group is fractional. However this difference will only be exacerbated in ‘normal’ size schools.

So what does this actually mean then? Well, a couple of things:

- Averages don’t tell the whole story – well, duh – but…
- A higher proportion of lower attainers in a school has a greater negative effect on the number of students you would expect to make 3LP based on national figures, compared with the proportion of higher attainers and their potential positive effect.

In other words, two year groups might have, on average, achieved levels in line with national averages – but once you look deeper and consider the proportions of students making 3LP (and 4LP for that matter) at each level, it paints a very different picture.

In fact, we can go even simpler than this. **It does not matter how many higher attainers are in your school – if there’s a significantly greater proportion of lower attainers (L3 or lower from KS2) compared with the previous year, even if the average progress/attainment is pretty much equivalent between the years, then your results are likely to be worse than last year.**

Now I’m sure for the vast majority of readers you probably already know this. But if you’re having a battle with those monitoring your efforts in your department and need something to back up your arguments, it might be worthwhile sending them this way. This will probably need further explanation – and you want me help you in any way with this then let me know, as we’ve all got to support each other in this accountability culture.

I’m going to do a follow-up post on this in the near future, as this concept has big potential in terms of the idea of target setting, how they can be set relatively fairly, and how ‘aspirational targets’ need to take into account not the year group as a whole but the proportions of students that make them up.

Reblogged this on The Echo Chamber.

LikeLike

I’ve just been having this exact conversation today …. surely a ‘top set’ teacher shouldn’t be judged by the same criteria as a ‘bottom set’ teacher. In our school 80% of your class are expected to make 3lp no matter what ability – i always thought that top set should be judged on the % of pupils making 4lp. Set 2 could perhaps be judged on 80% making 3lp,set 3 perhaps 75% 3 lp. Set 4 60%, set 5 100% make at l3ast 2 levels progress – I’m just thinking of these numbers off of the top of my head … im sure that there are more evidence based figures.

Surely in an era of performance related pay, serious consideration must be given to this.

As a professional i would want all of my students to make the maximum progress, but when you are judging performance of teachers that take different sets it seems grossly unfair to judge every group of students in the same way?

LikeLike

Hi Rob, I completely agree with your points and I’m writing a follow-up post that explains my theories on target setting for year groups and classes. Thanks for the feedback and I hope you continue to enjoy my writing.

LikeLike

Pingback: Thinking Aloud: On Statistics, Again | Teaching at the edge of chaos

Rob, you’re absolutely correct

Oh, and you work for innumerate idiots.

LikeLike