Year 6 SATS and “playing the game” – the dysfunction of misaligned incentives

You juke the stats, and majors become colonels. I’ve been here before.
Roland Pryzbylewski – The Wire

Imagine this: Two ambitious teachers in the same school are desperately seeking promotion. They’re equally qualified and have similar levels of experience. Knowing that above all else the head teacher values dedication (and has a keen eye on the signing-in book), Teacher A decides to stay until 6 pm every night to clearly exhibit her dedication. In response, teacher B decides to stay until 6:30 pm the next night. This causes Teacher A to stay until 7 pm the night after that. This battle of one-upmanship continues until both are staying late into the evening every day. In this zero sum game, two people who would benefit from cooperating are forced into actions that leave neither person better off despite the cost to both. This is an example of a ‘race to the bottom‘, a phrase usually associated with economics. Thankfully, the hypothetical head teacher sees what is going on and intervenes: she states that she values time-management skills as much as hard work and actively discourages the two ambitious teachers from working late into the night. Problem solved.

There is a similar race to the bottom in primary schools with regards to year 6 SATS preparation, one that equally requires intervention from those in authority. It has taken place gradually, in small increments, so that the shift has hardly been noticed, but all involved seem to agree that something has gone wrong. Secondary teachers bemoan the unreliability of the results. Parents despair at the unnecessary pressure placed upon their children. Primary teachers complain about the narrowed curriculum, the Easter schools, the after-school SATs clubs that often begin in year 5 and the nudge-wink grey area between perfectly administered assessments and outright cheating. It is an open secret that this part of our education system is broken.

As with the example of the two ambitious teachers, the problem is that SATs results and league tables are a zero-sum game. No individual school benefits when the majority are pushing boundaries to tip SATs results in their favour, or “playing the game” as it is often called. To choose not to “play the game” is to burden one’s school with a significant disadvantage in the key measure against which every school stakeholder is judged. When a cohort consists of as few as 30 students, every small part of the game can have a large statistical impact. I’m sure there are head teachers who completely ignore the myriad ways – some subtle, some less so – that SATs results can be nudged in the right direction at the expense of children’s overall educational experience. However, head teachers are human beings, and human beings follow incentives. Rather than demanding that individual head teachers risk their careers for the sake of integrity – in a system that encourages them to compete against others that may not – why not change the incentives that have caused this mess?

Ofsted have taken a first small step towards fixing a problem that they inadvertently created. A greater focus on curriculum and less on statistically measurable outcomes is a start. (Nevertheless, one might argue that the Ofsted’s choice to ignore internal data – though done for sound reasons – may intensify the focus on SATs results.) The problem is that it takes effort to shift people from the status quo, and right now the status quo in primary schools is dysfunctional. Forceful new incentives are required. Here are three suggestions:

1. Ofsted could make it clear to schools that they will try to find out how much SATs-based revision has taken place, and communicate that this will be taken into account when looking at results. (I tweeted Sean Harford on this subject, and he referred me to the draft inspection handbook; however, unless I am mistaken, there is no clear and robust disincentive relating to this subject mentioned in the handbook.)

2. SATs could be administered at the start of year 7 with results feeding back to primary schools.[i] This would incentivise primary schools to – shock, horror – actually continue to teach core subjects properly all the way up until July. It would also incentivise primary schools to focus on long-term retention rather than cramming and exam strategies. (Naturally, end-of-year 6 summer schools would need to be disincentivised as these would no doubt start to crop up; such is the system that currently operates.)

3. Ofsted could get rid of the four grades. Schools should be judged as acceptable or not. Acceptable schools should be given improvement priorities that will contribute to the next Ofsted inspection. Unacceptable schools should be given immediate and wide-ranging support, and – obviously – where irremediable incompetence is found, people should lose their jobs. High Ofsted grades create complacency and low ones create despondency. Ditch them. Where Ofsted find particular practices that they think are potentially worthy of imitation, they should take note and share this information with other schools.

I’m sure there are other solutions to this mess, far more achievable than I could propose. Either way, the current state of affairs cannot be allowed to continue.

[i] One argument against point 2 is that secondary schools would not be able to meet the array of requirements that year 6 children have for these tests: readers, scribes, transcribing, 1:1 confidence-givers, quiet spaces for children who feel the need to read aloud, rest breaks, etc. However, given everything we know about working memory, I think that it is fair to say that some of these access arrangements are up for debate. For example, shouldn’t the ability to independently read and comprehend a question be part of what is assessed in a mathematics assessment? Is it appropriate for a child who can read 91 words per minute to have one hour for a reading test, while another student who reads 89 words per minute gets an additional 15 minutes. Is reading fluency not part of what is being assessed? This is a thorny topic and is beyond the scope of this post. It suffices to say that this is another grey area where schools are incentivised to do anything that might boost SATs results.

Protesting and the role of teachers

It happened towards the end of my NQT year in year six. Two pairs of ashamed eyes struggled to look at me as I asked the usual question: “What exactly happened on the playground?” I already knew some of it, but wanted to hear their interpretations. The boy spoke first. He was often in trouble so knew his best chance was to leave out key details:

“She pushed me into a bush.”

Nothing else. The girl’s turn:

“He followed me around for the whole of break time. He told me I was fat. And a hippo. And that no one liked me because I was fat. And because I was a hippo. I ignored him, but then he drew on my jumper on purpose in pen. He was right in my face.”

There was a pause before she struggled to say the last part:

“He said the reason my dad left was because I was so fat and ugly.”

Both children admitted to what they had done. She confessed because this was her first time she’d ever really broken the school rules. He confessed, reluctantly, because there were several witnesses to his crimes. Once I had spoken to the boy and sent him to sit in silence at the back of my classroom, I spoke to the girl who at this point had just managed to stop crying. The girl was the sort of child that makes teaching easy: polite, pleasant and attentive. Without being asked, she apologised for what she had done. In that moment, a part of me desperately wanted to say,

“You know what? Go back out to play. I’d probably have done the same thing, and I know you’ll never do anything like this again anyway.”

But this isn’t what I said. What I really said was something like this:

“I understand why you were so upset. There will be major consequences for the hurtful nonsense he said to you and the damage he did to your property. However, it’s never right to push someone, even if they say awful things. The moment he started saying nasty things you should’ve told an adult.”

I then dished out the school’s mandatory sanction for unwanted physical contact. In this moment, I didn’t get to express exactly how I felt. I didn’t get to admit that it would’ve taken a saint to have not reacted in that situation, that I felt bad for adding to her obvious feelings of guilt. (After all, she pushed him into a bush. She didn’t do – or intend to do – any actual damage.) No, in this situation it was my job to hold the line; to understand and sympathise, yes, but to hold the line on our school rules nonetheless. I’m wasn’t her father, her uncle, her grandfather or her friend. To her, I was the establishment.

We often forget that as teachers we are, whether we like it or not, establishment figures. Other than perhaps politicians and the police, teachers are the first group who come to mind when people conjur up their own personal representations of ‘the man.’ We all know or have been taught by teachers that would flinch at the thought of being representatives of the establishment – often the ones that call pupils ‘mate’ – but it is the truth. And being part of the establishment brings compromises. It means holding the line. It means that when children don’t turn up to school in order to protest – however worthy the cause – our role is not to encourage. The last thing they want, or need, are their teachers cheering them on. It is our role to hold children to account for their actions. It is our role to show them that civil disobedience comes at a personal cost and that they must carefully weigh up their actions without the guidance of authority figures. The tacit support of teachers for such actions would make the schools the protestors, not the students themselves. Like a mature parent, we provide the safe, responsible ballast against which young people can rebel. Part of being an authority figure is knowing that sometimes there has to be a difference between what we think and what we say.

*** Naturally, the details of the anecdote in the post above have been somewhat altered to protect the children involved.***

The puzzle of inference in reading

I was always jealous of people who could do cryptic crosswords. Observed up close, their completing of each clue seemed like a small act of genius, the answers appearing from nowhere, usually making some sense only in retrospect. I could see that there was a great deal of skill involved or at least I thought so. My attempts to learn the skill of crosswording began with guides to cryptic crosswords, which revealed that there were only a few different ways that a clue can be formed. For example, one clue type involves breaking down a word into chunks and then signaling each piece elsewhere in the clue. Here are a couple of examples:

Quiet bird has a sign on a strange occurrence (10) PHENOMENON (quiet = p; bird = hen; sign = omen; on = on)

Stop! Everyone on the road. (5) STALL (road = st; everyone = all)

I’ve had some experience of musical notation so the fact that the word ‘quiet’ might signify the letter p made sense. However, road signifying st (from street) caught me off guard. “What other crossword abbreviations might I need to know?” I thought. Fortunately, Wikipedia had me covered.[i] I found to my dismay that there were literally hundreds of abbreviations. Not only that, but the Wikipedia list – according to other crossworders – was nowhere near exhaustive, and this was just one type of clue. With each new clue type that I encountered, there appeared to be an entire language of signifiers and abbreviations. On top of this, solving each clue depended utterly on the limits of my vocabulary. What had appeared from the outset to be a single learnable skill was in fact a vast array of overlapping fragments of knowledge. It was like the famous art gallery scene in Ferris Bueller’s Day Off when Cameron, a teenager bunking off school, stares deeper and deeper into a pointillist masterpiece and sees the unified scene disintegrate into a multitude of individual coloured dots.[ii]

I know, I know: a truanting teenager choosing to go to the Art Institute of Chicago to contemplate existence in front of a Seurat painting is unrealistic. They’d choose a Rothko. Duh.

I think that my crossword experience can help us understand how to teach other skills that in reality consist of a vast array of overlapping fragments of knowledge, things like inference in reading…

Here is a statement from the national curriculum for year 5:

By the end of year 5, children should be able to understand what they read by drawing inferences such as inferring characters’ feelings, thoughts and motives from their actions, and justifying inferences with evidence.

Assuming that this drawing and justifying of inferences is to be performed using age-appropriate texts, a teacher is left with the same one question that they wrestle with most days: how on Earth do I teach this? Far too often, teachers’ understandable conclusion is to model the supposed skill of inference with one text and then have children mimic the skill with another text, one containing language and background knowledge with which the children are already familiar. This is the equivalent of teaching a novice crossworder by showing one clue and its solution and then giving them a clue of their own to solve, ensuring that the abbreviations, signifiers and vocabulary involved are already understood by the novice. In other words, in attempting to evidence a child’s understanding of a generic skill that doesn’t really exist, teachers – through necessity – avoid introducing children to the new language and background knowledge upon which inference is really based. If this all sounds slightly despairing, then fear not as perhaps our crossword example can shed some light on the methods through which we can develop children’s ability to make inferences while reading.

I found that by far the most effective way to learn how to solve crosswords was to place each question and solution side by side. For every clue that proved impenetrable – at first, this was all of them – I went straight to the answer and attempted to understand how it was derived. Sometimes the entire clue made sense; sometimes only parts did. Nevertheless, the more my own internal store of abbreviations and signifiers grew, the quicker I could isolate and understand those that were a little rarer. (Knowledge assists in the acquisition of more knowledge, a relationship that makes initial learning intimidating, but that eventually encourages mastery.) In short, to get better at answering crossword clues, I had to be shown thousands upon thousands of them, each one adding or reinforcing a fragment of relevant knowledge.

The parallels between this experience and learning to infer while reading are clear. Both crosswording and reading inference are the result of countless accumulated experiences of language, remembered and connected. In both cases we must remember that, first and foremost, we are building knowledge of language and of the world, something that has consequences for how we teach reading:

Ensure that your reading lessons revolve around children encountering and understanding *lots* of text. Do not underestimate the quantitative aspects of reading instruction.

2. Ensure children learn from a wide variety of texts, carefully chosen to develop children’s knowledge of language and of the world.

3. Make explicit connections to where children might have seen particular words or concepts before, perhaps in other texts you have read.

4. Ensure children become familiar with key cohesive devices in written language and how they work, specifically connectives and pronouns. (It might surprise you how rarely some children grasp what is being referred to by a given pronoun.)

5. How much independent work is appropriate will depend on where the children are in their learning journey and the difficulty of the text. When first starting out, the balance between independent practice and explicit explanation will strongly (but not entirely) favour the latter. There is much to value in simply explaining the meaning of some text, but children also need opportunities to grapple with meaning independently.

Thank you for reading. Constructive feedback is appreciated.

[i] https://en.wikipedia.org/wiki/Crossword_abbreviations

[ii] https://www.youtube.com/watch?v=ubpRcZNJAnE

[iii] https://learningspy.co.uk/featured/what-do-we-mean-by-skills/

[iv] https://schoolsweek.co.uk/how-i-wish-id-taught-maths-by-craig-barton/

Principle Skimmer

Anyone who has ever run an inset in a school is aware of the potential pitfalls. It’s either the first day of term or the last. Attention spans are shorter than usual and stomachs are rumbling as teachers begin to adjust to the abrupt change in their daily routine. You – the person entrusted to improve teaching in your school over the space of an hour or two – are under no illusions. You are only too aware of how little is retained a few months or even a few days after an inset, and you want yours to have some lasting impact. So, what do you do? The easy answer is create some key take-aways, some practical suggestions that the teachers can begin to use immediately. “Don’t worry about the principle,” you imply. “Here’s a useful strategy.” If you’re a leader in the school, perhaps this useful strategy becomes a non-negotiable expectation for some or all future lessons. Perhaps this even becomes the latest addition to the checklist of strategies that must be seen during the next round of lesson observations. The teachers’ practice will be observably altered, and you feel you’ve done your job. Congratulations! You’ve had an impact – that’s the good news; the bad news is that you and your colleagues may have been far better off if you just hadn’t bothered…

Attempting to enforce changes in behaviour often leads to unintended consequences. A famous example is that of Mexico City’s Hoy No Circular program.[i] In an attempt to encourage the use of public transport, the program banned older, less environmentally-friendly cars from being driven in the city on particular days, based on the last digit of the car’s licence plate. The plan backfired as car journeys were widely replaced by taxi journeys, and many people substituted ownership of one quite old car for two very old cars so that they could drive in the city on any day of the week. Mexico City’s air quality did not improve, though this has not stopped the same program being copied and rolled out across other cities; as with education, it is often the apparent simplicity of a strategy that appeals, rather than its efficacy. Forget the principle. Apply the strategy. In this way, schools are rarely any different.

Dylan Wiliam is clearly a man who recognises better than most the nuances in discussions on education, and I imagine that few principles are as widely accepted across primary schools as one he has promoted for decades: the need to clarify, share and understand learning intentions. So why does the man himself describe this element of formative assessment as “possibly…the least well done of all”?[ii] Perhaps it is because many schools have taken Wiliam’s principle of sharing clear learning intentions and turned it into non-negotiable strategies, usually in the form of written learning outcomes or objectives (LOs), that have been applied thoughtlessly with unfortunate consequences:

A small – but cumulatively significant – fraction of every lesson is wasted through the writing of LOs by children, sometimes in more than one language.
Being forced to write LO achieved at the end of each lesson unconsciously nudges teachers, especially inexperienced ones, towards the comforting but ultimately deluded view that there is no difference between what has been taught and what has been learned.
In some schools LOs have evolved into a way of determining in retrospect what a child has supposedly learned. “LO achieved? Oh, good. Tick that one off the list.” This is summative assessment at its weakest and a further waste of time.
The process of writing LO achieved or LO ongoing under pieces of work has undermined the use of research-informed ideas such as retrieval practice: as an inexperienced teacher once said to me, “I can’t cover that bit again. They’ve already got LO achieved in their book. How would that look?” The truth is obviously that every LO is ongoing until it is painstakingly committed to long-term memory. That is the nature of learning.

It goes without saying that I believe Wiliam understands – better than I ever could – how his eminently useful principle has at points been warped into a range of counter-productive strategies.

What do I think can be learned from this example? To those of us occasionally entrusted with the responsibility of improving teaching in our schools: We must not short-change our colleagues with ready-made, checklist-friendly, non-negotiable strategies. We owe it to our colleagues to trust them with the underlying principles of effective teaching and to let the individual strategies grow organically from these (though suggesting some strategies that have proven to be effective is a useful starting point).

(An addendum: Implicit in the above post is the idea of teacher autonomy. I’m in the fortunate position to work in a school that supports my desire to try new things and learn in the process. If you are a teacher and you work in a school that doesn’t allow for this autonomy, my recommendation is that you find one that does as soon as you can.)

[i] http://greeneconomics.blogspot.com/2006/10/unintended-consequences-of-driving.html

[ii] https://www.youtube.com/watch?v=fC29IyqPVr0