Is 8% a big number?

Is 8% a big number? To some extent it depends on context but most of us would agree that it is usually a significant number. Who would turn down a pay increase of 8%?
A major study (cost = $12 000 000 USD) of twins in Australia and the United States has found that variations in student performance that is attributable to classroom factors such as teacher behaviours is of the order of 8% in literacy performance in the first three years of schooling (“Teacher Effects” in Early Literacy Development:Evidence from a Study of Twins, Byrne, B., Coventry, W., et al, 2010).
You may want to read that last paragraph again – yes, that’s right – “they” spent twelve million dollars to determine that different teachers produce different results – and that difference in results is around eight per cent.
Eight per cent.
Is that all? Other studies have put the figure as being in the order of 30% - 40% (Teacher – September 2009, p. 20). What might account for the difference? Apart from the staggering cost of the study, what makes the new figure interesting is that it was conducted using twins in different classes. As identical twins have virtually the same biology and essentially the same home environment it was felt that they provided a good indication about the impact of different teachers. We know that we are all unique – but twins, especially ones raised together, have a lot in common. Any difference in school performance is likely to be due to different experiences in the different classrooms. The study goes to some lengths to acknowledge that the experience in a classroom may not be entirely due to the teacher – but in broad terms, much of what goes on in a classroom is under the control of the teacher. In simple terms, twins taught by the same teacher showed minimal variation in performance, but twins taught by different teachers did – and after the statistical contortions were performed this figure was revealed to be a difference of eight per cent.
Now – it doesn’t take a mathematician to spot that 8% is a significantly less than 30% - and much less than 40%. How to explain the differences? It appears that the “twin study” is as statistically robust as can be realistically achieved – does that mean that it is right? Were there errors in the previous studies – or is the difference in the sample populations so significant as to make comparison difficult? But does it really matter – except to the people who provided the $12 000 000 of course? What matters, and what is confirmed by both studies, is that differences in teachers lead to a difference in student outcomes. Interestingly, even a $12 000 000 study does not actually identify the exact behaviours that make the difference. (An online report of the study can be accessed here. )
So, what does this study really tell us? Surely “they” did not need to spend all that money to discover that teachers are not all the same and that some achieve better results than others – as measured by the performance of their pupils? Surely, even relatively casual acquaintance with the realities of school would indicate this. Or does it simply reveal the pernicious growth of the measurement myth – the notion that everything can be measured and given a distinct number?
Does 8% tell us anything of genuine significance? No. Would a confirmation of the 30 or 40% figure tell us more? No. In a sense the numbers are irrelevant. What the numbers tell us is simply this – that some teachers are more effective than others. I doubt we needed this study to tell us that. Teachers, all teachers, have an obligation to their students to be the best teacher that they can – to give 100% to every lesson every day. We owe it to our charges to continually improve – to be a better teacher at the end of the year than we were at the start. In short, teachers need to learners – not necessarily students, but definitely learners.

Sources:
Teacher, September, 2009, “Study questions “teacher effects”.
“Teacher Effects” in Early Literacy Development: evidence from a Study of Twins, Byrne, B., Coventry, W., et al, in J Educational Psychol. 2010 February 1; 102(1): 32-42.
Teacher image:
https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgBs2XrYV8naXQElVKWUK2DH_SnFZvQa4PaBoGjsdz3CEadquh6JC8CB7ywAq75zVB_6ZUHn3fxN5CCOjKjPCRfOyU56CDi87UqwRMmvUZ2275OTHrhTNw3xpfxJ5uAfje6Aoo0q8DUvAk/s1600/teacher.jpg