Something has been bothering me about education research for some time. It’s the tendency of people to see a big effect size and conclude that a way of doing things is the best choice for every single student.
Let me explain.
Suppose you have two teaching methods A and B, and you set up an experiment to help you decide which teaching method is better in some sense. So you teach one class using method A and another class with method B and give them a test, which you compare between class A and class B. Or perhaps you teach a class using method A and the same class at a different time using method B and do a test after both. Either way, you’ll do a t-test (unparied in the first case and paired in the second) to help you decide if the teaching method can predict a dfference in test score.
When you do a hypothesis test, such as a t-test, it will spit out a p-value which people use to declare whether an effect is “significant”. The word “significant” here means “signifying something” and importantly it doesn’t mean “important”. Indeed, an effect might be significant but actually be too small to be useful. For example, you may find a significant effect in the story above, but the difference in test scores is on average 1 percentage point. That probably doesn’t matter all that much, for all that the difference might be real.
In the previous paragraph, I decided the usefulness of the effect based on the literal size of the effect in comparison to the scale on which I was measuring. Another way to do it is to compare it not to the scale but to the variability of the individual’s results. For t-tests I’ve mostly seen people use “Cohen’s D”, which for the unpaired t-test is the difference in mean score divided by the standard deviation of scores, and for the paired t-test is the mean of the difference in score divided by the standard deviation. Many people say that a Cohen’s D of 0.25 is considered a small effect, a Cohen’s D of 0.5 is considered medium and a Cohen’s D of 1 is considered large. A Cohen’s D of 2 is very large.
But the problem is that Cohen’s D is all about averages. It doesn’t necessarily mean there is an effect for any pariticular individual.
Let me show you.
Imagine the situation where students under Teaching Method A get scores that are normally distributed with mean 4 and standard deviation 1, while under Teaching Method B they get scores that are normally distributed with mean 5 and standard deviation 1. Then the Cohen’s D is indeed a large effect of 1.
Let me choose 30 random students for each class and have a look at their scores.
In case you were wondering, the commands I wrote in Desmos to produce these scatters are:
A=(normaldist(4,1).random(30),1+uniformdist(-0.3,0.3).random(30))
B=(normaldist(5,1).random(30),2+uniformdist(-0.3,0.3).random(30))
The scatter of dots for Teaching Method B is definitely further to the right than the scatter of dots for Teaching Method A, but would you look at how much they overlap? Look at how many students in Teaching Method A got scores higher than someone in Teaching Method B.
So, even though the effect size was “large”, it doesn’t guarantee that all students perform better in Teaching Method A. Because effect size is about averages and averages aren’t about individuals but groups.
Let’s look at a very large effect size of 2, which is usually touted as almost a holy grail in education research, as if it somehow guarantees a better result. This is two random classes with mean scores 4 and 6 and standard deviation 1.
There are still many students in Teaching Method A with higher scores than students in Teaching Method B. The very large effect size isn’t enough to tell you what is best for every student.
The above experiments could be expected to have difficulty saying something about every individual because we didn’t put every individual in both situations. Let’s imagine a different experiment, where every student is taught using both Teaching Methods and tested both times, and we subtract the two scores to get the difference D = B – A. Then every student gets a difference. Suppose these differences were normally distributed with mean 1 and standard deviation 1. Then the Cohen’s D is indeed a large effect size of 1.
Here’s 30 random students in that distribution:
Look at how many students were below 0! For those students, their results in Teaching Method B are actually worse than Teaching Method A. Yes on average Teaching Method B tends to produce a score 1 more than Teaching Method A, but not for everyone individually.
What if the Cohen’s D was 2 instead? Then these are the differences for a random class of 30:
There’s still one student who did worse in Teaching Method B. Admittedly, there will often be random classes of 30 where everyone’s difference is indeed greater than 0, since the actual probability of a negative difference is about 2.3%, based on a normal distribution. But that will work out to a lot of people if you think of all the classes in your school district, or a university class of 500.
My point is that even if your effect size is large then you can’t guarantee that your teaching method makes a positive difference for everyone. Yes it might make a positive difference for most, so if the research has a big positive effect size, it’s certainly worth giving something a go, but remember to keep an eye out for those people that the effect size doesn’t speak for.
PS: There are any number of other things to consider when interpreting education research and I don’t need to be told what they are here. I am just talking about this one thing today.
Leave a Reply