We just had a research article posted in AERA Open (a fairly new online journal that you should be reading). I worked with Shana Shaw and Vincent Cho on this study, and you can access the article here. Read on, because it’s an interesting study. We were really stoked to do it.
We correlated teacher use of a data system with student achievement. In doing so, we actually counted the instructional actions that teachers took with the system, every day, all year, for two years. We didn’t count logons, logoffs, or other stuff that doesn’t matter to instruction. Instead, we counted things that were supposed to help them improve instruction – stuff like building a report, running a data query, looking at a student or class’s performance, or assigning work. The idea was that students might have better achievement if they learned from teachers use used the system a lot. We did the study right – we used complex, appropriate models, we looked at use longitudinally, we measured use two ways, we even had a pilot year.
So what did we find? Not much. We found one significance for one measure, with elementary reading. Overall, we weren’t able to demonstrate that higher uses of the system correlated with student achievement over time.
I gotta admit, I was pretty disappointed. If you’ve followed my work over the last billion years, you know that I was an early advocate for data systems and what they could provide for teachers. And I really wanted it for both the district and the vendor – they’ve worked hard on both sides of the relationship, throughout both organizations.
But here’s the deal: we aren’t yet ready to say this was the data system’s fault. We aren’t yet ready to say the results of this study mean that this or any other data system isn’t effective. Because in the end, it’s not about the data system – any more than the effects of data use are about the data themselves. It’s about what people do with the system, how we conceive of use, and how we support use of the system. Technology is absolutely necessary, but it’s a starting point. We wrote a pretty good Discussion section about this and I’m going to write about these things in my next blog post.
So go read the article if you can, and look for my next blog on this in a week or two. We’ll all learn. Thanks for reading.