Last week, the U.S. Department of Education released a study that found K-12 student test scores on state math and reading tests were not significantly impacted when students used, for one-year, various educational software products. It received major media attention, including a front page article in The Washington Post, pieces on NBC’s The Today Show and ABC’s Good Morning America, and now several op-ed pieces. The most recent op-ed was penned for Business Week by U.S. Rep. Ron Kind (D-WI), who declared: “This study attempts to show the ineffectiveness of educational technology. Instead it simply reveals the Bush Administration's tunnel vision on the uses and value of it. The benefits education technology offers students go beyond merely passing core-curricular tests. With it, students are gaining the technology skills and knowledge they will need to compete in the 21st-century economy. While ensuring that our students are proficient in reading, math, and science is critical to their academic and employment futures, their ability to use technology tools, mine the resources of the Internet, and collaborate virtually with peers around the world are skills that high-paying employers seek as well.”

This study, if allowed to stand unchallenged, has major implications for not just the educational software industry but also for all involved in education generally. If educational software is dismissed as not helping students achieve, other conversations about the importance of computers and Internet access in classrooms are bound to follow.

Over the past week, numerous press releases and articles have begun to poke holes in the study’s methodology and whether its conclusions can be applied broadly. The Consortium for School Networking, the International Society for Technology in Education and the State Educational Technology Directors Association fired the first broadsides against the study. The Software Information and Industry Association also weighed-in with concerns.

The major criticisms of the study boil down to five major points. First is the issue of who was studied and whether software, as a single tool, can be held responsible for student test scores. While the study spanned 132 schools and nearly 10,000 students, most of the schools participating were low-income and academically underachieving. This suggests that many of the conditions necessary for academic success were not present in those schools and that their absence, not the software’s presence, was largely responsible for the failure of the software products to improve student test scores. Keith Krueger, the CEO of the Consortium for School Networking, a major K-12 ed tech organization, stated: “It is important to remember that educational software, like textbooks, is only one tool in the learning process. Neither can be a substitute for well-trained teachers, leadership, and parental involvement.”

Second, the study only surveyed 16 educational software titles out of hundreds of such titles commercially available. Thus, it is very hard to state that all educational software does not impact learning. In fact, the study released only aggregate results for each academic area, not individual results for each software product. So, some of the software products may actually have improved test scores in a statistically significant way but those gains were hidden by the way the study’s authors chose to display the data.

Third, the study itself essentially admits that the teachers using the software did not feel comfortable with it, even after training. Nearly all teachers in the software implementation classrooms received some training from vendors in using the products, but most reported a low degree of confidence in that training after they began using the products.

Fourth, the teachers reported that they used the products only 10% to 15% of all classroom time over the course of one year devoted to a particular subject. In the case of first grade reading software usage, this percentage translates into teachers using reading software approximately 30 hours per year. It seems difficult to label the software as failing to raise student scores when it was not used all that much.

Fifth, the study’s results appear to contradict results from other USDE studies that show students making real academic gains through their use of education technology. In Utah, Missouri and Maine, the eMINTS program which provides schools and teachers with education technology, curriculum and over 200 hours of professional development, was responsible for students in the eMINTS classroom achieving over 10 to 20 percentage points higher than students in the control classroom. Additionally, in Iowa, after connecting teachers with sustainable professional development and technology-based curriculum interventions, student scores increased by 14 percentage points in 8th grade math, 16 points in 4th grade math, and 13 points in 4th grade reading, when compared with control groups.

This is a big deal for the tech industry and is not going away soon because Congress and the Administration must decide whether the federal education technology program in No Child Left Behind is reauthorized and continues to receive funding. Stay tuned.

Jon Bernstein is President of Bernstein Strategy Group, Washington, D.C.