Why Computer Adaptive Testing?

Dave Thomas, Utah State Board of Education District 4.

David L. Thomas is a member of the Utah State Board of Education, District 4. He is a former State Senator and Chair of the Senate Standing Committee on Education.

The Utah State Board of Education has led the school reform effort in Utah by advocating for a complete renovation of the Utah public education assessment system.  The traditional fixed testing was designed for the student of medium ability and thus was an inaccurate measure of those with either low or high abilities.  Additionally, these tests did not provide detailed data to teachers to inform instruction. Consequently, the fixed testing did nothing to improve college and career readiness test scores. 

A new innovative approach is needed.  The State Board believes that this new approach comes in the form of Computer Adaptive Testing or “CAT.”

The basic idea of CAT is that test items are selected by the computer to individually match the ability level of each student. In this manner, the test is tailored to each student. [...] With CAT, the computer begins by choosing an initial item, usually one at an intermediate level of difficulty, to administer to the student. If the student answers the item correctly, a more difficult item is selected next for administration. If the student’s answer is incorrect, an easier item is selected next for administration. This process of selection and evaluation is carried out by the computer throughout the test. By “adapting” the difficulty level of the items selected, the computer is able to accurately measure student ability using far fewer items than a traditional paper assessment. [...] The basis of CAT has been well-researched and its theoretical foundations are well established in the testing literature. Conceptually, because CAT capitalizes on the power of the computer to deliver a more efficient test, it is a compelling alternative to consider with the introduction of online assessments. – Walter D. Way, “Practical Questions in Introducing Computerized Adaptive Testing for K-12 Assessment,” Research Report 05-03 (March 2005).

The ultimate advantage of CAT lies in the ability of teachers to instantaneously use the data to inform instruction in real time. The data received accurately evaluates where each student in the class is succeeding or failing. Teachers can change lesson plans in order to direct teaching to these areas of weakness. Parents will know exactly where their child is in terms of grade level and competency. 

The real test to any reform, in my opinion, is the result of assessments. So how does CAT stack up? Utah has conducted a pilot program in various school districts over the past few years in order to test the efficacy of CAT.  Below you will see the 2009 CAT scores for the Sevier School District.

 

* Data graphed is the difference between District and National RIT scores for the 2008-09 school year, where zero equals the national score. Math results for grades 8, 9, and 10 are omitted because the table used as a source for this graph had these grades segregated into algebra and non-algebra groups for the Spring test only. Graph adapted from "NWEA Mean RIT Values" tables, Sevier School District.

The results are stunning. Student test scores increased at a remarkable rate as teachers were better able to adapt lesson plans and curriculum to the needs of their students. As an example, let’s look at the Sevier District’s second graders. These students began the year four points below the national average in Math, but by year’s end they were nine points above the national average, an impressive 13-point gain over the course of the school year. Similar gains in Math were also seen in the third and fourth grade. In fact, nearly all grade levels finished the year well above the national average in Math, Language and Reading. Sevier School District’s growth over the course of the school year also outpaced the growth in the national data, as illustrated in the graph below. These types of success stories are the norm for CAT. 

* Source: Sevier School District

One drawback of CAT was that the U.S. Department of Education did not recognize CAT as an acceptable test for NCLB.  However, in April 2011, Utah became the first state to win approval from the USDE for use of CAT to satisfy NCLB. We are the trailblazers for the rest of the country. The State Board intends to implement CAT as a replacement for CRTs statewide by 2014. The cost of this new testing structure is not cheap. It amounts to almost $6 million per year in additional funding to develop a test based upon the Utah Common CORE (which has been recognized as the Gold Standard of educational standards). Further, the State Board has determined that a 3:1 ratio of students to computers is needed to optimize the system. That amounts to another $30 million per year of additional funding. Notwithstanding the costs, the new assessment protocol is worth every cent. Our children must compete in the world market place. In order to compete, they need to have a solid educational foundation. CAT is a good way to ensure that they have that foundation.

Related posts:

2 comments to Why Computer Adaptive Testing?

  • Kent Harris

    Have you contemplated approaching Bill Gates or Steve Jobs with an offer ‘they shouldn’t refuse’. The collective futures of our kids and the nation are at stake and suggest we must make CAT available to every school in the nation. There is no bigger, more urgent priority. Every year in which we fail to teach our kids to read and do basic math at grade level is a year we doom millions of drop outs and under-educated graduates to lives of quiet desperation, literally. Our country needs leaders. Our country must do better in reducing an ever increasing ‘underclass’ that’s personally tragic and publicly unsupportable.

    Personalized learning assessed, adapted and delivered by computer is the only reasonable answer to the mandate of NCLB. You’ve shown CAT can be the beginning of a golden era of public education. Don’t let proprietary interests keep it out of the ‘minds’ of those that need it the most. Both Apple and Microsoft should see the potential for ‘updates’ and dominance in this wondrous application of data mining techniques already in existence. The kids, all the kids, should benefit. Good luck and congratulations on your accomplishment.

  • [...] federal permission to use it for reporting related to NCLB, but the Utah State Board of Education has embraced the idea and legislators are proposing a bill that would require schools statewide to implement [...]

Leave a Reply