The Utah State Board of Education has led the school reform effort in Utah by advocating for a complete renovation of the Utah public education assessment system. The traditional fixed testing was designed for the student of medium ability and thus was an inaccurate measure of those with either low or high abilities. Additionally, these tests did not provide detailed data to teachers to inform instruction. Consequently, the fixed testing did nothing to improve college and career readiness test scores.
A new innovative approach is needed. The State Board believes that this new approach comes in the form of Computer Adaptive Testing or “CAT.”
The basic idea of CAT is that test items are selected by the computer to individually match the ability level of each student. In this manner, the test is tailored to each student. [...] With CAT, the computer begins by choosing an initial item, usually one at an intermediate level of difficulty, to administer to the student. If the student answers the item correctly, a more difficult item is selected next for administration. If the student’s answer is incorrect, an easier item is selected next for administration. This process of selection and evaluation is carried out by the computer throughout the test. By “adapting” the difficulty level of the items selected, the computer is able to accurately measure student ability using far fewer items than a traditional paper assessment. [...] The basis of CAT has been well-researched and its theoretical foundations are well established in the testing literature. Conceptually, because CAT capitalizes on the power of the computer to deliver a more efficient test, it is a compelling alternative to consider with the introduction of online assessments. – Walter D. Way, “Practical Questions in Introducing Computerized Adaptive Testing for K-12 Assessment,” Research Report 05-03 (March 2005).
The ultimate advantage of CAT lies in the ability of teachers to instantaneously use the data to inform instruction in real time. The data received accurately evaluates where each student in the class is succeeding or failing. Teachers can change lesson plans in order to direct teaching to these areas of weakness. Parents will know exactly where their child is in terms of grade level and competency.
The real test to any reform, in my opinion, is the result of assessments. So how does CAT stack up? Utah has conducted a pilot program in various school districts over the past few years in order to test the efficacy of CAT. Below you will see the 2009 CAT scores for the Sevier School District.
The results are stunning. Student test scores increased at a remarkable rate as teachers were better able to adapt lesson plans and curriculum to the needs of their students. As an example, let’s look at the Sevier District’s second graders. These students began the year four points below the national average in Math, but by year’s end they were nine points above the national average, an impressive 13-point gain over the course of the school year. Similar gains in Math were also seen in the third and fourth grade. In fact, nearly all grade levels finished the year well above the national average in Math, Language and Reading. Sevier School District’s growth over the course of the school year also outpaced the growth in the national data, as illustrated in the graph below. These types of success stories are the norm for CAT.
One drawback of CAT was that the U.S. Department of Education did not recognize CAT as an acceptable test for NCLB. However, in April 2011, Utah became the first state to win approval from the USDE for use of CAT to satisfy NCLB. We are the trailblazers for the rest of the country. The State Board intends to implement CAT as a replacement for CRTs statewide by 2014. The cost of this new testing structure is not cheap. It amounts to almost $6 million per year in additional funding to develop a test based upon the Utah Common CORE (which has been recognized as the Gold Standard of educational standards). Further, the State Board has determined that a 3:1 ratio of students to computers is needed to optimize the system. That amounts to another $30 million per year of additional funding. Notwithstanding the costs, the new assessment protocol is worth every cent. Our children must compete in the world market place. In order to compete, they need to have a solid educational foundation. CAT is a good way to ensure that they have that foundation.