Value-added modeling
Encyclopedia
Value-added modeling is a method of teacher evaluation that measures the teacher's contribution in a given year by comparing current school year test scores of their students to the scores of those same students in the previous school year, as well as to the scores of other students in the same grade. In this manner, value-added modeling seeks to isolate the contribution that each teacher makes in a given year, which can be compared to the performance measures of other teachers. Critics say that the use of tests to evaluate individual teachers has not been scientifically validated, and much of the results are due to chance or conditions beyond the teacher's control, such as outside tutoring.
s use a student's past test scores to predict the student's future test scores, on the assumption that students usually score approximately as well each year as they have in past years. The student's actual score is then compared to the predicted score. The difference between the predicted and actual scores, if any, is assumed to be due to the teacher and the school, rather than to the student's natural ability or socioeconomic circumstances.
In this way, value-added modeling isolates the teacher's contributions from factors outside the teacher's control that are known to strongly affect student test performance, including the student's general intelligence
, poverty
, and parental involvement.
By aggregating all of these individual results, statisticians can determine how much a given teacher typically improves student achievement, compared to how much the typical teacher would have improved student achievement.
, New York City Department of Education
and District of Columbia Public Schools
. The rankings have been used to decide on issues of teacher retention and the awarding of bonuses, as well as a tool for identifying those teachers who would benefit most from teacher training. Under Race to the Top
and other programs advocating for better methods of evaluating teacher performance, districts have looked to value-added modeling as a supplement to observing teachers in classrooms.
Louisiana
legislator Frank Hoffmann
introduced a bill to authorize the use of value-added modeling techniques in the state's public schools schools as a means to reward strong teachers and to identify successful pedagogical methods, as well as providing a means to provide additional professional development for those teachers identified as weaker than others. Despite opposition from the Louisiana Federation of Teachers, the bill passed the Louisiana State Senate
on May 26, 2010, and was immediately signed into law by Governor Bobby Jindal
.
Experts do not recommend using value-added modeling as the sole determinant of any decision. Instead, they recommend using it as a significant factor in a multifaceted evaluation program.
Because each student's expected score is largely derived from the student's actual scores in previous years, it difficult to use this model to evaluate teachers of Kindergarten and first grade. Some research limits the model to teachers of third grade and above.
Schools may not be able to obtain new students' prior scores from the students' former schools, or the scores may not be useful because of the non-comparability of some tests. A school with high levels of student turnover may have difficulty in collecting sufficient data to apply this model. When students change schools in the middle of the year, their progress during the year is not solely attributable to their final teachers.
Value-added scores are more sensitive to teacher effects for mathematics than for language. This may be due to widespread use of poorly constructed tests for reading and language skills, or it may be because teachers ultimately have less influence over language development. Students learn language skills from many sources, especially their families, while they learn math skills primarily in school.
There is some variation in scores from year to year and from class to class. This variation is similar to performance measures in other fields, such as Major League Baseball
and thus may reflect real, natural variations in the teacher's performance. Because of this variation, scores are most accurate if they are derived from a large number of students (typically 50 or more). As a result, it is difficult to use this model to evaluate first-year teachers, especially in elementary school, as they may have only taught 20 students. A ranking based on a single classroom is likely to classify the teacher correctly about 65% of the time. This number rises to 88% if ten years' data are available. Additionally, because the confidence interval
is wide, the method is most reliable when identifying teachers who are consistently in the top or bottom 10%, rather than trying to draw fine distinctions between teachers that produce more or less typical achievements, such as attempting to determine whether a teacher should be rated as being slightly above or slightly below the median.
, an economist currently at Stanford University. It was subsequently analyzed by Richard Murnane
of Harvard University among others. The approach has been used in a variety of different analyses to assess the variation in teacher effectiveness within schools, and the estimation has shown large and consistent differences among teachers in the learning pace of their students.
Statistician William Sanders
, a senior research manager at SAS introduced the concept to school operations when he developed value-added models for school districts in North Carolina
and Tennessee
. First created as a teacher evaluation tool for school programs in Tennessee in the 1990s, the use of the technique expanded with the passage of the No Child Left Behind legislation in 2002. Based on his experience and research, Sanders argued that "if you use rigorous, robust methods and surround them with safeguards, you can reliably distinguish highly effective teachers from average teachers and from ineffective teachers."
A 2003 study by the RAND Corporation prepared for the Carnegie Corporation of New York
, said that value-added modeling "holds out the promise of separating the effects of teachers and schools from the powerful effects of such noneducational factors as family background" and that studies had shown that there was a wide variance in teacher scores when using such models, which could make value-added modeling an effective tool for evaluating and rewarding teacher performance if the variability could be substantiated as linked to the performance of individual teachers.
The Los Angeles Times
reported on the use of the program in that city's schools, creating a searchable web site that provided the score calculated by the value-added modeling system for 6,000 elementary school teachers in the district. United States Secretary of Education
Arne Duncan
praised the newspaper's reporting on the teacher scores citing it as a model of increased transparency, though he noted that greater openness must be balanced against concerns regarding "privacy, fairness and respect for teachers". In February, 2011, Derek Briggs and Ben Domingue of the National Education Policy Center
(NEPC) released a report reanalyzing the same dataset from the L.A. Unified School District, attempting to replicate the results published in the Times, and they found serious limitations of the previous research, concluding that the "research on which the Los Angeles Times relied for its August 2010 teacher effectiveness reporting was demonstrably inadequate to support the published rankings."
The Bill and Melinda Gates Foundation is sponsoring a multi-year study of value-added modeling with their Measures of Effective Teaching program. Initial results, released in December 2010, indicate that both value-added modeling and student perception of several key teacher traits, such as control of the classroom and challenging students with rigorous work, correctly identify effective teachers. The study about student evaluations was done by Ronald Ferguson
. The study also discovered that teachers who teach to the test
are much less effective, and have significantly lower value-added modeling scores, than teachers who promote a deep conceptual understanding of the full curriculum. Reanalysis of the MET report’s results conducted by Jesse Rothstein, an economist and professor at University of California, Berkeley
, dispute some of these interpretations, however. Rothstein argues that the analyses in the report do not support the conclusions, and that "interpreted correctly... [they] undermine rather than validate value-added-based approaches to teacher evaluation.”
in August 2010 recognized that "American public schools generally do a poor job of systematically developing and evaluating teachers" but expressed concern that using performance on standardized test
s as a measuring tool will not lead to better performance. The EPI report recommends that measures of performance based on standardized test scores be one factor among many that should be considered to "provide a more accurate view of what teachers in fact do in the classroom and how that contributes to student learning." The study called value-added modeling a fairer means of comparing teachers that allows for better measures of educational methodologies and overall school performance, but argued that student test scores were not sufficiently reliable as a means of making "high-stakes personnel decisions".
Edward Haertel, who led the Economic Policy Institute research team, wrote that the methodologies being pushed as part of the Race to the Top
program placed "too much emphasis on measures of growth in student achievement that have not yet been adequately studied for the purposes of evaluating teachers and principals" and that the techniques of valued-added modeling need to be more thoroughly evaluated and should only be used "in closely studied pilot projects".
Most experts recommend using multiple measures to evaluate teacher effectiveness.
Method
StatisticianStatistician
A statistician is someone who works with theoretical or applied statistics. The profession exists in both the private and public sectors. The core of that work is to measure, interpret, and describe the world and human activity patterns within it...
s use a student's past test scores to predict the student's future test scores, on the assumption that students usually score approximately as well each year as they have in past years. The student's actual score is then compared to the predicted score. The difference between the predicted and actual scores, if any, is assumed to be due to the teacher and the school, rather than to the student's natural ability or socioeconomic circumstances.
In this way, value-added modeling isolates the teacher's contributions from factors outside the teacher's control that are known to strongly affect student test performance, including the student's general intelligence
Intelligence
Intelligence has been defined in different ways, including the abilities for abstract thought, understanding, communication, reasoning, learning, planning, emotional intelligence and problem solving....
, poverty
Poverty
Poverty is the lack of a certain amount of material possessions or money. Absolute poverty or destitution is inability to afford basic human needs, which commonly includes clean and fresh water, nutrition, health care, education, clothing and shelter. About 1.7 billion people are estimated to live...
, and parental involvement.
By aggregating all of these individual results, statisticians can determine how much a given teacher typically improves student achievement, compared to how much the typical teacher would have improved student achievement.
Uses
, a few school districts across the United States had adopted the system, including the Chicago Public SchoolsChicago Public Schools
Chicago Public Schools, commonly abbreviated as CPS by local residents and politicians and officially classified as City of Chicago School District #299 for funding and districting reasons, is a large school district that manages over 600 public elementary and high schools in Chicago, Illinois...
, New York City Department of Education
New York City Department of Education
The New York City Department of Education is the branch of municipal government in New York City that manages the city's public school system. It is the largest school system in the United States, with over 1.1 million students taught in more than 1,700 separate schools...
and District of Columbia Public Schools
District of Columbia Public Schools
District of Columbia Public Schools is the traditional public school system of Washington, D.C. in the United States.- Composition and enrollment :...
. The rankings have been used to decide on issues of teacher retention and the awarding of bonuses, as well as a tool for identifying those teachers who would benefit most from teacher training. Under Race to the Top
Race to the Top
Race to the Top, abbreviated R2T, RTTT or RTT, is a $4.35 billion United States Department of Education competition designed to spur innovation and reforms in state and local district K-12 education...
and other programs advocating for better methods of evaluating teacher performance, districts have looked to value-added modeling as a supplement to observing teachers in classrooms.
Louisiana
Louisiana
Louisiana is a state located in the southern region of the United States of America. Its capital is Baton Rouge and largest city is New Orleans. Louisiana is the only state in the U.S. with political subdivisions termed parishes, which are local governments equivalent to counties...
legislator Frank Hoffmann
Frank Hoffmann
Frank Albert Hoffmann is a Republican member of the Louisiana House of Representatives from District 15 in Ouachita Parish, Louisiana. He is known as a leading legislative opponent of abortion...
introduced a bill to authorize the use of value-added modeling techniques in the state's public schools schools as a means to reward strong teachers and to identify successful pedagogical methods, as well as providing a means to provide additional professional development for those teachers identified as weaker than others. Despite opposition from the Louisiana Federation of Teachers, the bill passed the Louisiana State Senate
Louisiana State Senate
The Louisiana State Senate is the upper house of the state legislature of Louisiana. All Senators serve four year terms and are assigned multiple committees to work on. The Republicans control the State Senate following a Special Election Victory in District 26 by Jonathan W. Perry...
on May 26, 2010, and was immediately signed into law by Governor Bobby Jindal
Bobby Jindal
Piyush "Bobby" Jindal is the 55th and current Governor of Louisiana and formerly a member of the United States House of Representatives. He is a member of the Republican Party....
.
Experts do not recommend using value-added modeling as the sole determinant of any decision. Instead, they recommend using it as a significant factor in a multifaceted evaluation program.
Limitations
As a norm-referenced evaluation system, the teacher's performance is compared to the results seen in other teachers in the chosen comparison group. It is therefore possible to use this model to infer that a teacher is better, worse, or the same as the typical teacher, but it is not possible to use this model to determine whether a given level of performance is desirable.Because each student's expected score is largely derived from the student's actual scores in previous years, it difficult to use this model to evaluate teachers of Kindergarten and first grade. Some research limits the model to teachers of third grade and above.
Schools may not be able to obtain new students' prior scores from the students' former schools, or the scores may not be useful because of the non-comparability of some tests. A school with high levels of student turnover may have difficulty in collecting sufficient data to apply this model. When students change schools in the middle of the year, their progress during the year is not solely attributable to their final teachers.
Value-added scores are more sensitive to teacher effects for mathematics than for language. This may be due to widespread use of poorly constructed tests for reading and language skills, or it may be because teachers ultimately have less influence over language development. Students learn language skills from many sources, especially their families, while they learn math skills primarily in school.
There is some variation in scores from year to year and from class to class. This variation is similar to performance measures in other fields, such as Major League Baseball
Major League Baseball
Major League Baseball is the highest level of professional baseball in the United States and Canada, consisting of teams that play in the National League and the American League...
and thus may reflect real, natural variations in the teacher's performance. Because of this variation, scores are most accurate if they are derived from a large number of students (typically 50 or more). As a result, it is difficult to use this model to evaluate first-year teachers, especially in elementary school, as they may have only taught 20 students. A ranking based on a single classroom is likely to classify the teacher correctly about 65% of the time. This number rises to 88% if ten years' data are available. Additionally, because the confidence interval
Confidence interval
In statistics, a confidence interval is a particular kind of interval estimate of a population parameter and is used to indicate the reliability of an estimate. It is an observed interval , in principle different from sample to sample, that frequently includes the parameter of interest, if the...
is wide, the method is most reliable when identifying teachers who are consistently in the top or bottom 10%, rather than trying to draw fine distinctions between teachers that produce more or less typical achievements, such as attempting to determine whether a teacher should be rated as being slightly above or slightly below the median.
Research
The idea of judging the effectiveness of teachers based on the learning gains of students was first introduced into the research literature in 1971 by Eric HanushekEric Hanushek
Eric Alan Hanushek is a Paul and Jean Hanna Senior Fellow at the Hoover Institution of Stanford University. He is also an expert on educational policy. His main area of interest is the economics of education, focusing on controversial areas of education policy including the class size reduction,...
, an economist currently at Stanford University. It was subsequently analyzed by Richard Murnane
Richard Murnane
Richard Murnane is an economist and the Juliana W. and William Foss Thompson Professor of Education and Society at the Harvard Graduate School of Education. He has made important contributions to our understanding of education policy and the relationship between the economy and education. He has...
of Harvard University among others. The approach has been used in a variety of different analyses to assess the variation in teacher effectiveness within schools, and the estimation has shown large and consistent differences among teachers in the learning pace of their students.
Statistician William Sanders
William Sanders (statistician)
Dr. William L. Sanders is a senior research fellow with the University of North Carolina. He developed the Tennessee Value-Added Assessment System , also known as the Educational Value-Added Assessment System , a method for measuring a teacher's effect on student performance by tracking the...
, a senior research manager at SAS introduced the concept to school operations when he developed value-added models for school districts in North Carolina
North Carolina
North Carolina is a state located in the southeastern United States. The state borders South Carolina and Georgia to the south, Tennessee to the west and Virginia to the north. North Carolina contains 100 counties. Its capital is Raleigh, and its largest city is Charlotte...
and Tennessee
Tennessee
Tennessee is a U.S. state located in the Southeastern United States. It has a population of 6,346,105, making it the nation's 17th-largest state by population, and covers , making it the 36th-largest by total land area...
. First created as a teacher evaluation tool for school programs in Tennessee in the 1990s, the use of the technique expanded with the passage of the No Child Left Behind legislation in 2002. Based on his experience and research, Sanders argued that "if you use rigorous, robust methods and surround them with safeguards, you can reliably distinguish highly effective teachers from average teachers and from ineffective teachers."
A 2003 study by the RAND Corporation prepared for the Carnegie Corporation of New York
Carnegie Corporation of New York
Carnegie Corporation of New York, which was established by Andrew Carnegie in 1911 "to promote the advancement and diffusion of knowledge and understanding," is one of the oldest, largest and most influential of American foundations...
, said that value-added modeling "holds out the promise of separating the effects of teachers and schools from the powerful effects of such noneducational factors as family background" and that studies had shown that there was a wide variance in teacher scores when using such models, which could make value-added modeling an effective tool for evaluating and rewarding teacher performance if the variability could be substantiated as linked to the performance of individual teachers.
The Los Angeles Times
Los Angeles Times
The Los Angeles Times is a daily newspaper published in Los Angeles, California, since 1881. It was the second-largest metropolitan newspaper in circulation in the United States in 2008 and the fourth most widely distributed newspaper in the country....
reported on the use of the program in that city's schools, creating a searchable web site that provided the score calculated by the value-added modeling system for 6,000 elementary school teachers in the district. United States Secretary of Education
United States Secretary of Education
The United States Secretary of Education is the head of the Department of Education. The Secretary is a member of the President's Cabinet, and 16th in line of United States presidential line of succession...
Arne Duncan
Arne Duncan
Arne Duncan is an American education administrator and currently United States Secretary of Education. Duncan previously served as CEO of the Chicago Public Schools.-Early years and personal:...
praised the newspaper's reporting on the teacher scores citing it as a model of increased transparency, though he noted that greater openness must be balanced against concerns regarding "privacy, fairness and respect for teachers". In February, 2011, Derek Briggs and Ben Domingue of the National Education Policy Center
National Education Policy Center
The National Education Policy Center is a non-profit education policy research center located in the School of Education at the University of Colorado at Boulder. It was founded in 2010 and is funded by a variety of governmental organizations, NGOs, and foundations...
(NEPC) released a report reanalyzing the same dataset from the L.A. Unified School District, attempting to replicate the results published in the Times, and they found serious limitations of the previous research, concluding that the "research on which the Los Angeles Times relied for its August 2010 teacher effectiveness reporting was demonstrably inadequate to support the published rankings."
The Bill and Melinda Gates Foundation is sponsoring a multi-year study of value-added modeling with their Measures of Effective Teaching program. Initial results, released in December 2010, indicate that both value-added modeling and student perception of several key teacher traits, such as control of the classroom and challenging students with rigorous work, correctly identify effective teachers. The study about student evaluations was done by Ronald Ferguson
Ronald Ferguson (economist)
Ronald F. Ferguson is an economist who researches factors that affect educational achievement. Major themes in his work include the race-related achievement gap in the United States and how to improve schools and identify effective teachers....
. The study also discovered that teachers who teach to the test
Teaching to the test
Teaching to the test is an educational practice where the curriculum is centered primarily around an end assessment or standardized test. The practice is designed to give students a set range of knowledge or skills that will allow them to enhance their performance on tests...
are much less effective, and have significantly lower value-added modeling scores, than teachers who promote a deep conceptual understanding of the full curriculum. Reanalysis of the MET report’s results conducted by Jesse Rothstein, an economist and professor at University of California, Berkeley
University of California, Berkeley
The University of California, Berkeley , is a teaching and research university established in 1868 and located in Berkeley, California, USA...
, dispute some of these interpretations, however. Rothstein argues that the analyses in the report do not support the conclusions, and that "interpreted correctly... [they] undermine rather than validate value-added-based approaches to teacher evaluation.”
Criticism and concerns
A report issued by the Economic Policy InstituteEconomic Policy Institute
The Economic Policy Institute is a 501 non-profit, liberal, nonpartisan think tank that seeks to broaden the public debate about strategies to achieve a prosperous and fair economy...
in August 2010 recognized that "American public schools generally do a poor job of systematically developing and evaluating teachers" but expressed concern that using performance on standardized test
Standardized test
A standardized test is a test that is administered and scored in a consistent, or "standard", manner. Standardized tests are designed in such a way that the questions, conditions for administering, scoring procedures, and interpretations are consistent and are administered and scored in a...
s as a measuring tool will not lead to better performance. The EPI report recommends that measures of performance based on standardized test scores be one factor among many that should be considered to "provide a more accurate view of what teachers in fact do in the classroom and how that contributes to student learning." The study called value-added modeling a fairer means of comparing teachers that allows for better measures of educational methodologies and overall school performance, but argued that student test scores were not sufficiently reliable as a means of making "high-stakes personnel decisions".
Edward Haertel, who led the Economic Policy Institute research team, wrote that the methodologies being pushed as part of the Race to the Top
Race to the Top
Race to the Top, abbreviated R2T, RTTT or RTT, is a $4.35 billion United States Department of Education competition designed to spur innovation and reforms in state and local district K-12 education...
program placed "too much emphasis on measures of growth in student achievement that have not yet been adequately studied for the purposes of evaluating teachers and principals" and that the techniques of valued-added modeling need to be more thoroughly evaluated and should only be used "in closely studied pilot projects".
Alternatives
Several alternatives for teacher evaluation have been implemented:- Absolute standardized test scores: Teachers and schools are presumed to be effective if their students score well on standardized tests, and ineffective if they don't. In practical terms, this means that teachers of affluent, white, and Asian students are declared to be high-performing, and teachers of English language learners, poor, black, Latino, Native American students are declared to be low-performing.
- Evaluation by school principals: The school principal makes a pre-announced visit to the classroom, to observe a specially prepared lesson. The observation usually lasts for less than one hour, and happens once or twice a year. Afterwards, the principal issues a written report, often containing a checklist and a narrative evaluation, that almost always declares the teacher's overall performance to be satisfactory. In some school districts, the evaluation may be performed by senior teacher rather than, or in addition to, the principal. Rarely, independent observers, typically managed by the district office rather than the individual school, conduct teacher evaluations. Principal-led evaluations are criticized for perceived favoritism and for sometimes giving passing scores to more than 99% of teachers in a district.
- Evaluation by students: If asked validated questions, students as young as fourth graders can accurately identify effective teachers. Course evaluationCourse evaluationA course evaluation is a paper or electronic questionnaire, which requires a written or selected response answer to a series of questions in order to evaluate the instruction of a given course...
s are common in universities, but rarely count for more than a trivial fraction in a decision to retain or fire a teacher. - Activities outside the classroom: Part of a teacher's evaluation typically includes participation in staff training events. For example, a teacher who completes a master's degree is almost always paid more, even though holding a master's degree has no effect on student achievement.
Most experts recommend using multiple measures to evaluate teacher effectiveness.
See also
- Ipsative assessment, in which students are compared to their own previous performance
External links
- Los Angeles Teacher Ratings database and news stories by the Los Angeles Times