Liberty Street Economics

« | Main | »

November 28, 2011

Unintended Consequences in School Accountability Policies

Rajashri Chakrabarti and Noah Schwartz*

Over the past two decades, state and federal education policies have tried to hold schools more accountable for educating their students. A common criticism of these policies is that they may induce schools to “game the system” with strategies such as excluding certain types of students from computation of school average test scores. In this post, based on our recent New York Fed staff report, “Vouchers, Responses, and the Test Taking Population: Regression Discontinuity Evidence from Florida,” we investigate whether Florida schools resorted to such strategic behavior in response to a voucher program. We find some evidence that Florida’s schools strategically reclassified weak students into exempt categories, and we draw some lessons that are applicable to New York City’s education policies.


    The Florida Opportunity Scholarship Program, enacted in June 1999, made all students of a public school eligible for vouchers (or “opportunity scholarships”) to attend private schools if the school received two “F” grades in a period of four years. The program can be viewed as a “threat of voucher” program. A school receiving an “F” grade for the first time was exposed to the threat of vouchers, but vouchers were implemented only if the school received a second “F” grade within the next three years. Since vouchers were associated with a loss in revenue and negative publicity, threatened schools had a strong incentive to try to avoid a second “F” grade.

    While the scores of all regular students were included in the computation of school grades, scores of certain limited-English-proficient (LEP) students were excluded, as were scores of students in many categories of special education. Did this provision of the law induce schools to reclassify their low-performing students into exempt categories so as to artificially inflate scores?

    To examine whether the threat of vouchers induced schools to reclassify students in this manner, we consider the fact that there was a sharp discontinuity in how the threat of vouchers was applied. Schools that scored below a fixed cutoff received an “F” grade, and thus the threat, while schools that scored above the cutoff did not. By comparing the schools that fell just below the cutoff to those just above, we get an estimate of the effect of the threat of vouchers. These two groups were nearly identical in terms of socioeconomic and demographic characteristics. Thus, we are comfortable with the supposition that the only difference between them was that one group was subjected to the threat of vouchers while the other was not.

    Using this strategy, we find evidence that schools threatened by vouchers tended to classify a larger percentage of their students into the excluded LEP category. We focus on the elementary grades; grades 4 and 5 were the tested grades during this time period in Florida. The figure below shows the share of students classified into the excluded LEP category in threatened schools (which were just below the cutoff) versus non-threatened schools (which were just above the cutoff) in various grades, in the first year after the program went into effect (1999-2000). In Grade 4—the first tested grade—the threatened schools classified an additional 0.31 percent of their students into the excluded LEP category than did non-threatened schools. In addition, we find that in grade 3 (the entry grade to the tested grades), the threatened schools classified an additional 0.36 percent of their students into the excluded LEP category.  Both numbers are statistically different from zero. These findings suggest that F-schools were attempting to remove certain students from the test-taking pool by classifying them into the excluded LEP category. When we compare these two groups of schools in the year before the voucher program went into effect (1998-99), we observe no evidence of any additional classification by the soon-to-be-threatened group in the excluded LEP category (red bars in the figure below) in any of the grades, supporting the notion that the observed spike in excluded LEP classification in 1999-2000 was induced by the program.

LEP_chart

    The numbers above imply that the threatened schools tended to classify an additional 2.6 students in grade 3 and an additional 2.3 students in grade 4 into the excluded LEP category. These numbers are modest. While this strategy alone would probably not enable lower-end F-schools to make a “D” grade, schools close to and just below the cutoff could see this strategy make a difference. Note also that the strategy we use captures the effect of the program on schools that were close to the cutoff. Thus, schools situated far from the cutoff may not have taken these strategic steps.

    In contrast, we find no evidence of reclassification of students into excluded special education categories. This is not surprising because there were larger costs associated with such reclassification compared to reclassification into LEP. The main cost was posed by Florida’s McKay Scholarship Program for Students with Disabilities. This program made every disabled student in Florida public schools eligible for vouchers. So reclassification posed the risk that schools would lose the reclassified students and the revenue associated with them.

    The Florida experience yields important lessons for accountability policies in our region. New York City’s accountability policy, the Progress Report policy, and the federal No Child Left Behind (NCLB) law were both modeled on the Florida program, but with important differences. As in Florida, New York City assigns schools letter grades on a scale of A to F, based in part on student performance on standardized tests. These grades are published in annual Progress Reports and result in credible rewards and sanctions. However, unlike the Florida program, New York City’s program holds schools accountable for all English language learners (ELLs) and Special Education students. Moreover, New York City Progress Reports give schools extra credit for achieving progress for ELL and Special Education, as well as other high-needs groups (for example, students in the lowest third citywide). Like New York City’s Progress Reports, the federal NCLB law holds schools accountable for the performance of all students, including LEP students and students with disabilities. These program features are important steps forward in that they serve to eliminate adverse incentives for the type of strategic reclassification that took place in Florida.

    The general lesson is that policymakers must be careful when designing exemptions or special allowances for certain groups of students, as these accommodations can create adverse incentives and unintended consequences. While accountability policies must acknowledge the challenges schools face in educating students with limited English proficiency, disabilities, and other special needs, excluding these students entirely from accountability measures may induce struggling schools to reclassify low-performing students into the exempted categories. Such strategic steps may not improve the quality of education for the groups in question or for the reclassified students.

*Noah Schwartz is a former assistant economist in the Federal Reserve Bank of New York’s Research and Statistics Group.

Disclaimer
The views expressed in this post are those of the authors and do not necessarily reflect the position of the Federal Reserve Bank of New York or the Federal Reserve System. Any errors or omissions are the responsibility of the authors.

About the Blog

Liberty Street Economics features insight and analysis from New York Fed economists working at the intersection of research and policy. Launched in 2011, the blog takes its name from the Bank’s headquarters at 33 Liberty Street in Manhattan’s Financial District.

The editors are Michael Fleming, Andrew Haughwout, Thomas Klitgaard, and Asani Sarkar, all economists in the Bank’s Research Group.

Liberty Street Economics does not publish new posts during the blackout periods surrounding Federal Open Market Committee meetings.

The views expressed are those of the authors, and do not necessarily reflect the position of the New York Fed or the Federal Reserve System.

Economic Research Tracker

Image of NYFED Economic Research Tracker Icon Liberty Street Economics is available on the iPhone® and iPad® and can be customized by economic research topic or economist.

Economic Inequality

image of inequality icons for the Economic Inequality: A Research Series

This ongoing Liberty Street Economics series analyzes disparities in economic and policy outcomes by race, gender, age, region, income, and other factors.

Most Read this Year

Comment Guidelines

 

We encourage your comments and queries on our posts and will publish them (below the post) subject to the following guidelines:

Please be brief: Comments are limited to 1,500 characters.

Please be aware: Comments submitted shortly before or during the FOMC blackout may not be published until after the blackout.

Please be relevant: Comments are moderated and will not appear until they have been reviewed to ensure that they are substantive and clearly related to the topic of the post.

Please be respectful: We reserve the right not to post any comment, and will not post comments that are abusive, harassing, obscene, or commercial in nature. No notice will be given regarding whether a submission will or will
not be posted.‎

Comments with links: Please do not include any links in your comment, even if you feel the links will contribute to the discussion. Comments with links will not be posted.

Send Us Feedback

Disclosure Policy

The LSE editors ask authors submitting a post to the blog to confirm that they have no conflicts of interest as defined by the American Economic Association in its Disclosure Policy. If an author has sources of financial support or other interests that could be perceived as influencing the research presented in the post, we disclose that fact in a statement prepared by the author and appended to the author information at the end of the post. If the author has no such interests to disclose, no statement is provided. Note, however, that we do indicate in all cases if a data vendor or other party has a right to review a post.

Archives