The Disaster of Centrally Planned Student Accountability

Most of the discussion around public school accountability has to do with school leaders and teachers.  In theory, if we reward or punish school employees for the student outcomes that we care about, the employees will have incentives to improve student level measures.  However, should individual students not also be held accountable for their own performance?  After all, individual students are best able to control their own behavior, so the optimal accountability standard may be set at the student level.  Nonetheless, even an omniscient and benevolent standard setter in the traditional system of schooling would harm many children for the rest of their lives.

In order to graduate high school and obtain a diploma, many students in the United States need to pass a standardized exit exam.[1]  More loosely, all high schools require their students to meet a bundle of standards through 12th grade in order to obtain a diploma.  If standards capture student level productivity, stronger graduation requirements translate to higher diploma value.  If the high school diploma held by the student is valued more by the job market, the student will be able to obtain a higher standard of living.  On the other hand, if standards are raised uniformly for all students within a traditional public school, many children will be harmed as a result of failing to meet the minimum requirements.

Winners & Losers

The beneficiaries of an increase in graduation standards will be those who would have graduated anyway and those who would have not graduated anyway.  The value of the diploma is greater, so those who would have graduated anyway receive a more valuable certificate, and those who would not have graduated can at least say they failed at a prestigious institution.  Those harmed by such a policy change would be those at the margin, who would have graduated with slightly lower standards, but fail to graduate because of the higher standards.  Further, higher standards could discourage some students from putting in extra effort to meet the requirements.

Julian Betts and Robert Costrell (2001) point out that increasing student standards results in groups of winners and losers, just like almost all technological change throughout history.[2]  When the industrial revolution occurred, almost all of society benefitted from technological progress; however, factory workers that lost their jobs to machinery were obviously worse off.[3]  I agree that setting standards in the traditional public school system creates winners and losers, but I believe that the analogy to technological progress is a little far-fetched.

We should not compare central planning optimization, through the choices of elected officials, to the system of voluntary exchange and customer-driven reform that leads to optimal living standards.  A better analogy would be that standard setting in public schools creates winners and losers similar to how democratic institutions create winners and losers in our agricultural industry through price manipulation.  Subsidization of agricultural products obviously helps those employed in the industry, but it harms every single consumer of fruits and vegetables.

Even an All-Knowing Standard Setter Must Fail

In theory, a benevolent and all-knowing dictator could set the optimal standard for each individual school district.  The standard setter would have all of the information necessary to solve the constrained optimization problem of what standard level would generate the largest overall benefit to society.  Even in this scenario, given the constraint of the current educational system, the dictator would have to make a conscious decision about harming many children.  Remember, the official is benevolent as well, so this task would be difficult, but necessary for the greater good, of course.

Betts and Costrell (2001) also argue that a decentralized system of standard setting could produce economically inefficient outcomes if the benefits of valuable diplomas are shared across districts.  After all, if I choose a low standard for my district, I could free ride off the highly valuable diplomas in neighboring districts, assuming that employers view all diplomas equally.  I agree that viewing diplomas equally could exist in the short run; however, it would vanish over time.  Since firms have the incentive to hire the best candidates at the lowest cost, they will notice differences in student productivity across districts.  If a company makes the heroic assumption that all diplomas are equally valuable, they will pay for doing so through lower productivity.  The companies that realize that students from different educational institutions are likely to have different productivity levels will be rewarded by the market.  Similarly, companies all around the world have recognized that successful graduates of Harvard University are typically highly productive and valuable. Therefore, in a system of voluntary exchange, and at least minimal levels of information, decentralized decisions will outperform centralized ones.

Removing the Constraint

If decentralized decisions outperform centralized ones, why should we stop at the district level of planning?  If we prefer localized knowledge sources and stronger incentives to perform well, we should set standards at the individual student level.  Standards should be set for each individual student, and the best way to align incentives and knowledge is to allow family units to decide what graduation requirements ought to consist of.  In this scenario, the standard setters (the parents) would be free of the constraint faced by the central planner.  Since parents would be choosing the standard for one student at a time, they would not have to choose who ought to be a winner and who ought to be a loser.  In this sense, the previously impossible problem could be solved by finding the right fit for each unique child.  In the system of voluntary educational experiences, every single child would have a strong chance to be a winner.

[1] Hemelt, S. W., & Marcotte, D. E. (2013). High school exit exams and dropout in an era of increased accountability. Journal of Policy Analysis and Management32(2), 323-349.

[2] Betts, J. R., Costrell, R. M., Walberg, H. J., Phillips, M., & Chin, T. (2001). Incentives and equity under standards-based reform. Brookings papers on education policy, (4), 9-74.

[3] Sale, K. (1996). Rebels against the future: the Luddites and their war on the Industrial Revolution: lessons for the computer age. Basic Books.

High Standards for Higher Education

In the United States, many students are graduating from college with enormous amounts of debt, little or no job experience, and a shortage of skills necessary to attain employment in their desired fields.  The most recent statistics show that over 44 million Americans hold over $1.3 trillion in student loan debt.[1]  This amount is equal to the gross domestic product of the country of South Korea.[2]  Some scholars argue that the potentially negative effects of higher education on many students necessitates state-driven accountability for colleges across the nation.  However, state-driven accountability systems designed to improve the education sector likely harm students and societies through homogenization, discrimination, rising costs, and deteriorated standards.

Proponents of top-down accountability measures in the K-12 education system claim that they are necessary since young children cannot make completely rational decisions about school attendance.  In addition, they lack the funding necessary to pay for private school tuition, and their parents may not have the resources, knowledge, or incentives to purchase homes based on school quality levels.  Importantly, families face colossal transaction costs associated with opting out of residentially-assigned public schools.  Oftentimes, perceived quality differentials are not excessive enough for families to make the move, so children must remain in low-quality educational institutions.  In this sense, top-down accountability driven by the state is necessary to ensure minimum quality levels in K-12 institutions.

Imperfect Rationality

Some scholars argue for top-down accountability systems in higher education as well.  After all, students are graduating from college with student loan payments of around $350 per month, and over a tenth of student loans become delinquent.  Since many students graduating from college end up arguably worse off than they would have been (buried in debt and four or more years of unexploited on-the-job training), we need to hold these institutions accountable to higher quality standards, they contend.  It is also argued that the standards be determined by state officials since young adults lack expert wisdom and doctorate degrees.

However, the rationality argument is not as strong as it is for K-12 settings.  College students are adults that can seek information on how to select higher education institutions.  Similarly, young adults can go to the grocery store without being nutritionists.  Sure, individuals with imperfect information may make suboptimal decisions; however, doing so can help them realize a level of responsibility and improve their future choices.  In addition, of course, dictating the types of minimum standards in place may prevent some students from making poor decisions.  Nevertheless, doing so would ensure that most students would be harmed in unintended ways.

Lowering the Bar

Obviously, having a state-driven standard requires that a small number of bureaucrats determine what the measure ought to be.  In the case of higher education, accountability standards have been based on simple measures such as retention, graduation rates, and job placement.[3]  These indicators all seem important at first glance.  We want students to persist through college, attain a degree, and finish with the skills necessary for desired employment.  The problem is strongly related to Campbell’s Law; in the political sphere, an incentivized measure can become useless through corruption and gaming of the system.  For example, if colleges were financially incentivized to have high graduation rates, they would be more likely to allow students to pass right through.  Similarly, job placement incentives could lower the bar for the types of employment that are pursued.

Nudging

If I were to pay you each time you smiled at your child, you would have an incentive to do so more often.  Therefore, at the margin, you would smile at your child even when they were not behaving as well as you would have otherwise liked.  The financial nudge may very well result in parents inadvertently conditioning children to behave worse in society.  The overall result is that the child is harmed through ineffective priming and society is harmed through costly rewards for smiling.

Perhaps most striking is the fact that these types of financial incentives push universities to admit students that have shown promise in the traditional K-12 setting.  If a disadvantaged student did not perform very well on standardized tests, or they are from a minority group that is less likely to graduate from college, a specific institution would have an incentive to discriminate against them.  Of course, the institution would believe they were being selective on merit; however, the measures could systemically favor non-minority students.  At the margin, institutions that would not admit students based on K-12 merit would now have to determine what merit means to the state.  Oftentimes, this type of selectivity comes in the form of cutting student off starting from the bottom of the distribution in terms of test scores or GPA.  In this way, students from disadvantaged households will be less likely to have the chance to attend a selective institution.  Importantly, colleges may be missing out on diverse talents that are not easily captured by ordinary academic achievement.

Less Specialization

The largest impact is on the supply side of higher education.  If all educational institutions are incentivized by state-driven accountability measures, we should expect colleges to construct similar educational products.  Similarly, if the government paid automobile manufacturers large sums of money to produce fuel-efficient cars, the market for cars would undoubtedly consist of automobiles with higher fuel efficiency, even if individual consumers did not care about that specific feature.  If the supply of higher education gravitates towards homogeneity, specialized instruction will be less likely to occur.  Since children are unique in their skills and interests, homogenization would limit the strength of the match between educators and students.  A less specialized educational product could lead to lower quality levels and could harm students and even employers seeking diverse talent.

If higher education institutions and their missions were uniformly shaped by top-down accountability measures, the K-12 system would be affected as well.  Traditional K-12 schools and their leaders would recognize the measures adopted by universities for admissions.  In addition, if K-12 institutions have the goal of preparing children for college, they will attempt to adopt a system that is the most efficient in doing so.  The result: students from the age of five to around twenty-two must narrowly focus on measures deemed important by a relatively small number of officials.  Instead, an individual could use those seventeen years of their lives determining what type of education will provide them with the skills necessary to gain desirable employment and a higher standard of living.

[1] https://studentloanhero.com/student-loan-debt-statistics/

[2] http://www.tradingeconomics.com/south-korea/gdp

[3] http://journals.sagepub.com/doi/pdf/10.1177/0002716214541042

It is Impossible for Our Current School System to be Accountable for Non-Cognitive Skills

It is becoming more and more obvious that student test scores do not capture all of educational quality.  Since educational institutions can shape non-cognitive skills such as self-control, determination, conscientiousness, effort, and respect, we need to find an effective way to hold schools accountable for these outcomes.  While we can attempt to hold residentially-assigned public schools accountable for shaping non-cognitive skills, I argue that the current system makes the act impossible.  Indeed, attempting to hold public schools accountable for these types of skills in the current system will destroy the measures and even harm individual students.

There is a growing school choice literature indicating a disconnect between student achievement and long-term outcomes.  Some evaluations have found small or zero impacts on test scores with large positive impacts on graduation and crime reduction (Chingos & Peterson, 2013; Cowen et al., 2013; DeAngelis & Wolf, 2016; Wolf et al., 2013).  Others have found large impacts on student test scores, but no impacts on long-term outcomes such as graduation (Angrist et al., 2014; Dobbie & Fryer, 2014; Tuttle et al., 2015; Unterman et al., 2016).  If test scores do not predict later life outcomes, it is not clear whether they are important for the overall well-being of individuals and societies.

This disconnect has led many scholars to identify ways to more-easily measure the things that we actually care about.  Angela Duckworth created a scale[1] to recognize non-cognitive skills of students through self-reported grit.  Zamarro, Hitt, and Mendez (2016) realized that survey and test answer patterns could measure non-cognitive skills.  In particular, they used survey non-response rates, survey careless answer patterns, and decline on tests to measure student effort on the PISA test and student survey.  While these are all honorable attempts to come up with ways to measure non-cognitive skills of students, using them to hold schools accountable will not work.

The Problems

The first problem with using such measures for school accountability has to do with Campbell’s Law.[2]  Whenever a measure is used for social decision-making, the measure becomes susceptible to corruption pressures, making it eventually useless.  Test scores are subject to this law; however, these non-cognitive measures are especially corruptible (Duckworth & Yeager, 2015).  For instance, a teacher could make sure that students do not skip questions and could entice them to describe themselves as hard-workers.  These measures would be very easy for a teacher to influence, but simple manipulation would certainly not indicate that students are actually improving character skills.  The very act of linking accountability to these non-cognitive measures would destroy them.

Even if these measures could not easily be manipulated, it is not clear what they are actually capturing.  Scholars claim that non-response rates and careless answering patterns could capture conscientiousness (Hitt, Trivitt, & Cheng, 2016), but it could very well be described as compliance or obedience.  In fact, we may not want to maximize these types of outcomes.  A highly obedient citizen is great in that they will be less likely to break the law and listen to authority; however, they will also be less likely to invent self-driving cars or other technologies.  In other words, altering the education of children in order to maximize these measures will result in large positive and negative externalities.  Similarly, Angela Duckworth’s self-reported grit scale could simply be capturing confidence, which is also beneficial, but only up to a certain level.  In addition, researchers have noted that reference group bias could harm our ability to use these types of measures for evaluation and accountability (West, 2016).

Obviously, when a teacher focuses on shaping certain skills that are deemed important by the experts, they put less effort into shaping other abilities.  As a result, focusing on things like obedience and respect could harm students academically and intellectually.  Similarly, focusing on compliance could even harm other non-cognitive skills that are arguably more important such as creativity or critical-thinking.  The biggest problem here is that the political process needs to determine a standard to hold teachers and schools accountable.  The creation of the standard assumes that all students need to focus on the same skills and abilities.  Clearly, some students will need more focus on character skills, while others will need more focus on academics.  Children are unique.  Assuming otherwise is toxic.

Only focusing on test scores assures that children will be harmed through receiving lower levels of character education.  Focusing on character education assures that the measures that we have will become obsolete, and that many children will maximize skills that are not socially optimal.  In addition, focusing on character education will ensure that many children will be deprived of academic achievement and some unmeasured skills such as creativity.  This is the dilemma.  So what else can we do?

The Only Solution

Understanding that the dilemma is inherent to the current system of residentially assigned public schooling is paramount.  All of the described problems illustrate the fact that we cannot centrally-plan what is best for every individual in society.  Even the best-intentioned leader cannot acquire the necessary knowledge, especially since children and families have largely diverse interests and needs.

The only way to make sure that needs are matched to desires is to allow individuals to choose their educational products.  If families value character skills such as self-control and determination, they will choose schools that are known to have a culture that fosters non-cognitive growth.  If, on the other hand, a particular family recognizes that they have the resources necessary to improve those skills at home, they can choose an institution that focuses more on academic achievement.  If a school is not providing the necessary non-cognitive development for its students, the institution will have to alter its model or shut down in the long-run.  If a school does a great job at providing character development, demand for their service will increase, and they will be rewarded with the ability to charge a higher price.  The raise in tuition will entice other educational institutions to adopt their best practices and even incentivize innovative schools to enter the market.

It is obvious that creating accountability standards centrally is an economically inefficient solution.  Nonetheless, should schools be held accountable to parents or bureaucrats?  After all, parents may choose schools that do not maximize social welfare since they are self-interested.  However, controlling the decisions of families in order to prevent wrong choices damages the lives of millions of children through forcing them to deal with the problems that are inherent to the political process.

 

 

 

 

Image Source:  http://static.tabatatimes.com/wp-content/uploads/2013/10/Sisyphus-Image-01C.jpg

[1] https://angeladuckworth.com/grit-scale/

[2] http://www.ethnography.com/index.php?s=Campbell%27s+Law

Did NCLB Leave Children Behind?

The No Child Left Behind Act (NCLB), passed in 2001, was a high stakes federal accountability reform for education systems in the United States.  The act mandated that a high percentage of students within a state had to meet minimum proficiency levels.  By 2014, this mandated proficiency level increased to around 100 percent of students.  There has been much debate about the potential merits and unintended consequences of NCLB.  Empirical evidence has found that NCLB may have improved student achievement; however, the act may have also decreased teacher job security while creating other perverse incentives.

Methodological Issues

The evidence on the effects of NCLB is mixed, and most studies have used difference in difference analytic strategies to attempt to dampen biased estimates.  Since the federal government did not randomly assign the given policy to treatment and control groups, no true experiments will ever exist on the topic.  Since this is the case, we must rely heavily on theory and examine the methodological rigor of the quasi-experimental evaluations.

Reback, Rockoff, and Schwartz (2014) performed the first nationwide study of the impacts of NCLB on individual incentives.  They created a measure of “risk of failure” among very similar schools in different states and argued that this captured the effect of the policy since schools in higher risk environments received stronger doses of the treatment.  Since we are comparing schools in completely different states, we should be skeptical about the strength of this study.  However, let us assume that the results are experimentally sound and true.

Empirical Results

They found that NCLB had positive impacts on student’s enjoyment of learning and academic achievement.  While a student survey could indicate that they enjoyed their learning environment more, it is not clear that this outcome is policy relevant or even informative.  Are students enjoying learning more due to increased stimulation, or because they only have to focus on standardized tests?  I enjoy my time when I skip the gym and eat chips while watching a movie; however, the task of exercising and eating healthy, while less joyful, is more rewarding to me and others in society.  The result that NCLB increased test scores may sound beneficial at first, but it is unsurprising and perhaps even harmful.

Opportunity Costs

If we focus our efforts on standardized testing, and attach rewards and punishments to scores, should we not expect scores to increase?  Suppose we allocated a billion dollars per student each year so that they could improve their test scores.  Would that be worth the investment?  At what point would diminishing returns set in, and what level of achievement is optimal for an individual?  Perhaps most importantly, what is the opportunity cost of allocating those funds towards academic achievement?

Obviously, if we forced all children to focus on standardized tests every day for thirteen years of their lives, they would realize higher scoresHowever, doing so would harm students’ unobservable non-cognitive skills such as effort, self-control, creativity, determination, and conscientiousness.  Those skills are arguably more important than test scores, so we need to realize that maximizing test scores could be harmful.  In addition, there is a huge disconnect in the literature between test scores and long-term outcomes.  Some studies show large test score impacts without changing graduation rates or incomes, while other studies show small test score effects (or none at all) with large impacts on

Indeed, Reback, Rockoff, and Schwartz (2014) found that NCLB caused teachers to allocate more time towards subject-specific instruction rather than whole-class instruction.  They also found that NCLB had negative impacts on teachers’ feelings of job security.  This result could go a few different ways.  It may be that teachers perform better when they feel like their job is less secure.  Since there is a specific punishment tied to job performance, teachers have the incentive to improve.  However, they would have a stronger incentive to improve according to what is measured on the tests.  Tests attempt to measure cognitive skills, so this could harm non-cognitive skills.  Furthermore, teachers make a cost-benefit decision to stay in or leave the workforce.  Since this policy would be an additional cost in the decision, higher-quality teachers, at the margin, may opt out of the schooling industry.  Instead, these teachers may pursue jobs in other industries with higher benefit / cost ratios, resulting in lower quality education for children in the long-run.

Ballou and Springer (2017) pointed out that proficiency levels, like the ones used in NCLB, created perverse incentives about teacher focus.  Theoretically, if there were students “at the bubble,” or just below the cutoff score, teachers would have an incentive to focus on those students to improve quality ratings.  This may be great for the bubble students, but could be disastrous for students at the tails of distributions.  Ballou and Springer (2017) do not find evidence to support this claim; however, they do point out that other evaluations have found the opposite result.  They use a grade-year fixed effects strategy to compare the effects of high-stakes testing on students in different locations of the distribution; however, they are not able to compare individual students to themselves over time.  Ballou and Springer (2017) also found suggestive evidence that NCLB could have had negative impacts on advantaged students at the top of the distribution of scores.

What Should We Do?

The empirical evidence suggests that NCLB improved test scores and student enjoyment, and decreased teacher job security.  These results follow intuition; if rewards and punishments are attached to standardized tests, student achievement (on those types of tests) will increase and teachers will feel more pressure.  Nonetheless, it is unclear whether this type of reform is beneficial overall, especially since schools can shape unobservable, non-cognitive skills as well.

Perhaps we need to focus on some other important questions instead.  What is the socially optimal level of student proficiency on standardized tests given our budget constraints?  What are the unintended impacts of focusing too much on academics?  How can we decide what amount is best for what types of children, and who ought to make that decision?  Should we force all students to focus on the same material, even though that is sure to produce negative impacts on millions of children; or should we allow individual families to decide how to reach the educational goals of their individual children?

 

Image Source:  https://s-media-cache-ak0.pinimg.com/564x/e4/c8/f1/e4c8f177bf38efa2c2e5476b5c98b7d1.jpg

How Much Value is there in Value-Added?

Because of numerous criticisms of standardized test scores being used to measure student ability, economists have attempted to enhance educational accountability through advanced statistical techniques.  Rather than holding teachers and schools accountable to mere examinations of test score levels, researchers have developed measures that account for student growth.  The most innovative system devised by scholars today is known as value-added measurementAlthough value-added is a substantial improvement to previous accountability measures, there are still many problems that the system cannot avoid.

What is Value-Added?

Value-added measurement usually relies on student fixed effects regression in order to determine the impacts of teachers and schools on student test scores.  Since many unobservable student characteristics are relatively constant over time, this technique can control for several factors that could otherwise produce biased estimates of the impacts of schooling.  The strongest types of value-added systems have information on individual students over time and a baseline measure of achievement.  As a result, the model can estimate how a student is predicted to perform, and compare that to how they actually perform.  If the student switches into a new year and performs better than expected, scholars interpret the difference as the effect of the teacher or school, depending on the model.

Random-Sorting & Predicting Scores

First, this framework assumes that students are exogenously switching from one teacher to another.  In other words, the model assumes that principals are not doing their jobs and are randomly assigning students to teachers.  We would hope that this is not the case, especially if we believe the principal should assign students to teachers based on their abilities, needs, interests, and learning styles.

In addition, these models examine students based on their observable characteristics (and baseline test scores) to predict how they ought to do the following year.  This is problematic since the model can only do so based on observable characteristics.  The model predicts a value based on things such as skin color, income level, and a crude measure of innate ability.  Obviously, two minority children coming from households with similar incomes and similar test scores in the previous year are different in ways that will affect their achievement in subsequent years.

Narrow Focused Tests Needed

Jacob and Rothstein (2016) discuss issues with other assumptions that researchers quickly make about these measures and their potentially damaging consequences.  For instance, if we want to use value-added to keep individual teachers accountable, the assessments must have an extremely narrow focus.  If we really want to attribute the growth of student achievement to individual subject-area teachers, we must ensure that the assessment does not capture information or skills that come from more than one classroom.  Additionally, to improve the measure, we would need the assessments to capture abilities that are malleable solely within individual classrooms.  In an attempt to hold teachers more accountable, by improving the value-added measure, we are likely to do much harm to students.

The Scaling Assumption

Perhaps even more importantly, even the best value-added models today assume that the test scores are internally scaled (Ballou, 2009).  In other words, receiving a test score gain from zero to 50 and a gain from 50 to 100 are assumed to result in an equal change in cognitive skills.  This is a heroic assumption; learning to recite the alphabet is not an equivalent jump in cognitive ability as going from being able to recite the alphabet to reading words.  Since the underlying attribute (aptitude or ability) that we care about is not directly measured, our crude measures do not allow us to make the assumption of equal scale across the points awarded on a given assessment.

However, test scores with non-internal scales could be used to predict other measures that are internally scaled, such as graduation and earnings.  Nevertheless, these long-term outcomes would take an enormous amount of time and effort to collect and analyze.  Further, even if these measures were costless to collect and analyze, such a model would require test scores to be able to predict graduation and earnings.  As we know, there is a huge disconnect in the literature between test score outcomes and long-term outcomes.  The voucher programs in Milwaukee and DC, for example, produced little or no test score gain, but large increases in high school graduation (Cowen et al., 2013; Wolf et al., 2013).  Alternatively, studies have shown some charter schools had a large test score impact with no change in high school graduation (Angrist et al., 2014; Dobbie & Fryer, 2014; Tuttle et al., 2015; Unterman et al., 2016).  In addition, this could cause us to overestimate achievement gaps since, on average, black students are more likely to graduate than white students with the same standardized test scores.

Important Unmeasurable Skills

And what about non-cognitive skills?  Even if we could, theoretically, perfectly measure the impacts that teachers and schools have on student cognitive skills, we would still miss the other half of the equation.  Focusing on cognitive, academic skills is probably beneficial for some students, but it can be harmful to many others.  If schools are meant to alter the life-trajectory of children, they must shape citizenship skills, determination, and conscientiousness.  Those skills cannot be measured accurately at this time, and probably will not be measured accurately in the future.

Even the most-refined measure that we have, value-added modeling, relies on the heroic assumptions of exogenous sorting and internally scaled assessments.  Even if the model somehow did not rely on these assumptions, the accountability system would require someone to decide what ought to be assessed, how much students ought to grow, and what the goals of education ought to be for all members of society. 

Uniform decisions such as these cannot, and will not, work for children with unique interests, abilities, and learning styles.  The only way to account for diverse children is to make schools accountable to the desires of families.  The only way to do that is to allow individual families to choose the type of education that is best for their children.

 

Image Source:  http://creativeelectron.com/wp-content/uploads/2015/03/Statistical-Analysis-X-Ray-sm.jpg

Unanswerable Questions in Educational Accountability

The debates surrounding the preferred structure of accountability systems in the education sector revolve around a few important questions.  As Figlio and Loeb (2011) point out, these discussions often focus on the level of accountability, whether or not to use extrinsic rewards, and the measurement of agent behavior.  While the merits to the various answers to these questions should be considered in depth, it is impossible to find optimal solutions to each of these problems.  In fact, it appears that the very quest to find the answers to these problems within the system of residentially assigned public schooling actually increases the number of remaining questions.

What Level?

First, at what level should we design accountability systems to operate at?  Scholars discuss the most preferred level for accountability systems, but any level is sure to result in principal-agent problems somewhere.  After all, the education sector is fraught with several complicated principal-agent relationships.  These relationships are intertwined between various actors such as parents, children, teachers, unions, school boards, principals, superintendents, mayors, and other representatives all the way up to the national level.  Since it would be extremely complex and costly to implement accountability systems for every single relationship, it is important that we at least discuss the merits of a couple of important types.

It may appear wise to set accountability standards at the school level.  If we do so, the leader of the school can make decisions within the school in order to create a successful institution.  Similarly, in the private industry, customers review the quality of different brands, and sometimes even review the quality of locations set up at specific geographic locations.  Thanks to historical customer reviews being available at our fingertips today, customers can determine if a specific fast food restaurant location is high or low quality.  However, it does not seem very useful to collect that information at the employee (or teacher) level.

Nevertheless, what about the school leader relationship with the teacher?  How is the school leader to know whether a specific teacher is high or low quality?  The principal needs a way to measure teacher quality other than how the given teacher treats the school leader.  Thus, an accountability measure is at least necessary at the teacher level.

A Problem with Measures

The problem here is that even the best econometric measures of teacher quality rely on a few important assumptions.  The often-praised measure of “value-added” requires that increases in the observed measure is desirable, and that maximizing the measure is preferable.

Furthermore, proponents of value-added measures often forget that the model assumes exogenous sorting of students and teachers.  As Derek Neal (2008) pointed out, the model assumes that school leaders are not doing their jobs at all.  After all, the job of the school principal is to find teachers that will help their specific types of students and to match students and teachers effectively.  Since (we hope) principals assign students to teachers that will best serve them, the value-added model fails the exogenous sorting assumption.  If the model fails one of its most important assumptions, how could we conscientiously assign rewards or punishments to teachers based on them?

Monetary Rewards and Opportunity Costs

The next question is whether we should give explicit rewards and/or punishments based on the measures.  For example, some scholars have proposed to use monetary incentives for teachers, or merit pay, in the educational setting.  Sure, monetary incentives allow the principal to shape the actions of the agent (or the teacher).  However, how does the school leader identify the optimal level of the measure, and, perhaps even more importantly, how do they determine if the additional costs incurred are worth the additional benefits?

This is a common theme in discussions of accountability systems: even bright scholars will forget the basic, but important, economic lesson of opportunity costs.  If a monetary incentive raises test scores, many people will argue that the proposed policy is ideal, especially since spending, in general, has not correlated with any improvement in educational outcomes that we care about (Hanushek, 1996).

However, those same people will completely disregard the other side of the equation by not asking the more important question: “Is the observed result worth the cost?  And what other programs could the same resources be allocated towards?”  In addition, do we want to maximize test scores, or do we simply want to reach a certain level and then focus on something else?  Surely, if we allocate 100% of our educational resources towards test scores, many children will suffer from decreased creativity levels and harmed social skills (Neal, 2010).

In addition, if we allocate a billion dollars to each student and find that they have magnificent test score gains in every subject, would the investment be worth it?  What if long-term outcomes also increased?  Suppose that we somehow knew, with certainty, that the billion-dollar investment, per student, doubled their lifetime earnings (an increase of a little over a million dollars, on average).  Obviously, this would be a horrible investment.  Instead, the investor should just give each student two million dollars.  The student would be better off (by one million dollars), and the investor would spend 99.8% less.

How to Fix These Problems

In short, it seems that an attempt to answer any of these accountability questions results in many more questions.  In this sense, answers to accountability problems within the system of residentially assigned public schooling do not reduce the number of remaining questions, but increase them exponentially.  Due to the complexity of relationships and diverse goals within societies, accountability systems cannot function within the public sector of schooling.  Perhaps most importantly, the very attempt to control the various mechanisms of these principal-agent relationships is likely to cause much unintended harm.  Instead, the schooling industry would perform much better in the complex, yet simple system of voluntary exchange.  All of the complicated relationships, goals, and desires would be efficiently communicated through the spontaneous formation of a single metric: price.

 

Image Source: http://dosminutos.complot.net/wp-content/uploads/2015/04/Problema.jpg

Why Force Children to Bear the Consequences of Political Institutions?

While the intentions of top-down accountability are unquestionable, the impacts of these types of systems are harmful to large groups of students.  Although top-down, government-controlled accountability systems within the education sector are beneficial to a few students, they are harmful to most.  The flaws inherent to democratic systems, in general, can explain this unintended consequence.  In particular, the problems result from unclear principal-agent relationships, uniform rules and measurement, concentrated benefits and dispersed costs, and the makeup of political participants.

The Control Problem

As Terry Moe (2002) points out, there are important problems that arise from the need to control actors within any given system.  For example, when there is a principal-agent relationship of any type, the principal needs to figure out how to control the agent.  Principals do so through measurement of agent performance and the creation of systems designed to incentivize agents for desirable performance.  In the current system of schooling, how could the school leader figure out how well the teachers are doing?  Additionally, how could superintendents figure out if the school leaders are doing a good job?  These relationships are numerous within any industry, but information is extremely limited in any institution that is controlled democratically, especially when students are coerced to attend a school based on their zip code.  Since principals are unable to measure agent performance through student selections, the principals are forced to create different, more complicated measures.

A proponent of public institutions will argue that locally-elected school boards hold schools accountable since rational voters participate in the political process.  However, this argument is seriously flawed.  First, less than ten percent of voters actually participate in local school board elections, so this could not possibly be representative of the entire society’s views.  Additionally, even if all people were legally forced to vote in these elections, they are not incentivized to be completely informed about the success of the candidates.  It takes a lot of time and energy to become a rational voter, and individual voters know that they are unlikely to determine the outcome of the election, so why would they become informed?  In fact, for most people, it would be irrational to take the steps necessary to become a rational voter.  To illustrate my point further, let us assume that everyone was forced to vote and to be a completely rational voter.  It is true that there would be more accountability of the elected officials; however, this gets to another problem that is inherent to democratic institutions.

Just as James Madison (2008) pointed out in Federalist no. 51, just as a monarchy leads to a tyranny of a king, democratic institutions can lead to a tyranny of the majority.  The democratic process if preferable, of course, since power is dispersed among more people.  However, the majority in a school board election, or any other election, is unlikely to represent the views of the entire society.  The goals that are pushed through democratic accountability represent the goals held by the majority of those participating in school board elections.  This is most likely the views of the most able, and most well informed voters within a society, so the least-advantaged citizens, and their children, are left behind as a result.

Public Choice Theory

Perhaps even more importantly, we must not forget that since people are inherently self-interested in their well-being, their votes may produce a result that is not socially desirable.  As James Buchanan and Robert Tollison (1984) indicated, public frameworks create the well-known problem of concentrated interests and dispersed costs.  In the context of the public education system, teachers have a huge incentive to mobilize and vote for benefits, while individual citizens bear small costs to cover teacher benefits since they are spread out among a large number of people.  In other words, when the costs of voting exceed the benefit of blocking the policy, ordinary citizens do not vote, and the socially undesirable policy passes.  As Rick Hess (2002) has discussed, out of rational self-interest, not greed, teachers have the incentive to pass policies that make their salaries and pensions higher while making their jobs easier.

Since democratic institutions are not accountable to the goals of society, the principals must come up with specific rules and measures to determine how to reward or punish their agents.  The problem with these measures, such as standardized test scores, is that teachers will have additional incentives to teach what is tested.  The basic problem here is one of opportunity costs.  If a teacher is focusing more efforts on standardized tests, they will have less time and energy to focus on building character skills such as determination and conscientiousness.  While this result may be okay for students that come from advantaged households, since they arguably receive more character building at home, this may be a disastrous policy for those students that need the most help.

The Knowledge Problem (And What to Do About It)

Of course, advantaged bureaucrats look at all of these policies from the lens created by their own past experiences, so they see no problem with focusing on rigorous academics.  It’s not that these officials are intentionally harming disadvantaged children.  Though these bureaucrats are doing what they think is right, they will not have the ability to determine what every kid needs to focus on.  It is a clear example of the basic knowledge problem: information and desires are constantly changing so much that it is impossible even for a benevolent central planner to create a favorable education system for every individual (Hayek, 1945).

Knowing that all of these problems exist in democratic institutions, why create a system of public schooling if one is not necessary?  Since schooling is excludable, it does not result in a free-rider problem, so government is not necessary for the operation of schools.  While democracy may be needed for services such as national defense, it is not needed for education, so we should not force children to bear all of the unintended consequences of democratic institutions.  We end up with a truly an ironic phenomenon: the very attempt to control the education of all children undermines the education of most children.  Instead, we should promote private school choice policies that allow children with diverse interests, abilities, needs, and goals pursue an education that is just as unique as they are.

 

Image Source:  http://news.images.itv.com/image/file/636404/stream_img.jpg

This Kind of School Choice Is Superior to Vouchers

Private school choice programs in the United States come in four basic forms: individual tax credits, tax credit scholarships, vouchers, and Education Savings Accounts (ESAs).  Voucher programs are the most well-known type of private school choice.  While voucher programs are desirable for individual students and the societies in which they reside, ESAs have a few important advantages that make them more effective, according to economic theory.

Voucher and ESA Definitions

Vouchers allow families to use a fraction of their public school funding for tuition at a private school of their choice.  ESAs allow families to allocate a portion of their public school funding amount to a government-authorized savings account, if they choose to opt out of their public school.  ESA funds can be used for various education-related expenditures such as private school tuition and fees, online learning, tutoring, and even college costs.

Five Economic Advantages of ESAs

While the flexibility of ESAs appears beneficial relative to vouchers, it is important to examine the economic implications of the policy.  Here are five reasons why ESAs are expected to outperform vouchers:

  1. Specialization:  Education and schooling are not the same thing.  While vouchers allow for private school choice, ESAs allow for educational choice.  Since all children are unique, the enhanced customization granted through ESAs should lead to a better match between students and their educational needs.
  2. Price Differentiation Incentivizes School Improvement:  When tuitions are fixed, as in a voucher program, the price of all schooling gravitates towards that amount.  Since schools will move towards the set price, quality will deteriorate.  Why take the risk of producing an innovative educational product if you know that you will not be compensated?  ESAs allow families to reward high quality private schools.
  3. Price Differentiation is Needed for Efficiency:  Since families know that they can save their ESA funding for other educational expenses and even college costs, they have a huge incentive to economize.  Just like with any other product, families will seek the best school that they can find at the lowest price.  The result?  Private schools are incentivized to keep tuitions low and quality high.
  4. Price Differentiation Entices More Providers:  Suppose a provider realizes that they can serve a certain group of students at the same level of quality for a lower price. The provider will have a better opportunity to do so in an ESA setting since families can spend less than the full amount on school tuition if they desire.  Additionally, providers of all educational services, such as tutoring, transportation, college preparation, and online learning, will have the same incentive.  The result of increased market entry: prices drop while quality increases for all educational services.
  5. Positive Externalities of College Enrollment:  Since ESAs allow families to save excess funding for college costs, children will be more likely to receive higher education.  Consequently, society may benefit from additional citizens attaining higher levels of education overall.

Not all school choice programs are created equal.  Since ESAs allow for enhanced specialization and price differentiation, I expect that they will outperform voucher programs.  More importantly, since ESAs are more well-grounded in accepted economic theory, I expect they will allow children to have access to an enhanced educational experience and a better life.

Corey DeAngelis


Corey DeAngelis

Corey DeAngelis is a Distinguished Doctoral Fellow, University of Arkansas.

 

This article was originally published on FEE.org. Read the original article.

Top-Down Accountability is Essential for Public Schools

Accountability systems driven by the political process are often discussed as desirable for individual students and the rest of society.  Of course, members of society want to at least attempt to ensure that certain standards are being met within any education system.  After all, if a given citizen did not attempt to hold educational institutions accountable, they would surely feel at least partially responsible for the result.  However, top-down, politically driven accountability processes are only necessary for public institutions.

Scholars discussing accountability schemes often focus on solving problems within education systems through coming up with well-defined measures.  The idea to design accountability systems in this way seems fine, and perhaps may result from using common sense.  However, creating rules ensures that uniform measures will arise and remain for a set of drastically diverse children.

Nevertheless, what if rules and measures were not in place within the current public system of education?  If teachers and school leaders were not held accountable to a measure, one could argue, they could get away with not teaching children much of anything.  Even if they were benevolent, they would not have a strong incentive to do so.  If you or I were in a position to earn a salary whether or not we did well, we would behave similarly, so I do not at all blame public school employees for humiliating results.  Because of the concern about rational human incentives, top-down accountability is necessary, but only in the system of public schooling.  The system itself creates an endless number of perverse incentives.  But why?

It is all very simple.  Since we live in a society with a relatively large population, we must figure out how to coordinate the actions of many people.  This is known as the coordination problem in economics, and there are only two ways to solve it (Hayek, 1988).  The way the coordination problem is solved for almost all industries that fail the economic definition of a public good is through voluntary exchange (i.e. private markets).  One of the few exceptions is the schooling industry.  Schooling is rival in consumption, and it is excludable, so it is not a public good, but it is still coordinated through central planning (i.e. the political process that includes officials coming up with relatively easy-to-manage rules and measures).

Since schooling is treated as a public good, even though it is not, our education system ends up with all of the problems inherent to the political process.  For example, since planners need to solve the problem of excess supply and demand within “free” institutions, they come up with feasible solutions such as residentially-assigned “free” schools.  This is a rational thing to do given the constraints, but residential-assignment results in extensive unintended consequences, including a public school finance monopoly, which leads to a system with enormous costs and low quality (Evers & Walberg, 2002; Hanushek & Raymond, 2002).

If families have huge transaction costs associated with leaving their assigned public school, they will be nudged to stay there, even if the institution is not up to their standards.  Imagine if restaurants, or any other business, operated the same way.  If you were residentially assigned to a restaurant, and had to pay for the food whether or not you ate there, it would take a large quality differential to get you to select another option.

Evers and Walberg (2002) claim that the education system is failing and that public schools need to be held accountable through top-down reforms.  They conclude that these types of reforms are the “best hope for improving American public schools.”  However, they made the same mistake as most education scholars; they assume that the coordination problem in education must be solved as it has historically: through central planning.  Additionally, the planning system requires overly simplistic rules that fail for systems with customers that have diverse goals, abilities, and interests.

Instead, decision-makers should realize that since schooling fails both parts of the public good definition, it could be coordinated more effectively, through the system of voluntary exchange (i.e. private school choice programs).  As a result of an increase in educational choice programs, private and public schools alike would be held accountable to the diverse set of individuals that they serve.  The public or private schools that do not cater to the unique needs of individual families would be held accountable since an exit threat would exist.  The standards in the proposed system would be specialized based on the unique interests, desires, and abilities of all children within the society.  In the current system, standards are set by officials sitting in their office, at a desk, hundreds of miles away.

 

Image Source:  http://www.avatargeneration.com/wp-content/uploads/2013/02/teacher-upset.jpg

Is Education a Public Good, Even if Schooling is Not?

I recently made the argument that schooling is not a public good because of the fact that it is rival in consumption and excludable.  In other words, the free-rider problem does not exist, so schooling should be provided privately.  However, I recognize that schooling may have positive externalities, so the government might have an interest in partially funding the service.

The Distinction

Schooling and education are two different things, although closely related. Education is the transmission of information to an individual or group. We receive an education in schools, while traveling on vacation, going to museums and theatres, and while talking to a diverse set people. Schooling is just one process to educate students.  Since education is more broad than schooling, is education itself a public good?

Postive Externalities of Education

Remember, education has positive externalities.  When I buy an educational service from a school, the school and I benefit.  However, the rest of society benefits from being around an educated individual without having to pay.  Therefore, government might have an interest in funding education, since individuals may consume less than the “socially optimal” level of education.

So, one may argue that government has an interest in partially funding schooling, educational travel, field trips, and other educational services such as tutoring. That is starting to happen with Education Savings Accounts (ESAs).  Currently, 5 states have ESAs, and Arkansas just proposed a fully-universal ESA a couple of days ago. These funds are put into an account and can be used for all of the educational services that I mentioned (even therapy for children that need special services).

Is Education a Public Good?

The simple answer to the question is that “it depends” on the channel through which education is transmitted. If it is in a school, then it is not a public good. If it is through tutoring, travel, museums, theatres, or any other related institution, it is not a public good. If it is through radio waves, however, it passes both parts of the economic definition, and the free-rider problem exists.

But why produce something such as education through radio (as a public good), if it can be done privately? You don’t want to create a free-rider problem if you don’t have one!

 

 

Image Source:  http://dataworks-ed.com/wp-content/uploads/2015/10/Teacher-Confused-000010368169.jpg