A New Shot at Testing and Accountability

Well, this is my first blog post ever.  I am used to just mulling things over in my head, grumbling to my pals, or telling students what to read.  Today, though, my topic is a few overlooked gems I would like to assign as reading to aides to members of Congress regarding decisions they will soon have to make about our nation�s testing policies.

My expectation is that Congress will, at least some time in 2013, resume debating whether to keep mandated state testing in grades 3 through 8 under No Child Left Behind, and what kind of stakes to require states to attach.  So once again, staffers have to figure out how to design a policy that works for all students.  Collectively, the readings below point to some clear flaws with current policy, and suggest possible alternatives. 

One is from five years back, a 2007 Education Week Commentary entitled �No Child Gets Ahead,� by Anthony P. Carnevale.  Colleen Donovan, David Figlio, and Mark Rush of the National Bureau of Economic Research used data from the federal early Childhood Longitudinal Study to analyze low to middle income, high-achieving students� educational attainment. Specifically, there were "more than a million grade school students from families making less than $85,000 a year who start out in the top half of their class but fall off the college track on the way to high school." Part of the story was that these achievers were being harmed by NCLB�s focus on the lowest-performing students in the schools they attended.  They found that teaching to the test �dulls creativity and generally ignores the students who can meet the standards.�

As Carnevale writes: "With lower standards on offer, many high-performing students from working families rush down to meet them.  They give in to lower standards because their college and career expectations are fragile and they get less support at home and at school than students born into affluent families." The way forward, he says, "is to move beyond uniform standards altogether, toward individualized standards."  Hmm, how does that fit with the onslaught of Common Core assessments?

The second is the 2011 report of the National Research Council, Incentives and Test-Based Accountability in Education, which interrogates what the behavioral and social sciences (in particular, economics and experimental psychology) tell us about the use testing and incentives to improve performance.  Based on 10 years of empirical work, this group of psychologists, economists, and testing experts concluded that "the available evidence does not give strong support for the use of test-based incentives to improve education and provides only minimal guidance about which incentive designs may be effective"(p. 91) The report explains the trade-offs in different kinds of accountability systems, and reviews the various considerations about incentives: target, performance measures, consequences, and support (p. 33).   NCLB, obviously, has provided "many ways for schools to fail" (p. 49); wouldn't it be better to have test scores instead serve as a trigger for a deeper examination of instructional and organizational norms inside the schools? 

Last but not least is a new piece by Andrew McEachin and Morgan Polikoff in the October 2012 Educational Researcher, "We are the 5%: Which schools would be held accountable under a proposed revision of the ESEA?" The authors model the bill�s proposed accountability criteria, which seek to identify lowest performing, largest within-school achievement gaps, and lowest performing subgroups, to schools in California, attempting to answer questions about the stability of the various classifications, as well as whether they identify the schools they were designed to identify. Based on their findings, they have numerous important policy recommendations, including �considering alternatives to the proposed Lowest Subgroup Achieving Schools [LSAS] criteria, which, as written, target schools serving significant numbers of students with disabilities,� such as stratifying the LSAS by subgroups, such as Hispanic, special education, etc. (p. 250).  They also note the importance of administering accountability separately by school level (elementary, middle, and high) � say, 15% of each type if the policy goal is to hold 15% of all schools accountable per year.   McEachin and Polikoff highlight the importance of state policymakers using 3-year averages of combined proficiency level and growth measures to give the most optimal picture of persistently low-achieving and low-growing schools.  The authors recommend that Congress should commission similar analyses from all states to look at possible implications.  

Now some may argue that the Common Coreassessments will, in time, solve some of the problems with low-level state standardized tests driving instruction down.  But does that tell aides to members of Congress what kind of testing and accountability system to enact next year, 2013?  What are the likeliest measures to build state capacity for intervention while not harming instruction?  The 1994 Improving America's Schools Act, with its mandate of testing just once in three grade intervals between the early grades and high school was too loose for many in the civil rights community, who pushed for the sub-group tight enforcement model.  How do you tend to the lowest-performing students without dragging down Carnevale's "low-hanging fruit" of high-performing, middle-income students (many of whom he points out are likely to become teachers and public servants themselves)?

If the answer is obvious, it has eluded me.  One thing I do know is that there is no substitute for good congressional deliberation, and that just might involve bringing some of these researchers to testify, run more models, answer questions, and even be permitted to debate each other as well as interact with state officials who have to run these programs. Aides, happy mid-air reading after you go flying over the cliff. 

By: Elizabeth DeBray

THE RESEARCH TRIANGLE. SORT OF.

As a student of education policy, my foremost personal goal is to keep my teacher sensibility and credibility from permanently draining away. Reading research reports, with their inevitable earnest calls for still more research, authoritative policy claims and suggestions, one understands that the base of the research-policy-practice triangle is practice, the apex being the conjunction of research and policy. Practice is a kind of collection pan (or audition stage) for ideas generated by scholarly investigation and clever policy-making.

Teachers serve as both subjects and objects of reform, but�with the exception of classroom-based action research, a phrase that causes your average doctoral student�s lip to curl�seldom go out looking to gather data or propose policy on a wider scale. They wait for the next finding, prescription or mandate to come down the pike, then either wrap the new guidelines around their old practice or attempt to ignore scholarship and policy altogether.

Some see this as evidence of a regrettable autonomy still present�despite the best efforts of education publishers�in the act of teaching. Others (people who work in schools, mainly) see these habits as a defense mechanism. No matter how many studies are conducted, no matter how large the data sets and innovative the statistical modeling, no matter how muscular the policy lever�kids keep coming to school, and teachers have no choice but to keep their heads down and teach them, somehow.

There�s been a little dustup over another one of Jay Greene�s papers, just released by the Manhattan Institute: Building on the Basics: The Impact of High-Stakes Testing on Student Proficiency in Low- Stakes Subjects. Greene�s shabby scholarship was roundly criticized about six months ago, when he and colleague Catherine Shock counted course titles in university catalogues, and developed a math-to-multiculturalism ratio, proving that Ed schools and teachers didn�t give a rip about mathematics achievement. Greene took some heat for that, eventually re-characterizing the data as �an amusement.�

This time, Greene and co-authors Marcus Winters and Julie Trivitt investigate the question of whether narrowing curriculum to put greater emphasis on two tested subjects (math and reading) in Florida schools might have a negative impact on student achievement in other subjects. Their answer? No.

Eduwonkette and other bloggers have raised lots of questions about technical aspects of the study�s research design, and the fact that the report was embargoed, not subject to traditional peer review, before hitting cyberspace; these are the kinds of inquiries that won�t be satisfied until long after the titular core message (�building on the basics� is a good and justifiable thing) has become conventional wisdom. Sherman Dorn cranked out an engaging essay on �the reworking of intellectual credibility in the internet age (which) will involve judgments of status as well as intellectual merit.� And an �Anonymous Peer Review� poster on the Ed Week blog set up, point by point, technical issues of concern in the piece, including some that are obvious even to research lightweights. For example, the study uses two years of math and reading achievement data and one year of test scores in science. That�s right�one year. I�m no statistician�but isn�t it hard to do growth comparisons with only one set of numbers?

Some tidbits from the report:

We find that students attending schools designated as failing in the prior year made greater gains on the state�s science exam than they would have done if their school had not received the F sanction.

There are two important reasons that we might expect schools deemed to be failing to respond positively. Those that have received an F grade for the first time may be shamed into improving their performance. Those that have received at least one failing grade may decide to raise their performance because they fear attrition of their student body.

Though there is some disagreement about which aspect of the accountability policy was effective (the threat of vouchers or the shame of an F grade), each of these analyses found that the policy improved the math and reading proficiency of students in public schools designated as failing.

While the hard-core researchers duke it out, let me step aside here and think like a teacher. Greene and his colleagues, through the Manhattan Institute (�turning intellect into influence�) have released a study strongly suggesting several things, some of which will be appealing to Florida legislators:

� Don�t worry too much about schools cutting back on science or other academic subjects to meet math and literacy targets, because it doesn�t really matter, in the long run. Science scores are likely to go up, statistically, if math and reading scores go up�and that�s good enough for us.

� Schools can �decide� to raise their performance after being shamed and threatened.

� The policy of giving schools failing grades improves their reading and math proficiency. And now we have evidence that it improves all subjects, whether we spend time teaching them or not. Failure, therefore, is a great motivator.

� Sanctions work, and are much less expensive than investing in improved instruction, engaging curriculum or retaining effective teachers.

Florida is a state with nearly 11,000 National Board Certified Teachers. The National Council on Teacher Quality, commenting acerbically on the recent�generally positive�National Research Council report on National Board Certification, said this:

Teachers from advantaged schools and states with financial incentives were more likely to participate in the certification process. Board-certified teachers are more likely to remain in the field than other teachers, and are more likely to move to assignments in high-performing schools with lower rates of poverty.

When policy and research �decide� that perhaps a robust science program really isn�t a necessity in a failing school, where are the Board-certified teachers going to migrate? Where might an accomplished science teacher seek refuge from the trickle-down effects of policy and research, along the bottom of the triangle?

Good News for the National Board

The National Research Council released a report today--a meta-analysis of the research on National Board Certification and teacher effectiveness:

WASHINGTON -- Advanced certification through the National Board for Professional Teaching Standards (NBPTS) is an effective way to identify highly skilled teachers, says a new congressionally mandated report from the National Research Council. Students taught by NBPTS-certified teachers make greater gains on achievement tests than students taught by teachers who are not board-certified, says the report. However, it is unclear whether the certification process itself leads to higher quality teaching.

"Earning NBPTS certification is a useful 'signal' that a teacher is effective in the classroom," said Milton Hakel, Ohio Board of Regents' Eminent Scholar in Industrial and Organizational Psychology at Bowling Green State University, and chair of the committee that wrote the report.

"But we don't know whether the certification process itself makes teachers more effective -- as they become familiar with the standards and complete the assessment -- or if high-quality teachers are attracted to the certification process."

The report recommends further research to investigate that question, as well as to determine whether NBPTS certification is having broader effects on the educational system, beyond individual classrooms. Studies so far suggest that many school systems are not supporting or making the best use of their board-certified teachers.

Link to the press release and the full report.


Full disclosure: I am a National Board Certified Teacher and worked, for two years, as a Teacher in Residence at NBPTS. I am fully conversant with the range of research on National Board Certification--both positive and less than glowing--and have noticed a trend, lately, toward a shorthand, knee-jerk opinion that there is no "proof" that National Board Certification is a signal of anything except a willingness to put oneself through the assessment wringer for a year (which, in itself, represents a huge break from business as usual). It's gratifying to see a highly respected entity take a, well, scientifically based look and come out with some qualified positives.

And not just from a research/editorial/scholarly standpoint, either. Most National Board Certified Teachers (over 90%, in fact) will tell you that the process changes them and their teaching. They know, in their gut, that it's been good for their practice--especially being required to articulate what, precisely, their students have learned and how they know that learning is real. It's enormously frustrating for a teacher who's clear about the benefits to their own professional learning to be told the research on National Board Certification is murky or negative, negating their first-hand experience.

I'm sure they're celebrating at the National Board, but the real winners are National Board Certified Teachers.

"Rethinking Educational Accountability" recording

An audio recording of the "Rethinking Educational Accountability" panel at YearlyKos is now available online. This session featured a discussion with Nebraska Education Commissioner Doug Christensen, Maryland teacher Ken Bernstein, and me (Sherman Dorn). The recording is 47 minutes long. You can listen with the online player or download the mp3 (which I think is 44 MB).