The Viability of Standarized Testing

I may not get to a full posting before Monday, but I will throw some ideas out. First, my experience is with entrance tests - SAT, ACT, GMAT, GRE, LSAT - and I will confine my comments to them alone. Second, I will modify my earlier exuberant language to say that standardized entrance tests are relatively better than other forms of assessment, and, I think they are pretty accurate as indicators of raw ability. Third, most of these entrance tests (particularly the graduate level ones) are only secondarily about content; the higher difficulty questions are really testing the student’s ability to reason deductively and apply general principles to unexpected situations. Most of my teaching consists of critical reasoning skills. I know that sounds strange, so I will try to post some examples next week.

There are certainly going to be flaws in any standardized model, but I do support the basic idea of standardized testing. We live in a society of strangers today. Because of industrialization, geographic mobility, and the resulting atomization of traditional forms of community, we spend most of our lives interacting with strangers at work, school, and in our neighborhoods.

This is different (even if only in degree) from the past. Imagine with me a Puritan community in Massachusetts during the mid to late-17th century. The township is small enough so that pretty much everybody in town knows everyone else. A majority of the inhabitants have lived in the town for many years and a signficant minority have been born and raised there (at least, relative to today). Almost every villager is a member of the local, established parish. In this town you would personally know the abilities or skills of each potential laborer. So, for example, if you were a shopkeeper you’d know which kids were adept with numbers and would make good assistants. You would not need a formal, standardized testing procedure in this community.

But today we no longer live, work, or go to school among neighbors (in the old sense). Because we interact with the unknown we need some way of measuring the reliability and skills of people with whom we have no acquaintance. If we do not develop some standard of comparison a community of strangers cannot, or at least cannot efficiently, comingle and interact. Communities are formed by bonds of trust. How do you trust a stranger?

For business transactions we’ve seen the evolution of contract law. In an intimate society, your word can be your bond. The community acts as an arbitrator and can enforce the verbal contract by shunning the transgressive party. In a society of strangers we need written contracts. That is the cost of doing business today. Without strict legal guidelines and our judicial system, our modern society would be impossible. The development of standardized testing in education is a similar response to our modern society of strangers.

If you are a college seeking the best and the brightest students, how do you determine whether a student is da bomb or a bust? They have to have some way of comparing students to each other. This criterion must be standardized, applicable to all students; you cannot make a comparison without a consistent measuring stick. This is the basic flaw behind using GPA as your main measure of student performance. Every school grades differently. Some schools are guilty of rampant grade inflation. A GPA of 4.0 at one school might be equivalent to a 3.0 at another. With thousands of different grading standards, colleges and graduate schools have no hope of using GPA as a consistent measure.

Thus the introduction of the SAT, ACT, LSAT, GRE, etc. They attempt to, however imperfectly, fulfill a legitimate need.

[Susan R] Paul- I only have a minute, but I wanted to reply quickly to one of your points. I’ve read a few studies (at least one of which I linked to in the OP) that indicate the best measure of future grades is the student’s previous grades, and that the SATs measure a student’s ability to retrieve information, not their ability to think deeply. Many colleges recognize this and are moving away from standardized tests- how do you account for this?

I think I can add something here. The standardized tests are definitely in favor of the quick thinkers. Standardized tests indicate future testing ability and raw cleverness more than diligence, perseverance, etc. So, these tests only give part of the picture, but they do contribute an important part. Standardized tests are primarily aimed at the critical thinking skills of students. They are not designed to test how much content the student has accumulated (except for the subject specific tests), but how well the student is able to apply familiar ideas to unfamiliar situations. That’s why the math caps at about Algebra II rather than going through Calculus, which is usually available to high schoolers. Here’s an example of a quantitative question from a GMAT (business school):

Team X won 40 basketball games. What percent of its basketball games did Team X win?

1) Team X played the same number of games as Team Y.

2) Team Y won 45 games, representing 62.5 percent of the basketball games it played

Answer Choices:

A) Statement (1) alone is sufficient, but statement (2) alone is not sufficient to answer the question asked

B) Statement (2) alone is sufficient, but statement (1) alone is not sufficient to answer the question asked

C) BOTH statements (1) and (2) TOGETHER are sufficient to answer the question asked, but NEITHER statement ALONE is sufficient

D) EACH statement ALONE is sufficient to answer the question asked

E) Statements (1) and (2) TOGETHER are NOT sufficient to answer the question asked, and additional data specific to the question are needed

The genius of this question type - data sufficiency - is that it pretends to be a math question while being nothing but pure logic. The student, after reading the question stem, must determine what information he/she needs to answer it, then evaluate the statements to see if they provide the information. Students who do unnecessary math, such as actually finding out the total number of games played by the two teams, are punished by the strict time limitations of the GMAT.

(For anyone who was wondering, the answer is C)

[Susan R] Paul- I only have a minute, but I wanted to reply quickly to one of your points. I’ve read a few studies (at least one of which I linked to in the OP) that indicate the best measure of future grades is the student’s previous grades, and that the SATs measure a student’s ability to retrieve information, not their ability to think deeply. Many colleges recognize this and are moving away from standardized tests- how do you account for this?

http://www.psychologytoday.com/articles/200401/testing-rip-the-sat

The person that the article quotes to say that high school grades are a better indicator of college success than standardized test scores, Robert Schaeffer, is the PR guy for an anti-standardized testing advocacy group backed by the NEA. So I question his authority; they have a giant axe to grind labeled “Ulterior Motives.”

Also, if you read the subtext of the article the reason why these schools are dropping the SAT has more to do with the non-politically correct results the test gives.

[Article] But its critics have gained momentum in recent years. They say that the SAT is a poor predictor of women’s, non-native English speakers and older students’ academic performance in college. Whites outscore African Americans on average by 206 points.

But great joy breaks through when…

[Article] “These schools have reported greater diversity among accepted students and higher educational quality since they changed their policies,” says Schaeffer. He also says that school grades are a better predictor of success in college than the SAT, despite the variability in high school curricula.

The better authority in the article is Robert Sternberg from Yale who notes that the SAT is limited, but useful. I concur.

On a tangential note, I find it ironic that as a homeschooler you would bite the standardized plan that feeds you. If it weren’t for standardized testing, homeschoolers would have significantly harder time cracking into higher education. Do you really believe that, in the absence of stadardized testing, a selective university would choose a homeschooled kid just because their parents gave them all A’s?

[Susan R] Everyone has ulterior motives. I don’t discount information or insight simply because it supports someone’s agenda. Teachers and school officials are NOT objective, and are just as capable of cheating- Google “teachers and cheating”, ifn’ you don’t believe me. IMO teachers have even more motivation to skew grades than parents. I can’t imagine a parent taking on the task of homeschooling, and then crippling their own child by fudging grades, but teachers with a classroom of other people’s children are under enormous pressure to teach to the test so that schools can continue to receive federal funds and get a higher NAEP score. How many times have you heard stories about some star basketball or football player being allowed to pursue his sports career even though his grades were abysmal? So I’d prefer to deal with the veracity of the information itself, rather than eliminate it based on whether or not I think they have an ax to grind or a career to protect.

What you are arguing would matter if Robert Schaeffer was actually citing data. But all that Schaeffer gives as evidence is his own opinion:

[Schaeffer] “These schools have reported greater diversity among accepted students and higher educational quality since they changed their policies,” says Schaeffer. He also says that school grades are a better predictor of success in college than the SAT, despite the variability in high school curricula.

Schaeffer’s opinion isn’t worth a hill of beans as evidence unless he is an unbiased expert. This is why expert witnesses in court are required to have some proof of expertise and must not have ties to the party for which they are testifying. Thus I believe that it is perfectly legitimate to note his position as a professional spinner for an anti-testing advocacy group. All we are given is his opinion, and as I’ve noted, that opinion is suspect because it is biased.

[Susan R] When we discuss standardized testing, we need to realize that it isn’t just the SATs we’re talking about- I mentioned in the OP that testing is mandatory in schools, and in some states for homeschoolers as well. But since I don’t follow the same scope and sequence as the local schools, and I’ve opted out of the system, why should I be required to comply?

This is misdirection. The question posed by this forum is the viability of standardized testing in general, not specifically whether or not homeschoolers should be required to take such tests. These two questions are distinct though both are indeed worthy of discussion.

[Susan R] In The Washington Post article Colleges Coveting Home-Schooled Students, I don’t see any grand importance given to SAT scores for homeschoolers.

I do. Look at the second page:

[Washington Post] The school’s admissions standards for home-schooled students are identical to those for traditional graduates _ minus the formal transcript requirement. Some colleges and universities, though, continue to require home-schoolers to earn a GED high-school equivalency diploma or take subject-specific SAT tests along with the standard requirements.

Indeed, they are more likely to require that homeschoolers take extra standardized tests above and beyond what non-homeschoolers have to take. This is logical. The less independent verification the college can get, the more of an emphasis they are likely to place on standardized testing.

[Susan R] If he was a product, like dishsoap or a handsaw, then standardization makes sense. But he is an individual person, and the idea of squeezing him out like toothpaste through a tube flies against everything we know about human beings as innovative and unique.

You have a gift for making things sound as unappealing as possible.(-; But what a standardized test tests for is not the person but what they know and how they think. Knowledge can be standardized. Either you know when Columbus sailed the ocean blue or you don’t. Beyond just measuring what you know, a well designed test can determine your ability to think in certain ways. Logical thought, analytical ability, and comprehension are measurable.

Still, you are right to note that standardized testing is imperfect, that there are exceptions, and that success at test-taking is no guarantee of future success. That being said, the flaws of standardized testing are not grounds for denying any and all merit.

[Susan R] BTW, where did you get the idea that Fairtest (Robert Schaeffer) was backed by the NEA? http://www.fairtest.org/about] Their site states that “FairTest is funded by grants from the Bay and Paul Foundation, Ford Foundation, Polk Brothers Foundation, Schott Foundation, United Church of Christ, Wiener Educational Foundation, Woods Fund of Chicago, and many individual donors.” Is one of those a front for the NEA or something?

They cohosted a conference back in May and they seem joined at the hip in anti-testing articles that I’ve seen. http://www.fairtest.org/fairtest-nea-state-assessment-reform-conference

[Susan R] Our current education system increasingly places emphasis on standardized tests- the stakes are incredibly high, with millions of dollars attached to student performance. The fact that homeschoolers are also being held to this standard is bizarre to me. IMO, the underlying assumption is that parents can’t provide a fair assessment of their child’s progress, but professional educators, who are always objective, can- and that is (here comes a really technical term) BALONEY.

You’re talking about two different things here. The whole point of a standardized test is to assess student performance more accurately than professional educators can. This is precisely why the NEA hates standardized testing…it reveals how bad a job many professional educators have done educating America’s children.

[Susan R] It is now almost unheard of for colleges to require that homeschoolers take the GED or submit to extra qualifying tests- something that was addressed by the The Higher Education Act Amendments of 1998 with regards to qualifying for student loans. The Army had no trouble with the transcripts I submitted to my son’s recruiting officer, and he did not have to take the GED to get in the Army or to take online college courses.

And I propose that the rise of the ACT and SAT is precisely why colleges no longer require a GED. Simply put, standardized testing is a superior criterion to high school equivalency GEDs for colleges which are looking to judge the abilities of strangers. A GED is limited because not all GEDs are created equal. The advantage that a test like the ACT or the SAT has is precisely that every ACT is equal. Since colleges have standardized tests to measure ability they no longer place as much value on GEDs and such. This is why I wrote the whole “bite the hand” comment. Standardized testing has made access easier for homeschoolers by and large, not harder.

But all that said, as you noted, “Yipee,” we agree on the basic issue. :-)

PS - Do you have a cheat sheet for all those fancy little emoticons I can borrow?

I think some broader factors should be considered here.

First, speaking objectively and not relative to some pragmatic goal (like getting into college), I’ve never read any respected educator who thinks well of objective tests

Take Jacques Barzun for example, a great model of a scholar, humanist (in the best sense), educator, and writer. Barzun wrote extensively on teaching and education, having served at Columbia for most of his life, including as Dean of the Graduate School. In numerous places, including his book Begin Here: The Forgotten Conditions of Teaching and Learning, he comes out explicitly against objective, multiple choice tests. He thinks they are worse than useless. Barzun, for people who know who he is, should mean a lot, but he’s hardly alone. The point here is that multiple choice, objective tests are intrinisically unhelpful in the process of education. Their only value, and this is key, is with respect to people other than the student, and with respect to goal’s other than education. Objective tests go against good education, as people like Barzun well know, but they fit with the standards of bureaucratized systems because they provide objective, shared assessments.

So, objective tests, of which standardized tests are instances, are not helpful for educating people nor are they evidence that someone is educated.

Second, the way we assess is provincial; many do not do it this way. When German exchange students, or students from many other countries (e.g. I know a guy from Latvia educated in the Russian system) come to my graduate school, the freak out because some classes have tests, which they simply do not use in their home system (the Latvian wrote essays even for mathematics exams; go figure - he’s good at math, by the way). So, the whole idea of exams and tests used the way we do is not only recent but it’s also far from universal. Indeed, anyone who knows anything about an Oxbridge education knows that, even today, practically everything rests on how well you do on your exams that you take at the end of you degree; those scores determine whether you get a First, Second, or Third Class degree.

Third, even given their value for bureaucrats, standarized, objective tests are still of highly dubious value for the people, ultimately, for whom they are most relevant: the teachers/employers of the people who take the test. I have yet to meet a respected academic or scholar in my field who thinks well of GRE scores; in fact, one of the leading scholars in the history of philosophy told me in correspondence that he found the GRE’s worthless for predicting student success. Again, this is not uncommon.

Fourth, if you think of all the really important means of asessments, like BAR and CPA exams, exams for people graduating for a degree from Oxford or Cambridge, the battery of tests use for comprehensive exams in academics, or the tests administered to people in the military/government jobs who are seeking to move up in rank, none of them center on (some don’t use any) objective, multiple-chooise tests because what such tests assess is normally utterly worthless for gaining a sense of people’s fitness at that level.

Fifth, a key question to ask about any test is: who is assessing the results? If only experts in the respective fields tested on are qualified to review and assess the test data, then you are likely to have a test that actually tells you something worthwhile; if any idiot can assess the test results, as is necessarily the case in objective, standarized tests, then you have a very good idea of how limited the data from that test is. This is why people like Barzun, by far among the historical majority, place the emphasis on essays. Objective tests cannot tell you any of the important things about a students’ knowledge; just knowing when something happened is, truly, worthless if you don’t know how the student thinks of that data. I am positive I know a great deal more about certain events than some people who may remember the dates of them better than I. My knowledge cannot be assessed on a standarized test; only an essay would adequately reflect it; their knowledge could be objectively assessed, and those are the types of students’ favored by objective exams.

Finally, I don’t think the solution is prior grade records. Educators know those are not very helpful because there are not uniform standards of grading, and grade-inflation is a problem that runs from the lowliest schools all the way into the Ivy League. As many graduate schools will point out on their websites, a 4.0 from PoeDunk U is worth far less than a 3.2 from the University of Chicago or a 3.5 from Princeton. That’s why admissions commitees so value the only two generally reliable indicators of student preparation: writing samples and (to a lesser extent if the person is unknown) recommendation letters.

So, I think objective tests have little to any intrinsic value; indeed, Barzun and others would say they are positively harmful as we use them; and at the scale at which we use them, I don’t know what sensible person could disagree - I know many teachers who are frustrated by the idiocy of having to “teach to the test,” instead of teaching their subject as it should be taught they teach it according to some standarized test requirements.

Although the nature and standards of helpful assessment vary from field to field, the principles do not change much; look at great schools with proven and deserved reputations and you will see patterns, patterns that don’t involve heave weighting (if they are used at all) of standarized tests. Oxford is a good example if you have never looked at the admissions process (it involves, among other things, formal examinations in you field as well as an interview that can essentially be an oral examination by an expert in the area to which you are applying).