In Literature, Juror Bias and Personality Can Dominate Discussion. How to Make a Juried Award Fair and Less-Biased?
Juried Awards are Prestigious, and Strange
I have been on 4 Awards Juries before. Two of them were through colleges I was attending, and one was a fairly prestigious literary award through a local writer's group. The fourth was an award you might recognize that has national clout (undeservedly, in my opinion...) and made drastic changes to its structure and form in the many years since I served on that jury. They were all short story awards, of some sort. Having seen the inner workings of a couple different awards juries, I have noticed three trends.
1) Unanimous juried awards might as well just ask the biggest, most-arrogant bastard on the jury for their choice, because what will happen is someone will pick their favorite, and the jury will have to either nominate that story, or none at all, because an individual ego is too great to imagine compromise.
2) Awards with a non-unanimous, majority vote are only slightly better, because the pool of jurors are very unlikely to have a majority of jurors selecting the same story. Much infighting occurs. At least everyone can gang up on that one guy (it's always a guy) who has planted his flag on his hill and won't budge. But, it also means that everyone is ready to isolate someone who may have a good point, so that's not so great.
Basically, there is no perfect system. Awards juries are handed a set of stories to read, and given vague instruction on who is to be the head juror, and whether we all have to agree or not. Without a good leader in the head juror, there is no hope for a desirable outcome for the award, and some really strange things result.
The best awards jury I served on was run, not by writers, but by librarians. They devised an excellent system to narrow down the focus of the jury, and truly select the best of the group, as everyone saw it.
First, Make a Spreadsheet
The first step is work for the head juror, but it will pay off in the end, as other jurors will be able to narrow their focus beyond "I like this one! It should win!"
This spreadsheet should have every entry on it, plus one column for a number ranking. For smaller awards, for instance, this librarians teen writing contest, 3 stars are likely enough to handle the ten or twenty entries in the contest. For larger awards, like the Pushcart or the O. Henry, likely up to ten stars would be necessary to get quality results.
That spreadsheet is handed off to every person on the jury, within which every story is given a number on the scale of choice for the size of the award.
After completing the spreadsheet, the jurors send the spreadsheet back to the head juror.
Let's Run the Numbers!
The Head Juror will cut and paste every spreadsheet into the master document, with the different jurors' names as column headers.
Now, go out to the edge of the different columns, and make a sum of the numbers, across the columns.
No single juror has said one word to each other, yet. This is important. No personalities have been involved in the debate. Each juror has their own reasons and aesthetics in the process, and each has created their own numerical byproduct of their personal preference.
From here, the head juror can tally up and decide if there should be discussion at all between the jurors. If the contest is small, and the winner obvious, rising up over the other entries, there is no need for discussion. For larger awards, with more entries, the top ten numerical entries can be pulled out of the database, and a new round of spreadsheets goes out, where every juror is required to rank the ten stories as 1 through 10. From this, another round of math leads to the winner.
The only reason to discuss at all, in this model, where egos get involved and dividing lines are drawn, is if there is a tie.
Different Awards Encourage Discussion at Different Points in the Process. When Should Jurors Discuss Their Reading?
When Should Discussion Occur Between Jurors?
Different awards are going to have different standards. To me, the question of when discussion should occur will depend on the size of the award. For very large awards, with numerous entries, the sooner discussion begins, the better, because it will be a challenge to read all of the material presented, and discussion can push jurors towards certain works and away from others. The smaller the award, the less valuable discussion becomes, in my opinion. For the librarians that hosted an award, with about 10-20 entries, the numerical weighting was just fine to produce an excellent outcome that consisted of all the opinions of all the jurors together. In other awards, I have seen discussion guide all the reading and all the selections, and it was something of a mess, where lots of the stories either fell through the cracks or jurors' opinions fell through the cracks as strong personalities took over.
Each juried award is suggested to create very clear guidelines on the process that jurors are to apply. The point where discussion begins is also the point where things are going to be thrown off the rails of good statistics, and pure, weighted opinions. There is a place for that in juried awards, definitely, as often the broadest appeal is not the best writing, but the point where discussion begins should be carefully considered, and it is strongly recommended to place that discussion as late in the process as is feasible.
Awards Juries Should Carefully Consider the Jury's Job Description Before Handing Over the Keys to a Prestigious Award
Most of the juries on which I have served - 3 of them - had no clear rules for the jurors other than pick a unanimous winner, or vote for a winner. There, juries often have limited institutional memory as the workload is high and volunteers must be found every year.
Laying out, in writing, very explicit and specific instructions for jurors is advisable to ensure that the operation of the award runs smoothly. Telling jurors "Here take these and pick a winner" is a sure way to have an award with limited consistency year-in, year-out, and some years the jury will get quite contentious with each other as everyone will have a clear set of ideas about what ought to be done and why. Laying out very clear, blow-by-blow instructions will set the limits on places where problems can flare up.
Also, it is difficult to track the activity of volunteer jurors. A spreadsheet system helps to ascertain whether jurors are actually trying to read everything they receive. I know at least one of the awards juries on which I served, one juror openly complained that he couldn't read possibly everything and picked his winner from one of his friends' books. In that case, being able to track the jurors in some way would help a head juror who may have the authority to decide that individual jurors just aren't doing their job, and a replacement may be necessary.
Some General Advice for Awards Juries
First, what happens in jury, stays in jury. Going public with tales of woe and domineering behavior may feel good, in the moment, but it will only do harm to the award, to the person who is chosen, and to the people of the jury who will not come out looking like grown-ups in the situation. No matter what happens, short of threats of violence, do not go public with the discussions that happen in the awards jury.
Second, make sure that there is always a way to remove a juror, but that it happens rarely. Removing a juror shouldn't happen casually, but there ought to be a clear set of rules and standards that will lay out how and why a juror should be removed, and whether a replacement juror is needed or not.
Third, if possible, remove all identifying data from the stories before they are passed along to jurors. One of the great problems happening in literature is the cultural bias towards the white and the male. Speaking as a white male, I am very concerned that when I am nominated for awards, as has happened, my race or gender are part of the equation, instead of the quality of the work itself. I am also concerned that some of the best authors I know are facing an uphill climb that they do not deserve simply because of gender and race bias. Removing all identifying markers from stories, if possible, before handing to the jurors will help ensure that history smiles upon the award, and our culturally-temporary biases will have no place in ascertaining the best of our little corner of the world, be it a high school literary magazine, a local library's annual festival, or the Pulitzer Prize. (In the short story awards I have judged, this was easily possible at the amateur level, but with work that appeared in magazines or anthologies, often the juries were sent the whole publication, which meant seeing all identifying markers and author biographies.)