We all need feedback to gauge how well we’re doing in our various life pursuits. FamilySearch indexers spoke loud and long for a tool that would help them gauge their indexing accuracy. So, FamilySearch created “My Accuracy” (now called “Arbitration Results”). The launch of Arbitration Results created both intended and unintended consequences. Indexers began receiving much-needed feedback on their work, and many expressed gratitude for the opportunity to learn how to improve. But some reported dismay at learning, for the first time, that arbitrators were changing what appeared to be accurate indexing work. The magnitude of the problem was also a revelation to FamilySearch.
Some of the first issues showed up in non-English batches that had been indexed by native or knowledgeable speakers of the batch language but were subsequently arbitrated by individuals with less knowledge of the batch language. More recently, incidents have been reported of arbitrators failing to follow project instructions, essentially undoing good indexing work.
Were it not for Arbitration Results, none of this would be known today. And though “ignorance was bliss” for everyone—both FamilySearch and its volunteers—we can use our newfound wisdom to do better work going forward. We’ll discuss how to do that in a moment, but first, a word to indexers about Arbitration Results.
What Is Arbitration Results, Really?
Arbitration Results is a measure of how often a volunteer’s indexing agrees with another volunteer’s arbitration. It’s not a measure of absolute accuracy against some truth handed down from on high. It’s a reflection of one volunteer’s best judgment compared against someone else’s best judgment. Once in a while, but hopefully not often, the indexer may be more accurate than the arbitrator. Shocking, isn’t it?
So what should you take from this? First of all, don’t put too much stock in your Arbitration Results score. It’s a guide, not a grade. Nobody’s going to be impressed if you put it on your resume. And you’re not likely to get an award for being in the “99 Percent Club” at your next family reunion. Use your score for yourself, in order to get better and to study where you might be making indexing mistakes.
Further, don’t let a bad score on a batch ruin your day, or worse, make you want to quit indexing. Sure, it would be nice to say you’re a 100 percent accurate indexer, but again, 100 percent compared to what? In some rare situations your 90 percent might be more accurate than someone else’s 100 percent—it all depends on who’s doing the arbitration, the handwriting on the records, and the project difficulty level. One volunteer in a recent focus group said she had a 92 percent accuracy average, and she felt like a failure. The point is, it’s not a good idea to base your self-worth or even your competence as an indexer, on your arbitration results.
Now, since we’re trying to be transparent, let’s touch on another subject where the response may not be quite so comforting.
When you, as an indexer, review your arbitration results and find an obvious mistake in arbitration, your only recourse is to click Review Batch, which sends a generic message to what, from your perspective, is a faceless, nameless person at FamilySearch. From there, nothing more happens on your end. You receive no response, no resolution, and no further assurance that a clear mistake in arbitration is ever going to be corrected.
Does this sound like a problem to you? It does to us at FamilySearch, and we are determined to find a satisfactory solution. As you’ll read below and in other articles in this series, the underlying problem of poor arbitration is being dealt with on many fronts, which should alleviate much of the concern. What’s missing is the ability for an indexer to state his or her case or have any real say in correcting mistakes in arbitration.
For now, let’s just say FamilySearch is aware of and acknowledges this issue. We wish it were different and that we could offer a more immediate solution to indexers who care enough to draw our attention to the current need. But change takes time, and though we’ve been working on possible solutions for months, a permanent change won’t occur until a new version of the FamilySearch indexing system is launched. Meanwhile, we will continue to focus on the things that can be changed in the near term, even if they are only incremental steps to improvement.
Qualified to Arbitrate?
Despite what some would have you believe, bad arbitration is not the rule; in fact, it’s far from it. Tens of millions of records are published on FamilySearch.org every month, and the vast majority of those are indexed and arbitrated accurately.
That doesn’t ease the pain when it’s your indexed records that get incorrectly arbitrated, but think about it: how many of your batches or fields have been “victims”? One percent? Two percent? If it’s higher than that, don’t assume that’s the norm, because it’s not—despite what the vocal minority would have you believe.
Most indexers receive an invitation to consider arbitrating when they complete their 100th indexing batch. Others are recommended by their group administrator or stake indexing director. Some think the bar to become an arbitrator is set too low, and perhaps they’re right. Some have suggested that 10,000 records would be a good level, and others have said they wouldn’t feel qualified to arbitrate with even that amount of experience.
The fact is there is no single right level of qualification for an arbitrator. Everyone is different. Some may never feel (or be) ready to arbitrate while others, especially those with many years of genealogical experience, might be ready much sooner.
If the level to qualify to arbitrate were set arbitrarily high, far fewer records would be published on FamilySearch.org. If it were set arbitrarily low, too many who were not prepared to arbitrate might attempt it before they should. Finding the right balance is challenging, to say the least. When an indexer reaches the 100-batch level, it suggests that the indexer is committed and serious about indexing, has had a fair amount of practice reading handwritten records, and has likely achieved a high level of quality, at least by the measures at our disposal. This rule of thumb may not be perfect, but so far these guidelines have proven to be reliable for producing a lot of near-perfect work.
So are we saying, “Get used to the way things are because they’re not going to change”? Not hardly. We’re smart enough to know that, as Albert Einstein is credited as having said, “The definition of insanity is doing the same thing over and over and expecting different results.”
So What’s Changing?
If the expected resolution to the arbitration conundrum hasn’t come as quickly as the discovery of the issue, it’s because experience has shown decisions made in haste often result in problems that are worse. A measured approach is what FamilySearch will always opt for. That measured approach includes the following current initiatives:
- Greater emphasis on providing training resources for arbitrators.
- Greater emphasis on training group administrators and stake indexing directors to identify qualified arbitrators and to train and counsel them more actively and consistently.
- More reminders to arbitrators to stay with projects they are familiar with and to understand that accuracy, not speed, is job number one.
- Closer monitoring of indexer feedback and occasionally revoking arbitration rights in rare cases where we see a pattern of carelessness.
In the end, personal care and concern for quality will make the biggest difference. Improvement will come incrementally as we all commit ourselves to a consistent pattern of excellence in our indexing and arbitration work. FamilySearch is committed to doing its part, and we welcome your feedback and suggestions. Please send your thoughts and ideas to firstname.lastname@example.org.
*This is the fifth post in a series of articles about arbitration.
- Now Is Not the Time to Get Cold Feet about Arbitration: Over 3 Million Images Are Waiting to Be Published
- Learning to Like the Referee: Why Arbitrators are Necessary and Deserve Some Respect
- “Pay No Attention to That Man Behind the Curtain”: Revealing the Truth about Arbitrators
- Ten Commandments for Arbitrators: We’re Doing Great, But We Can Always Do Better
- Arbitration Results Ruining Your Day?: What They Mean and How You Can Help
- What “Final” Really Means: Is Arbitration Really Data’s Last Chance?
- Frequently Asked Questions: Answers to the Most Common Questions Asked about Arbitration