WBC Scoresheets and Transparency

Those of you following Colin Harmon’s blog – and if you don’t you should, as it’s excellent – will have noticed that Colin has uploaded his WBC scoresheets for all to see online.

While we were travelling together earlier in the month Colin, Gwilym and I chatted about uploading our scoresheets. I said that I’d upload mine if he uploaded his – and I think Gwilym is up for it too.

I wanted to upload the ones from Tokyo, but I honestly can’t find them. If anyone reading has a copy then please let me know/post a link in the comments! What I did find were my sheets from Berne. They are available to download as a zip of 7 jpg images.

I should add some commentary to these particular sheets:

[download id="1"]

This was one of the greatest learning experiences of my career. I walked offstage a mixture of happy, proud and also despondent at the knowledge that I didn’t really understand espresso and I really didn’t feel in control of it. I felt that I hadn’t served the shots I wanted to, but didn’t know enough to fix the problems I had had.

My signature drink was (quite rightly) savaged. It might have been innovative in its presentation but it wasn’t delicious and too much was (and had to be) prepared ahead of time.

All of the low scores I got, I deserved – no question.

I should also add that the rules were quite different then, as you can see from the scoresheets.

So why upload the sheets? I think it is interesting, I think (hope anyway) that it makes competition more accessible. I also see no reason why the judges should not be open to the same public scrutiny as the baristas. I’ll go out on a limb here and say that it would probably improve the scoresheets (on average) of judges if they knew that they would be a matter of public record. I’ve only judged around 100 competitors but I’d certainly have no objection to them uploading what I wrote/how I scored.

I also invite anyone else who has access to their scoresheets to upload them online – doesn’t matter which competition, how far they got, or where they placed. If you do please post a link in the comments. I know I wish I could have seen more scoresheets before I competed the first time.

Once again – all respect to Colin for going first, and hopefully setting a precedent that others will follow.

20 Comments

  1. The download link doesn’t work for me, can you verify?

  2. Working on some computers but not others – will fix asap!

  3. This is fantastic. If first time competitors are able to better break down scoresheets and actually find out what judges are looking for, they’ll be better prepared for competition at grass roots level and won’t be so disillusioned when they get destroyed on technical.

  4. I think that in principle the point of needing to improve judging transparency is a great one, but saying that means all past scoresheets should be made public glosses over some important issues.

    1. The judges wrote these scoresheets under the general assumption that they would be shared with the barista when the judge was present. As such, judges may have approached the notes section as only one part of the overall evaluation they give a barista, i.e. putting in notes to jog their own memory as well as notes for the barista, etc.

    2. Judges wrote these scoresheets assuming that they would be consumed by people who had experienced the sensory aspects of a presentation, either other judges/the Head Judge, or the competitor and their training team. This of course can lead to radically different evaluations then people who simply watched the presentation, but in the context of the scoresheet, judges were not trying to justify their sensory reactions for an audience with no familiarity of how a particular coffee or drink component actually tasted, but instead were justifying based on a common familiarity with at least the outline of the sensory experience.

    I think #1 and #2 have historically combined to make judges feel more comfortable saying things on a score sheet that they wouldn’t say in a completely public forum, at least not in the same way.

    Judges probably should produce score sheets up to the standard of 100% public consumption. However I think it is necessary to be sensitive to the caveats of presenting past score sheets that were produced under a non-public understanding of the score sheet.

  5. I guess I should add that the points I’m trying to make are much more relevant at the regional/national judging level than the more heavily disciplined WBC judging level.

  6. I think everything that contributes to improving the barista profession should be done: in transparency, scoring, writing the rules clearly, making it as objective as it can be.

    Judges are to support the barista and experience everything the barista has already experienced…as baristas get more knowledgeable about coffee, so must judges.

    All these comments can help achieve those goals, not only in competition but also providing consumers with better opportunities to tasting great coffee.

  7. Bravo to you James (and Colin) for posting your scoresheets.

    I think it’s important that the WBC and the USBC post the finalists scoresheets for review – whether by the public or the other competitors.

    Within the ranks (both as a competitor and a judge) there’s always discussion about improving the competition and elevating the talent. Providing the opportunity for non-finalist baristas to review the winning scoresheets can provide an invaluable tool towards improving their own presentations.

    Alex touches on a good point about the scoresheets. Many of the judges out there are essentially incompetent. They either don’t know enough about coffee and cuisine to be competent or they simply are weak/poor judges who provide little to no feedback to the competitor – much of these points are evident in the judges scoresheets.

    A judge is expected to write notes to help the competitor understand the basis of the score. The judges I have the most confidence in are the ones who are competent and write clear and concise feedback on the scoresheets.

    While it is ideal that you, as a judge, will meet face-to-face with the competitor, this doesn’t always happen and the competitor then will rely solely on the scoresheet for reference.

    Incompetent judges should be outed and a publishing of these scoresheets is a potential avenue to that result. Because as a competitor, I want to know who are the good/strong judges and who are the charlatans.

  8. Hahaha, thank you for getting to the heart of the matter as usual Jay.

    I couldn’t agree more, many judges are charlatans, and that’s a problem. And it’s likely that the quality of a judge is pretty accurately represented in their scoresheets.

    The only point I’m making is that the idea of outing “charlatan judges” based on how the internet-masses construe their scoresheet is problematic because the scoresheet as I understand it is not designed as an instrument for judges to prove their competency to the public but as a semi-private way for judges to communicate with competitors.

    That doesn’t mean scoresheets shouldn’t be made public. It just means that scoresheets that were written with the belief they wouldn’t be generally public should be taken with a grain of salt.

  9. Hi all. My score sheets appearing soon. they have been almost public for a while, with me passing them on to competitors who requested them
    Sometimes when viewing performances barista’s will focus on areas that may get attention but few points, analyzing the routine with the score sheets is more useful. For me, making my scoresheets public is not about charlatans or poor judging but helping competitors focus on what it was that got the points…. Usually the taste of the coffee

  10. First time I have dived into these scores which is crazy really now I would ask for them, but nieve times.

    Super cool to see, lots of memories in these coffees.

    Great to see scores out there

  11. Alex – point taken.

    Gwilym – I think your post brings up something that’s pertinent to the issue, and that is: timeliness.

    In order to help future competitors understand the methods of being judged under the WBC by providing scoresheets for review, they need to be disseminated in a timely manner.

    While James providing his WBC 2006 Berne scoresheets is a fantastic resource, the WBC should be providing finalist scoresheets immediately following the WBC. Sadly, a set of old scoresheets is of limited value due to rules and scoresheet changes (as well as focus changes by the judges) that occurs over the years.

    A timely set of scoresheets gives the future competitor insight into what’s important to the judges, advances the competitive barista and demonstrates the competence of the judge.

    It’s a win all the way around.

  12. I agree and respect the fact that you are willing to be transparent. I think there are many opportunities for this transparency in other sports/competitions as well and I commend you and Colin for being first out the gate!

    “All of the low scores I got, I deserved – no question.” – Love how you take responsibility. This is probably why you were so successful the following year.

    I agree with Jay C completely,

    “A timely set of scoresheets gives the future competitor insight into what’s important to the judges, advances the competitive barista and demonstrates the competence of the judge.

    It’s a win all the way around.”

  13. This is radical cool. I have just two levels of competition under my belt and I have to say that up until all this revealing of scoresheets I figured I probably wouldn’t compete again because of (what I considered to be) incompetent judging by one judge I had during my USBC performance. Maybe it was my naivete, but I honestly didn’t expect that to be the case at the USBC. Such a strong field of barista competitors. I expected an equally strong field of judges top to bottom. But maybe I just got the one bad apple. :)

    Granted, I made my own mistakes. I own that fully. Had I taken care of business I might have even been able to overcome a one-off judge. But seriously, also, when one sensory judge’s scores seemed woefully different from the other three sensory judges–with what I considered to be useless notes given–I got pretty depressed.

    To make matters worse, this judge was NOWHERE to be found afterward for critique review. I was on the lookout for him. Not to meanfully confront him. But honestly so I could hear his rationale. Perhaps he had perfectly sound rationale. I was ready to hear it. But nothing. So I was left with poor scores, poor notes and poor judgemanship.

    Worse still, I’d love to upload my scoresheets so that some other new competition barista won’t make the same mistakes this new comp barista made. But alas, I have never received my scoresheets for my regional or USBC performances. I guess they are there for the asking.

    Not bitter. Really. Just a bit disillusioned about those parts of competition.

    Nameless judge (whose name I don’t have without my scoresheets!), please show yourself so we can have a little chat about transparency.

  14. Thanks for the scoresheets !!!… It’s a great resource for any barista interested in the competition. I wish that for total transparency, every competition made all the scoresheets public, and not only the ones of the finalists. I think it would be very constructive to talk with the other competitors scoresheets in hand, or to look at videos with them, etc.

Submit a comment