Barista Competition Judging

I know barista competitions come in for a lot of criticism online, and I don’t really want this to be one of those posts. However having seen a bit of both sides there are some things I’d love to see changed.

I don’t really have an issue with the rules of the competition as they are now, and I am not out to suggest a better competition or anything. I am really looking at this from the angle of former competitor.

A few suggestions:

Get rid of the words explaining the numbers

At the bottom of the score sheet there is a key to the numbers:
0 – Unacceptable
1 – Acceptable
2 – Average
3 – Good
4 – Very Good
5 – Excellent
6 – Extraordinary

I think this creates a big misunderstanding around the lower scores. To get a 2, as a barista, is pretty disappointing. It feels like criticism, but a judge could be meaning it was ok. Not bad, not great but average. Equally I think it makes scoring higher difficult as a judge going by the words – What exactly is the difference between Very Good and Excellent.

In the UK we’ve tried to use the mantra of “what more do I want?” as judges because this pretty much mirrors the question you ask as a barista when you pull a great shot of great coffee and get a 4. We’ve tried to ignore the words and just use the numbers. This leads me to my next point:

Explanations are mandatory for scores below 3

If we are to use barista competitions to promote the craft and help baristas improve then your scoresheet at the end of this needs to be very useful. On the day only one person gets the trophy. To some extent all the rest get back beyond the experience are the sheets themselves. Getting a blank sheet full of low scores back is not only disheartening but frustrating. I think judges need to be held accountable for their scores and there is plenty of room on the sheet for notes.

An online barista competition wiki

I think as barista competitions mature around the world there still needs to be some resource available for a variety of things. For visual things it would be great to see if we all rate drinks similarly – be it crema colour, latte art or exactly how a great traditional looks. Useful for both judges and baristas.

Then it would be good if there were somewhere to ask questions about things not really covered in the rules. There are several things that the rules state as being illegal with no indication of the correct way to punish them. There will always be new and surprising things happening and it would be great if the JCC were available for questions on rules (like using a side table to pour capps at the judges table) as well as other judges weighing in with their opinion. I think it is easy to feel that unless you have some level of inside knowledge then you can never achieve the highest points. A little transparency would go a long way.

I am not ought to criticise or rewrite the rules – I think if you chose to compete you agree to play by the rules and if you don’t like them no one is forcing you to play.
However I’d love to see the competition keep evolving in a way that has a positive impact on the baristas entering and on the coffee they serve.

I’d love to hear opinions of both competitors and judges on this as long as we don’t drift into what people think is wrong with the competition format as that is a separate discussion altogether.

24 Comments

  1. I certainly agree with the number descriptions. As a judge i’d guess it’s very hard to rate something as ‘acceptable’ yet only score it a 1/6, I pull ‘acceptable’ shots at work sometimes but I wouldn’t give them a 1/6. If everyone is judged along the same parameters then I guess it’s a fair enough system but everyone (including judges) have different opinions on what is acceptable/excellent so I would like to see descriptions removed.

    The wiki is also an excellent idea, James you know first hand that I have lots of questions about competition and there are few resources other than previous competitors or judges themselves.

    I think stricter rules on sig drinks as well. Personally I think the sig drink should be judged purely on whatever is in the cup, extras such as nibbles, scents, dancing cats should be part of the overall impression for the head judge.

    Other than that I think the overall format is very good, knowing and having your scoresheet is very useful, i’ve had some bad experiences going into cocktail comepetions blind!

    PS Your preview post plugin isn’t working properly in safari, it’s all scrunched up.

  2. Let me tell you what I think it wrong…

    But seriously, your point about certain “no” things in the competition not having an explicit penalty should be addressed. What I would also like to see is more continuity in the judging from competition to competition (at least within each individual nation).

    For example, if you compete in a US Regional and a rule is interpreted at that level in a particular way, then it is reasonable for the competitors to expect that this has set precedence for the remaining competitions.

    Let’s say a barista (under the old rules) used a particular shape of cappuccino cup that was deemed “acceptable” at a regional – this same cup cannot (or perhaps should not) be deemed “unacceptable” at another regional or national event that same competition year.

  3. I don’t have a problem with the the score descriptions. If there were no descriptions then it would be very much up to the judges own opinions and views on a certain drink/category scores, whilst it is important that judges use their own views it is also important that all the judges score equally so there need to be guides for the judges to follow, surly the descriptions help here?

    From my point of view I want to know how the judges will score my drinks so possibly even more detail in to the descriptions would be helpful. Perhaps there needs to be more transparency in to how the scores are given. Notes for both high and low scores would be very useful to improve but having a better understanding before hand would be even better. So the competition wiki would be a very useful tool.

    The signature drink is a great opportunity to show off abit so I think the scores here are well balanced here, you still get the most points for the taste of the drink anyway.

  4. I remember when the descriptions – or guide to scoring, or scale as it’s usually referred to was first introduced. For the first couple of years the WBC scoresheets existed with a 1 – 6 ranking, with no explanation for what was what. So judges from different countries and different ideologies gave vastly different scores. One judge gave me a 4 another gave me a 1 for the same beverage. Even with explanation this is totally confusing for any barista who is trying to improve their craft. So the scale was put into the scoresheets to try and at least put some perspective as to what each number might represent. Better words might be chosen to explain each word, but I think the scale is fundamental given the scope and breadth the competition now entails.

    Truth be told, many first time competitors are out in the cold, and getting a poor score especially without explanation is devastating. It either enspires (sometimes) or provokes an agressive negative feeling towards competition. We’ve had both here in Australia. So this is where the responsibility of judging comes in. It’s an incredibly hard job, and you have to be honest and prepared to stand by what you’ve said, and most of all sit with a barista and tell them why you’ve made a particular decision about something they’re very passionate about. At least by giving the numbers some meaning there is a platform everyone can start from.

  5. Sorry I also forgot to say – I think baristas would benefit greatly from participating in the judges training sessions, and judges from going through the ropes of a competition.

    Judges (at WBC level at least & I know here in Aus to) are expected to make legible and constructive comments on every competitor. Blank sheets are not accepted – in fact they are rejected by the head judge as the notes are so important in a debrief, especially when you’re assessing a lot of baristas. Judges who fail to follow basic respect for competitors, simply do not judge again!

  6. hi James,
    I would like it to be mandatory that a competitor supply a sample(extra) espresso cup and capp cup and for these to have their volumes measured by tekkies as a matter of course after each performance. I’d also like to see a series of photos of 3 espresso at level 1, 2, 3 etc. available as a guideline during judges calibration.(I seem to remember having seen something like this on a certain macbook) Similarly for cappuccino. But also competitors need to remember judges are voluntarily doing their best, and in my experience are readily available and very willing to help competitors in de-breifing.Experience and knowledge count for a lot in competitions and competing should be regarded as a learning curve.

  7. I’ve been thinking about the descriptions abit more, to me they don’t seem to be very balanced. This often leads to a competitor feeling negative about a low score. If you score a 4 in every category and get every single point available your score will be 700. Correct me if I’m wrong but in last years UKBC and WBC 700 points would have got you in to the top 3. A ’4′ is only described as ‘very good’. I would have thought that the best Baristas in the world would be considered excellent, amazing or even extrodinary!
    Perhaps if the descriptions took this in to account we would all feel better about what score we achieved.

  8. James,
    It seems that any sort of judged activity that has both an element of fundamental and discernable measurements mixed with subjectivity must always be under the microscope. I like the idea of accountability to low scoring because it causes judges to be clear in their expectations and evaluations. I’ve always wondered how well anybody does in judging an espresso that is made well but does not suit their tastes. Nevertheless, with the competition really still in its infancy (8 or 9 years if I recall correctly) there must be a constant evaluation process. Thanks for your post

  9. As long as the judging is consistant between the competitors at an event the numbers don’t really matter.

    Presented with ten espressos pulled by ten different baristas, anyone would be able to rank them relatively. Most espresso enthusiasts will be able critique a shot, I know I spend all day doing just that to my own.

    I’d love to have had more idea of what I was doing right or wrong. That’s where a wiki page would be brilliant. Bring it on…. but not before the finals this year!

  10. I’ve just been rifling through the score sheets myself in order to find those extra points. It seems to me that the way the sheets are set up now it is possible to charm your way into the higher points scores. Is that not what being a great barista is about? It isn’t just the technical side of it, nor is it just about creating a ‘perfect’ tasting drink, whatever that is. If you connect with your customers and they enjoy the experience then surely you are their best barista. Is that not what we are doing in competition, persuading the judges/customers that we are the best? (not that I’ve been anywhere near that may I add).

    It seems to me that you can either go down the route of deciding what the best drink is/performance is and have competitors try their best to replicate that or leave it like it is and let the guys use the skills that they have acquired in their jobs.

    I remember a section on one of Nick and Jay’s Podcast discussing what the perfect cafe is that seems to sum it up for me. It’s not one single thing that makes a great cafe it’s a combination of everything. Does this not apply to the barista? You definitely need the human edge.

    Hey.. if that follows through you might even see some Italians doing well in comps. Or, are Italians too human!

    Shoot me down please!

  11. Hi James,
    Some interesting contributions here. I think that the human element in all things is always the best and worst of it.May be rather than a callibration day and learning of rules, judges should have to pass something? Also an interesting phrase that keeps cropping up is “the competition is in its infancy, after only 9 years. I wonder what most of the current entrants were doing then?
    I’m not that far from 40, so I can pretend to sound old and experienced!

  12. mentess/stuartlee:

    If competition were about replicating what we do daily in our cafes, then your first paragraph would be spot on. This is something I thought about when I competed in 07. However, how are you supposed connect with and “charm” the judges as you would your customers/guests when the only acknowledgement they are allowed to give is when you greet them as you introduce yourself at the start of your performance time?

  13. Morning Anthony

    Granted, it is a one way conversation. I think that what I am trying to do, and what I’ve seen other do well, Jim especially, is put the judges in an frame of mind where it is easy for them to award good points scores. There is definatly a leeway there to take advantage of.

  14. I’ve judged in the U.S. and I think the judge’s training and the calibration during score review does a good job defining what drinks deserve what score.

    You could argue the words used could be better somehow, but judges learn what they mean and apply them consistently.

    A problem could be that the competitors aren’t trained in the meanings in the same way the judges are, and this leads to confusion and disappointment when they don’t share the “judges definition”.

    Judges should be giving good feedback on the scores regardless if they are below a certain number or not.

  15. Just a few thoughts:

    Scoring: I don’t think that there is a problem with the score system, provided that the judges apply it fairly to everyone. Competitors can just add four in their head to get to a score out of ten. That way, “average” works out to be 5/10, which is probably more intuitively satisfying. I guess it’s sort of like cupping on the traditional scale where 50 points are always added and cuppers use the cuppers correction to take away points for anything that they want to score below 50. So we could do something like re-writing the scoresheets with the /6 scores as /10 scores, then just subtract 4 before multiplying each so that the numbers used remain the same. Frankly, though, seeing as this doesn’t change the information that the numbers convey, I don’t really see that it’s a big issue.

    Feedback: I agree 100% that judges need to give as much feedback as is humanly possible. Even if a competitor comes dead last, that competitor should have a positive experience that sets him or her on the path to improvement. Given that the comments fields on the score sheets are so tiny, it’s not unreasonable to expect that they are filled for every competitor. Justin Metcalf also sets the expectation that all judges stay back after the event until every competitor has had a chance to talk with them. That way, everyone gets to ask “what more did you want?”

    Transparency, insider information and consistency: It’s so true that everyone feels that they have to have insider information and, imho, that’s by far the worst part of the current competition format. I don’t think that anyone expects that judges will automatically know everything. It’s perfectly reasonable that they might need to confer to decide how to interpret something, given that people are always trying new things at competitions. Once that happens, everyone needs to judge consistently, though. It is utterly ludicrous, for example, for one judge to be permitted to dock points from a signature drink for it not having a spoon with it, whilst the other three judges don’t. It’s either one or the other. It’s ridiculous for some judges to dock points for knowledge of grinder because someone chooses to use a Mahlkoenig (thanks for eliminating that problem, James ;P) or a grinder with a timer. If these sorts of inconsistencies exist, the head judge is not doing their job.

    Basically, what I would like to see is a competition culture where competitors are able to contact the head judge to ask questions before the competition, when it would be too late to change anything. Once a ruling is given, that should be binding for that particular competition and the ruling should be published on the relevant specialty coffee association’s webpage – a similar idea to the wiki, I guess. To prevent last minute problems, there should be a cutoff for questions a few weeks before each competition – after that, it’s fair enough to let competitors take their chances.

    Judging the judges: Comparing each judge’s scores with each other judge’s scores isn’t novel, nor is it particularly difficult. I think that each specialty coffee association should maintain a register and use the information to decide which people judge the finals. They might not give the same absolute scores as one might get at the worlds, but at least it would remove some of the random variation to help to make sure that the finalists feel that the final ranking is right.

    Cheers,

    Luca

  16. Oh, before I forget, I think that Keith from AASCA has made an awesome spreadsheet that works out how differently judges are judging from the mean and corrects for judges who judge too harshly or generously. That’s only really important where the judges change between competitors, though.

  17. As much as I like Luca’s point about the grinder docking and the belief that James’ use of the Mahlkonig has “corrected” that problem, I don’t know if that will be the case. Unfortunately, there is no continuity in the judging. Just because it was “okay” to use at the WBC, doesn’t not mean that it will be “okay” at nationals or regionals around the world. This is an area the WBC needs to address.

    I would also like to see the score revealed after the competitors’ performance. Currently, scores are held until after the competition – effectively eliminating a teams’ chance to challenge or contest a ruling. This creates a severe problem when a competitor may have access to insider information – a problem that seemed to have occured during the 2006 USBC.

  18. Hi James
    And well said.
    I feel all judges should write the reason a drink was scored, whatever the score may be, and stand by it. Lower than a three yes, but also why was my drink very good? I understand as a former competitor and WBC Judge, how important the feedback is! Back in 2002 you never recived any feedback or ever saw your judging sheets. So it has moved forward and evolved over the years, yet there is still vast room for improvement. I feel that the segregation of the judges and competitors can sometimes be unhealthy, but in turn understand the reason why, the old catch 22! As a competitor you only get chance to connect or share thoughts and opinions after the events with the judges, which often has limited time constraints. The Judges must also be able to connect and understand the competitor, and sometimes this may not happen. The Judge may understand the elements but have they stood behind the machine? I’ll probably get told off for that one, but hey, if not! why not?

  19. I wish Brazil had someone opened minded to organize this kind of events…

    I’m not a competitor nor a Judge, but I’ve been watching this Championships the last 3 years and got very disappointed with it..

  20. I just judged my first competition. WOW!!! I saw a LOT of great stuff produced.

    The training was an all day affair with a fair amount of time spent with three different baristas pulling shot after shot/capp and us tasting according the the competition sheets. It even included a mock competition (minus Sig drink) in real time.

    I tasted several drinks that I liked, but had to score lower according to the rules. The reverse was also true.

    As much as humanly possible, the rules for scoring remove the personal preference bias. All competitors have access to the score sheets that will be used on them in competition. Some choose to push/break the rules in one area because they feel they can make it up in other areas. That is their choice.

    What the seasoned judges kept telling us was to concentrate on the words next to the numbers we were to use for scoring, NOT the numbers themselves.

    A couple of times I wanted to give a high number on a Sig drink. I had some that were really enjoyable, but the coffee part of the equation got lost in everything that was put in it. That, and the fact that so many competitors forgot to connect the ingredients in their Sig drink to the COFFEE.

    I know after going through the Judges Training I am going to be much more critical of my performance in the shop. I know that getting a “3″ (means GOOD) is not a bad thing, but I don’t want to serve just ‘GOOD’ to my customers. I want to serve better than that.

    Right now I don’t know if anything does need to be done to change the scoring format. Things may need to be “tweaked” from time to time, but it seems pretty good to me. As long as people can commit to judging by the score sheets, and their rules, instead of personal bias for drink taste, or the person competing, then it should be fine. But as small as the coffee world is getting here in the States, the only way to try and avoid this would be to fly people in from completely different regions for judging. But I don’t see that happening any time soon unless someone extreeeeeemely wealthy sets up an endowment for air fair and hotels.

    I like the challenge of judging and hope to do it again as soon as I can get the time away from the shop.

  21. Hey James,

    I’m going to leave the rules in peace for now, though I do think they should be changed, and talk about blank score sheets.

    From my experience of judging both regionals and national competitions, the sad truth is that the baristas just don’t care to listen. I happen to be one of those judges who take meticulous notes to explain my scores, and I’ve written extensively about the experience of judging and my perception of how baristas should approach a competition to improve their performance. Sadly, there’s nobody out there. In the U.S., for the past two years, they have introduced a post competition debriefing in which judges are available to go over the score sheet with the baristas. It is a great opportunity not only to explain the score, but to point out where improvement is needed, and how to do better next time. I could count on one hand with a few fingers to spare, the number of baristas that stay for the debriefing.

  22. I have competed and judged and organized competitions in the past. I have issues with the regional support that comes from the WBC. My first regional comp that I help I had sanctioned from the SCAA. There was no guidelines that I had to follow. It was assumed that I knew what I was doing!? Having been through the western Canadians a few times I can say that the judging and general organization needs some help. This is where the WBC can and should get involved. It is fine to say that the rules must be followed , but who is making sure that this is happening? The judges are picked by a qualified head judge. The judges though are not necessarily sanctioned or qualified. This year was a good example of this.
    -One set of judges for 12 competitor back to back .
    -one round of competing, no finals.
    -one tech judge (at least one that I am sure of) has no judging experience at all. Not even sensory!
    -one sensory judge who is a good roaster is a self professed cafe hermit (never goes anywhere for coffee). How can this person judge with no references out side their own coffee profile? This person was asked because of their relationship with the head judge and the fact that at the last minute there was no one else to ask.
    -the other judge is the owner of a cafe where they buy the coffee of the two top competitors. The unexperienced tech judge is his employee and the M.C. is his partner.

    I could go on but I do not want to be thought of as a bitcher. I highlight this to emphasize the fact that the WBC needs to take control of the regionals or run the risk of loosing the faith of the public. There are a lot of great things about the competitions, but this will be the last time that I participate in the fun. I am looking forward to the time I can support a “blind” competition. I think this will be the future of Barista (espresso) comps. That is my two bits, anyone have any great ideas for the next generation of competitions?

Submit a comment