WBC Machine Testing – Temperature Testing

Vancouver is a lovely city, but it seems it likes to rain. So we’ve all been cooped up inside testing the six machines submitted (I am glad I am not the one paying to run 6 three groups all day!)

People probably want to know how this works and I will try and explain, but I warn you it isn’t massively exciting. As a tester I have no idea how the machines performed. The Scace devices run straight into a datalogger and then into a laptop which is kept out of view. I will never get to see the data, unless a manufacturer chooses to publish. Also a manufacturer will only see their data and not the other machines.

The testing protocol is pretty simple. Each round consists of 14 shots pulled at varying intervals, with the device removed at certain points to simulate grinding, as well as knocking out the puck and flushing. Each machine has four rounds of testing: middle group only, middle group and steaming, 1st and 3rd group only and 1st and 3rd gorup with steaming. With the latter rounds two people worked the machine, even though it was replicating one barista using this. We did this to make sure we could be as consistent as possible in the collection of data across machines. With the steaming we had tested each machine to see the time it took to steam 10oz of milk up to 60C, and then that time was used during the testing.

It hasn’t been the most fascinating testing to watch – every manufacturer was present with a technician during the testing and I think the constant counting down each action, second by second, was close to driving everyone a little crazy!

I think the testing has been very fair, I think the protocols are very good (though I don’t think anyone is claiming that it is beyond improvement). If anyone has any questions I will try and answer them though if you have issues or complaints I am probably the wrong person to speak to. Everyone seems very happy with the fairness of it, which is good.

Next up is grinders which will take a while, and I will write a bit more about that after we have done it.

16 Comments

  1. Yeah, gotta admit it’s pretty boring to watch too. I shot about 15 mins of video and am debating on whether it’s worth the time to stitch up and put up on YouTube.

    I’ll mention something I don’t think I’ve seen you mention (just glanced at the commentary so far): Greg Scace is doing a fantastic job of this; I believe the testing protocols are mainly his, and they are very, very well designed. As you said, there’s always room for improvement, but it’s a good sight better than the last time around (3 years ago).

    Re Jay’s comments the previous post – in a refreshing surprise, by and large politics seem to be devoid at this thing. It’s nice to see the manufacturers chit chatting and being very cordial and nice to each other.

    Re Alistair. I have the model names in my camera’s photos and notes, but off the top of my head:

    – Dalla Corte’s newest machine
    – Brasilia’s WBC spec machine
    – La Cimbali’s M39 Dosatron, WBC Spec’ed (with a carbon fibre body shell!!!)
    – Faema with their high end machine, but off the shelf (E61 styled too)
    – Nuova Simonelli’s Aurellia semi-auto, latest version, no special WBC spec.
    – La Marzocco FB80, WBC spec with paddle group controls!!!

    Mark

  2. Forgot to add – two no-shows: CMA’s Astoria (bit o’ a surprise, since they got the dough), and La Spaziale.

    Re grinders, I’ve asked a few times so far who the grinder submissions are, and couldn’t get a complete answer; that said, I saw

    – Compak
    – Nuova Simonelli (two, their Mythos and a traditional one)
    – Rossi / La Cimbali?
    – Er, that’s about it. There’s most likely more
    No Anfim. No Mazzer. The Mazzer absense… well, it’s Mazzer. They’re the #1 seller, maybe they felt their didn’t need to get the title. Anfim though is a grinder that should have been entered, IMO, but maybe politics played a part there, I don’t know.

  3. Mark said: “Greg Scace is doing a fantastic job of this; I believe the testing protocols are mainly his, and they are very, very well designed.”

    Of course if Greg is involved the testing will be accurate, impartial and very professional. But do you know if the standards have been changed, or are the machines still required to produce the same flat profiles that (IMHO) defy both common sense and the predominant Italian preference?

  4. The machines were never required to produce flat line profiles, not in this test or any other WBC test (contrary to popular belief).

    The data is analysed in a number of ways but ultimately the search is for repeatability, not for a certain profile.

    Everyone has a different opinion – in fact one manufacturer has brought a machine designed to brew at 14 bars at a lower temp for a fuller extraction. (we’ve only tested for 9 bars at this stage though, in an effort to remain consistent.)

  5. Added – Malkhonig (I can never spell that right) had two grinders, a K30 (James’ choice at the WBC), and a double grinder, the K31?; Nuova Simonelli decided at the last minute to pull one of their two grinders, and go with the Mythos as their submission. Rossi’s looks interesting, and the Compak look pretty much like the Compak we know from the previous WBCs. Tasting’s in about 9 hours, I need to get some ZZZs to be ready.

  6. Hi Jim,

    It all sounds very interesting but how do they decide the winner? Is it purely on performance in the testing stages? or is there any other criteria on a more basic level like ease of use for the barista etc. or the ability for the manafacturer to support the event?

    It will be interesting to see what tests you do on the grinders.

  7. Oops, I think I linked to page 2 of the discussion. Please go back to page 1 to read the standard.

  8. If you read the testing then what is looked at is the average brew temperature for the shot, and then this value is compared across the 14 shots for the samples taken on the machines.

    Of course there is the possibility for varying profiles to produce close averages but it is unlikely, and would be spotted in the graphing.

    As long as a humped curve is the same each time then it should have the same average temperature and therefore would be considered stable/repeatable. The shape of the curve is relatively irrelevant.

    I don’t want to speak on anyone’s behalf but this is just my understanding of the process.

    (The reason my answers seem so prepared is that I am asking the exact same questions!)

    At least one manufacturer has stated they will publish their data regardless which I think is pretty exciting and would love to see happen – a step closer to increased transparency and all…

    Ed – the machines are also scored ergonomically and will go through a taste testing procedure (which we will do later today). However a machine could theoretically be discounted on temperature testing alone. I think there is going to be a lot of feedback from the baristas directly to the manufacturers (i.e. we sit down here and talk to them) which is pretty cool.

    Once we start on the grinders I will post about them too.

  9. Hi James, This seems really progressive (if that isn’t a little 1970’s phrase) just wish the whole industry could see the findings to expell a few myths and also make some informed decissions.

  10. > If you read the testing then what is looked at is the average brew temperature for the shot,
    > and then this value is compared across the 14 shots for the samples taken on the machines.
    >
    > As long as a humped curve is the same each time then it should have the same average
    > temperature and therefore would be considered stable/repeatable.
    > The shape of the curve is relatively irrelevant.

    That’s great.

    The measurement you describe is what that 2005 document specified in section “6.3 Brew Temperature Reproducibility.” It indeed makes the shape of the curve irrelevant.

    In 2005 they also made another measurement specified in section “6.4 Espresso Machine Temperature Stability.” This section was about getting a profile that was as flat as possible. If they no longer bother with this measurement, then we have moved beyond the flat earth theory. Cheers!

  11. If you are still looking for a Ditting Grinder I may have a DEAL for you. Get back to me quickly.
    tres

  12. I know i`m a bit too curious, but is the manufacturer that promised to publish their machines data DallaCorte? No obligation whatsoever in answering..

Submit a comment