ExtremeTech’s “Which Web Browser is King?” browser comparison scores don’t tell the full story…

ExtremeTech recently tested several browsers, pitting them against each other in various tests, in an article called "Which Web Browser is King?".

While I could comment on how they are using the V8 Benchmark Suite for which V8 is specifically optimized whereas other browsers aren't, what I actually want to look at is how they distribute points for each tests. You see, slighly changing to a (IMHO) better way to distribute points changes the picture somewhat. …

ExtremeTech's final score is based on giving 10 points to the winner and nothing to the rest. This results in Chrome as #1 with 30 points, Firefox as #2 with 20 points, and Opera as #3 with 10 points. Safari and IE didn't win any tests, so they didn't get any points at all.

This seems to be an odd way to give out scores, so I quickly calculated how things would look if the winner of each round got 5 points, the 2nd 4 points, and down to the 5th spot with 1 point (the number of points, with the ranking for each test in parentheses):

While Chrome still comes out on top, the difference isn't as great as what ExtremeTech's original score would indicate. Opera actually places second, while Firefox comes in third. Surprisingly, Safari, which has received a lot of praise as "the new best browser" from a lot of sites, places behind everyone except Internet Explorer.

If giving points to the loser of each round doesn't sound like a good idea, here's how it looks if you only give points to the first three spots:

As you can see, Chrome still wins, and Opera still comes in second. However, Firefox and Safari are now tied.

I think both of these ways to calculate the score are much better than the way ExtremeTech did it. When it comes to performance, being good at it will help a lot even if you aren't always the fastest. Performance isn't always all-or-nothing, but often gradual improvements over time.

ExtremeTech promises to revisit these benchmarks in the future, when Microsoft and Mozilla launch their new versions. I hope they will take that opportunity to make their scores more reflective of each browser's performance throughout the test. And I hope, of course, that Opera 10 will be included as well 🙂

Advertisements

14 thoughts on “ExtremeTech’s “Which Web Browser is King?” browser comparison scores don’t tell the full story…

  1. Yeah! Give it to us already!Promise, we will test it against everything ourselves. :DP.S. Yeah, I also found their final score calculations misguided after I read the article yesterday.

  2. OK, it becomes better and better: no more "WIR" for Opera 10, now we got status named "SE" (soon enough). 🙂

  3. Good grief, Opera 10! Opera is so much better than these other browsers, in my opinion, already. Opera 10 coming "soon enough" shows that Opera is unconcerned about the other guys' (the competition, that is) feelings. When will the world wake up and start using Opera instead of all the other pretenders?

  4. Haavard, a good read! I'd just wish the titles from your RSS feed stopped showing up like this in Google Reader: "ExtremeTech's "Which Web Browser is King?" browser comparison scores don't tell the full story…"

  5. Is Chrome more "specifically optimized" for the V8 Benchmark than Safari is for the SunSpider benchmark? I guess both parties have selected benchmarks that represent the code they want to optimize for, and then wrote code that did just that (i.e., SunSpider excells at representing crappy code found on the internet :).Me, I use Chrome for GMail, Firefox for Roborunner, and Opera for actally browsing. Speed only really matters for online applications. Everything but IE7 is "fast enough" for plain browsing.

  6. Anonymous writes:

    I like to score based on averaging where the browsers placed in each subtest. For example at http://tinyurl.com/2008browsers there are 8 subtests and based on placings in each the ranking is as follows:1. Chrome – 2nd,3rd,2nd,1st,1st,1st,2nd,2nd = 1.752. Firefox- 1st,1st,1st,2nd,2nd,4th,4th,3rd = 2.253. Opera – 4th,2nd,3rd,4th,3rd,3rd,1st,1st = 2.6254. IE – 3rd,3rd,4th,3rd,4th,2nd,3rd,4th = 3.25(IE and Chrome tied for 3rd in the second subtest)

  7. Of course the tests in question don't take into account the real test of speed: how long it takes your agent to render something after you go back. In such a test Chrome is woefully slow, as bad as IE if not worse. Considering how often people go back, forward and back again, I would have thought this would be an area of prime concern.

  8. That's no kind of test, MTKnight… In Opera, I can't "see" anything happening! The "old" page is just "there"… 😀

  9. Funny… I had the exact same feeling when I read the Extremetech article, the fact that the scoring, or to be more precise the weighting of the scoring, was somewhat unbalanced.Prior to reading this, I read the article and applyied a fairer scoring metric to each of the tests. For me, Opera also came out joint second (with safari), not forgetting the obvious caveat that V8 tests are webkit (safari/chrome) optimised.Overall I don't think the ET article is a useful comparative review of browser performance let alone browsers themselves.Seb 🙂

  10. Opera 10 Alpha 1SunSpider JavaScript Benchmark ResultsRESULTS (means and 95% confidence intervals)——————————————–Total: 4610.6ms +/- 0.8%Very fast 🙂

Comments are closed.