Beware of misleading comments from Microsoft on JavaScript test suite “test262” and IE10

I notice that yet another "benchmark" blog post from Microsoft is making the rounds on the internet.

This time they are bragging about having the highest score for JavaScript test suite "test262".

As one might expect, the impression people are getting when reading the blog is not one that matches reality. The results are misleading unless you read the "fine print", which seems to escape most people who comment on it. You see, while the IE10 preview appears to come out on top, the results are not what most people think they are. …

A high score does not show overall compliance. It only shows whether you pass the currently available tests or not. This is a flashback to Microsoft's blog post on HTML5 compliance last year.

Here's what the actual test suite page says (emphasis mine):

test262 is not yet complete. It is still undergoing active development. Some portions of the ES5 specification have very complete test coverage while other portions of the specification have only partial test coverage. Some tests may be invalid or may yield false positive or false negative results. A perfect passing score on test262 does not guarantee that a JavaScript implementation perfectly supports ES5. Because tests are being actively added and modified, tests results from different days or times may not be directly comparable.

Summary:

  • The test suite is not complete
  • It is in active development
  • It does not cover all parts of the specification
  • There may be invalid tests
  • Tests may yield false positive or false negative results
  • A perfect score does not imply a perfect ES5 implementation
  • Tests are being added and removed all the time

If you read the Microsoft blog post carefully, they do mention that it is a work in progress. But the way it is being reported on it's obvious that the message people read was "IE10 is the most compliant". The blog post also reveals that Microsoft submitted thousands of tests (which they should receive praise for), so it does not seem to be coincidental that they are reporting on it now.

In other words: While Microsoft briefly mentions in "fine print" that the test is not complete, it seems that most people are misled, and think the tests results say something that they don't.

Advertisements

26 thoughts on “Beware of misleading comments from Microsoft on JavaScript test suite “test262” and IE10

  1. Originally posted by BS-Harou:

    on the other hand, Opera does really suck in ES5 support

    +1

  2. Originally posted by BS-Harou:

    on the other hand, Opera does really suck in ES5 support 😥

    And doesn't appear to be at all active on ES Discuss 😦

  3. Originally posted by BS-Harou:

    on the other hand, Opera does really suck in ES5 support

    Yeh, that's why a lot of the tests fail in Opera.

  4. Thanks for sharing and jeeping a vigilant eye.There's lies, goddamn lies and communication strategies …. ;):knight: :sherlock: :jester:

  5. So Microsoft stretches the facts againlots of them have no errorpause resume button might be buggynow it is randomly stoping now the test just broke itself

  6. Originally posted by BS-Harou:

    on the other hand, Opera does really suck in ES5 support 😥

    Originally posted by Chas4:

    Google's http://sputnik.googlelabs.com has different results tho thye don't have IE 9 listed since it's not for xp also not listed a current test of ff or chrome

    Wasn't Opera bragging about their Sputnik score before?Oh btw Sputnik was integrated into test262 http://code.google.com/p/sputniktests/We need a JS overhaul for multi threading, among many other updates like the webm/webp codecs, sqlite version update, and zlib/openssl license text needs updating in opera:about if your using the newer versions.

  7. Heh, I've ran sputnik test with opera, chromium and firefox, and at the end opera scored with less fails of all. Opera 11.50 results:Total: 5246Succeeded: 5166Failed: 80Chromium 13 results:Total: 5246Succeeded: 5115Failed: 131Firefox 5.0 results:Total: 5246Succeeded: 5065Failed: 181all three browsers are fresh installs.

  8. Sunspider javascript test:Opera 11.50 RESULTS (means and 95% confidence intervals)——————————————–Total: 322.9ms +/- 0.8%——————————————– 3d: 45.4ms +/- 1.7% cube: 12.8ms +/- 2.4% morph: 17.7ms +/- 2.0% raytrace: 14.9ms +/- 2.7% access: 41.7ms +/- 1.6% binary-trees: 3.6ms +/- 10.3% fannkuch: 21.2ms +/- 1.4% nbody: 10.4ms +/- 3.5% nsieve: 6.5ms +/- 5.8% bitops: 15.1ms +/- 3.5% 3bit-bits-in-byte: 1.7ms +/- 20.3% bits-in-byte: 2.5ms +/- 15.1% bitwise-and: 2.0ms +/- 16.8% nsieve-bits: 8.9ms +/- 2.5% controlflow: 4.7ms +/- 7.3% recursive: 4.7ms +/- 7.3% crypto: 21.0ms +/- 2.3% aes: 12.6ms +/- 2.9% md5: 5.0ms +/- 0.0% sha1: 3.4ms +/- 10.9% date: 40.8ms +/- 1.4% format-tofte: 18.8ms +/- 2.4% format-xparb: 22.0ms +/- 1.5% math: 29.7ms +/- 2.5% cordic: 8.1ms +/- 5.0% partial-sums: 16.1ms +/- 1.4% spectral-norm: 5.5ms +/- 6.8% regexp: 14.0ms +/- 0.0% dna: 14.0ms +/- 0.0% string: 110.5ms +/- 0.8% base64: 10.8ms +/- 2.8% fasta: 19.5ms +/- 1.9% tagcloud: 34.0ms +/- 1.0% unpack-code: 23.5ms +/- 1.6% validate-input: 22.7ms +/- 2.1%Firefox 5.0 RESULTS (means and 95% confidence intervals)——————————————–Total: 324.1ms +/- 0.7%——————————————– 3d: 53.8ms +/- 1.4% cube: 21.6ms +/- 3.2% morph: 9.5ms +/- 4.0% raytrace: 22.7ms +/- 1.5% access: 49.4ms +/- 2.2% binary-trees: 6.0ms +/- 5.6% fannkuch: 25.1ms +/- 2.5% nbody: 7.6ms +/- 4.9% nsieve: 10.7ms +/- 3.2% bitops: 22.8ms +/- 1.3% 3bit-bits-in-byte: 1.0ms +/- 0.0% bits-in-byte: 12.0ms +/- 0.0% bitwise-and: 2.9ms +/- 7.8% nsieve-bits: 6.9ms +/- 3.3% controlflow: 3.8ms +/- 7.9% recursive: 3.8ms +/- 7.9% crypto: 24.1ms +/- 2.9% aes: 13.4ms +/- 2.8% md5: 6.6ms +/- 5.6% sha1: 4.1ms +/- 5.5% date: 41.5ms +/- 2.5% format-tofte: 28.1ms +/- 1.4% format-xparb: 13.4ms +/- 6.3% math: 30.6ms +/- 1.6% cordic: 7.3ms +/- 4.7% partial-sums: 15.0ms +/- 0.0% spectral-norm: 8.3ms +/- 4.2% regexp: 18.2ms +/- 1.7% dna: 18.2ms +/- 1.7% string: 79.9ms +/- 1.2% base64: 5.6ms +/- 6.6% fasta: 16.5ms +/- 2.3% tagcloud: 23.2ms +/- 1.9% unpack-code: 25.7ms +/- 1.3% validate-input: 8.9ms +/- 2.5%Chromium 13 RESULTS (means and 95% confidence intervals)——————————————–Total: 376.0ms +/- 2.3%——————————————– 3d: 51.0ms +/- 11.7% cube: 16.0ms +/- 15.2% morph: 16.5ms +/- 2.3% raytrace: 18.5ms +/- 24.0% access: 43.0ms +/- 5.2% binary-trees: 3.4ms +/- 10.9% fannkuch: 24.2ms +/- 5.9% nbody: 7.9ms +/- 5.1% nsieve: 7.5ms +/- 19.2% bitops: 49.3ms +/- 8.2% 3bit-bits-in-byte: 3.1ms +/- 7.3% bits-in-byte: 7.9ms +/- 2.9% bitwise-and: 26.6ms +/- 1.4% nsieve-bits: 11.7ms +/- 35.6% controlflow: 4.6ms +/- 15.0% recursive: 4.6ms +/- 15.0% crypto: 23.3ms +/- 3.8% aes: 11.8ms +/- 9.8% md5: 6.1ms +/- 6.7% sha1: 5.4ms +/- 6.8% date: 56.7ms +/- 1.5% format-tofte: 19.3ms +/- 3.9% format-xparb: 37.4ms +/- 2.4% math: 31.0ms +/- 5.4% cordic: 9.3ms +/- 10.9% partial-sums: 17.1ms +/- 3.1% spectral-norm: 4.6ms +/- 25.6% regexp: 13.5ms +/- 6.7% dna: 13.5ms +/- 6.7% string: 103.6ms +/- 0.9% base64: 6.8ms +/- 4.4% fasta: 17.6ms +/- 2.1% tagcloud: 25.7ms +/- 1.9% unpack-code: 35.9ms +/- 1.7% validate-input: 17.6ms +/- 2.1%

  9. Originally posted by Asires:

    Also, Sunspider is outdated. Use Kraken.

    still will depend on the hardware and how updated the drivers are, I have seen a 100 ms difference on sunspider between an older and new machine, using the same browser

  10. Originally posted by toyotabedzrock:

    Wasn't Opera bragging about their Sputnik score before?

    Really? You mean this, where the Sputnik creators are being praised for publishing the test like that, and where there is no bragging whatsoever?And of course the difference here, where Sputnik was not made by Opera at all, while test262 had a significant number of tests recently submitted by Microsoft.So are you going to apologize now?

  11. Yes, Opera still has "gaping holes" in ES5 support… So what – these are the features not yet used by web-app developers anyway…Most of the other so called "errors" I see are:1) Not throwing an exception where it should be thrown — this increase JS engine performance, BTW…2) Errors which are, if fixed, may cause a lot of loose code to fail unexpectedly on a lot of random websites…UPD: 1000+ errors are caused by not implemented "Strict Mode"… This test is so not uniform… Some features give you 1000 points and others only 10-20…

  12. these are the features not yet used by web-app developers anyway…

    not true, plenty of modern websites and apps are using it. They often provide ES3 fallback but that is not good solution.

  13. Um, Opera doesn't have "holes" in our ES5 support. We have just about no ES5 support at all, the shiny new things like strict mode are simply not implemented yet.They will be..

  14. Originally posted by hallvors:

    We have just about no ES5 support at all,

    Not true, Opera supports Array extras from ES5, and some other things like Array.isArray, Date.prototype.toISOString, String.prototype.trim, etc.

  15. Yeap. trim and array extras are from ES5.Originally posted by hallvors:

    They will be..

    Now that's a good news ! We eagerly await :spock:

  16. Anyway, Opera is the slower browser now. Opera was fast about 2 years ago, but not, it doesn't. I still love Opera, but I have changed to Chrome because Chrome is fast. I can notice it, not just some miliseconds, but some seconds in some pages.

  17. Originally posted by macusking:

    Anyway, Opera is the slower browser now. Opera was fast about 2 years ago, but not, it doesn't.

    Complete and utter nonsense. Opera is now faster than any version before it, and one of the fastest browsers on the market.

    I still love Opera, but I have changed to Chrome because Chrome is fast.

    Good job. You were fooled by Google's aggressive marketing.

  18. No, I was Not. I'm just waiting for the Opera w/ hardware acceleration! I miss Opera so much!Originally posted by Chirpie:

    Originally posted by macusking:

    Anyway, Opera is the slower browser now. Opera was fast about 2 years ago, but not, it doesn't.

    Complete and utter nonsense. Opera is now faster than any version before it, and one of the fastest browsers on the market.

    I still love Opera, but I have changed to Chrome because Chrome is fast.

    Good job. You were fooled by Google's aggressive marketing.

Comments are closed.