Malware report from NSS Labs manipulates statistics?

A Microsoft sponsored study concluded that IE8 catches the most malware, and that Opera catches nearly nothing. But can the report really be trusted? …

We will never know what's really going on because important part of their selection and methodology is simply not revealed. As such, there is no way to verify or falsify their claims, possibly pushing this into the realm of pseudoscience.

One can also question the objectivity of NSS Labs when they make statements like this:

We were impressed by the stability of IE8 (RC1).

An interesting observation is that the report is from March 12th, 2009. They claim to have done 24/7 testing for 12 days, meaning that they must have started before Opera 9.64 was released, even though it's in their report!

There are other problems as well:

  • Safari 4 and Firefox 3.1 were left out, while IE8 RC, a non-final version, is allowed
  • The report says that 7% of the threats were blocked by all browsers, but Opera is claimed to have blocked only 5%
  • They started out with 150 000 URLs, but ended up with only 492 in the final test
  • Out of the 492 final tests, the same site could have up to 10% of the URLs, meaning that in a "worst case scenario", 10 unique sites were tested! If a browser did particularly well on one of these sites making up more than 10% of the test, their score would obviously be inflated (the report mentions that a number of sites were pruned after reaching their limit)
  • According to the "Malware URL Response" table on page 3, Opera catches 15% on hour 0, and 33% after 5 days. And yet the final rate was set to only 5%
  • According to the same table, Chrome consistently catches 25% or more, but the final score is only 16%
  • The same table shows that IE8 never reaches 69% even once in the table, and yet its final score is raised to 69%
  • On the other hand, IE7 has a total score of 17% in the table, but the final score is lowered to 4%

One could almost get the impression that NSS Labs is setting the test up in a very specific way to exaggerate the results in a certain direction:

The computations are done based on this "So if it is blocked early on, it will improve the score. If it continues to be missed, it will detract from the score." Sounds like a typical statistical trick to exaggerate differences found between browsers – those that do well will further improve their score, those that do less well will further decrease their score. With proper selection of the algorithm, one can maximise the resulting difference. This is how an absolute score of 33% for Opera is changed to a score of 5% after the statistical manipulation.

As mentioned above, these anomalies do not only affect Opera, but all the browsers in the test. What could in reality be a tiny and insignificant difference turns into what seems to be a huge gap in the final report. Do other browsers really report less than half of what IE8 does?

The test also measures success at preventing malicious sites from being downloaded, but if Opera only shows the warning after downloading the page, it will automatically fail on a lot of tests even though the user is actually warned. It is not the downloading which is dangerous, but the report does not take that into consideration.

This report is receiving quite a bit of attention from the media. From a quick glance at the numbers, my preliminary conclusion is that this is just another Microsoft marketing trick. By carefully manipulating methods and statistics, you can make a set of numbers show just about anything.

I wonder if the other browser vendors have investigated the report, and if they plan to respond in an official manner. I don't know if we will offer any official statements on this, but I don't think what appears to be rather obvious manipulation of the numbers to exaggerate differences should go unquestioned.

It does seem that I am not the only person who is not convinced. Are there any other reports out there that don't simply repeat Microsoft's claims without question?

In order to look more closely into the claims in this report, I have mailed NSS Labs and requested the URL list. Check back later for any updates. (Update: They never sent any URL lists.)

Advertisements

29 thoughts on “Malware report from NSS Labs manipulates statistics?

  1. The report says: "Data in this report spans a testing period of just over 12 days, from February 26 through March 10". (it's on page 6, I think) Does that mean they started testing on 26 february? That's even before the 1st march :eyes:And well it's 'a Microsoft sponsored study'. That says enough to me…

  2. I also wonder why the IE team never fully patches any version of IEIE5IE6IE7The malware protection does not work right if you can't even patch your browserSafari uses google as its phishing/malware protection so not that much strength (only one source data) So if Google goes down then Safari has not fraud protection

  3. Sorry, what does it mean "150K URLs" in following sentence?They also started out with more than 150K URLs, but they ended up with only 492 in the final test.

  4. Yes, they started several days before 9.64 was released. I edited the text a bit to correct and clarify.

  5. The biggest problem is the idea of fraud prevention as malware protection. They are mostly 2 very different things. Most web browsers are not anti-malware programs, and they really shouldn't be asked to be (see Comodo's CEOs comments on this). Malware is really not something you can necessarily detect before downloading it, much of the time you have to have the file to do a signature match, or do behavior analysis, or however you detect it.Most reasonable people I've talked to realize that a user who is the administrator of their PC (I don't mean logged in as admin, just IS the administrator) can't be protected from themselves by software.

  6. jp, it isn't just about malware, but frauds n deceptions in general. My 60yr old dad, although he's pretty familiar with technology, he was arguing with me about the validity of a site which claimed it could track your mobile by your phnoe number… Of course is of no use for experienced users.Apart from that, I m not sure why an advertisement deserves such attention. Who is NSS labs exactly? What's the value of this particular authority? Now, an official MS site linking to them, although it wouldnt surprise me, it would be a nice grab for a blog post.-Btw,I couldnt find the original report. Is this report valid at all?

  7. It's hard to trust their statistics. As you said, it's not clear how they got their hands in Opera 9.64 before it was even released. That info is extremely relevant because that was when Opera started supporting the security measures "Data Execution Prevention (DEP)" and "Address Space Layout Randomization (ASLR)" in the first place. Of course, they offer OS-level security but still, I think that would make a difference.I wonder if there is any improvement in malware detection planned for Peregrine. Does anyone know?

  8. Well, what I meant deadHarlequin was that they are passing off general anti-fraud technologies as anti-malware technologies. That's like claiming the flue vaccine will protect you against a con man. It's a misuse of terms to quite confuse the entire discussion.I argue that while it'd be nice to catch a malware distribution site or drive by download site at the phishing filter, it's not what that filter is targetted at. Just downloading malware isn't going to infect your PC unless, perhaps, it's with IE6 or you've got a known vulnerability and no layered defense. And once it *is* on the PC, beyond the browser not just executing it automatically, it's not the browser's job to do anything there.If you save a file from a site, even if it's malware, once it's on the PC, your anti-malware ought to be catching it.More clearly, I don't think Opera should try to be Comodo Internet Security (anti-malware/firewall/HIPS, whatever) nor do I think the anti-malware software ought to try and be a browser. It's stupid, and going far outside core competencies.It is important to deliniate tasks between software targetted at that task. Part of the reason IE6 was so bad was it tried to *also* be an OS component, and a file manager, instead of being a web browser. When you blur task responsibility, you just confuse the issue, and there's more chances for malware to get in.None of the above adresses phishing though, which is (I think) what the report actually was trying to test. I maintain that no current technology is going to somehow protect a credulous, gullable non-technical user from being scammed. It's just not possible – one HTML site (putting aside malware installers, just talking about fake paypal sites etc) is going to look much like another one to the browser. Sure, certificates can help, but the browser can't really know if you meant to go to site A or site S (the typoed site)… I wouldn't be suprised if most users don't like the computer telling them "what they really want". Heck, most users I support don't like their local IT people pointing them towards what will actually accomplish what they want to do vs the first thing they got off of google that they think does what they want.

  9. Come on jp, comodo's chief cant be considered objective and he isn't, he sells protection to home users.What he actually says is that prevention isnt sth browsers can/should do, instead we need special super-duper-macho scanning software. But I think measures prohibiting the downloading and execution of malware in the first place are a much better connotation of the word 'prevention'.The fact that he tries to attack common web browsers for buffer overflow exploits is laughable. Has he ever had a look on how often his crappy Comodo firewEll had been in security lists for overflows and remote exploits? I guess not. He is even ridiculously attacking sandboxing methods, claiming that they arent 100% safe. Well, sure, there is no thing being 100% safe.But speaking about relative safeness, let's have a look on the software he sells. Lets say comodo antivirus can catch 99%(yes, i am a good person) of the malware. Trying to run a new program a week(or using the claimed 'unsafe' browsers) gives the user a protection equal to:P= 0.99^52 = 0.6 !!!….or just 60% chance of protecting him in a year. His logic simply doesnt work. We need true prevention. Executing everything you browse upon and hoping your antivirus will save you is stupid at best, and for sure cant be called 'prevention'.I ve never used an antivirus for the past 10 years, I have never had any virus problems. Instead I prefer to run Opera with limited privileges using a simple Software Restriction Policy, and that's what I consider a nice prevention method for example.

  10. deadHarlequin, I guess we'll have to disagree then. First, Comodo isn't selling software to home users… Secondly, HIPS, which is what he's suggesting, is quite similar but more interactive than a software restriction policy (and can home users even do this?).And while browsers should be written well, I think the argument is subtler than that (though maybe his isn't) – The browser should prevent exploits of it's code, but no browser is perfect and you need layered defense. Core competencies make sense, rather than trying to get every app perfectly to have no buffer overflows (we've seen how well that works – it doesn't), have one app (or the OS) just watch for and prevent buffer overflows.Opera makes a great browser. But I don't expect them to make an anti-malware with it. How does Opera know if that file you just said to download is a virus? Before it's downloaded anything? No app will. And I argue that it's not the Browsers job to guess if your selected download is malicious or not, that's why you ought to have security software.Finally, your statistics are specious, or are you claiming that the user is going to always download malware? Also, detection of the malware isn't going to be the same percentage chance for each piece of malware code, it will vary. Also, you'll notice that he's suggesting that any one program / vendor as your defence isn't enough, you need layered security. He'd like to provide some layers.

  11. There was no manipulation. But the poster has certainly mis-interpreted the report data, and made some assumptions that we would not agree with, as detailed below. But, I wonder what % of fresh malware sites the poster expected Opera would block after all? Literally, there are hundreds of thousands if not millions of infected sites. It takes some resources to tackle this. 1. What are we not disclosing specifically? The included test methodology is pretty darn complete.2. Opera 9.6.4 was released on 3/3/09. We updated from 9.6.3 to 9.6.4 during the test.3. It appears we need to educate some people on the difference between a Release Candidate and a BETA product. By definition, a Release Candidate is published after the BETA test is completed. If Safari or Firefox had a Release Candidate, it would have been included. At the time of testing, Safari 4 was BETA product as was Firefox 3.1. IE8 RC1 was Microsoft’s FINAL Release Candidate. We made a conscious decision not to including BETA products (with known bugs and stability issues) in our test as it would have been unfair to the Browsers (Firefox and Safari).4. The 5% result was what users can expect at any given point in time. To get this number, we added up the number of blocks by each browser (counting multiple blocks or misses by each browser as individual results), and divided by the total number of URLs looked up over the period of the test. The 7% number (like the histogram) is the overall block number. Meaning for each URL, we counted a URL blocked if it was blocked once throughout the test. We then divided by the number of unique URLs tested (492).5. Part of the reason we ended up with 492 is that Opera (in particular) kept being exploited and crashing. If we had excluded Opera, there would have been over 10,000 results. Also, the test included Phishing, Clickjacking, and so-called “drive-by downloads” (where the web page contains an exploit against a browser and the payload of that exploit is malware that is automatically installed).6. You are misunderstanding – there were 492 UNIQUE URLs included in the test. The 10% weighting number applies to the larger set of 1779, not the 492 UNIQUE URLs.7. See response #48. See response #49. See response #410. See response #411. (Response to next paragraph) – Still you are misunderstanding. There is a distinction between TOTAL catch rate, and the catch rate at any given point in time (with 141 consecutive tests). The reason we tested OVER TIME is to highlight this point. People use products over time, so the more data we collect, the more accurate the results.12. Our test accounted for this behavior. We waited 2 seconds before recording the results. Thus, if the site was first loaded, and then redirected to a warning page, it counted as a “block”. Having said that, don’t you think it is a problem if the site contains an exploit and the Warning comes AFTER the browser has been exploited?13. Let it be known, that we contacted you. At the time you did not disclose any of these concerns or questions which you've not hesitated to post here. If you wanted an answer to them, you could have directed the questions to us. We even have screenshots of missed web pages. 14. As a general note, this test was about SOCIALLY ENGINEERED MALWARE. It did NOT cover Phishing, so-called “drive-by” exploits/downloads, or Clickjacking. Why? Because 53% of malware is now downloaded by users via Socially Engineered Malware. Meaning they BELIEVE they are downloading a free copy of Fantasy Baseball Stat-Tracker. Or a game of some sort. Thus, the topic has a high degree of relevance as we discussed in the report.

  12. Reading the comments on http://www.thetechherald.com/article.php/200912/3268/Can-you-trust-the-NSS-Labs-report-touting-the-benefits-of-IE8, someone named 'Rick' put up a post containing this:Originally posted by Rick:

    The opera blog comments are all interesting misinterpretations by the author and we're responding to that. And yes, opera started at 9.63 but updated itself to 9.64 without user intervention.

    My question: How did opera 9.63 update itself without user intervention? I never succeeded in doing that… :eyes: When opera 9.63 asked me to update, it brought me to the website where I click the 'download' button manually… and run the downloaded file manually :eyes: I though the auto updater was only in opera 10 Alpa?

  13. Nice excuses from NSStest. I sure hope this troll doesn't represent the amateurs at NSS :lol:The RC comment is just gold. NSS is conveniently ignoring the fact that RCs are unreleased software as well. Let's just insult people who point out flaws instead of admitting the fact that we used unfinished software only when convenient for Microsoft! :lol:Never mind the fact that NSS refuses to share its methods and URLs with anyone, and that their numbers don't add up, except if you do some fantastic circus acts with the numbers.Never mind the fact that the test still assumes that a download equals an infection.Laughable. I can understand why Microsoft picked NSS to lie for them, though 😆

  14. Yep. Another NSS lie. They keep spinning a web of lies. Automatic updates are only in Opera 10.

  15. Nsstest:14.As a general note, this test was about SOCIALLY ENGINEERED MALWARE. It did NOT cover Phishing, so-called “drive-by” exploits/downloads, or Clickjacking. Why? Because 53% of malware is now downloaded by users via Socially Engineered Malware. Meaning they BELIEVE they are downloading a free copy of Fantasy Baseball Stat-Tracker. Or a game of some sort. Thus, the topic has a high degree of relevance as we discussed in the report.So this test was about trojans? Who, before you, has expected web browsers to protect users against trojan horses? There are specialized apps for this, as well as Antimalware software. The browser isn't some sort of walled garden system, how are they supposed to prevent trojan horses? This is a bizarre test for browser security, i.e. it's like rating cars on their ability to avoid or prevent carjackings… No one normally expects the car to do anything for that.

  16. Originally posted by NSStest:

    1.What are we not disclosing specifically? The included test methodology is pretty darn complete.

    The list of URLs at the very least ?Originally posted by NSStest:

    5.Part of the reason we ended up with 492 is that Opera (in particular) kept being exploited and crashing.

    You could have been a bit credible if you hadn't posted this. Obviously this statement is just a flamebait: Opera keeps crashing and being exploited ? Oh my ! Then if opera kept being exploited, you should have kept those URLs and claim opera protects against 0.069% of threats. But, it's just a flame bait.And kept crashing ? The 9.6x release cycles has went through a severe QA, testing and real world usage, and it's too much stable to keep "crashing" constantly on your obscure Microsoft-sponsored malware test. If you got one or two crashes, that would be understandable. No browser is crash free.Originally posted by NSStest:

    13.Let it be known, that we contacted you. At the time you did not disclose any of these concerns or questions which you've not hesitated to post here.

    Well, if someone posts publicly FUD about Opera, I think it's at the very least legitimate to reply also publicly. But now you are the victims and Opera are the bad guys ? Originally posted by NSStest:

    We even have screenshots of missed web pages.

    Please send them, including the FULL URL set, not the 492, but the 60k.Originally posted by NSStest:

    14.As a general note, this test was about SOCIALLY ENGINEERED MALWARE.

    Question: if I go to a website and Opera flags it as fraud site, but then I bypass the warning and click a link that downloads a file, is Opera supposed to block that download even after the user bypassed the warning ? It seems that you are actually implying this.

  17. Hi there NSStest, and thanks for your comments.I was not aware that you had contacted Opera Software. As this is my personal blog, I'm writing on behalf of myself, and not Opera Software. No one I asked at Opera had received anything from you. Maybe I didn't ask the right people.I must admit, though, that I find it strange that you were not asked by your contacts at Opera to supply the contact person with more data, such as URLs.I'm sure the relevant people at Opera would interested in hearing about these exploits you are referring to. There must be a lot of unknown security holes in the latest version! Sadly, it doesn't look like any information is available to people who want to see if the claims in the report can be verified.Unfortunately, your commend doesn't quite convince me that your report wasn't manipulated. Perhaps actually publishing data which allows people to verify (or falsify) your report would help. For now, all we have for it is your claim, and considering Microsoft's track record of funding fake reports, I remain skeptical.

  18. quote from rick"And yes, opera started at 9.63 but updated itself to 9.64 without user intervention."And from NSStest "2.Opera 9.6.4 was released on 3/3/09. We updated from 9.6.3 to 9.6.4 during the test."yay a contradiction, at least its from two different people.now check this out:"Also, the test included Phishing, Clickjacking, and so-called “drive-by downloads” (where the web page contains an exploit against a browser and the payload of that exploit is malware that is automatically installed)."Then,"It did NOT cover Phishing, so-called “drive-by” exploits/downloads, or Clickjacking."Ok its bad enough to contradict yourself to start with, but to contradict yourself in the same block of writing is just downright retarded.Because of this, I have 0 respect for your company and its BS. I think a lot of people out there will agree with me.

  19. Never had any doubts that this test was bogus, especially the Opera part. When I used IE7 and later Firefox I got three viruses per day from simply browsing. When I switched to Opera last year… all what I got was tracking cookies, no Trojans (if at all), and tracking cookies aren't too hard to get rid of anyway.

  20. So how about that Opera 10.x vulnerability list? Oh wait, what list? Okay, now public list for Firefox 3.x? 3.5x? Well, what about IE8?Huh. Well, isn't that something. Thanks for dialing back to this, Haavard, I must have missed it.

  21. Originally posted by hellspork:

    So how about that Opera 10.x vulnerability list? Oh wait, what list? Okay, now public list for Firefox 3.x? 3.5x? Well, what about IE8?

    Huh? What are you talking about?

  22. Levity, o creature of darkness.Thus far, Opera's 10.x generation has been close to bulletproof. Firefox 3.5x has had its exploits, and a fair run of emergency patches. IE8 is…well, it's another Internet explorer generation.The report only talked about protecting users from social engineering, which utterly depends on third-party "trust" lists and cannot be controlled by the browser vendor.The browser's part in maintaining security, is to prevent attacks of opportunity or hijacks of the operating system. So the report was basely false at its most fundamental level.

Comments are closed.