Antivirus Comparison: Sophos Wins, But Detects Less Than Half Of New Attacks

Cloudmark recently conducted a study of the effectiveness of different antivirus (AV) products in detecting malware distributed via email spam. We found that even the best performing AV product in our study, Sophos, only detected 48% of all the samples that were eventually flagged as malicious, and that many big names in the AV industry did substantially worse. Cloudmark receives several million spam reports a day from spam traps and trusted users. Between May 28th and August 12th of 2015 we evaluated all suspicious file attachments from this data. We defined suspicious as executable or html files, or compressed files containing executables or html. We did not look at PDFs, or MS Office documents. Though these sometimes contain malicious content they have too many legitimate uses to be automatically regarded as suspicious. We calculated a hash from the file and checked to see if we had already seen it. If not then we submitted it to VirusTotal.com (VT) for analysis. VT is an invaluable service that runs submitted samples against fifty or more different AV programs and tells you which of them flag each sample as malicious. We used the VT API, which queues requests and may take an hour or more before the scans are performed. We recorded the results from VT, and if none of the AV engines flagged it as malicious, we requested a rescan twelve hours later. We submitted a total of 33,688 suspicious files and 93% of those were flagged as malicious on either the first or second scan. However, not all of the AV engines caught all of the malware. The chart below shows the percentage of confirmed malware detected on the first scan by each of the AV engines in VT's system. Click on the image for an enlarged version.
AV Comparison
As you can see, Sophos was the most effective, detecting 48.35% of new threats with ESET-NOD32 close behind with 47.60%. Some other big names in the AV industry did not fare so well. McAfee caught 36%, Symantec caught 22%, Kaspersky caught 21%, and Microsoft caught just 13%. Based on this sample Sophos and ESET-NOD32 would appear to provide the best first line of defense against malware distributed via spam. However, we should add some caveats:
  • Malware samples were tested when first received by Cloudmark. We would expect detection rates to be significantly better once the AV companies have had time to produce signatures for each new variant.
  • Static file analysis is only the first step in protection against malware. A good AV program will also detect malicious behavior such as updating registry keys or system DLLs which provides a second line of defense. This was not tested in our study.
  • Detection rates may change based on the particular attacks that are current at any given time. We will continue to collect this data and may update this study in future.
  • Malware may also be delivered via other means: watering hole attacks (malicious or compromised web sites), malvertising (malicious web advertising), etc. These were not addressed in our study.
Having an AV engine with an up-to-date signature set installed on your computer is still an important part of maintaining security, but it is not a complete solution. Don't assume that it will protect your enterprise in all cases, or even most cases. It should be combined with good spam filtering, applying all software and firmware updates when they are released, network monitoring, DNS monitoring, and employee training as part of a complete security solution.