One of the most effective techniques available for combating spam is the widespread application of collaborative filtering, where members of a community identify spam messages. We present data and analysis of our success in applying a collaborative filter, originally designed for anti-spam, to the anti-virus problem. We also present our results from specific case studies, including an analysis of the CME-24 outbreak of early 2006. We show that not only is a collaborative filter effective for filtering viruses, but also that the community begins filtering the virus within minutes of its initial detection-and with an extremely low false positive rate.
Conventional anti-virus software relies on a staff of researchers to isolate and analyze viruses, and then write and test rules to block them. This process can take up to 24 hours and often blindly blocks many legitimate messages with attached executable code. In contrast, Cloudmark uses a fingerprinting algorithm to identify each incoming message, combined with a reputation-based, trusted community of users to accurately identify malicious viruses in real-time. Using the Cloudmark Trust Evaluation System™ (TES), Cloudmark is able to corroborate reports from the Cloudmark Global Threat Network. Cloudmark's virus fingerprinting algorithm automates the time-intensive "reverse engineering" analysis of conventional technologies allowing its system to identify and stop new worms and virus strains in zero-hours.
In this paper, we describe the collaborative, reputation-based approach to fighting spam that has been developed by Cloudmark. Two core components of our techonlogy are described in depth, namely the Global Threat Network and the Trust Evaluation System. The Global Threat Network rapidly gathers and correlates reports of suspect messages from millions of globally-distributed honeypots, customers, and various other sensors. The Trust Evaluation System (TES) analyzes the reports, assigns a reputation to the suspect messages based upon the reputation of the reporters, and then determines a new reputation for each reporter based upon their historical performance at correctly identifying unwanted content. Because the reputation-based collaborative system does not draw blanket conclusions about terms, hosts, or people, this approach reduces false positives and false criticals, while simultaneously decreasing administration costs.
The Cloudmark Global Threat Network contains a vast number of honeypots and human reporters who are themselves targets of phishing attacks. The Trust Evaluation System automtically identifies and only listens to those reporters who have an excellent “phishing recognition” track record, thereby minimizing the possibility of false positives. In addition to this network of trusted users, Cloudmark employs a message fingerprinting algorithm to identify each incoming message in its system. Once enough users confirm a message as an attack, any message with a matching fingerprint is moved into every user's spam folder. A single fingerprint is enough to match all messages generated from the same phisher – even polymorphic mutations of the same messaging attack – thus allowing Cloudmark to provide immediate protection against new and emerging phishing attacks.
Cloudmark Network Feedback System™ (CNFS) enables Cloudmark ISP and enterprise customers to participate in the Cloudmark Global Threat Network, improving message filtering accuracy by automatically submitting feedback on misclassified messages identified by honeypots and other reporters. The CNFS Reporting Interface offers a suite of reports that provide system administrators with statistics and in depth information on the feedback customers have submitted. By examining the feedback reporting trends, system administrators can proactively address accuracy issues.