Anti Spam

The Evolution Of Antispam Technology

Unsolicited Commercial email (or UCE or “spam”) has been a growing problem for the Internet since the mid 1990s. At that time, the Internet was rapidly becoming commercialized as well as accessible to consumers. Initially perceived merely as a nuisance created by a handful of unethical advertisers, spam currently accounts for the majority of email traffic over the Internet. Spam has a negative impact on the ability of businesses and consumers to benefit from the use of email technology and the Internet, and it poses a serious and growing threat to the reliability, efficiency, and security of corporate electronic messaging systems and the Internet.

Spam not caused by technology

Most of the discussions about spam focus on technology issues. However, the fundamental factors driving the increase in spam are economic. The technology of Internet email is such that the recipient and intermediate service providers pay the vast majority of the cost involved in delivering spam messages.

Analyst firm Ferris Research has estimated that, in 2004, spam cost U.S. companies over $10 billion per year. At the same time, spam represents the least expensive way of advertising to literally millions of businesses and consumers. The cost to spammers of sending millions of email messages over the Internet can be quickly recovered through a small number of sales, thus even if the response rate for spam is extremely low compared with legitimate advertising media, (e.g., 100 sales in response to 10 million email messages), it remains profitable to send spam.

As long as the economics of spam hold true, and as long as it is technologically possible to advertise through spam without being held accountable for the true costs, the volume of spam will certainly continue its increase.

Messaging industry slow to respond

By the late 1990s, spam had become a major problem for Internet Service Providers (ISPs) and businesses. ISPs and enterprises were forced to take steps to stop spam from overwhelming their email servers and flooding computer networks.

Although the problem was widely recognized in the messaging industry, there were relatively few anti-spam tools and technologies. The messaging industry, comprising vendors of email infrastructure software and related products, as well as standards bodies, was slow to respond to the spam problem–having initially underestimated the scale and technical complexity of the spam problem.

In particular, standards bodies such as the Electronic Messaging Association (now the OpenGroup Messaging Forum) and the Internet Engineering Task Forces (IETF) failed to effectively address the spam problem, although the IETF did eventually create the Anti-spam Research Group (ASRG) in 2003.

The evolution of anti-spam technology (continued)

At the same time, spammers rapidly multiplied, became far more sophisticated, and began to operate through quasi-legitimate ISPs and to move operations offshore (substantially as a result of legal risks made apparent through high profile law suits such as CompuServe Incorporated vs. Cyber Promotions Inc. and Sanford Wallace in 1997).

In the mid 1990s, the methods used to slow the flow of spam were based upon crude technological capabilities, such as restricting the “mail relay” feature of Internet email servers, technology fixes that were not designed specifically to address the spam problem.

In the late 1990s, the first true anti-spam technologies emerged, such as the unsolicited bulk email filter built into the Netscape Messaging Server product. However, most messaging infrastructure products did not have such capabilities and, by 2000, it was clear that businesses would have to play a role controlling spam in order to protect their own server networks.

It was also clear at that time that the existing capabilities of messaging infrastructure software and associated products, such as email anti-virus gateways, were inadequate and that that activities of industry standards bodies were not keeping up with the problem. This situation, combined with the fact that U.S. court cases related to spam, such as the 1997 court case mentioned above, had no apparent deterrent effect on spammers, presented a potential business opportunity for anti-spam technologies.

Precursors of anti-spam technology

Virtually every Simple Mail Transfer Protocol (SMTP) Message Transfer Agent (MTA) has some native ability to control communications with other MTAs over the Internet. For example, sendmail, the most widely used SMTP MTA in the world, has the ability to disallow relaying and to limit the networks from which it accepts messages.

These features, referred to below as precursors and first-generation technologies, were not created specifically to address the spam problem, nor do they constitute an anti-spam technology. In the past few years, messaging product vendors seeking to address the spam problem have implemented rudimentary capabilities like these and crude features such as simple “white lists” and “black lists” into basic anti-spam feature sets. The latter approach, which is still evident in many products that include SMTP MTAs such as IBM Lotus Domino, does not truly represent an anti-spam technology.

True anti-spam technologies, referred to below as second and third generation technologies, are a relatively new development that followed the advent of spam in the mid 1990s. Anti-spam technologies are entirely new and are sharply differentiated from basic MTA controls. In particular, Bayesian algorithms and more recent advances in text analysis, for example, using Artificial Intelligence, are a radical departure from pre-existing mechanisms characterized or repurposed as anti-spam tools.