Continual feature selection: a cost effective method for enhancing the capabilities of enterprise spam solutions
Vipul Sharma, John Gardiner Myers, Steve Lewis Proofpoint
The effectiveness of content-based spam filters is directly related to the quality of the features used in the filter's
classification model. Features are the specific attributes examined by the spam filter. Highly effective filters
may employ an extremely large number of such features (on the order of hundreds of thousands), which can consume a
significant amount of both storage space and classification time. In the ongoing battle between spammers and spam
filter developers, new techniques and technologies are continually being introduced by both sides. This means that the
number and importance of the features needed to classify spam accurately is subject to continual change. A given
feature might be very important at one point in time, but become irrelevant after a few months as spam campaigns and
their associated techniques change. Discarding on a regular basis features that have become ineffective
('bad features') will benefit the spam filter with reduced classification time (reduced model training time and email
delivery time), reduced storage requirements, increased spam detection accuracy and a reduced risk of over-fitting of
the model.
In this paper we discuss and benchmark several statistical methods for feature selection in spam filtering.
We also discuss the properties of good and bad features in spam filtering. We report a significant improvement in both
the filter's performance and effectiveness.
