Evaluating Email Security Services: Methods, Challenges and Best Practices

Evaluating an email security service is crucial for organizations looking to mitigate email-borne threats. While monitoring email security solutions can be a challenging, costly and somewhat murky process, any level of testing is better than none.

With nearly 25 years in the email security industry, I’ve worked closely with organizations conducting pre-purchase testing and ongoing assessments of their security solutions. And my biggest takeaway from my time in the field is that understanding the potential flaws in each approach is essential to making informed decisions and enhancing email security. 

That’s why I’ve outlined the three primary methods of evaluating an email security service, and the key questions you need to help you land on the right vendor or strategy.

The ins and out of each testing methods 

Published tests

Published tests are conducted by organizations that compare multiple vendors and periodically release their findings. However, as email security has matured, demand for these tests has declined and now only a handful of organizations test regularly. Major email security vendors often opt out due to resource constraints, disagreements over test methodologies and disputes regarding results. Consequently, these tests feature emerging or niche companies looking to establish credibility.

Email penetration testing

Third-party email penetration testing vendors charge a fee to test your email security service. These tests involve sending a set of sample threat emails through the service and reporting on detection rates. However, many vendors rely on manufactured test articles rather than real-world threats, and are often reluctant to share samples for independent analysis, making it difficult to assess the validity of their findings.

In-house testing

Conducting internal tests allows organizations to control every aspect of the process, ensuring relevant and realistic evaluations. However, this approach can be costly, time-consuming and require specialized skills. Despite these challenges, in-house testing is beneficial for both selecting the best product and continuously monitoring its effectiveness. 

7 questions you need to consider  

Whether you’re exploring a third-party vendor or taking the testing on yourselves, it is crucial to pose or consider these seven key questions before diving in.

1. Who is funding the test?

Testing email security solutions is expensive. The most reliable tests are fully independent and financed by reputable publications. Tests where all vendors contribute a set fee can also be valuable, but it is essential to understand why certain companies may have chosen not to participate. Tests initiated by vendors themselves should be approached with skepticism, as they may manipulate configurations to produce favorable results.

2. Are vendors allowed to adjust configurations for the test?

Vendors may fine-tune their settings to enhance detection rates for a test. For example, they could increase sensitivity to capture more threats, but this may lead to unacceptable false positives in a real-world deployment scenario. Some vendors may also classify legitimate bulk emails (e.g., newsletters) as spam to inflate their success rates.

3. Can vendors challenge test results before publication?

While it’s common for vendors to review test results before publication, transparency is key. Vendors may dispute specific results due to sample sources, sample age or artificially generated test threats that don’t accurately represent real-world attacks. Understanding how vendors can influence the final report is crucial when interpreting results.

4. How old are the test samples?

The difference between an average and an excellent email security solution can be measured in seconds. Most solutions can block threats that are 24 hours old, but the best ones can heuristically detect brand-new threats. A gold-standard test would assess real-time email traffic, though this is challenging to execute fairly across multiple vendors.

5. Are test samples real-world threats or artificially generated?

Manufactured test threats, commonly used by email penetration testing companies, are often designed around specific vulnerabilities. However, these samples may lack malicious payloads and use legitimate sender domains, making them harder to convict using traditional threat detection methods. Test companies may also reuse vulnerabilities in different formats to artificially inflate failure rates for certain vendors.

6. Where do the test samples originate?

Many security vendors and testing organizations rely on third-party sources for threat samples. If a vendor happens to use the same source as the testing company, they may achieve artificially high success rates compared to competitors using different data feeds. Understanding sample sources helps assess the validity of test outcomes.

7. Does the test methodology favor one vendor over another?

Some vendors offer companies a "review" of their email security by scanning journaled email traffic (copies of scanned emails). They then present a report highlighting threats missed by the incumbent solution but detected by their product. Since false positives are ignored in these reports, the reviewing vendor may artificially increase sensitivity to boost their detection numbers. Additionally, they may classify bulk emails as threats to pad their report. This one-sided evaluation does not allow for a fair comparison of missed detections across both solutions.

Ultimately, it pays to be informed

Here’s my two cents about each testing method:

• Published tests offer broad vendor comparisons but may lack participation from leading providers.
• Email penetration testing can provide insights but may use artificial threats that do not reflect real-world risks.
• In-house testing offers the most control and accuracy but requires time and expertise. 

While no testing method is perfect, being aware of potential biases and methodological flaws will help you make informed decisions. Ultimately, you should adopt a combination of these methods and remain critical of the results to ensure your email security remains as effective as possible.

Looking for other ways to shore up your email security? Explore everything we have to offer.