The 2022 MITRE ATT&CK Engenuity results have just been released and vendors are now sharing how they performed. In this blog post, we help you to cut through the marketing noise and read about the key statistics from the tests.

What are MITRE Engenuity ATT&CK Evaluations?

MITRE Engenuity evaluates cyber security products using an open methodology based on the ATT&CK knowledge base. It is a great way to get visibility into different vendors, and how they actually perform against real-world scenarios.

Goals of the evaluation:

• Empower end-users with objective insights into how to use specific commercial security products to detect known adversary behaviours.
• Provide transparency around the true capabilities of security products and services to detect known adversary behaviours.
• Drive the security vendor community to enhance their capability to detect known adversary behaviours.

What is tested?

This year’s MITRE Engenuity ATT&CK evaluations emulated two threat groups, Wizard Spider and Sandworm.

• Wizard Spider is a financially motivated criminal group that has been conducting ransomware campaigns since at least August 2018 against a variety of organisations, ranging from major corporations to hospitals.
• Sandworm Team is a destructive Russian threat group that has been attributed to Russian GRU Unit 74455 by the US Department of Justice and the UK National Cyber Security Centre. Sandworm Team's most notable attacks include the 2015 and 2016 targeting of Ukrainian electrical companies and 2017's NotPetya attacks. Sandworm Team has been active since at least 2009.

Both threat groups abuse Data Encrypted for Impact.

Who participated?

Similar to 2021, this year was another with a high number of participants. In total 30 vendors participated in the evaluations including SentinelOne, Crowdstrike, Microsoft, Palo Alto and McAfee.

‍

Who performed best?

As MITRE ATT&CK do not publish rankings it can be difficult to determine which vendor has performed the best. Be mindful that all vendors will put their own twist on the results so make sure you check where they are getting their facts from.

To see the full results of all 30 vendors and see where the statistics come from, take a look at the results here.

Below is a summary of some of the key results from 20 of the key vendors who participated.

AhnLab, ESET, Deep Instinct and Fortinet did not have a Linux agent deployed so only 14 attack steps, 90 substeps and 8 protection tests are relevant to these vendors.

Visibility:

One of the most important statistics to review from the results is visibility. Most of the vendors were able to detect all 19 attack steps tested.

The graph below goes into more detail and shows the number of substep detections, in total, there were 109 substeps (90 for vendors without a Linux agent).

‍

‍

Protection:

Perhaps the most important statistics to review is how the vendors performed against the 9 protection tests. The majority of vendors were able to protect against all 9 tests. Carbon Black, McAfee, Deep Instinct, Fortinet, Sophos, AhnLab, ESET and CyCraft opted out of participating in the Linux protection test.

‍

Detection by type:

This part of the results shows how each vendor performed on different detection types. The detections are put into two different categories: Telemetry or Analytic. Analytic detections are the most valuable to have as these produce a detailed view of what took place, why, and how. Whereas telemetry detections produce minimally processed data.

‍

Configuration changes:

Another important statistic to consider is the number of configuration changes each vendor had. MITRE Engenuity gives participants is the ability to change configuration settings in a security product once an evaluation has already begun, this may be done to show additional data can be collected and/or processed. Ideally, vendors will have as few configuration changes as possible.

‍

Delayed detections:

Another important consideration is detection speed. MITRE also evaluate if vendors had any delayed detections. Time to detect is a crucial factor in responding to a threat, therefore any delayed detections mean a slower response and greater risk to your organisation. The graph below shows the vendors who had any delayed detections.

‍

What can you do with these results?

These results can be a great starting point for evaluating vendors as it gives you full visibility of how their products perform in a real-life scenario.

Although MITRE does not publish an overall "winner" or performance ranking, it is easy to tell which vendors performed the best. In previous years there have been stand-out vendors however, this year has been mixed, as a significant number of solutions have similar results. Some of the most notable vendors this year have been SentinelOne, Cybereason, Palo Alto Networks and Cynet.

There is a lot more to consider when choosing the right vendor, especially what works best for your organisation and other factors such as Gartner Peer Reviews and Magic Quadrants. The best way to understand how well a solution might fit your business is a demo.

We are proud of the performance of our partners. Here is how some of our partners responded to the results:

The results from all four years of the ATT&CK Evaluations highlight how the SentinelOne solution maps directly to the ATT&CK framework to deliver unparalleled detection of advanced threat actor Tactics, Techniques, and Procedures (TTPs). Organisations can immediately benefit from exceptional protection and detection capabilities and autonomous and one-click response options to stop and contain the most advanced cyberattacks. - Find out more about SentinelOne

Cynet performed strongly in this year’s MITRE ATT&CK Evaluation, outperforming the majority of vendors in several key areas. Cynet provided 100% visibility and detection across each of the 19 MITRE ATT&CK steps evaluated. That is, Cynet was able to detect every one of the 19 unique attack steps. - Find out more about Cynet.

Want to book a demo?

To book a demo of SentinelOne, click here.

To book a demo of Cynet, click here.

Talk to us about the results.

We can help you figure out what the MITRE ATT&CK evaluations results could mean for your organisation. Contact us here.

‍