Now that the MITRE ATT&CK Engenuity results for 2021 have been released, participating vendors will be publishing their results. However, it can be hard to understand how well a vendor has actually performed as MITRE do not produce scores and rankings. Vendors tend to form their own results by only sharing specific results and shaping the messaging behind the results. In this blog post we have put together some of the key results to help you cut through all the noise.
What are MITRE Engenuity ATT&CK Evaluations?
MITRE Engenuity evaluates cyber security products using an open methodology based on the ATT&CK knowledge base. It is great way to get visibility into different vendors, which otherwise can be confusing due to the power of marketing.
Goals of the evaluation:
Empower end-users with objective insights into how to use specific commercial security products to detect known adversary behaviours.
Provide transparency around the true capabilities of security products and services to detect known adversary behaviours.
Drive the security vendor community to enhance their capability to detect known adversary behaviours.
This year’s evaluation had the largest number of participants:
BlackBerry Cylance, Broadcom, Check Point, Cisco, CrowdStrike, Cybereason, CyCraft, Cynet, Elastic, ESET, F-Secure, Fidelis, FireEye, Fortinet, GoSecure, Malwarebytes, McAfee, Micro Focus, Microsoft, OpenText, Palo Alto Networks, ReaQta, SentinelOne, Sophos, Trend Micro, Uptycs, and VMware.
The ATT&CK Evaluations team chose to emulate Carbanak and FIN7 because they both target a wide range of industries for financial gain, whereas prior emulated groups were more focused on espionage. Previously evaluations were on APT3, Gothic Panda (2019) and APT 29, Cozy Bear (2020). Take a look at the previous results here.
Who performed best?
As MITRE ATT&CK do not publish rankings it can be difficult to determine which vendor has performed the best. Be mindful that all vendors will put their own twist on the results so make sure you check where they are getting their facts from.
To see the full results of all 29 vendors and see where the statistics come from, take a look at the results here.
This table shows the 15 vendors who had the largest number of detections and you can see overall these vendors all performed well across the board.
Although a high number of detections is good, it also depends on the type of detections.
Here are the different type of detections:
Techniques - Gives the analyst information on how the action was performed or helps answer the question "what was done"
Tactics - Gives the analyst information on the potential intent of the activity or helps answer the question "why this would be done".
General - Processed data specifying that malicious/abnormal event(s) occurred, with relation to the behaviour under test. No or limited details are provided as to why the action was performed for how the action was performed.
Telemetry - Minimally processed data collected by the capability showing that event(s) occurred specific to the behaviour under test that satisfies the assigned detection criteria.
None and not applicable
Configuration change and delayed - Configuration change indicates when a vendor change their configuration in the middle of the test. Delayed indicates when a detection was not immediately available to the test proctors due to some delay in processing.
Some consider detection types such as Techniques and Tactics to be the most valuable.
Vendor Detection Types Results:
This graph shows the different number of detection types of the vendors.
Vendors such as SentinelOne overall performed well in terms of high number of analytic detections compared to Telemetry and General detections. Whereas other vendors like BlackBerry Cylance and CyCraft have a larger number of Telemetry and General detections.
Linux Technique Detection Results:
This year MITRE introduced a detection evaluation for the Linux operating system, normally it has only been for Windows environments. 12 substeps were applied Linux environments, some vendors did not have an agent deployed to the Linux environment. Those vendors were Cisco, AhnLab, F-Secure, GoSecure, ESET, Sophos, Fortinet, OpenText EnCase and Malwarebytes.
In this graph you can see of the top 15 vendors, 8 were able to provide 100% technique detection on Linux environments.
Visibility of Substeps:
Here you can see the standout vendor having detected all substeps being SentinelOne. Being able to detect all the substeps is obviously something you would look for in a vendor. The other vendors all have similar scores.
What can you do with these results?
These results can be a great starting point for evaluating vendors as it gives you full visibility of how their products perform in a real life scenario.
There is a lot more to consider when choosing the right vendor, especially what works best for your organisation and other factors such as Gartner Peer Reviews and Magic Quadrants. The best way to understand how well a solution might fit your business is a demo.
We are proud of the performance of our partners. Here is how some of the vendors responded:
"Cynet’s MITRE Engenuity ATT&CK Evaluation results confirm the powerful protection capabilities of the Cynet XDR platform, including broad visibility and fast detection and prevention. Beyond the protection capabilities demonstrated in the MITRE ATT&CK evaluation, Cynet provides the most extensive set of Response Automation capabilities to fully automate investigation and response actions, including automated root cause analysis and attack scope, followed by extended threat remediation actions across the entire environment. " - Find out more about Cynet.
“MITRE Engenuity ATT&CK Evaluations continues its stellar record in pushing the security industry forward and brings much-needed visibility and independent testing to the EDR space as practitioners sort through a complex threat and vendor landscape. Participating in all the evaluations has become an essential practice that we have used to improve our products further. At SentinelOne, we continue to be enthusiastic supporters for the work MITRE Engenuity is doing to painstakingly define and continually expand a common cybersecurity language that describes how adversaries operate.” - Find out more about SentinelOne
"In the MITRE ATT&CK round 3 evaluation, Cortex XDR delivered 100% threat protection and 97%+ detection visibility. The MITRE ATT&CK evaluations test the detection capabilities of leading security solutions by emulating the real-world attack sequences of the world’s most sophisticated advanced persistent threat (APT) groups." - Find out more about Palo Alto Cortex XDR
Want to book a Cynet or SentinelOne demo?
To book a demo of SentinelOne, click here.
To book a demo of Cynet, click here.
Talk to us about the results.
We can help you figure out what the MITRE ATT&CK evaluations results could mean for your organisation. Contact us here.
