My thoughts on the “2021 Gartner Market Guide for Vulnerability Assessment”. What about the quality? The Gartner Vulnerability Management Reports are one of the few marketing reports that I try to read regularly. This started back in the days when I was working for a VM vendor doing competitive analysis. Gartner is one of the few organizations that think about Vulnerability Assessment and Vulnerability Management and clearly articulate where we are and where we are going.

I got a free reprint of “2021 Gartner Market Guide for Vulnerability Assessment” from the Tenable website. Thanks a lot to them for that.

Let’s start with what I liked:

1. It’s great that Gartner has made vulnerability prioritization technology (VPT) a separate class of solutions, that do not detect vulnerabilities themselves, but work with them. For example, Kenna or my Vulristics. And it could be additional functionality like Tenable VPR.
2. I liked the focus on EDR as a promising VM replacement. Especially, Microsoft solutions (Defender for Endpoint or as was mentioned in the report Microsoft’s Threat & Vulnerability Management, TVM).
3. It’s nice that various areas related to Vulnerability Management have been mentioned: Pentest, Bug Bounty, Breach and Attack Simulation (BAS).
4. An interesting diagram that shows that VA is primarily about “Assess” and “Asset Management”, VPT is primarily about “Prioritize” and “Workflow Management”, BAS is primarily about “Compensate” and “Security Controls”.

Now what I didn’t like. I have one pain point – the quality of the scanning. And here, on the one hand, something was said, but on the other, it was not enough and not as definite as I would like. Market Direction is the most interesting section of the document. And it was the most painful to read.

They write the following:

“Vulnerability assessment against common platforms/major operating systems, network devices and popular third-party applications is universally covered in the market, with only minute differences between solutions in terms of scope and coverage from the leading vendors. Differentiating solutions based on these criteria is seldom possible. Vendors can be difficult to differentiate based on scanning accuracy and performance alone.”

A minute difference? And who checked it? It seems to me that no one did it. Even a simple CVE comparison shows a big difference in the knowledge base of scanners, especially for “popular third-party applications”. But nobody cares about that.

Immediately further in the report, you can read:

“Gartner sees competition increasingly based on pricing, rather than features, along with the addition of scanning other asset types such as the cloud, containers, OT and IoT.”

You see, there is no interest in quality. It is easier to show more lines in the list of systems with which these products can work. And they don’t work very well with them either. And this is a sad truth.

Further in the report, Gartner states that it will never be better:

“Gaps in coverage — for example, for less-common technologies or third-party applications — will persist, because they are difficult to convert into new sales and are not widely deployed by clients. Developing and maintaining these capabilities also requires the same R&D overhead as more-common technologies to perform assessments.”

And then in separate sections you can read where the situation is worst:

“In-depth assessments of databases and applications, such as ERP systems (e.g., SAP or Oracle), are not widely supported in traditional VA solutions. Most VA vendors provide some OT security capabilities. But for specific mature use cases, dedicated OT VA vendors should be evaluated.”

“Coverage for OT assets and technologies — such as supervisory control and data acquisition (SCADA) or industrial control system (ICS) devices — is lacking in the wider VA market. Many VA vendors claim SCADA or ICS support for their solutions. OT requires careful consideration, since its balance of “business risk” and “security risk” is vastly different from IT”.

Well, it’s good that they write this directectly. It’s bad that they don’t develop this idea further. Because otherwise, you have to say that all VA marketing is now based on the belief that sufficient scan quality has already been achieved and there is no need to even talk about it. And the reality is that vulnerability detection is often implemented primitively and with a bunch of assumptions.

A simple example. Yes, vulnerability scanners can detect Linux vulnerabilities in software packages installed from the official repository. How about custom packages? How about software installed from source? “Well, you know, this is out of scope”. But why? Just because it’s not cool enough? And this is just one example.

But instead of this you hear the beautiful words about the miracles that VA/VM vendors can do using the raw vulnerability data: VPT, BAS, whatever. But the fact is that it uses at least incomplete data at the input.

I think it would be much better to see the real limitations of Vulnerability Assessment tools. Ok, the VA product found something during the scan. But it would be great if this product showed which systems it cannot process adequately.

End users now have to compensate for core functionality limitations with multiple VA tools and home-grown automation. And it seems that only end-users are interested in the quality of these core features. Perhaps the developers too. But not the managers. Because it is much easier for them to declare support for yet another type of system. Even if all this support is purely formal.

I want to end with a quote from Recommendations for “Security and risk management leaders responsible for security operations”. They should:

“Evaluate VA solutions’ capabilities for assessment coverage and depth and support of stand-alone product integrations to fill in the gaps across the vulnerability life cycle, and to assist in remediation automation”.

And also

“Select a VA tool with comprehensive coverage. Most VA tools are strongest in identifying and scanning vulnerabilities. Select a vendor that is able to align with your organization’s computing architecture to provide wide support for your IT assets (in terms of numbers of classes of assets, such as endpoints, servers, storage, networking, mobile and security)”.

It is not clear how to do this, because no marketing materials can be trusted. But at least try to do this until you spend money on the product.