Hello everyone! This video was recorded for the VMconf22 Vulnerability Management conference. I want to talk about the blind spots in the knowledge bases of Vulnerability Scanners and Vulnerability Management products.

This report was presented in Russian at Tenable Security Day 2022. The video is here.

Potential customers rarely worry about the completeness of the Knowledge Base when choosing a Vulnerability Scanner. They usually trust the VM vendors’ claims of the “largest vulnerability base” and the total number of detection plugins. But in fact the completeness is very important. All high-level vulnerability prioritization features are meaningless unless the vulnerability has been reliably detected. In this presentation, I will show the examples of blindspots in the knowledge bases of vulnerability management products, try to describe the causes and what we (as customers and the community) can do about it.

Who am I

My name is Alexander Leonov. I have been working in the field of Vulnerability Management since 2009. For the first 6 years I worked in a VM vendor and then I started working on the customer side. I have a blog avleonov.com and a telegram channel avleonovcom.

The goal of this presentation

First of all, I want to make it clear that these presentation is not targeted against any particular VM vendor. I like them all. My sole purpose is to highlight existing issues in order to improve the overall level of Vulnerability Assessment.

One of the biggest problems, in my opinion, is that Vulnerability Management solutions are often presented in marketing as a kind of magic wand that can detect every vulnerability in any system. Like it’s easy and obvious. But, of course, this is not true. It takes a lot of intensive daily work of hundreds of people to understand how the software update process works on different systems and how to correctly detect vulnerabilities in them. And treating it like a commodity, like something natural and uninteresting, devalues ​​the work of these people. I’ve been developing vulnerability detection checks for years and this hurts my feelings too.

What vulnerability detections are added first?

It is clear that when the Security Content developers of the VM vendor are working on new checks to detect vulnerabilities, they have some plan and some priorities. What will they work on first?

1. Most likely they will add the most critical and known vulnerabilities first. These vulnerabilities are the most dangerous for customers, and on the other hand, they can be effectively demonstrated in marketing. “See how quickly we added the log4shell detections”.
2. Then there will be vulnerabilities for software that is used or may be used by large customers of this VM vendor. Most likely, vulnerabilities in enterprise-level software products.
3. If for some systems, vulnerability detection checks can be generated automatically, for example, based on security bulletins, these systems will also be supported in priority.

But in any case, to achieve full coverage of all vulnerabilities is an extremely difficult task, rather even impossible. So the question is: how to assess whether the coverage is sufficient?

Comparison of Knowledge Bases based on CVE IDs

One way to evaluate the coverage of Vulnerability Management products is to compare the knowledge bases by CVE IDs. We simply take sets of CVE IDs that each of the scanners can detect, then overlap these sets and see which scanner can detect more vulnerabilities. Naturally, this approach has limitations, which I will demonstrate myself.

If the CVE is in the KB it may not be enough

• Okay, CVE exists in the knowledge base of the Vulnerability Management solution. But we do not take into account the type of detection check it is associated with. It can be authentication-based, banner-based, or exploitation-based. This is an important factor because different checks provide different quality of detection and require different development resources.
• It happens that one CVE affects many products at once. As in the case with Log4Shell. When comparing by CVE, we cannot say exactly for which products the VM solution can detect this particular CVE ID.
• In addition, we should not forget that there are vulnerabilities without a CVE ID. And there are many. If we compare only by CVE, we leave these vulnerabilities out of scope, which is also not good.

If there is no CVE, this is definitely a problem

• This may mean that the VM solution is not able to detect a vulnerability at all.
• This may mean that the VM solution can detect this vulnerability, but the CVE ID is not specified in the vulnerability description. This is not a super-critical problem, but still a problem. For example, this can affect the prioritization of vulnerabilities.

How bad is it that the VM solution is unable to detect some set of vulnerabilities?

I often hear from representatives of VM vendors: “Well, yes, we do not detect some CVEs. So what? We detect everything important. Let’s talk about the VM process and our dashboards.” Of course, for the most part it is some kind of sales technique, objection handling and NLP. But there is some truth in this. Indeed, it can be pretty hard to tell if a VM solution should detect some set of CVEs. But do VM solutions really detect everything that is needed? Let’s check it out. Let’s start with the most obvious case.

CISA Known Exploited Vulnerabilities Catalog

Last year, The Cybersecurity and Infrastructure Security Agency released the Known Exploited Vulnerabilities Catalog. It contains relatively few entries. At the time of my research, February 8, 2022, there were 352 CVEs. There are definitely exploits for them. These vulnerabilities are definitely used by attackers. There are also strict Patch Deadlines for US Federal Civilian Executive Branch (FCEB) agencies to fix these vulnerabilities. In other words, these are really critical vulnerabilities relevant to the enterprise environment. It would be logical to assume that all of them can be detected by VM solutions. Is it so?

Comparison

To make the comparison, I used my open source project VulnKBDiff. I compared OpenVAS (officially renamed as GVM with GCF – free Greenbone Community Feed), Tenable Nessus and NVD databases. For those who want to repeat the analysis, I must clarify that VulnKBDiff obtains information about Nessus plugins from Vulners.com and this requires a commercial or researcher license. I have a researcher license, thanks a lot to Vulners Team for that! The rest of the data I take directly from open sources.

On February 8, 2022, I took 352 CVEs from the CISA Known Exploited Vulnerabilities Catalog and used VulnKBDiff to see which ones are in Nessus and OpenVAS (Full report: 2022-02-09_cisa_exploited.html).

We can see that the commercial Nessus scanner can detect more vulnerabilities than the free OpenVAS. This is not surprising. Although the presence of 9 vulnerabilities that Nessus does not detect, but OpenVAS detects is interesting.

There are also 47 vulnerabilities that neither Nessus nor OpenVAS can detect. It’s about 13%. I will not analyze all the vulnerabilities, but I would like to outline some theses.

OpenVAS

Does it make sense to use OpenVAS/GVM with the free Greenbone community feed to detect vulnerabilities in enterprise infrastructure? Yes, but you need to understand that it will not detect vulnerabilities in some products, such as Cisco IOS, Cisco ASA, Atlassian Confluence, VMWare solutions. You probably remember last year there was a lot of noise about the Kaseya VSA and Accellion FTA vulnerabilities. Also just recently everyone was talking about the Microsoft Exchange “ProxyShell” RCE. Well, OpenVAS cannot detect all this. As you can read on Greenbone website: “However, the GCF receives no new NVTs for features for enterprise environments since September 4th 2017. This distinction is regarded an adequate balance between community needs and commercial needs.” Keep this in mind.

Nessus

Now let’s see what Nessus can’t detect. First, it does not know how to detect some of the vulnerabilities that OpenVAS detects. rConfig and Apache Struts can certainly be found on corporate networks.

What about vulnerabilities that both Nessus and OpenVAS cannot detect. At first I thought that there will be vulnerabilities that are usually out of scope of VM solutions. For example, vulnerabilities in smart phones. But actually I see only two vulnerabilities for Android. The rest are quite critical vulnerabilities in enterprise-level products. Although there could be no other vulnerabilities in the CISA Known Exploited Vulnerabilities Catalog.

By the way, these 2 vulnerabilities, which are in the catalog and are detected by OpenVAS and Nessus, but are not present in NVD, demonstrate a problem with NVD. Vulnerabilities now get there with a very long delay.

NVD CVEs Published in 2021

The next step I wanted to increase the scope to show the scale of the problem. I took all the CVEs published last year 2021. By the way, you can’t get them by the mask “CVE-2021-*”. The correct way to do this is using the NVD Publication Date. A total of 20131 CVE IDs were published is 2021. 3944 of them are “abnormal”. For example CVE-2002-20001, CVE-2002-2438, CVE-2007-5967, CVE-2008-2544, CVE-2008-3280, etc. It is difficult to say what the reason is, apparently decentralization and uncoordinated work of Mitre and CVE Numbering Authorities (CNAs).

Comparison

So, on February 8, 2022, I took 20131 CVEs published in 2021 and used VulnKBDiff to see which ones are in Nessus and OpenVAS KBs (full report: 2022-02-09_nvd_published2021.html). Nessus can detect more vulnerabilities than the free OpenVAS. However, even Nessus and OpenVAS together detect less than 1/3 of all NVD CVEs published in 2021. Nessus can’t detect 14606 vulnerabilities (full list: vulristics_task_cve_2021_list.txt), so I decided to take a closer look at them.

NVD CVEs 2021 Filtering Vulristics

To analyze these vulnerabilities, I used my open source project Vulristics. It allows you to prioritize vulnerabilities using different data sources. To make the report more readable, I identified 1389 (full list: vulristics_task_cve_2021_list_filtered.txt) of 14606 vulnerabilities that have a public exploit on Vulners.com and made a report only for those vulnerabilities (full report: cve2021_report_with_comments_ext_img.html 3,6 mb).

Here is how I did it:

$ python3.8 vulristics.py --report-type "cve_list" --cve-project-name "CVE2021" --cve-list-path "vulristics_task_cve_2021_list.txt" --cve-comments-path "vulristics_task_cve_2021_comment.txt" --cve-data-sources "vulners" --rewrite-flag "False"
$ grep -nr "\"public_exploit\": true" data/vulners_processed/ | awk -F":" '{print $1}' | sed 's/.json//' | sed 's|data/vulners_processed/||' > test.txt
$ python3.8 vulristics.py --report-type "cve_list" --cve-project-name "CVE2021" --cve-list-path "test.txt" --cve-comments-path "vulristics_task_cve_2021_comment.txt" --cve-data-sources "vulners" --rewrite-flag "False"

I specifically did not groom the report in terms of detecting the type of vulnerability and vulnerable product, since they are detected based on the description from NVD and this is not such an obvious process. Sorry for that. But it’s still interesting to look through it to understand the limitations of Nessus. Let’s start from the very top.

Remote Code Execution in Micro Focus Operation Bridge Reporter (OBR) an enterprise metrics solution. This vulnerability is used by Mirai botnet and there is an exploit for this vulnerability in Metasploit. And plugins for detecting this vulnerability in Nessus and in general in Tenable products are not yet available. You can check it at https://www.tenable.com/cve/<CVE_ID>.

Remote Code Execution in Apache Tapestry, an open-source framework for creating web applications in Java. There is also a public exploit on github and signs of exploitation in the wild.

Further on the slide there are no detailed proofs, but you can check everything yourself in the full report. Remote Code Executions in routers and video surveillance systems of various vendors.

What other vulnerabilities are there? For example, there are many CMS vulnerabilities, especially WordPress plugins. Why don’t Tenable detect them, if a WordPress blog is a completely normal situation for companies? It’s probably just not a priority.

In conclusion

There is a problem, we have to live with it and somehow compensate.

• Consumers of VM solutions, look at vulnerability detection more broadly, try to use several detection tools. Just because you scanned a host and don’t see vulnerabilities in the report doesn’t mean they aren’t there. And it’s not about 0days at all.
• VM vendors, let’s be friends! Let’s talk about this problem and improve vulnerability detection. Analyzing your knowledge base in the simple way that I have presented can help you better prioritize security content developers’ tasks, highlight issues, and possibly get more resources. If you work for a VM vendor, show this to the manager responsible for developing vulnerability detection checks. I am open to dialogue, ready to help you analyze your knowledge bases publicly or not. I believe that together we can make a difference.

Thank you very much for your time! Links to full reports are in my blog and avleonovcom telegram channel.