Parsing Nessus v2 XML reports with python

Previous post about Nessus v2 reports I was writing mainly about the format itself. Now let’s see how you can parse them with Python.

Please don’t work with XML documents the same way you process text files. I adore bash scripting and awk, but that’s an awful idea to use it for XML parsing. In Python you can do it much easier and the script will work much faster. I will use lxml library for this.

So, let’s assume that we have Nessus xml report. We could get it using Nessus API or SecurityCenter API. First of all, we need to read content of the file.

#!/usr/bin/python

f = open('scanreport.nessus', 'r')
xml_content = f.read()
f.close()

Now I want to make a dict of vulnerabilities from this xml file. The key of this dict will have structure “host|plugin_id|port”. So, vulnerabilities["host|plugin_id|port"] will return me a dict with all parameters of vulnerability (Nessus plugin). Moreover, I want to see there not only information about particular plugin, but also information about the host: os, network interfaces, MAC, scan configuration. It won’t be  an optimal way of storing the scan data, but it will make processing much easier, because you will see the context of any vulnerability.

If we look at the Nessus XML report structure we see that actual results are in Report section:

And in Report section will be ReportHost blocks containing some information about the host in HostProperties and some information about vulnerabilities in several ReportItem blocks.

So, I should get to the ReportItem, read all the data I need to produce a key, initialize vulnerability structure with this key and add all data from the HostProperties. Here is the code:


from lxml import etree

vulnerabilities = dict()
root = etree.fromstring(xml_content)
for block in root:
    if block.tag == "Report":
        for report_host in block:
            host_properties_dict = dict()
            for report_item in report_host:
                if report_item.tag == "HostProperties":
                    for host_properties in report_item:
                        host_properties_dict[host_properties.attrib['name']] = host_properties.text
            for report_item in report_host:
                if 'pluginName' in report_item.attrib:
                    vulner_id = report_host.attrib['name'] + "|" + report_item.attrib['pluginID'] + "|" + report_item.attrib['port']
                    vulnerabilities[vulner_id] = dict()
                    vulnerabilities[vulner_id]['port'] =  report_item.attrib['port']
                    vulnerabilities[vulner_id]['pluginName'] =  report_item.attrib['pluginName']  
                    vulnerabilities[vulner_id]['pluginFamily'] =  report_item.attrib['pluginFamily']                                
                    vulnerabilities[vulner_id]['pluginID'] =  report_item.attrib['pluginID']
                    for param in report_item:
                        if param.tag == "risk_factor":
                            risk_factor = param.text
                            vulnerabilities[vulner_id]['host'] =  report_host.attrib['name']
                            vulnerabilities[vulner_id]['riskFactor'] =  risk_factor
                        else:
                            vulnerabilities[vulner_id][param.tag] = param.text
                    for param in host_properties_dict:
                        vulnerabilities[vulner_id][param] = host_properties_dict[param]

As you can see in the code, I get the root of XML document, then I check it’s child blocks in cycle. I can read block tag name (tag), attributes (attrib dict) and text in the block. If we are in block with tag name “Report” I make two new cycles: first one to initialize the host_properties_dict with information about the host, the second one to produce the key (vulner_id), to add information about each plugin to the vulnerability dictionary and copy parameters from host_properties_dict to the vulnerability dictionary.

To see the results in pretty print form:

import pprint                   
ids = vulnerabilities.keys()
pp = pprint.PrettyPrinter(indent=4)
pp.pprint(vulnerabilities["117.121.13.14|73570|445"])

Output:

{   'Credentialed_Scan': 'true',
    'HOST_END': 'Thu Dec 29 12:13:17 2016',
    'HOST_START': 'Thu Dec 29 12:03:53 2016',
    'LastAuthenticatedResults': '1483002797',
    'bid': '66920',
    'bios-uuid': '155C0A00-5BCB-11D9-8E9C-5404A6BFD7AB',
    'cpe': 'cpe:/o:microsoft:windows',
    'cpe-0': 'cpe:/o:microsoft:windows_8_1::gold',
    'cpe-1': 'cpe:/a:wireshark:wireshark:1.12.6 -> Wireshark 1.12.6',
    'cpe-10': 'cpe:/a:oracle:jre:1.8.0:update112',
    'cpe-11': 'cpe:/a:oracle:jre:1.8.0:update112',
    'cpe-12': 'cpe:/a:videolan:vlc_media_player:2.2.4',
    'cpe-2': 'cpe:/a:adobe:acrobat_reader:11.0.18.21',
    'cpe-3': 'cpe:/a:adobe:adobe_air:23.0.0',
    'cpe-4': 'cpe:/a:adobe:flash_player:24.0.0.186',
    'cpe-5': 'cpe:/a:adobe:flash_player:23.0.0.205',
    'cpe-6': 'cpe:/a:opera:opera_browser:42.0',
    'cpe-7': 'cpe:/a:microsoft:ie:11.0.9600.18538',
    'cpe-8': 'cpe:/a:mozilla:firefox:50.1.0',
    'cpe-9': 'cpe:/a:oracle:jre:1.7.0:update51',
    'cve': 'CVE-2014-2428',
    'cvss_base_score': '10.0',
    'cvss_temporal_score': '9.0',
    'cvss_temporal_vector': 'CVSS2#E:POC/RL:U/RC:ND',
    'cvss_vector': 'CVSS2#AV:N/AC:L/Au:N/C:C/I:C/A:C',
    'description': 'The version of Oracle (formerly Sun) Java SE or Java for Business installed on the remote host is earlier than 8 Update 5, 7 Update 55, 6 Update 75, or 5 Update 65.  It is, therefore, potentially affected by security issues in the following components :\n\n  - 2D\n  - AWT\n  - Deployment\n  - Hotspot\n  - JAX-WS\n  - JAXB\n  - JAXP\n  - JNDI\n  - JavaFX\n  - Javadoc\n  - Libraries\n  - Scripting\n  - Security\n  - Sound',
    'exploit_available': 'true',
    'exploitability_ease': 'Exploits are available',
    'fname': 'oracle_java_cpu_apr_2014.nasl',
    'host': '117.121.13.14',
    'host-fqdn': 'user3273c3.corporation.com',
    'host-ip': '117.121.13.14',
    'hostname': 'USER3273C3',
    'local-checks-proto': 'smb',
    'mac-address': '54:15:A6:BF:18:AB',
    'netbios-name': 'USER3273C3',
    'netstat-established-tcp4-0': '117.121.13.14:135-117.121.13.11:37440',
...
    'netstat-established-tcp4-20': '127.0.0.1:61179-127.0.0.1:8191',
    'netstat-listen-tcp4-0': '0.0.0.0:135',
...
    'netstat-listen-tcp6-38': '[::1]:30523',
    'netstat-listen-udp4-39': '0.0.0.0:123',
...
    'netstat-listen-udp6-94': '[::]:15000',
    'operating-system': 'Microsoft Windows 8.1 Pro',
    'os': 'windows',
    'osvdb': '105899',
    'patch-summary-cve-num-6b52ad5d58bdbf4d9ae49271ad3ae15f': '30',
    'patch-summary-cve-num-6baf8f308323d224984bd832e424c820': '202',
    'patch-summary-cve-num-7b052ee02101353b71c18c498e71a2b7': '18',
    'patch-summary-cve-num-c78dfc9faff6e53e76ec9d3b3fa9a0d2': '26',
    'patch-summary-cve-num-eb2933669984591a7b9aa0b30d7ed02b': '43',
    'patch-summary-cve-num-fec1888dda0cda68ca7a95076d7f0cde': '30',
    'patch-summary-cves-6b52ad5d58bdbf4d9ae49271ad3ae15f': 'CVE-2016-0602, ... CVE-2010-5298',
    'patch-summary-cves-6baf8f308323d224984bd832e424c820': 'CVE-2016-5597, ... CVE-2015-7830',
    'patch-summary-cves-fec1888dda0cda68ca7a95076d7f0cde': 'CVE-2015-8104, ... CVE-2010-5298',
    'patch-summary-total-cves': '333',
    'patch-summary-txt-6b52ad5d58bdbf4d9ae49271ad3ae15f': 'Oracle VM VirtualBox < 4.3.36 / 5.0.14 Multiple Vulnerabilities (January 2016 CPU): Upgrade to Oracle VM VirtualBox version 4.3.36 / 5.0.14 or later as referenced in the January 2016 Oracle Critical Patch Update advisory.',
    'patch-summary-txt-6baf8f308323d224984bd832e424c820': 'Oracle Java SE Multiple Vulnerabilities (October 2016 CPU): Upgrade to Oracle JDK / JRE 8 Update 111 / 7 Update 121 / 6 Update 131 or later. If necessary, remove any affected versions.\n\nNote that an Extended Support contract with Oracle is needed to obtain JDK / JRE 6 Update 95 or later.',
    'patch-summary-txt-7b052ee02101353b71c18c498e71a2b7': 'WinSCP 5.x < 5.5.5 Multiple Vulnerabilities: Upgrade to WinSCP version 5.5.5 or later.',
    'patch-summary-txt-c78dfc9faff6e53e76ec9d3b3fa9a0d2': 'Adobe Flash Player <= 23.0.0.207 Multiple Vulnerabilities (APSB16-39): Upgrade to Adobe Flash Player version 24.0.0.186 or later.',
    'patch-summary-txt-eb2933669984591a7b9aa0b30d7ed02b': 'Wireshark 1.12.x < 1.12.13 Multiple DoS: Upgrade to Wireshark version 1.12.13 or later.',
    'patch-summary-txt-fec1888dda0cda68ca7a95076d7f0cde': 'Oracle VM VirtualBox < 4.0.36 / 4.1.44 / 4.2.36 / 4.3.34 / 5.0.10 Multiple Vulnerabilities (January 2016 CPU): Upgrade to Oracle VM VirtualBox version 4.0.36 / 4.1.44 / 4.2.36 / 4.3.34 / 5.0.10 or later as referenced in the January 2016 Oracle Critical Patch Update advisory.',
    'patch_publication_date': '2014/04/15',
    'pluginFamily': 'Windows',
    'pluginID': '73570',
    'pluginName': 'Oracle Java SE Multiple Vulnerabilities (April 2014 CPU)',
    'plugin_modification_date': '2016/05/20',
    'plugin_name': 'Oracle Java SE Multiple Vulnerabilities (April 2014 CPU)',
    'plugin_output': '\nThe following vulnerable instance of Java is installed on the\nremote host :\n\n  Path              : C:\\Program Files\\Java\\jdk1.7.0_51\\jre\n  Installed version : 1.7.0_51\n  Fixed version     : 1.5.0_65 / 1.6.0_75 / 1.7.0_55 / 1.8.0_5\n',
    'plugin_publication_date': '2014/04/16',
    'plugin_type': 'local',
    'policy-used': 'CorporationCred',
    'port': '445',
    'riskFactor': 'Critical',
    'script_version': '$Revision: 1.13 $',
    'see_also': 'http://www.nessus.org/u?1e3ee66a\nhttp://www.nessus.org/u?e09f916a\nhttp://www.nessus.org/u?6de19bd1\nhttp://www.nessus.org/u?726f7054\nhttp://www.nessus.org/u?6086d976',
    'smb-login-used': 'corporation\\scan-windows',
    'solution': 'Update to JDK / JRE 8 Update 5, 7 Update 55, 6 Update 75, or 5 Update 65 or later and, if necessary, remove any affected versions.\n\nNote that an Extended Support contract with Oracle is needed to obtain JDK / JRE 5 Update 65 or later or 6 Update 75 or later.',
    'synopsis': 'The remote Windows host contains a programming platform that is potentially affected by multiple vulnerabilities.',
    'system-type': 'general-purpose',
    'traceroute-hop-0': '117.121.13.2',
    'traceroute-hop-1': '117.121.252.74',
    'traceroute-hop-2': '?',
    'vuln_publication_date': '2014/04/15',
    'xref': 'OSVDB:105899'}

It is a typical Java vulnerability. All important plugin data is in bold here, including CVE references CVSS (base and temporal) and actual plugin output. You can see here host information: OS type and version, CPEs, patches, vulnerabilities, network configuration. You can also see that scan was authenticated, what policy (CorporationCred), transport (smb) and credentials (scan-windows) were used.

What’s next? You can make JSON and easily export this vulnerability structure to Splunk SIEM (“Export anything to Splunk with HTTP Event Collector“). Or you can process it by your own python scripts. For example, to find all critical vulnerabilities with public exploits and Network access vector:

for vulner_id in vulnerabilities:
    if "riskFactor" in vulnerabilities[vulner_id].keys() and "cvss_vector" in vulnerabilities[vulner_id].keys() and "exploit_available" in vulnerabilities[vulner_id].keys():
        if vulnerabilities[vulner_id]["riskFactor"] == "Critical" and "AV:N" in vulnerabilities[vulner_id]["cvss_vector"] and  vulnerabilities[vulner_id]["exploit_available"] == "true":
            print(vulner_id + " " + vulnerabilities[vulner_id]["plugin_name"])

Output:

117.121.11.51|74011|445 Adobe Acrobat < 10.1.10 / 11.0.07 Multiple Vulnerabilities (APSB14-15)
117.121.31.33|84824|445 Oracle Java SE Multiple Vulnerabilities (July 2015 CPU) (Bar Mitzvah)
117.121.21.10|84824|445 Oracle Java SE Multiple Vulnerabilities (July 2015 CPU) (Bar Mitzvah)
117.121.13.33|73570|445 Oracle Java SE Multiple Vulnerabilities (April 2014 CPU)

7 thoughts on “Parsing Nessus v2 XML reports with python

  1. Pingback: Nessus v2 xml report format | Alexander V. Leonov

  2. Nikolay

    Hi,

    i’m glad to see you have started to use Python. XML is good point but there is another way is to export to ‘csv’. I prefer last one

    Also there is some very cool package:
    https://github.com/tenable/nessrest

    I’ve used it before to automate download/exporting and converting all this unmanageable Nessus staff )))

    Reply
    1. Alexander Leonov Post author

      Hi Nikolay,

      Thanks for comment! Yeah, I know Python a bit 😉
      Unfortunately, Nessus CSV doesn’t contain some important information about vulnerabilities. I made a comparison here.

      Nessrest is awesome. It’s very cool that Tenable keeps it up to date.
      However, for me it was easier to work with API directly.

      Reply
  3. Pingback: Tenable.IO VM: connected scanners and asset UUIDs | Alexander V. Leonov

  4. Wiles

    Great post! thanks for sharing this!

    Have you tried to work with the ‘plugin_output’ of the nessus file? I always find it to be a challenge. I am currently learning Python and trying to get the data into more readable format. I started to modify your code above and am able to filter on a specific plugin and get the plugin output however it is still not in the format I’d like.

    If you have any pointers or links to share that would be great!

    Thanks

    Reply
    1. Alexander Leonov Post author

      Thanks, Wiles!

      Yeah, parsing “plugin_output” is always tricky. Every plugin may have, in theory, it’s own format of output =) I parse “plugin_output” for some plugins, for example to get versions from detection plugins, but it requires me to write separate piece of code for every plugin.
      I think it might be a good idea to analyze code of all nasl plugins and classify them by output format. It will make getting data from “plugin output” much easier. However, maybe Tenable should do it by themselves 😉

      Reply

Leave a Reply

Your email address will not be published. Required fields are marked *