Processing Vulners collections using Python

Vulners collection is a zip archive containing all available objects of some type (e.g. CentOS security bulletins or OpenVAS detection plugins) from the Vulners Knowledge Base. Let’s see how to work with this data using powerful Python scripting language. You can read more about Vulners itself at “Vulners – Google for hacker“.

Vulners Collections and python

All collections are listed at https://vulners.com/#stats:

Vulners Stats

Note a gray icon with black arrow. Press it to download particular vulners collection.

OpenVAS collection link: https://vulners.com/api/v3/archive/collection/?type=openvas

If you need to get all objects for further analysis, you don’t need to make huge amount simmilar Search API requests. You just need to download one file. It’s takes less time and efforts and makes less load on Vulners service.

To download file in Python you may use:

import urllib
urllib.urlretrieve("https://vulners.com/api/v3/archive/collection/?type=openvas", "openvas.zip")

for Python 3+ use ‘import urllib.request’ and urllib.request.urlretrieve

In Python you can read directly from the zip archive:

from zipfile import ZipFile
archive = ZipFile("openvas.zip")
archived_file = archive.open(archive.namelist()[0])
archive_content = archived_file.read()
archived_file.close()

Vulners collection files contains data in JSON. If you do “print(archive_content)“, you will see:

...
  {
      "_index": "bulletins",
      "_type": "bulletin",
      "_id": "OPENVAS:892605",
      "_score": 0.0,
      "_source": {
      "sourceData": "# OpenVAS Vulnerability Test\n# $Id:[...] {\n    security_message(data:report);\n} else if (__pkg_match) {\n    exit(99); # Not vulnerable.\n}\n",
      "naslFamily": "Debian Local Security Checks",
      "title": "Debian Security Advisory DSA 2605-2 (asterisk - several issues)",
      "bulletinFamily": "exploit",
      "reporter": "Copyright (c) 2013 Greenbone Networks GmbH http://greenbone.net",
      "description": "Several vulnerabilities were discovered in Asterisk, a PBX and telephony\ntoolkit, that allow remote attackers to perform denial of service\nattacks.",
      "lastseen": "2016-09-26T20:40:29",
      "references": [
        "http://www.debian.org/security/2013/dsa-2605.html"
      ],
      "type": "openvas",
      "published": "2013-01-19T00:00:00",
      "hashmap": [
        {
          "hash": "708697c63f7eb369319c6523380bdf7a",
          "key": "bulletinFamily"
        },
        {
          "hash": "7f867f04ef59aa5cade6911e81c14a6f",
          "key": "cvelist"
        },
        {
          "hash": "84813b1457b92d6ba1174abffbb83a2f",
          "key": "cvss"
        },
        {
          "hash": "6767bcda3f63a278d6ce365bec2419bf",
          "key": "description"
        },
        {
          "hash": "805b7b8dfbbd4d9d41ec7f38ec45197d",
          "key": "href"
        },
        {
          "hash": "e29ab3248acfd6527d1025d945e184c6",
          "key": "modified"
        },
        {
          "hash": "74562d71b087df9eabd0c21f99b132cc",
          "key": "naslFamily"
        },
        {
          "hash": "56765472680401499c79732468ba4340",
          "key": "objectVersion"
        },
        {
          "hash": "980e8fbb7a34d92c9977698d6b4a6e38",
          "key": "pluginID"
        },
        {
          "hash": "81ce5c6faae6a195a646ae91c15b0cc8",
          "key": "published"
        },
        {
          "hash": "2d83a0110462ad2c6311bad5832c72fd",
          "key": "references"
        },
        {
          "hash": "5474798d22021d679c1738be31fc4947",
          "key": "reporter"
        },
        {
          "hash": "ecdde94497f1901046a45666767ead49",
          "key": "sourceData"
        },
        {
          "hash": "2972d92c6acaa39b8d3b96678f4423db",
          "key": "title"
        },
        {
          "hash": "47c1f692ea47a21f716dad07043ade01",
          "key": "type"
        },
        {
          "hash": "cfcd208495d565ef66e7dff9f98764da",
          "key": "viewCount"
        }
      ],

      ],
      "href": "http://plugins.openvas.org/nasl.php?oid=103593",
      "cvelist": [
        "CVE-2012-5192",
        "CVE-2012-5193"
      ],
      "cvss": {
        "vector": "AV:NETWORK/AC:LOW/Au:NONE/C:PARTIAL/I:NONE/A:NONE/",
        "score": 5.0
      },
      "objectVersion": "1.2",
      "pluginID": "103593",
      "modified": "2016-04-14T00:00:00",
      "hash": "ae028cee0fc54c8eef6b544331442af742787cdf75cb67a8e815179968e96495",
      "id": "OPENVAS:103593",
      "edition": 1
    }
  },
...
]

We can trivially present this json file as a list of dictionaries:

import json
vulners_objects = json.loads(archive_content)
print(vulners_objects[10]["_id"])
print(vulners_objects[10]["_source"]["title"])
print(vulners_objects[10]["_source"]["cvelist"])

Output:

OPENVAS:71963
Slackware Advisory SSA:2011-224-01 bind 
[u'CVE-2011-2464', u'CVE-2011-1910']
AV:NETWORK/AC:LOW/Au:NONE/C:NONE/I:NONE/A:PARTIAL/

If you need, for example, a dict, from which you can easily get OpenVAS Plugin data by it’s ID (openvas_objects["OPENVAS:71963"]) and you want to know all CVEs that are mentioned in OpenVAS Plugin references (openvas_cves), you can run:

openvas_cves = set()
openvas_objects = dict()
for obj in vulners_objects:
    openvas_objects[obj["_id"]] = obj
    for cve in set(obj["_source"]["cvelist"]):
        openvas_cves.add(cve)

One thought on “Processing Vulners collections using Python

  1. Pingback: Fast comparison of Nessus and OpenVAS knowledge bases | Alexander V. Leonov

Leave a Reply

Your email address will not be published. Required fields are marked *