Export anything to Splunk with HTTP Event Collector. In a previous post I described how to export Nessus scan reports to Splunk server using standard app. Today let’s see how to export any structured data presented in JSON, including of course Nessus scan reports, to Splunk using HTTP Event Collector.

First of all, we should create new HTTP Event Collector

http://your_splunk_host:8000/en-US/manager/launcher/http-eventcollector

And press “New Token” button

I set only name “New_HTTP_token”. Press “Next” button.

In “Settings” you need to specify in which index exported data will be stored. I created a new one “test_index”.

Reviewing our very basic settings.

And finally we get a token we can use in for authentication in our request.

HTTP Event Collector uses 8088 tcp port. Don’t forget to open it in your firewall.

For FirewallD in CentOS7:
# firewall-cmd --add-port=8088/tcp --permanent
# firewall-cmd --reload
# firewall-cmd --list-all-zones
...
public (default, active)
interfaces: enp3s0f0
sources:
services: dhcpv6-client ssh
ports: 8088/tcp
masquerade: no
forward-ports:
icmp-blocks:
rich rules:
...

For CentOS 6 and iptables you can find an example here: Making vulnerable OpenSSL scanning target.

Or, now we are ready to send something to our Splunk server. All our request should have parameters: time, host, and event.

$ echo '{"time":"1471613569", "host":"test_host", "event":{"test_key":"test_value"}}' > temp.json
$ curl -k https://your_splunk_host:8088/services/collector -H 'Authorization: Splunk 8DEE8A67-7700-4BA7-8CBF-4B917CE2352B' -d @temp.json
{"text":"Success","code":0}

Here it is in Splunk search interface:

But putting one event per request might be slow. Splunk lets you send a package of several events in the json, one per line. Not a very nice format, but why not.

$ echo -e '{"time":"1471613579", "host":"test_host", "event":{"test_key":"test_line1"}}\n{"time":"1471613580", "host":"test_host", "event":{"test_key":"test_line2"}}' > temp.json
$ curl -k https://your_splunk_host:8088/services/collector -H 'Authorization: Splunk 8DEE8A67-7700-4BA7-8CBF-4B917CE2352B' -d @temp.json
{"text":"Success","code":0}

Here we can see two events:

Now we will try to send something from Nessus vulnerability scan report.

Where can I get a timestamp?

I take timestamp of the scan start via API:

...
"creation_date": 1444816022,
...

Read more at “Retrieving scan results through Nessus API”

Where can I get a host name?

I take it in scan report:

...
<ReportHost name="192.168.0.1">
...

Read more at “Nessus v2 xml report format”

I make event from <ReportItem>…</ ReportItem> converting it to JSON. So, the complete request look like this:

$ echo -e '{"time":"1444816022", "host":"192.168.0.1", "event":{"cvss_temporal_vector":["CVSS2#E:ND/RL:OF/RC:C"], "protocol":"tcp", "cvss_base_score":"6.8", "exploitability_ease":["No known exploits are available"], "solution":"Upgrade to VLC version 2.0.9 or later.", "cvss_temporal_score":["5.9"], "plugin_output":" Path : C:Program Files (x86)VideoLANVLC[newline] Installed version : 2.0.7[newline] Fixed version : 2.0.9", "port":"445", "xref":["OSVDB:98063"], "severity":"2", "script_version":"$Revision: 1.1 $", "osvdb":["98063"], "patch_publication_date":"2013/08/25", "synopsis":"The remote Windows host contains a media player that is affected by a buffer overflow vulnerability.", "risk_factor":"Medium", "pluginID":"70560", "fname":"vlc_2_0_9.nasl", "svc_name":"cifs", "description":"The version of VLC media player installed on the remote host is earlier than 2.0.9. It is, therefore, affected by a buffer overflow vulnerability in 'packetizer/mpeg4audio.c'.", "see_also":["http://www.videolan.org/developers/vlc-branch/NEWS"], "bid":["62724"], "plugin_modification_date":"2013/10/23", "plugin_name":"VLC < 2.0.9 Buffer Overflow", "plugin_publication_date":"2013/10/23", "cvss_vector":"CVSS2#AV:N/AC:M/Au:N/C:P/I:P/A:P", "vuln_publication_date":["2013/09/30"], "cpe":["cpe:/a:videolan:vlc_media_player"], "pluginName":"VLC < 2.0.9 Buffer Overflow", "exploit_available":["false"], "cve":["CVE-2013-4388"], "pluginFamily":"Windows", "plugin_type":"local"}}' > temp.json
{"text":"Success","code":0}

And here it is in Splunk.

You can compare it with “nessus:scan” from “Splunk Add-on for Tenable“. It’s much more informative. You can see all attributes nasl plugin has as well as plugin output. On the right panel statistics for attributes is be displayed. You can make requests the same way you do it in Nessus/SecurityCenter GUI.

Warning! If you try to send big files to Splunk via HTTP Event Collector you will probably get max_content_length error.

You change the limit (in bytes) in $SPLUNK_HOME/etc/system/default/limits.conf (/opt/splunk/etc/system/default/limits.conf)

[http_input]
# The max number of tokens reported by logging input metrics.
max_number_of_tokens = 10000
# The interval (in seconds) of logging input metrics report.
metrics_report_interval = 60
# The max request content length.
#max_content_length = 1000000
max_content_length = 10000000000

And then restart:

http://your_splunk_host:8000/en-US/manager/launcher/control

“Click the button below to restart Splunk”.