Making Splunk searches using REST API. When you have already learned how to make search requests in Splunk GUI, it may be nice to figure out how do the same from your own scripts using the Splunk REST API.

It’s really easy!

Ok, we have a Splunk SIEM account:

user="user"
pass="Password123"

And we want to execute this search request:

search='search index="index_nessus" host="192.168.56.50"'

First of all we need to get ID of our search request (make sure that 8089 port is open! ?):

curl -u $user:$pass -k https://192.168.56.101:8089/services/search/jobs -d search="$search"

<?xml version="1.0" encoding="UTF-8"?>
<response>
<sid>1490878950.3029</sid>
</response>


Now, having this ID we can check if the results of this search request are available:

curl -u $user:$pass -k https://192.168.56.101:8089/services/search/jobs/1490878950.3029

This command will return a huge a xml. We need to figure out if it is finished or not:

curl -s -u $user:$pass -k https://192.168.56.101:8089/services/search/jobs/1490878950.3029  | grep "dispatchState"
<s:key name="dispatchState">DONE</s:key>

When dispatchState is DONE, we can get the results:

curl -u $user:$pass -k https://192.168.56.101:8089/services/search/jobs/1490878950.3029/results/ --get -d output_mode=csv

This command will return 200 lines of text like this:

...
"index_nessus~11~C780C931-CD4E-4CE4-ACEC-6240AF49DAB2","11:3468150",1490703404,"{""scan_group"":""vm_moscow"", ""plugin_output"":""Port 135/tcp was found to be open"", ""protocol"":""tcp"", ""severity"":""0"", ""script_version"":""$Revision: 1.73 $"", ""risk_factor"":""None"", ""solution"":""n/a"", ""plugin_modification_date"":""2017/03/27"", ""pluginName"":""Netstat Portscanner (WMI)"", ""agent"":[""windows""], ""pluginFamily"":""Port scanners"", ""synopsis"":""The list of open ports could be retrieved via netstat."", ""pluginID"":""34220"", ""plugin_name"":""Netstat Portscanner (WMI)"", ""fname"":""wmi_netstat.nbin"", ""plugin_publication_date"":""2008/09/16"", ""plugin_type"":""local"", ""svc_name"":""epmap"", ""port"":""135"", ""description"":""Using the WMI interface, it was possible to get the open ports by running the netstat command remotely.""}",14,"192.168.56.101"
...

It’s some plugin data from Nessus scan results.

And if we use “| table ” operator, in results we also get great parsable csv table. Therefore, you do not need to mess around with the creation of reports, as in the case of searching in Splunk graphical interface.

Upd. 04.10.2018 Recently I made some python scripts to automate Splunk searches via API. This functionality can be very useful to debug the searches for dashboards. Or you even can get some events from Splunk and put them back with some different data, for example if some connector was broken for a day.

Let’s say we want to get all events from ‘important_index’ for 10/04/2018. First of all, we need to get ID for the search:

import requests

search='index="important_index" earliest=10/04/2018:0:0:0 latest=10/04/2018:23:59:00'

data = {'search': search, 'max_count':'10000000'}
response = requests.post('https://splunk.corporation.com:8089/services/search/jobs', data=data,
                         auth=(splunk_user, splunk_user_password), verify=False)

It will return an xml, so we can parse it with Python etree module:

import xml.etree.ElementTree as ET

root = ET.fromstring(response.text)
for tag in root:
    job_id = tag.text

Now we should check the status of the job until it is ‘DONE’:

import xml.etree.ElementTree as ET

def get_dispatch_state(xml_text):
    root = ET.fromstring(xml_text)
    dispatchState = ""
    for tag in root:
        if "content" in tag.tag:
            for tag2 in tag:
                for tag3 in tag2:
                    if tag3.attrib['name'] == "dispatchState":
                        dispatchState = tag3.text
    return (dispatchState)

status = "UNKNOWN"
while status != "DONE":
    response = requests.post('https://splunk.corporation.com:8089/services/search/jobs/' + job_id,
                             auth=(splunk_user, splunk_user_password), verify=False)
    status = get_dispatch_state(response.text)
    print(status)
    time.sleep(5)

Output:

PARSING
DONE

And finally we should repetitive results of the job. We can get no more than 50000 event in one request, so it will be necessary to make several requests and use an offset:

data = {'output_mode': 'json'}
offset = 0
results_fullness = False
events = list()

while not results_fullness:
    response = requests.get('https://splunk.corporation.com:8089/services/search/jobs/' + job_id + \
                            '/results?count=50000&offset='+str(offset),
                            data=data,
                            auth=(splunk_user, splunk_user_password), verify=False)
    response_data = json.loads(response.text)
    print("Count: " + str(len(response_data['results'])))
    events += response_data['results']
    if len(response_data['results']) == 0: #This means that we got all the results
        results_fullness = True
    else:
        offset += 50000
print("All: " + str(len(events)))

Output:

Count: 50000
Count: 2751
Count: 0
All: 52751

Now we will have all the events in events variable.