Creating Splunk Alerts using API. As I mentioned in “Accelerating Splunk Dashboards with Base Searches and Saved Searches“, Splunk Reports are basically the Saved Searches. Moreover, Splunk Alerts are also the same Saved Searches with some additional parameters.

The question is what parameters you need to set to get the right Alert.

Here is the official documentation, which describes all the parameters. Honestly, I don’t find it sufficient. So I just used a request that Splunk GUI makes to create an Alert in my python script. It worked. But Splunk GUI sets a lot of obviously redundant parameters and I minimized them a little bit by trial and error.

I assume that we already know how to get app_author and app_name (see the previous post). The value for splunk_server is something like https://splunk.corporation.com:8089

Let’s say we want to receive email Alert if there were no events in some important_index for the last 24 hours. That means something broke.

search_name = 'No events in important_index for the last 24 hours'

Delete Alert

First of all, I want make sure that there are no Alerts with such name, so I make a delete request:

data = {'output_mode': 'json'}
response = requests.delete(splunk_server + '/servicesNS/' + app_author + '/' + app_name + '/saved/searches/' + search_name, data=data, auth=(user, password), verify=False)

Create Alert

Now I create a new Alert that will for events in important_index at 5:00 am. If the count of such events will be equal to 0, rhe email to j.smith@corporation.com will be sent.

email = 'j.smith@corporation.com'
search = 'index="important_index" earliest=-1d'
alert_comparator = 'equal to'
alert_threshold = 0
cron = '0 5 * * *'

Here is somehow minimized set of parameters and the request:

data = {
'output_mode': 'json',
'action.email.bcc': '',
'action.email.cc': '',
'action.email.content_type': 'plain',
'action.email.message.alert': 'The alert condition for \'$name$\' was triggered.',
'action.email.message.report': 'The scheduled report \'$name$\' has run.',
'action.email.to': email,
'action.email.sendresults': '1',
'action.email.inline': '0',
'action.email.format': 'csv',
'actions': 'email',
'alert.digest_mode': '1',
'alert.expires': '24h',
'alert.managedBy': '',
'alert.severity': '3',
'alert.suppress': '0',
'alert.suppress.fields': '',
'alert.suppress.period': '',
'alert.track': '0',
'alert_comparator': alert_comparator,
'alert_condition': '',
'alert_threshold': alert_threshold,
'alert_type': 'number of events',
'allow_skew': '0',
'cron_schedule': cron,
'description': '',
'disabled': '0',
'displayview': '',
'is_scheduled': '1',
'is_visible': '1',
'max_concurrent': '1',
'name': search_name,
'realtime_schedule': '1',
'restart_on_searchpeer_add': '1',
'run_n_times': '0',
'run_on_startup': '0',
'schedule_priority': 'default',
'schedule_window': '0',
'search': search,
'action.email': '1'
}
response = requests.post(splunk_server + '/servicesNS/' + app_author + '/' + app_name + '/saved/searches', data=data, auth=(user, password), verify=False)

You can see how the Alert looks in Splunk GUI in the image above.

Note that I included action.email.sendresults, action.email.inline, action.email.format parameter to send search results as csv. It’s not necessery for this particular example, but can be quite useful for the other Alerts, when some data should be reviewed by the analyst.

Alexander Leonov

Hi! My name is Alexander and I am a Vulnerability Management specialist. You can read more about me here. Currently, the best way to follow me is my Telegram channel @avleonovcom. I update it more often than this site. If you haven’t used Telegram yet, give it a try. It’s great. You can discuss my posts or ask questions at @avleonovchat.

А всех русскоязычных я приглашаю в ещё один телеграмм канал @avleonovrus, первым делом теперь пишу туда.