Creating Splunk Alerts using API

As I mentioned in “Accelerating Splunk Dashboards with Base Searches and Saved Searches“, Splunk Reports are basically the Saved Searches. Moreover, Splunk Alerts are also the same Saved Searches with some additional parameters.

Creating Splunk Alerts using API

The question is what parameters you need to set to get the right Alert.

Here is the official documentation, which describes all the parameters. Honestly, I don’t find it sufficient. So I just used a request that Splunk GUI makes to create an Alert in my python script. It worked. But Splunk GUI sets a lot of obviously redundant parameters and I minimized them a little bit by trial and error.

I assume that we already know how to get app_author and app_name (see the previous post). The value for splunk_server is something like https://splunk.corporation.com:8089

Let’s say we want to receive email Alert if there were no events in some important_index for the last 24 hours. That means something broke.

search_name = 'No events in important_index for the last 24 hours'

Delete Alert

First of all, I want make sure that there are no Alerts with such name, so I make a delete request:

data = {'output_mode': 'json'}
response = requests.delete(splunk_server + '/servicesNS/' + app_author + '/' + app_name + '/saved/searches/' + search_name, data=data, auth=(user, password), verify=False)

Create Alert

Now I create a new Alert that will for events in important_index at 5:00 am. If the count of such events will be equal to 0, rhe email to j.smith@corporation.com will be sent.

email = 'j.smith@corporation.com'
search = 'index="important_index" earliest=-1d'
alert_comparator = 'equal to'
alert_threshold = 0
cron = '0 5 * * *'

Here is somehow minimized set of parameters and the request:

data = {
'output_mode': 'json',
'action.email.bcc': '',
'action.email.cc': '',
'action.email.content_type': 'plain',
'action.email.message.alert': 'The alert condition for \'$name$\' was triggered.',
'action.email.message.report': 'The scheduled report \'$name$\' has run.',
'action.email.to': email,
'action.email.sendresults': '1',
'action.email.inline': '0',
'action.email.format': 'csv',

'actions': 'email',
'alert.digest_mode': '1',
'alert.expires': '24h',
'alert.managedBy': '',
'alert.severity': '3',
'alert.suppress': '0',
'alert.suppress.fields': '',
'alert.suppress.period': '',
'alert.track': '0',
'alert_comparator': alert_comparator,
'alert_condition': '',
'alert_threshold': alert_threshold,
'alert_type': 'number of events',
'allow_skew': '0',
'cron_schedule': cron,
'description': '',
'disabled': '0',
'displayview': '',
'is_scheduled': '1',
'is_visible': '1',
'max_concurrent': '1',
'name': search_name,
'realtime_schedule': '1',
'restart_on_searchpeer_add': '1',
'run_n_times': '0',
'run_on_startup': '0',
'schedule_priority': 'default',
'schedule_window': '0',
'search': search,
'action.email': '1'
}
response = requests.post(splunk_server + '/servicesNS/' + app_author + '/' + app_name + '/saved/searches', data=data, auth=(user, password), verify=False)

You can see how the Alert looks in Splunk GUI in the image above.

Note that I included action.email.sendresults, action.email.inline, action.email.format parameter to send search results as csv. It’s not necessery for this particular example, but can be quite useful for the other Alerts, when some data should be reviewed by the analyst.

2 thoughts on “Creating Splunk Alerts using API

  1. Pingback: Retrieving data from Splunk Dashboard Panels via API | Alexander V. Leonov

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.