For those using log tools such as Splunk, you can setup alerts. These will run queries every so often and trigger actions when conditions are met e.g. count of events surpasses a threshold.
Hey there, The signup process is not automated. It is reviewed on a frequent basis (usually once a week, but people are people). That means that the signup is not instant, but you should receive an invite within two weeks. “Splunk is a key part of Slack’s ability to operate a zero trust network,” Ryder says. “Because Splunk gives us the visibility into all the activity that’s happening across all of our cloud services.” For Slack, staying secure has been essential as it continues to innovate and stays in lockstep with its customers.
This post is about pushing those alerts from Splunk to Slack. When an alarm is triggered, a JSON request is made by Splunk to a URL you provide. This gives the benefit of your own customisation of messages.
Reasons
You can find applications on the Splunk app store to post alerts to Slack. However you may have issues when running such apps on a cluster. Or you may have permission issues in an enterprise environment. In my situation, I ran into both problems.
Flow
- Splunk invokes an AWS API Gateway endpoint.
- The AWS API Gateway endpoint invokes an AWS Lambda function.
- The AWS Lambda function sends a message to a Slack inbound web hook (and delivers the message your own desired Slack channel).
Step 1 - Create Slack Inbound Webhook
Go to the following page, whilst logged-in to your Slack workspace:
Or alternatively, from Slack’s website:
- Configure apps
- Custom integrations (sidebar)
- Incoming WebHooks
More information about inbound web hooks can be found here:
Once you’ve setup an inbound webhook, copy the Webhook URL for the next step. Let’s pretend it’s:
Step 2 - Setup AWS Lambda Function
Create a new Lambda function from scratch:
For this example, we’ll call it splunk alert.
Then paste in the following, but edit the request options with parts of the Webhook URL from the previous step:
Towards the end is the JSON object message, which is the Slack message written to the Slack inbound webhook. This can be heavily customised.
Docs on Slack message format:
Once you’re happy with your Lambda, publish it. At the top, go to Actions and select Publish new version.
Step 3 - Create AWS API Gateway Trigger
On your AWS Lambda function page, go to the Triggers tab and add a trigger.
You will then see a dotted box, click it and select API Gateway.
Then will in the form with an API name (such as splunk-alerts) and set the Security to Open (although you may want to change this later):
You should now have API Gateway available as a trigger. Click the arrow icon to show the endpoint’s URL.
Step 4 - Test Trigger
Let’s say the trigger URL is:
Just make a POST request with the following test payload:
In Chrome I use the Postman app, but this is simple enough to achieve with cURL as well:
Step 5 - Splunk Alert
Run a query on the Splunk search application. Once it has finished loading, select Save As and then Alert.
Configure the alert as needed; useful docs:
After the alert is setup, add a Webhook action and set the URL to the endpoint created earlier.
Summary
Splunk Slack Channel
You should now have monkeyboy to save the day: