How to use the Web API tile with Splunk

Overview

This article covers how to use the [Web API tile](#/v3/Walkthroughs/Tiles/HowToUseTheWebAPITile to query live API data from Splunk and display this in a dashboard or perspective.

Squared Up uses the Splunk account that you configure and queries will be updated as the page updates, by default every minute.

You can also watch the Webinar: Integrating Splunk

This article covers three areas:

  1. Adding a Splunk provider in Squared Up.

  2. Adding a Splunk query to a dashboard.

  3. Adding a Splunk query to a perspective using variables.

Prerequisites

  • Squared Up v3 with an Enterprise Application Monitoring license – this gives you the Web API tile. The Squared Up server must have access to your Splunk instance, since it is the server that connects to Splunk, and not your users’ browser.

  • A Splunk instance. You may need some help from a Splunk admin to get an account with the correct privileges.

Adding a Splunk provider in Squared Up

Here we will create a simple provider, using an authorization header specifying basic APIkey, and enabling ignore invalid ssl if using self-signed certificates.

  1. Log on to Squared Up and navigate to the right-hand menu ☰ then system and then click on the web api perspective.

  2. Click Add new provider.

  3. Leave the provider type as simple and in the service name box type in a suitable name, e.g. Splunk

  4. The base URL should look something like https://instancename:8089/services/search/jobs where instancename is your instance. This URL is prepended to every request URL.

    The port number 8089 is the Splunk API port which is different to the port used when browsing Splunk normally, which would be port 8000.

    If you are using Splunk with Windows authentication you will need to log in to Splunk and add the Squared Up application pool account to the list of authorized users.

    If you are using Splunk’s built-in authentication follow the steps below.

  5. Change ignore invalid ssl to on if you are using a self-signed certificate.

  6. Click the add button under default headers.

  7. In the first box type authorization

  8. In the second box paste in the word basic followed by a space and then your username:password which has been Base64 encoded, as described below.

  9. On the Start button type:

    powershell

  10. Open PowerShell and type:

    $bytes = [System.Text.Encoding]::ASCII.GetBytes("username:password")

  11. Then type:

    [system.convert]::ToBase64String($bytes)

  12. Copy the output. On the Squared Up Splunk provider page after the word basic and a space paste your Base64 encoded username:password, so that your provider in Squared Up looks similar to this:

    provider

  13. Click add provider.

Walkthrough: Adding a Splunk query to a dashboard

This walkthrough takes you through using a simple Splunk query on a dashboard.

It is important to use http post mode and to set the key path in the response data section to results.
  1. On the top navigation bar in Squared Up click the + to create a new dashboard, and give your dashboard a title.

  2. Click on the Web API tile.

    Web Api tile

  3. Select Web API (Grid).

    Web Api Grid

  4. In the provider section select the Splunk provider that you created earlier.

    Provider section

  5. In the http mode section click on post to change the http method from get to post.

    http mode section

    The URL shown here is the one you specified in the Web API provider settings above, and is the correct URL for search jobs.

  6. The headers & data section is where you can pass key variables through to Splunk. Click the add button to add all the data items below:

    name: search value: search index=WinEventLog SourceName=HealthService Type=Error

    search is mandatory because this is where you input your Splunk search query. You need to type search followed by a space and then your Splunk search query string. We recommend you configure your search query in Splunk to check the data returned, then copy the query into Squared Up. In this example the WinEventLog is an index we have created in Splunk, SourceName=HealthService specifies data from the SCOM HealthService, and Type=Error specifies error messages.

    name: exec_mode value: oneshot

    exec_mode is mandatory as this tells Splunk how to execute the search and what to do with the results. In this case oneshot tells it to search and then send the results back (as opposed to storing them for later).

    name: output_mode value: json

    output_mode is mandatory and must always be set as json so the results are in the correct format. You need this setting in order to see any results.

    name: timeout value: 60

    timeout is mandatory and tells Splunk how long to keep the data after the search has completed. The value is in seconds, and with it set to only 60 we are preventing the Splunk server from filling up with loads of active searches. The default is 24 hours.

    name: earliest_time value: -1h

    earliest_time is how far back the search should go. You can also use UTC strings.

    name: latest_time value: now

    latest_time is when it should run to, so in this case with earliest_time as -1h and latest_time as now, it will return logs from the last hour.

    name: max_time value: 30

    max_time is the query timeout limit i.e. 30 seconds.

  7. Click apply changes and the headers & data section should look like this:

    headers & data section

    At this point you should see some data displayed in the tile. The next steps take you into the results data to show you more detail.

  8. Click next

  9. In the response data section, type results into the box.

    response data section

  10. Click next to move to the grid columns section.

    Here we are going to use custom labels to make the data more meaningful. For more information see How to use custom labeling

  11. Rename the host column to Computer.

  12. Click edit next to the Computer column and paste the following in to the custom template box:

    {{value.split('.')[0]}}

    This splits the computer name on the fullstop and only shows the first part.

  13. Click done.

  14. Rename the _time column to Time Logged.

  15. Click edit next to the Time Logged column and paste the following in to the custom template box:

    {{timeago(value)}}

    This shows how long ago the event occurred.

    timeago

  16. Click done.

  17. Click edit next to the Message column and paste the following in to the custom template box:

    {{value.substr(0,230)}}

    This shows only the first 230 characters.

  18. Click done.

  19. Hide all the other columns to leave just the three you have edited.

  20. You may wish to change the order the columns to show Computer, Time Logged, Message by dragging them into position.

  21. Leave the grid options section as it is.

  22. Click done.

It can be useful to use the clone tile button at the top right of the section to copy the Web API tile you have configured and then make a few changes to the search terms in the headers & data section to add another similar Splunk query to the dashboard.

Walkthrough: Adding a Splunk query to a perspective using variables

The configuration for a perspective is very similar to that for a dashboard, as described above, except that as a perspective can be shown for many objects you can use variables in the search query to inject SCOM properties.

  1. Browse or search for to the object for which you wish to add a new perspective.

  2. On the perspectives ribbon click the + to create a new perspective.

  3. Give your perspective a title, configure the perspective scope and click done

  4. Click on the Web API tile.

  5. Select Web API (Grid).

  6. In the provider section select the Splunk provider that you created.

  7. In the http mode section click on post to change the http method from get to post.

  8. In the headers & data section click the add button to add the following settings. This is where you can add SCOM properties as variables in the search query.

    name: search value: search index=WinEventLog ComputerName="{{displayName}}"

    This search string uses the SCOM property displayName as a variable. So if you browse to this perspective for a different computer then the query will use that computer name, and the results will be specific to that computer. You can view a list of available properties on the monitored entity perspective.

    name: exec_mode value: oneshot

    name: output_mode value: json

    name: timeout value: 60

    name: earliest_time value: -1h

    name: latest_time value: now

    name: max_time value: 30

  9. Click apply changes

  10. Click next

  11. In the response data section, type results into the box.

  12. Click next to move to the grid columns section.

  13. Rename the _time column to Time.

  14. Click edit next to the Time column and paste the following in to the custom template box:

    {{timeago(value)}}

  15. Click done.

  16. Rename the source column to Log.

  17. Click edit next to the Log column and paste the following in to the custom template box:

    {{value.substr(16,100)}}

  18. Click done.

  19. Click edit next to the Message column and paste the following in to the custom template box:

    {{value.substr(0,200)}}

    For more help customizing the columns of data see How to use the Grid designer when configuring tiles.

  20. Click done.

  21. Hide all the other columns to leave just the three you have edited.

  22. You may wish to change the order of the columns to show Log, Time, Message, by dragging the column handles.

  23. Leave the grid options section as it is.

  24. Click done.

Reference: headers & data

Each Web API tile you add to a dashboard or perspective can run a different Splunk query. We recommend you configure your search query in Splunk to check the data returned is what you want, then copy the query into Squared Up. The Splunk search query is specified in the headers & data section, along with other data options, which are described for reference below, and covered step by step in the walkthroughs.

Name Description Mandatory?
search search followed space and then your Splunk search query yes
exec_mode Must specify oneshot yes
output_mode Must specify json yes
timeout How long Splunk should keep results for (in seconds, e.g.60) yes
earliest_time The earliest result (useful for last x hours, e.g. -1h) no
latest_time The latest result, usually just now no
max_time Query processing timeout limit (in seconds, e.g.30) no

Hints and Tips

  • Look at the Splunk API Documentation or speak to your Splunk expert.

  • Test your query with PowerShell/Curl/Postman to see the results directly. You can also browse to the URL that you used in your Splunk provider (right-hand menu ☰ > system > web api) which will be something like https://instancename:8089/services/search/jobs (where instancename is your Splunk instance) to see a visual API explorer. Click on services > search jobs you can see the job numbers of any recurring jobs in Splunk, which can then be used in Squared Up see API explorer.

  • Users who are not SCOM administrators will need the construct-sensitive-queries permission to be able to create or edit Web API tiles, which should only be given to trusted users. See How to manage user profiles

  • Use custom labels to improve data formatting see How to use custom labeling

  • You can view a list of the properties available for the search query by going to the monitored entity perspective. (Although displayName is not listed, it is also available for you to use).

Webinar: Integrating Splunk

Splunk REST API documentation

An Introduction to the Splunk REST API

Splunk REST API Tutorials

[How to use the Web API tile](#/v3/Walkthroughs/Tiles/HowToUseTheWebAPITile

How to use custom labeling

Webinar: Custom Labeling

Webinar: Introduction to perspectives

How to use the Web API tile with ServiceNow

How to use the Grid designer when configuring tiles label: How to use the Web API tile with Splunk keywords: Splunk Web API tile providers connection querying job authorisation authorization