How to use the Web API tile with Splunk

This article covers how to use the Web API tile to query live API data from Splunk and display this in a dashboard or perspective.

SquaredUp uses the Splunk account that you configure and queries will be updated as the page updates, by default every minute.

You can also watch the Webinar: Integrating Splunk

This article covers four areas:

1. Adding a Splunk provider in SquaredUp.
2. Adding a Splunk query to a dashboard.
3. Adding a Splunk query to a perspective using a single variable.
4. Adding a Splunk query to a perspective using an array of variables.

Prerequisites

• SquaredUp v4 with the Enterprise or EAM edition – this gives you the Web API tile. The SquaredUp server must have access to your Splunk instance, since it is the server that connects to Splunk, and not your users' browser.
• A Splunk instance. You may need some help from a Splunk admin to get an account with the correct privileges.

Adding a Splunk provider in SquaredUp

These steps allow you to create a basic auth provider so your specified username and password will be Base64 encoded automatically.

1. Log on to SquaredUp and navigate to the right-hand menu ☰ then system and then click on the web api perspective.
2. Click Add new provider.

3. Change the provider type to basic auth.

   

4. In the service name box type in a suitable name, e.g. Splunk

5. The base URL should look something like https://instancename:8089/services/search/jobs where instancename is your instance. This URL is prepended to every request URL.

   The port number 8089 is the Splunk API port which is different to the port used when browsing Splunk normally, which would be port 8000.

6. If you are using Splunk with Windows authentication you will need to log in to Splunk and add the SquaredUp application pool account to the list of authorized users.
7. If you are using Splunk's built-in authentication enter the username and password for your Splunk instance into the appropriate boxes and they will be Base64 encoded automatically.
8. Change ignore invalid ssl to on if you are using a self-signed certificate.
9. Click add provider.

Walkthrough: Adding a Splunk query to a dashboard

This walkthrough takes you through using a simple Splunk query on a dashboard.

It is important to use http post mode and to set the key path in the response data section to results.

1. On the top navigation bar in SquaredUp click the + to create a new dashboard, and give your dashboard a title.

2. Click on the Web API tile.
   
   

3. Select Web API (Grid).
   
   

4. Leave the scope section (v4.2 and above) blank for this walkthrough. (The scope is used in last walkthrough in this article).

5. In the provider section select the Splunk provider that you created earlier.

   

6. In the http mode section click on post to change the http method from get to post.

   

   The URL shown here is the one you specified in the Web API provider settings above, and is the correct URL for search jobs.

7. The headers & data section is where you can pass key variables through to Splunk. Click the add button under data to add the following data name and value settings:

   name: search value: search index=WinEventLog SourceName=HealthService Type=Error

   search is mandatory because this is where you input your Splunk search query. You need to type search followed by a space and then your Splunk search query string. We recommend you configure your search query in Splunk to check the data returned, then copy the query into SquaredUp. In this example the WinEventLog is an index we have created in Splunk, SourceName=HealthService specifies data from the SCOM HealthService, and Type=Error specifies error messages.

   name: exec_mode value: oneshot

   exec_mode is mandatory as this tells Splunk how to execute the search and what to do with the results. In this case oneshot tells it to search and then send the results back (as opposed to storing them for later).

   name: output_mode value: json

   output_mode is mandatory and must always be set as json so the results are in the correct format. You need this setting in order to see any results.

   name: timeout value: 60

   timeout is mandatory and tells Splunk how long to keep the data after the search has completed. The value is in seconds, and with it set to only 60 we are preventing the Splunk server from filling up with loads of active searches. The default is 24 hours.

   name: earliest_time value: -1h

   earliest_time is how far back the search should go. You can also use UTC strings.

   name: latest_time value: now

   latest_time is when it should run to, so in this case with earliest_time as -1h and latest_time as now, it will return logs from the last hour.

   name: max_time value: 30

   max_time is the query timeout limit i.e. 30 seconds.

8. Click apply changes and the headers & data section should look like this:

   

   At this point you should see a row of data displayed in the tile. The next steps take you into the results data to show more detail. If the tile output is blank check that the http mode section is set to post, and run the query in Splunk to check that data is returned.

9. Click next

10. In the response data section, type results into the box.

    

11. Click next to move to the grid columns section.

    Here we are going to use custom labels to make the data more meaningful. For more information see How to use custom labelling

12. Rename the host column to Computer.

13. Click edit next to the Computer column and paste the following in to the custom template box:

    {{value.split('.')[0]}}

    This splits the computer name on the fullstop and only shows the first part.

14. Click done.
15. Rename the _time column to Time Logged.

16. Click edit next to the Time Logged column and paste the following in to the custom template box:

    {{timeago(value)}}

    This shows how long ago the event occurred.
    
    

17. Click done.

18. Click edit next to the Message column and paste the following in to the custom template box:

    {{value.substr(0,230)}}

    This shows only the first 230 characters.

19. Click done.
20. Hide all the other columns to leave just the three you have edited.
21. You may wish to change the order the columns to show Computer, Time Logged, Message by dragging them into position.
22. Leave the grid options section as it is.
23. Click done.

It can be useful to use the clone tile button at the top right of the section to copy the Web API tile you have configured and then make a few changes to the search terms in the headers & data section to add another similar Splunk query to the dashboard.

Walkthrough: Adding a Splunk query to a perspective using a single variable

The configuration for a perspective is very similar to that for a dashboard, as described above, except that a single perspective can be configured to show for many objects. You can use a variable in the search query to inject a SCOM property to make the Splunk results relevant to the object being shown by the perspective. This walkthrough uses a single variable in the search query which is ideal when browsing to a server and injecting that server name into the Splunk query. The next walkthrough uses an array of variables which is more suited to an Enterprise Application (EA), where you may wish to inject the names of all the servers appearing in the EA map.

1. Browse or search for to the object for which you wish to add a new perspective. For this walkthrough, browse to a Windows server monitored by Splunk.
2. On the perspectives ribbon click the + to create a new perspective.
3. Give your perspective a title.
4. Check that the auto-populated class is Windows Server. This is so that your new perspective appears for all Windows servers, rather than just this one server.
5. Close the perspective settings.

6. Click on the Web API tile.
   
   

7. Select Web API (Grid).
   
   

8. Leave the scope section (v4.2 and above) blank for this walkthrough. (The scope is used in last walkthrough in this article).
9. In the provider section select the Splunk provider that you created.
10. In the http mode section click on post to change the http method from get to post.

11. In the headers & data section click the add button under data to add the following data name and value settings:

    name: exec_mode value: oneshot

    name: output_mode value: json

    name: timeout value: 60

    name: earliest_time value: -1h

    name: latest_time value: now

    name: max_time value: 30

    name: search value: search index=WinEventLog ComputerName="{{displayName}}"

    This search string uses the SCOM property displayName as a variable. So if you browse to this perspective for a different server then the query will use that server name, and the results will be specific to that server. You can view a list of available properties on the monitored entity perspective.

12. Click apply changes
13. Click next and in the response data section, type results into the box.
14. Click next to move to the grid columns section.
15. Rename the _time column to Time.

16. Click edit next to the Time column and paste the following in to the custom template box:

    {{timeago(value)}}

17. Click done.
18. Rename the source column to Log.

19. Click edit next to the Log column and paste the following in to the custom template box:

    {{value.substr(16,100)}}

20. Click done.

21. Click edit next to the Message column and paste the following in to the custom template box:

    {{value.substr(0,200)}}

    For more help customizing the columns of data see How to use the Grid designer when configuring tiles.

22. Click done.
23. Hide all the other columns to leave just the three you have edited.
24. You may wish to change the order of the columns to show Log, Time, Message, by dragging the column handles.
25. Leave the grid options section as it is.
26. Click done.

Walkthrough: Adding a Splunk query to a perspective using an array of variables

This walkthrough uses an array of variables in the search query, utilising the scope section of the Web API tile (v4.2 and above). This is ideal when browsing to an Enterprise Application (EA) and then injecting the names of all the servers on the EA map into the Splunk query.

1. Browse or search for to the object for which you wish to add a new perspective. For this walkthrough, browse to an Enterprise Application (EA) so you can go on to show Splunk information for the servers which are on EA map.
2. On the perspectives ribbon click the + to create a new perspective.
3. Give your perspective a title.
4. Remove the auto-populated class, and start typing Enterprise Application (v1) and select this from the list to add this class. This is so that your new perspective appears for all EAs, rather than just this one.
5. Close the perspective settings.

6. Click on the Web API tile.
   
   

7. Select Web API (Grid).
   
   

8. The scope section (v4.2 and above) allows you to specify a scope of SCOM objects, which can be used later in the tile configuration to insert SCOM object properties into the query. For an EA, you want to scope to the servers that are specified on the EA map by selecting something similar to the following:

   This /<YourApplicationName> Map / ... / Windows Computer

   The above will scope the tile to all the objects of class Windows Computer on this EA's map.

   

   Next we need to adjust the specified scope to allow it to work for all EAs, rather than just this one.

9. In the scope section click custom.

10. Click on the text <YourApplicationName> Map (children) which is your first scope step. This will expand the scope step so you can edit it.
    
    

11. Remove the auto-populated class <YourApplicationName> Map by clicking the cross x next to it.
    
    

12. Start typing Enterprise Application - Map and select this from the list to add this class. This is so that this tile scope will work for all EAs, rather than just this one EA.
    
    

13. In the provider section select the Splunk provider that you created.
14. In the http mode section click on post to change the http method from get to post. At this point you will still see an error in the tile output. This will be resolved in the next step.

15. In the headers & data section click the add button under data to add the following data name and value settings:

    name: exec_mode value: oneshot

    name: output_mode value: json

    name: timeout value: 60

    name: earliest_time value: -1h

    name: latest_time value: now

    name: max_time value: 30

16. If you apply changes at this point you should see a single line table. Next we will add the search query to return the results.

17. Still within the headers & data > data section click the add button to add the following:

    name: search value: search index=opp_events ComputerName IN () sourcetype=WMI:WinEventLog:Security

18. Place your cursor within the brackets () after ComputerName IN and click the mustache insert dynamic value button.

    

19. Click on the first snippet listed.

    So that the search term shows the following:

    search index=opp_events ComputerName IN ({{scope.map(item => '"'+item.displayName+'"').join(',')}}) sourcetype=WMI:WinEventLog:Security

    This search string uses scope.map to match the scope you have set, and the SCOM property displayName, but you can edit this as required.
    
    

20. The Headers & Data section should now look like this, and you can click apply changes:

    

21. Click next and in the response data section, type results into the box.

22. Click done and you should see a table of results.

    

Reference: headers & data

Each Web API tile you add to a dashboard or perspective can run a different Splunk query. We recommend you configure your search query in Splunk to check the data returned is what you want, then copy the query into SquaredUp. The Splunk search query is specified in the headers & data section, along with other data options, which are described for reference below, and covered step by step in the walkthroughs.

Hints and Tips

• Look at the Splunk API Documentation or speak to your Splunk expert.
• Test your query with PowerShell/Curl/Postman to see the results directly. You can also browse to the URL that you used in your Splunk provider (right-hand menu ☰ > system > web api) which will be something like https://instancename:8089/services/search/jobs (where instancename is your Splunk instance) to see a visual API explorer. Click on services > search jobs you can see the job numbers of any recurring jobs in Splunk, which can then be used in SquaredUp see API explorer.
• Only SCOM administrators are able to create or edit Web API tiles.
• Use custom labels to improve data formatting see How to use custom labelling
• You can view a list of the properties available for the search query by going to the monitored entity perspective. (Although displayName is not listed, it is also available for you to use).