Apica Alerts can be delivered to any service through a Custom Webhook Integration.

Custom Webhook Targets allow you to

Set up incident alerts to any system that supports HTTP(S)
Create calls with multiple steps for both Trigger and Resolution of incidents
Trigger multiple API calls to execute a set of actions

Overview

Important: Define your Webhook Placeholders first
Add the Custom Target

Note: For delivery to work, there needs to exist a receiving Webhook to handle the request.

Step

Screenshot

Add WebHook Target

Locate the Webhook symbol which can be found next to any alarm recipient.
Click the symbol to start to Add WebHook Target
A Service and Target Name dialog will appear at the bottom of the Recipient.
Select Custom from the drop-down box.

The Custom Webhook Target Dialog

The detailed steps for filling in this dialog box have their own page, Adding a Custom Service Target.

Adding a Custom Service Target

If your Webhook-enabled application is not listed (ServiceNow, OpsGenie, Slack, Splunk), you can easily create a custom Webhook target for delivering alerts to any predefined Webhook destination.

For each target/destination, there are options to send various alerts, in selected formats to them.

PagerDuty
Webhook
Email
SMS

In the following screenshot, the DocDemo Target/User has a few alerts now being sent to him, including a custom Webhook-enabled alert.

The Webhook icon indicates the kind of alert it is.

You can also see the PagerDuty, SMS, and email alerts icons next to the names of the alerts.

Steps to create a Custom Webhook Target

Note: Before starting, for Custom Webhook delivery to work, you need to have a working Webhook in your service.

Step	Screenshot
Create a Custom Service Target by clicking the WebHook button
A blank target is created
Select the Service
The Service section allows you to select the service type that we are going to send an alert to. The service should already be established to send the Webhook to and the authentication for the webhook service to use should already be enabled.	Open the Service menu Choose the Custom service
The following steps refer to the following Custom Service dialog
Set a Target Name The target name identifies the target in Synthetic Monitoring. Enter a Service Name for display. Required! This field is required in Apica Synthetic. The exact contents of the field may depend on the requirements of your service.	General - Service Name In the Service Name section, you can provide the ID or name of the service that should receive the alerts.
Add Target Details
Set Authentication Set your Authentication, including Basic Authentication OAuth2	Global Authentication In the Global Authentication section, you can choose and configure the type of authentication to use for alert delivery.
Request Headers Define the HTTP Header Type and value pairs to attach to different request types.
Request Types Send the Service if the request is a Trigger, Resolution, Authentication, or a refreshed access token	T (Trigger): Trigger an alert. R (Resolve): Resolve an alert. A (Authenticate): Authenticate using the access token. F (Refresh): Request with refresh token for a new access token.
Header Content Multiple headers can be added by clicking the Add button	Name: The name of the header. Value: The value of the header.
Response Parameters Example: A RESTful service has the `incident_id` parameter. This parameter should be saved for use when resolving the incident. Multiple parameters can be added by clicking the Add button	The Response Parameters section lets you define parameters in responses to store for use in future requests. XPath: The XPath for the parameter in the response. Example: `/response/sessionKey`. Placeholder: A Webhook Placeholder to be replaced with the actual value. Example: `#SESSION_KEY#` Is Auth?: Indicates (1) if the parameter is used for authentication and (2) that the parameter has an expiration time. Any time the Alerter fails to authenticate, it will clean up all values for this parameter.
Trigger Sequence	A procedure page on entering the Trigger Request Sequence configures the steps to begin a Webhook alert handling instance.
Resolve Sequence	A procedure page on entering the Trigger Resolve Sequence configures the steps to begin a Webhook alert handling instance when the alert has been resolved. i.e. to notify when the situation has been corrected or the event has ended.
(Done) Add Custom Webhook Target	When you are happy with all the settings, you can create the target. Click the Add Custom WebHook Target button

Trigger Sequence

The Trigger Request Sequence

This Trigger Request Sequence configures the steps to begin a Webhook alert handling instance. Basically,

What service do you want to send a Webhook to?
What information do you want to send to that service?
Under what Severity condition do you want to send it?
Confirmation: Do you want to send HTTP requests to the application to see if it returns an expected response?
- Do you want this to repeat? If so, under what conditions should it stop repeating?

Step	Screenshot
1. Define the Trigger URL What Webhook-enabled Trigger URL field is Apica going to use to trigger the alert? It will send one of the selected Methods to this URL. Trigger URL is the URL to Access Method: HTTP Methods to use for calling the authentication server. `POST`/`GET`/`PUT`/`DELETE`/`PATCH`
2. Add the Webhook Payload for the Alert Target You can specify a content type in the Payload section to include as a payload for the trigger request. Content-Type: The MIME Type for the data. `application/json / www-form-urlencoded` Data: Payload content. The data field defines the message content for the payload. You may use Message Placeholders in the data. Severity: A combination of Severity indicators. The severity field lets you select which severity type(s) to attach to the alert. F: Fatal E: Error W: Warning
Data Payload Example 01	A string containing key-value pairs for the content. `username=user&title=Resolved Alert!` {`"message": "%SEV% Resolved Alert for check '%CHECK_NAME%' at %TT%: %M%"`}
Data Payload Example 02	A simple alert text string: `Resolved! Notification of a Resolved Error`
3. Add Sub Requests As Needed You can add additional URL calls to the request sequence in the Sub Requests section. Some services require more than one call to their REST API to trigger an alert event. Multiple sub-requests can be added by clicking the Add button URL: URL to access. Method: HTTP Methods to use for calling the authentication server. `POST/GET/PUT/DELETE/PATCH` Repeat: Indicates if the call should be repeated until the Condition expression is matched. Condition: An XPath expression representing the condition to match when the Repeat checkbox is marked. Example: `/data/success=true`.
4. Add if Sub Requests Repeat and if they have a Repeat Condition Checking the Repeat box setting makes the subrequest step repeat until a Condition is met. When you want to do this, you need to set repeat on so a GET will be issued until the desired response is collected. Note: The condition needs to be an XPath expression indicating the response's desired value.
5. Add Headers You can define an HTTP Header in the Headers section to attach to the subrequest. Name: The name of the header. Value: The value of the header. Multiple headers for each sub-request can be added by clicking the green Add button.

Resolve Sequence (Optional)

The Resolve Request Sequence (Optional)

This Resolve Request Sequence configures the steps to begin a follow-up Webhook alert handling instance to handle a resolution. Basically: this allows a service to handle a resolution of an earlier Trigger alert.

This is an optional Sequence. If you only want your receiving service to handle the resolve, you do not need to add a resolve request.

It asks for most of the same information as in the Trigger Sequence except without a Severity.

What service do you want to send a Webhook to?
What information do you want to send to that service?
Confirmation: Do you want to send HTTP requests to the application to see if it returns an expected response?
- Do you want this to repeat? If so, under what conditions should it stop repeating?

Step	Screenshot
1. Define the Resolve URL What Webhook-enabled Resolve URL field is Apica going to use to stop the alert? It will send one of the selected Methods to this URL. The Resolve URL is the URL to Access Method: HTTP Methods to use for calling the authentication server. `POST`/`GET`/`PUT`/`DELETE`/`PATCH`
2. Add the Webhook Payload for the Alert Target You can specify a content type in the Payload section to include as a payload for the resolve request. Content-Type: The MIME Type for the data. `application/json / www-form-urlencoded` Data: Payload content. The data field defines the message content for the payload. You may use Message Placeholders in the data.
Data Payload Example 01	A string containing key-value pairs for the content. `username=user&title=Alert!` {`"message": "%SEV% alert for check '%CHECK_NAME%' at %TT%: %M%"`}
Data Payload Example 02	A simple alert text string: `Alert! Notification of an Error`
3. Set Resolve Sub Requests As Needed You can add additional URL calls to the request sequence in the Sub Requests section. Some services require more than one call to their REST API to resolve an alert event. Multiple sub-requests can be added by clicking the Add button. URL: URL to access. Method: HTTP Methods to use for calling the authentication server. `POST/GET/PUT/DELETE/PATCH` Repeat: Indicates if the call should be repeated until the Condition expression is matched. Condition: An XPath expression representing the condition to match when the Repeat checkbox is marked. Example: `/data/success=true`.
4. Add if Sub Requests Repeat and if they have a Repeat Condition Checking the Repeat box setting makes the subrequest step repeat until a Condition is met. When you want to do this, you need to set repeat on so a GET will be issued until the desired response is collected. Note: The condition needs to be an XPath expression indicating the response's desired value.
5. Send Sub Request Headers You can define an HTTP Header in the Headers section to attach to the subrequest. Name The name of the header. Value: The value of the header. Multiple headers for each sub-request can be added by clicking the green Add button.

Custom Webhook Examples

Custom Webhook w/ message placeholder (Microsoft Teams)

For instructions on setting up the URL which will be used as both the “Trigger URL” and the “Resolution URL”, refer to the official Microsoft documentation here.

Trigger URL: https://apicasystem.webhook.office.com/webhookb2/{longGuidString} /IncomingWebhook/{longGuidString}

Resolution URL: https://apicasystem.webhook.office.com/webhookb2/{longGuidString} /IncomingWebhook/{longGuidString}

Data: For both the trigger and resolution request sequences, the placeholder %CHECK_ID% is used along with a custom message to inform the user that the alert has been triggered/resolved.

Example alert trigger/resolution within a configured Microsoft Teams channel:

Custom Webhook w/ message placeholder (OpsGenie)

Trigger URL: https://api.opsgenie.com/v2/alerts?apiKey={opsGenieApiKey} (opsGenieApiKey is generated on the OpsGenie end)

Data:

{
   "message":"%CHECK_NAME%",
   "alias":"%CHECK_ID%",
   "description":"%CHECK_DESCRIPTION%",
   "responders":[
      {
         "name":"SAT",
         "type":"team"
      }
   ],
   "visibleTo":[
      {
         "name":"SAT",
         "type":"team"
      }
   ],
   "priority":"P1",
   "user":"ASM"
}

This JSON data is an example which we’ve created based on your current OpsGenie integration. The payload can be customized by referring to this documentation: https://docs.opsgenie.com/docs/alert-api#create-alert

%CHECK_NAME%, %CHECK_ID%, and %CHECK_DESCRIPTION% are message placeholders. Alias must be %CHECK_ID% in order for the resolution request to function correctly.

Resolve URL: https://api.opsgenie.com/v2/alerts/%CHECK_ID%/close?identifierType=alias&apiKey={opsGenieApiKey} (opsGenieApiKey is generated on the OpsGenie end)

%CHECK_ID% is a string which dynamically grabs the check ID of the check, which has ALSO been set as the Alias of the alert. It is possible to close Alerts via alert Alias, which we defined to be the Check ID in the Trigger Sequence request body. Thus, by specifying the Check ID as both the alias of the alert, we are able to dynamically identify and close the alert we created before. “identifierType=alias” is required.

Data (resolution request):

{“user”:”ASM”}

The response body of the resolution alert cannot be empty. Thus, at least one property must be present in the JSON body, although the POST call does not require any data from the OpsGenie end. It is possible to add a resolution note as well; see https://docs.opsgenie.com/docs/alert-api#close-alert for more details concerning closing alerts.

Custom Webhook w/ message placeholder, custom webhook placeholders (Slack)

Response Parameters: although the UI asks for the XPath for finding XML, it is also capable of finding JSON properties with the XPath. In this instance, we want to capture the value of the URL property of the HTTP response of our Sub Request (the api-wpm.apicasystem.com request).

As such, we will use #url# to display the “url” property of the api-wpm API call in our resolution message.

Trigger Request: the initial API call we make when the Alert is Triggered. Generated from the Slack end. Example: https://hooks.slack.com/services/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

Data: Note that the data contains a Message Placeholder, %CHECK_NAME%, which allows us to display dynamic data in the Slack message body.

Severity: We can have different Payload data (that is, different request messages) for different severities if we so desire. That is not needed for our use case, so we will use one payload for Warning, Error, and Fatal.

Sub Requests: After the initial request to https://hooks.slack.com is made, we want to get ADDITIONAL information about the check from an API call which gives us more check data. We store that data in the Response Parameter we explained above. Example: https://api-wpm.apicasystem.com/v3/checks/{check_id}?auth_ticket={auth_ticket}

Resolution Request: the final API call we make when the Alert is Resolved. Note the usage of both Message and Webhook placeholders to give our resolution message that dynamic data we need. Generated from the Slack end. Example: https://hooks.slack.com/services/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

Here are examples of the request/resolution messages that are sent to Slack based on the above configuration:

Custom Target (Webhook Integration)

Overview

Add WebHook Target

The Custom Webhook Target Dialog

Adding a Custom Service Target

Steps to create a Custom Webhook Target

Set a Target Name

General - Service Name

Set Authentication

Global Authentication

Request Headers

Request Types

Header Content

Response Parameters

Trigger Sequence

Resolve Sequence

(Done) Add Custom Webhook Target

Trigger Sequence

The Trigger Request Sequence

1. Define the Trigger URL

2. Add the Webhook Payload for the Alert Target

Data Payload Example 01

Data Payload Example 02

3. Add Sub Requests As Needed

4. Add if Sub Requests Repeat and if they have a Repeat Condition

5. Add Headers

Resolve Sequence (Optional)

The Resolve Request Sequence (Optional)

1. Define the Resolve URL

2. Add the Webhook Payload for the Alert Target

Data Payload Example 01

Data Payload Example 02

3. Set Resolve Sub Requests As Needed

4. Add if Sub Requests Repeat and if they have a Repeat Condition

5. Send Sub Request Headers

Custom Webhook Examples

Custom Webhook w/ message placeholder (Microsoft Teams)

Custom Webhook w/ message placeholder (OpsGenie)

Custom Webhook w/ message placeholder, custom webhook placeholders (Slack)