Webhooks v3 event delivery process

In the ​Akamai​ Identity Cloud, events are generated on a regular basis. These events, which include information that specifies the organization and the application associated with each event, are transported across the network to an Identity Cloud data warehouse. From there, ​Akamai​ personnel can review the event data to do such things as look for anomalies and problems, fine-tune performance and availability, and help plan for new product features and capabilities. 

As they travel to the data warehouse, events are inspected by apps designed to pick out certain events. Among these apps are webhook subscriptions. Webhook subscriptions are configured to look for specific events generated for a specific customer. For example, you might have a subscription that looks for all entityCreated, entityUpdated, and entityDeleted events associated with the customer 00000000-0000-0000-0000-000000000000. Let’s suppose that the following set of events come down the event pipeline

Customer IDEvent Type
11111111-0000-0000-0000-000000000000emailSent
22222222-0000-0000-0000-000000000000entityDeleted
11111111-0000-0000-0000-000000000000entityUpdated
11111111-0000-0000-0000-000000000000accessGrantIssued
22222222-0000-0000-0000-000000000000captureApplicationDelete

What will our subscription do when it sees these events? Nothing: after all, none of those events involve customer 00000000-0000-0000-0000-000000000000 and one of the specified event types (entityCreated, entityUpdated, or entityDeleted). Like any good event subscription, our example subscription ignores everything that isn’t of interest to it. 

Now, let’s suppose that the next set of events to pass through the pipeline include these:

Application IDEvent Type
22222222-0000-0000-0000-000000000000captureApplicationDeleted
22222222-0000-0000-0000-000000000000entityDeleted
00000000-0000-0000-0000-000000000000entityUpdate
00000000-0000-0000-0000-000000000000entityDeleted
11111111-0000-0000-0000-000000000000emailSent

As you can see, this time there are two events of interest: an entityUpdated event associated with customer 00000000-0000-0000-0000-000000000000and an entityDeleted event associated with the same customer. That means that our webhooks subscription is going to spring into action.

And what does it mean for a webhooks subscription to “spring into action?” Well, for one thing, it means that the event will be packaged as a security event token (SET). For another, the event state will be changed to awaiting-executing before the event is deposited in the event store. Changing the event state is an indication that the event is ready to be delivered to a customer’s listener endpoint.

At that point the event simply waits for delivery, with wait time typically measured in a matter of seconds. After this short wait, the event is sent (as an HTTP request) to the listener endpoint specified in the webhooks subscription. In turn, one of three things will happen:

  • The request is accepted and delivery is made.
  • The request is accepted, but delivery is not made.
  • The request is not accepted and delivery is not made.

How does Webhooks v3 respond in these situations? In situation 1 (the request is accepted and delivery is made) the response is pretty straightforward. If delivery succeeds, the event’s event state is changed to success and the case is closed: the system ignores any events marked as success. (Why? That’s right: because those events have already been delivered.) 

Things get a little more complicated when delivery fails: the actions taken by Webhooks v3 depend on the reason why delivery failed. (For a more detailed look at which failure types result in which actions, see Undelivered events.) Based on the reason for the failure, one of the following happens:

  • The event state is changed to awaiting-retry. Effectively, this is the same as setting the event state to awaiting-executing: it means that the event is ready for delivery. So why not just set the event state to awaiting-executing? As it turns out, there’s a limit on how many times a delivery attempt will be tried before the event state is set to failure: setting the state to awaiting-retry helps differentiate new events from existing events that have failed delivery at least once.

  • The event state is changed to failure. When an event is marked as failure, no additional delivery attempts are made; that’s because the response from the webhooks listener suggests that delivery can’t be made. When an event enters the failure state, event deliverers simply ignore it. However, there is a 7-day period during which a failed event can still be viewed in the event store, and can still be retrieved by using the event redelivery service

The different Webhooks event states are described in more detail in the following table:

StateDescription
awaiting-executingThe event has been packaged as a webhooks notification and is ready to be delivered. When it’s time to deliver this event, the state will almost always be changed to executing. The one exception? If it turns out that the associated webhooks subscription has been disabled since the time the event entered the event store. In that case, the event is immediately be marked as failure and no delivery attempt is made.

executingThe event is currently being delivered. After the delivery attempt is made, one of three things will happen:

* If delivery was made, the event state is changed to success.
* If delivery was not made but an HTTP status code of 4xx or 5xx was returned (and the allowed number of retry attempts have not been exhausted), the event state is changed to awaiting-retry.
* If delivery was not made and: 1) the retry attempts have been exhausted; or, 2) a status code other than 4xx or 5xx was returned, the event state is changed to failure.

If a subscription is disabled while delivery is in progress, Webhooks v3 will attempt to complete the delivery. If delivery fails, however, then the event state will immediately be changed to failure.

successThe event has been delivered, and Webhooks v3 takes no further action.

awaiting-retryThe event has gone through at least one failed delivery attempt, and has been scheduled for another try. The awaiting-retry state functions exactly like the awaiting-executing state: the only difference is that setting the state to awaiting-retry tells you that the event has been involved in at least one failed delivery attempt.

failureNo attempt is made to deliver this event. An event enters the failure state if any of the following conditions are true:

* Webhooks v3 has tried, and failed, to deliver the event 6 times.
* Webhooks v3 tried to deliver the event, and a status code other than 4xx or 5xx was returned.
* The target webhooks subscription has been disabled.

Here’s a typical webhooks flow, from start to finish:

Event delivery frequency

Officially, events are delivered as they occur, and in “near real-time.” That means two things. First, events are not queued up and sent in batches. When a new user account is created, an entityCreated event is generated. If there’s a webhooks subscription listening for entityCreated events then a webhooks notification is sent. If another account is created a second or two later, a second webhooks notification is sent. If 1,000 more accounts are created in the next hour or so then 1,000 more webhooks notifications will be sent. That’s the nature of webhooks: each event triggers its own notification.


📘

Which is one reason why you might want to limit event subscriptions to a few specified events, and not set up subscriptions to respond to every event generated for your applications.


Let’s say an event of interest just occurred; how long will it take before Webhooks v3 sends a notification of that event? To be honest, the answer to that can be a bit complicated. Under the covers, things aren’t always as easy as: 

  1. An event occurs.
  2. A webhook notification is sent and received.

What kinds of things can complicate the process? For one, the network might be congested at the Akamai end: that means it could take longer for events to get pushed out to the customer. Alternatively, things might be congested at your end. For example, suppose your listener endpoint can only handle 10 notifications per minute (which, admittedly, is an unrealistically-low value). Let’s further suppose that 100 people just registered on your website. That means that 100 event notifications are on their way to an endpoint that can only handle 10 such notifications per minute. In a case like that, delays are inevitable.

We should also note that ​Akamai​ can’t guarantee that notifications will be delivered in the exact order in which those events occurred. Suppose events 1, 2, 3, 4, and 5 occur in rapid-fire succession. Typically you’ll receive your event notifications in that exact same order: 1, 2, 3, 4, and 5. However, it’s possible that you might receive the notifications in this order: 1, 2, 3, 5, 4. That’s just the way the system works.

To help keep the system moving as efficiently, and as quickly, as possible, the Identity Cloud constantly monitors for such things as:

  • The number of events currently in the event pipeline.
  • The number of outgoing notifications.
  • The success and failure rates for webhook deliveries.
  • The distribution rate for delivery times.