Introduction to Webhooks v3 bulk export of failed events

📘
Webhooks v3 bulk export of failed events is a new Identity Cloud feature currently in limited availability. Because the feature isn't in general release, that means that both the product, and this documentation, are subject to change at any time.

Bad things sometimes happen to good people, and to good companies. For example, despite your best efforts and despite your best intentions, circumstances – network connectivity issues, hardware failures, power outages – can conspire to prevent webhook notifications from being delivered to your listener endpoint. Yes, someone did just create an account on your website, but even though you’ve configured a webhook subscription to notify you any time that happens the notification never reached your listener endpoint. Needless to say, that defeats the whole purpose of using webhooks in the first place.

Failed deliveries like this are few and far between, but they can happen. Because of that, Webhooks v3 has a /redelivery endpoint that enables you to retrieve events that couldn’t be delivered. For example, suppose a user creates a new account on your site, but, thanks to a glitch of some kind, the corresponding webhook notification isn’t delivered. That’s an inconvenience, yes, but events that can’t be delivered don’t disappear, at least not right away. Instead, copies of all your events – included failed events – are stored in the Webhooks v3 event store for 7 days. At any time during those 7 days you can use the /redelivery endpoint to access those failed events and schedule them for redelivery.

That makes the /redelivery endpoint extremely useful … but only if you have a handful of undelivered event notifications. However, suppose a power failure resulted in several hundred, or maybe even several thousand, events going undelivered. Can you use the /redelivery endpoint to retrieve those events? In theory, yes, you can. But only as long as you’re willing to make several hundred, or several thousand API calls, one for each failed event. (The /redelivery endpoint is limited to returning one undelivered event per API call.) In case of a catastrophic failure, that’s not exactly the ideal solution.

Fortunately, the new process (released in March 2023) provides a way to retrieve all your failed events, and to do so by making just a couple API calls. Here’s how the process works:

When an event occurs, an event notification is sent your way, and a copy of the event is sent to the event store. If the notification can’t be delivered (following the Webhooks v3 event delivery process), the event store event is marked as failure.

None of that’s new. What is new, however, is the fact that a copy of each failed event is now sent to the failed events table as well. As the name implies, this table only stores failed events: events that were delivered or events that are still in the process of being delivered aren’t copied to the failed events table. The failed events table also serves as the repository for the feature.

For example, suppose Webhooks sends you six notifications: four of these notifications were successfully delivered and two were not. In that case, your event store and your failed events table will look something like this:

Events remain in the event store and in the failed events table for 7 days before being deleted. That means that you have some time, but not an unlimited amount of time, to retrieve events that couldn’t be delivered.
As noted, after 7 days events are automatically removed from the event store and from the failed events table. However, at any time during those 7 days you can use the /export endpoint and the POST method to export all the events in the failed events table to an Amazon Web Services S3 bucket. “Exporting” the failed events means that Webhooks v3 generates a file (using the NDJSON format) that contains the payload of those failed events and then makes that file available for downloading.
After the failed events have been exported you can use the /download endpoint to retrieve the file. And after the file has been retrieved you can parse out the individual event notifications and then do whatever it is you typically do with these notifications (e.g., store them in a webhooks database).

Here’s a more graphical look at the process:

Although that’s a somewhat-cursory overview, that’s basically how the process works. In addition to that, however, here are a few other things to keep in minds:

For rate limiting purposes, jobs are based on your subscription ID, and each subscription can only have one active job at any given time. For example, suppose you kick off an job for webhook subscription A, then immediately try to start a second job, this one for webhook subscription B. That second API call fails: you can’t start a new job until the previous job for that subscription has completed. (Fortunately, you can use the /exports endpoint and the GET method to see if you have any active jobs.)
Jobs are carried out on a subscription-by-subscription basis For example, suppose you have 3 webhook subscriptions: subscriptions A, B, and C. If you need to retrieve failed events for all 3 of those subscriptions you’ll need to make 3 different API calls, one for each subscription. There’s no way to make a single API call and retrieve the failed events for all your subscriptions.
At the risk of repeating ourselves. events remain in the failed events table for 7 days. After that, these older events are automatically deleted. That means that you can’t export an event that occurred three weeks ago: by that time the event has already been deleted.
The download files written to the Amazon S3 bucket also expire after 7 days. If you do an export you’ll need to download that file sometime in the next 7 days. If you wait three weeks then, again, the file won’t be there: it will have automatically been deleted.
After an event has been exported that event is marked with a timestamp indicating that it’s been exported. When Webhooks v3 carries out an export operation it ignores any events in the failed events table that have already been exported. That prevents you from inadvertently exporting the exact same event multiple times.
Although you can only export an event once you can download the file that contains that event as many times as you wish. For example, suppose you download file A and then accidentally delete that file. That’s fine: just download file A a second time.

Note, however, that Webhooks v3 doesn’t keep track of how many times you’ve downloaded a given file, assuming you’ve even downloaded it at all. It’s up to you to make sure you get copies of all your download files, and to make sure that you don’t upload the same download file multiple times. That would result in you having duplicate events in your events database.
Files containing your exported events are limited to a maximum size of 6 GB. That’s unlikely to be a problem: it would take a lot of event notifications to create a file larger than 6 GB. For example, a sample file containing 10,000 events checked in at 6.3 MB. Still, it’s something to keep in mind.

See also