Bulk export of failed webhook events

Overview

Identity Cloud Webhooks provides a way for you to build integrations by receiving notifications at your endpoint. If your endpoint is having a short outage, Webhooks will automatically retry 5 times for up to 1 day. If you missed a handful of notifications beyond the retry window, you can trigger single redelivery through the API. However, if you had a long outage and missed many notifications, you have the option to initiate a bulk export through the API.

Bulk export is meant for disaster recovery, and will include the payload of all failed notifications within the last 7 days. Once the failed events have been exported, you will be able to download them as a single file. You can then ingest those events back into your system in an offline manner.

Limitations

You can only export events that failed delivery within the last 7 days.
You can only export events that have reached the failure state.
You can only export a failed event once.
You can only complete a bulk export once in 24 hours, but you can download multiple times.
You can only have one active bulk export at a time.
If the system is too busy, you may need to retry the bulk export later.

API

The bulk export API is based on asynchronous jobs. After starting a bulk export job, you will receive an ID you can use to check the export status. Once available, you can download a single file with all exported events.

Action	Method	Endpoint
List export jobs	GET	/{customer-id}/webhooks/subscriptions/{subscription-id}/exports
Start bulk export	POST	/{customer-id}/webhooks/subscriptions/{subscription-id}/exports
View status of bulk export	GET	/{customer-id}/webhooks/subscriptions/{subscription-id}/exports/{export-id}
Cancel a bulk export	DELETE	/{customer-id}/webhooks/subscriptions/{subscription-id}/exports/{export-id}
Download exported events	GET	/{customer-id}/webhooks/subscriptions/{subscription-id}/exports/{export-id}/download

Export time window

When you start a bulk export, all events that have been created in the system within the last 7 days and are in a failed state will be exported. There is no option to change the time window.

Export states

An export job may be in one of the following states:

processing: The system is working on the export job.
succeeded: The export has completed and the events are ready for downloading.
canceled: The export has been manually canceled.
failed: The export job could not be completed, and you may try again. If the problem persists, please contact Akamai Identity Cloud Support.

Bulk export states

Export format

The exported events will be in NDJSON format, where each entry (separated by a newline) contains the decoded body of the webhook SET payload. This means that each line is an event in JSON format that you can read directly, without having to decode or verify any signatures.

For example:

{"iss":"https://v1.api.us.janrain.com/00000000-0000-0000-0000-000000000000/webhooks","iat":1568312175,"jti":"519a08c4-3016-4cad-9b10-feb49fcd133a","aud":["https://example.com/path/to/endpoint"],"txn":"00000000-0000-0000-0000-000000000000","toe":1559372000,"events":{"entityCreated":{"captureApplicationId":"aaaaaaaaaaaaaaaaaaaaaaaaaa","captureClientId":"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa","entityType":"user","globalSub":"capture-v1://us.janraincapture.com/aaaaaaaaaaaaaaaaaaaaaaaaaa/user/00000000-0000-0000-0000-000000000000","sub":"00000000-0000-0000-0000-000000000000","id":"00000000-0000-0000-0000-000000000000"}}}  
{"iss":"https://v1.api.us.janrain.com/00000000-0000-0000-0000-000000000000/webhooks","iat":1568312175,"jti":"519a08c4-3016-4cad-9b10-feb49fcd133a","aud":["https://example.com/path/to/endpoint"],"txn":"00000000-0000-0000-0000-000000000000","toe":1559372000,"events":{"entityDeleted":{"captureApplicationId":"aaaaaaaaaaaaaaaaaaaaaaaaaa","captureClientId":"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa","entityType":"user","globalSub":"capture-v1://us.janraincapture.com/aaaaaaaaaaaaaaaaaaaaaaaaaa/user/00000000-0000-0000-0000-000000000000","sub":"00000000-0000-0000-0000-000000000000","id":"00000000-0000-0000-0000-000000000000"}}}  
{"iss":"https://v1.api.us.janrain.com/00000000-0000-0000-0000-000000000000/webhooks","iat":1568312175,"jti":"519a08c4-3016-4cad-9b10-feb49fcd133a","aud":["https://example.com/path/to/endpoint"],"txn":"00000000-0000-0000-0000-000000000000","toe":1559372000,"events":{"entityUpdated":{"captureApplicationId":"aaaaaaaaaaaaaaaaaaaaaaaaaa","captureClientId":"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa","entityType":"user","globalSub":"capture-v1://us.janraincapture.com/aaaaaaaaaaaaaaaaaaaaaaaaaa/user/00000000-0000-0000-0000-000000000000","sub":"00000000-0000-0000-0000-000000000000","id":"00000000-0000-0000-0000-000000000000"}}}

After formatting, the first entry would look like this:

{  
  "iss": "https://v1.api.us.janrain.com/00000000-0000-0000-0000-000000000000/webhooks",  
  "iat": 1568312175,  
  "jti": "519a08c4-3016-4cad-9b10-feb49fcd133a",  
  "aud": [  
    "https://example.com/path/to/endpoint"  
  ],  
  "txn": "00000000-0000-0000-0000-000000000000",  
  "toe": 1559372000,  
  "events": {  
    "entityCreated": {  
      "captureApplicationId": "aaaaaaaaaaaaaaaaaaaaaaaaaa",  
      "captureClientId": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",  
      "entityType": "user",  
      "globalSub": "capture-v1://us.janraincapture.com/aaaaaaaaaaaaaaaaaaaaaaaaaa/user/00000000-0000-0000-0000-000000000000",  
      "sub": "00000000-0000-0000-0000-000000000000",  
      "id": "00000000-0000-0000-0000-000000000000"  
    }  
  }  
}

Walkthrough

Listing export jobs

We start by checking if we already have any bulk export jobs running for our subscription. We do this by sending a GET request to the /exports endpoint. Because we can only run one export job at a time, it can be helpful to know if a job is already in progress. In the example below, we do not have any jobs yet (1).

Request:

curl --request GET \
  --header 'Authorization: Bearer Yh1cEVzUOy0lZNaBxcGbQRQ6qFaRvW0UBqTD81p76ymSNN73P6eulMZFX2-MkN1h' \
  --url https://v1.api.us.janrain.com/00000000-0000-0000-0000-000000000000/webhooks/subscriptions/c8c7fcb5-d44c-4226-b728-9d5660192143/exports

Response:

{
  "total": 0,  // (1)
  "_links": {
    "self": {
      "href": "/00000000-0000-0000-0000-000000000000/webhooks/subscriptions/c8c7fcb5-d44c-4226-b728-9d5660192143/exports"
    }
  },
  "_embedded": {}
}

Starting an export job

Next we initiate a bulk export job. We do this by sending a POST request to the /exports endpoint. The response will contain information about the export, including its initial state (1) and a link to the specific job for more details (2).

Request:

curl --request POST \
  --header 'Authorization: Bearer Yh1cEVzUOy0lZNaBxcGbQRQ6qFaRvW0UBqTD81p76ymSNN73P6eulMZFX2-MkN1h' \
  --url https://v1.api.us.janrain.com/00000000-0000-0000-0000-000000000000/webhooks/subscriptions/c8c7fcb5-d44c-4226-b728-9d5660192143/exports

Response:

{
  "id": "ffec4894-633f-4512-9ea6-5b680a584720",
  "createdAt": "2024-04-25T17:30:23.020304Z",
  "updatedAt": "2024-04-25T17:30:23.020304Z",
  "state": "processing", // (1)
  "_links": {
    "self": {
      "href": "/00000000-0000-0000-0000-000000000000/webhooks/subscriptions/c8c7fcb5-d44c-4226-b728-9d5660192143/exports/ffec4894-633f-4512-9ea6-5b680a584720" // (2)
    },
    "exports": {
      "href": "/00000000-0000-0000-0000-000000000000/webhooks/subscriptions/c8c7fcb5-d44c-4226-b728-9d5660192143/exports"
    }
  }
}

Keep in mind that if you already have a bulk export running or the system is too busy, you will receive HTTP Status 409, and you will have to try again later.

Checking the export job status

To get an up-to-date status of the job, we follow that job link. In the example below, the job has succeeded (1), and we have a link to download the exported events (2).

Request:

curl --request GET \
  --header 'Authorization: Bearer Yh1cEVzUOy0lZNaBxcGbQRQ6qFaRvW0UBqTD81p76ymSNN73P6eulMZFX2-MkN1h' \
  --url https://v1.api.us.janrain.com/00000000-0000-0000-0000-000000000000/webhooks/subscriptions/c8c7fcb5-d44c-4226-b728-9d5660192143/exports/ffec4894-633f-4512-9ea6-5b680a584720

Response:

{
  "id": "ffec4894-633f-4512-9ea6-5b680a584720",
  "createdAt": "2024-04-25T17:30:23.020304Z",
  "updatedAt": "2024-04-25T17:30:23.362008Z",
  "state": "succeeded", // (1)
  "exportedCount": 1,
  "byteCount": 319,
  "_links": {
    "self": {
      "href": "/00000000-0000-0000-0000-000000000000/webhooks/subscriptions/c8c7fcb5-d44c-4226-b728-9d5660192143/exports/ffec4894-633f-4512-9ea6-5b680a584720"
    },
    "exports": {
      "href": "/00000000-0000-0000-0000-000000000000/webhooks/subscriptions/c8c7fcb5-d44c-4226-b728-9d5660192143/exports"
    },
    "download": {
      "href": "/00000000-0000-0000-0000-000000000000/webhooks/subscriptions/c8c7fcb5-d44c-4226-b728-9d5660192143/exports/ffec4894-633f-4512-9ea6-5b680a584720/download" // (2)
    }
  }
}

Downloading

Once a job succeeds, we can fetch the exported events as a single file through the /download endpoint. Your exported events will be available for download for 7 days.
Note that here we are saving the output directly to a file, since there could be thousands or even millions of events exported.

curl --request GET \
  --header 'Authorization: Bearer Yh1cEVzUOy0lZNaBxcGbQRQ6qFaRvW0UBqTD81p76ymSNN73P6eulMZFX2-MkN1h' \
  --url https://v1.api.us.janrain.com/00000000-0000-0000-0000-000000000000/webhooks/subscriptions/c8c7fcb5-d44c-4226-b728-9d5660192143/exports/ffec4894-633f-4512-9ea6-5b680a584720/download \
  --output events.ndjson

When you check the resulting file, you may see something like the following:

{"aud": ["https://example.com/path/to/endpoint"], "iat": 1714066353, "iss": "https://v1.api.us.janrain.com/00000000-0000-0000-0000-000000000000/webhooks", "jti": "3b1ae254-6560-4996-8581-426a6ecd50a7", "toe": 1714066353, "events": {"test": {"id": "e460be5e-4cb4-40ec-83ce-68e928a83a7a"}}}
{"aud": ["https://example.com/path/to/endpoint"], "iat": 1714066355, "iss": "https://v1.api.us.janrain.com/00000000-0000-0000-0000-000000000000/webhooks", "jti": "6a4c2fb5-e293-4e99-9f04-253fcb3e76e9", "toe": 1714066355, "events": {"test": {"id": "186877ac-560a-4afc-a8ae-187adc70d86f"}}}

Next steps

Now that you have a file with failed events, you will need to ingest them into your system. How to do that is out of scope for this document, since it will change from system to system.

FAQ

Can I export non-failed events?

No. Since this feature is meant for disaster recovery, only failed events can be exported.

Can I choose the start/end period for exporting?

No. For convenience, we always export all failed events over the last 7 days.

Can I change the format of the exported events?

No. We always export in NDJSON format, with events already decoded and separated by a newline character.

Can you resend the failed events to my endpoint?

No. To avoid flooding your endpoint and causing another outage, we only support bulk export for offline processing. You can then ingest the events in a way that does not impact your system.

Can I re-export failed events?

No, but once you've exported failed events, you may download them multiple times for 7 days.

Can I download the failed events more than once?

Yes. You can only export failed events once per subscription, but once exported the file will be available for download for 7 days. You can download your exported events multiple times.

Can I clean up past export jobs?

Export jobs will be automatically cleaned up after 7 days.

Can I resume a canceled job?

No, but you can start a new export job after canceling.

What happens if there were no failed events to export?

If there were no failed events available for export, then the job status will show a count of zero. You will still get a download link, but it will be empty.

Why am I receiving HTTP Status 409 when starting a bulk export?

Webhooks only allows one bulk export to run at a time. If the system is too busy, you may also receive that error. In either case, you may retry the bulk export later.

Updated 3 months ago