The NDJSON file format

📘
Webhooks v3 bulk export of failed events is a new Identity Cloud feature currently in limited availability. Because the feature isn't in general release, that means that both the product, and this documentation, are subject to change at any time.

The files you download following a job are formatted by using NDJSON: newline-delimited JSON. What’s NDJSON and why is this used for export files? To answer that question, let’s start by taking a look at a standard JSON file. For illustration purposes, we’ve chosen a simple file containing two fields and two records (firstName and lastName) and two records:

[
  {
    "firstName": "Karim",
    "lastName": "Nafir"
  },
  {
    "firstName": "Maria",
    "lastName": "Torres"
  }
]

And here’s that same file in NDJSON format:

{"firstName":"Karim","lastName":"Nafir"}
{"firstName":"Maria","lastName":"Torres"}

As you can see, the NDJSON file contains the same two fields and the same two records. With this format, however, most of the indenting and most of the white space commonly found in a JSON file has been eliminated. Each record is still formatted using JSON, but each record has also been “compressed” so that it fits on a single line. In an NDJSON file each line is a separate JSON object separated from the next JSON object by using the newline character (\n). (Which, of course, is why it’s called newline-delimited JSON: individual lines are delimited by using the newline character.)

Admittedly, that might not seem like that big of a deal. But consider a more complicated example. At a glance, how many records are in this file:

[
  {
  "type": "text",
  "name": "myCustomTextField",
  "schemaAttribute": "displayName",
  "socialProfileData": "profile.displayName",
  "label": {
    "key": "b6ced670-7140-4446-9839-da3474860b1a"
	},
  "tip": {
    "key": "8b93448a-6f00-448c-952b-0f1536107cf7"
	},
  "placeholder": {
    "key": "6e15067b-2ca5-43c3-af96-930766d63375"
	},
  "validation": [{
    "rule": "required",
    "value": true,
    "message": {
      "key": "bb2b21a1-98df-4dce-84f7-534013c46225"
	  }
	},
	{
  "rule": "unique",
  "value": true,
  "message": {
    "key": "63f7c18f-521a-4b4f-94c4-0b04b870c82e"
	  }
	}
   ]
}
]

As it turns out, there’s just one record, something that’s difficult to discern with all the nested values. And if there were several hundred records similar to this? Well ….

Now here’s that the same file (albeit truncated to fit on a single line) in NDJSON format:

{"type":"text"..."message":{"key":"63f7c18f-521a-4b4f-94c4-0b04b870c82e"}}]}

That’s the value of NDJSON. With the NDJSON format it’s easy to count the records in a file (just count the number of lines) and, because each line is a separate and distinct record, it’s easy to know where one record ends and the next begins. And you get this ease of use without losing any of the advantages that come along with using JSON for storing data.

So couldn’t you just use the comma-separated values CSV) format to accomplish the same thing? Maybe. However, CSV files have some definite disadvantages compared to NDJSON files, in large part because there’s no agreed-upon format for CSV files. For example, some CSV files include fields name; some don’t. Some CSV files support Unicode characters; some don’t. Different CSV files have different ways of handling commas and blank spaces, and it’s often difficult to tell whether a value such as "true" is meant to be a string value or a Boolean value. NDJSON files overcome most of these limitations.

See also