Create data streams

Suggest Edits

Set up data streams, monitoring, and analytical components. Use them to gather delivery performance and health metrics about your traffic across multiple properties for better insight and improved security.

Get your properties

The properties you use in DataStream integrations need to have room to pick up a new data stream and be active on a network.

You can assign up to three data streams to a single property. To see how many data streams are assigned to a property, review its DataStream rule.
A single data stream can support up to 100 properties. Use the Get-DataStream command to get a list of its assigned properties.

Note: If you need, create a new property and activate it on a network.

Get a list of your properties for a specific group and determine which of them you'll use with your data stream.

Get-DatastreamProperties -GroupID 12345

propertyId   : 12345
propertyName : my-property-1
productId    : Object_Delivery
productName  : Object Delivery
hostnames    : {example-1.com}
contractId   : C-0N7RAC7

propertyId   : 98765
propertyName : my-property-2
productId    : Download_Delivery
productName  : Download Delivery
hostnames    : {example-2.com}
contractId   : C-0N7RAC7

Alternatively, you can run the Get-Property command from the Akamai.Property module to get the properties for a specific contract and group.

Get-Property -GroupID 12345 -ContractID 'C-0N7RAC7'

accountId         : act_A-CCT1234
contractId        : ctr_C-0N7RAC7
groupId           : grp_12345
propertyId        : prp_12345
propertyName      : my-property-1
latestVersion     : 1
stagingVersion    :
productionVersion :
assetId           : aid_12345

accountId         : A-CCT9876
contractId        : ctr_C-0N7RAC7
groupId           : grp_98765
propertyId        : prp_98765
propertyName      : my-property-2
latestVersion     : 1
stagingVersion    : 2
productionVersion :
assetId           : aid_98765

Make a note of the property you want to use to update its rule tree after you create a data stream.

Create a data stream

Average processing time: 90 minutes – 3 hours

To create a data stream, configure the following:

Properties to monitor in the stream.
Data sets that shape what's monitored and collected about your traffic. See Choose data sets for the available data set IDs.
A log destination. See Choose a destination for the available destinations and their configurations.
Other optional settings if needed.

Argument	Description
Required
`contractId`	Your contract's ID.
`groupId`	Your group's ID.
`properties`	A list of properties the data stream monitors. Data can only be logged on active properties.
`streamName`	The name for your stream.
`datasetFields`	A set of IDs for the data set fields within the product for which you want to receive logs. The order of the IDs defines their order in the log lines. For values, use the Data set parameters page to get the available fields for your product. For a complete list of field name IDs, see the Choose data sets section. You can also run the `Get-DatastreamDatasets` command for those details and optionally add the `-ProductID` parameter to this command to get data sets for a specific product.
`destination`	A destination to send log files to. See the Destinations page for features available for each destination. To set up each destination type, see the Choose a destination section.
`deliveryConfiguration`	A set that provides configuration information for the logs. Includes: `format`. Required. The format in which you want to receive log files, `STRUCTURED` or `JSON`. If you've used a delimiter, the format must be `STRUCTURED`. `frequency`. Required. A set that includes `intervalInSeconds`. The time in seconds after which the system bundles log lines into a file and sends the file to a destination. Possible values are `30` and `60`. `uploadFilePrefix`. The log file prefix to send to a destination. Maximum characters, 200. If unspecified, it defaults to `ak`. `uploadFileSuffix`. The log file suffix to send to a destination. Maximum characters, 10. If unspecified, it defaults to `ds`. `fieldDelimiter`. Sets a space as a delimiter to separate data set fields in log lines. Value is `SPACE`. If used, you must also use the format argument set to `STRUCTURED`.
Optional
`collectMidgress`	Whether to set the collection of midgress data. `true` to enable it.
`notificationEmails`	A list of email addresses to which the data stream's activation and deactivation status are sent.

Define your data stream settings in a variable as a hashtable, PSCustomObject, or JSON string, and then pass the variable to the -Body parameter.

Optionally, you can decide whether to activate the stream on making the request or later using the -Activate switch parameter. Note that only active streams can collect and send logs to their destinations.

Important: Because the data stream creation process can take a bit, provide the -Activate switch parameter as it removes a second round of processing for activation.

$myDataStream = @{
  streamName = "my-stream"
  groupId = 12345
  contractId = "C-0N7RAC7"
  notificationEmails = @(
    "jsmith@email.com"
  )
  properties = @(
    @{
      propertyId = 12345
    }
  )
  datasetFields = @(
    @{
      datasetFieldId = 1000
    },
    @{
      datasetFieldId = 1002
    },
    @{
      datasetFieldId = 1023
    }
  )
  deliveryConfiguration = @{
    frequency = @{
      intervalInSeconds = 30
  }
  format = "STRUCTURED"
  fieldDelimiter = "SPACE"
  }
  destination = @{
    destinationType = 'HTTPS'
    displayName = 'my-destination'
    compressLogs = true
    authenticationType = 'NONE'
    endpoint = 'https://example.com/post'
    contentType = 'application/json'
  }
  collectMidgress = true
}

New-DataStream -Body $myDataStream

$myDataStream = {
  "streamName": "my-stream",
  "groupId": 12345,
  "contractId": "C-0N7RAC7",
  "notificationEmails": [
    "jsmith@email.com"
  ],
  "properties": [
    {
      "propertyId": 12345
    }
  ],
  "datasetFields": [
    {
      "datasetFieldId": 1000
    },
    {
      "datasetFieldId": 1002
    },
    {
      "datasetFieldId": 1023
    }
  ],
  "deliveryConfiguration": {
    "frequency": {
      "intervalInSeconds": 30
    },
    "format": "STRUCTURED",
    "fieldDelimiter": "SPACE"
  },
  "destination": {
    "destinationType": "HTTPS",
    "displayName": "my-destination",
    "compressLogs": true,
    "authenticationType": "NONE",
    "endpoint": "https://example.com/post",
    "contentType": "application/json"
  },
  "collectMidgress": true
}

New-DataStream -Body $myDataStream

streamName            : my-stream
streamId              : 12345
streamVersion         : 1
groupId               : 12345
contractId            : C-0N7RAC7
notificationEmails    : {jsmith@email.com}
properties            : {@{propertyId=12345; propertyName=my-property-1}}
datasetFields         : {@{datasetFieldId=1000; datasetFieldName=CP code; datasetFieldJsonKey=cp}, 
                        @{datasetFieldId=1002; datasetFieldName=Request ID; datasetFieldJsonKey=reqId}, 
                        @{datasetFieldId=1023; datasetFieldName=Cookie; datasetFieldJsonKey=cookie}}
destination           : @{destinationType=HTTPS; displayName=my-destination; authenticationType=NONE; 
                        endpoint=https://example.com/post; compressLogs=False; contentType=application/json}
deliveryConfiguration : @{fieldDelimiter=SPACE; frequency=; format=STRUCTURED}
latestVersion         : 1
productId             : Object_Delivery
streamStatus          : INACTIVE
createdBy             : 1abcdefgh2ijklmn
createdDate           : 2/18/2025 3:50:47 PM
modifiedBy            : 1abcdefgh2ijklmn
modifiedDate          : 2/18/2025 3:50:47 PM

You can also define your data stream settings in a JSON file, load the file into a variable, convert it into a PowerShell object, and pipe it to the command.

$myDataStream = Get-Content ./myStream.json | ConvertFrom-Json -Depth 100

$myDataStream | New-DataStream

If you haven't activated your data stream upon create, you can run a separate command New-DataStreamActivation to do that by specifying a stream ID you want to activate.

New-DataStreamActivation -StreamID 12345

streamName            : my-stream
streamId              : 12345
streamVersion         : 1
groupId               : 12345
contractId            : C-0N7RAC7
notificationEmails    : {jsmith@email.com}
properties            : {@{propertyId=12345; propertyName=my-property-1}}
datasetFields         : {@{datasetFieldId=1000; datasetFieldName=CP code; datasetFieldJsonKey=cp}, 
                        @{datasetFieldId=1002; datasetFieldName=Request ID; datasetFieldJsonKey=reqId}, 
                        @{datasetFieldId=1023; datasetFieldName=Cookie; datasetFieldJsonKey=cookie}}
destination           : @{destinationType=HTTPS; displayName=my-destination; authenticationType=NONE; 
                        endpoint=https://example.com/post; compressLogs=False; contentType=application/json}
deliveryConfiguration : @{fieldDelimiter=SPACE; frequency=; format=STRUCTURED}
latestVersion         : 1
productId             : Object_Delivery
streamStatus          : ACTIVATING
createdBy             : 1abcdefgh2ijklmn
createdDate           : 2/18/2025 3:55:47 PM
modifiedBy            : 1abcdefgh2ijklmn
modifiedDate          : 2/18/2025 3:55:47 PM

Update a data stream

If you don't know your data stream ID, use the Get-DataStream to return the list of all data streams available to you.

For a specific data stream version, pass also the -StreamID parameters in the command. Optionally, pass the -Version parameter; without specifying it, the latest version is returned by default.

# Get all
Get-DataStream

# Get one
Get-DataStream -StreamID 12345 -Version 1

streamName    : my-stream-1
streamId      : 12345
streamVersion : 3
groupId       : 12345
contractId    : C-0N7RAC7
properties    : {@{propertyId=12345; propertyName=my-property-1}}
latestVersion : 3
productId     : Obj_Delivery
streamStatus  : ACTIVATED
createdBy     : 1abcdefgh2ijklmn
createdDate   : 2/18/2025 3:55:47 PM
modifiedBy    : 1abcdefgh2ijklmn
modifiedDate  : 2/18/2025 3:55:47 PM

streamName    : my-stream-2
streamId      : 98765
streamVersion : 1
groupId       : 12345
contractId    : C-0N7RAC7
properties    : {@{propertyId=98765; propertyName=my-property-2}}
latestVersion : 1
productId     : Adaptive_Media_Delivery
streamStatus  : INACTIVE
createdBy     : jasmith
createdDate   : 1/9/2025 6:00:17 AM
modifiedBy    : jasmith
modifiedDate  : 1/9/2025 6:00:17 AM

streamName            : my-stream-1
streamId              : 12345
streamVersion         : 1
groupId               : 12345
contractId            : C-0N7RAC7
notificationEmails    : {jsmith@email.com}
properties            : {@{propertyId=12345; propertyName=my-property-1}}
datasetFields         : {@{datasetFieldId=1000; datasetFieldName=CP code; datasetFieldJsonKey=cp}, 
      @{datasetFieldId=1002; datasetFieldName=Request ID; datasetFieldJsonKey=reqId}, 
      @{datasetFieldId=1023; datasetFieldName=Cookie; datasetFieldJsonKey=cookie}}
destination           : @{destinationType=HTTPS; displayName=my-destination; authenticationType=NONE; 
      endpoint=https://example.com/post; compressLogs=False; contentType=application/json}
deliveryConfiguration : @{fieldDelimiter=SPACE; frequency=; format=STRUCTURED}
latestVersion         : 1
productId             : Object_Delivery
streamStatus          : ACTIVATED
createdBy             : 1abcdefgh2ijklmn
createdDate           : 2/18/2025 3:55:47 PM
modifiedBy            : 1abcdefgh2ijklmn
modifiedDate          : 2/18/2025 3:55:47 PM

Once you have the required data stream, run one of these commands to make changes:

Set-DataStream. To update the latest version of the stream.

First, get a specific stream, save its output locally, and edit it according to your needs.

Notes:

The function doesn’t return sensitive attributes, such as a password or access key. If you need to update them, add them to your JSON file.

If you happen to have the 2051 (midgress status) data set field ID in your loaded JSON, remove that information before piping your edited object as it can’t be passed on the data set level.

Get-DataStream -StreamID 12345 | ConvertTo-Json -Depth 100 | Out-File -FilePath ./myStream.json

{
  "streamName": "my-updated-stream",
  "streamId": 12345,
  "streamVersion": 1,
  "groupId": 12345,
  "contractId": "C-0N7RAC7",
  "notificationEmails": [
    "jsmith@email.com"
  ],
  "properties": [
    {
      "propertyId": 12345,
      "propertyName": "my-property-1"
    },
    {
      "propertyId": 98765,
      "propertyName": "my-property-2"
    }
  ],
  "datasetFields": [
    {
      "datasetFieldId": 1002,
      "datasetFieldName": "Request ID",
      "datasetFieldJsonKey": "reqId"
    },
    {
      "datasetFieldId": 1019,
      "datasetFieldName": "Accept-Language",
      "datasetFieldJsonKey": "acceptLanguage"
    },
    {
      "datasetFieldId": 1023,
      "datasetFieldName": "Cookie",
      "datasetFieldJsonKey": "cookie"
    },
    {
      "datasetFieldId": 1032,
      "datasetFieldName": "Referer",
      "datasetFieldJsonKey": "referer"
    }
  ],
  "destination": {
    "destinationType": "HTTPS",
    "displayName": "my-destination",
    "authenticationType": "BASIC",
    "password": "p@ssw0rd",
    "userName": "jsmith",
    "endpoint": "https://example.post/post",
    "compressLogs": true,
    "contentType": "application/json"
  },
  "collectMidgress": true,
  "deliveryConfiguration": {
    "fieldDelimiter": "SPACE",
    "frequency": {
      "intervalInSeconds": 60
    },
    "format": "STRUCTURED"
  },
  "latestVersion": 1,
  "productId": "Object_Delivery",
  "streamStatus": "INACTIVE",
  "createdBy": "1abcdefgh2ijklmn",
  "createdDate": "2025-02-20T09:32:23Z",
  "modifiedBy": "1abcdefgh2ijklmn",
  "modifiedDate": "2025-02-20T09:32:23Z"
}

Then convert it back to a PowerShell object and pipe it to the Set-DataStream command.

$myUpdatedStream = Get-Content ./myStream.json | ConvertFrom-Json -Depth 100

$myUpdatedStream | Set-DataStream -StreamID 12345

streamName            : my-updated-stream
streamId              : 12345
streamVersion         : 2
groupId               : 12345
contractId            : C-0N7RAC7
notificationEmails    : {jsmith@email.com}
properties            : {@{propertyId=12345; propertyName=my-property-1},@{propertyId=98765; propertyName=my-property-2}}
datasetFields         : {@{datasetFieldId=1002; datasetFieldName=Request ID; datasetFieldJsonKey=reqId}, @{datasetFieldId=1019; datasetFieldName=Accept-Language; datasetFieldJsonKey=accLang}, 
 @{datasetFieldId=1023; datasetFieldName=Cookie; datasetFieldJsonKey=cookie}, @{datasetFieldId=1032; datasetFieldName=Referer; datasetFieldJsonKey=referer}}
destination           : @{destinationType=HTTPS; displayName=my-updated-destination; authenticationType=BASIC; endpoint=https://example.com/post; compressLogs=False; contentType=application/json}
deliveryConfiguration : @{fieldDelimiter=SPACE; frequency=; format=STRUCTURED}
latestVersion         : 2
productId             : Object_Delivery
streamStatus          : ACTIVATING
createdBy             : 1abcdefgh2ijklmn
createdDate           : 2/20/2025 9:45:50 AM
modifiedBy            : 1abcdefgh2ijklmn
modifiedDate          : 2/20/2025 11:42:03 AM

Update-DataStream. To update selected details of an existing stream using JSON Patch syntax. Currently, you can patch a stream using only the REPLACE operation.

When updating configuration objects, such as destination or deliveryConfiguration, pass the complete object in the -Body parameter. Thanks to this, you can avoid overwriting current details with the default values for omitted parameters, such as tags, uploadFilePrefix, and uploadFileSuffix.

$myUpdatedStream = @(
 @{
  op = "REPLACE"
  path = "/streamName"
  value = "my-updated-stream"
 },
 @{
  op = "REPLACE"
  path = "/destination"
  value = @{
   authenticationType = "BASIC"
   destinationType = "HTTPS"
   displayName = "my-updated-destination"
   endpoint = "https://example.com/post"
   password = "p@ssW0rd"
   userName = "jsmith"
  }
 }
)

Update-DataStream -StreamID 12345 -Body $myUpdatedStream -Activate

$myUpdatedStream = '[
 {
  "op": "REPLACE",
  "path": "/streamName",
  "value": "my-updated-stream"
 },
 {
  "op": "REPLACE",
  "path": "/destination",
  "value": {
   "authenticationType": "BASIC",
   "destinationType": "HTTPS",
   "displayName": "my-updated-desination",
   "endpoint": "https://example.com/post",
   "password": "p@ssw0rd",
   "userName": "jsmith"
  }
 }
]'

Update-DataStream -StreamID 12345 -Body $myUpdatedStream -Activate

streamName            : my-updated-stream
streamId              : 12345
streamVersion         : 2
groupId               : 12345
contractId            : C-0N7RAC7
notificationEmails    : {jsmith@email.com}
properties            : {@{propertyId=12345; propertyName=my-property-1}}
datasetFields         : {@{datasetFieldId=1002; datasetFieldName=Request ID; datasetFieldJsonKey=reqId}, @{datasetFieldId=1019; datasetFieldName=Accept-Language; datasetFieldJsonKey=accLang}, @{datasetFieldId=1023; datasetFieldName=Cookie; 
      datasetFieldJsonKey=cookie}, @{datasetFieldId=1032; datasetFieldName=Referer; datasetFieldJsonKey=referer}}
destination           : @{destinationType=HTTPS; displayName=my-updated-desination; authenticationType=BASIC; endpoint=https://example.com/post; compressLogs=False}
deliveryConfiguration : @{fieldDelimiter=SPACE; frequency=; format=STRUCTURED}
latestVersion         : 2
productId             : Object_Delivery
streamStatus          : ACTIVATING
createdBy             : 1abcdefgh2ijklmn
createdDate           : 2/20/2025 9:45:50 AM
modifiedBy            : 1abcdefgh2ijklmn
modifiedDate          : 2/20/2025 2:51:09 PM

Running any of these two commands creates a new stream version that replaces the current one.

For both commands, you need to add the -Activate switch parameter when updating active streams, and optionally for inactive streams if you want to activate them upon request.

Add rules and behaviors

Get your property's rule tree as a JSON file. The output is a JSON file at your current location containing your rule's configurations.
```
Get-PropertyRules -PropertyName 'my-property-1' -PropertyVersion 'latest' -OutputToFile
```

In the property's rule tree, add the DataStream rule and behavior and any additional behaviors required by a data set. For the options' configuration, see the DataStream behavior.

For more than one property, loop through each property to add the rule.
If you use includes in your rule tree, activate them before you activate your property.

{
 "name": "Datastream",
 "children": [],
 "behaviors": [
  {
   "name": "datastream",
   "options": {
    "streamType": "LOG",
    "logStreamTitle": "",
    "logEnabled": true,
    "logStreamName": [
     "12345"
    ],
    "samplingPercentage": 100,
    "collectMidgressTraffic": false
   }
  }
 ],
 "criteria": [],
 "criteriaMustSatisfy": ""
}

Add your updated rules to your property, pointing to the appropriate file.

Send your entire rule tree back. Rules not returned are removed from your property.
```
Set-PropertyRules -PropertyName 'my-property-1' -PropertyVersion latest -InputFile './main.json' -VersionNotes 'Adding an access key'
```
The output returns your rule tree along with any errors or warnings on upload.

Activate a property

To activate the property on a network to start collecting data with your data stream, provide a value for the network, staging or production, and activate your property.

New-PropertyActivation -PropertyName 'my-property-1' -PropertyVersion 'latest' -Network 'Staging' -NotifyEmails 'jsmith@email.com'

activationLink                                                                                      activationId
--------------                                                                                      ------------
/papi/v1/properties/prp_12345/activations/atv_67890?contractId=ctr_C-0N7RAC7&groupId=grp_12345      atv_67890

Remove a data stream

To remove a data stream, first, you need to deactivate it. Run the New-DataStreamDeactivation command to do that. Deactivating a stream takes about an hour.

New-DataStreamDeactivation -StreamID 12345

streamName            : my-stream
streamId              : 12345
streamVersion         : 2
groupId               : 12345
contractId            : C-0N7RAC7
notificationEmails    : {jsmith@email.com}
properties            : {@{propertyId=12345; propertyName=my-property-1}}
datasetFields         : {@{datasetFieldId=1002; datasetFieldName=Request ID; datasetFieldJsonKey=reqId}, @{datasetFieldId=1019; 
                        datasetFieldName=Accept-Language; datasetFieldJsonKey=accLang}, @{datasetFieldId=1023; 
                        datasetFieldName=Cookie; datasetFieldJsonKey=cookie}, @{datasetFieldId=1032; datasetFieldName=Referer; 
                        datasetFieldJsonKey=referer}}
destination           : @{destinationType=HTTPS; displayName=my-desination; authenticationType=BASIC; 
                        endpoint=https://example.com/post; compressLogs=False}
deliveryConfiguration : @{fieldDelimiter=SPACE; frequency=; format=STRUCTURED}
latestVersion         : 2
productId             : Object_Delivery
streamStatus          : DEACTIVATING
createdBy             : 1abcdefgh2ijklmn
createdDate           : 2/20/2025 9:45:50 AM
modifiedBy            : 1abcdefgh2ijklmn
modifiedDate          : 2/21/2025 4:25:16 PM

Once you've deactivated the data stream, run the Remove-DataStream command to remove it completely.

Remove-DataStream -StreamID 12345

Updated about 12 hours ago