Create data streams

Version 5.0 subprovider update

We've updated our DataStream subprovider to provide a better developer experience and support new platform capabilities.

The improvements introduce changes incompatible with the previous version.

Version 4.1 will continue to work for existing integrations until January 2024. To update your integration, see the migration information.

Data streams are monitoring and analytical components you set up independently of and later assign to your properties.

This component-rule relationship lets you configure multiple data streams, each focused on gathering a specific type of data. When combined, data streams deliver customized metrics and health insights about your traffic across multiple properties.

What you'll do

Configure a data stream, set up the push of raw data logs to a given destination, and connect it to your properties.

Get your properties

The properties you use in DataStream integrations must have room to pick up a new data stream and be active on a network.

• You can assign up to three data streams to a single property. To see how many data streams are assigned to a property, review its Datastream rule.
• A single data stream can support up to 100 properties. Use the Data streams resource to get a list of its assigned properties.

Note: If you need, create a new property and activate it on a network.

1. Get a list of your properties and determine which of them you'll use with your data stream.

   QueryStandard output

   data "akamai_properties" "my_properties" {
       group_id    = "12345"
       contract_id = "C-0N7RAC7"
   }
   
   output "my_properties" {
     value = data.akamai_properties.my_properties
   }
   

   + my_properties = {
         + contract_id = "C-0N7RAC7"
         + group_id    = "12345"
         + id          = "grp_12345ctr_C-0N7RAC7"
         + properties  = [
             + {
                 + contract_id        = "ctr_C-0N7RAC7"
                 + group_id           = "grp_12345"
                 + latest_version     = 8
                 + note               = "Added hostname."
                 + product_id         = "prd_Adaptive_Media_Delivery"
                 + production_version = 8
                 + property_id        = "prp_12345"
                 + property_name      = "my_property1"
                 + rule_format        = ""
                 + staging_version    = 3
               },
               + {
                 + contract_id        = "ctr_C-0N7RAC7"
                 + group_id           = "grp_12345"
                 + latest_version     = 3
                 + note               = "File type update."
                 + product_id         = "prd_Object_Delivery"
                 + production_version = 3
                 + property_id        = "prp_98765"
                 + property_name      = "my_property2"
                 + rule_format        = ""
                 + staging_version    = 2
               },
           ]
       }
   

2. Export all chosen properties using the Terraform CLI.

   The CLI export command places property configuration files in your current directory. To export them to a different location, add the --tfworkpath <path> flag to the command.

   When exporting more than one, loop through the CLI command with the --tfworkpath <path> flag, changing the property name and export location every iteration.

   Shell

   akamai terraform --edgerc <edgerc-file-location> --section <edgerc-section> export-property <property-name>
   

3. Run the included import script to populate your Terraform state. This prevents Terraform from attempting to recreate your assets.

After you create a data stream, you'll update your properties' rule trees.

Create a data stream

To create a data stream, configure a log destination and choose data sets that shape what's monitored and collected about your traffic.

Average processing time 90 minutes - 3 hours

Provide a destination

You can use a custom HTTPS endpoint or a third-party object storage location. Supported storage locations:

Create a connector block for your data destination. Use the argument column's heading as is and add it to _connector to name your connector's block, for example, gcs_connector.

Include all required arguments for your destination.

gcs_connector {
    bucket               = "my_bucket"
    display_name         = "my_connector_name"
    path                 = "akamai/logs"
    private_key          = "-----BEGIN PRIVATE KEY-----\nprivate_key\n-----END PRIVATE KEY-----\n"
    project_id           = "my_project_id"
    service_account_name = "my_service_account_name"
  }

Choose data sets

Data set fields represent the types of data collected and returned in your log files.

Choose data set field and add their IDs as a comma separated list of integers in the dataset_fields argument. The order in which you place these determines the order in which they appear in log files.

🚧

For fields that require additional behaviors, wait to adjust your property configuration until after data stream creation.

Construct the resource

Add your connector block and data stream fields and property lists to the remaining required arguments to create a data stream.

resource "akamai_datastream" "my_datastream" {
  active = true
  delivery_configuration {
    field_delimiter = "SPACE"
    format          = "STRUCTURED"
    frequency {
      interval_in_secs = 30
    }
    upload_file_prefix = "prefix"
    upload_file_suffix = "suffix"
  }
  contract_id = "C-0N7RAC7"
  dataset_fields = [
    1000, 1002, 1102
  ]
  group_id = 12345
  properties = [
    12345, 98765
  ]
  stream_name = "Datastream_Example1"
  gcs_connector {
    bucket               = "my_bucket"
    display_name         = "my_connector_name"
    path                 = "akamai/logs"
    private_key          = "-----BEGIN PRIVATE KEY-----\nprivate_key\n-----END PRIVATE KEY-----\n"
    project_id           = "my_project_id"
    service_account_name = "my_service_account_name"
  }
  notification_emails = [
    "example1@example.com",
    "example2@example.com",
  ]
  collect_midgress = true
}

There is no default standard output as the attribute values are sensitive, but you can get your data stream's ID from the last line of the process log or by using the data stream data source.

akamai_datastream.my_datastream: Creation complete after 1h20m16s [id=12345]

Use the ID to connect your data stream to your property in the data stream rule.

Add rules and behaviors

Add the DataStream rule and behavior and any additional behaviors required by a data set to your properties' default rule. For options configuration, see the datastream behavior.

• For more than one property, loop through each property to add the rule.
• If you use includes in your rule tree, activate them before you activate your property.

{
    "name": "Datastream",
    "children": [],
    "behaviors": [
        {
            "name": "datastream",
            "options": {}
        }
    ],
    "criteria": [],
    "criteriaMustSatisfy": ""
}

Activate properties

Activate your properties on a network to start collecting data with your data stream.

The required arguments for a property activation are property_id, contact, and version. If you don't specify a network, the default action targets staging.

// Change the network value to production for the production network
resource "akamai_property_activation" "my_activation" {
     property_id                    = "prp_12345"
     network                        = "staging"
     contact                        = ["jsmith@example.com"]
     note                           = "Sample activation"
     version                        = "1"
     auto_acknowledge_rule_warnings = true
}