Configure DataStream 2 to get traffic data and metrics on edge request-response cycles in bundled log files. You can use log data to monitor the performance and behavior on your property, troubleshoot incidents, and improve security.

Your streams collect, bundle, and upload log records to a destination of your choice at selected time intervals. Each time you create or edit a stream, you can choose the data set fields (including custom log fields and Reporting metrics), change the order of these fields in the log lines, and the delivery frequency.

Data format

Depending on the destination, DataStream 2 can send logs either in the Structured (space- or tab-delimited) or JSON log file format. Destinations offer other features, including custom log filename prefix and suffix, custom header details sent with the log requests, and integration with other products, such as EdgeWorkers. See Stream logs to a destination for the complete list of available features for each destination and log format support.

When you Create a stream or Edit a stream in DataStream 2, you can use the Data Sets screen in the stream creation wizard to change the order of data set fields as their values appear in the log lines, and generate generate sample log lines. For steps, see Choose data parameters.

Sample log lines

See the sample log lines for both DataStream log formats below. Note that in each format the stream version and stream ID come before the values for the data set fields in the log line.

Structured:

1 12345 123456 1239f220 1573840000 4995 128.147.28.68 206 HTTPS test.hostname.net GET /path1/path2/file.ext 443 5000 text/html Mozilla%2F5.0+%28Macintosh%3B+Intel+Mac+OS+X+10_14_3%29 0 TLSv1 484 484 232 0 param=value //BC/%5Ba=23.33.41.20,c=g,k=0,l=1%5D en-US cookie-content 37334-42356 https%3A%2F%2Ftest.referrer.net%2Fen-US%2Fdocs%2FWeb%2Ftest 8.47.28.38 3600 3 ERR_ACCESS_DENIED|fwd_acl 11 125 50 1 23.50.51.173 IN Virginia HERNDON SG 8 1 ULnR_28976|3900000:3900001:3900005:3900006:BOT-ANOMALY-HEADER| //4380/4.0/1/-/0/4/#1,2\//4380/4.0/4/-/0/4/#0,0\//4380/4.0/5/-/1/1/#0,0 c:4380:7:161:162:161:n:::12473:200|C:4380:3:0:4:0:n:::6967:200|R:4380:20:99:99:1:n:::35982:200 any-custom-value

JSON:

{
  "version": 1,
  "streamId": "12345",
  "cp": "123456",
  "reqId": "1239f220",
  "reqTimeSec": "1573840000",
  "bytes": "4995",
  "cliIP": "128.147.28.68",
  "statusCode": "206",
  "proto": "HTTPS",
  "reqHost": "test.hostname.net",
  "reqMethod": "GET",
  "reqPath": "/path1/path2/file.ext",
  "reqPort": "443",
  "rspContentLen": "5000",
  "rspContentType": "text/html",
  "UA": "Mozilla%2F5.0+%28Macintosh%3B+Intel+Mac+OS+X+10_14_3%29",
  "tlsOverheadTimeMSec": "0",
  "tlsVersion": "TLSv1",
  "objSize": "484",
  "uncompressedSize": "484",
  "overheadBytes": "232",
  "totalBytes": "0",
  "queryStr": "param=value",
  "breadcrumbs": "//BC/%5Ba=23.33.41.20,c=g,k=0,l=1%5D",
  "accLang": "en-US",
  "cookie": "cookie-content",
  "range": "37334-42356",
  "referer": "https%3A%2F%2Ftest.referrer.net%2Fen-US%2Fdocs%2FWeb%2Ftest",
  "xForwardedFor": "8.47.28.38",
  "maxAgeSec": "3600",
  "reqEndTimeMSec": "3",
  "errorCode": "ERR_ACCESS_DENIED|fwd_acl",
  "turnAroundTimeMSec": "11",
  "transferTimeMSec": "125",
  "dnsLookupTimeMSec": "50",
  "lastByte": "1",
  "edgeIP": "23.50.51.173",
  "country": "IN",
  "state": "Virginia",
  "city": "HERNDON",
  "serverCountry": "SG",
  "billingRegion": "8",
  "cacheStatus": "1",
  "securityRules": "ULnR_28976|3900000:3900001:3900005:3900006:BOT-ANOMALY-HEADER|",
  "ewUsageInfo": "//4380/4.0/1/-/0/4/#1,2\\//4380/4.0/4/-/0/4/#0,0\\//4380/4.0/5/-/1/1/#0,0",
  "ewExecutionInfo": "c:4380:7:161:162:161:n:::12473:200|C:4380:3:0:4:0:n:::6967:200|R:4380:20:99:99:1:n:::35982:200",
  "customField": "any-custom-value"
}