DataStream 2 provides traffic data and metrics on edge request-response cycles in bundled log files. You can use log data to monitor the performance and behavior on your property, troubleshoot incidents, and improve security.

Your streams collect, bundle, and upload log records to a destination of your choice at selected time intervals. DataStream 2 lets you choose the data set fields you monitor in your logs (including custom log fields), the order of these fields in the log lines, and the delivery frequency.

Depending on the destination, DataStream 2 can send logs either in the Structured (space-delimited) or JSON log file format. Destinations offer other features, including custom log filename prefix and suffix, custom header details sent with the log requests, and integration with other products, such as EdgeWorkers. See Stream logs to a destination for the complete list of available features for each destination and log format support.

When you Create a stream or Edit a stream in DataStream 2, you can move the order of data set fields as their values appear in the log lines. You can use the Data Sets screen in the stream creation wizard to move the order of data set fields and generate sample log lines.

Sample log lines

See the sample log lines for both DataStream log formats below. Note that in each format the stream version and stream ID come before the values for the data set fields in the log line.

Structured:

1 12345 123456 1239f220 1573840000 4995 128.147.28.68 206 HTTPS test.hostname.net GET /path1/path2/file.ext 443 5000 text/html Mozilla%2F5.0+%28Macintosh%3B+Intel+Mac+OS+X+10_14_3%29 0 TLSv1 484 484 232 0 param=value //BC/%5Ba=23.33.41.20,c=g,k=0,l=1%5D en-US cookie-content 37334-42356 https%3A%2F%2Ftest.referrer.net%2Fen-US%2Fdocs%2FWeb%2Ftest 8.47.28.38 3600 3 ERR_ACCESS_DENIED|fwd_acl 11 125 50 1 23.50.51.173 IN Virginia HERNDON SG 8 1 ULnR_28976|3900000:3900001:3900005:3900006:BOT-ANOMALY-HEADER| //4380/4.0/1/-/0/4/#1,2\//4380/4.0/4/-/0/4/#0,0\//4380/4.0/5/-/1/1/#0,0 c:4380:7:161:162:161:n:::12473:200|C:4380:3:0:4:0:n:::6967:200|R:4380:20:99:99:1:n:::35982:200 any-custom-value

JSON:

{
  "version": 1,
  "streamId": "12345",
  "cp": "123456",
  "reqId": "1239f220",
  "reqTimeSec": "1573840000",
  "bytes": "4995",
  "cliIP": "128.147.28.68",
  "statusCode": "206",
  "proto": "HTTPS",
  "reqHost": "test.hostname.net",
  "reqMethod": "GET",
  "reqPath": "/path1/path2/file.ext",
  "reqPort": "443",
  "rspContentLen": "5000",
  "rspContentType": "text/html",
  "UA": "Mozilla%2F5.0+%28Macintosh%3B+Intel+Mac+OS+X+10_14_3%29",
  "tlsOverheadTimeMSec": "0",
  "tlsVersion": "TLSv1",
  "objSize": "484",
  "uncompressedSize": "484",
  "overheadBytes": "232",
  "totalBytes": "0",
  "queryStr": "param=value",
  "breadcrumbs": "//BC/%5Ba=23.33.41.20,c=g,k=0,l=1%5D",
  "accLang": "en-US",
  "cookie": "cookie-content",
  "range": "37334-42356",
  "referer": "https%3A%2F%2Ftest.referrer.net%2Fen-US%2Fdocs%2FWeb%2Ftest",
  "xForwardedFor": "8.47.28.38",
  "maxAgeSec": "3600",
  "reqEndTimeMSec": "3",
  "errorCode": "ERR_ACCESS_DENIED|fwd_acl",
  "turnAroundTimeMSec": "11",
  "transferTimeMSec": "125",
  "dnsLookupTimeMSec": "50",
  "lastByte": "1",
  "edgeIP": "23.50.51.173",
  "country": "IN",
  "state": "Virginia",
  "city": "HERNDON",
  "serverCountry": "SG",
  "billingRegion": "8",
  "cacheStatus": "1",
  "securityRules": "ULnR_28976|3900000:3900001:3900005:3900006:BOT-ANOMALY-HEADER|",
  "ewUsageInfo": "//4380/4.0/1/-/0/4/#1,2\\//4380/4.0/4/-/0/4/#0,0\\//4380/4.0/5/-/1/1/#0,0",
  "ewExecutionInfo": "c:4380:7:161:162:161:n:::12473:200|C:4380:3:0:4:0:n:::6967:200|R:4380:20:99:99:1:n:::35982:200",
  "customField": "any-custom-value"
}