DataStream 2 provides traffic data and metrics on edge request-response cycles in bundled log files. You can use log data to monitor the performance and behavior on your property, troubleshoot incidents, and improve security.
Your streams collect, bundle, and upload log records to a destination of your choice at selected time intervals. DataStream 2 lets you choose the data set fields you monitor in your logs (including custom log fields), the order of these fields in the log lines, and the delivery frequency.
Depending on the destination, DataStream 2 can send logs either in the Structured (space-delimited) or JSON log file format. Destinations offer other features, including custom log filename prefix and suffix, custom header details sent with the log requests, and integration with other products, such as EdgeWorkers. See Destinations for the complete list of available features for each destination and log format support.
When you Create a stream, Edit a stream, or Patch a stream in the DataStream 2 API, the order of data set fields listed in the request body is the order of their values in the log line. You can use this feature to move data set fields in the log files you receive as you need.
Sample log lines
See the sample log lines for both DataStream log formats below. Note that in each format the stream version and stream ID (1 and 12345 in the examples) come before the values for the data set fields in the log line.
You can also use the DataStream application to move the order of data set fields and generate sample log lines.
Structured:
1 12345 123456 1239f220 1573840000 4995 128.147.28.68 206 HTTPS test.hostname.net GET /path1/path2/file.ext 443 5000 text/html Mozilla%2F5.0+%28Macintosh%3B+Intel+Mac+OS+X+10_14_3%29 0 TLSv1 484 484 232 0 param=value //BC/%5Ba=23.33.41.20,c=g,k=0,l=1%5D en-US cookie-content 37334-42356 https%3A%2F%2Ftest.referrer.net%2Fen-US%2Fdocs%2FWeb%2Ftest 8.47.28.38 3600 3 ERR_ACCESS_DENIED|fwd_acl 11 125 50 1 23.50.51.173 IN Virginia HERNDON SG 8 1 ULnR_28976|3900000:3900001:3900005:3900006:BOT-ANOMALY-HEADER| //4380/4.0/1/-/0/4/#1,2\//4380/4.0/4/-/0/4/#0,0\//4380/4.0/5/-/1/1/#0,0 c:4380:7:161:162:161:n:::12473:200|C:4380:3:0:4:0:n:::6967:200|R:4380:20:99:99:1:n:::35982:200 any-custom-value
JSON:
{
"version": 1,
"streamId": "12345",
"cp": "123456",
"reqId": "1239f220",
"reqTimeSec": "1573840000",
"bytes": "4995",
"cliIP": "128.147.28.68",
"statusCode": "206",
"proto": "HTTPS",
"reqHost": "test.hostname.net",
"reqMethod": "GET",
"reqPath": "/path1/path2/file.ext",
"reqPort": "443",
"rspContentLen": "5000",
"rspContentType": "text/html",
"UA": "Mozilla%2F5.0+%28Macintosh%3B+Intel+Mac+OS+X+10_14_3%29",
"tlsOverheadTimeMSec": "0",
"tlsVersion": "TLSv1",
"objSize": "484",
"uncompressedSize": "484",
"overheadBytes": "232",
"totalBytes": "0",
"queryStr": "param=value",
"breadcrumbs": "//BC/%5Ba=23.33.41.20,c=g,k=0,l=1%5D",
"accLang": "en-US",
"cookie": "cookie-content",
"range": "37334-42356",
"referer": "https%3A%2F%2Ftest.referrer.net%2Fen-US%2Fdocs%2FWeb%2Ftest",
"xForwardedFor": "8.47.28.38",
"maxAgeSec": "3600",
"reqEndTimeMSec": "3",
"errorCode": "ERR_ACCESS_DENIED|fwd_acl",
"turnAroundTimeMSec": "11",
"transferTimeMSec": "125",
"dnsLookupTimeMSec": "50",
"lastByte": "1",
"edgeIP": "23.50.51.173",
"country": "IN",
"state": "Virginia",
"city": "HERNDON",
"serverCountry": "SG",
"billingRegion": "8",
"cacheStatus": "1",
"securityRules": "ULnR_28976|3900000:3900001:3900005:3900006:BOT-ANOMALY-HEADER|",
"ewUsageInfo": "//4380/4.0/1/-/0/4/#1,2\\//4380/4.0/4/-/0/4/#0,0\\//4380/4.0/5/-/1/1/#0,0",
"ewExecutionInfo": "c:4380:7:161:162:161:n:::12473:200|C:4380:3:0:4:0:n:::6967:200|R:4380:20:99:99:1:n:::35982:200",
"customField": "any-custom-value"
}