Configure DataStream 2 to get traffic data and metrics on edge request-response cycles in bundled log files. You can use log data to monitor the performance and behavior on your property, troubleshoot incidents, and improve security.
Your streams collect, bundle, and upload log records to a destination of your choice at selected time intervals. Each time you create or edit a stream, you can choose the data set fields (including custom log fields and Reporting metrics), change the order of these fields in the log lines, and the delivery frequency.
Log format
Depending on the destination, DataStream 2 can send logs either in the Structured (space- or tab-delimited) or JSON log file format. Destinations offer other features, including custom log filename prefix and suffix, custom header details sent with the log requests, and integration with other products, such as EdgeWorkers. See Destinations for the complete list of available features for each destination and log format support.
When you Create a stream, Edit a stream, or Patch a stream in the DataStream 2 API, the order of data set fields listed in the request body is the order of their values in the log line. You can use this feature to move data set fields in the log files you receive as you need.
Note
In JSON logs, DataStream omits data set fields reported in Structured logs as
-ornullby default.If any other fields are missing from your logs, Get a stream to check the list of logged parameters.
Sample log lines
See the sample log lines for both DataStream log formats below. Note that in each format the stream version and stream ID (1 and 12345 in the examples) come before the values for the data set fields in the log line.
You can also use the DataStream application to move the order of data set fields and generate sample log lines.
Structured:
1 12345 123456 1239f220 1573840000 4995 128.147.28.68 206 HTTPS test.hostname.net GET /path1/path2/file.ext 443 5000 text/html Mozilla%2F5.0+%28Macintosh%3B+Intel+Mac+OS+X+10_14_3%29 0 TLSv1 484 484 232 0 param=value //BC/%5Ba=23.33.41.20,c=g,k=0,l=1%5D en-US cookie-content 37334-42356 https%3A%2F%2Ftest.referrer.net%2Fen-US%2Fdocs%2FWeb%2Ftest 8.47.28.38 3600 3 ERR_ACCESS_DENIED|fwd_acl 11 125 50 1 23.50.51.173 IN Virginia HERNDON SG 8 1 ULnR_28976|3900000:3900001:3900005:3900006:BOT-ANOMALY-HEADER| //4380/4.0/1/-/0/4/#1,2\//4380/4.0/4/-/0/4/#0,0\//4380/4.0/5/-/1/1/#0,0 c:4380:7:161:162:161:n:::12473:200|C:4380:3:0:4:0:n:::6967:200|R:4380:20:99:99:1:n:::35982:200 any-custom-value
JSON:
{
"version": 1,
"streamId": "12345",
"cp": "123456",
"reqId": "1239f220",
"reqTimeSec": "1573840000",
"bytes": "4995",
"cliIP": "128.147.28.68",
"statusCode": "206",
"proto": "HTTPS",
"reqHost": "test.hostname.net",
"reqMethod": "GET",
"reqPath": "/path1/path2/file.ext",
"reqPort": "443",
"rspContentLen": "5000",
"rspContentType": "text/html",
"UA": "Mozilla%2F5.0+%28Macintosh%3B+Intel+Mac+OS+X+10_14_3%29",
"tlsOverheadTimeMSec": "0",
"tlsVersion": "TLSv1",
"objSize": "484",
"uncompressedSize": "484",
"overheadBytes": "232",
"totalBytes": "0",
"queryStr": "param=value",
"breadcrumbs": "//BC/%5Ba=23.33.41.20,c=g,k=0,l=1%5D",
"accLang": "en-US",
"cookie": "cookie-content",
"range": "37334-42356",
"referer": "https%3A%2F%2Ftest.referrer.net%2Fen-US%2Fdocs%2FWeb%2Ftest",
"xForwardedFor": "8.47.28.38",
"maxAgeSec": "3600",
"reqEndTimeMSec": "3",
"errorCode": "ERR_ACCESS_DENIED|fwd_acl",
"turnAroundTimeMSec": "11",
"transferTimeMSec": "125",
"dnsLookupTimeMSec": "50",
"lastByte": "1",
"edgeIP": "23.50.51.173",
"country": "IN",
"state": "Virginia",
"city": "HERNDON",
"serverCountry": "SG",
"billingRegion": "8",
"cacheStatus": "1",
"securityRules": "ULnR_28976|3900000:3900001:3900005:3900006:BOT-ANOMALY-HEADER|",
"ewUsageInfo": "//4380/4.0/1/-/0/4/#1,2\\//4380/4.0/4/-/0/4/#0,0\\//4380/4.0/5/-/1/1/#0,0",
"ewExecutionInfo": "c:4380:7:161:162:161:n:::12473:200|C:4380:3:0:4:0:n:::6967:200|R:4380:20:99:99:1:n:::35982:200",
"customField": "any-custom-value"
}