FAQ

The following are frequently asked questions and known issues about DataStream 2.

Question	Answer
What is DataStream's output data format?	The output log file format for DataStream is structured (space-delimited) or JSON. For sample schemas and data sets, see Log format or the DataStream 2 API.
How many endpoints does DataStream send logs from?	DataStream sends logs to customer destinations from multiple source endpoints. Typically, the number of source endpoints range from 10s to 100s and can change without prior notice.
What happens if log files fail to upload to a destination?	Streaming logs to third-party destinations may temporarily fail due to issues such as latency or connection problems. To prevent data loss during temporary upload failures, DataStream retries data upload three times. After three failed upload retries, the data will not be saved.
Why are multiple small log files uploaded to my destination?	The push frequency limits (each 30 or 60 seconds, 4 MB file size, and 32 000 log lines) apply on an uploader machine level, and not at the overall system level. When the uploader machine collecting the log lines hits any of these limits, it creates a file and uploads it to the destination. Since there are many uploader machines, you might see multiple smaller files getting uploaded for the same push frequency interval.
Should I use log data for billing?	We recommend using raw log data for basic traffic analysis and monitoring CDN health. In case of issues with third-party destinations, such as latency or connection problems, data will be lost after three failed retries to connect.
Does DataStream store log data in case of stream failures?	DataStream doesn't store data streamed to third-party destinations. In case of stream failure, the stream attempts to reconnect three times every five minutes. After three failures over 15 minutes, the log data is lost.
I don't use a log analytics tool. Can I still use DataStream?	You need a tool to parse and visualize the DataStream output. You could choose lower cost, open source stacks for data parsing and visualization in human readable formats.
Does DataStream support security event logs?	DataStream is a log delivery product for all transactional events and associated metrics. You can use the SIEM Integration product to deliver security logs. See SIEM Integration.
Why does one or more custom fields return `^` in log files?	DataStream requires enabling custom fields, such as User-Agent, Accept-Language, Cookie, Referrer or X-Forwarded-For in the Log Delivery Service behavior in your property configuration. If log lines return `^` , first enable the fields in Property Manager.