FAQ

Check the list of frequently asked questions (FAQs) and known issues in DataStream 2:

Question	Answer
What is DataStream's output data format?	The output log file format for DataStream is structured (space- or tab-delimited) or JSON. For sample schemas and data sets, see Log format or the DataStream 2 API.
How many endpoints does DataStream send logs from?	DataStream sends logs to customer destinations from multiple source endpoints. Typically, the number of source endpoints range from 10s to 2500s and can change without prior notice.
What happens if log files fail to upload to a destination?	Streaming logs to third-party destinations may temporarily fail due to issues such as latency or connection problems. To prevent data loss during temporary upload failures, DataStream retries data upload up to 10 attempts in 5 minutes. This applies to responses with 429 and 5XX error codes. For details, see Delivery retry.
Why are multiple small log files uploaded to my destination?	The push frequency limits (each 30 or 60 seconds, 4 MB file size, and 32 000 log lines) apply on an uploader machine level, and not at the overall system level. When the uploader machine collecting the log lines hits any of these limits, it creates a file and uploads it to the destination. Since there are many uploader machines, you might see multiple smaller files getting uploaded for the same push frequency interval.
Should I use log data for billing?	We recommend using raw log data for basic traffic analysis and monitoring CDN health. In case of issues with third-party destinations, such as latency or connection problems, data will be lost after three failed retries to connect.
Does DataStream store log data in case of stream failures?	DataStream doesn't store backup copies of data streamed to third-party destinations. In case of stream upload failure, the system will retry uploading logs up to 10 attempts in 5 minutes. This applies to responses with 429 and 5XX error codes. For details, see Delivery retry.
I don't use a log analytics tool. Can I still use DataStream?	You need a tool to parse and visualize the DataStream output. You could choose lower cost, open source stacks for data parsing and visualization in human readable formats.
Does DataStream support security event logs?	DataStream is a log delivery product for all transactional events and associated metrics. You can use the SIEM Integration product to deliver security logs. See SIEM Integration.
Why does one or more custom fields return `^` in log files?	DataStream requires enabling custom fields, such as User-Agent, Accept-Language, Cookie, Referrer or X-Forwarded-For in the Log Delivery Service behavior in your property configuration. If log lines return `^` , first enable the fields in Property Manager.

If you want to troubleshoot issues related to DataStream 2, see Troubleshooting.