Stream logs to a custom HTTPS endpoint

DataStream supports sending logs to a secure HTTPS endpoint to allow on-premise software to receive and process logs.

Depending on your choice, DataStream 2 can upload either uncompressed or gzip-compressed log files.

For security reasons, DataStream sends logs over TLS even if the endpoint’s policies allow insecure requests. Optionally, you can upload a client certificate to enable mTLS authentication to improve stream security and prevent data delivery failures. The custom header feature allows you to optionally choose the content type passed in the log file, and enter the name and value for the header that your destination accepts.

Before you begin

  • Deploy a dedicated HTTPS endpoint that supports URL token authentication.
  • Enable TLS transport for the endpoint to receive your data stream's logs.
  • Optionally, configure an username or password in your custom endpoint. This applies if you want Basic authentication for log streaming.

How to

  1. In Destination, select Custom HTTPS.

  2. In Name, enter a human-readable description for the destination.

  3. In Endpoint URL, enter the secure URL where you want to send and store your logs.

📘

Endpoint URL requirements

Provide an endpoint URL that supports POST requests. If you want to choose Basic authentication, make sure your endpoint supports it.

Enter an URL that is not an IPv4 or IPv6 hostname.

  1. In Authentication, select:

    • Basic if you want to authenticate log streaming to your custom destination. Provide the Username and Password you set in your custom HTTPS endpoint for authentication.
    • None for no authentication.
  2. If you want to send compressed gzip files to your destination, check the Send compressed data box.

  3. Click Validate & Save to validate the connection to the destination and save the details you provided.

    As part of this validation process, the system uses the provided credentials to push a sample request to the provided endpoint to validate the write access. In case you chose the Structured log format, the sample data appears in the 0,access_validation format. For JSON logs, the data follows the {"access_validation":true} format. You can see the data only if the destination validates, and you can access the destination storage.

Additional options

  1. Optionally, click Additional options to add mTLS certificates for additional authentication. In Client certificate, enter the:
    • TLS hostname matching the Subject Alternative Names (SANs) present in the SSL certificate for the endpoint URL. If not provided, DataStream 2 fetches the hostname from the URL.
    • CA certificate that you want to use to verify the origin server's certificate. DataStream requires a CA certificate, if you provide a self-signed certificate or a certificate signed by an unknown authority. Enter the CA certificate in the PEM format for verification.
    • Client certificate in the PEM format that you want to use to authenticate requests to your destination. If you want to use mutual authentication, provide both the client certificate and the client key.
    • Client key you want to use to authenticate to the backend server in the PEM (non-encrypted PKCS8) format. If you want to use mutual authentication, provide both the client certificate and the client key.

📘

When enabling mTLS authentication for a custom destination, configure the endpoint for all settings required for authentication with a valid client certificate.

  1. Optionally, go to Custom header and provide the details of the custom header for the log file:
    • In Content type, set the content type to pass in the log file header. application/json is the only supported content type at this time.
    • If your destination accepts requests only with certain headers, enter the Custom header name and Custom header value. The custom header name can contain the alphanumeric, dash, and underscore characters.

🚧

Forbidden custom header values

DataStream 2 does not support custom header user values containing:

  • Content-Type
  • Encoding
  • Authorization
  • Host
  • Akamai
  1. Click Validate & Save to validate the connection to the destination and save the details you provided.

Request examples

Depending on the configuration, including the authentication type you choose, requests to your destination may look differently. See the request header examples below:

None authentication

Basic authentication

Host: pdxsqalinuxvm.eastus2.cloudapp.azure.com:8102
User-Agent: Go-http-client/1.1
Connection: close
Transfer-Encoding: chunked
Accept-Encoding: gzip

Host: pdxsqalinuxvm.eastus2.cloudapp.azure.com:8002
User-Agent: Go-http-client/1.1
Connection: close
Transfer-Encoding: chunked
Authorization: Basic Og==
Accept-Encoding: gzip


Did this page help you?