Apache Kafka (cluster)

Deploy Apache Kafka, a robust, scalable, and high-performance system for managing real-time data streams. Kafka provides a reliable foundation for data pipelines, event-driven architectures, and stream processing applications, from log aggregation to real-time analytics.

This app deploys a Kafka cluster using Kafka's native consensus protocol, KRaft. A few things to note about the deployment:

• The cluster is configured with mTLS for authentication. Inter-broker communication and client authentication are established via certificate identity.
• The minimum cluster size is three, with three controllers configured at all times for fault tolerance.
• Clients connecting to the cluster need their own valid certificate. All certificates are signed with a self-signed Certificate Authority (CA). Client keystores and the truststore are on the first Kafka node in /etc/kafka/ssl/keystore and /etc/kafka/ssl/truststore.
• The CA key and certificate pair are on the first Kafka node in /etc/kafka/ssl/ca.

📘

Cluster deployment

This app deploys three, five, or seven Linode instances to create a distributed Apache Kafka cluster, each with the plan type and size you select.

Each of these Linode instances will appear on your invoice as separate items.

1. Deploy Apache Kafka cluster

 Estimated deployment time: 5–7 minutes

Software included

Software installed on your Linode when deploying the app.

Base distribution and plan

• Supported distribution: Ubuntu 24.04 LTS
• Recommended plan: All plan types and sizes can be used depending on your storage needs.

Configuration options

StackScript ID: 1377657

Note: After the app deployment completes, the password for your limited sudo user is generated and stored in the .credentials file in the home directory, along with application-specific passwords. Log in to your instance as root through the Lish console or SSH, then run cat /home/$USERNAME/.credentials to view its contents.

Use API, CLI, or Terraform

curl --location 'https://api.linode.com/v4/linode/instances' \
    --header 'Content-Type: application/json' \
    --header 'Accept: application/json' \
    --header 'Authorization: Bearer abc123def456hij789klm' \
    --data-raw '{
        "region": "us-east",
        "type": "g6-standard-4",
        "image": "linode/ubuntu24.04",
        "label": "my-kafka-cluster-one-click-app",
        "root_pass": "@C0mpl3x#P@ssw0rd",
        "stackscript_id": 1377657,
        "stackscript_data": {
            "token_password": "abc123def456hij789klm",
            "sudo_username": "jsmith",
            "client_count": "1",
            "cluster_size": "5",
            "country_name": "US",
            "state_or_province_name": "Massachusetts",
            "locality_name": "Cambridge",
            "organization_name": "Akamai Technologies",
            "email_address": "jsmith@example.com",
            "ca_common_name": "Kafka RootCA"
        }
    }'

linode-cli linodes create \
    --region us-east \
    --type g6-standard-4 \
    --label my-kafka-cluster-one-click-app \
    --image linode/ubuntu24.04 \
    --root_pass @C0mpl3x#P@ssw0rd \
    --stackscript_id 1377657 \
    --stackscript_data '{"token_password":"abc123def456hij789klm", "sudo_username":"jsmith", "client_count":"1", "cluster_size":"5", "country_name":"US", "state_or_province_name":"Massachusetts", "locality_name":"Cambridge", "organization_name":"Akamai Technologies", "email_address":"jsmith@example.com", "ca_common_name":"Kafka RootCA"}'

resource "linode_instance" "my-linode" {
    region         = "us-east"
    type           = "g6-standard-4"
    label          = "my-kafka-cluster-one-click-app"
    image          = "linode/ubuntu24.04"
    root_pass      = "@C0mpl3x#P@ssw0rd"
    stackscript_id = 1377657
    stackscript_data = {
        "token_password"         = "abc123def456hij789klm"
        "sudo_username"          = "jsmith"
        "client_count"           = "1"
        "cluster_size"           = "5"
        "country_name"           = "US"
        "state_or_province_name" = "Massachusetts"
        "locality_name"          = "Cambridge"
        "organization_name"      = "Akamai Technologies"
        "email_address"          = "jsmith@example.com"
        "ca_common_name"         = "Kafka RootCA"
    }
}

2. Connect to Kafka

Get the keystore and truststore

Once the deployment is complete, get your client certificates from the first Kafka node.

Authenticating clients requires a keystore and truststore. Access the first Kafka server to get these from /etc/kafka/ssl/keystore and /etc/kafka/ssl/truststore. Client certificates start with the word client followed by a number, depending on the number of clients requested, for example client1 and client2.

Use a secure method such as SSH or an encrypted HTTPS web UI when transferring the client certificates to the Kafka consumer/producer

Test the cluster

Run a quick test from any of the Kafka nodes using Kafka's utilities found in /etc/kafka/bin.

1. Create a client.properties file with the following content, replacing the placeholder values with your own:

   File: client.properties

   security.protocol=SSL
   ssl.truststore.location=/etc/kafka/ssl/truststore/server.truststore.jks
   ssl.truststore.password=CHANGE-ME
   ssl.keystore.location=/etc/kafka/ssl/keystore/client1.keystore.jks
   ssl.keystore.password=CHANGE-ME

2. Create a topic to test the connection and authentication.

   The output returns a message confirming the topic with the specified name was created.

   CommandOutput

   /etc/kafka/bin/kafka-topics.sh --create --topic test-ssl --bootstrap-server kafka3:9092 --command-config client.properties

   Created topic test-ssl.

3. Publish a message to the topic as a producer.

   Bash

   echo "Kafka rocks!" | /etc/kafka/bin/kafka-console-producer.sh --topic test-ssl --bootstrap-server kafka3:9092 --producer.config client.properties

4. Read the message as a consumer.

   The output returns the previously published message.

   CommandOutput

   /etc/kafka/bin/kafka-console-consumer.sh --topic test-ssl --from-beginning --bootstrap-server kafka3:9092 --consumer.config client.properties

   Kafka rocks!

Additional resources

For more information about the installed packages, see their official documentation.

• Apache Kafka Documentation