Apache Spark (cluster)

Deploy Apache Spark, an open source unified analytics engine for large-scale data processing. It's designed for both batch and streaming data processing.

This app deploys an Apache Spark standalone cluster with one master node and two worker nodes, configured with secure web access and ready to run in a distributed environment.

1. Deploy Apache Spark cluster

 Estimated deployment time: 5–7 minutes

Software included

Software installed on your Linode when deploying the app.

Base distribution and plan

• Supported distribution: Ubuntu 24.04 LTS
• Recommended plan: 4GB RAM

Configuration options

StackScript ID: 1403818

Notes:

• After the app deployment completes, the password for your limited sudo user is generated and stored in the .credentials file in the home directory, along with application-specific passwords. Log in to your instance as root through the Lish console or SSH, then run cat /home/$USERNAME/.credentials to view its contents.
• You can automatically configure a custom domain (optional) during deployment. To do that:
  1. First, configure your domain to use Linode’s name servers via your registrar. See Configure your domain's authoritative name servers for details.
  2. Once you complete that, specify the Linode API token, subdomain, and domain for the app in Cloud Manager or via API/CLI/Terraform.

Use API, CLI, or Terraform

curl --location 'https://api.linode.com/v4/linode/instances' \
    --header 'Content-Type: application/json' \
    --header 'Accept: application/json' \
    --header 'Authorization: Bearer abc123def456hij789klm' \
    --data-raw '{
        "region": "us-east",
        "type": "g6-standard-2",
        "image": "linode/ubuntu24.04",
        "label": "my-apache-spark-cluster-one-click-app",
        "root_pass": "@C0mpl3x#P@ssw0rd",
        "stackscript_id": 1403818,
        "stackscript_data": {
            "token_password": "abc123def456hij789klm",
            "soa_email_address": "jsmith@example.com",
            "sudo_username": "jsmith",
            "spark_version": "3.5.5",
            "spark_user": "jsmith-admin"
        }
    }'

linode-cli linodes create \
    --region us-east \
    --type g6-standard-2 \
    --label my-apache-spark-cluster-one-click-app \
    --image linode/ubuntu24.04 \
    --root_pass @C0mpl3x#P@ssw0rd \
    --stackscript_id 1403818 \
    --stackscript_data '{"token_password":"abc123def456hij789klm", "soa_email_address":"jsmith@example.com", "sudo_username":"jsmith", "spark_version":"3.5.5", "spark_user":"jsmith-admin"}'

resource "linode_instance" "my-linode" {
    region         = "us-east"
    type           = "g6-standard-2"
    label          = "my-apache-spark-cluster-one-click-app"
    image          = "linode/ubuntu24.04"
    root_pass      = "@C0mpl3x#P@ssw0rd"
    stackscript_id = 1403818
    stackscript_data = {
        "token_password"    = "abc123def456hij789klm"
        "soa_email_address" = "jsmith@example.com"
        "sudo_username"     = "jsmith"
        "spark_version"     = "3.5.5"
        "spark_user"        = "jsmith-admin"
    }
}

2. Access Apache Spark

Once the Apache Spark cluster is deployed, you can access the Apache Spark UI. To do that:

1. Log in to your Linode instance as a root user using either the Lish console or SSH.

2. After logging in, you can find the URL to the Spark UI in the motd (message of the day).

   The URL is either the domain you entered when deploying the cluster or the rDNS value of the master node. To learn more about viewing the rDNS value, see Manage IP addresses on a Linode.

   Bash

   cat /etc/motd

   To delete the motd, run rm /etc/motd.

3. Then run cat /home/$SUDO_USER/.credentials to see the file contents. It contains the credentials for the sudo user you created. For example, if you created a user named admin, the credentials file can be found in /home/admin/.credentials.

4. Provide your username and password from the .credentials file on the Spark UI login page.

   Notes:

   • The Apache Spark cluster needs access to external storage, such as Amazon S3 or S3-compatible storage, HDFS, Azure Blob Storage, Apache HBase, or your local filesystem. For more details on this, see Integration with Cloud Infrastructures.
   • The spark-shell is included with the installation. It's an interactive command-line shell provided by Apache Spark that allows you to interact with and perform data queries on a Spark cluster in real-time.

Additional resources

For more information about the installed packages, see their official documentation.

• Apache Spark Documentation