Fei's Blog
0%

Install the standalone Spark Cluster

Posted on In

Checklist

  • Ensure all nodes can resolve each other by hostnames/ips

  • Enable SSH with no need of password

  • Install JDK on each node

Export JAVA_HOME and SPARK_HOME in ~/.bashrc on each node

1
2
3
4
export JAVA_HOME=/home/feihu/jdk1.8.0_171
export PATH="$JAVA_HOME/bin:$PATH"
export SPARK_HOME=/home/feihu/spark-2.3.1-bin-hadoop2.7
export PATH="$SPARK_HOME/bin:$PATH"

Configure spark-defaults.conf

1
spark.master                     spark://codait-gpu2:7077

Add data nodes to slaves

Some scripts may be useful in the future (provided by Bryan)

node-install.sh 

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
#!/bin/bash

curl "https://repo.continuum.io/miniconda/Miniconda2-latest-Linux-x86_64.sh" > "Miniconda2-latest-Linux-x86_64.sh"

bash "Miniconda2-latest-Linux-x86_64.sh" -bf

DIR="/root/miniconda2/bin"
CONDA="$DIR"/conda
ACTIVATE="$DIR"/activate

"$CONDA" remove -y -n tf --all
"$CONDA" create -y -n tf -c conda-forge pyarrow=0.8 numpy pandas pip setuptools tensorflow

#"$CONDA" remove -y -n pyarrow --all
#"$CONDA" create -y -n pyarrow numpy pandas pip setuptools
#source "$ACTIVATE" pyarrow
#"$CONDA" install -y -c conda-forge pyarrow=0.8

setup-nodes.sh

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
#"apt-get install -y bzip2"
#"mkdir -p /opt/ibm && pushd /opt/ibm && tar -zxf ~/$SPARK_DIST && ln -sf spark* spark && popd"

CMDS=(
"conda remove --name tf --all"
"bash ~/node-install.sh"
)

for i in $(seq 1 $NUM_NODES);
do
    NODE="$CLUSTER$i.***.***.com"
    echo "setting up: $NODE"

    for F in "${FILES[@]}";
    do
        echo "copying $F"
        scp "$F" "$NODE":~/
    done

    for CMD in "${CMDS[@]}";
    do
        echo "executing: $CMD"
        ssh "$NODE" "$CMD"
    done
done