Creating Elasticsearch Cluster

Elasticsearch is a search engine based on the Lucene library. It provides a distributed, multitenant-capable full-text search engine with an HTTP web interface and schema-free JSON documents. Elasticsearch is developed in Java. This tutorial guides you on how to build the cluster.

We use 5 nodes

  • master1
  • master2
  • master3
  • data1
  • data2

Settings the hostname in all nodes

nano /etc/hosts

10.105.105.11 elastic-master1
10.105.105.12 elastic-master2
10.105.105.13 elastic-master3
10.105.105.14 elastic-data1
10.105.105.15 elastic-data2

Install Java in all nodes

sudo apt install openjdk-8-jdk -y
java -version

Install elasticsearch in all nodes

wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | apt-key add -
apt-get install apt-transport-https -y
echo "deb https://artifacts.elastic.co/packages/6.x/apt stable main" | tee -a /etc/apt/sources.list.d/elastic-6.x.list
apt-get update
apt-get install elasticsearch -y

Master configuration

Create elasticsearch configuration

cat > /etc/elasticsearch/elasticsearch.yml << EOF
cluster.name: production
bootstrap.memory_lock: true

node.name: \${HOSTNAME}
node.master: true
node.data: false

network.host: 0.0.0.0
http.port: 9200

path.logs: /var/log/elasticsearch

discovery.zen.ping.unicast.hosts: ["elastic-master1", "elastic-master2", "elastic-master3"]
discovery.zen.minimum_master_nodes: 2
EOF
cat > /etc/default/elasticsearch << EOF
ES_STARTUP_SLEEP_TIME=5
MAX_OPEN_FILES=65536
MAX_LOCKED_MEMORY=unlimited
EOF

Ensure that pages are not swapped out to disk by requesting the JVM to lock the heap in memory by setting LimitMEMLOCK=infinity

nano /usr/lib/systemd/system/elasticsearch.service
[Service]
LimitMEMLOCK=infinity

Increase the limit on the number of open files descriptors to user elasticsearch of 65536 or higher

cat > /etc/security/limits.conf << EOF
elasticsearch soft memlock unlimited
elasticsearch hard memlock unlimited
EOF

Increase the value of the mmap counts as elasticsearch uses mmapfs directory to store its indices:

sysctl -w vm.max_map_count=262144
sysctl -p /etc/sysctl.conf 

Reload the systemd daemon, enable and start elasticsearch

systemctl daemon-reload
systemctl enable elasticsearch
systemctl restart elasticsearch

Testing

netstat -tulpn | grep 9200
curl http://127.0.0.1:9200/_cluster/health?pretty
curl http://127.0.0.1:9200/_cat/nodes?v

Data Nodes

Create data folder

mkdir /data
mkdir /data/nodes
chown -R elasticsearch:elasticsearch /data
chown -R elasticsearch:elasticsearch /data/nodes

Create elasticsearch configuration

cat > /etc/elasticsearch/elasticsearch.yml << EOF
cluster.name: production
bootstrap.memory_lock: true

node.name: \${HOSTNAME}
node.master: false
node.data: true

network.host: 0.0.0.0
http.port: 9200

path.logs: /var/log/elasticsearch
path.data: /data

discovery.zen.ping.unicast.hosts: ["elastic-master1", "elastic-master2", "elastic-master3"]
discovery.zen.minimum_master_nodes: 2
EOF
cat > /etc/default/elasticsearch << EOF
ES_STARTUP_SLEEP_TIME=5
MAX_OPEN_FILES=65536
MAX_LOCKED_MEMORY=unlimited
EOF

Ensure that pages are not swapped out to disk by requesting the JVM to lock the heap in memory by setting LimitMEMLOCK=infinity

nano /usr/lib/systemd/system/elasticsearch.service
[Service]
LimitMEMLOCK=infinity

Increase the limit on the number of open files descriptors to user elasticsearch of 65536 or higher

cat > /etc/security/limits.conf << EOF
elasticsearch soft memlock unlimited
elasticsearch hard memlock unlimited
EOF

Reload the systemd daemon, enable and start elasticsearch

systemctl daemon-reload
systemctl enable elasticsearch
systemctl restart elasticsearch

Testing

netstat -tulpn | grep 9200
curl http://127.0.0.1:9200/_cluster/health?pretty
curl http://127.0.0.1:9200/_cat/nodes?v

Leave a Comment

Your email address will not be published.