如何部署Elasticsearch 8.x集群?
- By : Will
- Category : Big Data Framework, Elastic Stack

1 基础知识
1.1 集群的介绍
– Elasticsearch集群允许节点单点故障
– Elasticsearch集群出现单点故障时,正常的节点会替代非正常的节点继续提供服务
1.2 集群角色
– master role,即主角色,拥有该角色的节点为其他节点的主控节点
– data role,即数据角色,拥有该角色的节点负责保存集群的数据和相关的操作,可细分为如下角色,
— data_content role,即内容数据角色,拥有该角色的节点保存用户创建的内容
— data_hot role,即热数据角色,拥有该角色的节点会存储时间序列数据并具备快熟的数据读写能力(需硬件支持)
— data_warm role,即暖数据角色,拥有该角色的节点索引无需更新,但可查询
— data_cold role,即冷数据角色,拥有该角色的节点索引只读且访问频率较低(可用性能较低硬件)
– ingest role,即提取角色,拥有该角色的节点提供一个以上的管道处理能力
– ml role,即机器学习( Machine learning)角色,拥有该角色的节点拥有机器学习能力
– remote_cluster_client role,即远程节点角色,拥有该角色的节点拥有远程连接其他集群的能力
– transform,即转换角色,拥有该角色的节点具有转换能力(转换索引为汇总索引)。
1.3 集群的节点
1.3.1 集群的节点类型
– Master-eligible node,即及格主节点(拥有主角色),由其他节点选举产生的集群主控及格节点
– Data node,即数据节点(拥有数据角色),保存集群数据和相关操作(例如CRUD、search、aggregations)的节点
– Ingest node,即提取节点(拥有提取角色),提供一个以上的管道处理节点
– Remote-eligible node,即及格远程节点(拥有远程节点角色),承担远程连接其他集群的及格节点
– Machine learning node,即机器学习节点(拥有机器学习角色),承担机器学习功能的节点
– Transform node,即转换节点(拥有机器学习角色),承担转换(转换索引为汇总索引)功能的节点
1.3.2 弹性集群的节点要求
– 至少有一个被选举出及格主节点
– 每个角色至少分配到一个节点中
– 每个复制都必须有一个分片
基于以上,集群必须具备以下条件,
– 集群必须至少有三个节点(节点数量要求奇数)
– 每个角色必须分配到两个节点
– 每个分片至少有两个副本(一个主副本和一个以上的从副本)
2 最佳实践
2.1 集群环境
2.1.1 集群信息
Host Name = elasticsearch[01-03].cmdschool.org
IP Address = 10.168.0.[100 – 102]
OS = Oracle Linux 9.x x86_64
Elasticsearch Version = 8.11.3
2.1.2 节点的部署
如果你尚未部署节点,烦请参阅如下章节部署每个集群节点,
2.1.3 名称解析配置
In es[001-003],
echo '10.168.0.100 elasticsearch01 elasticsearch01.cmdschool.org' >> /etc/hosts echo '10.168.0.101 elasticsearch02 elasticsearch02.cmdschool.org' >> /etc/hosts echo '10.168.0.102 elasticsearch03 elasticsearch03.cmdschool.org' >> /etc/hosts
2.1.4 配置公钥认证(可选)
In elasticsearch01,
请参阅如下章节配置公钥认证,
然后,通过如下命令测试登录,
ssh elasticsearch02 ssh elasticsearch03
2.2 配置集群
2.2.1 修改配置
In elasticsearch01(主节点A),
vim /etc/elasticsearch/elasticsearch.yml
配置修改如下,
cluster.name: esCluster01 node.name: elasticsearch01 node.attr.rack: r1 path.data: /data/elasticsearch path.logs: /var/log/elasticsearch network.host: _ens192:ipv4_ http.port: 9200 discovery.seed_hosts: ["elasticsearch01", "elasticsearch02", "elasticsearch03"] cluster.initial_master_nodes: ["elasticsearch01", "elasticsearch02", "elasticsearch03"] action.destructive_requires_name: true xpack.security.enabled: false xpack.security.enrollment.enabled: false xpack.security.http.ssl: enabled: false keystore.path: certs/http.p12 xpack.security.transport.ssl: enabled: false verification_mode: certificate keystore.path: certs/transport.p12 truststore.path: certs/transport.p12 http.host: 0.0.0.0
– 参数“cluster.name”声明集群的名称
– 参数“node.name”声明节点名称
– 参数“node.attr.rack”声明节点所在的机架
– 参数“path.data”声明节点数据存储的路径
– 参数“path.logs”声明节点日志的输出路径
– 参数“network.host”声明节点的主机地址(支持特殊设置,例如“_ens192_”或_ens192:ipv4_”等)
– 参数“http.port”声明http服务的端口
– 参数“discovery.seed_hosts”声明可见的集群成员主机名称,用户形成集群(推荐三个)
– 参数“cluster.initial_master_nodes”声明集群主节点的列表
– 参数“action.destructive_requires_name”声明破坏性的请求只能通过集群初始化或滚动重启来设置
In elasticsearch02(主节点B),
vim /etc/elasticsearch/elasticsearch.yml
配置修改如下,
cluster.name: esCluster01 node.name: elasticsearch02 node.attr.rack: r2 path.data: /data/elasticsearch path.logs: /var/log/elasticsearch network.host: _ens192:ipv4_ http.port: 9200 discovery.seed_hosts: ["elasticsearch01", "elasticsearch02", "elasticsearch03"] cluster.initial_master_nodes: ["elasticsearch01", "elasticsearch02", "elasticsearch03"] action.destructive_requires_name: true xpack.security.enabled: false xpack.security.enrollment.enabled: false xpack.security.http.ssl: enabled: false keystore.path: certs/http.p12 xpack.security.transport.ssl: enabled: false verification_mode: certificate keystore.path: certs/transport.p12 truststore.path: certs/transport.p12 http.host: 0.0.0.0
In elasticsearch03(数据节点),
vim /etc/elasticsearch/elasticsearch.yml
配置修改如下,
cluster.name: esCluster01 node.name: elasticsearch03 node.attr.rack: r3 path.data: /data/elasticsearch path.logs: /var/log/elasticsearch network.host: _ens192:ipv4_ http.port: 9200 discovery.seed_hosts: ["elasticsearch01", "elasticsearch02", "elasticsearch03"] cluster.initial_master_nodes: ["elasticsearch01", "elasticsearch02", "elasticsearch03"] action.destructive_requires_name: true xpack.security.enabled: false xpack.security.enrollment.enabled: false xpack.security.http.ssl: enabled: false keystore.path: certs/http.p12 xpack.security.transport.ssl: enabled: false verification_mode: certificate keystore.path: certs/transport.p12 truststore.path: certs/transport.p12 http.host: 0.0.0.0
2.2.2 重启服务使配置生效
systemctl restart elasticsearch
2.2.3 检查端口倾听
In es[001-003],
for i in `pgrep -u elasticsearch java`; do netstat -anp | grep $i; done
其中一个节点显示如下,
unix 3 [ ] STREAM CONNECTED 69134 8127/java tcp6 0 0 10.10.200.100:9300 :::* LISTEN 8185/java tcp6 0 0 :::9200 :::* LISTEN 8185/java unix 3 [ ] STREAM CONNECTED 71765 8185/java unix 3 [ ] STREAM CONNECTED 71764 8185/java unix 2 [ ] STREAM CONNECTED 71762 8185/java
注:以上可见“9200”与“9300”端口已经倾听于主机的接口地址,允许与非本机通讯。
2.2.4 开启服务端口
In es[001-003],
firewall-cmd --permanent --add-port 9200/tcp --add-port 9300/tcp firewall-cmd --reload firewall-cmd --list-all
端口开启后,可使用如下命令测试,
curl http://elasticsearch01:9200 http://elasticsearch02:9200 http://elasticsearch03:9200
可见如下显示,
{ "name" : "elasticsearch01", "cluster_name" : "esCluster01", "cluster_uuid" : "tMf9ASzLQ0eqwBAdIdXPFw", "version" : { "number" : "8.11.3", "build_flavor" : "default", "build_type" : "rpm", "build_hash" : "64cf052f3b56b1fd4449f5454cb88aca7e739d9a", "build_date" : "2023-12-08T11:33:53.634979452Z", "build_snapshot" : false, "lucene_version" : "9.8.0", "minimum_wire_compatibility_version" : "7.17.0", "minimum_index_compatibility_version" : "7.0.0" }, "tagline" : "You Know, for Search" } { "name" : "elasticsearch02", "cluster_name" : "esCluster01", "cluster_uuid" : "tMf9ASzLQ0eqwBAdIdXPFw", #... } { "name" : "elasticsearch03", "cluster_name" : "esCluster01", "cluster_uuid" : "tMf9ASzLQ0eqwBAdIdXPFw", #... }
2.2.5 初始化集群节点
In elasticsearch01,
ssh elasticsearch02 systemctl stop elasticsearch ssh elasticsearch02 rm -rf /data/elasticsearch/* ssh elasticsearch02 systemctl start elasticsearch ssh elasticsearch03 systemctl stop elasticsearch ssh elasticsearch03 rm -rf /data/elasticsearch/* ssh elasticsearch03 systemctl start elasticsearch
注:以上删除操作目的是防止新节点加入集群前存有旧数据,影响节点加入集群(elasticsearch01不用操作)。
2.2.6 检查集群状态
curl -X GET "elasticsearch01:9200/_cluster/health?pretty" "elasticsearch02:9200/_cluster/health?pretty" "elasticsearch03:9200/_cluster/health?pretty"
可见如下显示,
{ "cluster_name" : "esCluster01", "status" : "green", "timed_out" : false, "number_of_nodes" : 3, "number_of_data_nodes" : 3, "active_primary_shards" : 0, "active_shards" : 0, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 0, "delayed_unassigned_shards" : 0, "number_of_pending_tasks" : 0, "number_of_in_flight_fetch" : 0, "task_max_waiting_in_queue_millis" : 0, "active_shards_percent_as_number" : 100.0 } { "cluster_name" : "esCluster01", "status" : "green", "timed_out" : false, "number_of_nodes" : 3, "number_of_data_nodes" : 3, "active_primary_shards" : 0, "active_shards" : 0, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 0, "delayed_unassigned_shards" : 0, "number_of_pending_tasks" : 0, "number_of_in_flight_fetch" : 0, "task_max_waiting_in_queue_millis" : 0, "active_shards_percent_as_number" : 100.0 } { "cluster_name" : "esCluster01", "status" : "green", "timed_out" : false, "number_of_nodes" : 3, "number_of_data_nodes" : 3, "active_primary_shards" : 0, "active_shards" : 0, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 0, "delayed_unassigned_shards" : 0, "number_of_pending_tasks" : 0, "number_of_in_flight_fetch" : 0, "task_max_waiting_in_queue_millis" : 0, "active_shards_percent_as_number" : 100.0 }
2.3 集成elastic其他服务
2.3.1 集成Kibana
2.3.2 集成Filebeat
2.3.3 集成Logstash
参阅文档
===========================
集群支持矩阵
—————–
https://www.elastic.co/cn/support/matrix#matrix_compatibility
集群的安装配置
——————
https://www.elastic.co/guide/en/elasticsearch/reference/8.11/rpm.html
集群健康API
—————
https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-health.html
配置文件解析
—————
https://www.elastic.co/guide/en/elasticsearch/reference/7.x/important-settings.html
elasticearch集群索引
———————-
https://www.elastic.co/guide/en/elasticsearch/reference/current/high-availability.html
elasticsearch文档索引
——————
https://www.elastic.co/guide/en/elasticsearch/reference/current/index.html
数据角色
———-
https://www.elastic.co/guide/en/elasticsearch/reference/8.11/data-tier-shard-filtering.html
节点角色的分配设置
———–
https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-node.html
elastic github
————–
https://github.com/elastic
没有评论