如何部署Elasticsearch 8.x集群？
- By : Will
- Category : Big Data Framework, Elastic Stack
1 基础知识
1.1 集群的介绍
– Elasticsearch集群允许节点单点故障
– Elasticsearch集群出现单点故障时，正常的节点会替代非正常的节点继续提供服务
1.2 集群角色
– master role，即主角色，拥有该角色的节点为其他节点的主控节点
– data role，即数据角色，拥有该角色的节点负责保存集群的数据和相关的操作，可细分为如下角色，
— data_content role，即内容数据角色，拥有该角色的节点保存用户创建的内容
— data_hot role，即热数据角色，拥有该角色的节点会存储时间序列数据并具备快熟的数据读写能力（需硬件支持）
— data_warm role，即暖数据角色，拥有该角色的节点索引无需更新，但可查询
— data_cold role，即冷数据角色，拥有该角色的节点索引只读且访问频率较低（可用性能较低硬件）
– ingest role，即提取角色，拥有该角色的节点提供一个以上的管道处理能力
– ml role，即机器学习（ Machine learning）角色，拥有该角色的节点拥有机器学习能力
– remote_cluster_client role，即远程节点角色，拥有该角色的节点拥有远程连接其他集群的能力
– transform，即转换角色，拥有该角色的节点具有转换能力（转换索引为汇总索引）。
1.3 集群的节点
1.3.1 集群的节点类型
– Master-eligible node，即及格主节点（拥有主角色），由其他节点选举产生的集群主控及格节点
– Data node，即数据节点（拥有数据角色），保存集群数据和相关操作（例如CRUD、search、aggregations）的节点
– Ingest node，即提取节点（拥有提取角色），提供一个以上的管道处理节点
– Remote-eligible node，即及格远程节点（拥有远程节点角色），承担远程连接其他集群的及格节点
– Machine learning node，即机器学习节点（拥有机器学习角色），承担机器学习功能的节点
– Transform node，即转换节点（拥有机器学习角色），承担转换（转换索引为汇总索引）功能的节点
1.3.2 弹性集群的节点要求
– 至少有一个被选举出及格主节点
– 每个角色至少分配到一个节点中
– 每个复制都必须有一个分片
基于以上，集群必须具备以下条件，
– 集群必须至少有三个节点（节点数量要求奇数）
– 每个角色必须分配到两个节点
– 每个分片至少有两个副本（一个主副本和一个以上的从副本）
2 最佳实践
2.1 集群环境
2.1.1 集群信息
Host Name = elasticsearch[01-03].cmdschool.org
IP Address = 10.168.0.[100 – 102]
OS = Oracle Linux 9.x x86_64
Elasticsearch Version = 8.11.3
2.1.2 节点的部署
如果你尚未部署节点，烦请参阅如下章节部署每个集群节点，
2.1.3 名称解析配置
In es[001-003]，
echo '10.168.0.100 elasticsearch01 elasticsearch01.cmdschool.org' >> /etc/hosts echo '10.168.0.101 elasticsearch02 elasticsearch02.cmdschool.org' >> /etc/hosts echo '10.168.0.102 elasticsearch03 elasticsearch03.cmdschool.org' >> /etc/hosts
2.1.4 配置公钥认证（可选）
In elasticsearch01，
请参阅如下章节配置公钥认证，
然后，通过如下命令测试登录，
ssh elasticsearch02 ssh elasticsearch03
2.2 配置集群
2.2.1 修改配置
In elasticsearch01（主节点A）,
vim /etc/elasticsearch/elasticsearch.yml
配置修改如下，
cluster.name: esCluster01 node.name: elasticsearch01 node.attr.rack: r1 path.data: /data/elasticsearch path.logs: /var/log/elasticsearch network.host: _ens192:ipv4_ http.port: 9200 discovery.seed_hosts: ["elasticsearch01", "elasticsearch02", "elasticsearch03"] cluster.initial_master_nodes: ["elasticsearch01", "elasticsearch02", "elasticsearch03"] action.destructive_requires_name: true xpack.security.enabled: false xpack.security.enrollment.enabled: false xpack.security.http.ssl: enabled: false keystore.path: certs/http.p12 xpack.security.transport.ssl: enabled: false verification_mode: certificate keystore.path: certs/transport.p12 truststore.path: certs/transport.p12 http.host: 0.0.0.0
– 参数“cluster.name”声明集群的名称
– 参数“node.name”声明节点名称
– 参数“node.attr.rack”声明节点所在的机架
– 参数“path.data”声明节点数据存储的路径
– 参数“path.logs”声明节点日志的输出路径
– 参数“network.host”声明节点的主机地址（支持特殊设置，例如“_ens192_”或_ens192:ipv4_”等）
– 参数“http.port”声明http服务的端口
– 参数“discovery.seed_hosts”声明可见的集群成员主机名称，用户形成集群（推荐三个）
– 参数“cluster.initial_master_nodes”声明集群主节点的列表
– 参数“action.destructive_requires_name”声明破坏性的请求只能通过集群初始化或滚动重启来设置
In elasticsearch02（主节点B）,
vim /etc/elasticsearch/elasticsearch.yml
配置修改如下，
cluster.name: esCluster01 node.name: elasticsearch02 node.attr.rack: r2 path.data: /data/elasticsearch path.logs: /var/log/elasticsearch network.host: _ens192:ipv4_ http.port: 9200 discovery.seed_hosts: ["elasticsearch01", "elasticsearch02", "elasticsearch03"] cluster.initial_master_nodes: ["elasticsearch01", "elasticsearch02", "elasticsearch03"] action.destructive_requires_name: true xpack.security.enabled: false xpack.security.enrollment.enabled: false xpack.security.http.ssl: enabled: false keystore.path: certs/http.p12 xpack.security.transport.ssl: enabled: false verification_mode: certificate keystore.path: certs/transport.p12 truststore.path: certs/transport.p12 http.host: 0.0.0.0
In elasticsearch03（数据节点）,
vim /etc/elasticsearch/elasticsearch.yml
配置修改如下，
cluster.name: esCluster01 node.name: elasticsearch03 node.attr.rack: r3 path.data: /data/elasticsearch path.logs: /var/log/elasticsearch network.host: _ens192:ipv4_ http.port: 9200 discovery.seed_hosts: ["elasticsearch01", "elasticsearch02", "elasticsearch03"] cluster.initial_master_nodes: ["elasticsearch01", "elasticsearch02", "elasticsearch03"] action.destructive_requires_name: true xpack.security.enabled: false xpack.security.enrollment.enabled: false xpack.security.http.ssl: enabled: false keystore.path: certs/http.p12 xpack.security.transport.ssl: enabled: false verification_mode: certificate keystore.path: certs/transport.p12 truststore.path: certs/transport.p12 http.host: 0.0.0.0
2.2.2 重启服务使配置生效
systemctl restart elasticsearch
2.2.3 检查端口倾听
In es[001-003]，
for i in `pgrep -u elasticsearch java`; do netstat -anp | grep $i; done
其中一个节点显示如下，
unix 3 [ ] STREAM CONNECTED 69134 8127/java tcp6 0 0 10.10.200.100:9300 :::* LISTEN 8185/java tcp6 0 0 :::9200 :::* LISTEN 8185/java unix 3 [ ] STREAM CONNECTED 71765 8185/java unix 3 [ ] STREAM CONNECTED 71764 8185/java unix 2 [ ] STREAM CONNECTED 71762 8185/java
注：以上可见“9200”与“9300”端口已经倾听于主机的接口地址，允许与非本机通讯。
2.2.4 开启服务端口
In es[001-003]，
firewall-cmd --permanent --add-port 9200/tcp --add-port 9300/tcp firewall-cmd --reload firewall-cmd --list-all
端口开启后，可使用如下命令测试，
curl http://elasticsearch01:9200 http://elasticsearch02:9200 http://elasticsearch03:9200
可见如下显示，
{
"name" : "elasticsearch01",
"cluster_name" : "esCluster01",
"cluster_uuid" : "tMf9ASzLQ0eqwBAdIdXPFw",
"version" : {
"number" : "8.11.3",
"build_flavor" : "default",
"build_type" : "rpm",
"build_hash" : "64cf052f3b56b1fd4449f5454cb88aca7e739d9a",
"build_date" : "2023-12-08T11:33:53.634979452Z",
"build_snapshot" : false,
"lucene_version" : "9.8.0",
"minimum_wire_compatibility_version" : "7.17.0",
"minimum_index_compatibility_version" : "7.0.0"
},
"tagline" : "You Know, for Search"
}
{
"name" : "elasticsearch02",
"cluster_name" : "esCluster01",
"cluster_uuid" : "tMf9ASzLQ0eqwBAdIdXPFw",
#...
}
{
"name" : "elasticsearch03",
"cluster_name" : "esCluster01",
"cluster_uuid" : "tMf9ASzLQ0eqwBAdIdXPFw",
#...
}
2.2.5 初始化集群节点
In elasticsearch01，
ssh elasticsearch02 systemctl stop elasticsearch ssh elasticsearch02 rm -rf /data/elasticsearch/* ssh elasticsearch02 systemctl start elasticsearch ssh elasticsearch03 systemctl stop elasticsearch ssh elasticsearch03 rm -rf /data/elasticsearch/* ssh elasticsearch03 systemctl start elasticsearch
注：以上删除操作目的是防止新节点加入集群前存有旧数据，影响节点加入集群（elasticsearch01不用操作）。
2.2.6 检查集群状态
curl -X GET "elasticsearch01:9200/_cluster/health?pretty" "elasticsearch02:9200/_cluster/health?pretty" "elasticsearch03:9200/_cluster/health?pretty"
可见如下显示，
{
"cluster_name" : "esCluster01",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 3,
"number_of_data_nodes" : 3,
"active_primary_shards" : 0,
"active_shards" : 0,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 100.0
}
{
"cluster_name" : "esCluster01",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 3,
"number_of_data_nodes" : 3,
"active_primary_shards" : 0,
"active_shards" : 0,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 100.0
}
{
"cluster_name" : "esCluster01",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 3,
"number_of_data_nodes" : 3,
"active_primary_shards" : 0,
"active_shards" : 0,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 100.0
}
2.3 集成elastic其他服务
2.3.1 集成Kibana
2.3.2 集成Filebeat
2.3.3 集成Logstash
参阅文档
===========================
集群支持矩阵
—————–
https://www.elastic.co/cn/support/matrix#matrix_compatibility
集群的安装配置
——————
https://www.elastic.co/guide/en/elasticsearch/reference/8.11/rpm.html
集群健康API
—————
https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-health.html
配置文件解析
—————
https://www.elastic.co/guide/en/elasticsearch/reference/7.x/important-settings.html
elasticearch集群索引
———————-
https://www.elastic.co/guide/en/elasticsearch/reference/current/high-availability.html
elasticsearch文档索引
——————
https://www.elastic.co/guide/en/elasticsearch/reference/current/index.html
数据角色
———-
https://www.elastic.co/guide/en/elasticsearch/reference/8.11/data-tier-shard-filtering.html
节点角色的分配设置
———–
https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-node.html
elastic github
————–
https://github.com/elastic
没有评论