如何部署Apache ZooKeeper 3.4.x集群?

Apache-Hadoop

1 基础知识

1.1 ZooKeeper的简介

– ZooKeeper是用于分布式应用程序的分布式开放源代码协调服务
– ZooKeeper公开一组简单的原语(具有特定功能的常驻内存中执行且不允许中断的程序段)
– ZooKeeper分布式程序基于原语实现同步、配置维护以及组和命名更高级别的服务。
– ZooKeeper的设计易于编程并使用文件系统熟悉的目录树结构样式设置数据模型
– ZooKeeper基于Java开发,可与Java和C绑定

1.2 ZooKeeper的实现机理

– ZooKeeper通过共享分布式进程之间的分层名称空间实现相互协调
– ZooKeeper的名称空间的组织方式类似于标准文件系统,名称空间由数据寄存器(znodes)组成
– ZonKeeper的znodes类似于标准文件系统文件和目录,不同点在于znodes存于内存中(具有高吞吐和低延迟特性)

1.3 ZooKeeper的特点

– ZooKeeper具有高性能的特性(能适应大型分布式系统)
– ZooKeeper具有高可用的特性(避免单点故障)
– ZooKeeper具有严格有序的访问特性(客户端实现复杂的同步原语)

1.4 ZooKeeper的复制


如上图所示,
– ZooKeeper将所有成员服务器节点视为一个集合,节点之间形成复制关系
– ZooKeeper维护节点的内存镜像、持久存储的事务日志和快照(大部分节点可用,则服务可用)
– ZooKeeper客户端与服务器之间使用TCP连接(包括发送请求、获取响应、监视事件和心跳)
– ZooKeeper客户端与单个节点服务器的TCP连接不可用,则客户端将于其他节点服务器建立连接

1.5 ZooKeeper的分层名称空间


– ZooKeeper名称空间与标准文件系统名称空间类似
– ZooKeeper名称空间由具有归属关系的节点(也称“Znode”)组成树状结构,父节点与子节点可使用“/”分隔
– ZooKeeper根节点通常使用“/”表示
– ZooKeeper节点都有唯一的路径(例如“/app1/p_1”)
– ZooKeeper节点都有与之关联的关联数据和子节点(与文件系统有差异)
– ZooKeeper节点用于存储协调的数据(包括状态、配置、位置)
– ZooKeeper节点的数据数据通常很小(字节到千字节之间)
– ZooKeeper节点可临时使用称为临时节点,当会话结束时,临时节点会被删除
– Znodes维护具有状态数据结构(包括数据更改、ACL更改和时间戳的版本号)
– Znodes数据每次更新时同时也会更新版本号(版本信息用于缓存验证和协调更新)
– Znodes的客户端检索数据的同时也会检索数据的版本号
– Znodes上可设置一次性的观察事件监视数据变化,当特定的数据变化事件发生时客户端会收到一个通知数据包

1.6 ZooKeeper的保障机制

ZooKeeper作为构建负责服务的基础服务(例如同步),它提供以下的保障机制,
– 顺序一致性(保证来自客户端的更新按照发送顺序进行应用)
– 事务特性(更新要么成功要么失败,不存在部分更新的结果)
– 统一的系统镜像(无论客户端连接那个数据节点,都会取得相同的服务视图)
– 可靠性(应用更新后,更新会持续到客户端更新完毕为止)
– 及时性(确保系统的客户端视图在特定时间范围内是最新的)

1.7 ZooKeeper的简单API

ZooKeeper的编程接口非常简单,支持以下操作,
– create,创建节点(位于树中的某个位置的节点)
– delete,删除节点
– exists,测试节点是否存在
– get data,读取节点数据
– set data,设置节点数据
– get children,检索节点的子节点列表
– sync,同步所有节点的数据

1.8 ZooKeeper的服务原理


如上图所示,
– ZooKeeper集群自有原子消息传递协议(原子层),该协议可保证各节点的本地副本不会出现分歧
– ZooKeeper集群通过消息传递层选举出主(leader)节点,其余作为从(followers)节点
– ZooKeeper集群主节点故障时,消息传递层会重新选举出新的主节点并保持新的主从同步
– ZooKeeper集群的数据库分内存和磁盘两部分,存储数据时先写入硬盘后存于内存
– ZooKeeper集群的所有节点均可为客户端提供服务,客户端只需与任一节点交互即可提交读写请求
– ZooKeeper集群的读请求只需直接从连接节点的本地数据副本返回(因为每个节点数据都保持一致)
– ZooKeeper集群的写请求需要由协议协商处理
– ZooKeeper集群收到写请求会先转发给主服务器,主服务器同意后分发到从服务器

2 部署要求

2.1 组件与系统的兼容性

Operating System Client Server Native Client Contrib
GNU/Linux Development and Production Development and Production Development and Production Development and Production
Solaris Development and Production Development and Production Not Supported Not Supported
FreeBSD Development and Production Development and Production Not Supported Not Supported
Windows Development and Production Development and Production Not Supported Not Supported
Mac OS X Development Only Development Only Not Supported Not Supported

相关的组件解析如下,
– Client,用于连接到ZooKeeper集合的Java客户端库
– Server,ZooKeeper集群节点上运行的Java服务器
– Native Client,用于连接到ZooKeeper集合的C语言客户端
– Contrib,代表多个可选的附加组件

2.2 组件与Java的兼容性

– ZooKeeper需要Java 1.7或更高版本
– ZooKeeper运行于FreeBSD需要openJDK7

2.3 运行要求

– ZooKeeper集群的最小集群是3台服务器(建议使用奇数的节点数量,例如“5”,“7”,“9”等)
– ZooKeeper集群节点数量偶数时,当故障节点等于正常节点数量,集群脑裂(无法投票界定故障节点)且不可用
– ZooKeeper集群节点建议接冗余网络,只接单个交换机集群容易因交换机单点故障而瘫痪
– ZooKeeper建议单台服务器只运行ZooKeeper,不建议混合其他角色
– ZooKeeper集群的Yahoo节点配置实例是双核处理器、2GB内存、80GB的IDE硬盘
– ZooKeeper集群应设置Java堆大小避免使用虚拟内存而导致性能严重下降(4GB机器保守设置最大堆大小为3GB)

3 最佳实践

3.1 环境信息

3.1.1 角色Hadoop HDFS的基本信息

hostname = hd0[1-5].cmdschool.org
ipaddress = 10.168.0.10[1-5] OS = centOS 7.6 x86_64

详细的角色分布如下,
Apache Hadoop HDFS NameNode(hdfs-nn) = hd01.cmdschool.org
Apache Hadoop HDFS SecondaryNameNode(hdfs-snn) = hd02.cmdschool.org
Apache Hadoop HDFS DataNode(hdfs-snn) = hd0[3-5].cmdschool.org
注:
– 如果你尚未配置HDFS,请参阅https://www.cmdschool.org/archives/5579
– 本章HDFS实例由“hadoop-2.6.5”提供

3.1.2 角色Apache Hadoop YARN的基本信息

hostname = hd0[1-5].cmdschool.org
ipaddress = 10.168.0.10[1-5] OS = centOS 7.6 x86_64

详细的角色分布如下,
Apache Hadoop YARN ResourceManager(yarn-rm) = hd01.cmdschool.org
Apache Hadoop YARN NodeManager(yarn-nm) = hd0[1-5].cmdschool.org
注:
– 如果你尚未配置YARN,请参阅https://www.cmdschool.org/archives/8313
– 本章YARN实例由“hadoop-2.6.5”提供

3.1.3 角色Apache ZooKeeper的基本信息

hostname = hd[06-10].cmdschool.org
ipaddress = 10.168.0.1[06-10] OS = centOS 7.6 x86_64

详细的角色分布如下,
Apache Hadoop ZooKeeper(zoo) = hd[06-10].cmdschool.org
注:
– ZooKeeper的“leader”与“followers”角色由集群投票自行选举
– 本章ZooKeeper实例的版本为“zookeeper-3.4.14”

3.2 系统环境配置

3.2.1 更新系统

In hd0[6-10],

yum install -y update

3.2.2 安装常用工具

In hd0[6-10],

yum install -y vim wget tree

3.3 软件环境配置

In hd0[6-10],

3.3.1 安装JDK

请参阅以下方法安装jdk-8u121-linux-x64,
https://www.cmdschool.org/archives/397
安装完成后,请使用如下命令检查JDK的安装,

java -version

命令显示如下,

java version "1.8.0_121"
Java(TM) SE Runtime Environment (build 1.8.0_121-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.121-b13, mixed mode)

3.3.2 配置名称解析

In hd0[1-5],

echo '10.168.0.106 hd06 hd06.cmdschool.org' >> /etc/hosts
echo '10.168.0.107 hd07 hd07.cmdschool.org' >> /etc/hosts
echo '10.168.0.108 hd08 hd08.cmdschool.org' >> /etc/hosts
echo '10.168.0.109 hd09 hd09.cmdschool.org' >> /etc/hosts
echo '10.168.0.110 hd10 hd10.cmdschool.org' >> /etc/hosts

In hd0[6-10],

echo '10.168.0.101 hd01 hd01.cmdschool.org' >> /etc/hosts
echo '10.168.0.102 hd02 hd02.cmdschool.org' >> /etc/hosts
echo '10.168.0.103 hd03 hd03.cmdschool.org' >> /etc/hosts
echo '10.168.0.104 hd04 hd04.cmdschool.org' >> /etc/hosts
echo '10.168.0.105 hd05 hd05.cmdschool.org' >> /etc/hosts
echo '10.168.0.106 hd06 hd06.cmdschool.org' >> /etc/hosts
echo '10.168.0.107 hd07 hd07.cmdschool.org' >> /etc/hosts
echo '10.168.0.108 hd08 hd08.cmdschool.org' >> /etc/hosts
echo '10.168.0.109 hd09 hd09.cmdschool.org' >> /etc/hosts
echo '10.168.0.110 hd10 hd10.cmdschool.org' >> /etc/hosts

注:以上仅用于测试,生产环境请使用DNS代替

3.3.3 配置管理节点到从节点的root用户公钥认证

In hd01,

ssh-copy-id -i ~/.ssh/id_rsa.pub root@hd06
ssh-copy-id -i ~/.ssh/id_rsa.pub root@hd07
ssh-copy-id -i ~/.ssh/id_rsa.pub root@hd08
ssh-copy-id -i ~/.ssh/id_rsa.pub root@hd09
ssh-copy-id -i ~/.ssh/id_rsa.pub root@hd10

配置完毕后,请使用如下命令测试,

ssh root@hd06
ssh root@hd07
ssh root@hd08
ssh root@hd09
ssh root@hd10

3.4 部署Apache ZooKeeper软件包

In hd[06-10],

3.4.1 配置运行用户

groupadd zookeeper
useradd -g zookeeper -d /var/lib/zookeeper/ -s /bin/false zookeeper

3.4.2 下载二进制安装包

cd ~
wget https://mirrors.tuna.tsinghua.edu.cn/apache/zookeeper/zookeeper-3.4.14/zookeeper-3.4.14.tar.gz

其他版本请从以下链接下载,
https://www.apache.org/dyn/closer.cgi/zookeeper/

3.4.3 解压安装包

cd ~
tar -xf zookeeper-3.4.14.tar.gz

3.4.4 部署软件包

cd ~
mv zookeeper-3.4.14 /usr/
chown zookeeper:zookeeper -R /usr/zookeeper-3.4.14
chmod 775 -R /usr/zookeeper-3.4.14

3.4.5 创建目录的软连接

mkdir -p /etc/zookeeper
ln -s /usr/zookeeper-3.4.14/conf /etc/zookeeper/conf
ln -s /usr/zookeeper-3.4.14 /usr/zookeeper

注:以上目的是管理方便

3.5 配置Apache ZooKeeper软件包

In hd[06-10],

3.5.1 配置环境公共的变量

vim /etc/profile.d/zookeeper.sh

加入如下配置,

export ZOOKEEPER_HOME=/usr/zookeeper-3.4.14
export PATH=${ZOOKEEPER_HOME}/bin:$PATH
export ZOOCFGDIR=/etc/zookeeper/conf
export ZOO_LOG_DIR=/var/log/zookeeper
export ZOOPIDFILE=/var/run/zookeeper/zookeeper_server.pid

创建完毕后,请使用如下命令导入环境变量,

source /etc/profile.d/zookeeper.sh

另外,我们建议根据配置文件创建目录,

mkdir -p /var/log/zookeeper/
chown zookeeper:zookeeper /var/log/zookeeper/
chmod 775 /var/log/zookeeper/

mkdir -p /var/run/zookeeper/
chown zookeeper:zookeeper /var/run/zookeeper/
chmod 775 /var/run/zookeeper/

3.5.2 配置运行所需的环境变量

In hd[06-10],

vim /etc/zookeeper/conf/java.env

加入如下内容,

. /etc/profile.d/jdk.sh
. /etc/profile.d/zookeeper.sh

注:以上两行用于导入JDK与zookeeper所需的环境变量

3.5.3 创建配置文件

In hd[06-10],

cp /etc/zookeeper/conf/zoo_sample.cfg /etc/zookeeper/conf/zoo.cfg
vim /etc/zookeeper/conf/zoo.cfg

修改配置如下,

tickTime=2000
dataDir=/var/lib/zookeeper
clientPort=2181
initLimit=5
syncLimit=2
server.1=hd06:2888:3888
server.2=hd07:2888:3888
server.3=hd08:2888:3888
server.4=hd09:2888:3888
server.5=hd10:2888:3888

根据以上配置,需要为主机创建“myid”,详细使用如下命令,
In hd06,

echo 1 > /var/lib/zookeeper/myid

In hd07,

echo 2 > /var/lib/zookeeper/myid

In hd08,

echo 3 > /var/lib/zookeeper/myid

In hd09,

echo 4 > /var/lib/zookeeper/myid

In hd10,

echo 5 > /var/lib/zookeeper/myid

3.5.4 测试配置

In hd0[6-10],

sudo -u zookeeper /usr/zookeeper-3.4.14/bin/zkServer.sh status /etc/zookeeper/conf/zoo.cfg

如果输出如下则配置可用,

ZooKeeper JMX enabled by default
Using config: /etc/zookeeper/conf/zoo.cfg
Error contacting service. It is probably not running.

3.5.5 手动测试程序启动

In hd0[6-10],

sudo -u zookeeper /usr/zookeeper-3.4.14/bin/zkServer.sh start /etc/zookeeper/conf/zoo.cfg

启动过程中的日志可使用如下命令查看,

cat /var/log/zookeeper/zookeeper.out

可使用以下命令查看启动的倾听端口,

netstat -antp | grep `pgrep -u zookeeper java`

可见如下显示,

tcp6       0      0 10.168.0.106:3888       :::*                    LISTEN      7991/java
tcp6       0      0 :::8080                 :::*                    LISTEN      7991/java
tcp6       0      0 :::44725                :::*                    LISTEN      7991/java
tcp6       0      0 :::2181                 :::*                    LISTEN      7991/java

可使用如下命令手动停止服务,

sudo -u zookeeper /usr/zookeeper-3.4.14/bin/zkServer.sh stop /etc/zookeeper/conf/zoo.cfg

3.5.6 配置服务控制脚本

In hd0[6-10],

vim /usr/lib/systemd/system/zoo.service

可加入如下配置,

[Unit]
Description=Apache ZooKeeper manager
Wants=network.target
Before=network.target
After=network-pre.target
Documentation=https://zookeeper.apache.org/doc/current/index.html

[Service]
Type=forking
ExecStartPre=/bin/sh -c 'mkdir -p /var/run/zookeeper;chown zookeeper:zookeeper /var/run/zookeeper;chmod 775 /var/run/zookeeper'
ExecStartPre=/bin/sh -c 'mkdir -p /var/log/zookeeper;chown zookeeper:zookeeper /var/log/zookeeper;chmod 775 /var/log/zookeeper'
ExecStart=/usr/bin/sudo -u zookeeper /usr/zookeeper-3.4.14/bin/zkServer.sh start /etc/zookeeper/conf/zoo.cfg
ExecStop=/usr/bin/sudo -u zookeeper /usr/zookeeper-3.4.14/bin/zkServer.sh stop /etc/zookeeper/conf/zoo.cfg
PIDFile=/var/run/zookeeper/zookeeper_server.pid
Restart=on-success

[Install]
WantedBy=multi-user.target

修改完脚本后,你需要使用如下命令重载服务,

systemctl daemon-reload

你可使用如下命令控制服务和查询状态,

systemctl start zoo.service
systemctl status zoo.service
systemctl stop zoo.service
systemctl restart zoo.service

测试完毕,建议你使用如下命令设置服务自动启动,

systemctl enable zoo.service

3.5.7 配置防火墙

In hd0[6-10],

firewall-cmd --permanent --add-port 2181/tcp --add-port 2888/tcp --add-port 3888/tcp
firewall-cmd --reload
firewall-cmd --list-all

3.5.8 使用客户端测试登录

In hd0[6-10],

zkCli.sh -server 127.0.0.1:2181

可见如下显示,

Connecting to 127.0.0.1:2181
2019-12-13 10:19:59,241 [myid:] - INFO  [main:Environment@100] - Client environment:zookeeper.version=3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT
2019-12-13 10:19:59,250 [myid:] - INFO  [main:Environment@100] - Client environment:host.name=hd10
2019-12-13 10:19:59,250 [myid:] - INFO  [main:Environment@100] - Client environment:java.version=1.8.0_121
2019-12-13 10:19:59,251 [myid:] - INFO  [main:Environment@100] - Client environment:java.vendor=Oracle Corporation
2019-12-13 10:19:59,252 [myid:] - INFO  [main:Environment@100] - Client environment:java.home=/usr/java/jdk1.8.0_121/jre
2019-12-13 10:19:59,252 [myid:] - INFO  [main:Environment@100] - Client environment:java.class.path=/usr/zookeeper-3.4.14/bin/../zookeeper-server/target/classes:/usr/zookeeper-3.4.14/bin/../build/classes:/usr/zookeeper-3.4.14/bin/../zookeeper-server/target/lib/*.jar:/usr/zookeeper-3.4.14/bin/../build/lib/*.jar:/usr/zookeeper-3.4.14/bin/../lib/slf4j-log4j12-1.7.25.jar:/usr/zookeeper-3.4.14/bin/../lib/slf4j-api-1.7.25.jar:/usr/zookeeper-3.4.14/bin/../lib/netty-3.10.6.Final.jar:/usr/zookeeper-3.4.14/bin/../lib/log4j-1.2.17.jar:/usr/zookeeper-3.4.14/bin/../lib/jline-0.9.94.jar:/usr/zookeeper-3.4.14/bin/../lib/audience-annotations-0.5.0.jar:/usr/zookeeper-3.4.14/bin/../zookeeper-3.4.14.jar:/usr/zookeeper-3.4.14/bin/../zookeeper-server/src/main/resources/lib/*.jar:/etc/zookeeper/conf:.:/usr/java/jdk1.8.0_121/lib:/usr/java/jdk1.8.0_121/jre/lib
2019-12-13 10:19:59,252 [myid:] - INFO  [main:Environment@100] - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2019-12-13 10:19:59,252 [myid:] - INFO  [main:Environment@100] - Client environment:java.io.tmpdir=/tmp
2019-12-13 10:19:59,252 [myid:] - INFO  [main:Environment@100] - Client environment:java.compiler=
2019-12-13 10:19:59,252 [myid:] - INFO  [main:Environment@100] - Client environment:os.name=Linux
2019-12-13 10:19:59,252 [myid:] - INFO  [main:Environment@100] - Client environment:os.arch=amd64
2019-12-13 10:19:59,252 [myid:] - INFO  [main:Environment@100] - Client environment:os.version=3.10.0-957.10.1.el7.x86_64
2019-12-13 10:19:59,252 [myid:] - INFO  [main:Environment@100] - Client environment:user.name=root
2019-12-13 10:19:59,252 [myid:] - INFO  [main:Environment@100] - Client environment:user.home=/root
2019-12-13 10:19:59,252 [myid:] - INFO  [main:Environment@100] - Client environment:user.dir=/root
2019-12-13 10:19:59,253 [myid:] - INFO  [main:ZooKeeper@442] - Initiating client connection, connectString=127.0.0.1:2181 sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@7aec35a
2019-12-13 10:19:59,292 [myid:] - INFO  [main-SendThread(localhost:2181):ClientCnxn$SendThread@1025] - Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
Welcome to ZooKeeper!
JLine support is enabled
2019-12-13 10:19:59,404 [myid:] - INFO  [main-SendThread(localhost:2181):ClientCnxn$SendThread@879] - Socket connection established to localhost/127.0.0.1:2181, initiating session
2019-12-13 10:19:59,437 [myid:] - INFO  [main-SendThread(localhost:2181):ClientCnxn$SendThread@1299] - Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x50003fc2c570000, negotiated timeout = 30000

WATCHER::

WatchedEvent state:SyncConnected type:None path:null
[zk: 127.0.0.1:2181(CONNECTED) 0]

然后,你可以通过以下命令获取命令的使用帮助,

[zk: 127.0.0.1:2181(CONNECTED) 0] help

可见如下显示,

ZooKeeper -server host:port cmd args
        stat path [watch]
        set path data [version]
        ls path [watch]
        delquota [-n|-b] path
        ls2 path [watch]
        setAcl path acl
        setquota -n|-b val path
        history
        redo cmdno
        printwatches on|off
        delete path [version]
        sync path
        listquota path
        rmr path
        get path [watch]
        create [-s] [-e] path data acl
        addauth scheme auth
        quit
        getAcl path
        close
        connect host:port

然后,你可以通过以下命令取得集群配置,

[zk: 127.0.0.1:2181(CONNECTED) 0] ls /

可见如下显示,

[zookeeper]

由于命令不是本章重点,这里不再演示, 详细请参阅官方的以下链接,
https://zookeeper.apache.org/doc/current/zookeeperStarted.html

参阅文档
=====================

zookeeper的官方首页
———————–
https://zookeeper.apache.org/

zookeeper wiki
———————–
https://cwiki.apache.org/confluence/display/ZOOKEEPER/Index

zookeeper doc
—————
https://zookeeper.apache.org/doc/current/index.html

zookeeper over
—————-
https://zookeeper.apache.org/doc/current/zookeeperOver.html

zookerper administrator’s guide
——————————–
https://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_systemReq

zookeeper download
——————-
http://zookeeper.apache.org/releases.html

原语的概念
—————-
https://baike.baidu.com/item/%E5%8E%9F%E8%AF%AD/3794081?fr=aladdin

没有评论

发表评论

Apache-Hadoop
如何部署HBase的集群?

1 基础知识 1.1 HBase的简介 – HBase是一个分布式的可扩展的大数据存储 …

Apache-Hadoop
如何部署Apache ZooKeeper 3.5.x集群?

1 基础知识 1.1 ZooKeeper的简介 – ZooKeeper是用于分布式应用程 …

Apache-Hadoop
如何部署Apache Hadoop YARN集群?

1 基础知识 1.1 YARN的介绍 – YARN英文全称为“Yet Another R …