如何配置nginx-plus的HA?

Highly available

1 基础知识

1.1 基本概念

1.1.1 访问限制

– 访问限制分连接限制、请求限制和带宽限制
– 连接限制是指根据定义的键值(例如IP)限制每个键值的链接数量
– 请求限制是指根据定义的键值(例如IP)限制请求的频率(单位时间内超过固定的频率请求将被推迟处理)
– 带宽限制是指根据定义的键值(例如IP)限制请求的带宽
– 连接限制由“http_limit_conn_module”模块实现
– 请求限制由“http_limit_req_module”模块实现
– 带宽限制由“ngx_http_core_module”模块的“limit_rate”指令实现

1.1.2 会话持久

– 会话持久即将NGINX识别的用户会话保持到同一台上游服务器
– 会话持久有三种实现方式,即cookie、route和learn
– Nginx将会话cookie加到上游组的首次响应中并标识响应服务器,后续请求根据客户端cookie值将请求路由到之前的服务器
– Nginx首次请求时为客户端分配“路由”,后续所有请求都与“route”指令参数(cookie或url)比较出之前的服务器并送达
– Nginx首次请求时检查请求和响应来生成会话标识,然后通过学习会话标识与服务器的对应关系并准确送达之前的服务器

1.1.3 键值数据存储

– Nginx支持使用后端数据库存储IP地址的黑白名单
– Nginx的后端数据库存储通过“ngx_http_keyval_module”模块实现

1.2 集群配置同步

1.2.1 集群配置同步的简介


– 集群配置同步适用于部署两台以上NGINX Plus设备的HA集群环境
– 集群配置同步允许管理员将配置从集群中的一台设备推动送到其他设备

1.2.2 部署的步骤

– 集群配置同步通过nginx-sync软件包实现(其实是基于rsync开发的同步脚本)
– 集群配置同步需要配置ssh的公钥认证(免密码访问其他副节点)

1.3 集群状态共享

1.3.1 集群节点的状态数据共享

NGINX Plus允许集群中的节点之间共享一些状态数据

1.3.2 集群状态数据共享功能

– 集群支持节点之间粘性学习会话持久
– 集群支持节点之间访问限制
– 集群支持节点之间键值存储数据

2 最佳实践

2.1 服务器环境

2.1.1 服务器信息

OS = CentOS 7.8 x86_64
hostname = rproxy0[1-3].cmdschool.org
ip address = 10.168.0.8[1-3]

2.1.2 客户端信息

OS = CentOS 7.8 x86_64
hostname = client.cmdschool.org
ip address = 10.168.0.8

2.1.3 安装nginx-plus

In rproxy0[1-3],
如果你的环境尚未部署nginx-plus,你可以参阅如下链接部署,
https://www.cmdschool.org/archives/7931

2.1.4 配置名称解析

In rproxy0[1-3],

echo '10.168.0.81 rproxy01 rproxy01.cmdschool.org' >> /etc/hosts
echo '10.168.0.82 rproxy02 rproxy02.cmdschool.org' >> /etc/hosts
echo '10.168.0.83 rproxy03 rproxy03.cmdschool.org' >> /etc/hosts

2.2 软件环境

In rproxy01,

2.2.1 生成公钥和私钥

ssh-keygen

注:以上一路回车,不需要设置任何参数

2.2.2 复制公钥到群集中的其他机器

ssh-copy-id -i ~/.ssh/id_rsa.pub rproxy02
ssh-copy-id -i ~/.ssh/id_rsa.pub rproxy03

2.2.3 测试公钥认证

ssh root@rproxy02
ssh root@rproxy03

2.3 配置集群配置同步

2.3.1 创建配置文件

In rproxy0[1-3],

yum install -y nginx-sync

2.3.2 创建配置文件

In rproxy01,

vim /etc/nginx-sync.conf

加入如下配置,

NODES="rproxy02 rproxy03"
CONFPATHS="/etc/nginx/nginx.conf /etc/nginx/conf.d"
EXCLUDE="default.conf"

参数解析,
– 参数“NODES”定义接收配置的远程主机(多个主机使用空格分隔)
– 参数“CONFPATHS”定义复制到远程主机的配置文件或目录(多个文件或目录使用空格分隔)
– 参数“EXCLUDE”定义不复制到远程主机的配置文件或目录(多个文件或目录使用空格分隔)

2.3.3 获取命令行帮助

nginx-sync.sh -h

可见如下显示,

nginx-sync.sh  [-h | -c node_address| -C] [-r] [-d] [-u] [-l logfile]
nginx-sync.sh  without arguments synchronizes configs from the master node
to the slave nodes
  -c compares local configs with configs on node_address
  -C compares local configs with configs of other nodes
  -r enables verbose rsync output
  -d enables verbose diff output
  -u run without sudo if started from non root user
  -l saves script output to the logfile

Prerequisites:
 * config file /etc/nginx-sync.conf
 * set up ssh key authentication between nodes

Config variables define a list of values, one per line:
NODES - list of the slave nodes
CONFPATHS - paths to directories and files to be synchronized
EXCLUDE - patterns to be excluded from synchronization
POSTSYNC - filename|sed_expression to be run after synchronization

To setup ssh key authentication it is required:
 * to generate key pair on the master node run
   'ssh-keygen -t rsa -b 2048' command from the root user
 * copy public key /root/.ssh/id_rsa.pub to the slave nodes as
   /root/.ssh/authorized_keys
 * add directive 'PermitRootLogin without-password', that allows
   only key authentication to log in with root credentials, to the
   /etc/ssh/sshd_config on the slave nodes and reload sshd
 * to ensure key authentication works run from the master node
   ssh  echo test
   command, where  is slave node's name or ip address

 Sample config file content:

NODES="node2.example.com
node3.example.com"
CONFPATHS="/etc/nginx
/etc/ssl/nginx"
EXCLUDE="default.conf"

2.3.4 创建测试配置文件

In rproxy01,

vim /etc/nginx/conf.d/www.cmdschool.org_80.conf

配置如下虚拟服务器,

server {
    listen       80;
    server_name  www.cmdschool.org;

    location / {
        root   /var/www/www.cmdschool.org;
        index  index.html index.htm  index.php;
    }

    error_page   500 502 503 504  /50x.html;
    location = /50x.html {
        root   /usr/share/nginx/html;
    }
}

2.3.5 对比主服务器与其他服务器的配置差异

In rproxy01,

nginx-sync.sh -C

可见如下提示,

 * Synchronization started at Tue Oct 22 16:06:03 UTC 2019

 * Checking prerequisites

 * Comparing local and remote configs

 * Comparing configs on master node and rproxy02:

Only in local:/etc/nginx/conf.d: www.cmdschool.org_80.conf
 * Comparing local and remote configs

 * Comparing configs on master node and rproxy03:

Only in local:/etc/nginx/conf.d: www.cmdschool.org_80.conf

 * Synchronization ended at Tue Oct 22 16:06:29 UTC 2019

以上提示意思为配置文件只存在于本地,你可以使用如下命令验证,

ssh rproxy02 ls /etc/nginx/conf.d/www.cmdschool.org_80.conf
ssh rproxy03 ls /etc/nginx/conf.d/www.cmdschool.org_80.conf

可见如提示,

ls: cannot access /etc/nginx/conf.d/www.cmdschool.org_80.conf: No such file or directory

另外,如果要单台对比,请使用如下命令,

nginx-sync.sh -c rproxy02
nginx-sync.sh -c rproxy03

2.3.6 同步并重载其他主机的配置

In rproxy01,

nginx-sync.sh

可见如提示,

 * Synchronization started at Tue Oct 22 16:33:20 UTC 2019

 * Checking prerequisites

 * Testing local nginx configuration file

nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
nginx version: nginx/1.17.3 (nginx-plus-r19)
 * Deleting remote backup directory

 * Backing up configuration on rproxy02

 * Updating configuration on rproxy02

 * Testing nginx config on rproxy02

nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
Redirecting to /bin/systemctl status nginx.service
Redirecting to /bin/systemctl reload nginx.service
nginx version: nginx/1.17.3 (nginx-plus-r19)
 * Deleting remote backup directory

 * Backing up configuration on rproxy03

 * Updating configuration on rproxy03

 * Testing nginx config on rproxy03

nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
Redirecting to /bin/systemctl status nginx.service
Redirecting to /bin/systemctl reload nginx.service

 * Synchronization ended at Tue Oct 22 16:35:04 UTC 2019

你可以使用如下命令验证,

ssh rproxy02 ls /etc/nginx/conf.d/www.cmdschool.org_80.conf
ssh rproxy03 ls /etc/nginx/conf.d/www.cmdschool.org_80.conf

可见如提示,

/etc/nginx/conf.d/www.cmdschool.org_80.conf

2.3.7 创建测试配置

In rproxy01,

mkdir -p /var/www/www.cmdschool.org
ssh rproxy02 mkdir -p /var/www/www.cmdschool.org
ssh rproxy03 mkdir -p /var/www/www.cmdschool.org
echo rproxy01.cmdschool.org > /var/www/www.cmdschool.org/index.html
ssh rproxy02 'echo rproxy02.cmdschool.org > /var/www/www.cmdschool.org/index.html'
ssh rproxy03 'echo rproxy03.cmdschool.org > /var/www/www.cmdschool.org/index.html'

请使用如下命令验证配置,

cat /var/www/www.cmdschool.org/index.html
ssh rproxy02 cat /var/www/www.cmdschool.org/index.html
ssh rproxy03 cat /var/www/www.cmdschool.org/index.html

可见如下输出,

rproxy01.cmdschool.org
rproxy02.cmdschool.org
rproxy03.cmdschool.org

2.3.8 测试配置

In client,

vim /etc/hosts

加入如下配置,

10.168.0.81 www.cmdschool.org

命令行测试访问,

curl http://www.cmdschool.org

可见如下显示,

rproxy01.cmdschool.org

然后,再次修改以下配置文件,

vim /etc/hosts

配置修改如下,

# 10.168.0.81 www.cmdschool.org
10.168.0.82 www.cmdschool.org

命令行测试访问,

curl http://www.cmdschool.org

可见如下显示,

rproxy02.cmdschool.org

然后,再次修改以下配置文件,

vim /etc/hosts

配置修改如下,

# 10.168.0.81 www.cmdschool.org
# 10.168.0.82 www.cmdschool.org
10.168.0.83 www.cmdschool.org

命令行测试访问,

curl http://www.cmdschool.org

可见如下显示,

rproxy03.cmdschool.org

2.4 配置集群状态共享

2.4.1 增加区域同步配置

In rproxy01,

vim /etc/nginx/nginx.conf

加入如下配置,

stream {
    resolver 127.0.0.1 valid=20s;

    server {
        listen 127.0.0.1:9000;

        zone_sync;
        zone_sync_server rproxy01.cmdschool.org:9000;
        zone_sync_server rproxy02.cmdschool.org:9000;
        zone_sync_server rproxy03.cmdschool.org:9000;
    }
}

注:由于前面使用host代替名称解析,所以“resolver”配置可以指向本机或注解掉

2.4.2 测试区域同步配置

In rproxy01,

nginx -t

2.4.3 同步并重载其他主机的配置

In rproxy01,

nginx-sync.sh

2.4.3 检查倾听的端口

In rproxy0[1-3],

netstat -antp | grep nginx

可见如下显示,

tcp        0      0 0.0.0.0:9000            0.0.0.0:*               LISTEN      4742/nginx: master
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      4742/nginx: master

注:注意到“rproxy01”的“9000”端口没有启用,所以也无需开放他的“9000”端口

2.4.4 配置的防火墙

In rproxy0[2-3],

firewall-cmd --permanent --add-port 9000/tcp
firewall-cmd --reload
firewall-cmd --list-all

参阅文档
========================

配置文件同步
————-
https://docs.nginx.com/nginx/admin-guide/high-availability/configuration-sharing/

配置同步模块
————–
http://nginx.org/en/docs/stream/ngx_stream_zone_sync_module.html

会话持久的概念
—————
https://docs.nginx.com/nginx/admin-guide/load-balancer/http-load-balancer/#sticky
https://blog.csdn.net/agangdi/article/details/41087921

资源限制
———————
https://docs.nginx.com/nginx/admin-guide/security-controls/controlling-access-proxied-http/#limit_req
https://www.jb51.net/article/143682.htm

键值存储数据
————
https://nginx.org/en/docs/http/ngx_http_keyval_module.html?&_ga=2.127248910.224040967.1571884787-1953014643.1569045545#keyval_zone
https://blog.csdn.net/kikajack/article/details/79339632

没有评论

发表评论

Nginx
如何设置Nginx HTTP-only Cookie?

1 基础知识 1.1 HTTP cookies 1.1.1 HTTP cookies的概念 &#82 …

Nginx
如何附加Nginx Cookie标签模块?

1 基础知识 “nginx_cookie_flag_module”为上游响应标头中的cookie设置 …