如何部署Docker EMQX(MQTT)社区版集群？

Docker

1 基础知识

管理理论知识，我建议你可以参考如下章节，

如何部署EMQX(MQTT)社区版集群？

2 最佳实践

2.1 准备环境

2.1.1 部署Docker集群环境

如何部署Oracle Linux 10.x Docker生产集群环境？

2.1.2 部署Nginx反向代理

如何部署Docker Nginx 1.20.2?

2.1.3 部署前的准备

In hd01，

cd /data/docker/images/
docker load -i emqx-emqx:4.3.8.tar
docker tag emqx/emqx:4.3.8 hd01.cmdschool.org:5000/emqx/emqx:4.3.8
docker push hd01.cmdschool.org:5000/emqx/emqx:4.3.8

另外，安装包的离线下载请在能上网的docker环境的机器上使用如下命令，

docker pull emqx/emqx:4.3.8
docker save emqx/emqx:4.3.8 -o emqx-emqx:4.3.8.tar

另外，其他版本请从以下链接下载，
https://hub.docker.com/r/emqx/emqx

2.2 部署EMQX

In hd01

2.2.1 创建服务配置文件

In hd01，

vim /data/docker/yml/emqx-stack.yml

加入如下配置，

version: '3.6'
services:
    emqx-01:
        image: hd01.cmdschool.org:5000/emqx/emqx:4.3.8
        volumes:
            - /etc/localtime:/etc/localtime
        deploy:
            replicas: 1
            placement:
                constraints: [node.labels.emqx == 01]
        hostname: emqx01.mes
        environment:
            EMQX_LOG__TO: 'both'
            #DEBUG: 'true'
            EMQX_NAME: 'emqx'
            EMQX_HOST: 'emqx01.mes'
            EMQX_CLUSTER__NAME: 'emqxcl'
            EMQX_CLUSTER__DISCOVERY: 'static'
            EMQX_CLUSTER__STATIC__SEEDS: 'emqx@emqx01.mes,emqx@emqx02.mes,emqx@emqx03.mes'
        networks:
            - mes-cloud-net
            
    emqx-02:
        image: hd01.cmdschool.org:5000/emqx/emqx:4.3.8
        volumes:
            - /etc/localtime:/etc/localtime
        deploy:
            replicas: 1
            placement:
                constraints: [node.labels.emqx == 02]
        hostname: emqx02.mes
        environment:
            EMQX_LOG__TO: 'both'
            #DEBUG: 'true'
            EMQX_NAME: 'emqx'
            EMQX_HOST: 'emqx02.mes'
            EMQX_CLUSTER__NAME: 'emqxcl'
            EMQX_CLUSTER__DISCOVERY: 'static'
            EMQX_CLUSTER__STATIC__SEEDS: 'emqx@emqx01.mes,emqx@emqx02.mes,emqx@emqx03.mes'
        networks:
            - mes-cloud-net
            
    emqx-03:
        image: hd01.cmdschool.org:5000/emqx/emqx:4.3.8
        volumes:
            - /etc/localtime:/etc/localtime
        deploy:
            replicas: 1
            placement:
                constraints: [node.labels.emqx == 03]
        hostname: emqx03.mes
        environment:
            EMQX_LOG__TO: 'both'
            #DEBUG: 'true'
            EMQX_NAME: 'emqx'
            EMQX_HOST: 'emqx03.mes'
            EMQX_CLUSTER__NAME: 'emqxcl'
            EMQX_CLUSTER__DISCOVERY: 'static'
            EMQX_CLUSTER__STATIC__SEEDS: 'emqx@emqx01.mes,emqx@emqx02.mes,emqx@emqx03.mes'
        networks:
            - mes-cloud-net
            
networks:
  mes-cloud-net:
    external: true
    name: mes-cloud-net

根据配置文件的要求，你需要手动创建如下网络，

docker network create --opt encrypted --driver overlay --attachable --subnet=172.16.100.0/24  mes-cloud-net

根据配置的，你需要设置运行标记，

docker node update --label-add emqx=01 hd01.cmdschool.org
docker node update --label-add emqx=02 hd02.cmdschool.org
docker node update --label-add emqx=03 hd03.cmdschool.org

2.2.2 部署服务

In hd01，

docker stack deploy -c /data/docker/yml/emqx-stack.yml mes-platform

2.3 本地化EMQX

2.3.1 部署服务

vim /data/docker/yml/emqx-stack.yml

你需要加入如下配置，

version: '3.6'
services:
    emqx-01:
        image: hd01.cmdschool.org:5000/emqx/emqx:4.3.8
        volumes:
            - /data/docker/container-data/emqx/conf:/opt/emqx/etc
            - /data/docker/container-data/emqx/log:/opt/emqx/log
            - /data/docker/container-data/emqx/data:/opt/emqx/data
            - /etc/localtime:/etc/localtime
        deploy:
            replicas: 1
            placement:
                constraints: [node.labels.emqx == 01]
        hostname: emqx01.mes
        environment:
            EMQX_LOG__TO: 'both'
            #DEBUG: 'true'
            EMQX_NAME: 'emqx'
            EMQX_HOST: 'emqx01.mes'
            EMQX_CLUSTER__NAME: 'emqxcl'
            EMQX_CLUSTER__DISCOVERY: 'static'
            EMQX_CLUSTER__STATIC__SEEDS: 'emqx@emqx01.mes,emqx@emqx02.mes,emqx@emqx03.mes'
        networks:
            - mes-cloud-net
            
    emqx-02:
        image: hd01.cmdschool.org:5000/emqx/emqx:4.3.8
        volumes:
            - /data/docker/container-data/emqx/conf:/opt/emqx/etc
            - /data/docker/container-data/emqx/log:/opt/emqx/log
            - /data/docker/container-data/emqx/data:/opt/emqx/data
            - /etc/localtime:/etc/localtime
        deploy:
            replicas: 1
            placement:
                constraints: [node.labels.emqx == 02]
        hostname: emqx02.mes
        environment:
            EMQX_LOG__TO: 'both'
            #DEBUG: 'true'
            EMQX_NAME: 'emqx'
            EMQX_HOST: 'emqx02.mes'
            EMQX_CLUSTER__NAME: 'emqxcl'
            EMQX_CLUSTER__DISCOVERY: 'static'
            EMQX_CLUSTER__STATIC__SEEDS: 'emqx@emqx01.mes,emqx@emqx02.mes,emqx@emqx03.mes'
        networks:
            - mes-cloud-net
            
    emqx-03:
        image: hd01.cmdschool.org:5000/emqx/emqx:4.3.8
        volumes:
            - /data/docker/container-data/emqx/conf:/opt/emqx/etc
            - /data/docker/container-data/emqx/log:/opt/emqx/log
            - /data/docker/container-data/emqx/data:/opt/emqx/data
            - /etc/localtime:/etc/localtime
        deploy:
            replicas: 1
            placement:
                constraints: [node.labels.emqx == 03]
        hostname: emqx03.mes
        environment:
            EMQX_LOG__TO: 'both'
            #DEBUG: 'true'
            EMQX_NAME: 'emqx'
            EMQX_HOST: 'emqx03.mes'
            EMQX_CLUSTER__NAME: 'emqxcl'
            EMQX_CLUSTER__DISCOVERY: 'static'
            EMQX_CLUSTER__STATIC__SEEDS: 'emqx@emqx01.mes,emqx@emqx02.mes,emqx@emqx03.mes'
        networks:
            - mes-cloud-net
            
networks:
  mes-cloud-net:
    external: true
    name: mes-cloud-net

根据配置的要求，你需要将配置文件复制到本地，

docker cp $(docker container ls | grep emqx | awk '{print $1}'):/opt/emqx/etc /data/docker/container-data/emqx/conf
ssh hd02 "docker cp \$(docker container ls | grep emqx | awk '{print \$1}'):/opt/emqx/etc /data/docker/container-data/emqx/conf"
ssh hd03 "docker cp \$(docker container ls | grep emqx | awk '{print \$1}'):/opt/emqx/etc /data/docker/container-data/emqx/conf"

然后，根据配置需求创建数据存放目录，

mkdir -p /data/docker/container-data/emqx/log
mkdir -p /data/docker/container-data/emqx/data
chown 1000:docker /data/docker/container-data/emqx/{data,log}
ssh hd02 mkdir -p /data/docker/container-data/emqx/log
ssh hd02 mkdir -p /data/docker/container-data/emqx/data
ssh hd02 chown 1000:docker /data/docker/container-data/emqx/{data,log}
ssh hd03 mkdir -p /data/docker/container-data/emqx/log
ssh hd03 mkdir -p /data/docker/container-data/emqx/data
ssh hd03 chown 1000:docker /data/docker/container-data/emqx/{data,log}

2.3.2 重新部署服务使配置生效

In hd01，

docker stack deploy -c /data/docker/yml/emqx-stack.yml mes-platform

2.4 启动EMQX的模块

In hd01

2.4.1 确定公用的ssl配置已经存在

vim /data/docker/container-data/emqx/data/loaded_plugins

加入如下参数，

{emqx_management, true}.
{emqx_dashboard, true}.
{emqx_modules, false}.
{emqx_recon, true}.
{emqx_retainer, true}.
{emqx_telemetry, true}.
{emqx_rule_engine, true}.
{emqx_bridge_mqtt, false}.

2.4.2 复制配置到其他节点

scp /data/docker/container-data/emqx/data/loaded_plugins hd02:/data/docker/container-data/emqx/data/loaded_plugins
scp /data/docker/container-data/emqx/data/loaded_plugins hd03:/data/docker/container-data/emqx/data/loaded_plugins

2.4.3 重新部署服务使配置生效

docker stack rm mes-platform
docker stack deploy -c /data/docker/yml/emqx-stack.yml mes-platform

2.5 部署EMQX代理

2.5.1 确定公用的ssl配置已经存在

In hd01

cat /etc/nginx/public/cmdschool.org_ssl.conf

加入如下配置，

ssl_certificate wildcard.cmdschool.org.crt;
ssl_certificate_key wildcard.cmdschool.org.key;
ssl_protocols TLSv1.2;
ssl_ciphers ECDHE-RSA-AES256-GCM-SHA512:DHE-RSA-AES256-GCM-SHA512:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384;
ssl_prefer_server_ciphers on;

2.5.2 部署EMQX MQTT服务代理配置

In hd01

vim /etc/nginx/stream.d/mesweb.cmdschool.org_emqx_1883_8883.conf

加入如下配置，

#emqx cluster
upstream emqx_cluster {
        zone tcp_servers 64k;
        hash $remote_addr;
        server emqx-01:1883 max_fails=2 fail_timeout=30s;
        server emqx-02:1883 max_fails=2 fail_timeout=30s;
        server emqx-03:1883 max_fails=2 fail_timeout=30s;
}

server {
        listen 1883;
        proxy_pass emqx_cluster;
        proxy_buffer_size 4k;
}

server {
        listen 8883 ssl;
        proxy_pass emqx_cluster;
        proxy_buffer_size 4k;
        include /etc/nginx/public/cmdschool.org_ssl.conf;
}

根据上面的配置，你需要创建如下别名解析，

mesweb.cmdschool.org --> hd01.cmdschool.org

2.5.3 部署EMQX Dashboard服务代理配置

In hd01

vim vim /etc/nginx/conf.d/mesweb.cmdschool.org_emqx_dashboard_18083.conf

加入如下配置，

upstream emqx_dashboard {
  least_conn;
  server emqx-01:18083 max_fails=3 fail_timeout=30s;
  server emqx-02:18083 max_fails=3 fail_timeout=30s;
  server emqx-03:18083 max_fails=3 fail_timeout=30s;
}

server {
        listen 18083 ssl;
        server_name mesweb.cmdschool.org;
        include /etc/nginx/public/cmdschool.org_ssl.conf;

        location / {
                proxy_pass http://emqx_dashboard;
        }
}

根据上面的配置，你需要创建如下别名解析，

mesweb.cmdschool.org --> hd01.cmdschool.org

2.5.4 同步配置到其他节点

In hd01

nginxtool sync

如果你已经安装集群管理工具，则使用以上命令，否则使用如下命令手动同步配置，

rsync -avP --delete /data/docker/container-data/nginx/conf/ hd02:/data/docker/container-data/nginx/conf/
rsync -avP --delete /data/docker/container-data/nginx/conf/ hd03:/data/docker/container-data/nginx/conf/

需要注意的是，之前我们已经将“/data/docker/container-data/nginx/conf”软连接到“/etc/nginx”

2.5.5 应用节点配置

In hd0[1-3] 

nginxtool reload

如果你已经安装集群管理工具，则使用以上命令，否则使用如下命令手动重载配置，

docker exec -it $(docker container ls | grep nginx | awk '{print $1}') bash
nginx -t
/etc/init.d/nginx reload

2.5.6 配置端口映射

In hd01

vim /data/docker/yml/nginx-stack.yml

增加如下配置，

version: '3.8'
services:
  nginx:
    image: 'hd01.cmdschool.org:5000/nginx:1.20.2'
    ports:
      - target: 443
        published: 443
      - target: 80
        published: 80
      - target: 1883
        published: 1883
      - target: 8883
        published: 8883
      - target: 18083
        published: 18083
#...
    networks:
       - elknet
       - mes-cloud-net
networks:
    elknet:
        external: true
        name: elknet
    mes-cloud-net:
        external: true
        name: mes-cloud-net

然后你需要执行如下命令重启服务使配置生效，

nginxtool restart

如果你已经安装集群管理工具，则使用以上命令，否则使用如下命令手动重启，

docker stack rm nginx
docker stack deploy -c /data/docker/yml/nginx-stack.yml nginx

2.6 测试EMQX服务

2.6.1 测试EMQX Dashboard

https://mesweb.cmdschool.org:18083/

请使用如下初始账号和密码登陆，
user: admin
password: public

2.6.2 集群MQTT通讯测试测试

mqttx-cli sub -t 'testtopic/#' -q 1 -h hd01.cmdschool.org -p 1883 'public' -v
mqttx-cli pub -t 'testtopic/1' -q 1 -h hd03.cmdschool.org -p 1883 -m 'from MQTTX CLI'
没有评论

发表回复

MQTT
如何部署EMQX(MQTT)社区版集群？

1 基础知识 1.1 Erang/OTP分布式编程 1.1.1 Erang/OTP分布式编程简介 & …

MQTT
如何安装部署EMQX(MQTT)社区版？

1 MQTT基础知识 1.1 MQTT的概念 – MQTT即英文“Message Que …

MQTT
如何安装部署Ubuntu mqttx-cli？

1 前言 一个问题，一篇文章，一出故事。 笔者今天部署EMQX服务端，因为需要mqtt的客户端来测试 …