1 基础知识

1.1 Apache Ignite的角色

– Apache Ignite启动时将节点分配为服务端节点或客户端节点之一
– Apache Ignite服务端是集群的主体，负责存储数据、执行计算任务等
– Apache Iginite客户端作为常规节点加入拓扑，但不存储数据。
– Apache Iginte客户端负责将数据流传输到集群中并执行用户查询

1.2 集群发现机制

1.2.1 发现机制的介绍

– Apache Ignite节点间可以自动相互发现并组成集群
– Apache Ignite自动发现便于横向扩展而不必重启集群
– Apache Ignite支持两种发现机制
— TCP/IP发现（针对百以内节点的集群发现）
— ZooKeeper发现（支持千以内节点的集群发现）

1.2.2 TCP/IP发现

– Apache Ignite集群节点间通过DiscoverSpi相互发现
– Apache Ignite DiscoverSpi是集群默认的相互发现

1.2.3 组播IP探测器

– TcpDiscoveryMulticastIpFinder使用组播来发现其他节点
– TcpDiscoveryMulticastIpFinder是默认的IP探测器
– TcpDiscoveryMulticastIpFinder的配置范例如下，

<bean class="org.apache.ignite.configuration.IgniteConfiguration">

    <property name="discoverySpi">
        <bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
            <property name="ipFinder">
                <bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.multicast.TcpDiscoveryMulticastIpFinder">
                    <property name="multicastGroup" value="228.10.10.157"/>
                </bean>
            </property>
        </bean>
    </property>

</bean>

1.2.4 静态IP探测器

– 静态IP探测器实现了TcpDiscoveryVmIpFinder
– 静态IP探测器可以指定一组IP地址和端口
– 静态IP探测器将检测指定的IP地址和端口发现其他节点
– 静态IP探测器配置范例如下，

<bean class="org.apache.ignite.configuration.IgniteConfiguration">
    <property name="discoverySpi">
        <bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
            <property name="ipFinder">
                <bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">
                    <property name="addresses">
                        <list>
                            <!--
                              Explicitly specifying address of a local node to let it start and
                              operate normally even if there is no more nodes in the cluster.
                              You can also optionally specify an individual port or port range.
                              -->
                            <value>1.2.3.4</value>
                            <!--
                              IP Address and optional port range of a remote node.
                              You can also optionally specify an individual port.
                              -->
                            <value>1.2.3.5:47500..47509</value>
                        </list>
                    </property>
                </bean>
            </property>
        </bean>
    </property>
</bean>

1.2.5 静态和组播IP探测器

– 静态和组播IP探测器基于组播和静态IP发现（TcpDiscoveryMulticastIpFinder）
– 静态和组播IP探测器TcpDiscoveryMulticastIpFinder支持接收组播IP地址和预定义的静态IP地址
– 静态和组播IP探测器范例如下，

<bean class="org.apache.ignite.configuration.IgniteConfiguration">
    <property name="discoverySpi">
        <bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
            <property name="ipFinder">
                <bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.multicast.TcpDiscoveryMulticastIpFinder">
                    <property name="multicastGroup" value="228.10.10.157"/>
                    <!-- list of static IP addresses-->
                    <property name="addresses">
                        <list>
                            <value>1.2.3.4</value>
                            <!--
                              IP Address and optional port range.
                              You can also optionally specify an individual port.
                             -->
                            <value>1.2.3.5:47500..47509</value>
                        </list>
                    </property>
                </bean>
            </property>
        </bean>
    </property>
</bean>

1.2.6 集群隔离

– 集群隔离许同一组主机内启动两个相互隔离的集群
– 集群隔离可使用TcpDiscoverySpi与TcpCommunicationSpi结合不相交的本地端口范围实现
– 集群隔离配置范例如下，

<bean class="org.apache.ignite.configuration.IgniteConfiguration">
    <!--
    Explicitly configure TCP discovery SPI to provide list of
    initial nodes from the first cluster.
    -->
    <property name="discoverySpi">
        <bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
            <!-- Initial local port to listen to. -->
            <property name="localPort" value="48500"/>

            <!-- Changing local port range. This is an optional action. -->
            <property name="localPortRange" value="20"/>

            <!-- Setting up IP finder for this cluster -->
            <property name="ipFinder">
                <bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">
                    <property name="addresses">
                        <list>
                            <!--
                            Addresses and port range of nodes from
                            the first cluster.
                            127.0.0.1 can be replaced with actual IP addresses
                            or host names. Port range is optional.
                            -->
                            <value>127.0.0.1:48500..48520</value>
                        </list>
                    </property>
                </bean>
            </property>
        </bean>
    </property>

    <!--
    Explicitly configure TCP communication SPI changing local
    port number for the nodes from the first cluster.
    -->
    <property name="communicationSpi">
        <bean class="org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi">
            <property name="localPort" value="48100"/>
        </bean>
    </property>
</bean>

另外一个隔离集群范例如下，

<bean id="ignite.cfg" class="org.apache.ignite.configuration.IgniteConfiguration">
    <!--
    Explicitly configure TCP discovery SPI to provide list of initial
    nodes from the second cluster.
    -->
    <property name="discoverySpi">
        <bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
            <!-- Initial local port to listen to. -->
            <property name="localPort" value="49500"/>

            <!-- Changing local port range. This is an optional action. -->
            <property name="localPortRange" value="20"/>

            <!-- Setting up IP finder for this cluster -->
            <property name="ipFinder">
                <bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">
                    <property name="addresses">
                        <list>
                            <!--
                            Addresses and port range of the nodes from the second cluster.
                            127.0.0.1 can be replaced with actual IP addresses or host names. Port range is optional.
                            -->
                            <value>127.0.0.1:49500..49520</value>
                        </list>
                    </property>
                </bean>
            </property>
        </bean>
    </property>

    <!--
    Explicitly configure TCP communication SPI changing local port number
    for the nodes from the second cluster.
    -->
    <property name="communicationSpi">
        <bean class="org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi">
            <property name="localPort" value="49100"/>
        </bean>
    </property>
</bean>

1.2.7 JDBC探测器

– JDBC探测器支持将数据库作为集群初始时节点IP地址的共享存储
– JDBC探测器启动时将集群节点IP地址写入数据库
– JDBC探测器通过TcpDiscoveryJdbcIpFinder实现
– JDBC的配置范例如下，

<bean class="org.apache.ignite.configuration.IgniteConfiguration">

  <property name="discoverySpi">
    <bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
      <property name="ipFinder">
        <bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.jdbc.TcpDiscoveryJdbcIpFinder">
          <property name="dataSource" ref="ds"/>
        </bean>
      </property>
    </bean>
  </property>
</bean>

<!-- Configured data source instance. -->
<bean id="ds" class="some.Datasource">

</bean>

1.2.8 共享文件系统IP探测器

– 共享文件系统IP探测器支持共享文件系统作为集群初始时节点IP地址的共享存储
– 共享文件系统IP探测器将集群节点IP地址写入共享存储
– 共享文件系统IP探测器通过TcpDiscoverySharedFsIpFinder实现
– 共享文件系统IP探测器的配置范例如下，

<bean class="org.apache.ignite.configuration.IgniteConfiguration">
    <property name="discoverySpi">
        <bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
            <property name="ipFinder">
                <bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.sharedfs.TcpDiscoverySharedFsIpFinder">
                  <property name="path" value="/var/ignite/addresses"/>
                </bean>
            </property>
        </bean>
    </property>
</bean>

1.2.9 ZooKeeper IP探测器

– ZooKeeper IP探测器支持TcpDiscoveryZookeeperIpFinder
– ZooKeeper IP探测器需要启用ignite-zookeeper模块
– ZooKeeper IP探测器.NET/C#/C++目前尚不支持
– ZooKeeper IP探测器的配置范例如下，

<bean class="org.apache.ignite.configuration.IgniteConfiguration">

    <property name="discoverySpi">
        <bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
            <property name="ipFinder">
                <bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.zk.TcpDiscoveryZookeeperIpFinder">
                    <property name="zkConnectionString" value="127.0.0.1:2181"/>
                </bean>
            </property>
        </bean>
    </property>
</bean>

2 最佳实践

2.1 环境信息

Cluster Node Host Name = ignite-node[1-3].cmdschool.org
Cluster Node IP Address = 10.168.0.11[1-3]

2.2 节点部署

请参阅以下文档部署节点的软件包，推荐使用Yum源部署，

如何yum部署Apache Ignite？

可选二进制压缩包部署，

如何二进制部署Apache Ignite?

2.3 安装前准备

2.3.1 检查NTP服务

In ignite-node[1-3]，

systemctl status chronyd.service

2.3.2 配置名称解析

In ignite-node[1-3]，

echo '10.168.0.111 ignite-node1 ignite-node1.cmdschool.org' >> /etc/hosts
echo '10.168.0.112 ignite-node2 ignite-node2.cmdschool.org' >> /etc/hosts
echo '10.168.0.113 ignite-node3 ignite-node3.cmdschool.org' >> /etc/hosts

2.3.3 配置公钥认证（可选）

In ignite-node1，

ssh-keygen

以上一直回车即可生成公钥和私钥，然后通过以下命令复制公钥到被登录的节点，

ssh-copy-id -i ~/.ssh/id_rsa ignite-node1
ssh-copy-id -i ~/.ssh/id_rsa ignite-node2
ssh-copy-id -i ~/.ssh/id_rsa ignite-node3

然后，通过如下命令测试登录，

ssh ignite-node1
ssh ignite-node2
ssh ignite-node3

2.4 配置集群

2.4.1 停止集群其他节点

In ignite-node1，

ssh ignite-node2 systemctl stop apache-ignite.service
ssh ignite-node3 systemctl stop apache-ignite.service

2.4.2 修改集群配置

In ignite-node1，

vim /etc/apache-ignite/default-config.xml

配置修改如下，

<?xml version="1.0" encoding="UTF-8"?>

<!--
  Licensed to the Apache Software Foundation (ASF) under one or more
  contributor license agreements.  See the NOTICE file distributed with
  this work for additional information regarding copyright ownership.
  The ASF licenses this file to You under the Apache License, Version 2.0
  (the "License"); you may not use this file except in compliance with
  the License.  You may obtain a copy of the License at

       http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License.
-->

<!--
    Ignite configuration with all defaults and enabled p2p deployment and enabled events.
-->
<beans xmlns="http://www.springframework.org/schema/beans"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xmlns:util="http://www.springframework.org/schema/util"
       xsi:schemaLocation="
        http://www.springframework.org/schema/beans
        http://www.springframework.org/schema/beans/spring-beans.xsd
        http://www.springframework.org/schema/util
        http://www.springframework.org/schema/util/spring-util.xsd">
    <bean abstract="true" id="ignite.cfg" class="org.apache.ignite.configuration.IgniteConfiguration">
        <!-- Set to true to enable distributed class loading for examples, default is false. -->
        <property name="peerClassLoadingEnabled" value="true"/>

        <!-- Enable task execution events for examples. -->
        <property name="includeEventTypes">
            <list>
                <!--Task execution events-->
                <util:constant static-field="org.apache.ignite.events.EventType.EVT_TASK_STARTED"/>
                <util:constant static-field="org.apache.ignite.events.EventType.EVT_TASK_FINISHED"/>
                <util:constant static-field="org.apache.ignite.events.EventType.EVT_TASK_FAILED"/>
                <util:constant static-field="org.apache.ignite.events.EventType.EVT_TASK_TIMEDOUT"/>
                <util:constant static-field="org.apache.ignite.events.EventType.EVT_TASK_SESSION_ATTR_SET"/>
                <util:constant static-field="org.apache.ignite.events.EventType.EVT_TASK_REDUCED"/>

                <!--Cache events-->
                <util:constant static-field="org.apache.ignite.events.EventType.EVT_CACHE_OBJECT_PUT"/>
                <util:constant static-field="org.apache.ignite.events.EventType.EVT_CACHE_OBJECT_READ"/>
                <util:constant static-field="org.apache.ignite.events.EventType.EVT_CACHE_OBJECT_REMOVED"/>
            </list>
        </property>

        <!-- Explicitly configure TCP discovery SPI to provide list of initial nodes. -->
        <property name="discoverySpi">
            <bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
                <property name="ipFinder">
                    <!--
                        Ignite provides several options for automatic discovery that can be used
                        instead os static IP based discovery. For information on all options refer
                        to our documentation: http://apacheignite.readme.io/docs/cluster-config
                    -->
                    <!-- Uncomment static IP finder to enable static-based discovery of initial nodes. -->
                    <!--<bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">-->
                    <bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.multicast.TcpDiscoveryMulticastIpFinder">
                        <property name="addresses">
                            <list>
                                <!-- In distributed environment, replace with actual host IP address. -->
                                <value>ignite-node1.cmdschool.org:47500..47509</value>
                                <value>ignite-node2.cmdschool.org:47500..47509</value>
                                <value>ignite-node3.cmdschool.org:47500..47509</value>
                            </list>
                        </property>
                    </bean>
                </property>
            </bean>
        </property>
    </bean>
</beans>

然后，使用如下命令分发配置到集群其他节点，

scp /etc/apache-ignite/default-config.xml ignite-node2:/etc/apache-ignite/
scp /etc/apache-ignite/default-config.xml ignite-node3:/etc/apache-ignite/

2.4.3 重启服务使配置生效

In ignite-node1，

ssh ignite-node1 systemctl restart apache-ignite.service
ssh ignite-node2 systemctl restart apache-ignite.service
ssh ignite-node3 systemctl restart apache-ignite.service

2.4.4 激活集群

In ignite-node1，

control.sh  --set-state ACTIVE

可见如下显示，

Warning: the command will change state of cluster with name "39feaf63a71-372b8ed1-502c-496e-8670-43d2319be7f9" to ACTIVE.
Press 'y' to continue . . . y
Control utility [ver. 2.9.1#20201203-sha1:adcce517]
2020 Copyright(C) Apache Software Foundation
User: root
Time: 2021-06-23T01:37:51.084
Command [SET-STATE] started
Arguments: --set-state ACTIVE
--------------------------------------------------------------------------------
Cluster state changed to ACTIVE
Command [SET-STATE] finished with code: 0
Control utility has completed execution at: 2021-06-23T01:38:00.987
Execution time: 9903 ms

然后，可以使用如下命令查看集群状态，

control.sh --state

可见如下显示，

Control utility [ver. 2.9.1#20201203-sha1:adcce517]
2020 Copyright(C) Apache Software Foundation
User: root
Time: 2021-06-23T01:40:05.148
Command [STATE] started
Arguments: --state
--------------------------------------------------------------------------------
Cluster  ID: ea494dce-2c1b-473b-a879-271aa82792a2
Cluster tag: unruffled_torvalds
--------------------------------------------------------------------------------
Cluster is active
Command [STATE] finished with code: 0
Control utility has completed execution at: 2021-06-23T01:40:05.573
Execution time: 425 ms

2.4.5 查看集群节点

control.sh --baseline

可见如下显示，

Control utility [ver. 2.9.1#20201203-sha1:adcce517]
2020 Copyright(C) Apache Software Foundation
User: root
Time: 2021-06-23T02:35:59.017
Command [BASELINE] started
Arguments: --baseline
--------------------------------------------------------------------------------
Cluster state: active
Current topology version: 11
Baseline auto adjustment enabled: softTimeout=0
Baseline auto-adjust are not scheduled

Current topology version: 11 (Coordinator: ConsistentId=0:0:0:0:0:0:0:1%lo,10.168.0.111,127.0.0.1:47500, Order=7)

Baseline nodes:
    ConsistentId=0:0:0:0:0:0:0:1%lo,10.168.0.111,127.0.0.1:47500, State=ONLINE, Order=7
    ConsistentId=0:0:0:0:0:0:0:1%lo,10.168.0.112,127.0.0.1:47500, State=ONLINE, Order=9
    ConsistentId=0:0:0:0:0:0:0:1%lo,10.168.0.113,127.0.0.1:47500, State=ONLINE, Order=11
--------------------------------------------------------------------------------
Number of baseline nodes: 3

Other nodes not found.
Command [BASELINE] finished with code: 0
Control utility has completed execution at: 2021-06-23T02:35:59.462
Execution time: 445 ms

2.5 配置集群数据持久化

如何配置Apache Ignite集群数据持久？

参阅文档
==================
https://www.ignite-service.cn/doc/java/Clustering.html