Centos7：rabbitmq3.8+haproxy+keepalived(搭建高可用RabbitMQ镜像模式集群)

https://blog.csdn.net/xjjj064/article/details/110539442
一、环境描述

注：三台服务器selinux设置为了disabled，firewalld已经stop，且已分别设置了主机名对应如下

[root@server3 lib]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.6.101 server1
192.168.6.102 server2
192.168.6.103 server3

二、部署jdk环境+erlang环境
此前已经部署过一台服务器
参考：Centos7下安装部署rabbitmq3.8.9+erlang23.0
server1已经部署过一套rabbitmq3.8.9+erlang23.0
所以将server1上的jdk、erlang、rabbitmq3.8.9 传输给server2、server3，并设置好环境变量。
传输erlang包

[root@server1 lib]# scp erlang.tar.gz [email protected]:/usr/lib/
[email protected]'s password: 
erlang.tar.gz                                                                                                                                                                      100%   55MB  11.1MB/s   00:05    
[root@server1 lib]# scp erlang.tar.gz [email protected]:/usr/lib/
[email protected]'s password: 
erlang.tar.gz                                                                                                                                                                      100%   55MB   9.2MB/s   00:06

传输jdk包：

[root@server1 src]# scp jdk-8u121-linux-x64.gz [email protected]:/usr/local/src/
[email protected]'s password: 
jdk-8u121-linux-x64.gz                                                                                                                                                             100%  175MB   7.9MB/s   00:22    
[root@server1 src]# scp jdk-8u121-linux-x64.gz [email protected]:/usr/local/src/
[email protected]'s password: 
jdk-8u121-linux-x64.gz                                                                                                                                                              84%  148MB  11.8MB/s   00:02 ETAljdk-8u121-linux-x64.gz                                                                                                                                                             100%  175MB   9.7MB/s   00:18 

server2、server3设置环境变量

unset i
unset -f pathmunge
source /opt/rh/devtoolset-9/enable
export JAVA_HOME=/usr/local/jdk
export CLASSPATH=.:$JAVA_HOME/jre/lib/rt.jar:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
export PATH=$JAVA_HOME/bin:$PATH
export PATH=$PATH:/usr/lib/erlang/bin:/usr/local/rabbitmq/sbin

测试erlang环境和jdk环境
erlang

[root@server2 src]# erl
Erlang/OTP 23 [erts-11.0] [source] [64-bit] [smp:4:4] [ds:4:4:10] [async-threads:1] [hipe]

Eshell V11.0  (abort with ^G)
1> 

jdk

[root@server2 src]# java -version
java version "1.8.0_121"
Java(TM) SE Runtime Environment (build 1.8.0_121-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.121-b13, mixed mode)

统一 erlang.cookie 文件中 cookie 值

[root@server1 ~]# scp .erlang.cookie [email protected]:/root
[email protected]'s password: 
.erlang.cookie                                                                                                                                                                     100%   20    11.8KB/s   00:00    
[root@server1 ~]# scp .erlang.cookie [email protected]:/root
[email protected]'s password: 
.erlang.cookie                                                                                                                                                                     100%   20    20.5KB/s   00:00 

三、安装并启动rabbitmq

传送rabbitmq包到server2、server3

[root@server1 ~]# cd /usr/local/src/
[root@server1 src]# ls
jdk-8u121-linux-x64.gz  otp_src_23.0  otp_src_23.0.tar.gz  rabbitmq-server-generic-unix-3.8.9.tar  redis-6.0.5  redis-6.0.5.tar.gz
[root@server1 src]# scp rabbitmq-server-generic-unix-3.8.9.tar [email protected]:/usr/local/src/
[email protected]'s password: 
rabbitmq-server-generic-unix-3.8.9.tar                                                                                                                                             100%   15MB   8.0MB/s   00:01    
[root@server1 src]# scp rabbitmq-server-generic-unix-3.8.9.tar [email protected]:/usr/local/src/
[email protected]'s password: 
rabbitmq-server-generic-unix-3.8.9.tar                                                                                                                                             100%   15MB  10.7MB/s   00:01 

解压到usr/local/rabbitmq 目录

[root@server2 src]# ls -l /usr/local/rabbitmq/
total 176
drwxr-xr-x. 2 root root   124 Sep 25 02:39 escript
drwxr-xr-x. 3 root root    22 Sep 25 02:39 etc
-rw-r--r--. 1 root root    75 Sep 25 02:39 INSTALL
-rw-r--r--. 1 root root 19534 Sep 25 02:39 LICENSE
-rw-r--r--. 1 root root 11354 Sep 25 02:39 LICENSE-APACHE2
-rw-r--r--. 1 root root 11360 Sep 25 02:39 LICENSE-APACHE2-excanvas
-rw-r--r--. 1 root root 11358 Sep 25 02:39 LICENSE-APACHE2-ExplorerCanvas
-rw-r--r--. 1 root root 10851 Sep 25 02:39 LICENSE-APL2-Stomp-Websocket
-rw-r--r--. 1 root root  1207 Sep 25 02:39 LICENSE-BSD-base64js
-rw-r--r--. 1 root root  1469 Sep 25 02:39 LICENSE-BSD-recon
-rw-r--r--. 1 root root  1274 Sep 25 02:39 LICENSE-erlcloud
-rw-r--r--. 1 root root  1471 Sep 25 02:39 LICENSE-httpc_aws
-rw-r--r--. 1 root root   757 Sep 25 02:39 LICENSE-ISC-cowboy
-rw-r--r--. 1 root root  1084 Sep 25 02:39 LICENSE-MIT-EJS
-rw-r--r--. 1 root root  1087 Sep 25 02:39 LICENSE-MIT-EJS10
-rw-r--r--. 1 root root  1057 Sep 25 02:39 LICENSE-MIT-Erlware-Commons
-rw-r--r--. 1 root root  1069 Sep 25 02:39 LICENSE-MIT-Flot
-rw-r--r--. 1 root root  1075 Sep 25 02:39 LICENSE-MIT-jQuery
-rw-r--r--. 1 root root  1076 Sep 25 02:39 LICENSE-MIT-jQuery164
-rw-r--r--. 1 root root  1087 Sep 25 02:39 LICENSE-MIT-Mochi
-rw-r--r--. 1 root root  1073 Sep 25 02:39 LICENSE-MIT-Sammy
-rw-r--r--. 1 root root  1076 Sep 25 02:39 LICENSE-MIT-Sammy060
-rw-r--r--. 1 root root 16725 Sep 25 02:39 LICENSE-MPL
-rw-r--r--. 1 root root 16726 Sep 25 02:39 LICENSE-MPL-RabbitMQ
-rw-r--r--. 1 root root  1471 Sep 25 02:39 LICENSE-rabbitmq_aws
drwxr-xr-x. 2 root root  4096 Sep 25 02:39 plugins
drwxr-xr-x. 2 root root   192 Sep 25 02:39 sbin
drwxr-xr-x. 3 root root    17 Sep 25 02:39 share

server2、server3 启动rabbitmq服务
此处只截图server2

[root@server2 src]# rabbitmq-server -detached       
[root@server2 src]# netstat -ntlp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      1056/sshd           
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      1233/master         
tcp        0      0 0.0.0.0:25672           0.0.0.0:*               LISTEN      1909/beam.smp       
tcp        0      0 0.0.0.0:4369            0.0.0.0:*               LISTEN      2005/epmd           
tcp6       0      0 :::22                   :::*                    LISTEN      1056/sshd           
tcp6       0      0 ::1:25                  :::*                    LISTEN      1233/master         
tcp6       0      0 :::5672                 :::*                    LISTEN      1909/beam.smp       
tcp6       0      0 :::4369                 :::*                    LISTEN      2005/epmd  

server1上查看rabbitmq服务状态

[root@server1 src]# rabbitmqctl status
Status of node rabbit@server1 ...
Runtime

OS PID: 1647
OS: Linux
Uptime (seconds): 5992
Is under maintenance?: false
RabbitMQ version: 3.8.9
Node name: rabbit@server1
Erlang configuration: Erlang/OTP 23 [erts-11.0] [source] [64-bit] [smp:4:4] [ds:4:4:10] [async-threads:64] [hipe]
Erlang processes: 447 used, 1048576 limit
Scheduler run queue: 1
Cluster heartbeat timeout (net_ticktime): 60

Plugins

Enabled plugin file: /usr/local/rabbitmq/etc/rabbitmq/enabled_plugins
Enabled plugins:

 * rabbitmq_management
 * amqp_client
 * rabbitmq_web_dispatch
 * cowboy
 * cowlib
 * rabbitmq_management_agent

Data directory

Node data directory: /usr/local/rabbitmq/var/lib/rabbitmq/mnesia/rabbit@server1
Raft data directory: /usr/local/rabbitmq/var/lib/rabbitmq/mnesia/rabbit@server1/quorum/rabbit@server1

Config files


Log file(s)

 * /usr/local/rabbitmq/var/log/rabbitmq/[email protected]
 * /usr/local/rabbitmq/var/log/rabbitmq/rabbit@server1_upgrade.log

Alarms

(none)

Memory

Total memory used: 0.085 gb
Calculation strategy: rss
Memory high watermark setting: 0.4 of available memory, computed to: 3.2805 gb

code: 0.0278 gb (32.67 %)
other_proc: 0.0238 gb (28.03 %)
other_system: 0.0143 gb (16.86 %)
allocated_unused: 0.0076 gb (8.97 %)
reserved_unallocated: 0.0037 gb (4.33 %)
other_ets: 0.0032 gb (3.78 %)
plugins: 0.0023 gb (2.7 %)
atom: 0.0014 gb (1.67 %)
mgmt_db: 0.0003 gb (0.32 %)
metrics: 0.0002 gb (0.25 %)
binary: 0.0002 gb (0.2 %)
mnesia: 0.0001 gb (0.09 %)
quorum_ets: 0.0 gb (0.06 %)
connection_other: 0.0 gb (0.04 %)
msg_index: 0.0 gb (0.04 %)
connection_channels: 0.0 gb (0.0 %)
connection_readers: 0.0 gb (0.0 %)
connection_writers: 0.0 gb (0.0 %)
queue_procs: 0.0 gb (0.0 %)
queue_slave_procs: 0.0 gb (0.0 %)
quorum_queue_procs: 0.0 gb (0.0 %)

File Descriptors

Total: 2, limit: 927
Sockets: 0, limit: 832

Free Disk Space

Low free disk space watermark: 0.05 gb
Free disk space: 46.5476 gb

Totals

Connection count: 0
Queue count: 0
Virtual host count: 1

Listeners

Interface: [::], port: 15672, protocol: http, purpose: HTTP API
Interface: [::], port: 25672, protocol: clustering, purpose: inter-node and CLI tool communication
Interface: [::], port: 5672, protocol: amqp, purpose: AMQP 0-9-1 and AMQP 1.0
[root@server1 src]# 

四、Rabbitmq 集群添加节点

①、将server2、server3加入server1组集群
因为server1建过管理账号，所以将server2、server3加入到server1，加入后通过登录server2、server3可以看到在后两台服务器没有新建管理账号的情况下，admin账号已经同步到server2和server3服务器。

[root@server2 ~]# rabbitmqctl stop_app
[root@server2 ~]# rabbitmqctl join_cluster --ram rabbit@server1
[root@server2 ~]# rabbitmqctl start_app
[root@server2 ~]# rabbitmq-plugins enable rabbitmq_management


[root@server3 ~]# rabbitmqctl stop_app
[root@server3 ~]# rabbitmqctl join_cluster --ram rabbit@server1
[root@server3 ~]# rabbitmqctl start_app
[root@server3 ~]# rabbitmq-plugins enable rabbitmq_management

②、打开网页管理页面查看 nodes

③、搭建 rabbitmq 的镜像高可用模式集群
镜像队列机制就是将队列在 N 个节点之间设置主从关系，消息会在 N 个节点之间进行自动同步，且如果其中一个节点不可用，并不会导致消息丢失或服务不可用的情况，提升 MQ 集群的整体高可用性。
这里在rabbitmq1节点操作（在任意节点操作都是可以的）
这一节要参考的文档是：http://www.rabbitmq.com/ha.html
首先镜像模式要依赖 policy 模块，这个模块是做什么用的呢？
policy 中文来说是政策，策略的意思就是要设置那些Exchanges或者queue的数据需要如何复制，同步。

[root@server1 ~]# rabbitmqctl set_policy ha-all "^" '{"ha-mode":"all"}'

ha-all：为策略名称。
：为匹配符，只有一个代表匹配所有，^qfedu为匹配名称为qfedu的exchanges或者queue。
ha-mode：为匹配类型，他分为3种模式：
all-所有（所有的 queue），
exctly-部分（需配置ha-params参数，此参数为int类型, 比如3，众多集群中的随机3台机器），
nodes-指定（需配置ha-params参数，此参数为数组类型比如[“rabbit@F”,“rabbit@G”]这样指定为F与G这2台机器。）。

[root@server1 src]# rabbitmqctl set_policy ha-all "^" '{"ha-mode":"all"}'
Setting policy "ha-all" for pattern "^" to "{"ha-mode":"all"}" with priority "0" for vhost "/" ...

五、部署haproxy，实现访问5672端口时，轮询访问3个节点（明日补上）

六、部署Keepalived，实现主从热备、秒级切换（明日补上）

参考：https://blog.csdn.net/m0_49821830/article/details/108686163
参考：https://blog.csdn.net/qq_23191379/article/details/107065591
参考：https://blog.51cto.com/11134648/2155934