redis哨兵模式的原理及部署

2024-04-16 11:34 由 misakivv 发表于 #数据库

一、什么是哨兵模式

Redis Sentinel 是一个分布式系统，为Redis提供高可用性解决方案。可以在一个架构中运行多个 Sentinel 进程(progress)，这些进程使用流言协议 (gossip protocols) 来接收关于主服务器是否下线的信息，并使用投票协议(agreement protocols)来决定是否执行自动故障迁移，以及选择哪个从服务器作为新的主服务器。

1、为什么需要哨兵机制

在没有哨兵机制的情况下，Redis主从集群中若主节点发生故障，需要运维人员手动介入，识别故障、选择合适的从节点提升为主节点，并更新所有客户端的连接配置，使其指向新的主节点。这一过程既繁琐又耗时，尤其是在紧急情况下可能导致服务长时间不可用。

2、哨兵架构拓扑

3、Redis Sentinel的功能：

对Redis节点进行监控
故障判断
故障转移
故障通知

二、搭建哨兵架构

1、涉及主机

角色	主机名	IP地址
主节点	master	192.168.112.40
从节点	slave2	192.168.112.50
从节点	slave1	192.168.112.60

2、拓扑结构

3、设置一主两从

所有节点：

[root@master ~]# vim /apps/redis/etc/redis.conf
bind 0.0.0.0
masterauth centos
requirepass centos
[root@master ~]# echo -e "net.core.somaxconn = 1024\nvm.overcommit_memory = 1" >> /etc/sysctl.conf
[root@master ~]# sysctl -p
net.core.somaxconn = 1024
vm.overcommit_memory = 1
net.core.somaxconn = 1024
vm.overcommit_memory = 1
[root@master ~]# echo never > /sys/kernel/mm/transparent_hugepage/enabled
[root@master ~]# echo "echo never > /sys/kernel/mm/transparent_hugepage/enabled" >> /etc/rc.d/rc.local
[root@master ~]# chmod +x /etc/rc.d/rc.local
[root@master ~]# systemctl restart redis

slave节点：

[root@slave1 ~]# echo "replicaof 192.168.112.40 6379" >> /apps/redis/etc/redis.conf
[root@slave1 ~]# systemctl restart redis
[root@slave2 ~]# echo "replicaof 192.168.112.40 6379" >> /apps/redis/etc/redis.conf
[root@slave2 ~]# systemctl restart redis

4、master服务器状态

[root@master ~]# redis-cli -a centos --no-auth-warning
127.0.0.1:6379> info replication
# Replication
role:master
connected_slaves:2
slave0:ip=192.168.112.60,port=6379,state=online,offset=518,lag=0
slave1:ip=192.168.112.50,port=6379,state=online,offset=518,lag=0
master_replid:4f28bb9953f9850a433ae73a943fa02e607f691f
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:518
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:1
repl_backlog_histlen:518

5、编辑哨兵的配置文件

Sentinel实际上是一个特殊的redis服务器,有些redis指令支持,但很多指令并不支持.默认监听在26379/tcp端口

哨兵可以不和Redis服务器部署在一起，但一般部署在一起，所有redis节点使用相同的配置文件

如果是编译安装，在源码目录有sentinel.conf，复制到安装目录即可

master节点：

[root@master ~]# cp redis-5.0.9/sentinel.conf /apps/redis/etc/
[root@master ~]# egrep -v "^#|^$" /apps/redis/etc/sentinel.conf
port 26379
daemonize no
pidfile /var/run/redis-sentinel.pid
logfile ""
dir /tmp  #工作目录
sentinel monitor mymaster 127.0.0.1 6379 2
#指定当前mymaster集群中master服务器的地址和端口
#2为法定人数限制(quorum)，即有几个sentinel认为master down了就进行故障转移，一般此值是所有sentinel节点(一般总数是>=3的 奇数,如:3,5,7等)的一半以上的整数值，比如，总数是3，即3/2=1.5，取整为2,是master的ODOWN客观下线的依据
sentinel auth-pass <master-name> <password>
#mymaster集群中master的密码，注意此行要在上面行的下面
sentinel down-after-milliseconds mymaster 30000
#(SDOWN)判断mymaster集群中所有节点的主观下线的时间，单位：毫秒，建议30000
sentinel parallel-syncs mymaster 1
#发生故障转移后，同时向新master同步数据的slave数量，数字越小总同步时间越长，但可以减轻新master的负载压力
sentinel failover-timeout mymaster 180000
#所有slaves指向新的master所需的超时时间，单位：毫秒
sentinel deny-scripts-reconfig yes
#禁止修改脚本

修改所有的哨兵服务器配置文件

修改配置文件前记得备份

#所有的哨兵服务器都是如下配置，以master为例

[root@master ~]# egrep -v "^#|^$" /apps/redis/etc/sentinel.conf
port 26379
daemonize no
pidfile /apps/redis/run/redis-sentinel.pid
logfile "/apps/redis/log/sentinel.log"
dir /tmp
sentinel monitor mymaster 192.168.112.40 6379 2
sentinel auth-pass mymaster centos
sentinel down-after-milliseconds mymaster 30000
sentinel parallel-syncs mymaster 1
sentinel failover-timeout mymaster 180000
sentinel deny-scripts-reconfig yes

6、启动哨兵

所有主机

#添加哨兵服务
cat << EOF > /lib/systemd/system/redis-sentinel.service
[Unit]
Description=Redis Sentinel
After=network.target
After=network-online.target
Wants=network-online.target
[Service]
ExecStart=/apps/redis/bin/redis-sentinel /apps/redis/etc/sentinel.conf --supervised systemd
ExecStop=/usr/libexec/redis-shutdown redis-sentinel
Type=notify
User=redis
Group=redis
RuntimeDirectory=redis
RuntimeDirectoryMode=0755
[Install]
WantedBy=multi-user.target
EOF
[root@master ~]# chown redis.redis /apps/redis/etc/sentinel.conf

#重载配置文件
[root@master ~]# systemctl daemon-reload
# 确保每个哨兵主机myid不同
#在master上
[root@master ~]# systemctl daemon-reload
[root@master ~]# systemctl enable --now redis-sentinel
Created symlink from /etc/systemd/system/multi-user.target.wants/redis-sentinel.service to /usr/lib/systemd/system/redis-sentinel.service.
[root@master ~]# grep myid /apps/redis/etc/sentinel.conf
sentinel myid 331200834d2dbc801dfe3714051861683517b037
#在slave1上
[root@slave1 tmp]# systemctl daemon-reload
[root@slave1 tmp]# systemctl enable --now redis-sentinel
Created symlink from /etc/systemd/system/multi-user.target.wants/redis-sentinel.service to /usr/lib/systemd/system/redis-sentinel.service.
[root@slave1 tmp]# grep myid /apps/redis/etc/sentinel.conf
sentinel myid d67683381ceb20bf91fc965540bd88a76594b7f5
#在slave2上
[root@slave2 tmp]# systemctl daemon-reload
[root@slave2 tmp]# systemctl enable --now redis-sentinel
Created symlink from /etc/systemd/system/multi-user.target.wants/redis-sentinel.service to /usr/lib/systemd/system/redis-sentinel.service.
[root@slave2 tmp]# grep myid /apps/redis/etc/sentinel.conf
sentinel myid 4223fba33129fa40e8af93590900b2f3e5a8f81f

7、验证哨兵端口

ss -tnl

8、查看哨兵日志

[root@master ~]# tail /apps/redis/log/sentinel.log
1628:X 16 Apr 2024 09:43:19.697 # You requested maxclients of 10000 requiring at least 10032 max file descriptors.
1628:X 16 Apr 2024 09:43:19.697 # Server can't set maximum open files to 10032 because of OS error: Operation not permitted.
1628:X 16 Apr 2024 09:43:19.697 # Current maximum open files is 4096. maxclients has been reduced to 4064 to compensate for low ulimit. If you need higher maxclients increase 'ulimit -n'.
1628:X 16 Apr 2024 09:43:19.698 * Running mode=sentinel, port=26379.
1628:X 16 Apr 2024 09:43:19.702 # Sentinel ID is 331200834d2dbc801dfe3714051861683517b037
1628:X 16 Apr 2024 09:43:19.702 # +monitor master mymaster 192.168.112.40 6379 quorum 2
1628:X 16 Apr 2024 09:43:19.703 * +slave slave 192.168.112.50:6379 192.168.112.50 6379 @ mymaster 192.168.112.40 6379
1628:X 16 Apr 2024 09:43:19.703 * +slave slave 192.168.112.60:6379 192.168.112.60 6379 @ mymaster 192.168.112.40 6379
1628:X 16 Apr 2024 09:44:54.907 * +sentinel sentinel d67683381ceb20bf91fc965540bd88a76594b7f5 192.168.112.50 26379 @ mymaster 192.168.112.40 6379
1628:X 16 Apr 2024 09:45:47.791 * +sentinel sentinel 4223fba33129fa40e8af93590900b2f3e5a8f81f 192.168.112.60 26379 @ mymaster 192.168.112.40 6379

9、验证当前sentinel状态

master节点：

[root@master ~]# redis-cli -p 26379
127.0.0.1:26379> info sentinel
# Sentinel
sentinel_masters:1
sentinel_tilt:0
sentinel_running_scripts:0
sentinel_scripts_queue_length:0
sentinel_simulate_failure_flags:0
master0:name=mymaster,status=ok,address=192.168.112.40:6379,slaves=2,sentinels=3

在sentinel状态中尤其是最后一行，涉及到masterIP是多少，有几个slave，有几个sentinels，必须是符合全部服务器数量

三、故障转移

1、redis sentinel故障转移的步骤：

1.当某个master发生故障，多个sentinel会监控到这个异常，这些sentinel会按照一定规则从多个slave中选中一个做为新的master,并通知别的slave从新的master中同步数据
2.当某个slave转换为新的master，sentinel会记录新的master的地址信息和slave的地址信息，通知Redis cli
3.Redis cli接收到新的master和slave的信息，就会向新的master写入数据，从slave中读取数据
4.等到原来的master重启之后，会变成新的master的slave,并从新的master同步数据

2、停止redis master

yum install -y psmisc
[root@master ~]# killall redis-server

3、查看各个节点哨兵信息

slave1：

[root@slave1 ~]# redis-cli -p 26379
127.0.0.1:26379> info sentinel
# Sentinel
sentinel_masters:1
sentinel_tilt:0
sentinel_running_scripts:0
sentinel_scripts_queue_length:0
sentinel_simulate_failure_flags:0
master0:name=mymaster,status=ok,address=192.168.112.60:6379,slaves=2,sentinels=3

slave2：

[root@slave2 ~]# redis-cli -p 26379
127.0.0.1:26379> info sentinel
# Sentinel
sentinel_masters:1
sentinel_tilt:0
sentinel_running_scripts:0
sentinel_scripts_queue_length:0
sentinel_simulate_failure_flags:0
master0:name=mymaster,status=ok,address=192.168.112.60:6379,slaves=2,sentinels=3

4、查看redis配置文件

[root@slave1 ~]# grep "^replicaof" /apps/redis/etc/redis.conf
replicaof 192.168.112.60 6379

[root@slave1 ~]# grep monitor /apps/redis/etc/sentinel.conf
sentinel monitor mymaster 192.168.112.60 6379 2

5、查看新master状态

[root@slave2 ~]# redis-cli -a centos --no-auth-warning
127.0.0.1:6379> info replication
# Replication
role:master
connected_slaves:1
slave0:ip=192.168.112.50,port=6379,state=online,offset=392291,lag=1
master_replid:c647c550c9394542e4f43b5538b1bfc1f7d5fedd
master_replid2:b54e20a0f991557145e9df7367777e1da8256a6f
master_repl_offset:392434
second_repl_offset:238694
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:1
repl_backlog_histlen:392434

6、恢复故障的原master重新加入redis集群

[root@master ~]# systemctl restart redis
[root@master ~]# ss -tnl
State       Recv-Q Send-Q Local Address:Port               Peer Address:Port             
LISTEN      0      128              *:22                           *:*
LISTEN      0      100      127.0.0.1:25                           *:*
LISTEN      0      511              *:6379                         *:*
LISTEN      0      511              *:26379                        *:*
LISTEN      0      128           [::]:22                        [::]:*
LISTEN      0      100          [::1]:25                        [::]:*
LISTEN      0      511           [::]:26379                     [::]:*
[root@master ~]# grep "^replicaof" /apps/redis/etc/redis.conf
replicaof 192.168.112.60 6379
[root@master ~]# redis-cli -a centos --no-auth-warning
127.0.0.1:6379> info replication
# Replication
role:slave
master_host:192.168.112.60
master_port:6379
master_link_status:up
master_last_io_seconds_ago:0
master_sync_in_progress:0
slave_repl_offset:446146
slave_priority:100
slave_read_only:1
connected_slaves:0
master_replid:c647c550c9394542e4f43b5538b1bfc1f7d5fedd
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:446146
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:438489
repl_backlog_histlen:7658

四、sentinel运维

1、手动让主节点下线

# 指定优先级,值越小sentinel会优先将之选为新的master,默为值为100
[root@master ~]# vim /apps/redis/etc/redis.conf
replica-priority 10
[root@master ~]# systemctl restart redis
[root@master ~]# redis-cli -p 26379
127.0.0.1:26379> sentinel failover mymaster            #手动发起哨兵切换
OK
127.0.0.1:26379> info sentinel
# Sentinel
sentinel_masters:1
sentinel_tilt:0
sentinel_running_scripts:0
sentinel_scripts_queue_length:0
sentinel_simulate_failure_flags:0
master0:name=mymaster,status=ok,address=192.168.112.40:6379,slaves=2,sentinels=3