背景简介
Redis 提供的如下技术「Redis Sentinel『主从切换』、Redis Cluster『分片』」,有效实现了 Redis 的高可用、高性能、高可伸缩性,本文对以上技术进行亲自动手实践。
1. Redis Sentinel「主从切换」
- 监控主从节点的在线状态,并根据配置自行完成切换「基于raft协议」。
- 主从复制从容量角度来说,还是单机。
2. Redis Cluster「分片」
- 通过一致性 hash 的方式,将数据分散到多个服务器节点:设计了 16384 个哈希槽,并分配到多台 redis-server。
- 当需要在 Redis Cluster 中存取一个 key 时,Redis 客户端先对 key 使用 CRC16 算法计算一个数值,然后对 16384 取模,这样每个 key 都会对应一个编号在 0-16383 之间的哈希槽,然后在此槽对应的节点上操作。
一、主从复制
设置详情
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
|
docker run -it --name redis-6380 -p 6380:6379 redis docker exec -it redis-6380 /bin/bash redis-cli -h 172.17.0.1 -p 6380
docker run -it --name redis-6381 -p 6381:6379 redis docker exec -it redis-6381 /bin/bash redis-cli -h 172.17.0.1 -p 6381 replicaof 172.17.0.1 6380
docker run -it --name redis-6382 -p 6382:6379 redis docker exec -it redis-6382 /bin/bash redis-cli -h 172.17.0.1 -p 6382 replicaof 172.17.0.1 6380
|
之后可查看 master 节点的信息,在 master-redis 下,执行:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
| > info Replication
role:master connected_slaves:2 slave0:ip=172.17.0.1,port=6379,state=online,offset=686,lag=0 slave1:ip=172.17.0.1,port=6379,state=online,offset=686,lag=1 master_replid:79187e2241015c2f8ed98ce68caafa765796dff2 master_replid2:0000000000000000000000000000000000000000 master_repl_offset:686 second_repl_offset:-1 repl_backlog_active:1 repl_backlog_size:1048576 repl_backlog_first_byte_offset:1 repl_backlog_histlen:686
|
之后操作 master 节点,slave 节点会自动同步。
slave-redis 下执行 replicaof no one
可重新改为主节点。
关键点
- 查看网络相关信息:
1 2
| docker network ls docker network inspect bridge
|
- 容器之间互访问,可使用内部端口号,也可使用外部映射端口号;
- 执行
docker network inspect bridge
后,可查看到网关 IP 以及各容器 IP,可使用 网关 IP : 外部映射端口
,或 容器 IP : 6379
访问 Redis;
参考资料
- 命令:SLAVEOF
二、Sentinel 高可用
当前状态:
- 网关IP:172.17.0.1
- master端口:6390
- slave端口:6391,6392
操作步骤
1. 重新创建 redis 的 docker 容器:
redis.conf 配置内容如下:
1 2 3 4 5 6 7 8
| port 6390
bind 0.0.0.0
daemonize no
|
变更监听端口号,并重新创建 redis 容器:
1 2 3 4 5 6 7 8 9 10 11 12 13
| docker run -p 6390:6390 -v D:\develop\shell\docker\redis\conf6390:/usr/local/etc/redis --name redis-conf-6390 redis redis-server /usr/local/etc/redis/redis.conf docker exec -it redis-conf-6390 /bin/bash redis-cli -h 172.17.0.1 -p 6390
docker run -p 6391:6391 -v D:\develop\shell\docker\redis\conf6391:/usr/local/etc/redis --name redis-conf-6391 redis redis-server /usr/local/etc/redis/redis.conf docker exec -it redis-conf-6391 /bin/bash redis-cli -h 172.17.0.1 -p 6391 slaveof 172.17.0.1 6390
docker run -p 6392:6392 -v D:\develop\shell\docker\redis\conf6392:/usr/local/etc/redis --name redis-conf-6392 redis redis-server /usr/local/etc/redis/redis.conf docker exec -it redis-conf-6392 /bin/bash redis-cli -h 172.17.0.1 -p 6392 slaveof 172.17.0.1 6390
|
之后可查看 master 节点的信息,可看到 master 获取到的 slave 的端口号恢复了正常。在 master-redis 下,执行:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
| > info Replication
role:master connected_slaves:2 slave0:ip=172.17.0.1,port=6391,state=online,offset=84,lag=0 slave1:ip=172.17.0.1,port=6392,state=online,offset=84,lag=0 master_replid:ed2e513ceed2b48a272b97c674c99d82284342a1 master_replid2:0000000000000000000000000000000000000000 master_repl_offset:84 second_repl_offset:-1 repl_backlog_active:1 repl_backlog_size:1048576 repl_backlog_first_byte_offset:1 repl_backlog_histlen:84
|
2. 创建配置文件
创建 sentinel.conf
,文件中写入如下内容:
1 2 3 4
| sentinel monitor bitkylin-master 172.17.0.1 6390 2 sentinel down-after-milliseconds bitkylin-master 5000 sentinel failover-timeout bitkylin-master 10000 sentinel parallel-syncs bitkylin-master 1
|
命令详解:指示 Sentinel 去监视一个名为 bitkylin-master 的主服务器,将这个主服务器标记为客观下线至少需要 2 个 Sentinel 同意;
响应超时 5 秒标记为主观下线,主观下线后就开始了迁移流程,超时 10 秒为迁移超时,暂不知用途。
3. 再创建两个 redis-docker 容器
将配置文件复制到 docker 容器内,共两个容器需要复制该文件:
1 2 3 4
| docker run -it --name redis-6490 redis docker run -it --name redis-6491 redis docker cp ./sentinel.conf dcbd015dbc0e:/data/sentinel.conf docker cp ./sentinel.conf 7c8307730bcc:/data/sentinel.conf
|
4. 执行 redis-sentinel 命令
1
| redis-sentinel sentinel.conf
|
5. 最终效果
此时任意启停 redis 容器,可以看到 sentinel 自动完成 redis 的主从切换,主从配置等不需要人工操作。
参考资料
- Redis 的 Sentinel 文档
- Docker 容器的文件操作
> 覆盖写入; >> 追加写入
三、Cluster 集群
操作步骤
1. 更新 redis 配置文件
主要追加集群配置信息,示例配置文件如下:
1 2 3 4 5 6 7 8 9 10 11 12 13 14
| port 6390
bind 0.0.0.0
daemonize no
cluster-enabled yes cluster-config-file nodes.conf cluster-node-timeout 5000 appendonly yes
|
2. 创建 6 个容器
以第二节作为基础,基于最新的配置文件,创建 6 个容器,注意新增集群总线端口映射:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23
| docker run -p 6390:6390 -p 16390:16390 -v D:\develop\shell\docker\redis\conf6390:/usr/local/etc/redis --name redis-conf-6390 redis redis-server /usr/local/etc/redis/redis.conf docker exec -it redis-conf-6390 /bin/bash redis-cli -h 172.17.0.1 -p 6390
docker run -p 6391:6391 -p 16391:16391 -v D:\develop\shell\docker\redis\conf6391:/usr/local/etc/redis --name redis-conf-6391 redis redis-server /usr/local/etc/redis/redis.conf docker exec -it redis-conf-6391 /bin/bash redis-cli -h 172.17.0.1 -p 6391
docker run -p 6392:6392 -p 16392:16392 -v D:\develop\shell\docker\redis\conf6392:/usr/local/etc/redis --name redis-conf-6392 redis redis-server /usr/local/etc/redis/redis.conf docker exec -it redis-conf-6392 /bin/bash redis-cli -h 172.17.0.1 -p 6392
docker run -p 6393:6393 -p 16393:16393 -v D:\develop\shell\docker\redis\conf6393:/usr/local/etc/redis --name redis-conf-6393 redis redis-server /usr/local/etc/redis/redis.conf docker exec -it redis-conf-6393 /bin/bash redis-cli -h 172.17.0.1 -p 6393
docker run -p 6394:6394 -p 16394:16394 -v D:\develop\shell\docker\redis\conf6394:/usr/local/etc/redis --name redis-conf-6394 redis redis-server /usr/local/etc/redis/redis.conf docker exec -it redis-conf-6394 /bin/bash redis-cli -h 172.17.0.1 -p 6394
docker run -p 6395:6395 -p 16395:16395 -v D:\develop\shell\docker\redis\conf6395:/usr/local/etc/redis --name redis-conf-6395 redis redis-server /usr/local/etc/redis/redis.conf docker exec -it redis-conf-6395 /bin/bash redis-cli -h 172.17.0.1 -p 6395
|
3. 直接通过命令创建集群
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54
| > redis-cli --cluster create 172.17.0.1:6390 172.17.0.1:6391 172.17.0.1:6392 172.17.0.1:6393 172.17.0.1:6394 172.17.0.1:6395 --cluster-replicas 1
>>> Performing hash slots allocation on 6 nodes... Master[0] -> Slots 0 - 5460 Master[1] -> Slots 5461 - 10922 Master[2] -> Slots 10923 - 16383 Adding replica 172.17.0.1:6394 to 172.17.0.1:6390 Adding replica 172.17.0.1:6395 to 172.17.0.1:6391 Adding replica 172.17.0.1:6393 to 172.17.0.1:6392 >>> Trying to optimize slaves allocation for anti-affinity [WARNING] Some slaves are in the same host as their master M: a9678b062663957e59bc3b4beb7be4366fa24adc 172.17.0.1:6390 slots:[0-5460] (5461 slots) master M: 41a4976431713cce936220fba8a230627d28d40c 172.17.0.1:6391 slots:[5461-10922] (5462 slots) master M: 1bf83414a12bad8f2e25dcea19ccea1c881d28c5 172.17.0.1:6392 slots:[10923-16383] (5461 slots) master S: 3d65eadd3321ef34c9413ae8f75d610c4228eda7 172.17.0.1:6393 replicates 41a4976431713cce936220fba8a230627d28d40c S: b604356698a5f211823ada4b45a97939744b1d57 172.17.0.1:6394 replicates 1bf83414a12bad8f2e25dcea19ccea1c881d28c5 S: 2c1cc93221dc3830aa1eb28601ac27e22a6801cc 172.17.0.1:6395 replicates a9678b062663957e59bc3b4beb7be4366fa24adc Can I set the above configuration? (type 'yes' to accept): yes >>> Nodes configuration updated >>> Assign a different config epoch to each node >>> Sending CLUSTER MEET messages to join the cluster Waiting for the cluster to join . >>> Performing Cluster Check (using node 172.17.0.1:6390) M: a9678b062663957e59bc3b4beb7be4366fa24adc 172.17.0.1:6390 slots:[0-5460] (5461 slots) master 1 additional replica(s) S: b604356698a5f211823ada4b45a97939744b1d57 172.17.0.1:6394 slots: (0 slots) slave replicates 1bf83414a12bad8f2e25dcea19ccea1c881d28c5 M: 41a4976431713cce936220fba8a230627d28d40c 172.17.0.1:6391 slots:[5461-10922] (5462 slots) master 1 additional replica(s) S: 3d65eadd3321ef34c9413ae8f75d610c4228eda7 172.17.0.1:6393 slots: (0 slots) slave replicates 41a4976431713cce936220fba8a230627d28d40c M: 1bf83414a12bad8f2e25dcea19ccea1c881d28c5 172.17.0.1:6392 slots:[10923-16383] (5461 slots) master 1 additional replica(s) S: 2c1cc93221dc3830aa1eb28601ac27e22a6801cc 172.17.0.1:6395 slots: (0 slots) slave replicates a9678b062663957e59bc3b4beb7be4366fa24adc [OK] All nodes agree about slots configuration. >>> Check for open slots... >>> Check slots coverage... [OK] All 16384 slots covered.
|
集群创建成功
注意点
- 需要开放集群总线端口号,默认为
业务端口号 + 10000
cluster reset
命令可以将当前节点从集群中移除
参考资料
- redis-cluster 集群 - 安装与状态验证
- Redis 集群教程
- Redis 命令参考 - 集群教程