Redis master-slave + KeepAlived achieve high availability

Redis is a non-relational database that we currently use more frequently. It can support diverse data types, multi-threaded high concurrency support, and redis running in memory with faster read and write. Because of the excellent performance of redis, how can we ensure that redis can deal with downtime during operation?

So today’s summary of the construction of the redis master-slave high-availability system, with reference to online bloggers of some great gods, found that many are pitted, so I share this one time, hoping to help everyone.

Redis Features
Redis is completely open source free, complies with the BSD protocol and is a high performance key-value database.

Redis and other key-value cache products have the following three characteristics:

Redis supports the persistence of data, can keep the data in memory on the disk, and can be loaded again for use when restarted.

Redis not only supports simple key-value types of data, but also provides data structures such as: Strings, Maps, Lists, Sets, and sorted sets. Storage.

Redis supports data backup, that is, data backup in master-slave mode.

The Redis advantage
is extremely high – Redis can read 100K+ times/s and write at 80K+ times/s.

Rich data types – Redis supports Strings, Lists, Hashes, Sets, and Ordered Sets data type operations for binary cases.

Atoms – All operations of Redis are atomic, and Redis also supports atomic execution of all operations after all operations.

Rich features – Redis also supports publish/subscribe, notifications, key expiration, and more.

Prepare environment

CentOS 7 –> 172.16.81.140 –> Master Redis –> Master Keepalived

CentOS7 –> 172.16.81.141 –> From Redis –> Prepared Keepalived

VIP –> 172.16.81.139

Redis (normally 3.0 or later)

KeepAlived (direct online installation)

Redis compile and install

cd /opt
tar -zxvf redis-4.0.6.tar.gz
mv redis-4.0.6 redis
cd redis
makeMALLOC=libc
make PREFIX=/usr/local/redis install

2, configure the redis startup script

vim /etc/init.d/redis

#!/bin/sh

#chkconfig:2345 80 90
# Simple Redisinit.d script conceived to work on Linux systems
# as it doeSUSE of the /proc filesystem.

# Configure the redis port number
REDISPORT=6379
# Configure the redis startup command path
EXE=/usr/local/redis/bin/ redisserver
# Configure the redis connection command path
CLIEXE=/usr/local/redis/bin/redis-cli
# Configure
Redis Run PID path PIDFILE=/var/run/redis_6379.pid
# Configure the path of redis configuration file
CONF=”/etc/redis/redis.conf”
# Configure the connection authentication password for redis
REDISPASSWORD=123456
function start () {
if [ -f $PIDFILE ]

then

echo “$PIDFILE exists,process is already running or crashed”

else

echo “Starting Redisserver…”

$EXE $CONF &

fi
}

function stop () {
if [ ! -f $PIDFILE ]

then

echo “$PIDFILE does not exist, process is not running”

else

PID=$(cat $PIDFILE)

echo “Stopping …”

$CLIEXE -p $REDISPORT -a $REDISPASSWORD shutdown

while [ -x /proc/${PID} ]

do

echo “Waiting forRedis to shutdown …”

sleep 1

done

echo “Redis stopped”

fi
}

function restart () {
stop

sleep 3

start
}

case “$1” in
start)
start
;;
stop)
stop
;;
restart)
restart
;;
*)
echo -e “\e[31m Please use $0 [start|stop|restart] asfirst argument \e[0m”
;;
esac

Grant execution permissions:

chmod +x /etc/init.d/redis

Add boot start:

chkconfig –add redis

chkconfig redis on

See: chkconfig –list | grep redis

The test closed the firewall and selinux beforehand. The production environment is recommended to open the firewall.

3, add redis command environment variables

#vi /etc/profile #Add the
next line of parameters
exportPATH=”$PATH:/usr/local/redis/bin” #Environment
variables become effective
source /etc/profile

4. Start redis service

Service redis start #Check
startup

ps -ef | grep redis

Note: After we perform the same operation on our two servers, the installation completes redis. After the installation is completed, we directly enter the configuration master-slave environment.

Redis master-slave configuration

To extend back to the previous design pattern, our idea is to use 140 as the master, 141 as the slave, and 139 as the VIP elegant address. The application accesses the redis database through the 6379 port of the 139.

In normal operation, when the master node 140 goes down, the VIP floats to 141, and then 141 will take over 140 as the master node, and 140 will become the slave node, continuing to provide read and write operations.

When 140 returns to normal, 140 will perform data synchronization with 141 at this time, 140 the original data will not be lost, and the data that has been written into 141 between synchronization machines will be synchronized. After the data synchronization is completed,

The VIP will return to the 140 node and become the master node because of the weight. 141 will lose the VIP and become the slave node again, and restore to the initial state to continue providing uninterrupted read and write services.

1, configure the redis configuration file

Master-140 configuration file

vim /etc/redis/redis.conf
bind 0.0.0.0
port 6379
daemonize yes
requirepass 123456
slave-serve-stale-data yes
slave-read-only no

Slave-141 configuration file

vim /etc/redis/redis.conf
bind 0.0.0.0
port 6379
daemonize yes
slaveof 172.16.81.140 6379
masterauth 123456
slave-serve-stale-data yes
slave-read-only no

2. Restart the redis service after the configuration is complete! Verify that the master and slave are normal.

Master node 140 terminal login test:

[root@localhost ~]# redis-cli -a 123456
127.0.0.1:6379> INFO
.
.
.
# Replication
role:master
connected_slaves:1
slave0:ip=172.16.81.141,port=6379,state=online,offset=105768,lag=1
master_replid:f83fcc3c98614d770f2205831fef1e877fa3f482
master_replid2:1f25604997a4ad3eb8344e8155990e78acd93312
master_repl_offset:105768
second_repl_offset:447
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:447
repl_backlog_histlen:105322

Login test from node 141 terminal:

[root@localhost ~]# redis-cli -a 123456
127.0.0.1:6379> info
.
.
.
# Replication
role:slave
master_host:172.16.81.140
master_port:6379
master_link_status:up
master_last_io_seconds_ago:5
master_sync_in_progress:0
slave_repl_offset:105992
slave_priority:100
slave_read_only:0
connected_slaves:0
master_replid:f83fcc3c98614d770f2205831fef1e877fa3f482
master_replid2:1f25604997a4ad3eb8344e8155990e78acd93312
master_repl_offset:105992
second_repl_offset:447
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:239
repl_backlog_histlen:105754
3, synchronous test

Master node 140

The masters and slaves of this redis have been completed!

KeepAlived configuration to achieve dual hot standby

Use Keepalived to implement VIP, and achieve disaster recovery through notify_master, notify_backup, notify_fault, notify_stop.

1, configure Keepalived configuration file

Master Keepalived Profile

vim /etc/keepalived/keepalived.conf
! Configuration File for keepalived

global_defs {
notification_email {
acassen@firewall.loc
failover@firewall.loc
sysadmin@firewall.loc
}
notification_email_from Alexandre.Cassen@firewall.loc
smtp_server 192.168.200.1
smtp_connect_timeout 30
router_id redis01
}

vrrp_script chk_redis {
script “/etc/keepalived/script/redis_check.sh”
interval 2
}

vrrp_instance VI_1 {
state MASTER
interface eno16777984
virtual_router_id 51
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}

track_script {
chk_redis
}
virtual_ipaddress {
172.16.81.139
}

notify_master /etc/keepalived/script/redis_master.sh
notify_backup /etc/keepalived/script/redis_backup.sh
notify_fault /etc/keepalived/script/redis_fault.sh
notify_stop /etc/keepalived/script/redis_stop.sh
}

Spare Keepalived Profile

vim /etc/keepalived/keepalived.conf
! Configuration File for keepalived

global_defs {
notification_email {
acassen@firewall.loc
failover@firewall.loc
sysadmin@firewall.loc
}
notification_email_from Alexandre.Cassen@firewall.loc
smtp_server 192.168.200.1
smtp_connect_timeout 30
router_id redis02
}

vrrp_script chk_redis {
script “/etc/keepalived/script/redis_check.sh”
interval 2
}

vrrp_instance VI_1 {
state BACKUP
interface eno16777984
virtual_router_id 51
priority 99
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}

track_script {
chk_redis
}
virtual_ipaddress {
172.16.81.139
}

notify_master /etc/keepalived/script/redis_master.sh
notify_backup /etc/keepalived/script/redis_backup.sh
notify_fault /etc/keepalived/script/redis_fault.sh
notify_stop /etc/keepalived/script/redis_stop.sh
}

2, configure the script

Master KeepAlived — 140

Create a script directory: mkdir -p /etc/keepalived/script/

cd /etc/keepalived/script/

[root@localhost script]# cat redis_check.sh
#!/bin/bash

ALIVE=`/usr/local/redis/bin/redis-cli -a 123456 PING`

if [ “$ALIVE” == “PONG” ];then

echo $ALIVE

exit 0

else

echo $ALIVE

exit 1

fi

[root@localhost script]# cat redis_master.sh
#!/bin/bash
REDISCLI=”/usr/local/redis/bin/redis-cli -a 123456″
LOGFILE=”/var/log/keepalived-redis-state.log”
sleep 15
echo “[master]” >> $LOGFILE
date >> $LOGFILE
echo “Being master….” >>$LOGFILE 2>&1
echo “Run SLAVEOF cmd …”>> $LOGFILE
$REDISCLI SLAVEOF 172.16.81.141 6379 >>$LOGFILE 2>&1
if [ $? -ne 0 ];then
echo “data rsync fail.” >>$LOGFILE 2>&1
else
echo “data rsync OK.” >> $LOGFILE 2>&1
fi

Sleep 10 # delay 10 seconds later to cancel synchronization after the data synchronization is complete

echo “Run SLAVEOF NO ONE cmd …”>> $LOGFILE

$REDISCLI SLAVEOF NO ONE >> $LOGFILE 2>&1
if [ $? -ne 0 ];then
echo “Run SLAVEOF NO ONE cmd fail.” >>$LOGFILE 2>&1
else
echo “Run SLAVEOF NO ONE cmd OK.” >> $LOGFILE 2>&1
fi

[root@localhost script]# cat redis_backup.sh
#!/bin/bash

REDISCLI=”/usr/local/redis/bin/redis-cli -a 123456″

LOGFILE=”/var/log/keepalived-redis-state.log”

echo “[backup]” >> $LOGFILE

date >> $LOGFILE

echo “Being slave….” >>$LOGFILE 2>&1

Sleep 15 #delay 15 seconds to wait until the data is synchronized to the other side and then switch the role of master-slave

echo “Run SLAVEOF cmd …”>> $LOGFILE

$REDISCLI SLAVEOF 172.16.81.141 6379 >>$LOGFILE 2>&1

[root@localhost script]# cat redis_fault.sh
#!/bin/bash

LOGFILE=/var/log/keepalived-redis-state.log

echo “[fault]” >> $LOGFILE

date >> $LOGFILE

[root@localhost script]# cat redis_stop.sh
#!/bin/bash

LOGFILE=/var/log/keepalived-redis-state.log

echo “[stop]” >> $LOGFILE

date >> $LOGFILE

Slave KeepAlived — 141

mkdir -p /etc/keepalived/script/

cd /etc/keepalived/script/

[root@localhost script]# cat redis_check.sh
#!/bin/bash

ALIVE=`/usr/local/redis/bin/redis-cli -a 123456 PING`

if [ “$ALIVE” == “PONG” ]; then

echo $ALIVE

exit 0

else

echo $ALIVE

exit 1

fi

[root@localhost script]# cat redis_master.sh
#!/bin/bash

REDISCLI=”/usr/local/redis/bin/redis-cli -a 123456″

LOGFILE=”/var/log/keepalived-redis-state.log”

echo “[master]” >> $LOGFILE

date >> $LOGFILE

echo “Being master….” >>$LOGFILE 2>&1

echo “Run SLAVEOF cmd …”>> $LOGFILE

$REDISCLI SLAVEOF 172.16.81.140 6379 >>$LOGFILE 2>&1

sleep 10 #

echo “Run SLAVEOF NO ONE cmd …”>> $LOGFILE

$REDISCLI SLAVEOF NO ONE >> $LOGFILE 2>&1

[root@localhost script]# cat redis_backup.sh
#!/bin/bash

REDISCLI=”/usr/local/redis/bin/redis-cli -a 123456″

LOGFILE=”/var/log/keepalived-redis-state.log”

echo “[backup]” >> $LOGFILE

date >> $LOGFILE

echo “Being slave….” >>$LOGFILE 2>&1

sleep 15 #

echo “Run SLAVEOF cmd …”>> $LOGFILE

$REDISCLI SLAVEOF 172.16.81.140 6379 >>$LOGFILE 2>&1

[root@localhost script]# cat redis_fault.sh
#!/bin/bash

LOGFILE=/var/log/keepalived-redis-state.log

echo “[fault]” >> $LOGFILE

date >> $LOGFILE

[root@localhost script]# cat redis_stop.sh
#!/bin/bash

LOGFILE=/var/log/keepalived-redis-state.log

echo “[stop]” >> $LOGFILE

date >> $LOGFILE

systemctl start keepalived

systemctl enable keepalived

ps -ef | grep keepalived

Install
——-
http://download.redis.io/redis-stable.tar.gz

$ wget http://download.redis.io/redis-stable.tar.gz
$ tar xvzf redis-stable.tar.gz
$ cd redis-stable
$ make
$ make test # optional
————–
# yum install wget tcl
# wget http://download.redis.io/releases/redis-3.2.5.tar.gz
# tar xzf redis-3.2.5.tar.gz
# cd redis-3.2.5
# make
# make test
————–
$ sudo make install
OR
$ sudo cp src/redis-server /usr/local/bin/
$ sudo cp src/redis-cli /usr/local/bin/
$ sudo mkdir /etc/redis
$ sudo mkdir /var/redis
$ sudo cp utils/redis_init_script /etc/init.d/redis_6379
$ sudo cp redis.conf /etc/redis/6379.conf
$ sudo mkdir /var/redis/6379
$ sudo update-rc.d redis_6379 defaults # OR sudo chkconfig –add redis_6379
$ sudo /etc/init.d/redis_6379 start
————–

TERMS
—–
RESP (REdis Serialization Protocol)
RESP, the type of some data depends on the first byte:
Simple Strings “+”
Errors “-”
Integers “:”
Bulk Strings “$”
Arrays “*”
Redis append-only file feature (AOF)

COMMANDS
——–
$ redis-server # start server
/etc/init.d/redis_PORT start
$ redis-cli [-p PORT] shutdown # stop server
/etc/init.d/redis_PORT stop
$ redis-cli ping # check if working
$ redis-cli –stat [-i interval] # continuous stats mode
$ redis-cli –bigkeys # scan for big keys
$ redis-cli [-p port ] –scan [–pattern REGEX] # get a list of keys
$ redis-cli [-p port ] monitor # monitor commands
$ redis-cli [-p port ] –latency # monitor latency of instances
$ redis-cli [-p port ] –latency-history [-i interval]
$ redis-cli [-p port ] –latency-dist # spectrum of latencies
$ redis-cli –intrinsic-latency [-p port ] [test-time] # latency of system
$ redis-cli –intrinsic-latency 5
$ redis-cli –rdb <dest-filename> # remote RDB backup ($?=0 success)
$ redis-cli –rdb /tmp/dump.rdb
$ redis-cli –slave # slave mode (monitor master -> slave writes)
$ redis-cli –lru-test 10000000 # Least Recently Used (LRU) simulation
# used to help configure ‘maxmemory’ for LRU
$ redis-cli save # save dump file (dump.rdb) to $dir
$ redis-cli select <DB_NUMBER> # select DB
$ redis-cli dbsize # show size of DB
$ redis-cli connect <SERVER> <PORT> # connect to different servers/ports
$ redis-cli debug restart
$ redis-cli –version
$ redis-cli pubsub channels [PATTERN]
$ redis-cli pubsub numsub [channel1 … channelN]
$ redis-cli subscribe/psubscribe/publish
$ redis-cli slowlog get [N]|len
a. unique progressive identifier for every slow log entry.
b. timestamp at which the logged command was processed.
c. amount of time needed for its execution, in microseconds.
d. array composing the arguments of the command.

FILES
—–
config file: /etc/redis/6371.conf
dbfilename dump.rdb
dir /var/lib/redis/6371
pidfile /var/run/redis/6371.pid
DB saved to:
/var/lib/redis/6371/dump.rdb

OPTIONS
——-
–raw, –no-raw

Configuration
————-
redis.conf # well documented
default ports: 6379 / 16379 (cluster mode) / 26379 (Sentinel)

daemonize no
pidfile /var/run/redis_6379.pid
port 6379
loglevel info
logfile /var/log/redis_6379.log
dir /var/redis/6379

keyword argument1 argument2 … argumentN

slaveof 127.0.0.1 6380
requirepass “hello world”
maxmemory 2mb
maxmemory-policy allkeys-lru
masterauth <password>
daemonize no # when run under daemontools

Administration
————–
/etc/sysctl.conf
—
vm.overcommit_memory = 1 # sysctl vm.overcommit_memory=1
echo never > /sys/kernel/mm/transparent_hugepage/enabled

# passing arguments via CLI
$ ./redis-server –port 6380 –slaveof 127.0.0.1 6379

redis> config get GLOB
redis> config set slaveof 127.0.0.1 6380
redis> config rewrite

— —
Actual config
—
daemonize yes
pidfile /var/run/redis/6371.pid
port 6371
tcp-backlog 511
bind 0.0.0.0
timeout 0
tcp-keepalive 0
loglevel notice
logfile “”
syslog-enabled yes
syslog-ident redis
syslog-facility USER
databases 16
save 900 1
save 300 10
save 60 10000
stop-writes-on-bgsave-error yes
rdbcompression yes
rdbchecksum yes
dbfilename dump.rdb
dir /var/lib/redis/6371
slaveof 10.200.18.115 6371 # only on slave(s)
slave-serve-stale-data yes
slave-read-only yes
repl-disable-tcp-nodelay no
slave-priority 100
maxclients 10000
maxmemory-policy noeviction
appendonly no
appendfilename “appendonly.aof”
appendfsync everysec
no-appendfsync-on-rewrite no
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb
lua-time-limit 5000
slowlog-log-slower-than 10000
slowlog-max-len 128
notify-keyspace-events “”
hash-max-ziplist-entries 512
hash-max-ziplist-value 64
list-max-ziplist-entries 512
list-max-ziplist-value 64
set-max-intset-entries 512
zset-max-ziplist-entries 128
zset-max-ziplist-value 64
activerehashing yes
client-output-buffer-limit normal 0 0 0
client-output-buffer-limit slave 256mb 64mb 60
client-output-buffer-limit pubsub 32mb 8mb 60
hz 10
aof-rewrite-incremental-fsync yes
—

Replication
———–
redis> config set masterauth <password>
persistence = enabled OR automatic-restarts = disabled

slaveof 192.168.1.1 6379
repl-diskless-sync
repl-diskless-sync-delay
slave-read-only
masterauth <password> # config set masterauth <password>
min-slaves-to-write <number of slaves>
min-slaves-max-lag <number of seconds>
slave-announce-ip 5.5.5.5
slave-announce-port 1234

Redis Sentinel (26379)
————–
– Monitoring
– Notification
– Automatic failover
– Configuration provider

redis-sentinel /path/to/sentinel.conf
OR
redis-server /path/to/sentinel.conf –sentinel

# typical minimal config
—
# sentinel monitor <master-group-name> <ip> <port> <quorum>
# sentinel down-after-milliseconds <master-name> <milliseconds> # default 30 secs
# sentinel failover-timeout <master-name> <milliseconds> # default 3 minutes
# sentinel parallel-syncs <master-name> <numslaves>
—
# example typical minimal config:
—
sentinel monitor mymaster 127.0.0.1 6379 2
sentinel down-after-milliseconds mymaster 60000
sentinel failover-timeout mymaster 180000
sentinel parallel-syncs mymaster 1
—
# additional configs
—
# bind 127.0.0.1 192.168.1.1
# protected-mode no
# sentinel announce-ip <ip>
# sentinel announce-port <port>
# dir <working-directory>
# syntax: sentinel <option_name> <master_name> <option_value>
# sentinel auth-pass <master-name> <password>
# sentinel down-after-milliseconds <master-name> <milliseconds> # default 30 secs
# sentinel parallel-syncs <master-name> <numslaves>
# sentinel failover-timeout <master-name> <milliseconds> # default 3 minutes
# sentinel notification-script <master-name> <script-path>
# passed: <event type> <event description>
# sentinel client-reconfig-script <master-name> <script-path>
# passed: <master-name> <role> <state> <from-ip> <from-port> <to-ip> <to-port>
# <state> is currently always “failover”
# <role> is either “leader” or “observer”
— — —
# actual config
—
/etc/redis/sentinel_26371.conf
# redis-sentinel 2.8.9 configuration file
# sentinel_26371.conf
daemonize no
dir “/var/lib/redis/sentinel_26371”
pidfile “/var/run/redis/sentinel_26371.pid”
port 26371
bind 0.0.0.0
sentinel monitor iowa_master_staging 10.200.18.115 6375 2
sentinel config-epoch iowa_master_staging 0
sentinel leader-epoch iowa_master_staging 0
sentinel known-slave iowa_master_staging 10.200.20.234 6375
logfile “”
syslog-enabled yes
syslog-ident “sentinel_26371”
syslog-facility user
————-
# sentinel messages/events
—
+monitor master <group-name> <ip> quorum <N>

# Testing
$ redis-cli -p PORT
127.0.0.1:PORT> SENTINEL master mymaster # info about master
127.0.0.1:PORT> SENTINEL slaves mymaster # info about slave(s)
127.0.0.1:PORT> SENTINEL sentinels mymaster # info about sentinel(s)
127.0.0.1:PORT> SENTINEL get-master-addr-by-name mymasterer # get address of master
$ redis-cli -p 6379 DEBUG sleep 30 # simulate master hanging
—
ping
SENTINEL masters # get list of monitored masters and their state
SENTINEL master <master name>
SENTINEL slaves <master name>
SENTINEL sentinels <master name>
SENTINEL get-master-addr-by-name <master name>
SENTINEL reset <pattern> # reset all masters matching pattern
SENTINEL failover <master name> # force a failover
SENTINEL ckquorum <master name> # check if current config is able to failover
SENTINEL flushconfig # rewrite config file
SENTINEL monitor <name> <ip> <port> <quorum> # start monitoring a new master
SENTINEL remove <name> # stop monitoring master
SENTINEL SET <name> <option> <value>
—
# Commands
$

Redis Cluster
————-
port 7000
cluster-enabled yes
cluster-config-file nodes.conf
cluster-node-timeout 5000
appendonly yes

=====================================
redis1:$ WTFI redis-cli => /nb/redis/bin/redis-cli

redis-cli -h <hostname> -p <port> -r <repeat (-1=forever)> -i <interval (secs)> -n <DB_NUM> <COMMAND>
redis-cli -p 6371|26371 info [server|clients|memory|persistence|stats|replication|cpu|keyspace|sentinel]
redis-cli -p 6371 ping
=====================================
Upgrading or restarting a Redis instance without downtime
Check out: https://redis.io/topics/admin (bottom)
=====================================
for p in $(grep ^port /etc/redis/*|awk ‘{print $NF}’); do echo “—- port: $p —-“; /nb/redis/bin/redis-cli -p $p info | grep stat; done

========== tool (redis_monit.sh) [begin] ==========
#!/bin/bash
# get status of redis servers
REDIS_CLI_CMD=/nb/redis/bin/redis-cli
# get the list of ports configured
ports=$(ls /etc/redis/*.conf | tr -d ‘[a-z/.]’)
for port in $ports; do
echo “—- port: $port —-”
if [ -e /etc/redis/$port.conf ]; then
$REDIS_CLI_CMD -p $port info | grep stat
else
echo “no config (/etc/redis/$port.conf)”
fi
done
========== tool (redis_monit.sh) [end] ==========

# update the monitor hosts – “live”
sentinel monitor mymaster 127.0.0.1 6379 2
sentinel monitor redis2 10.204.21.219 6379 2
sentinel failover redis1
sentinel masters
sentinel slaves redis1

# manual failover
# CLUSTER FAILOVER [FORCE|TAKEOVER]

$ redis-cli -p 7002 debug segfault