How I performed a Redis fail over.

Recently we have encountered a failure, on one of our Redis nodes due to reaching the maximum number of clients connected. As an immediate action for resolving the incident, we wanted to modify the existing connection timeout property of the node and do a failover master node into a slave, since the master node is not responsive anymore.

The approach we followed for the first time.

We modify the redis.conf file on both master and the slave nodes. Then do a Redis service restart on the master node. Due to the service restart on the master, slave promoted itself into a master node.

The disadvantage of this approach is, once the master is restarting, it is losing any of the ongoing operations in the master node. This is not the appropriate approach recommended by the Redis.

The approach recommended by Redis.

Redis has inbuild command to failover master node into a slave node.
CLUSTER FAILOVER [FORCE|TAKEOVER]

we have used takeover option since both servers are running as expected and we just wanted to switch the master. Once we execute this command on the slave node, master stop consuming any new Redis connections and it waits till all existing connections complete their processing. Once completed, the master becomes the salve and hand over the master responsibilities to the other node. By following this approach, we did not lose any transactions like the previous approach.

Modify config values during the runtime.

We have used,
CONFIG SET
ex :
CONFIG SET timeout 70

For listing all the config, you can use

CONFIG GET *
ex:
CONFIG GET timeout

With this config set command, it will effect immediately on the instance but, keep in mind to change the Redis.conf file in order to apply the change in case of a service restart otherwise you will lose any changes you have done during the runtime.