Sentinel sounds complicated at first, but when you get into it turns out to be rather simple. This guide will cover apsects of the Sentinel system you should know if you itnend to use, or are considering using Sentinel to manage your Redis master config.

First Things First: What is Sentinel?

Sentinel is an option for the Redis server binary to manage multiple Redis instances configured to replicate and provide high(er) availability. Currently, there is a lack of an established name for “a Redis master instance with one or more slaves. For this article I will use “pod” to indicate such a grouping. To identfy a colelction of Sentinels monitoring one or more pods I will use the term “constellation”.

What Sentinel Does

Sentinel monitors one or more given Pods for availability by connecting to the pod’s master and pulling information about the pod, and handling pod reconfiguration in the event of a master failure.

Client connections can take advantage of Sentinel support by connecting to one of the Sentinels monitoring the master and picking the master (or for read-only uses the slaves) to connect to.

What Sentinel Does NOT do

Sentinel does not manage a Virtual IP, load balancers, or clients. It does not provide data partitioning or multi-master clustering.

How Sentinel Works

Sentinel monitors a master, all slaves and other sentinels monitoring the pod, for loss of service. Upon one or more sentinels determining a node to be down it publishes an event, known as +sdown. if the number of sentinels issueing a +sdown is greater or equal to the configured quorum requirment an +odown event is issued and an election of a new master from the pool of promotable slaves is called.

Upon election of a new master the master is promoted and all slaves are reconfigured to slave from the new master. A +switch-master event is then issued. Several events happen in this process. They were left out for brevity’s sake.

If and when the old master comes back online the Sentinel constellation will reconfigure it to be a slave of the current master.

All of this activity is coordinated via the constellation via Redis’ native PUBSUB mechanisms. We will see more of this later.

Using Sentinel For More

With the native capabilities of Sentinel you get management of the Redis instances themselves. This is not always enough. If all of your client connections support and are configured for Sentinel this might be all you need to handle availability. Otherwise you will need something to track and act on the new master’s state. This might mean updating a DNS entry or changing a VIP configuration, or some other avenue. For these you will need to integrate with some form of external system.

We also may need to have this information available in other systems for event tracking or to trigger actions such as alerting us someting is wrong. This is where we roll up our sleeves and Make Things Happen.

Integrating With Other Systems

There are two primary avenues to integrate the actions and events in Redis Sentinel with other systems: script calls and event listening.

Using Scripts

Using scripts is perhaps the simpler route. In the sentinel.conf file you can specify, on a per-pod basis, a script or other system command to execute on event notifications or other events.

An example for notifications would be

sentinel notification-script mypodname /usr/bin/sentinel-event.sh

When a notification event occurs, Sentinel will call the configured script with two arguments - the event type (such as +sdown) and the event description. This script might log an entry, create a monitorign ticket or alert, or send an email.

Sentinel can do custom client reconfiguration via a script as well. The config format for this is sentinel client-reconfig-script <master-name> <script-path>. The arguments Sentinel will pass to this command are <master-name> <role> <state> <from-ip> <from-port> <to-ip> <to-port>. This configured command will be responsible for connecting to the client and making necessary changes. Note this command must be idempotent - calling it twice in a row with the same params should not result in different results. The reason for this is you will be running multiple Sentinels and each will call their local copy of the command.

Script execution can be useful, but it my opinion, event handling is a better route. Script execution has the benefit of not running additional daemons to handle events, but has the penalty of firing up external commands which may take longer to execute.

Using PUBSUB and Monitoring Events

The recommended way to integrate with Sentinel is to use PUBSUB. Because Sentinel publishes events on a PUBSUB channel on the Sentinels, you can subscribe to these channels and take action when new events are published.

This has the (potential) disadvantage of running your event handler(s) as a service (though without a need to listen on any port). In return,. however, you get the benefits of being able to run the handlers on different systems from the Sentinels and faster response times since you are not firing up a new process for the events. An additional benefit is better control over filtering.

To subscribe to a given event, you subscribe to a channel with the same name. For example to trigger an event on every +sdown event you would subscribe to the “+sdown” channel. This makes filtering automatic and allows you to properly isolate your handlers into their own services. For example your monitoring handlers can run on your monitoring server(s) and susbcribe only to monitoring style events while your load balancers run a daemon which listens only to the “+switch-master” event to alter the configuration of the load balancer.

There are many uses for this feature. A full list of channels/events is available at The Sentinel Doc.

Other possible scenarios include using the +slave/-slave events to monitor the addition or removal of slave instances. This might be useful in a deployment system. By monitoring each event in the sequence for failover you could easily generate a webpage showing the current status, or log each step to track timings and state.

Better Monitoring of Redis Masters and Sentinels

One use of Sentinel event handlers is for tighter monitoring of events as well as configuration auditing. I think the uses for monitoring are self-evident so I will skip them here. However, with some planning we can also use Sentinel to audit our configuration.

Given Sentinel knows the quorum and the number of Sentinels needed to make quorum possible, by writing a handler which subscribes to changes in the sentinel count you can make a monitor which detects the loss of quorum capability. You could then, as a simple matter of coding, use this to raise an alert to operations or to spin-up more sentinels to provide the necessary membership count.

Redis Sentinel Docs

Redis Cluster vs. Sentinel

Tags: availability sentinel