Monitor Dalli Connection Changes + Failures

Memcached servers changing leads to a tiny split-brain scenario since some servers might read from different caches then the others … good to keep an eye on it and alert when it happens too often. Here is a tiny snippet to report when it happens.

# config/initializers/dalli.rb
# Whenever the alive-ness of a server changes we read keys from a different server
# which leads to stale keys on the old server and cache-misses on the new servers
# so this should not happen often
# see lib/dalli/server.rb
# reproduce: rails c + Rails.cache.get + zdi memcached stop & start
Dalli::Server.prepend( do
  def down!
    $statsd.increment "dalli.connection_changed", tags: ["state:down"] unless @down_at

  def up!
    $statsd.increment "dalli.connection_changed", tags: ["state:up"] if @down_at

  def failure!(*)
    $statsd.increment "dalli.failed"

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s