Memcached servers changing leads to a tiny split-brain scenario since some servers might read from different caches then the others … good to keep an eye on it and alert when it happens too often. Here is a tiny snippet to report when it happens.

# config/initializers/dalli.rb
# Whenever the alive-ness of a server changes we read keys from a different server
# which leads to stale keys on the old server and cache-misses on the new servers
# so this should not happen often
# see lib/dalli/server.rb
#
# reproduce: rails c + Rails.cache.get + zdi memcached stop & start
Dalli::Server.prepend(Module.new do
  def down!
    $statsd.increment "dalli.connection_changed", tags: ["state:down"] unless @down_at
    super
  end

  def up!
    $statsd.increment "dalli.connection_changed", tags: ["state:up"] if @down_at
    super
  end

  def failure!(*)
    $statsd.increment "dalli.failed"
    super
  end
end)