Memcached servers changing leads to a tiny split-brain scenario since some servers might read from different caches then the others … good to keep an eye on it and alert when it happens too often. Here is a tiny snippet to report when it happens.
# config/initializers/dalli.rb # Whenever the alive-ness of a server changes we read keys from a different server # which leads to stale keys on the old server and cache-misses on the new servers # so this should not happen often # see lib/dalli/server.rb # # reproduce: rails c + Rails.cache.get + zdi memcached stop & start Dalli::Server.prepend(Module.new do def down! $statsd.increment "dalli.connection_changed", tags: ["state:down"] unless @down_at super end def up! $statsd.increment "dalli.connection_changed", tags: ["state:up"] if @down_at super end def failure!(*) $statsd.increment "dalli.failed" super end end)