Monitor Dalli Connection Changes + Failures

Memcached servers changing leads to a tiny split-brain scenario since some servers might read from different caches then the others … good to keep an eye on it and alert when it happens too often. Here is a tiny snippet to report when it happens.

# config/initializers/dalli.rb
# Whenever the alive-ness of a server changes we read keys from a different server
# which leads to stale keys on the old server and cache-misses on the new servers
# so this should not happen often
# see lib/dalli/server.rb
#
# reproduce: rails c + Rails.cache.get + zdi memcached stop & start
Dalli::Server.prepend(Module.new do
  def down!
    $statsd.increment "dalli.connection_changed", tags: ["state:down"] unless @down_at
    super
  end

  def up!
    $statsd.increment "dalli.connection_changed", tags: ["state:up"] if @down_at
    super
  end

  def failure!(*)
    $statsd.increment "dalli.failed"
    super
  end
end)

Bundler + Rails: realpath expand_path and symlinked vendor/bundle cause ‘already loaded’ errors

Symlinked vendor/bundle results in double load errors since:

  • rails adds the realpath of each engines lib (and various other folders) to the $LOAD_PATH
  • bundler adds the symlinked version to the $LOAD_PATH
  • require_relative uses the realpath

which looks like: `already initialized constant Arturo::Middleware::MISSING_FEATURE_ERROR`

Reproduce:

  • bundle install –path vendor/bundle
  • mv vendor/bundle tmp
  • cd vendor && ln -s ../tmp/bundle bundle
  • enable eager_load + preload_frameworks in config/environment/development.rb
  • rails runner 1

Ruby issue

Patch:

# Gemfile
require_relative 'lib/bundler_realpath'
 
# lib/bundler_realpath.rb
# https://grosser.it/2017/08/19/bundler-rails-realpath-expand_path-and-symlinked-vendorbundle-cause-already-loaded-errors
Bundler::Runtime.prepend(Module.new do
  def setup(*)
    super
  ensure
    linked_bundle = File.expand_path(Bundler.bundle_path)
    real_bundle = File.realpath(Bundler.bundle_path)
    if linked_bundle != real_bundle
      $LOAD_PATH.each_with_index do |path, i|
        $LOAD_PATH[i] = path.sub(linked_bundle, real_bundle)
      end
    end
  end
end)