logo-full-color

Improving sparkle_formation method_missing

Sparkle formation has the habit of swallowing all typos, which makes debugging hard:

foo typo
dynanic! :bar
# ... builds
{
  "typo": {},
  "foo": "<#SparkleFormation::Struct",
  "dymanic!": {"bar": {}}
}

let’s make these fail:

  • no arguments or block
  • looks like a method (start with _ or end with !)
# calling methods without arguments or blocks smells like a method missing
::SparkleFormation::SparkleStruct.prepend(Module.new do
   def method_missing(name, *args, &block)
     caller = ::Kernel.caller.first

     called_without_args = (
       args.empty? &&
       !block &&
       caller.start_with?(File.dirname(File.dirname(__FILE__))) &&
       !caller.include?("vendor/bundle")
     )
     internal_method = (name =~ /^_|\!$/)

     if called_without_args || internal_method
       message = "undefined local variable or method `#{name}` (use block helpers if this was not a typo)"
       ::Kernel.raise NameError, message
     end
     super
   end
end)

Private gem leak / attack tester

A script to run on CI to make sure that:

  • no private gems are accidentally listed on rubygems.org (rake release happily does that for you)
  • nobody is trying to attack your private gems by releasing similar named ones

This is written for secure https://github.com/geminabox/geminabox via https://github.com/zendesk/geminastrongbox and might need to be modified to fit other gem servers.

#!/usr/bin/env ruby
def sh(command)
  result = `#{command}`
  raise "FAILED #{result}" unless $?.success?
  result
end

key = ENV.fetch('PRIVATE_SERVER_KEY')
host = ENV.fetch('PRIVATE_SERVER_HOST')
private_gem_names = sh("curl -fs 'https://#{key}@#{host}/gems'")
private_gem_names = private_gem_names.scan(%r{"#{host}/gems/gems/([^"]+)"}).flatten
puts "Found #{private_gem_names.size} private gems"
puts private_gem_names.join(", ")

exposed = sh("curl -fs 'https://rubygems.org/api/v1/dependencies?gems=#{private_gem_names.join(",")}'")
exposed = Marshal.load(exposed).map { |d| d[:name] }.uniq
puts "Found #{exposed.size} of them on rubygems.org"
puts exposed.join(", ")

if exposed.sort == ["LIST KNOW DUPLICATE HERE"].sort
  puts "All good!"
else
  raise "Hacked private gems !?: #{exposed.join(", ")}"
end

Avoiding Sidekiq Memory Leaks By Killing It Regularly

Memory leaks are hard to hunt down … but killing things is super easy :)
(you have a monitor that restarts them … right !?)

# memory leaks or config changes happen ...
# shutdown -> slowly kill all threads -> restarted by service
# log of all restarts is kept in tmp/shutdown
def trigger_worker_restart_after_interval
  Thread.new do
    interval = 30*60
    time = interval + rand(interval)
    puts "Planning restart in #{time}s -> #{(Time.now + time).to_s(:db)}"

    sleep time
    puts "Planned restart"
    FileUtils.mkdir_p("tmp")
    File.write("tmp/parent_pid", Process.pid)

    # end this thread to ensure a clean exit and kill yourself
    `bundle exec sidekiqctl stop tmp/parent_pid 60 2>&1 >> tmp/shutdown &`
  end
end

Rails spring on steroid: take spring test time from 6s to 0.5s

We run a large app and test times even with spring where around 6s, so I fixed that :)

  • Preload environment
  • Preload test_helper via forking_test_runner
  • Disable slow at_exit handler
  • Enable by line running via testrbl
  • Preload fixtures
  • Turn logging on by default
# config/environment.rb
Spring::Commands::Test.after_environment if ENV['SPRING_PRELOADED']


# config/spring.rb
if defined?(Spring)

  raise "Spring is out of date: gem install spring" if Gem::Version.new(Spring::VERSION) < Gem::Version.new("1.3.6")

  ENV['SPRING_LOG'] = 'log/spring.log' # turn logging mode on
  ENV['SPRING_PRELOADED'] = 'true'

  class Spring::Commands::Test
    def env(*)
      "test"
    end

    def call
      # running by line number ?
      if ARGV.first =~ /^(\S+):(\d+)$/
        file, line = $1, $2
        pattern = Testrbl.pattern_from_file(File.readlines(file), line)
        ARGV[0..0] = [file, "-n", "/#{pattern}/"]
      end

      # running with --changed flag ?
      if ARGV.delete("--changed")
        ARGV[0...0] = Testrbl.send(:changed_files)
      end

      # load all the tests
      ForkingTestRunner.send(:enable_test_autorun, 'lib/slug_ids') # require an innocent files since we handle require ourselves
      ARGV.each do |arg|
        break if arg.start_with?("-")
        require_test(File.expand_path(arg))
      end
    end

    def description
      <<-USAGE
Run a test.
                    test/unit/xxx_test.rb:123            # test by line number
                    test/unit                            # everything _test.rb in a folder (on 1.8 this would be test/unit/*)
                    xxx_test.rb yyy_test.rb              # multiple files
                    --changed                            # run changed tests
                    test/unit/xxx_test.rb -n "/hello/"   # test by name
      USAGE
    end

    # we need to do some initialization after everything is loaded
    # otherwise we end up with for example extra tests running because urls are not cleaned out
    def self.after_environment
      return unless Rails.env.test?

      require 'testrbl'

      # preload as much of the test environment as possible for fast test startup
      require "forking_test_runner"
      ForkingTestRunner.send(:disable_test_autorun)

      require_relative '../test/helpers/test_helper'
      if User.count.zero?
        puts "No fixture data loaded. preload using rake db:fixtures:load"
        puts "Falling back to slower fixture loading strategy..."
      else
        ForkingTestRunner.send(:preload_fixtures)
      end

      # make test process not hang 2+s after it is done with tests
      # minitest calls at_exit twice and we need to hook in after the second
      # this might lead to some tempfiles pollution or logs missing ... time will tell
      require 'minitest/unit'
      class << MiniTest::Unit
        def at_exit(&block)
          @at_exit_called ||= 0
          @at_exit_called += 1
          super

          if @at_exit_called == 2
            super do
              status = (($!.respond_to?(:status) && $!.status) || 1)
              exit! status
            end
          end
        end
      end
    end

    private

    def require_test(path)
      if File.directory?(path)
        Dir[File.join path, "**", "*_test.rb"].each { |f| require f }
      else
        require path
      end
    end
  end

  Spring.register_command "test", Spring::Commands::Test.new
end