Category Archives: Ruby

Private gem leak / attack tester

A script to run on CI to make sure that:

  • no private gems are accidentally listed on rubygems.org (rake release happily does that for you)
  • nobody is trying to attack your private gems by releasing similar named ones

This is written for secure https://github.com/geminabox/geminabox via https://github.com/zendesk/geminastrongbox and might need to be modified to fit other gem servers.

#!/usr/bin/env ruby
def sh(command)
  result = `#{command}`
  raise "FAILED #{result}" unless $?.success?
  result
end

key = ENV.fetch('PRIVATE_SERVER_KEY')
host = ENV.fetch('PRIVATE_SERVER_HOST')
private_gem_names = sh("curl -fs 'https://#{key}@#{host}/gems'")
private_gem_names = private_gem_names.scan(%r{"#{host}/gems/gems/([^"]+)"}).flatten
puts "Found #{private_gem_names.size} private gems"
puts private_gem_names.join(", ")

exposed = sh("curl -fs 'https://rubygems.org/api/v1/dependencies?gems=#{private_gem_names.join(",")}'")
exposed = Marshal.load(exposed).map { |d| d[:name] }.uniq
puts "Found #{exposed.size} of them on rubygems.org"
puts exposed.join(", ")

if exposed.sort == ["LIST KNOW DUPLICATE HERE"].sort
  puts "All good!"
else
  raise "Hacked private gems !?: #{exposed.join(", ")}"
end

Avoiding Sidekiq Memory Leaks By Killing It Regularly

Memory leaks are hard to hunt down … but killing things is super easy 🙂
(you have a monitor that restarts them … right !?)

# memory leaks or config changes happen ...
# shutdown -> slowly kill all threads -> restarted by service
# log of all restarts is kept in tmp/shutdown
def trigger_worker_restart_after_interval
  Thread.new do
    interval = 30*60
    time = interval + rand(interval)
    puts "Planning restart in #{time}s -> #{(Time.now + time).to_s(:db)}"

    sleep time
    puts "Planned restart"
    FileUtils.mkdir_p("tmp")
    File.write("tmp/parent_pid", Process.pid)

    # end this thread to ensure a clean exit and kill yourself
    `bundle exec sidekiqctl stop tmp/parent_pid 60 2>&1 >> tmp/shutdown &`
  end
end

Check in your gems Gemfile.lock

Advantages of checking in the Gemfile.lock when developing gems:

  • A known good state to fall back on and do incremental updates
  • Being able to instantly work on an old project
  • No random failures for every new contributor
  • No “Bundler could not find compatible versions for gem” errors when Gemfile changes and users have an old lock

To test against newest versions, add `before_install: rm Gemfile.lock` to .travis.yml … same coverage, less drama.

Print capistrano execution time per server

Are all servers slow or just one ?

Output
Example logging all times > 3 seconds.

executing 'ruby -e "r = rand(10); puts %{sleep #{r}}; sleep r"'
 ** [out :: dbadmin1] sleep 6
 ** [out :: app1] sleep 4
 ** [out :: work1] sleep 2
 ** [out :: mirror1] sleep 5
 ** Server dbadmin1 finished command in 4 seconds
 ** Server app1 finished command in 7 seconds

Code

module PerServerTime
  # set via Capistrano::Command.per_server_time_threshold = 111
  def self.included(base)
    class << base
      attr_accessor :per_server_time_threshold
    end
  end

  # whenever a command finishes on 1 server print how long it ran on this server
  def process_iteration
    return super if @channels.size < 2 # do not need per-server logging for 1 server

    @start_time ||= Time.now.to_f
    @closed_channels ||= []

    @channels.each do |channel|
      if channel[:closed] && !@closed_channels.include?(channel)
        @closed_channels < (self.class.per_server_time_threshold || 30)
          host = channel.connection.instance_variable_get(:@xserver).host
          logger.info("Server #{host} finished command in #{time} seconds")
        end
      end
    end

    super
  end
end

Capistrano::Command.send(:include, PerServerTime)

Omniauth / OAuth integration test with webmock

Took me a while to figure out all the stubs … maybe this is useful later 🙂
I’m using this to test our facebook / twitter signup flows.

def external_redirect(url)
  yield
rescue ActionController::RoutingError # goes to twitter.com/oauth/authenticate
  current_url.must_equal url
else
  raise "Missing external redirect"
end

it "signs up" do
  twitter_id = 12345
  visit "/"
  
  stub_request(:post, "https://api.twitter.com/oauth/request_token").
    to_return(body: "oauth_token=TOKEN&oauth_token_secret=SECRET&oauth_callback_confirmed=true")

  external_redirect "https://api.twitter.com/oauth/authenticate?x_auth_access_type=read&oauth_token=TOKEN" do
    click_link "Continue with Twitter"
  end

  # https://dev.twitter.com/oauth/reference/post/oauth/access_token
  stub_request(:post, "https://api.twitter.com/oauth/access_token").
    to_return(body: "oauth_token=TOKEN&oauth_token_secret=SECRET&user_id=#{twitter_id}&screen_name=twitterapi")

  # https://dev.twitter.com/rest/reference/get/account/verify_credentials
  stub_request(:get, "https://api.twitter.com/1.1/account/verify_credentials.json?include_entities=false&skip_status=true").
    to_return(body: %{{"profile_image_url_https": "image_normal.png"}})

  visit "/auth/twitter/callback"
end