Category Archives: Uncategorized

Parallelizing SparkleFormation execution

We generate each templates CloudFormation .json file and check them in to spot subtle diffs when refactoring. We also validate all configs on every PR. Doing this serially takes a lot of time, but it can be parallelized easily, while also avoiding ruby boot overhead.

This took us from ~5 minutes runtime to 30s, enjoy!

desc "Validate templates via cloudformation api"
task :validate do
  each_template do |template|
    execute_sfn_command :validate, template
  end
end

desc "Generates cloudformation json files"
task :generate, [:pattern] do |_t, args|
  pattern = /#{args[:pattern]}/ if args[:pattern]
  previous = Dir["generated/*.json"]

  used = each_template do |template|
    next if pattern && template !~ pattern
    output = execute_sfn_command :print, template

    generated = "generated/#{File.basename(template.sub('.rb', '.json'))}"
    File.write generated, output
    generated
  end

  (previous - used).each { |f| File.unlink(f) } unless pattern
end

...

require 'parallel'
require 'timeout'

def each_template(&block)
  # preload slow requires
  require 'bogo-cli'
  require 'sfn'

  # run in parallel, but isolated to avoid caches from being reused
  options = {in_processes: 10, progress: "Progress", isolation: true}
  Parallel.map(Dir["templates/**/*.rb"], options, &block)
end

private

def execute_sfn_command(command, template, *args)
  Timeout.timeout(20) do
    capture_stdout do
      Sfn::Command.const_get(command.capitalize).new({
        defaults: true, # do not ask questions about parameters
        file: template,
        retry: {type: :flat, interval: 1} # we will run into rate limits, ignore them quickly
      }, args).execute!
    end
  end
rescue StandardError
  # give users context when something failed
  warn "bundle exec sfn #{command} #{args.join(" ")} --defaults -f #{template}"
  raise
end

def capture_stdout
  old = $stdout
  $stdout = StringIO.new
  yield
  $stdout.string
ensure
  $stdout = old
end

Thoughts on sparkleformation

sfn provides a few nice features, but it is really hard to deal with all the other craziness that goes on …

method_missing (typos result in silently swallowed errors)
https://github.com/sparkleformation/sparkle_formation/pull/112

components being loaded multiple times
https://github.com/sparkleformation/sparkle_formation/issues/114

cli silently catching all kinds of template errors making debugging impossible for anyone not knowing how to open a gem …
https://github.com/sparkleformation/sfn/issues/168
Solution: use .sfn.rb

very little testing going on + no CI setup …
https://github.com/sparkleformation/sparkle_formation/pull/110

ruby code is written in a non-standard style

forcing a `Bundler.require` which makes everything slow when there are lots of gems in the Gemfile https://github.com/sparkleformation/sfn/issues/170
Solution: add a :sfn group that only includes sfn gems

PR comments being ignored https://github.com/sparkleformation/sparkle_formation/pull/117

It adds a lot of unnecessary work and bugs that it tries to re-invent libraries that already exist (thor/optparse/etc for cli interfaces) and fog for api abstraction.

I don’t mean to hate on chrisroberts … he fixed lots of bugs / is helpful / drives this project forward , I just feel like this project has basic flaws in it’s foundation that should be fixed instead of adding more and more features on top.

I want to like/use this project, but this makes it really hard … the more templates we add / the more members we introduce to sfn the worse I feel about this whole situation … and the more I want to replace it with something sane/verbose that ‘just works’ instead of being magically broken.

Sending configuration into a AWS Lambda created via Cloudformation

Lambdas can only have static code (see code upload via cloudformation), so passing in DynamoDB table names/SNS topic ARNs etc is not possible. But there is a neat workaround:

Make the lambda read the stacks output.

# my-stack.json
"Outputs": {
  "World": {
    "Value": {
      "Ref": "MySnsTopic"
    }
  },
  ....
}


var AWS = require('aws-sdk');
var stack = context.invokedFunctionArn.match(/:function:(.*)-.*-.*/)[1];

exports.handler = function(event, context) {
  var cf = new AWS.CloudFormation();
  cf.describeStacks({"StackName": stack}, function(err, data){
    if (err) context.done(err, 'Error!');
    else {
      var config = {}
      data.Stacks[0].Outputs.map(function(out){ config[out.OutputKey] = out.OutputValue });
      context.succeed('hello ' + config.World)
    }
  })
};


# output
"hello arn:aws:sns:ap-northeast-1:8132302344234:my-stack-MySnSTopic-211Z1K3GAGK9"

Rails spring on steroid: take spring test time from 6s to 0.5s

We run a large app and test times even with spring where around 6s, so I fixed that 🙂

  • Preload environment
  • Preload test_helper via forking_test_runner
  • Disable slow at_exit handler
  • Enable by line running via testrbl
  • Preload fixtures
  • Turn logging on by default
# config/environment.rb
Spring::Commands::Test.after_environment if ENV['SPRING_PRELOADED']


# config/spring.rb
if defined?(Spring)

  raise "Spring is out of date: gem install spring" if Gem::Version.new(Spring::VERSION) < Gem::Version.new("1.3.6")

  ENV['SPRING_LOG'] = 'log/spring.log' # turn logging mode on
  ENV['SPRING_PRELOADED'] = 'true'

  class Spring::Commands::Test
    def env(*)
      "test"
    end

    def call
      # running by line number ?
      if ARGV.first =~ /^(\S+):(\d+)$/
        file, line = $1, $2
        pattern = Testrbl.pattern_from_file(File.readlines(file), line)
        ARGV[0..0] = [file, "-n", "/#{pattern}/"]
      end

      # running with --changed flag ?
      if ARGV.delete("--changed")
        ARGV[0...0] = Testrbl.send(:changed_files)
      end

      # load all the tests
      ForkingTestRunner.send(:enable_test_autorun, 'lib/slug_ids') # require an innocent files since we handle require ourselves
      ARGV.each do |arg|
        break if arg.start_with?("-")
        require_test(File.expand_path(arg))
      end
    end

    def description
      <<-USAGE
Run a test.
                    test/unit/xxx_test.rb:123            # test by line number
                    test/unit                            # everything _test.rb in a folder (on 1.8 this would be test/unit/*)
                    xxx_test.rb yyy_test.rb              # multiple files
                    --changed                            # run changed tests
                    test/unit/xxx_test.rb -n "/hello/"   # test by name
      USAGE
    end

    # we need to do some initialization after everything is loaded
    # otherwise we end up with for example extra tests running because urls are not cleaned out
    def self.after_environment
      return unless Rails.env.test?

      require 'testrbl'

      # preload as much of the test environment as possible for fast test startup
      require "forking_test_runner"
      ForkingTestRunner.send(:disable_test_autorun)

      require_relative '../test/helpers/test_helper'
      if User.count.zero?
        puts "No fixture data loaded. preload using rake db:fixtures:load"
        puts "Falling back to slower fixture loading strategy..."
      else
        ForkingTestRunner.send(:preload_fixtures)
      end

      # make test process not hang 2+s after it is done with tests
      # minitest calls at_exit twice and we need to hook in after the second
      # this might lead to some tempfiles pollution or logs missing ... time will tell
      require 'minitest/unit'
      class << MiniTest::Unit
        def at_exit(&block)
          @at_exit_called ||= 0
          @at_exit_called += 1
          super

          if @at_exit_called == 2
            super do
              status = (($!.respond_to?(:status) && $!.status) || 1)
              exit! status
            end
          end
        end
      end
    end

    private

    def require_test(path)
      if File.directory?(path)
        Dir[File.join path, "**", "*_test.rb"].each { |f| require f }
      else
        require path
      end
    end
  end

  Spring.register_command "test", Spring::Commands::Test.new
end