Ruby Capture Stdout Without $stdout

Rake’s sh for example cannot be captured with the normal $stdount = StringIO.new trick, but using some old ActiveSupport code that uses reopen with a Temfile seems to do the trick.

# StringIO does not work with rubies `system` call that `sh` uses under the hood, so using Tempfile + reopen
# https://grosser.it/2018/11/23/ruby-capture-stdout-without-stdout
def capture_stdout
  old_stdout = STDOUT.dup
  Tempfile.open('tee_stdout') do |capture|
    STDOUT.reopen(capture)
    STDOUT.sync = true
    yield
    capture.rewind
    capture.read
  end
ensure
  STDOUT.reopen(old_stdout)
end

Listing Unmuted Datadog Alerts

Datadogs UI for alerting monitors also shows muted ones without an option to filter, which leads to overhead / confusion when trying to track down what exactly is down.

So we added an alerts task to  kennel  that lists all unmuted alerts and since when they are alerting, it also shows alerts that have no-data warnings even though Datadog UI shows them as not alerting.

bundle exec rake kennel:alerts TAG=team:my-team
Downloading ... 5.36s
Foo certs will expire soon🔒
https://app.datadoghq.com/monitors/123
Ignored cluster:pod1,server:10.215.225.122 39:22:00
Ignored cluster:pod12,server:10.218.176.123 31:41:00
Ignored cluster:pod12,server:10.218.176.123 31:41:00


Foobar Errors (Retry Limit Exceeded)🔒
https://app.datadoghq.com/monitors/1234
Alert cluster:pod2 19:05:16

Ruby: Waiting for one of multiple threads to finish

We build a small project that watches multiple metrics until one of them finds something, I found ThreadsWait in the stdlib and it was easy to use it. Also added error re-raising so the threads do not die silently and cleanup.

require 'thwait'

def wait_for_first_block_to_complete(*blocks)
  threads = blocks.map do |block|
    Thread.new do
      block.call
    rescue StandardError => e
      e
    end
  end
  waiter = ThreadsWait.new(*threads)
  value = waiter.next_wait.value
  threads.each(&:kill)
  raise value if value.is_a?(StandardError)
  value
end

wait_for_first_block_to_complete(
  -> { sleep 5 }, -> { sleep 1 }, -> { sleep 2 }
) # will stop after 1 second

 

Reading journald kernel logs from inside a kubernetes pod

We wanted a watcher that alerts us when bad kernel things happen and were able to deploy that as a DaemonSet using Kubernetes 🙂

  • Use a Debian base image (for example ruby:2.5-stretch)
  • Run as root user or as user that can read systemd logs like systemd-journal
  • Mount /run/log/journal
    spec:
      containers:
      - name: foo
        ...
        volumeMounts:
        - name: runlog
          mountPath: /run/log/journal
          readOnly: true
      volumes:
      - name: runlog
        hostPath:
          path: /run/log/journal
  • Use systemd-journal to read the logs
    require 'systemd/journal'
    journal = Systemd::Journal.new
    journal.seek(:tail)
    journal.move_previous
    journal.filter(syslog_identifier: 'kernel')
    journal.watch { |entry| puts entry.message }

Building single/partial steps from cloudbuild.yaml

We have a single project that builds our GCR/gcloud base-images, it has a lot of reuse between steps so it is nice to not have multiple repos, but locally testing the build became super long since there are so many different images.

Here is a little script to build only a single step and it’s dependencies.

 


remote = ARGV.delete("–remote")
id = ARGV.pop
abort "Usage: ruby build.rb <id|all> [–remote]" unless ARGV.empty?
unless system('which container-builder-local')
abort "Run: gcloud components install container-builder-local"
end
def dependencies(steps, id)
step = steps.detect { |s| s.fetch("id") == id }
[id] + (step["waitFor"] || []).flat_map { |w| dependencies(steps, w) }
end
# make sure to use all variables to avoid:
# "invalid build: key in the substitution data is not matched in the template"
def echo_step
{
"id" => "echo",
"name" => "gcr.io/cloud-builders/wget",
"entrypoint" => "echo",
"args" => ["_IMAGE_DIR is $_IMAGE_DIR"]
}
end
require 'yaml'
config = YAML.load_file("cloudbuild.yaml")
config.delete("images")
unless id == "all"
steps = config.fetch("steps")
keep = dependencies(steps, id)
steps.select! { |s| keep.include? s.fetch("id") }
steps.unshift(echo_step)
puts "Running steps #{steps.map { |s| s.fetch("id")}.join(", ")}"
else
puts "Running all steps"
end
require 'tempfile'
Tempfile.open("docker-images-base-test") do |f|
f.write config.to_yaml
f.close
command = if remote
"gcloud container builds submit –config=#{f.path} –substitutions=_IMAGE_DIR=tmp ."
else
"container-builder-local –config=#{f.path} –dryrun=false –substitutions=_IMAGE_DIR=tmp ."
end
puts command
system(command)
exit $?.exitstatus
end

view raw

build.rb

hosted with ❤ by GitHub