Automated Sudo Password Prompt with SshKit

Basically what sshkit-sudo gem promises, but:

  • 1 hack instead of multiple layers
  • obvious how to debug
  • 1 global password
  • does not capture the password prompt
  • does not print the output when capturing
  • works when not using SshKit::DSL

Hint: You might want to start with an extra “puts data” to see how your password prompt looks like.

SSHKit::Command.prepend( do
  def on_stdout(channel, data)
    if data.include? "[sudo] password for "
      @@password ||= `echo password: 1>&2 && read -s PASSWORD && printf \"$PASSWORD\"`
      channel.send_data(@@password + "\n")

on servers do
  capture :sudo, "ls"

Datadog: Show Brittle Monitors

With the help of datadogs unofficial search_events endpoint we can see which of our monitors fail the most, a great place to start when trying to reduce alert spam.
(using Kennel)

rake brittle TAG=team:foo
analyzing 104 monitors ... 10.9s
Foo too high🔒
Frequency: 18.95/h
success: 56x
warning: 44x


desc "Show how brittle selected teams monitors are TAG="
task brittle: "kennel:environment" do
  monitors = Kennel.send(:api).list("monitor", with_downtimes: false, monitor_tags: [ENV.fetch("TAG")])
  abort "No monitors found" if monitors.empty?

  hour = 60 * 60
  interval = 7 * 24 * hour
  now =
  max = 100

  data = Kennel::Progress.progress "analyzing #{monitors.size} monitors" do
    Kennel::Utils.parallel monitors do |monitor|
      events = Kennel.send(:api).list("monitor/#{monitor[:id]}/search_events", from_ts: now - interval, to_ts: now, count: max, start: 0)
      next if events.empty?

      duration = now - (events.last.fetch(:date_detected) / 1000)
      amount = events.size
      frequency = amount * (hour / duration.to_f)
      [monitor, frequency, events]

  # spammy first
  data.sort_by! { |_, frequency, _| -frequency }

  data.each do |m, frequency, events|
    groups = events.group_by { |e| e.fetch(:alert_type) }
    groups.sort_by(&:first) # sort by alert_type

    puts m.fetch(:name)
    puts "{m.fetch(:id)}"
    puts "Frequency: #{frequency.round(2)}/h"
    groups.each do |type, grouped_events|
      puts "#{type}: #{grouped_events.size}x"

Ruby Capture Stdout Without $stdout

Rake’s sh for example cannot be captured with the normal $stdount = trick, but using some old ActiveSupport code that uses reopen with a Temfile seems to do the trick.

# StringIO does not work with rubies `system` call that `sh` uses under the hood, so using Tempfile + reopen
def capture_stdout
  old_stdout = STDOUT.dup'tee_stdout') do |capture|
    STDOUT.sync = true

Listing Unmuted Datadog Alerts

Datadogs UI for alerting monitors also shows muted ones without an option to filter, which leads to overhead / confusion when trying to track down what exactly is down.

So we added an alerts task to  kennel  that lists all unmuted alerts and since when they are alerting, it also shows alerts that have no-data warnings even though Datadog UI shows them as not alerting.

bundle exec rake kennel:alerts TAG=team:my-team
Downloading ... 5.36s
Foo certs will expire soon🔒
Ignored cluster:pod1,server: 39:22:00
Ignored cluster:pod12,server: 31:41:00
Ignored cluster:pod12,server: 31:41:00

Foobar Errors (Retry Limit Exceeded)🔒
Alert cluster:pod2 19:05:16