Tag Archives: Ruby

Automated Sudo Password Prompt with SshKit

Basically what sshkit-sudo gem promises, but:

  • 1 hack instead of multiple layers
  • obvious how to debug
  • 1 global password
  • does not capture the password prompt
  • does not print the output when capturing
  • works when not using SshKit::DSL

Hint: You might want to start with an extra “puts data” to see how your password prompt looks like.

SSHKit::Command.prepend(Module.new do
  def on_stdout(channel, data)
    if data.include? "[sudo] password for "
      @@password ||= `echo password: 1>&2 && read -s PASSWORD && printf \"$PASSWORD\"`
      channel.send_data(@@password + "\n")
    else
      super
    end
  end
end)

on servers do
  capture :sudo, "ls"
end

Datadog: Show Brittle Monitors

With the help of datadogs unofficial search_events endpoint we can see which of our monitors fail the most, a great place to start when trying to reduce alert spam.
(using Kennel)

rake brittle TAG=team:foo
analyzing 104 monitors ... 10.9s
Foo too high🔒
https://app.datadoghq.com/monitors/12345
Frequency: 18.95/h
success: 56x
warning: 44x

 

desc "Show how brittle selected teams monitors are TAG="
task brittle: "kennel:environment" do
  monitors = Kennel.send(:api).list("monitor", with_downtimes: false, monitor_tags: [ENV.fetch("TAG")])
  abort "No monitors found" if monitors.empty?

  hour = 60 * 60
  interval = 7 * 24 * hour
  now = Time.now.to_i
  max = 100

  data = Kennel::Progress.progress "analyzing #{monitors.size} monitors" do
    Kennel::Utils.parallel monitors do |monitor|
      events = Kennel.send(:api).list("monitor/#{monitor[:id]}/search_events", from_ts: now - interval, to_ts: now, count: max, start: 0)
      next if events.empty?

      duration = now - (events.last.fetch(:date_detected) / 1000)
      amount = events.size
      frequency = amount * (hour / duration.to_f)
      [monitor, frequency, events]
    end.compact
  end

  # spammy first
  data.sort_by! { |_, frequency, _| -frequency }

  data.each do |m, frequency, events|
    groups = events.group_by { |e| e.fetch(:alert_type) }
    groups.sort_by(&:first) # sort by alert_type

    puts m.fetch(:name)
    puts "https://zendesk.datadoghq.com/monitors/#{m.fetch(:id)}"
    puts "Frequency: #{frequency.round(2)}/h"
    groups.each do |type, grouped_events|
      puts "#{type}: #{grouped_events.size}x"
    end
    puts
  end
end

Ruby Capture Stdout Without $stdout

Rake’s sh for example cannot be captured with the normal $stdount = StringIO.new trick, but using some old ActiveSupport code that uses reopen with a Temfile seems to do the trick.

# StringIO does not work with rubies `system` call that `sh` uses under the hood, so using Tempfile + reopen
# https://grosser.it/2018/11/23/ruby-capture-stdout-without-stdout
def capture_stdout
  old_stdout = STDOUT.dup
  Tempfile.open('tee_stdout') do |capture|
    STDOUT.reopen(capture)
    STDOUT.sync = true
    yield
    capture.rewind
    capture.read
  end
ensure
  STDOUT.reopen(old_stdout)
end

Listing Unmuted Datadog Alerts

Datadogs UI for alerting monitors also shows muted ones without an option to filter, which leads to overhead / confusion when trying to track down what exactly is down.

So we added an alerts task to  kennel  that lists all unmuted alerts and since when they are alerting, it also shows alerts that have no-data warnings even though Datadog UI shows them as not alerting.

bundle exec rake kennel:alerts TAG=team:my-team
Downloading ... 5.36s
Foo certs will expire soon🔒
https://app.datadoghq.com/monitors/123
Ignored cluster:pod1,server:10.215.225.122 39:22:00
Ignored cluster:pod12,server:10.218.176.123 31:41:00
Ignored cluster:pod12,server:10.218.176.123 31:41:00


Foobar Errors (Retry Limit Exceeded)🔒
https://app.datadoghq.com/monitors/1234
Alert cluster:pod2 19:05:16