Author Archives: pragmatig

Reading journald kernel logs from inside a kubernetes pod

We wanted a watcher that alerts us when bad kernel things happen and were able to deploy that as a DaemonSet using Kubernetes 🙂

  • Use a Debian base image (for example ruby:2.5-stretch)
  • Run as root user or as user that can read systemd logs like systemd-journal
  • Mount /run/log/journal 
    spec:
  containers:
  - name: foo
    ...
    volumeMounts:
    - name: runlog
      mountPath: /run/log/journal
      readOnly: true
  volumes:
  - name: runlog
    hostPath:
      path: /run/log/journal
  • Use systemd-journal to read the logs 
    require 'systemd/journal'
journal = Systemd::Journal.new
journal.seek(:tail)
journal.move_previous
journal.filter(syslog_identifier: 'kernel')
journal.watch { |entry| puts entry.message }

Running multiple commands in docker in parallel

Went through foreman/goreman/forego and all of them either did not:
– support not printing the name
– support killing all when one finishes
– support sending signals to all children

But this does:

## Install parallel with `done` support
RUN \
  curl -sL http://ftp.gnu.org/gnu/parallel/parallel-20180422.tar.bz2 > /tmp/parallel.tar.bz2 && \
  cd /tmp && tar -xvjf /tmp/parallel.tar.bz2 && cd parallel* && \
  ./configure && make install && rm -rf /tmp/parallel*

# stream output and stop all commands if any of them finish/fail
parallel --no-notice --ungroup --halt 'now,done=1' {1} ::: 'sleep 10' 'sleep 20'

Building single/partial steps from cloudbuild.yaml

We have a single project that builds our GCR/gcloud base-images, it has a lot of reuse between steps so it is nice to not have multiple repos, but locally testing the build became super long since there are so many different images.

Here is a little script to build only a single step and it’s dependencies.

 


remote = ARGV.delete("–remote")
id = ARGV.pop
abort "Usage: ruby build.rb <id|all> [–remote]" unless ARGV.empty?
unless system('which container-builder-local')
abort "Run: gcloud components install container-builder-local"
end
def dependencies(steps, id)
step = steps.detect { |s| s.fetch("id") == id }
[id] + (step["waitFor"] || []).flat_map { |w| dependencies(steps, w) }
end
# make sure to use all variables to avoid:
# "invalid build: key in the substitution data is not matched in the template"
def echo_step
{
"id" => "echo",
"name" => "gcr.io/cloud-builders/wget",
"entrypoint" => "echo",
"args" => ["_IMAGE_DIR is $_IMAGE_DIR"]
}
end
require 'yaml'
config = YAML.load_file("cloudbuild.yaml")
config.delete("images")
unless id == "all"
steps = config.fetch("steps")
keep = dependencies(steps, id)
steps.select! { |s| keep.include? s.fetch("id") }
steps.unshift(echo_step)
puts "Running steps #{steps.map { |s| s.fetch("id")}.join(", ")}"
else
puts "Running all steps"
end
require 'tempfile'
Tempfile.open("docker-images-base-test") do |f|
f.write config.to_yaml
f.close
command = if remote
"gcloud container builds submit –config=#{f.path} –substitutions=_IMAGE_DIR=tmp ."
else
"container-builder-local –config=#{f.path} –dryrun=false –substitutions=_IMAGE_DIR=tmp ."
end
puts command
system(command)
exit $?.exitstatus
end

view raw

build.rb

hosted with ❤ by GitHub

Chef install google-cloud-sdk without package manager

A tiny chef snipped to install gcloud without using a package manager (to get the latest version without waiting)

gcloud_version = node["foo"]["google-cloud-sdk_version"]
gcloud_file = "google-cloud-sdk-#{gcloud_version}-linux-x86_64.tar.gz"
gcloud_folder = "https://dl.google.com/dl/cloudsdk/channels/rapid/downloads"
gcloud_installer = "https://dl.google.com/dl/cloudsdk/channels/rapid/install_google_cloud_sdk.bash"
execute "gcloud_install" do
  # clean up ... download installer but select the version we want ... install ... link
  command "rm -rf $CLOUDSDK_INSTALL_DIR/google-cloud-sdk && curl #{gcloud_installer} | sed 's;__SDK_URL_DIR=.*;__SDK_URL_DIR=#{gcloud_folder};' | sed 's/__SDK_TGZ=.*/__SDK_TGZ=#{gcloud_file}/' | bash && ln -sf $CLOUDSDK_INSTALL_DIR/google-cloud-sdk/bin/gcloud /usr/local/bin/gcloud"
  env(
    "CLOUDSDK_CORE_DISABLE_PROMPTS" => "true",
    "CLOUDSDK_INSTALL_DIR" => "/opt", # prefix
  )
  not_if { `true && gcloud -v`.include?(gcloud_version) } # ~FC048
end

fast npm install check to ensure it is up to date

Ensures that everyone has npm up to date without running “npm install”
Ideally this should be wrapped as “npm check” command, but we use a ruby/rake based workflow anyway.

  desc 'make sure npm is installed'
  task :ensure_npm do
    expected = JSON.parse(File.read('package-lock.json')).fetch('dependencies')
    satisfied = expected.all? do |name, data|
      expected_version = data.fetch('version')
      pack = "node_modules/#{name}/package.json"
      next unless File.exist?(pack)
      resolved = JSON.parse(File.read(pack))
      resolved.fetch('version') == expected_version || # regular
        resolved.fetch('_resolved') == expected_version # git
    end
    sh "npm install" unless satisfied
  end