Reading journald kernel logs from inside a kubernetes pod

We wanted a watcher that alerts us when bad kernel things happen and were able to deploy that as a DaemonSet using Kubernetes 🙂

  • Use a Debian base image (for example ruby:2.5-stretch)
  • Run as root user or as user that can read systemd logs like systemd-journal
  • Mount /run/log/journal
    spec:
      containers:
      - name: foo
        ...
        volumeMounts:
        - name: runlog
          mountPath: /run/log/journal
          readOnly: true
      volumes:
      - name: runlog
        hostPath:
          path: /run/log/journal
  • Use systemd-journal to read the logs
    require 'systemd/journal'
    journal = Systemd::Journal.new
    journal.seek(:tail)
    journal.move_previous
    journal.filter(syslog_identifier: 'kernel')
    journal.watch { |entry| puts entry.message }

Running multiple commands in docker in parallel

Went through foreman/goreman/forego and all of them either did not:
– support not printing the name
– support killing all when one finishes
– support sending signals to all children

But this does:

## Install parallel with `done` support
RUN \
  curl -sL http://ftp.gnu.org/gnu/parallel/parallel-20180422.tar.bz2 > /tmp/parallel.tar.bz2 && \
  cd /tmp && tar -xvjf /tmp/parallel.tar.bz2 && cd parallel* && \
  ./configure && make install && rm -rf /tmp/parallel*

# stream output and stop all commands if any of them finish/fail
parallel --no-notice --ungroup --halt 'now,done=1' {1} ::: 'sleep 10' 'sleep 20'

Bundler / Docker / alpine for a super small container

Took me a while to figure out how to get this running without installing ruby-dev or tools to compile native extensions, keeping my container nice and small.

Updated: using builtin packages for bundler + io-console now since they are smaller then installing manually and the logic is simpler
(apk add –update ruby ruby-io-console ruby-bundler)
… if latest bundler is needed below Dockerfile might still be useful.

FROM alpine

RUN apk add --update ruby && rm -rf /var/cache/apk/*

ENV BUNDLER_VERSION 1.12.3
RUN gem install bundler -v $BUNDLER_VERSION --no-ri --no-rdoc

# bundler wants some library that needs core extensions ... but it won't compile
RUN mkdir /usr/lib/ruby/gems/2.2.0/gems/bundler-$BUNDLER_VERSION/lib/io
RUN touch /usr/lib/ruby/gems/2.2.0/gems/bundler-$BUNDLER_VERSION/lib/io/console.rb

# bundler does not want to install as root
RUN bundle config --global silence_root_warning 1

RUN mkdir /app
WORKDIR /app

ADD Gemfile .
ADD Gemfile.lock .
RUN bundle