Ruby on Kubernetes: Memory GC OOMKilled

Ruby seems to always grow and then hit the memory limit … which triggers a SIGKILL … which means shutdown without cleanup.
This snippet helps to keep memory low and softly kills the process when memory gets to close to the limit.

# run GC periodically to reduce memory and report to airbrake if we run out (instead of kubernetes silently failing)
Thread.new do
  loop do
    sleep 60
    GC.start
    used = Integer(File.read('/sys/fs/cgroup/memory/memory.usage_in_bytes')) / 1024 / 1024
    max = Integer(`cat /sys/fs/cgroup/memory/memory.stat | grep hierarchical_memory_limit`.split.last) / 1024 / 1024
    puts "Ram: #{used}M / #{max}M"
    raise "Out of memory #{used}/#{max}" if used + 5 >= max
  end
end

A fun little tool that might help too: preoomkiller

Advertisements

One thought on “Ruby on Kubernetes: Memory GC OOMKilled

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s