Category Archives: Server

Add PATH= to beginning of a users crontab

Puts the correct PATH into a users crontab, so you no longer have to use absolute paths inside cron.
We executed it on all servers via puppet / console.

sudo su root
echo 'PATH=/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/bin:/sbin' > /tmp/cron
crontab -l -u deploy  | grep -v PATH= >> /tmp/cron
cat /tmp/cron | crontab - -u deploy

Installing MongoDb on Ubuntu Hardy from source with init.d and data dir

After setting up mongo by hand for 2 servers that are not deployed via puppet, here is a bash script to do it…


sudo su root

cd /opt/ && /usr/bin/curl -OL http://downloads.mongodb.org/linux/mongodb-linux-x86_64-1.4.1.tgz && tar -zxvf mongodb-linux-x86_64-1.4.1.tgz && ln -s /opt/mongodb-linux-x86_64-1.4.1 /opt/mongodb

useradd mongod -u 5005

curl http://gist.github.com/raw/413433/834a3d0ecc96cb36f8918f6d01b65fc082025cc0/gistfile1.sh > /etc/init.d/mongodb

chown mongod:mongod /etc/init.d/mongodb
chmod 750 /etc/init.d/mongodb

mkdir /etc/mongodb
chown mongod:mongod /etc/mongodb
chmod 750 /etc/mongodb

echo "dbpath=/opt/mongodb-data/" > /etc/mongodb/mongodb.conf
chown mongod:mongod /etc/mongodb/mongodb.conf
chmod 750 /etc/mongodb/mongodb.conf

mkdir /opt/mongodb-data
chown mongod:mongod /opt/mongodb-data
chmod 750 /opt/mongodb-data

one-liner to install nginx with passenger and ssl

Just build this little guy for our puppet script, maybe someone needs it too 🙂

export V=0.7.67 && wget -O /tmp/nginx-$V.tar.gz http://sysoev.ru/nginx/nginx-$V.tar.gz && cd /tmp && tar xzvf nginx-$V.tar.gz && sudo passenger-install-nginx-module --nginx-source-dir /tmp/nginx-$V --extra-configure-flags="--with-http_ssl_module" --auto --prefix=/opt/nginx-$V && rm /opt/nginx && sudo ln -s  /opt/nginx-$V /opt/nginx

(If you like the stock nginx without ssl, youll need the –auto-download option)

Sinatra on Nginx on Ubuntu on Passenger

Just wanted to share our setup, as reminder and help for others 😉

(do not forget to create a tmp and public folder inside your projects folder)

#/opt/nginx/conf/nginx.conf
server {
  listen 80;
  server_name xxx.yyy.com;

  access_log /var/log/xxx_access.log  main;
  error_log /var/log/xxx_error.log debug;
  root /srv/xxx/public;   # <--- be sure to point to 'public'!
  passenger_use_global_queue on;
}

#/etc/hosts
127.0.0.1       xxx.yyy.com

#config.ru
require 'app'
disable :run
set :root, Pathname(__FILE__).dirname
run Sinatra::Application

#app.rb
require 'rubygems'
require 'sinatra'

get "/" do
  "Hello world from xxx"
end

Fixing Varnish restarts after ESI backend dies or timeouts

Varnish died in peak hours, so we investigated and found out that varnish died only when the esi backend died. We reproduced this scenario by hitting our test-server, that had a simple sleep-10 ttl-0 esi action, with ab.

Error
And look what we found in /var/log/syslog …

Child (10211) Panic message: Missing errorhandling code in HSH_Prepare(), cache_hash.c line 188: Condition((p) != 0) not true. thread = (cache-worker)sp = 0x7fed9e312008 { fd = 14, id = 14, xid = 177614113, client = 127.0.0.1:36835, step = STP_LOOKUP, handling = hash, ws = 0x7fed9e312078 { overflow id = “sess”, {s,f,r,e} = {0x7fed9e312808,,+16364,(nil),+16384}, }, worker = 0x47002bc0 { }, vcl = { srcname = { “input”, “Default”, }, }, },

Searching aroud the interwebs suggested that session workspace should be increased, which helped a bit. Decreasing esi timeout (first_byte_timeout, not connection_timeout) and increasing the threads(min or pool, just getting more) also improved the situation.

Semi-Solution

# vcl
backend esi_backend {
  ...
  .first_byte_timeout = 3s;
}

# startup params
-p sess_workspace=524288
-p thread_pools=8 # should be number of cpus
-p thread_pool_min=500

This is for 2.0.4, there are some esi changes/fixes in trunk, so you might want to retest before simply using these settings in a newer version (e.g. sess_workspace is pretty high and will use more ram than necessary) (do not use 2.0.5 with esi)

Solution
Using SSI infront of Varnish, we could keep almost all configuration the same and instantly every problem was solved!
(we could also remove some unnecessary logic from Varnishs VCL, that normally handled adding/removing cookies after/before ESI)