Printing readable dates from the Nagios log

Just because I always forget...
ben@nagios1:/var/log/nagios3$ grep 'mycheck-or-host' nagios.log |perl -pe 's/(\d+)/localtime($1)/e'
The perl snippet will translate the epoch-style timestamp to one that is readable.

git lesson of the day (courtesy of Brad)

git rebase, reflog, and force push - the nice way to deal with your own branches and keep a clean git history.

Warning: don't do this on branches that other people are working on - the force push does nasty things. In that case, it's better to coordinate, collect changes, make a new branch, whatever.

So you've been working on a branch (let's call it ben.foo) and you're ready to squash a bunch of commits and push it back up to master.
* bring your checkout up to date by getting the heads of all branches

git fetch

* set your upstream to master

git branch --set-upstream ben.foo origin/master

* rebase all your changes that are different from master
** this brings your branch up to date and puts all your branch's commits after the most recent commit to master

git rebase

* squash all your commits
** this automatically chooses only the commits that don't exist in master

git rebase -i

* push them up, forcing your branch to remove your previous more verbose commits
** the + sign before the branch name means force push only this branch - much safer than --force, which forces all branches

git push origin +ben.foo

Whee!! You're done!

Let's do this another way.

We don't really care about master for most of this process - we're really just interested in squashing our own commits, then we can merge or do a PR or whatever like normal.

* Look for the commit before the place we started our branch. I'll assume it's visually easy to find - often the last commit by someone else before a long string of my own commits. Note the commit ID of the not-my-branch commit:

git log

* Do an interactive git rebase from that commit on

git rebase -i abcd1234

* squash all but the first commit (or however you want to collapse your commit history)

2Gv}:s/^pick/squash/<ret>:wq

* push this up, using the same +branch force in the previous example, overwriting the remote repository's messy history

git push origin +ben.foo

Voila! Your commit history is now clean

If you screw up any of these steps...

git reflog to the rescue.

git reflog ben.foo

This is a complete history of the changes you've made to the branch. Choose the commit ID you want to backtrack to (eg abcd1234) and run

git reset --hard abcd1234

Further Reading:
* Two good posts about git reflog
* Many thanks to Brad Kittenbrink for going over this with me.

Using attributes that might not exist in Chef

In chef, when testing a nested attribute that might not exist, you can use rescue as a modifier to an if statement. For example, assuming that only some of your nodes have node['foo']['bar']['baz'] defined:
 
if node['foo']['bar']['baz']
   do the stuff I want
end rescue NoMethodError

will avoid the chef run bailing with the dreaded "undefined method `[]' for nil:NilClass" error. Of course, if there are any legit method errors within your if block, you've just caught an exception you shouldn't, but that's another story.

I use this in cookbooks that have extra functionality if some other cookbook happens to be installed. For example, the apache cookbook doesn't depend on ganglia, but if ganglia's installed, I want it to report metrics.

if node[:ganglia][:grid_name]
  include_recipe "ganglia::default"
  ganglia_python "apache" do
    action :enable
  end
end rescue NoMethodError
If the node doesn't have ganglia isntalled, it just goes on its merry way. #win

Capistrano, nohup, and sleep

So a bunch of the random things that I deploy are not well-behaved services with init scripts and reload targets but random bits like a sinatra app or a hipchat bot. Being internal tools that are pretty stable, the easiest way to deploy these is to just nohup and background the command rather than setting everything up under monit or runit or something like that.

You'll see a number of posts talking about capistrano hanging when trying to use nohup (stackoverflow) but redirecting both stdout and stderr to /dev/null solves that one.

The problem I ran across is that the nohup would not actually succeed in launching the process. With my new favorite diagnostic tool watch -n 0.1 --differences 'ps auwwxf | grep -A3 -B3 ubuntu (aka refresh all the processes running as the ubuntu user every 1/10th second), I saw that the nohup appear and disappear but the process it was trying to launch never stuck around.

I believe what's actually happening is that capistrano is closing the ssh connection it used to launch the nohup process before nohup succeeds in detaching the process. Closing the ssh shell than kills the nohup process before it finishes detaching its child, so the child dies too.

The solution is simple; give nohup a little more time to finish detaching its child before closing the ssh connection by appnding sleep 2 to the line: