So a bunch of the random things that I deploy are not well-behaved services with init scripts and reload targets but random bits like a sinatra app or a hipchat bot. Being internal tools that are pretty stable, the easiest way to deploy these is to just nohup and background the command rather than setting everything up under monit or runit or something like that.
You'll see a number of posts talking about capistrano hanging when trying to use nohup (stackoverflow) but redirecting both stdout and stderr to /dev/null solves that one.
The problem I ran across is that the nohup would not actually succeed in launching the process. With my new favorite diagnostic tool watch -n 0.1 --differences 'ps auwwxf | grep -A3 -B3 ubuntu
(aka refresh all the processes running as the ubuntu user every 1/10th second), I saw that the nohup appear and disappear but the process it was trying to launch never stuck around.
I believe what's actually happening is that capistrano is closing the ssh connection it used to launch the nohup process before nohup succeeds in detaching the process. Closing the ssh shell than kills the nohup process before it finishes detaching its child, so the child dies too.
The solution is simple; give nohup a little more time to finish detaching its child before closing the ssh connection by appnding sleep 2
to the line: