God Gotchas

This is a reblog of a post from my new Ruby-themed tumblelog. I know it’s kind of cheap to repost your own stuff, but who cares.

I’ve spent a couple of hours today pulling my hair out while trying to get one of my background job scripts to work with god, the “easy to configure, easy to extend monitoring framework written in Ruby”.

Well, it took a while to make it work, tho. Yes, god is cool and simple and gets the job done. But there are some things that cost me hours and which I found out only by reading the source code.

So, I just want to quickly jot down some gotchas before I forget them again, running the risk of falling into the same traps again in the future.

My script used the constant LOG to keep my Logger instance. Logging worked fine when I ran the script by itself, yet when god took over, it didn’t anymore. Actually, the script died rather quickly. As there was no logging at all going on, and all STDOUT output was suppressed, I came rather close to losing it. Turns out god itself is declaring a LOG constant of its own, which was done before my script had the chance, so when it attempted to initialize it, it would actually try to re-declare an existing constant, and we all know how well that works. ;)
Once that was done, my script was logging just fine, but it didn’t produce any output whatsoever. Raah! Teeth were gnashed… there was definitely teeth gnashing going on. Even telling “my” Logger (the one inside my script) to write to a file didn’t produce anything. That was a fun hour, really. The reason for this behaviour: god is closing all open file descriptors when it sets up monitoring a script. Which included my script. Awesome! On the upside, it meant I could get rid of the part of my code dealing with different logger behaviours. Meaning less LOC! It doesn’t get any more agile than that, folks.
In case you actually want to capture anything your original script is sending to STDOUT, there’s the not-really-documented God::Watch#log. Set it inside your God.watch block to specify a log file. (See example below.)
If you need to set ENV variables, God::Watch#env is your friend. Accepts a hash with arbitrary key/value pairs. For example, I declare a few God.watch blocks, one for each value of an array (think “worker 1 to 5”), and I use God::Watch#env to pass the current value to the worker script. Works well.
When you do a sudo god stop <watch>, make sure to give it a few moments before running sudo god start <watch> again, or you might end up with orphaned unmonitored scripts running rampant in the background.
Running sudo god log without any further arguments will tell you that “You must specify a Task or Group name”. That’s actually a lie, as it only accepts task names. (A group is a number of related tasks. A task is a single monitoring watch.)

So, yeah.

Don’t get me wrong, please: It might not look like it, but once I had figured it all out, I’ve decided I actually like god. I like the feature set, it’s really easy to set up, and it works. I’m happy it exists.