We recently configured God to monitor our Sphinx search daemon. The application heavily depends on it and it’s the main reason the app is down from time to time. The problem is that God tries to restart Sphinx when it’s rebuilding the indexes (and thus not running). Let’s look at how Sphinx and God can play nicely together.

The idea is to temporarily unmonitor the Sphinx process while it’s rebuilding the indexes.

Sphinx configuration

Sphinx doesn’t do live index updates like some other search solutions (eg. Sunspot), so you’ll have to do it yourself. One way is to run a crontab which will rebuild the indexes each x hours. Make sure to use the thinking_sphinx:reindex rake task and not the rebuild one as the last one will stop the Sphinx search daemon which will wake up God which, in its turn, will try to start the daemon again, messy.

There is another way to update your Sphinx indexes, delayed delta’s, but in the scope of this post, it doesn’t really change a thing.

God configuration

As soon as the Sphinx search process stops, God will try to restart it. This is what we want but sometimes Sphinx needs to rebuild its indexes bringing down the process. We choose to rebuild the indexes during deploy to make sure we load the new configuration if it has been changed. Thinking Sphinx comes with some Capistrano tasks out-of-the-box so let’s use them. Add the following to your config/deploy.rb:

require 'thinking_sphinx/deploy/capistrano'

desc "Unmonitor and stop Sphinx"
task :before_update_code do
  run "god unmonitor sphinx"
  thinking_sphinx.stop
end

desc "Rebuild the Sphinx indexes and monitor"
task :after_update_code do
  thinking_sphinx.configure
  thinking_sphinx.index
  thinking_sphinx.start
  run "god monitor sphinx"
end

The sphinx param in the God commands is the same as the one you choose to define the God watch.

Caveat

Unmonitoring the Sphinx process is not without risk. What if something goes wrong during the reindexing? We think it’s not that big of a risk as we will probably notice this during deployment but it’s important to know.