We recently configured God to monitor our Sphinx search daemon. The application heavily depends on it and it’s the main reason the app is down from time to time. The problem is that God tries to restart Sphinx when it’s rebuilding the indexes (and thus not running). Let’s look at how Sphinx and God can play nicely together.
The idea is to temporarily unmonitor the Sphinx process while it’s rebuilding the indexes.
Sphinx doesn’t do live index updates like some other search solutions (eg. Sunspot), so you’ll have to do it yourself. One way is to run a crontab which will rebuild the indexes each x hours. Make sure to use the thinking_sphinx:reindex rake task and not the rebuild one as the last one will stop the Sphinx search daemon which will wake up God which, in its turn, will try to start the daemon again, messy.
There is another way to update your Sphinx indexes, delayed delta’s, but in the scope of this post, it doesn’t really change a thing.
As soon as the Sphinx search process stops, God will try to restart it. This is what we want but sometimes Sphinx needs to rebuild its indexes bringing down the process. We choose to rebuild the indexes during deploy to make sure we load the new configuration if it has been changed. Thinking Sphinx comes with some Capistrano tasks out-of-the-box so let’s use them. Add the following to your config/deploy.rb:
The sphinx param in the God commands is the same as the one you choose to define the God watch.
Unmonitoring the Sphinx process is not without risk. What if something goes wrong during the reindexing? We think it’s not that big of a risk as we will probably notice this during deployment but it’s important to know.
This post is open source. Did you spot a mistake? Ideas for improvements? Contribute to this post via Github. Thank you!