Staggering cron to distribute maintenance tasks

Whilst performance tuning a Drupal site recently, I read an article detailing reasons why cron is bad; How Drupal's cron is killing you in your sleep. Running cron hourly was purportedly killing our site's cache which would have the effect of slowing down browsing for users.

This is bad.

The page touts intelligent cron modules such as elysia cron or ultimate cron as a solution to ensure that the tasks that need to run very often do so without simultaneously running the tasks that require more grunt, wipe the cache, cause high loads and kill your site.

After whiteboarding all of the implementations of hook_cron (These functions run when cron does), I ordered them in order of importance and set about writing the cron script to place into the elysia cron settings. I also wrote an external script to call elysia's cron.php every 5 minutes instead of using the automatic cron provided with drupal 7, as the minimum frequency of calling it was once per hour. To ensure we had an active site responsive to reader feedback I decided to run it a little more frequently than default.

Doesn't running more frequently have negative effects?

Elysia cron runs intelligently. For each implementation of hook_cron a threshold may be set. If cron fires every minute and the threshold for system_cron is once a week then nothing will happen and no sites die!

The more important processes to run often are those which provide the user with action based feedback. If a user is directed to a post and sees it rise in the 'Popular Content' section of the site or notices hot topics posts fluctuating with new content posted regularly they will be more inclined to stay and return to this obviously active site.

Running cron this frequently on a normal site would have some slow down implications but elysia cron allows the administrative user to manage them. Taking slow background maintenance tasks and running them less frequently. It's not really too much of a problem to clear old log records from watchdog once a day/week. Checking for module updates too, can be left to a daily process.

So everything works now?

Once I had arranged the modules into groups to execute on the quarter of the hour, every six hours, every day and every week (depending on their importance) I ran into another issue. Every day/week there'll be a time, when all of the cron hooks run causing a momentary, but noticeable drop in performance.

This was brought to my attention by newrelic, a performance management tool I've used a few times to ensure I'm not writing inefficient code and to track where Drupal is slowing down sites.

I could see these regular, albeit infrequent blips in the performance graph accompanied by a reduction in apdex and a report showing cron.php to be the biggest hog of memory, time and processing power. Seeing these regular peaks made my mind jump instantly back to college math. Rather than endure these peaks, could we somehow spread the load out more to get more frequent, yet smaller peaks? It was my idea that this would result in an overall lower average apdex and a more responsive site overall.

The short story is, it worked.

The longer story, with pictures saying a thousand words shows lower overall site loads. The peaks are indeed more frequent but at the same time is on average lower with average load times being under 400ms; good by any website's standard!

An extract of some of my elysia cron timings is included below and available in an attachment for download. I'd be interested to see if people think my settings are too lean and stringent or if I could cut back even further. I'm aware of how important cron is for a healthy Drupal site, but perhaps some implementations of hook_cron are more equal than others.

aggregator_cron 10 3,9,15,21 * * *
apachesolr_cron 0 */6 * * *
captcha_cron 4 1 * * *
morning_news_cron 00 07-09 * * *
dblog_cron 50 4 1 * *
field_cron 40 0 * * 2
node_cron 4 2 * * *
poll_cron 4 3 * * *
radioactivity_cron */15 * * * *
redirect_cron 40 0 * *
rules_cron 30 * * * *
scheduler_cron 40 * * * *
system_cron 10 0 * * 6
update_cron 4 0 * * 0