So for now, instead of watching the PHP fastcgi process directly, I’m tracking its performance and usage from nginx’s perspective. You can log all kinds of data about upstream performance with nginx:

log_format upstream '$remote_addr - - [$time_local] "$request" $status '
    'upstream $upstream_response_time request $request_time '
    '[for $host via $upstream_addr]';

Then we log to a central upstream.log file from every location block which includes a fastcgi_pass parameter. For example:

location ~ \.php$ {
    include  fastcgi_params;
    access_log  /var/log/nginx/upstream.log  upstream;
    fastcgi_pass  fcgi_php;
    fastcgi_param  SCRIPT_FILENAME  $wordpress_root$fastcgi_script_name;
}

Now we know how many requests the PHP fastcgi process is handling, and how quickly it’s doing so. collectd‘s tail plugin can watch this log file…

<Plugin tail>
  <File "/var/log/nginx/upstream.log">
  Instance "nginx"
    <Match>
      Regex ".*"
      DSType "CounterInc"
      Type counter
      Instance "requests"
    </Match>
    <Match>
      Regex " upstream ([0-9.]*) "
      DSType GaugeAverage
      Type delay
      Instance "upstream"
    </Match>
  </File>
</Plugin>

… and turn it into something readable. First, the number of requests per second (which I only started watching at 14:30 this afternoon), then the delay for each request:

(Relatively boring statistics here, as it’s only monitoring the dynamic processing of my personal sites.)

Combining nginx’s flexible logging and collectd’s tail plugin makes it pretty easy to watch the usage and performance of whatever you’re running behind nginx, even if you can’t instrument the application itself.

… and thus far, I’m pretty happy with the performance, reliability and resource usage of nginx in front of PHP in fastcgi mode.

The problem with Apache processes is that they’re big: they chug a lot of memory, and take a long time to get started. Sure, Apache includes a number of slightly more modern process models than the traditional prefork MPM, unfortunately (and predictably enough), PHP keeps us mired in the land of suckage. So when it comes to memory abuse, I was delighted with the results of putting nginx in front of Apache, and giving it complete responsibility for static files:

Reduced memory usage, increased capacity for OS level disk caching, and of course, consistency. Consistency wins every time. Any sysadmin who has had to react to a sudden change in usage patterns will grok the importance of consistency — after all, it’s not being slashdotted or dugg that will kill you, it’s the hardware and software configuration designed for your everyday load.

That’s all very good for Mr. Server Administrator, but nothing is truly measured until we understand the impact on users. Friday lunch time? Business as usual…

… maaaaybe just a little quicker if they were paying attention. Not that you can tell from this chart.

So, I’m pretty happy with nginx sitting in front of Apache. It totally changes the performance characteristics of the server (and website!) for the better, while allowing me to fall back on somewhat more mature and extensive Apache features when I need to. For instance, nginx’s rewrite module is pretty good, but Apache’s mod_rewrite is truly the original and the best (read: most insane). Surprisingly, nginx doesn’t support old school CGI, but I suppose most of the cool kids using it are opting for better process models such as FastCGI and proxied application servers anyway.

The biggest win in this instance was the conversion of a caching mod_proxy frontend — lot of static files, inefficiently served by Apache — to the new, inbuilt caching module in nginx 0.7. Initially I had tried the apparently popular ncache — must be a lot of people talking about it hypothetically, methinks — but it is probably best described as an “out-of-tree design, configuration and implementation fail”. Once I sucked up the fear-of-trunk and tried the caching support in nginx 0.7, there was no looking back.

(Finally, a quick shout-out for collectd 4.6.2 and the relatively new collection3 web frontend for it… sadly, I switched to it after these charts were saved, so you don’t get to see the new hotness. Maybe you should try it on your server?)