Replacing Apache with nginx for static file serving

This chart represents the regular activity of an Apache server from Monday to Thursday — a massive spike of non-idle processes starting around lunch time — and then an eerily quiet Friday. No, it wasn’t a public holiday, it’s just evidence of win. Notice the gaps late on the 2nd and early on the 3rd? Happy infrastructure hacking in the witching hours!

nginx vs. apache: open/sending

The problem with Apache processes is that they’re big: they chug a lot of memory, and take a long time to get started. Sure, Apache includes a number of slightly more modern process models than the traditional prefork MPM, unfortunately (and predictably enough), PHP keeps us mired in the land of suckage. So when it comes to memory abuse, I was delighted with the results of putting nginx in front of Apache, and giving it complete responsibility for static files:

nginx vs. apache: memory

Reduced memory usage, increased capacity for OS level disk caching, and of course, consistency. Consistency wins every time. Any sysadmin who has had to react to a sudden change in usage patterns will grok the importance of consistency — after all, it’s not being slashdotted or dugg that will kill you, it’s the hardware and software configuration designed for your everyday load.

That’s all very good for Mr. Server Administrator, but nothing is truly measured until we understand the impact on users. Friday lunch time? Business as usual…

nginx vs. apache: traffic

… maaaaybe just a little quicker if they were paying attention. Not that you can tell from this chart. ;-)

So, I’m pretty happy with nginx sitting in front of Apache. It totally changes the performance characteristics of the server (and website!) for the better, while allowing me to fall back on somewhat more mature and extensive Apache features when I need to. For instance, nginx’s rewrite module is pretty good, but Apache’s mod_rewrite is truly the original and the best (read: most insane). Surprisingly, nginx doesn’t support old school CGI, but I suppose most of the cool kids using it are opting for better process models such as FastCGI and proxied application servers anyway.

The biggest win in this instance was the conversion of a caching mod_proxy frontend — lot of static files, inefficiently served by Apache — to the new, inbuilt caching module in nginx 0.7. Initially I had tried the apparently popular ncache — must be a lot of people talking about it hypothetically, methinks — but it is probably best described as an “out-of-tree design, configuration and implementation fail”. Once I sucked up the fear-of-trunk and tried the caching support in nginx 0.7, there was no looking back.

(Finally, a quick shout-out for collectd 4.6.2 and the relatively new collection3 web frontend for it… sadly, I switched to it after these charts were saved, so you don’t get to see the new hotness. Maybe you should try it on your server?)

This entry was posted in General and tagged , , , , , . Bookmark the permalink.

24 Responses to Replacing Apache with nginx for static file serving

  1. pirast says:

    btw. i am using php + apache2 worker w/o problems (fastcgi)

  2. What… you administer the crikey web servers now? Cool gig

  3. Jeff Waugh says:

    @pirast: Yeah, you can do so with fastcgi… but then you’re still using great big Apache processes to do the kind of work nginx can do with very little overhead.

    @Chris Sherlock: Yeah, I built their blogs site, am building their new main site, and managing the new servers they’re on. It’s fun! :-)

  4. Jordi says:

    Jeff, as I already suggested in Face^W^Welsewhere, depending on what rules you’re depending on, switching to PHP5 over fcgi using nginx rewrites entirely is not only feasible, but your charts will look even better.

    CGI? If you’re running Mailman and that’s one of the bits that tie you to Apache, not anymore! We’re using a cgiwrapper that nginx can launch to do the Mailman stuff. The wrapper is not uploaded to Debian because it lacks a copyright notice but just ping me for more info (and packages).

  5. Nick J says:

    Thank you, very interesting post. I hadn’t heard of collectd previously, installed it yesterday, it’s very nice. Still playing with it, but unfortunately the version in Ubuntu is a bit old (v4.4.1 in 8.10, and 9.04 currently only has 4.4.2), and has a bug that prevents UPS graphing via NUT from working on AMD64 until 9.04 is released and I upgrade ( https://bugs.launchpad.net/ubuntu/+source/collectd/+bug/238369 ).

    It’d also be nice if there was a graph of uptime (boring straight-line graph I know, but still useful perhaps), and some kind of “best-of” dashboard where you selected the most useful graphs and it put them all on one page could be handy too. Anyways, thanks for the heads up, it’s a much nicer way to look at this stuff.

    Oh and one other thing I’m not sure that I get: in your example “bits per second” graph, isn’t more bits per second transmitted better? … which would seem to imply there was more outgoing traffic before, which sounds like a bad thing, but I’m pretty I must be mis-interpreting the graph somewhere (e.g. is it “bits per second from apache only”? or is it mislabeled by collectd and it’s instead a client-side graph of delay between requesting data and receiving a response? or have I just got my wires crossed?)

  6. Jeff Waugh says:

    @Jordi: Yeah, definitely tempting… but will look at it after the current major change has settled in. When it comes to CGI, there’s a whole bunch of little bits and pieces, including the collectd graphs. ;-) Is your cgiwrapper basically a blob of PHP which handles the CGI script?

  7. Jeff Waugh says:

    @Nick J: The collection3 stuff (ready to use despite being in the examples directory in the package) has a nice little front page summary of (all) the charts… but I’m still using a simple HTML page which shows a bunch of the most relevant chart images.

    The interface chart is based on data from the actual network interface rather than Apache or nginx. In this case, it just demonstrates that the traffic pattern is “business as usual” despite the Apache pattern changing completely.

    Given that we’re pushing a similar amount of bits on Day 0 to what we were pushing on Day -1, everything’s fine. :-)

  8. Jeff Waugh says:

    @Nick J: btw, I built an nginx 0.7.47 package, based on the experimental packaging of 0.6.x in Debian sid.

  9. Ben Lisle says:

    Long time reader first time poster…

    Nice graphs! Cricket was getting a bit long in the tooth eh? :)

  10. Jeff Waugh says:

    @Ben Lisle: Yo! How’ve you been? Still in Canberra? I’m not too far away these days, in Yass. :-)

    I’d probably still be using Cricket for broader networking stuff (SNMP to routers and switches and whatnot — then again, I haven’t really been active in that area for a while, so there might be a better solution these days), but collectd is awesomely simple for the kinds of things you always want to watch on a single server, or a small cluster.

  11. Ben Lisle says:

    Yep! Working for TransACT still! How are things out your way? Ever find your way into good old Canberra?

  12. Jeff Waugh says:

    Sometimes it can’t be avoided… and Pia has just started a new job out there… so we should catch up some time!

  13. Holy crap – Ben Lisle commented as well?

    Truly you have an illustrious readership Jeff.

  14. Ben Lisle says:

    JW – Ubuntu user I am guessing? I see they have a server version!

    CS – Yep. Always handy to keep track on those who know more than you!

  15. Jeff Waugh says:

    @Ben Lisle: Still loving Ubuntu… been using it as a server ages before they slapped the label on it. It’s just a pretty Debian with a release schedule. ;-)

  16. Ben Lisle says:

    JW – I kind of disliked having to backport to Lenny before it was even released.

    Really loving the JEOS Ubuntu has though. Rocking stuff.

  17. Jordi says:

    Jeff, it’s a CGI wrapper written in C, which you hook to nginx as a proxy_pass to deal with your CGIs. Just ping me whenever you want to have a look, as I won’t upload to Debian just yet due to improper licence declaration in the upstream code.

  18. stuartm says:

    Hey Jeff – enjoyed your session today at WordCampNZ, hope you made your flight in time. I was the one who asked the question about whether it’s possible to run Nginx and Apache on the same host. I’m now looking for info on how to get this set up, but am struggling to find instructions for this specific use case. Most of the tutorials I’ve found deal with using Nginx as a web server by itself, or using solely as a reverse-proxy. Are you able to provide some more details on how you got it working with Apache?

  19. Jeff Waugh says:

    @stuartm: Thanks — I did make the flight on time, but it did not get in to Sydney in time for my second flight. ;-)

    So, first thing to do is change the address and port your Apache listens on and that your NameVirtualHosts are attached to. Just change them all to, say, 127.0.0.1:9000.

    Then in nginx, you proxy_pass to Apache (either with http://127.0.0.1:9000 in the proxy_pass directive, or within an upstream group — which you might find handy).

    Or were you struggling with other parts of the equation?

    (Might be best to email me rather than go back and forth in the comments here — I can blog a little tute based on your questions if it ends up being generally useful!)

  20. stuartm says:

    Thanks Jeff, I’ll send you an email with more details, but what I did was set up Apache to listen on port 80 on the localhost, and nginx listened on port 80 on the real IP address.

  21. Jordi says:

    Ahoy Jeff!

    I was playing with rrdtool for nginx today, and got reminded of this blog post.

    I was wondering, how far has nginx penetrated in your infrastructure. Ie, are still running the monster? I probably already told you by some other means, but fcgiwrap is in Debian squeeze and latest Ubuntus, and I’m running pyblosxom’s CGI (for now, not using a WSGI wrapper) just using that and nginx.


    # Pass blog CGI requests to fcgiwrap
    location /blog/ {
    # Disable gzip (it makes scripts feel slower since they
    # have to complete before getting gzipped)
    gzip off;

    # Set the root to /srv/oskuro.net (inside this location, this means
    # that we are giving access to the files under /srv/oskuro.net/cgi-bin)
    root /srv/oskuro.net/cgi-bin;

    # Include standar fastcgi parametres
    include fastcgi_params;

    # fcgiwrap's socket
    fastcgi_pass unix:/var/run/fcgiwrap.socket;

    # Set SCRIPT_FILENAME to the requested script
    fastcgi_param SCRIPT_FILENAME /srv/oskuro.net/cgi-bin$fastcgi_script_name;
    }

    For Mailman, that would be something like:


    location /mailman {
    # Disable gzip (it makes scripts feel slower since they have to
    # complete before getting gzipped)
    gzip off;

    # Set the root to /usr/lib/cgi-bin (inside this location this
    # means that we are giving access to the files under
    # /usr/lib/cgi-bin/mailman)
    root /usr/lib/cgi-bin;

    # Fastcgi socket
    fastcgi_pass unix:/var/run/fcgiwrap.socket;

    # Fastcgi parameters, include the standard ones
    include /etc/nginx/fastcgi_params;
    # Adjust non standard parameters (SCRIPT_FILENAME)
    fastcgi_param SCRIPT_FILENAME /usr/lib/cgi-bin$fastcgi_script_name;

    And a suitable alias directive to handle /usr/share/images/mailman.

    HTH!

    Jordi

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>

Comments will be sent to the moderation queue.