October 25, 2017

A Very Odd Couple: Apache Web Server Response Time and IPFilter

As those who have been with me on this miserable excuse for a website for awhile can likely attest, I am a fairly unabashed fan of *nix systems. I have been running FreeBSD at home for more than sixteen years and, for several years during that period, it was the only operating system I used. No Windows; no Macs; and sometimes no graphical user interfaces. Back in my truly hardcore days (i.e., Berkeley, circa 2002), I didn’t even bother to own a router; why waste the money on some piece of shit hardware (and back then, they really were pieces of shit, even more so than now) when I could route and NAT all the traffic through a computer I had built? Hell, I even flirted with OpenBSD, for crying out loud!

Of course, not all is beer and skittles when it comes to open source software: hardware incompatibility, broken device drivers, lacking (or worse, inaccurate) documentation, and maddeningly cryptic bugs will all sap your will to live if you spend enough time with it. And, so it shouldn’t come as a surprise that over the years, my once unwavering dedication to all things *nix began to waver. I gave up running my own homebrew router sometime in 2004, and tearfully bade farewell to hosting my own e-mail in 2007.

So much the worse that I did, as those sysadmin skills would have undoubtedly come in handy in debugging one of the most cryptic bugs with my systems that I have encountered to date: massive Apache webserver (httpd) performance degradation resulting from a IPFilter glitch. For those interested, more on the problem, the symptoms by which it manifest itself, and also the eventual solution is below.