Apache memory usage bogosity
For several months I've been seeing some very strange behaviour from Apache: As each day progressed, the child processes would use more and more memory, resetting to their initial size each morning when Apache restarted (due to daily HTTP log rotation), resulting in a very clear sawtooth pattern in my MRTG graphs of CPU and memory usage. This wasn't an unmanageable problem -- by setting a MaxRequestsPerChild limit in httpd.conf, I instructed Apache to "recycle" its children, which eventually recovered the lost memory -- but it was irritating enough that I wanted to find a better solution.I found that solution this morning. Working via the procfs pseudo-filesystem (also known as "rootmefs" due to its long history of security flaws), I compared the memory maps of a 2MB Apache process and a 9MB Apache process and discovered that, as expected, all of the increased memory usage was within the region used by malloc(3) calls. Inspecting the contents of this memory, I was surprised to find that it was almost all used by a long list of files names -- specifically, the names of the 18,000+ files in the /f/ directory on the portsnap mirror.
Since the growth in memory usage took the form of individual processes suddenly growing much larger, rather than all the processes slowly growing, it was clear that I was looking for a single request which caused Apache to generate a long list of file names... at which point the answer is immediately obvious. Looking at the access logs, I confirmed my guess: Someone was issuing a request which mapped to the directory, and Apache was helpfully responding with a directory listing -- after first allocating 7MB of memory to store the list of files in memory so that they could be sorted. Apache was presumably "freeing" this memory afterwards, but since it was not advising the kernel that the memory was unneeded, the additional megabytes continued to be marked as "in use" (and would ultimately be paged to disk when memory pressure made it necessary).
The simplest fixes are the best fixes, and this one is certainly simple: In the .htaccess file at the root of portsnap2.freebsd.org, I added the line "options -Indexes". No more large indices being generated; no more large memory allocations; and in the past 14 hours the memory usage on the server has remained nearly exactly constant.