This is the story about how ULYSSIS managed to get rid of its old mod_php-based shared webhosting setup and replaced it with a new and improved PHP-FPM-based setup, without breaking our users’ sites¹. If you’re not interested in the story, rationale and background information, you can skip ahead to the “how”. If you’re interested in the story and the “why” though, read right along.

The problem

Something had been bugging me ever since I joined ULYSSIS back in 2010: in our web setup, PHP ran as the user Apache runs as. On our Ubuntu servers, that user is www-data. This was probably not unlike how other shared hosting providers do it, but there are quite a few downsides to this:

1. First and foremost, it’s a security issue: if all of the PHP has to be processed by the www-data user, this user needs read access—and sometimes even write access—to all of the PHP files of a user. As a result, the installation process of many CMSs, like Drupal, involves changing permissions to certain files to allow all users to write to them: 666 for files, 777 for directories². This would for example allow users to craft special scripts that do things like access other users’ database information. Also, because ULYSSIS offers shell access to its users, this kind of abuse gets even easier. We have some security through obscurity tricks in place to hide users’ home directories, by automounting them, and disallowing directory listings by setting 701 rights on every home directory. However, if a user is nefarious and crafty enough, they could still easily gain access and wreak havoc on other users.
2. There’s an accountability problem: it’s harder to see what user is eating RAM and using a lot of CPU. This becomes much easier if the PHP scripts are actually being run by those users themselves.
3. It’s an inconvenience to our users. Whenever PHP would create a file, it would do so as www-data. When the user then tried to remove the file they would not succeed, because they wouldn’t have the rights to do so. This also caused those files not to be counted towards that user’s quota. We used to have a cronjob in place to fix this, but that’s definitely not the cleanest solution, and we’d have to wait until the cronjob fired.
4. It hampers Apache’s performance, especially when serving static content. This is because mod_php only works in conjunction with the prefork multi process module (MPM). The prefork MPM spawns separate processes, one process for each connection with a client. Whenever these processes would get stuck for some reason—which happened quite often—we  quickly ran out of available processes, and the entire Apache server would hang until restarted. Often, that wasn’t even sufficient, and we had to actually force kill all of the hanging child processes. The more modern event MPM is much more efficient: it also forks a few child processes, but every child process keeps a pool of threads around to handle requests. These threads will then only be occupied when they’re handling a user’s request. Once a request is handled, the thread returns to the pool.

First attempt: ITK

The first solution I tried was MPM-ITK³. This is an alternative third-party MPM that functions very similarly to the prefork MPM, but instead of running its forks as www-data, it would change to the user configured for a particular virtual host. However, this is still a prefork type MPM, and actually adds a bit of extra overhead. It also requires Apache to run as root. Finally, it doesn’t play well with mod_fcgid, which we need to support other programming languages apart from PHP. It was quickly clear, then, that MPM-ITK would not be the right solution.

PHP-FPM

There seemed to be no solutions that involved PHP actually being run inside of Apache as a module, so I started looking at solutions where Apache functions as a proxy. An obvious one is plain old CGI. In combination with suexec, it allows you to run your scripts as a particular user. CGI is a very inefficient solution to the issue, however, because it starts up a new PHP interpreter process on every request. The only other solution appeared to have to involve FastCGI somehow. There are again many ways to serve PHP over FastCGI, but PHP-FPM appears to be the most flexible and popular option, and PHP-FPM allows to define different pools with different user ids, which is just what we needed. The most popular alternative to mod_php appears to be nginx combined with PHP-FPM, but switching to nginx is not an option for us, because that would break our users’ sites.

So we’re using PHP-FPM, that’s sorted. Now, how to link it to Apache? If you look at the various ways that Apache can talk FastCGI, it’s easy to get confused. There’s no less than at least three different Apache modules that talk FastCGI in some way. Here’s an overview of the three:

• mod_fastcgi: this is an old module. It does the job, but it isn’t really being maintained anymore. We actually tried mod_fastcgi for a while, because it was the only one we got to work without some sort of breakage⁴. However, its age is showing: when I tried to use a separate CustomLog directive for every user with a different log format, but the same file, I mistakenly assumed that it would open only one file handle. In reality, this would open a new file handle for every user, which actually caused mod_fastcgi to fail, since its reliance on the select system call makes it incompatible with file descriptors over 1024.
• mod_fcgid: we do have mod_fcgid installed, so that users can use other programming languages than PHP, but mod_fcgid is more geared towards launching and managing FastCGI processes from Apache and we already have PHP-FPM doing that for us.
• mod_proxy_fcgi: this is the youngest of the three, and the most integrated one with Apache. Instead of providing new custom configuration directives, it simply hooks into Apache’s existing mod_proxy system. It’s also programmed in a modern, event-based way. We tried mod_proxy_fcgi for a short time before switching to mod_fastcgi for a while because there was some breakage⁵. The problem stems from the fact that mod_proxy’s ProxyPassMatch directive matches on URL (regardless of whether this URL corresponds to an actual file), whereas we really needed something that matches only actual existing php files. After SetHandler support for proxies was introduced in Apache 2.4.10, we switched back to using mod_proxy_fcgi.

At long last: the solution

Now we’ve come to the point where I’m actually going to show you how ULYSSIS actually has it set up. At ULYSSIS, we use LTS versions of Ubuntu. At the time of writing, the latest LTS is 16.04. I’m sure people using another server OS will be able to adapt. All of these commands have to be executed as root, so you can either switch to the root user, or put sudo before every command.

Update your package index:

Install PHP with PHP-FPM and Apache 2:

Press enter to continue when prompted.

If you install both Apache and PHP-FPM, it will have already figured out that you probably want the event MPM, so that will be enabled by default.

Now, you have to configure your virtual hosts to have their own PHP-FPM pool. In this example we’ll just use the example.com virtual host. In reality, ULYSSIS generates all of the virtual hosts from a similar template. We won’t touch on any other configuration tweaks, because that’s outside of the scope of this article.

Let’s create a pool for example.com, for the “example” user. Let’s create that user:

Let’s give him a www directory:

Let’s test it with an index.php file that just shows the info page:

Let’s make sure that only the example user can read this file:

Now, we’ll create the file /etc/php/7.0/fpm/pool.d/example.conf with the contents:

Restart PHP-FPM:

Now, let’s create the vhost in /etc/apache2/sites-available/example.conf with contents:

Now, for testing purposes, let’s disable the default vhost, and enable the example vhost:

Then, enable mod_proxy_fcgi, and restart Apache:

Now, go to the server in your web browser. You’ll see a PHP info page. The Server API should be FPM/FastCGI, and when you check for running php-fpm processes with ps aux | grep php, it should list the example pool running as the example user!

What about php_flag and php_value?

There was one minor outstanding issue still: with mod_php it was possible to use php_flag and php_value in .htaccess files to modify some PHP parameters, e.g. how and where errors are logged. We needed to support these, if we wanted the transition to be perfectly seamless. Luckily, it turns out that one of the ULYSSIS alumni wrote a PECL package called htscanner that does exactly that. It scans .htaccess files for these directives, and applies them.

Unfortunately, at the time of writing, htscanner does not support PHP 7. I wrote a patch to add PHP 7 support, but it hasn’t been merged yet. This means we’ll have to patch and compile htscanner ourselves.

To compile PHP modules, we’ll need the PHP 7.0 dev package:

Download htscanner:

Unpack it and switch to its directory:

Download the PHP 7.0 patch:

Apply the patch:

phpize the directory:

Configure htscanner:

And finally, build and install it:

Then, to activate this extension, create the file /etc/php/7.0/mods-available/htscanner.ini with the contents:

Then, enable this config with:

Then, restart PHP-FPM:

One more thing: Apache will not recognize php_value and php_flag directives, and fail with a 500 error if these are not guarded with <IfModule>. For this, there is an equivalent mod_htscanner for Apache. You’ll need to install apache2-dev for this:

Then, still in the directory where you unpacked htscanner, build it with:

The -c flag means “compile”, the -i flag means “install”, and the -a flag means “activate”, so this will install and activate mod_htscanner2.

Restart Apache to activate it:

Now, Apache will simply ignore php_value and php_flag directives instead of failing with an error.

Are we done?

Are all issues solved now? Not quite. The main problem we’re still dealing with is the fact that the event MPM is not quite event-driven. When handling a request, the thread will still block. This means that it’s still possible to lock up the webserver by keeping it waiting for something. For example, you could write a PHP file like such:

If enough requests are made to this PHP script, it will lock up Apache, because all of its worker threads will be occupied waiting for this infinite script to return. Eventually, these requests will time out, but in the meantime, Apache will not be able to handle new requests. It goes without saying that you should not pull these kinds of shenanigans on our servers. We will immediately disable your account if you do. Remember, because every PHP pool runs as the user it belongs to, we can easily see which user is tying up all of our resources.

So, did anything break?

Did we finally get what we wanted? Well, we got close enough. None of our users reported any issues, except for any issues that stemmed from our upgrade to PHP 7.0. The one breakage we experienced is that Apache when acting as a proxy adds REDIRECT_ to the front of environment variables, causing certain PHP scripts to cease functioning. For large CMSs, this is usually fine, but some of our modules and custom scripts did break. Overall, we can say that the switch has been a success. One thing we’re not sure about is the influence on webserver performance, because the switch coincided with an upgrade to better hardware. In any case, the webservers don’t get stuck as often anymore, and we don’t have to deal with any pesky permissions issues anymore.

1. Some things did break on our own sites, more on that at the end of the article, but we haven’t had users report any issues.
2. If you’re not familiar with how these permissions work, here’s some more info.
3. Yeah, I have no idea what ITK means either.
4. It still wasn’t quite trivial, though, more information about that here.
5. More info on this issue in this ServerFault question.