Million of visitors per day with a super cheap php mysql server using nginx and varnish

These instructions are the rather verbose, but hopefully easy enough to follow, steps to build a new Linux server using Varnish and Nginx to build a php application on a cheap server , all costing under $15 a month, capable of sustaining million hits per day, (we use blitz.io to test it).

Million hits per day on a cheap virtual server

Install Ubuntu 11.10 (Oneiric) on a new virtual private server- it needs to be 11.10 for all the packages that I’m going to list to work out of the box, but it’s all possible to do with other distributions.

You can use Amazon EC2 to build the test server, but Linode are also very good.

Let's start

Public DNS Name: server1.yourserver.com (example)
Public IP Address: 123.123.255.255 (example)

Login to the server as root

login as ubuntu via ssh, then run the sudo -i command to enter as root

ssh [email protected]
sudo -i

Configure a firewall first

It’s important to configure a firewall, since we’re going to be installing various network services which by default listen on all interfaces.For Ubuntu, this is easy, simply use the ufw package.

Execute the following commands to configure it:

ufw allow ssh
ufw allow http
ufw logging off
ufw enable

Once this is done, your server has a relatively secure firewall, though it’s worth looking at fail2ban to prevent brute force password attacks.

Note:If you’re using Amazon EC2, you’ll also need to open the Security Group to allow traffic on port 80. You can do this using the AWS Security Groups Console, you might need to change the region. Select the security group you used when you started the instance, and then click “Inbound”, then select “HTTP” from the drop down menu, then finally click “Add Rule”. You don’t need to restart the instance for it to take effect.

Install and Configure MySQL

apt-get update
apt-get install mysql-server

When prompted, set a mysql “root” user password

mysql -u root -p

When prompted, enter your newly set root password

At the mysql> prompt, run the following 4 commands, create your database and replacing ENTER_A_PASSWORD with a password of your own

CREATE DATABASE mydatabase;
GRANT ALL PRIVILEGES ON mysatabase.* TO "my_user"@"localhost" IDENTIFIED BY "ENTER_A_PASSWORD";
FLUSH PRIVILEGES;
EXIT

That’s MySQL installed, ready for the PHP and Web server installation (nginx).

Install and configure PHP

We need to install not just PHP, but the PHP FPM system, APC, and the MySQL module

apt-get install php5-fpm php-pear php5-common php5-mysql php-apc php5-gd

Edit /etc/php5/fpm/php.ini and add these lines at the bottom:

[apc]
apc.write_lock = 1
apc.slam_defense = 0

Edit /etc/php5/fpm/pool.d/www.conf

Replace

listen = 127.0.0.1:9000

with

listen = /dev/shm/php-fpm-www.sock

Below that, insert these 3 lines

listen.owner = nginx
listen.group = nginx
listen.mode = 0660

Then, further down in the same file, replace these 2 lines

user = www-data
group = www-data

with

user = nginx
group = nginx

Save the file, PHP FPM is now complete, but it won’t work until we install nginx, so don’t worry about starting it now.

Install and Configure Nginx

Instructions based on the Nginx website.

Download the nginx secure key to verify the package

cd /tmp/
wget http://nginx.org/keys/nginx_signing.key
apt-key add /tmp/nginx_signing.key

Add the sources to the APT sources file by running these 2 commands (the >> is important!)

echo "deb http://nginx.org/packages/ubuntu/ lucid nginx" >> /etc/apt/sources.list
echo "deb-src http://nginx.org/packages/ubuntu/ lucid nginx" >> /etc/apt/sources.list

Download and install nginx by running

apt-get update
apt-get install nginx

When that completes, nginx will be installed, but needs configuring for your php app

nginx configuration files are in /etc/nginx

First, edit /etc/nginx/nginx.conf

Inside the http section , insert the following line so that when you later add varnish in front, things don’t break all over the place:

port_in_redirect off;

Next, cd to /etc/nginx/conf.d and create a new file, /etc/nginx/conf.d/drop with the contents of the following (configuration good for wordpress too)

# Most sites won't have configured favicon or robots.txt
# and since its always grabbed, turn it off in access log
# and turn off it's not-found error in the error log
location = /favicon.ico { access_log off; log_not_found off; }
location = /robots.txt { access_log off; log_not_found off; }
location = /apple-touch-icon.png { access_log off; log_not_found off; }
location = /apple-touch-icon-precomposed.png { access_log off; log_not_found off; }

# Rather than just denying .ht* in the config, why not deny
# access to all .invisible files
location ~ /\. { deny  all; access_log off; log_not_found off; }

Then, replace /etc/nginx/conf.d/default.conf with the contents of the following changing all entries for domainname.com with your own domain name (there’s 3 entries, including 1 near the bottom)

server {
    ## Your website name goes here.
    server_name domainname.com www.domainname.com;
    ## Your only path reference.
    root /var/www/;
    listen 80;
    ## This should be in your http block and if it is, it's not needed here.
    index index.html index.htm index.php;

    include conf.d/drop;

        location / {
                # This is cool because no php is touched for static content
			try_files $uri $uri/ /index.php?q=$uri&$args;
        }

        location ~ \.php$ {
            fastcgi_buffers 8 256k;
            fastcgi_buffer_size 128k;
            fastcgi_intercept_errors on;
            include fastcgi_params;
            fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
            fastcgi_pass unix:/dev/shm/php-fpm-www.sock;

        }

}

Make a directory, /var/www/ and set the ownership:

mkdir -p /var/www/
chown nginx:nginx /var/www/
chmod 775 /var/www

That’s nginx configured, restart it and the PHP FPM service by running:

service nginx restart
service php5-fpm restart

Now, you’re actually ready to start your php app!

Change the owner of your app directory

chown -R nginx:nginx /var/www

Once it’s done, if you’ve not had any errors and your domain name is pointing at the right IP (this is important!), then you’ll be able to visit your domain and see your homepage.

Even if the server is running, but it’s still too slow for our goal

Time to install varnish 3!

apt-get install varnish

Edit /etc/varnish/default.vcl replace the contents with this content

# This is a basic VCL configuration file for varnish.  See the vcl(7)
# man page for details on VCL syntax and semantics.
# 
# Default backend definition.  Set this to point to your content
# server.
# 
# THIS IS VALID FOR WORDPRESS CONFIGURATION
# 

backend default {
    .host = "127.0.0.1";
    .port = "8080";
	.connect_timeout = 600s;
	.first_byte_timeout = 600s;
	.between_bytes_timeout = 600s;
	.max_connections = 800;
}


acl purge {
        "localhost";
}

sub vcl_recv {
	set req.grace = 2m;

  # Set X-Forwarded-For header for logging in nginx
  remove req.http.X-Forwarded-For;
  set    req.http.X-Forwarded-For = client.ip;


  # Remove has_js and CloudFlare/Google Analytics __* cookies.
  set req.http.Cookie = regsuball(req.http.Cookie, "(^|;\s*)(_[_a-z]+|has_js)=[^;]*", "");
  # Remove a ";" prefix, if present.
  set req.http.Cookie = regsub(req.http.Cookie, "^;\s*", "");



# Either the admin pages or the login
if (req.url ~ "/wp-(login|admin|cron)") {
        # Don't cache, pass to backend
        return (pass);
}


# Remove the wp-settings-1 cookie
set req.http.Cookie = regsuball(req.http.Cookie, "wp-settings-1=[^;]+(; )?", "");

# Remove the wp-settings-time-1 cookie
set req.http.Cookie = regsuball(req.http.Cookie, "wp-settings-time-1=[^;]+(; )?", "");

# Remove the wp test cookie
set req.http.Cookie = regsuball(req.http.Cookie, "wordpress_test_cookie=[^;]+(; )?", "");

# Static content unique to the theme can be cached (so no user uploaded images)
# The reason I don't take the wp-content/uploads is because of cache size on bigger blogs
# that would fill up with all those files getting pushed into cache
if (req.url ~ "wp-content/themes/" && req.url ~ "\.(css|js|png|gif|jp(e)?g)") {
    unset req.http.cookie;
}

# Even if no cookies are present, I don't want my "uploads" to be cached due to their potential size
if (req.url ~ "/wp-content/uploads/") {
    return (pass);
}

# Check the cookies for wordpress-specific items
if (req.http.Cookie ~ "wordpress_" || req.http.Cookie ~ "comment_") {
        # A wordpress specific cookie has been set
    return (pass);
}



	# allow PURGE from localhost
	if (req.request == "PURGE") {
		if (!client.ip ~ purge) {
			error 405 "Not allowed.";
		}
		return (lookup);
	}


	# Force lookup if the request is a no-cache request from the client
	if (req.http.Cache-Control ~ "no-cache") {
		return (pass);
	}


# Try a cache-lookup
return (lookup);

}

sub vcl_fetch {
	#set obj.grace = 5m;
    set beresp.grace = 2m;

}

sub vcl_hit {
        if (req.request == "PURGE") {
                purge;
                error 200 "Purged.";
        }
}

sub vcl_miss {
        if (req.request == "PURGE") {
                purge;
                error 200 "Purged.";
        }
}

Edit /etc/default/varnish

Change the section

DAEMON_OPTS="-a :6081 \
-T localhost:6082 \
-f /etc/varnish/default.vcl \
-S /etc/varnish/secret \
-s malloc,256m"

to

DAEMON_OPTS="-a :80 \
-T localhost:6082 \
-f /etc/varnish/default.vcl \
-S /etc/varnish/secret \
-s malloc,64m"

Next, we need to edit the nginx configuration to listen on port 8080, instead of port 80 – Varnish is going to be running on port 80 instead.

Edit /etc/nginx/conf.d/default.conf and replace

listen 80;

with

listen 8080;

Save the file, then run

service nginx restart
service varnish restart

There it is, million hits per day on a box costing less than $20 a month, all thanks to varnish and nginx, easy!