Ubuntu Server AWS EC2 ELB Apache Log Fix

If you’ve launched AWS EC2 server instances behind a load balancer, then you’ve probably noticed that your Apache logs are showing the internal ip address of the load balancer for all entries.

I’ve seen a few blogs that explain that the reason for this is that the ELB load balancer is configured as a reverse proxy and the actual ip address of the origin is masked.  Aside from the cosmetics of seeing thousands of log entries all with the same ip address, this is actually bad for SEO as your sitemaps will also be showing all your visitors coming from one ip address.  Also, if you are using geoip the country filter won’t work as it will be looking at all visitors as coming from Amazon’s region that your EC2 instances are located.

Here’s the fix:
Very simple actually.  It worked for me on Ubuntu 12.04 LTS and I assume it will work on any Linux configuration.  Instead of logging the Host (which in our case is the internal ip), we want to log the X-Forwarded_for which is the origin ip for the entry.
Open your apache configuration file for edit.  On Ubuntu it’s at /etc/apache2/apache2.conf    You can also use locate to find it on your instance if it’s not on Ubuntu.  locate apache2.conf

sudo vi /etc/apache2/apache2.conf

Next within configuration file look for the section with LogFormat   There should be several lines beginning with LogFormat.  The top three should be for vhost_combined, combined, and common.   In all three lines where you see %h  (which is echoing to the apache log the host) change that to %{X-Forwarded-For}i

Before

LogFormat “%v:%p %h %l %u %t \”%r\” %>s %O \”%{Referer}i\” \”%{User-Agent}i\”” vhost_combined
LogFormat “%h %l %u %t \”%r\” %>s %O \”%{Referer}i\” \”%{User-Agent}i\”” combined
LogFormat “%h %l %u %t \”%r\” %>s %O” common

After

LogFormat “%v:%p %{X-Forwarded-For}i %l %u %t \”%r\” %>s %O \”%{Referer}i\” \”%{User-Agent}i\”” vhost_combined
LogFormat “%{X-Forwarded-For}i %l %u %t \”%r\” %>s %O \”%{Referer}i\” \”%{User-Agent}i\”” combined
LogFormat “%{X-Forwarded-For}i %l %u %t \”%r\” %>s %O” common

Then remember to reload apache

sudo service apache2 reload

Note of Caution: It’s relatively easy for someone to spoof the header and change the X-Forwarded-For to whatever address they want. That would mask from you their actual ip. Therefore, this solution is not intended for applications that have security implications.

Another tip. Within your web application if you are using the ip for any purpose, like Maxmind’s geoip, you’ll need to echo the X-Forwarded-For instead of the Remote host as well. e.g. $ip=$_SERVER[‘HTTP_X_FORWARDED_FOR’];

Let us know if you have any suggestions on how to improve this method.

Ubuntu Server – How to Sync Files Between Multiple Web Servers

Another installment in our How to Set up an Ubuntu Server Series:

Synchronizing multiple servers – You have more than one web server in your host configuration. What’s the easiest way to synch files among the multiple web servers?
Solution – rsync does the trick efficiently and easily.

This tutorial assumes you are comfortable on the linux command line.

Steps to Set up rsync for multiple servers synchronization:

Step One

First make sure rsync is installed on each server that you’ll be working with.  By typing rsync from a command prompt, you should get the rsync response asking you for parameters.  If rsync isn’t found, then install it.

sudo apt-get install rsync

Step Two

Test synchronizing files.  From a command prompt on your 2nd server, type in an rsync command to retrieve files from the 1st server.  Let’s assume you have a directory /var/www/ set up on both servers and you wish to synchronize the files from the first server to the 2nd. The first server acts as the master.  Any new files or changes to existing files on the master will overwrite existing files on the 2nd server.  Type in from the command prompt on server 2

sudo rsync -ra user@FirstServerIP:/var/www/ /var/www/

The first server in this example sends all the files in /var/www/ along with the (-r) recursive subdirectories. You can also sync only specific files, etc…
If this works, you are ready to move on to step 3. If not, check your connectivity with the FirstServer. Generally, if you can ssh to another server, you should be able to rsync with it as well. AWS EC2 Users If you are using AWS Amazon’s ec2 servers, which require a passkey, the rsync command will look like this:

rsync -rave “ssh -i /home/ubuntu/.ssh/passkey.pem” ubuntu@EC2ServerIP:/var/www/ /var/www/

Note that for AWS EC2, you will first need to copy your passkey over to your .ssh directory

Step Three

Next, let’s set this up to synchronize all the time automatically. We accomplish this with a cron job. Here again, we’re assuming you are familiar with how to set up and edit crons. From your root user (sudo su) edit your crontab

crontab -e

Then, add a line to the crontab that tells it to run the rsync command every minute (or any frequency that suits you).

* * * * * rsync -ar FirstServerIP:/var/www/* /var/www/

Test your configuration!  Go to your first server and make a change or add a file to the synchronized directory. Then go back to the 2nd server and see if the file is there. You can replicate this process to as many servers as you need, but keep in mind that frequency of sync and amount of data passing on your internal network can get costly and cause a hit on performance.

I hope this little tutorial was helpful.  Please drop me a note if you have any questions or comments.

 

AWS Micro Servers Are Not Cheap But You Have Options

Amazon Web Services (AWS) has an amazing array of web servers and options for everyone (Netflix is their biggest customer).
For a new customer, the first year includes a ‘free-tier‘ which generously reduces prices, and some services are even free.

I signed up and moved Techexposures.com to AWS a little over a year ago. You need familiarize yourself with their nomenclature and how to navigate their control panel. Once you get the hang of it, everything goes very smoothly. They even give you basic server monitoring tools for free during the first year. The free support is good too, if you’re willing to read through forums of how others solved your problems. You can pay for a higher level of support that gives you one-on-one responses.

In the back of my head I remembered that the first year was coming to a close. I thought the micro server cost (their smallest dedicated size) was under $10/month…
So, when I received an automated email from AWS reminding me that the year would end in July, I figured I’d let it continue until I had time to look into alternatives.

Well, today I had a look… They offer a very good online calculator for you to plan and know exactly what your monthly costs will be (actually, they bill in hourly segments. It actually gets that granular!). Bottom line, 1 micro (tiny) server at their Virginia location, with a dedicated ip address (“Elastic IP”) and one volume, currently costs about $18.50 / month with no long term commitments. You can even start up and terminate a server for a few hours to test something and only pay for the hours used.

There are ways to lower the annual cost for a dedicated server. For example, you can commit to a yearly contract, and pay a reduced upfront fee, and then a small monthly charge. Doing it that way drops the cost to less than half.

Here’s a link to the Amazon AWS Web Services Monthly Cost Calculator where you can calculate your own costs if you’re thinking of moving there.