I was recently building a distributed system which will run in Amazon EC2 cloud. It consisted of several instances of the same AMI that were going to communicate with each other using private IP addresses assigned by EC2.
One interesting scenario popped up in my head. What if, after initial discovery of each peer’s internal IP address, one of the instances goes down (let’s say it was at IP1) and at least one other instance fails to notice this fact and continues to communicate with IP1. EC2 assigns IP addresses dynamically, and as far as I can say, IP1 can get assigned to someone else’s instance within same minute. So my instance will be unknowingly communicating with someone else’s instance - not something that I want to allow.
A solution can be something what the military call Identification Friend or Foe (IFF). You can read about it in Wikipedia or here. Note that you may want to consider an IFF anytime you are running applications in any IaaS cloud that assigns IP addresses dynamically and/or has no way of predicting which IP address your next host is going to get.
My Basic IFF Solution
First of all, my instances do not have access to AWS credentials (here is why). Secondly, I set up a requirement that all instances that needed to communicate with each other were to be launched with the same user-data (from a running instance, you can obtain user-data from http://169.254.169.254/latest/user-data).
I then created 2 checksums (SHA1 or MD5) - 4633e65fce4cf3b40648f574f4b60070 was a checksum of user-data plus some file in the AMI (say /usr/share/doc/coreutils/NEWS.gz) and 7a66a9361b14e95c14d98522502b9487 was a checksum of user-data plus another file in the AMI (say /bin/rmdir). Note that if user-data on each instance are the same, these checksums will be the same, because the files I selected are the same.
Then in Apache, I have the following configuration:
<Location /4633e65fce4cf3b40648f574f4b60070> AuthType basic AuthName 7a66a9361b14e95c14d98522502b9487 AuthUserFile /etc/apache2/users Require valid-user </Location>
Before establishing communications to a peer instance (and regularly afterwards), I set up my instances to get HTTP headers from above location (without actually submitting HTTP auth username and password), check WWW-Authenticate header and look for the second checksum there. Easy and efficient. If both checksums match, the other instance is a friend. If not, a foe. In this case I also assume that if I didn’t get a response, it’s not a friend - an instance might have gone down or it might not have apache listening on 80 or its web server might not know what to do with my URI.
You can further enhance this solution by creating new checksums every N minutes - this should work reliably for as long as EC2 infrastructure has no trouble keeping the clock accurate. You can also embed the timestamp in data used to generate checksums. Furthermore, if you monitor your access logs for bad interrogations (for example, old checksum or wrong checksum), you might be able to easily detect attacks against your IFF system.
A More Scalable IFF
Peer-to-peer IFF implementation that I described above may not work well for large deployments or for enterprise. If you are within either of these categories, I can recommend that you take a look at VPN-Cubed, a product offered by my employer CohesiveFT. Its features essentially serve as a scalable encrypted IFF.