As most of you probably know, I work at CohesiveFT where I focus on VPN-Cubed product. In short, it’s a solution to build overlay networks in third-party clouds. Overlay networks in this case are based on redundant encrypted point-to-point connections from your regular servers to your VPN-Cubed servers called “managers” (that you run in the cloud); managers then act as virtual switches and routers of this overlay, which essentially sits above your physical network. In other words, an overlay network gives a customer effectively a LAN-like network where the servers can be located pretty much anywhere, including in the cloud.

However, not all people know what an overlay network is or what its benefits and strengths are. This holiday season, as we were putting up our outdoor decorations and holiday lighting, I realized that what my wife and I were doing was essentially building an overlay network. Let’s follow the similarities.

Imagine a regular house with a front yard where for the holidays you want to set up a bunch of lighted Christmas trees, deer and other holiday figures. All of them require electricity - but there is no power installed in the ground (parallel with VPN-Cubed overlay network: you are deploying servers to third-party cloud and want to continue using your IP addressing schemes, want to ensure that all communications are encrypted - but provider doesn't offer any of these services out of the box).

You don’t need power out on your front yard all year around - so there is usually no point in investing money in installing one. Cloud computing is all about elasticity. As a complement to clouds, VPN-Cubed is easy to set up and take down if necessary for an experiment, or it can be running for long periods of time.

There are several outdoor outlets on the front wall so you are deciding to power your decorations from these outlets (you have VPN devices installed on the edge of your network - you will use them to offer connectivity to your servers from your network using VPN). The first obvious solution is to run a power cord from each piece towards an outlet. While it’s possible in theory, it will turn out ugly in practice. Firstly, a lot of long outdoor power cords are expensive. Secondly, it will create a cabling mess near the outlet. Thirdly, if a cord goes bad, you need to trace where exactly it’s plugged in and replace it. Fourthly, the more stuff you have to power up, the more difficult this octopus made of power cords is going to be. Absolutely the same problems apply in our parallel use case.

So you come up with optimization #1 - you go out and buy several outdoor power strips with several outlets each. By placing these power strips where your lighted trees and deer are, you are reducing cabling issues, gain ability to use shorter power cords and most likely save money on power cords. That's your VPN-Cubed manager server instance. When you place it next to your cloud-based servers, you reduce latency for your endpoints and cut down on VPN connections from the edge of your network that you need to build and maintain.

If you are well prepared (i.e., have enough of everything), your composition will drive how many power cords and strips you will need and how long your cords need to be, not the other way around. Same with VPN-Cubed - you mold it to fit your use case, your desired topology or application - you don't adjust your application to be able to work within VPN-Cubed overlay network.

Outdoor power strips have additional protection to let them function outdoors in low temperatures. And so are VPN-Cubed manager instances - they are running a hardened OS, with minimal set of enabled services, behind firewall protection. You can grab a regular switch and make it work outdoors - but why waste your time when these things don’t cost that much? Same with VPN-Cubed.

But power strips may fail - and if they do, entire section of your composition will be turned off. So you get a cold standby sitting in your garage in case a primary goes out. Or better - you install 2 power strips next to each other, connect them and evenly plug in your endpoints. If one goes out, you switch all connections to the other strip and it’s back. VPN-Cubed allows you to deploy a hot spare with automatic failover capability, which can help balance the load as well. Your outdoor lighted Christmas tree is connected to one power strip at any given time, but if one fails it can be reconnected to another within a power cord distance. Same with VPN-Cubed - your servers are connected to a single manager at any given time, but if a manager becomes unavailable, your servers can automatically re-connect to another manager.

And what happens if one of your outlets goes bad? Moving a handful of cables to another outlet is much easier than moving a whole lot. Same with VPN-Cubed - if your network loses one entry point, you just re-connect VPN-Cubed to another.

There are many more parallels between the two. Most of us have been building overlay networks of decorations for quite some time. Building overlay networks for the cloud may be new, but CohesiveFT VPN-Cubed product makes it easy and fun. Don’t be stuck with long power cords - get yourself some nice outdoor power strips. And enjoy the holidays!

This past Thursday I had a chance to attend CloudCamp Boston that took place in Microsoft research center in Cambridge, as a representative of CohesiveFT. The event was very well attended, and I was able to meet a lot of smart interesting people working in cloud computing space.

The lightning talks section started with Microsoft representative giving an overview of Azure. Then, John Willis gave a talk titled What color is your cloud in which he talked about various types of IaaS clouds. George Reese gave a talk where he compared a successful cloud deployment to reaching the Emerald City and pointed out that the yellow brick road to this goal is not always an easy one (great analogy!). Iron Mountain representative gave a talk about how one needs to be aware where their data are in the cloud, and emphasized security measures at their datacenter. Intuit representative talked about their PaaS, which allows developers to easily reach millions of small businesses already running Intuit products. PaaS is not my thing, but the idea makes sense and if I understand correctly, is very similar to idea behind Salesforce platform - develop against something which many organizations already use. And finally, Cory Von Wallenstein of Dyn, operators of a well-known DynDNS.org, gave a talk about their enterprise features like anycast DNS, CDN etc. Interestingly, now that I think about it, DynDNS offered a way to update DNS programmatically way back when, which definitely qualifies them as one of the earliest cloud APIs out there.

After a break, I attended a cloud security talk titled Cloudifornication by Chris Hoff. I’ve seen the slides and video of this talk before (for example, see here), but seeing it live was more than worth it. This is a very good and important talk for all cloud practitioners and especially architects and developers, and I highly recommend it. I personally had 3 main takeaways. Firstly, information security is based on C-I-A (confidentiality, integrity, availability). Therefore, any outage or service disruption is classified by a customer as a security issue, not only as an SLA issue. I didn’t know about this until a couple of weeks ago when Chris explained it to me on Twitter, and the talk also emphasized this fact. Secondly, I loved a series of slides about increasing complexity of interconnects as more and more vendors, intermediaries and brokers are added to one’s cloud mix. We at CohesiveFT are very aware of this as an emerging issue, and our VPN-Cubed product is targeted at such cases, among other things.

And thirdly, Chris very skillfully highlighted the brittleness of the foundation on top of which we collectively as an industry are currently building out our cloud offerings. When Internet was designed as it was for world wide web and static pages, it was all good. When we started doing e-commerce and social media on top of the same infrastructure, the risk increased many-fold but was still somewhat manageable (after all, it’s only buying stuff online). But now with cloud computing we are putting absolutely everything (!) on top of the same brittle foundation, and the risks are truly enormous.

Then, I attended a session on private cloud led by John Willis, where we discussed various private cloud technologies and ideas. My main takeaway was that there is or there will be a huge demand for private externally-operated clouds for mid-sized organizations, and that’s where I think future of colo and hosting is going to be.

All in all, this was a great event and thanks to all organizers and sponsors for putting it together, and to all participants for interesting discussion.

Since earlier this year when I got my copy of Erlang book, I’ve wanted to do something unconventional with RabbitMQ source. I finally came up with an idea, which is somewhat interesting and maybe even useful, and could be done by an Erlang beginner like myself.

Some background first. Each Erlang program in general consists of multiple Erlang processes that send and receive messages to/from each other. These are not your regular processes - these processes are running inside the Erlang VM, and are not mapped to either processes or threads on the system level in any way. They are cheap to create, and communications between them are fast. Each process has a process ID (pid) associated with it. If multiple Erlang VMs share a cookie (a piece of text which is used as a security token), processes on any Erlang VM can freely talk to each other without any code modification.

RabbitMQ implements each queue as an Erlang process. You can think of AMQP exchanges as pieces of routing logic - when a message arrives, Rabbit applies the routing rules to its routing key and sends the message to all eligible queues. Since each queue is a process and has a pid, in a nutshell exchanges send the message to several pids.

In non-clustered mode, Rabbit routing function will return a list of local PIDs by selecting values stored in a mnesia table (rabbit_durable_queue and rabbit_queue). But since RabbitMQ can also run in clustered mode, developers already implemented a way to send messages to queues on remote nodes. So when I hacked rabbit_durable_queue and rabbit_queue tables and replaced PIDs pointing to local queues with PIDs pointing to remote queues, I got myself a remote queue forwarder.

How is this useful you might ask. RabbitMQ supports remote queues only in clustered mode. The way clustering is implemented today (using mnesia in distributed mode), it’s not recommended to run a rabbit cluster over non-LAN links. This is because if 2 rabbit nodes lose and then regain connectivity to each other, the cluster may enter a “partitioned network” state, which effectively means that rabbit cluster is not functional (in other words, mnesia sacrifices tolerance to partitioning in order to achieve consistency and availability - recall CAP theorem). With remote queue forwarding, you don’t need to set up clustering and hence “partitioned network” state won’t affect you by design - and that’s what mattered to me.

On the other hand, remote queue forwarding can potentially break some AMQP guarantees. For example, if a remote node is temporarily unavailable, Rabbit won’t queue the message for later re-delivery (because such situation is currently impossible in unhacked Rabbit). It means that YOU SHOULD NOT USE THIS HACK UNLESS YOU KNOW WHAT YOU ARE DOING.

The hack works for me though in the following scenario. I publish messages with immediate=true (indicates that payload is time sensitive). Messages are sent to N local queues. These N local queues are forwarded to N remote nodes, with a consumer attached to 127.0.0.1 on each node. In this project, I don’t rely on any AMQP guarantees - I discard basic.return commands and can tolerate occasional message not reaching some or even all consumers.

I doubt anyone will find this hack in its current form useful, but just in case I uploaded it to http://github.com/somic/rabbit_queue_forwarder.

PS. After I started this small project, Tony Garnock-Jones announced that his work on pluggable exchange types has been added to default branch. I still went ahead and published my post, but please note that pluggable exchange types could probably achieve similar effect more cleanly.

I have recently noticed that costs were no longer always touted as the main driver for cloud computing - some have been advocating agility as the primary reason (for example, see here). It’s one thing when this theme gets mentioned in a talk at a technology conference where a company is sharing their experiences. But it’s a bit different when we start seeing it in vendors’ pitches - whenever anybody is trying to sell something new for more, I like to really understand what it is that I am paying premium for.

There are at least 2 types of agility that cloud computing could potentially enable.

The first is being able to provision resources faster than it would take in a traditional environment. Time to market, how fast you can see that an idea is working or is not working - both are positively affected.

The second is being able to right-size your resources (sometimes this concept is called “elasticity”). It eliminates the need to over-provision - start small, scale up when demand goes up, scale down when resources are no longer needed. In this case, agility refers to speed with which you can implement the upsize/downsize operation. No doubt cloud computing opens new opportunities in this space.

However, the most important question is - do any of these potential agility benefits apply to your use case (workload)? To me the answer is obviously not yes for every single project. Not every IT project in the world requires agility, and furthermore - not every IT project in the world requires agility at a price (remember who stands to benefit if all of a sudden all IT organizations in the world start putting premium on agility and become willing to cover it in real money when paying for services).

Again - I am not saying that none of IT projects could benefit from better agility. I am saying that not all projects could.

If your workload does not benefit from increased agility, your main driver towards cloud computing in theory should be cutting costs. It can take a form of explicit reduction in expenditures, or it can be paying the same for more, or it can be avoiding some cost which you’d otherwise have to pay. I am very far from accounting, but it’s my understanding that there may also be some benefit in spreading out costs over a longer period of time instead of putting up capital up front.

If you end up using cloud computing without taking advantage of increased agility and you didn’t cut costs migrating the cloud, I can think of several possibilities.

First, it’s possible that there is some cost that you simply overlooked. My favorite example in this area is Internet bandwidth. If you are in the cloud, you have a very fat pipe to the Internet - not something that every project accounts for.

Secondly, you may be trying to future-proof your staff - let them do a project in the cloud to gain experience, since cloud development and cloud operations may require slightly different skill set than traditional equivalents. Similarly, there are many proof-of-concept projects where main goal is to see what it would take an organization to do a project in the cloud, to see what breaks, what doesn’t work well, etc.

Thirdly, you may anticipate you will need agility or may anticipate potential for cutting costs in future projects, and you may be stepping into the cloud to avoid having to migrate these apps in the future anyway.

And finally, if none of the above applies to you, you must be doing it due to the hype. There is nothing wrong with this, as long as you understand it.

To summarize: cloud computing offers potential for unmatched agility; you may end up paying premium to take advantage of it; if agility is not on your list, you should get benefit of reduced costs; if you are not after agility and you don’t cut your costs, you should stop and think why exactly you are using cloud computing.

Disclaimer 1: Despite its possibly ominous name, this is NOT a network vulnerability or an attack that could lead to unauthorized access. UDP hole punching requires cooperation between two hosts, and hence can't be easily used as an attack by itself (in other words, in order to run it, you most likely must already have gained access to the hosts).

Disclaimer 2: Conclusions reached at the end of this post are my educated guesses, and may turn out to be not true. They are based on my observations and not on actual knowledge how EC2 internals are designed or implemented.

I was once working on a setup in Amazon EC2 and came across an oddity, which when coupled with my interest in EC2 security groups mechanism, turned into this post.

UDP hole punching, in a nutshell, is a technique which allows two cooperating hosts, potentially located behind NAT and/or firewalls, to establish a peer-to-peer UDP communication channel directly to each other. It’s a technique used by Skype, for example, - you can read more about it in a Wikipedia article. If two hosts start sending UDP packets to each other on pre-agreed ports, bi-directional flow of packets leads NAT devices and firewalls to think that all these packets are a part of an established communication channel.

EC2 allows a lighter form of this technique because EC2 NAT never rewrites source port of outgoing packet (recall that in EC2, NAT is always 1-to-1 such that port rewriting isn’t necessary). We know with 100% certainty that a packet we are sending with a given source port X will be seen by remote instance with the same source port.

I wrote a small Python tool (available at http://gist.github.com/224795) to test UDP hole punching and set out to discover if it could work in EC2. My expectation was that it should work. Unless explicitly noted, I used ports above 45,000 and none of security groups explicitly allowed UDP traffic on these ports.

I was able to easily punch UDP holes between any two instances using each instance's public IP address - in line with my expectation. But I hit a major snag when using private IP addresses of 2 instances in the same region (I used EC2-US) - I couldn't get it to work no matter what I tried: same availability zone, different availability zones, same security groups, different security groups, same AWS account, different AWS accounts. I even tried punching a hole over port 53 (all EC2 instances support DNS name resolution which happens over this port without an explicit corresponding rule in security groups) - no luck (EC2 DNS servers are not located on 10.0.0.0/8 where all instances reside).

The only way I could get it to work using private IPs, is to allow my UDP port in security groups of at least one of the instances. When I did this, both hosts reported success.

This observation leads to several thoughts that might help uncover some aspects of EC2 firewall’s internal design (these are all more or less educated guesses):

• You can punch a UDP hole between any 2 instances using their public IPs, even if your security groups do not allow such communication.
• Private IP traffic is treated totally differently than traffic over public IPs.
• You can punch a UDP hole on port X using private IP addresses of 2 instances in the same region only if at least one of the instances allows port X in its security groups (can be used as a test if you don't have access to query EC2 API endpoint)
• EC2 firewall somehow implements more logic than "all outgoing packets are allowed" when dealing with traffic over private IPs (if it were not the case, hole punching should have worked - see below).
• If we assume that security group rules are applied at an instance's dom0 (as makes at least some sense and as this research implies), I now suspect that all dom0 hosts have entire view of all security groups in the region and are getting real time updates when a rule is added or deleted (modification of rules is currently not supported). This in fact was contrary to my expectation - initially I thought each dom0 "subscribes" to updates for only those security groups which correspond to instances running on this dom0 and I thought this was the reason why dynamic group membership changes were not possible (say I want to move an instance from "db" security group to "webapp" security group).

To clarify: under the above assumption, in order for hole punching to NOT work, an outgoing packet from instance A must not reach dom0 of instance B - and the only way it’s possible under “all outgoing packets are allowed” policy is if dom0 of instance A knows that dom0 of instance B will block this packet and somehow takes this into consideration - which in general case can only happen if all dom0 hosts have entire view of all security groups and permissions in the region.

I would love to hear your thoughts on what could possibly explain this behavior, please let me know in the comments below.

If you liked this post, you may also be interested in Probing Ports in Remote Security Groups in EC2.