Probing Ports in Remote Security Groups in EC2

This is the third part of my series on Amazon EC2 security groups. In part 1, I described how security groups are possibly the most underappreciated feature in EC2. In part 2, I described a UDP hole punching technique, which led to some interesting conclusions.

On several occasions, when troubleshooting a connectivity issue or verifying a deployment’s security, I needed to check local and/or remote security group rules from within my instances. I never have AWS credentials on the instances, so I couldn’t use the API. And EC2 metadata server (169.254.169.254) returns only the names of security groups, not the rules themselves. In other words, I needed to answer the following question - “do remote instance’s security groups allow or not allow communications from this instance on TCP/UDP port X.” Note that trying to open a connection and not getting a response doesn’t answer this question - packets could be blocked by security groups or by remote instance’s local firewall (iptables). It turns out there is a way…

Let me first start with a warning. You probably are OK using this technique against your own instances, or instances of your friends, partners, vendors and customers with their permission and when you know specific ports you need to probe, for verification or troubleshooting purposes. Probing someone else’s instances or random ports may get you in trouble - such activities could fall into a category prohibited by AWS Acceptable Use Policy. You have been warned.

Summary

This technique works using private IP addresses, so both instances must be running within the same EC2 region.

To check if remote instance’s security groups allow communications from your instance on UDP port $X, do this:

traceroute -p $X -w 1 -q 1 -m 1 $REMOTE_INSTANCE_PRIVATE_IP

To check if remote instance’s security groups allow communications from your instance on TCP port $X, do this:

tcptracerute -w 1 -q 1 -m 1 $REMOTE_INSTANCE_PRIVATE_IP $X

(tcptraceroute is not a standard tool installed by default; you can get it from here or apt-get install tcptraceroute on Debian or Ubuntu)

If you see ”1 {IP address of first hop} XX.XXX ms”, remote instance’s security groups allow such communications from your private IP address. If you see "1 *”, they don’t.

Background

This technique is based on one of the conclusions of my UDP hole punching post.

In it, it looked to me like each dom0 in the region knows about all security group rules in this region. Additionally, it looked like when dom0 knows that remote dom0 will not accept traffic, it doesn’t bother even sending it. If we assume they do it on dom0 with iptables, they probably simply DROP such traffic (at least I would if I were them). And if they do, they won’t send ICMP Time Exceeded packet back to us - and hence, traceroute won’t be able to report the first hop.

If we do get that ICMP packet from dom0 back, it means it didn’t drop our packet. Which means it knows the other dom0 currently has a rule in security groups allowing it.

I set max_ttl to 1, since first hop in traceroute is believed to represent dom0.

Obviously, I verified this theoretical hypothesis on a pair of EC2 instances in us-east-1, and didn’t see any indication that it could be wrong.

Protecting Against Such Probes

If you have instances in EC2, you may be wondering if you could somehow protect them from such probes. After all, AWS network monitoring is top-notch, but they have a lot of hosts to watch over, so some small amount of probing may still occur and you may not even know it.

I’ve got good and bad news on this front. The good news is that you can do it, the bad news is it’s going to be ugly. Essentially, to avoid disclosing the ports open in your security groups to such probes, you must configure your security groups not to allow connections from arbitrary instances in your EC2 region. In other words, you must exclude 10.0.0.0/8 from all IP-address-based rules (rules granting access by name of security group are not affected). It means that you have to replace each rule that references 0.0.0.0/0 with multiple rules individually referencing 128.0.0.0/1, 64.0.0.0/2, 32.0.0.0/3, 16.0.0.0/4, 1.0.0.0/5 (I would avoid 0.0.0.0 just in case - hence 1 as first octet), 12.0.0.0/6, 8.0.0.0/7 and 11.0.0.0/8. Yep, 8 rules instead of 1. You have to also repeat the same exercise for all IP subnets referenced in your rules that include 10.0.0.0/8.

Conclusion

In the future, of course, I hope security groups API could be extended to support functionality “allow connections from Internet excluding private IPs in this region”. I am going to add it to my previous wishlist.

Categories: cloud-computing |