Last week I came across an interesting post on Amazon EC2 spot price spikes published on GigaOm. In the comments, in response to a question from a reader, the author stated that “I don’t think anyone ever expected that the market would behave like this.” I have been interested in this expectation for some time and decided to kick off 2012 blogging season on somic.org with a post dedicated to this topic.
As I described before, a spot instance technically is not that much different from a regular on-demand instance - it has the same CPU and RAM capacity, same network traffic and bandwidth allowances. The only fundamental difference is that AWS can terminate it under certain circumstances (when spot price exceeds this instance’s bid) - because of this, most people rationally expect spot instances to trade at a discount to the price of regular on-demand.
Furthermore, somewhat similar to the concept known as law of one price, one’s intuition says that if the spot price exceeds on-demand price, people will stop bidding on spot and will start getting regular instances instead, until spot price comes down as a result of reduced demand.
But then we face a question. How is it possible that we see the following prices for m1.small/Linux in us-east-1 when its on-demand price is 0.085:
SPOTINSTANCEPRICE 0.500000 2011-11-16T14:39:39-0600 m1.smalLinux/UNIX SPOTINSTANCEPRICE 2.000000 2011-11-16T14:53:40-0600 m1.smalLinux/UNIX SPOTINSTANCEPRICE 3.000000 2011-11-16T15:32:37-0600 m1.smalLinux/UNIX SPOTINSTANCEPRICE 2.000000 2011-11-16T17:51:35-0600 m1.smalLinux/UNIX SPOTINSTANCEPRICE 3.000000 2011-11-16T20:33:19-0600 m1.smalLinux/UNIX SPOTINSTANCEPRICE 2.100000 2011-11-17T01:43:24-0600 m1.smalLinux/UNIX SPOTINSTANCEPRICE 3.000000 2011-11-17T02:38:30-0600 m1.smalLinux/UNIX SPOTINSTANCEPRICE 2.100000 2011-11-17T05:34:07-0600 m1.smalLinux/UNIX SPOTINSTANCEPRICE 2.000000 2011-11-17T10:05:29-0600 m1.smalLinux/UNIX SPOTINSTANCEPRICE 3.000000 2011-11-17T12:22:39-0600 m1.smalLinux/UNIX SPOTINSTANCEPRICE 2.000000 2011-11-17T13:39:53-0600 m1.smalLinux/UNIX SPOTINSTANCEPRICE 5.000000 2011-11-17T13:53:19-0600 m1.smalLinux/UNIX SPOTINSTANCEPRICE 1.000000 2011-11-17T14:50:19-0600 m1.smalLinux/UNIX SPOTINSTANCEPRICE 0.500000 2011-11-17T19:49:52-0600 m1.smalLinux/UNIX SPOTINSTANCEPRICE 1.000000 2011-11-18T07:05:59-0600 m1.smalLinux/UNIX SPOTINSTANCEPRICE 0.500000 2011-11-18T09:23:32-0600 m1.smalLinux/UNIX
Note that spot price in this timeframe fluctuated between 588% and 5,882% of on-demand regular price. (I am wondering if users who had spot instances running at these prices know how to spell “overpaid.”)
I think there are several possible explanations.
Firstly, it’s possible that whoever does actual bidding on spot instances is not the same person who pays the bill - for example, developers bid, while accounting department gets charged on its credit card. This may lead to careless bidding and drive the price to unnecessarily high level.
Secondly, it’s possible that some customers don’t have enough sophistication built into their automated bidding systems. In the price history snippet above, spot price remained extremely high for entire day of November 17 (Thursday) - it shouldn’t have gone unnoticed even by a semi-automated system (i.e., one with occasional human supervision and monitoring). The right course of action was to cancel one’s all outstanding bids and switch to using on-demand. Any automated bidding system must be monitoring spot price at all times and be prepared to switch new instance launches away from spot to on-demand if spot price remains elevated for long periods of time, as well as termine current spot instances.
Thirdly, it’s possible that EC2 decided to shut down spot market for this instance type by setting spot price above all bids (I am talking about $5 price).
And finally, it’s possible that some customers want to gamble. They could be bidding above their true price to be able to weather the spikes. Their thinking could go like this: “Over life of a spot instance, in normal times when spot price is below on-demand, we realize good savings. We could give back some of those savings to offset times when spot price spikes so that in the end we still come out ahead.”
Hence, in addition to recommendations of the GigaOm post’s author, here is my advice.
- Avoid spot instances for workloads that must run non-stop for the foreseeable future (especially in us-east-1 where spot prices seem to fluctuate and spike a lot more)
- Do not set spot price above on-demand price unless you really know what you are doing and have sufficient automated instrumentation in place to protect you in case spot price does go through the roof
- Do not submit spot instance bids for immediate execution if spot price is already significantly higher than on-demand
- Hoping that spot price will come down to below on-demand very soon is not a bidding strategy, it's gambling
You can read my other posts about EC2 spot instances here.