blog

Learn more about Opbeat
Sign up for product updates
You should

Amateur hour at AWS

You are probably sick of hearing of the Heartbleed bug by now. A lot has been said about the way disclosure of this bug was handled, so instead this post will focus on how not to handle remediation and communication to users.

We use Amazon Web Service’s (AWS) Elastic Load Balancer (ELB), and have generally been quite happy with it. When the Heartbleed Bug first surfaced, we updated our servers immidiately. ELB is a very central component in our infrastructure, and if it is still vulnerable, updating our backend servers will do very little to protect our customers.

The first indication that AWS were working on remediation was this notice:

First indication

Noticed there is no timestamp? Amateur hour at AWS. If this was a hot, new cloud startup that made it huge overnight, it might have been easier to cut them some slack, but AWS has been the leader for a very long time.

At this point there were already tools to easily determine if you were vulnerable or not, and while AWS merely stated they were “investigating any impact”, the impact for us was quite clear:

INSECURE - opbeat.com:443 has the heartbeat extension enabled and is vulnerable
INSECURE - opbeat.com:443 has the heartbeat extension enabled and is vulnerable
INSECURE - opbeat.com:443 has the heartbeat extension enabled and is vulnerable
INSECURE - opbeat.com:443 has the heartbeat extension enabled and is vulnerable
INSECURE - opbeat.com:443 has the heartbeat extension enabled and is vulnerable
INSECURE - opbeat.com:443 has the heartbeat extension enabled and is vulnerable
INSECURE - opbeat.com:443 has the heartbeat extension enabled and is vulnerable
INSECURE - opbeat.com:443 has the heartbeat extension enabled and is vulnerable
INSECURE - opbeat.com:443 has the heartbeat extension enabled and is vulnerable
INSECURE - opbeat.com:443 has the heartbeat extension enabled and is vulnerable
INSECURE - opbeat.com:443 has the heartbeat extension enabled and is vulnerable
INSECURE - opbeat.com:443 has the heartbeat extension enabled and is vulnerable
INSECURE - opbeat.com:443 has the heartbeat extension enabled and is vulnerable
INSECURE - opbeat.com:443 has the heartbeat extension enabled and is vulnerable

We used this script (link at the bottom):

do ./heartbleeder opbeat.com; sleep 5; done

As the day progressed we started seeing reports on Hacker News that many ELBs had been patched. We pay for the privilege of being able to contact AWS’ support. We rarely use it, but it is in situations like this you are really happy you do, right?

Quite useless

Note the “Please feel free to check back in at any time and I would be more than happy to assist.” at the bottom.

Much, much later, AWS posted an update that looked like this:

Within a few hours

Almost everything is great. The remaining ELBs will be updated within a couple of hours.

Again notice the complete lack of timestamp which makes it impossible to determine when to expect the rollout to be complete.

At that time, our loadbalancer still gave inconsistent responses:

SECURE - timed out while waiting for a response from opbeat.com:443
SECURE - timed out while waiting for a response from opbeat.com:443
SECURE - timed out while waiting for a response from opbeat.com:443
INSECURE - opbeat.com:443 has the heartbeat extension enabled and is vulnerable
INSECURE - opbeat.com:443 has the heartbeat extension enabled and is vulnerable
SECURE - timed out while waiting for a response from opbeat.com:443
INSECURE - opbeat.com:443 has the heartbeat extension enabled and is vulnerable
INSECURE - opbeat.com:443 has the heartbeat extension enabled and is vulnerable
SECURE - timed out while waiting for a response from opbeat.com:443
INSECURE - opbeat.com:443 has the heartbeat extension enabled and is vulnerable
SECURE - timed out while waiting for a response from opbeat.com:443
SECURE - timed out while waiting for a response from opbeat.com:443
INSECURE - opbeat.com:443 has the heartbeat extension enabled and is vulnerable
INSECURE - opbeat.com:443 has the heartbeat extension enabled and is vulnerable
SECURE - timed out while waiting for a response from opbeat.com:443
INSECURE - opbeat.com:443 has the heartbeat extension enabled and is vulnerable
INSECURE - opbeat.com:443 has the heartbeat extension enabled and is vulnerable

It is a known fact that ELB uses roundrobin DNS to do the first round of load balancing. This means when two requests are made, it might go to two different IP addresses. Machines responding to these addresses are supposed to be in separate availability zones. The response above would seem to indicate that some of the machines responding were patched, and some were not.

We tried corrolating the resolved IP addresses to the response, but seemingly, there was no correlation. This could indicate that IP addresses are not always routed to specific machines.

Another couple of hours went by until we started seeing this:

SECURE - timed out while waiting for a response from opbeat.com:443
SECURE - timed out while waiting for a response from opbeat.com:443
SECURE - timed out while waiting for a response from opbeat.com:443
SECURE - timed out while waiting for a response from opbeat.com:443
SECURE - timed out while waiting for a response from opbeat.com:443
SECURE - timed out while waiting for a response from opbeat.com:443
SECURE - timed out while waiting for a response from opbeat.com:443
SECURE - timed out while waiting for a response from opbeat.com:443
SECURE - timed out while waiting for a response from opbeat.com:443
SECURE - timed out while waiting for a response from opbeat.com:443
SECURE - timed out while waiting for a response from opbeat.com:443

However, due to nobody outside of AWS knowing exactly how ELB works, this could just mean that the machines currently responding to the requests are patched, but in the next request, it could hit an unpatched ELB machine. Another case of lack of transparency from AWS, making it impossible to make rational decisions.

We then wrote to support to hear if it was now safe to re-key the certificates, but did not hear from them for hours. Finally, AWS updated the security bulletin site to reflect that all ELBs were now patched:

Finally patched

Notice how they updated the page, making it impossible to understand the timeline and progression of prior events. Again, amateur hour at AWS.

A couple of final points:

  1. See if you can find the security bulletin on AWS from the start page: https://aws.amazon.com/
  2. No mention of anything security related on the official @awscloud Twitter feed until very, very late, and then only a retweet: http://cl.ly/image/1g0T2s1r2b2e
  3. Everything is awesome: http://cl.ly/image/1D3j3J3u3G42

This was a text book example of how to do bad communication on very serious issues. Additionally, it took way too long for our ELBs to get patched - we’ll be looking into replacing ELB with something like nginx or HAProxy.

As always, Heroku is leading the way in communicating issues to their users: https://status.heroku.com/incidents/606.


If you want to know more about Heartbleed Bug, read about it on heartbleed.com. If you want to know if you are vulnerable, you can use Filippo’s tool or this go program by Titanous.


Discuss on Hacker News

About the author
Ron Cohen is co-founder and CTO at Opbeat.
Follow him on Twitter or GitHub .
Sign up for updates from Opbeat
You should
comments powered byDisqus