Eliminating Dangling Elastic IP Takeovers with Ghostbuster
TL;DR
When you are deploying infrastructure to AWS, you may spin up EC2 instances which have an IP associated with them. When you create DNS records pointing to these IPs, but forget to remove the DNS records after the EC2 instance has been given a new IP or has been destroyed, you are susceptible to subdomain takeover attacks.
We’ve created a tool called Ghostbuster, which works by enumerating all the elastic/public IPs associated with every AWS account you own, and then checking if there are any DNS records pointing to elastic IPs that you don’t own in any of your AWS accounts. It is important to note that this tool requires that you have complete coverage over your AWS accounts in order to avoid false positive findings.
Further instructions on how to setup and use the tool can be found in the README file inside our Github repo for Ghostbuster.
Introduction
Over the last ten years, companies have truly adopted cloud providers such as AWS, Azure and GCP, rapidly spinning up infrastructure to keep up with the growing needs of their businesses. The move to cloud-native architectures has presented a number of challenges for organizations that have adopted it, ranging from visibility of the cloud attack surface to cloud specific security issues.
Although the major cloud providers do provide a number of security benefits as part of their service, they all operate on a “shared responsibility” model when it comes to security. We’ve seen a number of misconfiguration vulnerabilities come and go in AWS, ranging from Route53 takeovers, Cloudfront takeovers, ELB takeovers, and now Elastic IP takeovers. These misconfigurations were considered to fall on the customer side of the shared responsibility model and while AWS eventually deployed mitigations for Route53, Cloudfront and ELB takeovers in their platform, these issues were exploited for several years.
Notably, AWS isn’t alone in these issues, Microsoft Azure and Google Cloud Platform both also face this vulnerability in many of their cloud based offerings, as do a host of other cloud providers.
With the rate of development and the fluidity of cloud attack surfaces, it’s difficult to gain visibility into these exposures - particularly at scale. Couple this with cloud providers that are slow to deploy mitigations for issues they believe should be on the customer side of the shared responsibility model, and many organizations struggle to stay on top of these misconfiguration issues.
As part of developing our Attack Surface Management platform, we research these techniques and build automation to identify cloud security issues (amongst many other types of security issues) in real-time for our customers. Throughout the last year we have seen an increased prevalence of dangling elastic IP takeovers affecting companies both large and small.
In recognition of these challenges, we are open-sourcing a tool called Ghostbuster based on our platform automation which can be used to remediate potential dangling elastic IPs from your AWS environment.
What Is The Dangling Elastic IP Problem?
When you are deploying infrastructure to AWS, you may spin up EC2 instances which have an IP associated with them. When you create DNS records pointing to these IPs, but forget to remove the DNS records after the EC2 instance has been given a new IP or has been destroyed, you are susceptible to subdomain takeover attacks.
There has been a great amount of research done on elastic IP takeovers, where it is possible for attackers to continually claim elastic IPs until they obtain an IP associated with a subdomain of the company they are targeting.
While AWS frequently bans accounts that are attempting to perform this attack pattern, no long term fix has been released by AWS.
The impact of dangling elastic IP subdomain takeover attacks are more serious than a typical subdomain takeover where you can only control the content being served. With dangling elastic IP takeovers, it is possible for an attacker to do the following:
- Claim SSL certificates for the subdomain via ACME TLS challenges
- Listen for traffic on all ports (potentially discovering sensitive information still being sent to the subdomain)
- Run server-side scripts with the ability to steal HTTPOnly cookies, typically leading to a one-click account takeover attack when cookies are scoped to <span class="code_single-line">*.domain.com</span>
Previous Research
We’re not the first people to notice this problem, with the earliest publication about elastic IP takeovers dating back to 2015, from an ex-colleague of mine, Matthew Bryant: Fishing the AWS IP Pool for Dangling Domains.
Since 2015, a number of other people have attempted similar solutions to Matthew, sticking to the approach of allocating and deallocating elastic IPs until they find an elastic IP used by the company they are targeting.
Other PoCs include EIP Fish and Flying a False Flag - Cloud Racoon.
In particular, the issue of dangling elastic IPs has been heavily researched by a number of bug bounty hunters, some of which are making over $50k a month exploiting this issue on AWS customers. A lot of innovation has come out of the bug bounty scene, especially when it comes to cloud-specific attacks such as this one.
Outside of companies that invest in developing their own solutions, paying out via a bug bounty program or buying a product like our Continuous Security Platform (which is capable of detecting this issue proactively), the only protection we have against these attacks is AWS banning accounts that they have identified performing elastic IP takeovers. Unfortunately banning accounts is not a viable solution.
Assetnote’s Efforts
A few years ago, I had a dream where I had worked out whether or not an elastic IP was allocated through performing timing analysis on AWS’s EC2 APIs. After waking up from this dream, I spent days trying to perform timing analysis that would leak this information. Ultimately, I was unsuccessful.
But, we weren’t ready to give up. We spent over three months analysing each and every AWS API that would internally establish whether or not an Elastic IP is allocated or not. Our hope was that by triggering one such API for any given IP address, we could sense a small difference in the response timing as to whether the internal lookup of the Elastic IP resulted in “it’s allocated” or “it’s not allocated”. Why is this information so useful, you may be asking? Because if you can determine if a specific IP is allocated or not, you can detect if a subdomain pointing to that IP is vulnerable.
Unfortunately, after months of detailed analysis and research, we were unsuccessful at performing timing analysis against AWS APIs.
Following the previous work done in this field, we then moved onto allocating and deallocating elastic IPs from the pool and checking if the IPs belonged to our customers. This is referred to as the “lottery” approach, which we did find some success with, but were still deeply unsatisfied with because of the sparse results it produced.
Although this approach worked in finding elastic IP takeovers for our customers, AWS reached out to us and told us to not continue with this approach.
With the “lottery” approach now off the table, we moved further towards an approach that involved our customers integrating all of their AWS accounts in our Continuous Security Platform.
Although this requires integration and a little more setup, this approach has been successful for detecting these issues. Given the potential impact of these issues and the difficulties encountered by many in solving them, we decided to build and release an open-source tool, Ghostbuster, based on work in our product. This tool can be used to detect dangling elastic IPs, given that the AWS credentials used have the appropriate level of access to all of your relevant AWS accounts.
How AWS is Managing This Issue
We have had many discussions with various teams at AWS about this issue and our attempts to build tooling that helps companies find and mitigate dangling elastic IP takeovers.
Historically the approach AWS has taken is to effectively treat this as abuse of their public APIs and simply ban accounts that are allocating or deallocating elastic IPs too fast, or in patterns that AWS deems as abuse. Although this raises the bar for exploitation to some extent (particularly at scale), this is not an effective long-term mitigation to the underlying issue.
AWS have stated that they are working on a mitigation to this issue on their side, however the exact details of this mitigation and the timeline for its implementation remain unclear.
As it stands today, these techniques are still possible.
Working Towards a Resolution: Ghostbuster
The prevalence of this problem is widespread and the number of organisations affected by this attack vector keeps on increasing as companies continue to migrate more services to the public cloud, especially to AWS. The mistake of not removing DNS records after spinning down an EC2 instance is an easy one to make, exacerbated by automatic provisioning, and we wanted to make a conscious effort to be a part of the solution.
Until AWS releases any official fixes, we recommend that you check out our newly released open-source tool Ghostbuster which can be used to detect potential dangling elastic IPs.
You can install the tool by running: <span class="code_single-line">pip3 install ghostbuster</span>, and subsequently use the tool by using the <span class="code_single-line">ghostbuster</span> command.
The tool works by enumerating all the elastic/public IPs associated with every AWS account you own, and then checking if there are any DNS records pointing to elastic IPs that you don’t own in any of your AWS accounts. It is important to note that this tool requires that you have complete coverage over your AWS accounts in order to avoid false positive findings.
The tool is user-friendly, and uses your <span class="code_single-line">.aws/config</span> and <span class="code_single-line">.aws/credentials</span> files to iterate through every configured account and perform processing.
Additionally, if you use Cloudflare to manage your DNS instead of Route53, the tool also integrates with Cloudflare to pull zones and associated DNS records for analysis.
If you don’t use Route53 or Cloudflare to manage your DNS zones, you can provide manual input to the tool (CSV). It is also possible to configure a Slack webhook so that this tool will send a notification upon detecting a potential takeover.
This tool was designed in a way that it can be run as a cron job on a frequent basis, informing you of potential elastic IP takeovers over time.
The following options are available when using the tool:
The output of the tool looks like this:
Further instructions on how to setup and use the tool can be found in the README file inside our Github repo for Ghostbuster.
Conclusion
Managing the security of cloud architectures at scale can be difficult and companies leaving behind DNS records after EC2 resources are destroyed is extremely common.
Unlike typical subdomain takeover attacks where it is only possible to control the contents of the page, elastic IP takeovers hold a much higher risk profile, with attackers being able to claim SSL certs, listen to traffic and perform account takeover attacks in the case of broad cookie scoping.
The details of the intended mitigation from AWS and the timeline for its release remain unclear. In the meantime, it is possible to use Ghostbuster to get on top of dangling elastic IPs in your AWS environment.
Assetnote Is Hiring!
If you are interested in working on the leading Attack Surface Management platform that’s helping companies worldwide from the Fortune 100 to innovate startups secure millions of systems, please check out our careers page for current openings. We are always on the lookout for top talent, so even if there are no open roles in your field please feel free to drop us a line.
More Like This
Ready to get started?
Get on a call with our team and learn how Assetnote can change the way you secure your attack surface. We'll set you up with a trial instance so you can see the impact for yourself.
Level 10, 12 Creek Street, Brisbane QLD, 4000
Contact:
contact@assetnote.io
Press Inquiries:
press@assetnote.io
