Google Cloud’s predatory practices: A Cautionary tale
As a cloud user, you are always exposed to costs. It is well known cloud platforms will charge for almost everything, from data storage to data transfer, and even for API requests. However, some cloud providers have been known to engage in predatory practices that can lead to unexpectedly high costs for users. Here is a cautionary tale about my experience with Google Cloud and how I believe their platform’s design led me to significantly higher costs than expected.
Article index
A rogue script
Recently, a friend provided a script with a route optimisation feature which uses the Google Maps API. This script ran through a list of items in an excel sheet, calculating paths using the Distance Matrix API. I made some tweaks and ran the script with some expectation that its local caching system would limit the number of API calls.
After some tweaks to the script, I ran it with zero expectation that it would make a number of API calls beyond Google Maps Platform’s free tier. I was wrong. My expectation of max. ~20 USD of costs turned into a bill worth 100 times as much, well over 2700 USD. This was a shock, and I immediately contacted Google Cloud support to try to resolve the issue.
Google Maps Platform API key creation flow
In the Google Cloud console, if you do not have an API key created, you will immediately be prompted to create one when going to the Maps platform section.
The flow begins with a “Get Started” popup which looks like this:
Once you have copied the API key, you will be prompted to ‘Protect’ it, with pretty standard options:
This is great so far, good experience. However, if you are not extremely careful, this is where they catch you. Now you are thrown into this welcome screen, and you can now send an almost unlimited amount of requests to the Maps API:
API Quotas
Crucially absent from the API key creation flow. You are never prompted or even recommended to set quotas. Is this intentional?
Quotas can control your usage up to a certain limit. You can set quotas in the quotas menu:
As per Google Cloud support’s own admission, you should not expect quotas to limit your usage to the exact amount you have set and that there might be some excess charges above what you have set as a limit, albeit small:
When selecting your desired API to set quotas for, you will find that pretty much all enabled APIs have no caps or limits on daily usage and a very high value on the per minute usage:
This means you can send an enormous, to essentially unlimited amount of requests. The marginal costs for google to serve these requests is likely extremely low. They have the infrastructure to handle it (some of the largest data centers in the world), and this is obviously their line of business. However, this means that if you make a mistake, you can be on the hook for a massive bill which likely far exceeds your expectations, and which for google likely represents a marginal cost.
Cost delays
When you use the platform and start incurring costs, you will not see them immediately. In fact, they take hours to show up, giving even a cost-aware user the impression that maybe requests are not coming through or haven’t incurred a cost yet?.
The truth is that there is a delay of up to 24 - 32 hours before costs are reflected in your billing dashboard. As shown at the bottom of this documentation page:
Support is aware of this, so even if you have detected a mistake early, you might not be able to get it fixed before the costs are ultimately reflected. Which can be significantly higher than what you expect or have already been charged for:
In my case, it took around 18 hours for all costs to be fully reflected. As a result, you are not able to stop costs from accruing shortly after they occur, because they aren’t reflected in real time.
Language utilized
The way Google Cloud support expresses itself is very kind, everyone was extremely kind and understanding. However, the language is clearly showing that if you make a mistake, you are obviously 100% liable for it. Any adjustment is a goodwill gesture, with no guarantees:
Suggestions after the fact
After some questions about the project itself, to verify that it does indeed qualify for the adjustment:
The main requirement for this one-time goodwill gesture is that you setup quotas and secure your API keys:
This is obviously a good idea, but it is something that should be done before you start incurring costs, not after the fact. It is also something that should be strongly suggested during the API key creation flow, not something that you are expected to know about beforehand:
Support is meant to help you solve your case, and in my case they helped significantly, helping me achieve a 90% adjustment on the incurred costs. It is still, however, very likely that this adjustment is not 100% so that people do not expect that they can make mistakes and get everything adjusted. Nonetheless, this ‘90%’ adjustment, while apparently generous, could be far from ideal.
Putting costs into perspective
Assume you write a script that accidentally makes 2 million requests to the same API I used (e.g. If I left it running overnight, which I thankfully didn’t). This will incur a cost of approximately 10,000 USD. A 90% (goodwill, not even guaranteed to be 90%) adjustment will leave you with a 1000 USD bill, which is represents, on a per country basis (Asia source, Europe source, Americas source):
- India 🇮🇳: between 2.76 and 4.9 monthly wages
- Philippines 🇵🇭: ~2.2 monthly wages
- Indonesia 🇮🇩: ~3.1 monthly wages
- Bangladesh 🇧🇩: ~4.1 monthly wages
- Vietnam 🇻🇳: ~2.3 monthly wages
- Brazil 🇧🇷: ~2.0 monthly wages
- Mexico 🇲🇽: ~1.5 monthly wages
- Argentina 🇦🇷: ~2.5 monthly wages
- Nigeria 🇳🇬: ~3.6 monthly wages
And even in higher income countries, 1000 USD is still a significant amount of money:
- Germany 🇩🇪: 31% of avg. monthly wage
- France 🇫🇷: 38% of avg. monthly wage
- United Kingdom 🇬🇧: 36% of avg. monthly wage
- Spain 🇪🇸: 49% of avg. monthly wage
- Italy 🇮🇹: 47% of avg. monthly wage
- Poland 🇵🇱: 66% of avg. monthly wage
It is also known that Google will try to get their money, and while this is not a great example given that the poster here never attempted to contact support, it is clear that Google is very serious about getting paid for their services, even if usage is accidental and the cost is marginal.
Suggestion for Google Cloud
Google Cloud should absolutely change the API key creation flow to strongly suggest setting quotas and securing API keys. This should be a mandatory step before you can start using the Maps Platform API, or any other API for that matter.
Even though it is well known that cloud platforms can be expensive, it is not reasonable to expect users to know about all the intricacies of each platform, especially when:
- Budgets don’t hard cap usage
- Quotas cannot cannot fully prevent you from incurring costs above the set limit
- The default quota is essentially unlimited
- Costs are not reflected in real time, or even with a small delay and take nearly a full day (or more) to be fully reflected
- Quotas are not suggested or enforced during API key creation
This is a predatory practice that can lead to significant financial harm for small developers. We take responsibility for our mistake by covering the incurred cost, we are able luckily able to cover this cost, but what if you’re a user or company from a lower income country, or if you’re a small developer who cannot afford such a bill?
In my case, I use Google Workspace for my company and our bill never exceeds 20 USD per month. Even after adjustment, our costs for this incident represent about 20 months of our Google Workspace bill.