How to turn DDOS attacks into a business opportunity
Introduction
I’ve been working with Amazon AWS for two years now. During this time I have acquired an instinct of staying alert all the time. Amazon keeps coming up with new ingenious ways of extracting money. And boy, they did it again today! Welcome to AWS Lambda Function URLs!
Lambda URL Endpoints
The recently added Lambda URL is a great improvement. It makes it much easier to expose Lambda functions through a secure HTTPS endpoint. Before it was quite difficult to do. It required setting up the API Gateway, which is a very complicated thing in itself. There are a lot of configuration steps and moving parts. The first time I tried it, I spent three days before it worked. With the new Lambda URL configuration, it becomes much simpler. There’s a new dialog in function configuration. There you can create a public URL, configure authorization, CORS, etc. A few clicks and you can call the function over HTTPS.
So, where’s this new and exciting business opportunity for Amazon? Right here, under the Auth Type setting:
NONE: Lambda won’t perform IAM authentication on requests to your function URL. The URL endpoint will be public unless you put in place your own authorization logic in your function.
If the word about your public function gets out, you’re screwed. Amazon will charge you for every invocation of the function. It’s an open invitation to DDOS attacks which can cost a lot, in actual hard currency. The documentation suggests using custom authorization logic in such cases. But even then, every invocation of the function is charged even if the request ends up being denied.
There’s not a single reason why a sane person would make a Lambda URL public. Why is it there then, if not to profit from human gullibility?
How Much Does Lambda Cost, Exactly?
Truth is, you will never know until the monthly bill comes due. There is no way to predict exactly what your charges will be.
On the surface, lambda seems cheap. Everybody knows that usage is charged per request. One million requests costs around 0.20$, not much. What many don’t realize, is that there are extra charges, depending on:
- Request duration
- Allocated memory
- Logs usage
Imagine booking a hotel room. You check out and the clerk asks for extra fees for:
- opening the windows (0.002 EUR per event)
- and closing them
- looking inside the mini-bar (a separate fee for actual consumption applies)
- walking to the bathroom (to avoid the fee, don’t)
- the time spent on the toilet (the free tier includes peeing, but only if done standing)
- the length of the toilet paper used (the minimal charged amount is 10 meters)
It can bite you hard, as we learned in the past. Here are a few such cases:
Connection timeouts inside a lambda
We have Lambda functions writing to a MySQL database. One day unusually high traffic caused database connection pool timeouts. The default timeout was 10 seconds. Result: thousands of lambda calls per hour, normally executed within 50-80 ms, now billed by Amazon for the entire 10 seconds of doing nothing. Amazon charged us hundreds of EUR extra per day. So be careful with any calls outside the lambda, such as MySQL database, REDIS cache, sending emails etc. It can all time out, and you will be charged for the idle lambda. Keep your timeouts as short as possible and always review the defaults.
Memory allocation charges
Memory allocation by lambda is not dynamic. You’re not charged for what the function uses. The charge is for the maximal available memory. It does not matter if the function uses it or not. If only we knew that before… Our functions need between 80 and 100 MB. The default configuration is … 1024 MB. It’s buried deep, with no hint whatsoever of what it will cost. Until someone has noticed this, we were charged three times as much as needed, for a long time.
Logging inside the lambda
Careful with logging. Never forget that you’re charged per amount of log writes, and for the physical size of the logs. Our lambdas are called thousands of times per hour, which means a lot of logging. So I thought: once we’re done with development and testing, why not turn off the logs? Well, easier said than done:
See that greyed-out toggle? It is on but disabled, you simply cannot switch it off. It feels nearly like someone is making fun of me ;-) After long research, I found a workaround. It’s in the permissions of the IAM role under which the lambda runs. There, I could remove the permissions to write to CloudWatch logs. But it’s buried deep, so I still don’t quite understand what I’ve done. I wrote a long tutorial full of screenshots for myself and future generations.
DDOS-As-A-Business-Opportunity, Conclusions
It all accumulates when your application becomes a target of a DDOS attack. Amazon won’t ask questions. It will charge for anything extra: additional API Gateway calls and bandwidth, extra log space used, extra lambda calls including their time wasted on doing nothing because of the timeouts. It all comes on the bill, and before the billing alerts kick in at the end of the day.
Imagine that there’s a DDOS attack ongoing. All hands on deck, you’re busy defending your system from a predator. The last thing you want to think about is the AWS bill. Yet your cloud provider turns your disaster into a money-making opportunity. Hmmm …
In all fairness, I don’t have an issue with paying for the service. What I find annoying, is how opaque the AWS cost structure is. How often we’re surprised by yet another charge which we weren’t aware of. It makes one extremely cautious when adding new functionality. Because there’s no way of knowing the final price tag upfront. It costs time and money to defend yourself from this unpredictability, while I have better uses of my time than fighting off the AWS bill.
Is it too much to ask, a little floating window showing the expected extra costs, as I’m configuring an AWS service? Just like with booking an airline ticket? The obvious difference is that airlines are legally forced to inform us upfront about the exact costs, before we pay. I wish a similar requirement applied to the Big Cloud Tech.
All in all, be careful using a lambda back-end for any kind of high-intensity API endpoints. You’ll be likely better off setting up an EC2 instance and hosting the entire endpoint there, using own logging facilities, etc. The fees for EC2 are fixed, predictable, and often much lower. An extra benefit is that you’re no longer vendor-locked!