A few days ago I noticed a CPU peak on FreeIPAPI's application servers. I Quickly checked Cloudflare and saw that the amount of requests has doubled! By doubled I mean it reached 60 million requests a day!
Was it an attack?!
Well, I still don't have an answer to this but what I know (from Cloudflare) is they were all mobile devices and every device was sending between 5 to 10 requests in a second and they were from all countries in the world! All kinds of internet providers!
Similar to botnets?!
Scaling the infrastructure
Anyway, Attack or Massive usage, I decided to first scale the infrastructure.
Thanks to the architecture of FreeIPAPI (Which I am proud of it 😎) and using VitoDeploy I created 2 more application servers and connected them to the load balancer to help the other 2 servers to handle the requests.
It just took a few minutes to bring App3 and App4 into the game!
PS 1: App1 and App2 were able to handle the 60 million requests but the response time was increased because of the heavy load!
PS 2: Every app server uses 2 GB memory and 2 CPUs which costs less than 5 EUR per month!
PS 3: If you set multiple A records with the same name but different IPs, Then Cloudflare will act as a load balancer!
So the temp servers helped the system back to normal but the requests kept increasing!
Cloudflare's WAF to the rescue
After bringing the app into a stable situation I started looking at Cloudflare to see how I can prevent these requests from reaching my servers. Maybe I should've done this in the first step but believe me it was my first experience in such a situation haha!
Anyway, I found some cool features on Cloudflare to protect my servers from such requests and applied some WAF rules and everything came back to normal.
Another solution I considered
Since FreeIPAPI is a very small SaaS especially the API part of it. Anyone can rewrite it in any language in just a few hours. I did an MVP of writing the API part in Golang but after deploying it to a staging environment I did a few stress tests on it and ended up crashing the server.
But that wasn't the only reason I didn't continue the Golang MVP for this case, The actual reason was the response time! The current version which is a PHP Laravel one has a response time of 50ms and I was expecting the Golang version to be faster but it was the same! Maybe I did it wrong? Although I am new in Golang :)
What is the plan for the future?
Well, Although I've optimized the project as much as possible, I still think that there is room for more optimizations!
So I am going to give another try to Go and/or Some other in-memory web servers for PHP and Laravel like RoadRunner or Swoole!
What do you think? Feel free to drop a comment here ❤️