Making sure you can handle bad bots #2
Update March 2016: we have changed the 503 response code (temporarily unavailable) to the – now standard – 429 (too many requests).
A few of our customers experienced a scenario where all available PHP worker slots were full. Mostly caused by errant bots, this scenario typically results in a slow site in mild cases, or timeouts for clients in heavy cases. We started working on this problem in release 473. Now we roll out a solution.
After writing lots of code to be able to see inside PHPFPM, we finally were able to conclude that bad bots were about 80% of the problem. We split these bots into two categories: 1) Google, Bing, Pingdom, etc. and 2) all the others. Or: things you want and things you do not want.
All bots that are not well-known are rate-limited to one request per second. If they exceed this rate, they receive a 503 response, which has the meaning of “please come back later, we’re too busy for you”.
Note well: we do not rate-limit Google, nor Bing, nor Pingdom.
If you want to examine the requests that were rate limited, you can do so in the shell using the following command:
pnl –today –filter status=503
Many more tools and fixes
We’ve added a number of tools and fixed a few bugs:
-
- Magento Partner Brainstorm: We’ve added Magerun bash completion and the ZSH shell.
- You’ll get big red warnings in your shell if your custom Nginx configuration fails to load.
- Nginx upload body limit was too low for many sites. This limit is now increased to 100MB.
- All known HHVM bugs have now been fixed.
Edit: I misread the internal commit log. We just shuffled the fixes around.