In this release we have added functionality to the hypernode-api that allows users to easily and safely deploy some common NGINX on their Hypernode(s). The rules that we currently expose enable blocking known bot traffic and blocking traffic from some specific countries from which we’ve seen a lot of bot traffic but in which our customers mostly don’t do business.
On the Hypernode platform we have a sophisticated system for auto-healing which we call the emergency rescue. When our monitoring flags a node as down the system detects known issues and then tries to automatically fix them. Over time the list of issues that we can automatically detect and fix has grown because our on-call team proactively analyze patterns in previous incidents every week. When a common pattern emerges we try to automate it away so that the next time it will be fixed automatically.
A couple of the categories of issues that the emergency rescue can detect are certain types of excessive or unwanted bot traffic, brute-force attacks and vulnerability probes from hackers scouting for an attack vector. When a node goes down because it is possibly overloaded it makes sense to automatically crack down on unnecessary, unwanted and potentially dangerous traffic in order to prioritize servicing real customers with the server’s resources.
Of course there is traffic that we block across the board on all shops, but in some cases adaptive filtering is a better solution. For example, it might not make sense to block random SEO crawlers if you have enough server capacity anyway, but if those crawlers start getting in the way of real people visiting your site it is worth it to block them.
The emergency rescue process and these automated detection and blocking mechanisms are only triggered if your node goes down or is overloaded. Because of that there might still be cases where you might want to deploy any of the rules that we would automatically deploy otherwise yourself.
If for example you have a very large Hypernode with enough capacity to process anything a crawler throws at it, but that crawler does not even operate in the country you sell in, then our automation will not automatically block that traffic, while you still might want to block it yourself if you notice their requests filling up slots in livefpm (because then you might not need as much server resources).
So to make it easier for users to deploy these rules a POST can now be done to the block_attack endpoint of the hypernode-api to trigger an automated deployment of this NGINX configuration.
curl https://api.hypernode.com/v1/app/your_app_name/block_attack/ -X POST --data attack_name=attack_name --header "Authorization: Token your_hypernode_api_token"
For a list of possible values see:
curl https://api.hypernode.com/v1/app/block_attack_descriptions/ -X GET --header "Authorization: Token your_hypernode_api_token"
You can find your hypernode_api_token in /etc/hypernode/hypernode_api_token on your Hypernode.Note
The hypernode-systemctl command-line tool
To make using the hypernode-api more user-friendly we have implemented this functionality in the hypernode-systemctl CLI tool as well. To list the possible values you can run this on your Hypernode:
app@pup1l6-vdloo-magweb-cmbl:~$ hypernode-systemctl block_attack --help
usage: hypernode-systemctl block_attack BlockChinaBruteForce
The possible values are:
BlockUkraineBruteForce Attempts to deploy NGINX rules to block all IPs originating from Ukraine if not already configured. Also see https://www.hypernode.com/knowledgebase/block-your-site-for-specific-countries/
BlockRussiaBruteForce Attempts to deploy NGINX rules to block all IPs originating from Russia if not already configured. Also see https://www.hypernode.com/knowledgebase/block-your-site-for-specific-countries/
BlockAhrefsBot Attempts to deploy NGINX rules to block the AhrefsBot Web Crawler if not already configured. See https://www.hypernode.com/knowledgebase/fixing-bad-performance-caused-by-search-engines/
BlockSemrushBot Attempts to deploy NGINX rules to block the SEMrush Web Crawler if not already configured. See https://www.hypernode.com/knowledgebase/fixing-bad-performance-caused-by-search-engines/
BlockPhilippinesBruteForce Attempts to deploy NGINX rules to block all IPs originating from the Philippines if not already configured. Also see https://www.hypernode.com/knowledgebase/block-your-site-for-specific-countries/
BlockPageSpeedBruteForce Attempts to deploy NGINX rules to block known brute-force probes against URLs related to PageSpeed if not already configured
BlockGrapeshotBot Attempts to deploy NGINX rules to block the Grapeshot Web Crawler if not already configured. See https://www.hypernode.com/knowledgebase/fixing-bad-performance-caused-by-search-engines/
BlockMJ12Bot Attempts to deploy NGINX rules to block the MJ12Bot Web Crawler if not already configured. See https://www.hypernode.com/knowledgebase/fixing-bad-performance-caused-by-search-engines/
BlockDotBot Attempts to deploy NGINX rules to block the DotBot Web Crawler if not already configured. See https://www.hypernode.com/knowledgebase/fixing-bad-performance-caused-by-search-engines/
BlockChinaBruteForce Attempts to deploy NGINX rules to block all IPs originating from China if not already configured. Also see https://www.hypernode.com/knowledgebase/block-your-site-for-specific-countries/
BlockDownloaderBruteForce Attempts to deploy NGINX rules to block the Magento 1 /downloader endpoint. See https://www.hypernode.com/knowledgebase/how-to-protect-your-magento-store-against-brute-force/#2_Secure_downloader_and_rss
BlockRSSBruteForce Attempts to deploy NGINX rules to block known attacks against the RSS endpoint in Magento 1 if not already configured. See https://www.hypernode.com/changelog/release-5946-configurable-memory-management-policy-and-rss-bruteforce-detection/
BlockHongkongBruteForce Attempts to deploy NGINX rules to block all IPs originating from Hong Kong if not already configured. Also see https://www.hypernode.com/knowledgebase/block-your-site-for-specific-countries/
BlockRogerBot Attempts to deploy NGINX rules to block the Rogerbot Web Crawler if not already configured. See https://www.hypernode.com/knowledgebase/fixing-bad-performance-caused-by-search-engines/
positional arguments:
{BlockUkraineBruteForce,BlockRussiaBruteForce,BlockAhrefsBot,BlockSemrushBot,BlockPhilippinesBruteForce,BlockPageSpeedBruteForce,BlockGrapeshotBot,BlockMJ12Bot,BlockDotBot,BlockChinaBruteForce,BlockDownloaderBruteForce,BlockRSSBruteForce,BlockHongkongBruteForce,BlockRogerBot}
optional arguments:
-h, --help show this help message and exit
If for example you would then want to block all requests from China because you noticed some suspicious traffic from CN IPs in hypernode-fpm-status, then you can run:
app@pup1l6-vdloo-magweb-cmbl:~$ hypernode-systemctl block_attack BlockChinaBruteForce
Block attack job posted, see hypernode-log (or livelog) for job progress
After hypernode-log or livelog reports the newly posted block_attack job as finished the new rule should be deployed.
app@pup1l6-vdloo-magweb-cmbl:~$ hypernode-log | grep block_attack | tail -n 1
block_attack 2019-01-10T15:34:29Z 2019-01-10T15:34:31Z success 4/4 finished
The newly deployed NGINX rule will then be in the /data/web/nginx directory:
app@pup1l6-vdloo-magweb-cmbl:~$ cat nginx/server.block_cn
#Placed by Hypernode automation on 2019-01-11 12:18
if ($geoip_country_code = CN) { return 403; }
Safely deploying NGINX rules
Because this process is normally autonomously executed in our auto-healing automation we have made it so that the block attack job will only deploy the NGINX rules if it can do so without interfering with the previously configured NGINX rules.
Under the hood the nginx config reloader is used to ensure that whatever configuration we’re going to place does not cause a state where the new configuration will not validate. If the previously defined NGINX config was already invalid this process will do nothing as well. Also, when the file to be placed already exists the action will be skipped over. This means that if you later truncate the placed file you can prevent it from being (automatically) re-placed by the emergency rescue or by using this API endpoint.
These constraints make it possible for users to safely deploy these NGINX rules without intricate knowledge of NGINX or of the existing configuration. When the rules have been successfully deployed an email will be sent to the account owner to inform of this configuration change.
This new hypernode-systemctl option will be deployed over the course of the coming week.