Release 5946: Configurable memory management policy and RSS bruteforce detection

Configurable memory management policy

In this release we will amend the change we made earlier this month where we increased the limit for our non-essential process slaying mechanism in order to allow more one-off memory hungry processes to run without executing overly drastic measures. While this change worked as intended, it did however result in processes that would later become memory problems to be prevented from being culled in an early stage in shops with a certain type of memory footprint.

To get the best of both worlds we now offer a configurable setting that can be set using the hypernode-api or using the hypernode-systemctl command-line tool from the Hypernode. The default will be switched back to the way it was from before November the second because on average that seems to be the most effective setting. For a select number of Hypernodes where we observed the settings from the past month to be more effective we have toggled this setting to preserve the optimal behavior.

If you want to enable or disable this setting yourself, you can do so like this. But before you do, be sure to read this previous changelog for an in-depth explanation of what the setting exactly entails.

$ hypernode-systemctl settings permissive_memory_management --value True
Operation was successful and is being processed. Please allow a few minutes for the settings to be applied. Run 'livelog' to see the progress.

Note that even though this setting can give you some more leeway in regards to memory utilization on Hypernode and gives you the option to make a decision whether you value keeping the site online at the cost of killing non-essential processes early versus trading some risk in terms of stability for increasing the chance of memory hungry one-off processes to complete, if you notice structural Out Of Memory messages in your kern.log that often indicates a real problem in the shop or that it might be time to upgrade.

Automated RSS brute-force blocking

In other news, we have expanded our brute-force detection to detect RSS brute-forcing in case a node goes down due to high load. Our emergency rescue automation will now detect patterns in the PHP-FPM process list that indicate an RSS brute-force and will attempt to deploy a rule in NGINX to block the attack if possible without conflicting with the customer defined configuration. If that happens, you will receive a mail like this:

In January we removed mitigation for SUPEE-6285 on account of these mitigation rules causing configuration issues for various users and the vulnerability being long since patched (or taken care of in a user-level configuration). That mitigation included blocking known brute-forcible RSS endpoints in Magento 1.

But even though there is no longer a critical security threat for this three year old vulnerability, we still see more than one million probes per day on our platform. While these hack-attempts are relatively unsophisticated, blocking them still has some merit. The reason for that is that if NGINX does not block these requests at the front door, they might end up in PHP-FPM, wasting FPM slots and other precious server resources.

Having these bots enter Magento and receive a redirect to the front-page there or getting 404’d instead of blocking them before even reaching the application can have serious performance implications if there are other problems in the shop that prevent a rapid throughput on the limited amount of FPM workers. For this reason we will deploy this user-configurable NGINX configuration on any Hypernode where we detect an active brute-force during an emergency rescue. Additionally, this configuration will be deployed on any newly delivered Hypernode where it can be disabled by the user if desired.

cat /data/web/nginx/before_redir.rss_bruteforce
# Hypernode Platform RSS Brute force
# Only allow whitelisted countries on the lowercase (normal usage) URLs. Bots often try capitalized combinations.
location ^~ /rss/order/new {
    if ($geoip_country_code ~ (NL|BE|GB|DE|US) ) {
        echo_exec @handler;
    }
    return 403;
}
location ^~ /rss/catalog/notifystock {
    if ($geoip_country_code ~ (NL|BE|GB|DE|US) ) {
        echo_exec @handler;
    }
    return 403;
}
location ^~ /rss/catalog/review {
    if ($geoip_country_code ~ (NL|BE|GB|DE|US) ) {
        echo_exec @handler;
    }
    return 403;
}
location ~* /rss/order/new {
    return 403;
}
location ~* /rss/catalog/notifystock {
    return 403;
}
location ~* /rss/catalog/review {
    return 403;
}

For now we will not add this configuration to existing Hypernodes unless it is being ’emergency rescued’ to prevent unnecessary changes during the holiday season but we will place a commented out version of /data/web/nginx/before_redir.rss_bruteforce on all nodes so this rule can easily be enabled if that is preferred.

Changes will be deployed over the course of the next seven days.