In this release we have overhauled our hypernode-firewall and we’ve added a handy script for users to keep tabs on excessive MySQL memory usage.
Hypernode Firewall overhaul
In the past we have blogged about the self-learning Hypernode firewall. For all Hypernodes we aggregate data about suspicious requests in our central Elasticsearch so that we can automatically detect attacks on for example the Magento newsletter registration endpoint. We can then automatically block the assailants on the entire platform if the requests are sufficiently suspicious.
While that mechanism works great, there have been some cases where we wanted to whitelist specific IPs from ever being blocked again in the future. That functionality was already in the initial implementation, but in this new revision of the Hypernode Firewall we have made it so that instead of filtering the whitelisted IPs in our API-endpoint we do it directly in the server’s iptable. From the perspective of the user nothing is changed, but for us this will make the firewall a bit easier to maintain.
Release MySQL memory script
On smaller Hypernodes or on Hypernodes with databases with big datasets, over time sometimes too much memory is allocated by MySQL. The rest of the system will go in a state of undesirable memory pressure while MySQL is hogging a lot of memory that it might not even actively use. MySQL’s memory footprint can be configured with settings such as innodb_buffer_pool_size but correctly tweaking MySQL’s memory allocation is a complex topic and sometimes periodically restarting the service is the easiest solution.
If you have a Hypernode which is occasionally low on memory due to dormant MySQL caches building up after a while, it might be a smart move to periodically check if the server is below a certain memory free threshold and then restart the MySQL service. To facilitate that we have placed a script in /usr/local/bin/release_mysql_cached_memory on every Hypernode that users can configure to run in a cron if they have this problem.
app@levkcl-appname-magweb-cmbl:~$ cat /usr/local/bin/release_mysql_cached_memory
#!/bin/bash
read -r -d '' USAGE <&2 echo "Invalid action. $USAGE"; exit 1
esac
done
if [ -z $MEMFREETHRESHOLD ] || [ -z $COOLDOWNFORRESTART ]; then
>&2 echo "Invalid action. $USAGE"; exit 1
else
if ! find /tmp/release_mysql_cached_memory_cooldown -mmin -$COOLDOWNFORRESTART 2>/dev/null | grep -q .; then
echo "Checking if we're in a low memory situation"
if [ $(free -m | grep Mem | awk '{print$7}') -lt $MEMFREETHRESHOLD ]; then
touch /tmp/release_mysql_cached_memory_cooldown
echo "Low memory condition detected and we haven't restarted MySQL in the last $COOLDOWNFORRESTART minutes"
echo "Restarting MySQL..."
hypernode-servicectl restart mysql
else
echo "No low memory condition detected, have at least $MEMFREETHRESHOLD MB of mem available"
fi;
else
echo "Not checking low memory condition, already restarted MySQL because of low mem in the last $COOLDOWNFORRESTART minutes"
fi
fiYou can configure this script to run periodically with a cron like:
app@5viaxa-appname-magweb-cmbl:~$ crontab -l | grep release
*/5 * * * * /usr/local/bin/release_mysql_cached_memory -m 1024 -c 30
Then when the system reaches a configurable low watermark of memory available on the system, MySQL will automatically be restarted to free up some RAM. The second argument specifies a cooldown in minutes for how long there should be at least between any MySQL restart. Of course you don’t want to be restarting your database too often in a production environment.
In general you should not need to run a script like this to keep your memory usage at bay, but in some cases where you know that tweaking MySQL memory usage exactly correctly is difficult or if you’re working with limited resources this might be a good tool to have in your toolkit.