Multi-threaded cache warmer
In this release we’ve once again deployed a new version of magerun-hypernode. This time with a great new feature by the people at Tickles. This update makes the hypernode cache warmer multi-threaded so it can warm the cache for multiple pages at the same time. To specify the amount of concurrent requests to use, pass the new --batchsize flag.
magerun hypernode:performance --help
Usage:
hypernode:performance [options]
Options:
...
--batchsize[=BATCHSIZE] Batch size for multithreading [default: false]
This can greatly speed up the time it takes to get your site running at full capacity again after flushing the cache. Note that for Hypernodes there is a limit on the amount of concurrent requests a single IP can do before being rate-limited. This limit is at this moment of writing according to the following formula:
vcpus * 5 - 2Continuous deployment of the Hypernode control code
Our development process for Hypernode is vastly automated. We have one integrated pipeline all the way from submitting a change to the version control system to actually deploying code on production systems. Previously there were some steps that we chose to perform manually to limit the influence of side effects and provide great control over when a feature would go live. But over time it became clear that because of our rigorous automated testing environments, in practice pressing the actual deploy button was merely a mechanical step that no longer serves us a purpose in day to day operations.
In this release we have deployed two changes that are invisible to the user but will further automate our development workflow so we can make changes and push them to production faster and with even less effort than before.
The first one is that all merges to master of our control code will now be automatically be deployed to the production environment without human intervention. The second change is that every Hypernode will be assigned a deterministic and designated time slot in which an automatic system update will take place.
By removing this last step of manual system wide deployment we now employ full continuous deployment. Of course in case of particularly impactful changes we retain the flexibility to take over manually to exert greater precision.
More graceful package upgrades in low memory conditions
We have made some changes to the system that will allow interrupted package updates to proceed more gracefully. Sometimes when packages are being upgraded but the Hypernode is too busy to process the upgrade, the process can be interrupted in a way that will leave behind an inconsistent state preventing further updates.
This could result in situations where user settings and changes could not be propagated to the system until the inconsistent state was cleaned up. We have changed our playbooks to take into account more of these states so we can resolve them automatically more often. This change will speed up deployment on some nodes under certain circumstances.
Other changes
- PHP 7 update
PHP 7 has been updated from 7.0.23 to 7.0.24. You can find the changelog here.
- The benchmark tool will now only benchmark category and CMS pages for Magento 2.
Listing the product pages seemed to cause problems with some site configurations, so for now we will limit the benchmarkable URLs to these two URL types to increase compatibility.
- install python3-pip
The package pip3 is now installed on all Hypernodes. Note that we do not expose a virtualenvwrapper at this moment so all modules you install will be installed system-wide for your user.
- temp cleaner is now flocked
There is a process that cleans stale temporary files on each Hypernode. This periodic task was not locked per run, meaning that it could run multiple times on large nodes with a large amount of temp files if each run took longer than the interval between runs, causing a cascading effect. This has now been remedied.
All changes will be deployed gradually over all Hypernodes.