Release-3405: new hypernode-importer features, tweaks in memory management

Today we deployed some changes to the hypernode-importer which might end up saving you some time when migrating a Magento shop to a Hypernode or a local hypernode-vagrant development environment.

• The --set-maintenance-on-source flag

This new option for the hypernode-importer enables you to automatically put the Magento installation you are importing from in maintenance mode before copying the data. This will prevent new from data being added while the transfer is already in progress. When you are performing a real site migration instead of only copying the shop this feature might be handy.

If no orders can be made after the database dump and the file synchronization have been started, there won’t be any changes that can potentially be left behind. When the import has been completed successfully the maintenance flag will be removed in the new Magento installation on the Hypernode you imported the shop to.

If the migration fails it will leave maintenance mode on both the origin and the destination until you sort out the problem and try the import again. When the import succeeds, the maintenance flag of the new Magento installation on the Hypernode will be removed.

$ hypernode-importer --host vdloo.hypernode.io --set-maintenance-on-source
...
Setting maintenance mode, source shop will go offline now!
...
Copying files from remote site: 22.6k [1.12k/s]
Configuring database and user
...
Ensuring the maintenance flag is removed on the new local copy of the site
...
Profit: import completed!

• Exit codes

We started adding error codes for common errors to supplement the human readable logging with some machine readable exit codes. This can be helpful for scripting and providing feedback to the caller of the importer. The current list of exit codes:

0    - No error, command succeeded
1    - Catchall for general errors
150  - User aborted the importer with a keyboard interrupt
151  - Circular import was attempted. Can not import the same Hypernode to itself
152  - Authentication failed. Could not log in with either key auth or the provided password
153  - Unsupported special characters in SSH password while running in safe mode
154  - Key authentication failed and no alternative password was provided
155  - No Magento installation found on remote host
156  - Can not access the database defined in app/etc/local.xml
157  - Unsupported special characters in database credentials while running in safe mode
158  - No file copying tool on the remote host. Please install rsync

Other changes to the hypernode-importer:

1. We added the innodb_monitor table to the ignored tables list, this will make the sync a bit faster.
2. If no Paramiko SSH connection can be made when a jump-host is configured in the user SSH config, the importer will try again without the tunnel (using -oProxyCommand=none)
3. The importer now copies symlinks as if they were normal files and directories. (rsync will transform symlinks into referent files/dirs)
   note: if your shop has a complicated symlink construction that point to external directories with files that are not symlinked but do belong to the project, the import might be incomplete.
4. Perform sanity after getting SSH connection to see if remote host has required file transfer tooling (rsync)

Other platform changes:

• Tweaks relating to memory management

We have tweaked the Hypernode Out Of Memory monitor a bit to make it take OOMkills in the root memory space into account. Previously in some situations it could happen that a program in the cgroup we have defined for the processes of the app user could push a program outside of that group over the edge when it makes a claim on some previously allocated virtual memory pages.

This could cause the kernel OOM heuristic to kill certain processes with a higher priority than some other processes (which we might actually rather kill to free up memory). This is an edge case that does not happen very often because the borders of the defined memory cgroup on Hypernode work pretty well, but in the case that it does happen we need to make sure we deal with the resulting situation.

Previously if something was OOMkilled in the app cgroup, we would based on the special file descriptor of that cgroup’s Notification API perform some careful preventive process killing to allow the system to recover with a lower change of having slain crucial processes. Why and how this works is explained more expansively in this previous changelog. But in the case of a kernel OOMkill outside of the cgroup, the preventive measures would not be triggered. To deal with this we’ve changed our out of memory monitor to also watch for events in the global cgroup (/sys/fs/cgroup/memory/cgroup.event_control), which will hopefully make the Hypernode platform even more stable.