In the past month we re-wrote the hypernode-importer from scratch to fix a couple of issues, added some new functionalities and made it more efficient. Now it supports Magento 2, handles special characters in database passwords and imports the database and file-system in parallel.
Hypernode-importer3 – loads of new options
The original hypernode-importer will still be available on all Hypernodes, because some previously existing functionality was omitted from the new version. The new version of the importer has the name hypernode-importer3, you can check out all available options by running:
$ hypernode-importer3 --help
usage: hypernode-importer3 [-h] [--host HOST] [--port PORT] [--user USER]
[--path PATH] [--set-default-url]
[--skip-db-import] [--skip-file-import] [--dry-run]
[--skip-autodetect]
[--protected-file PROTECTED_FILE]
[--use-settings-file USE_SETTINGS_FILE] [--force]
[--single-threaded] [--verbose | --quiet]
optional arguments:
-h, --help show this help message and exit
--host HOST SSH hostname to fetch site from
--port PORT SSH port to fetch site from
--user USER Username used for SSH connection
--path PATH Remote path to look for the Magento directory
--set-default-url Set secure URL of default storefront and admin area to
http://vdloo.hypernode.io
--skip-db-import Skip importing the database. Might be handy if you
only want to sync files
--skip-file-import Skip importing the webroot. Might be handy if you only
want to sync the db
--dry-run Don't really import but check if the source is OK
--skip-autodetect Do not attempt to find Magento in the default
locations if no Magento can be found in the provided
path. The default locations we look in are ['$HOME',
'/var/www', '/var/html', '/www']
--protected-file PROTECTED_FILE
An optional file to pass as a 'protect' filter to
rsync to exclude files during the sync
--use-settings-file USE_SETTINGS_FILE
Parse a json file for keys and values that will be
used as arguments for the importer. Can bee handy for
scripting purposes. Example content: {"path":
"/data/web/public"}
--force Overwrite any local changes. If you run hypernode-
importer, edit files locally and run the import again,
all newer local files would be untouched by default.
This flag overwrites that and imports remote files
over newer local changes.
--single-threaded Normally we import the database and the filesystem at
the same time. If this is too rough on the source or
destination server you can choose to run those steps
serially instead of in parallel. This will take longer
but decrease the load on the servers.
--verbose, -v Display debug output
--quiet, -q Do not display debug output
Changelog
- Added Magento 2 import functionality
- Using special characters in database passwords is now supported
- By default, the database and file-system import now run in parallel to save time
- If you do not want to impose extra load on the servers by running the imports in parallel you can use new the
--single-threadedflag to run the database and file-system import sequentially like before - Previously the file-system synchronization always overwrote files on the destination. Now by default all newer files on the destination node remain untouched. This means you can import once, update a config file like
local.xmland then re-sync without losing your local changes - If you do want to overwrite the files on re-import like before, you can specify the new
--forceflag - The importer now runs on Python 3.5
- Automatically attempting to import peripheral WordPress databases has been removed
- The
--import-from-local-filesystemflag has not been implemented inhypernode-importer3 - The
--use-sql-for-dbflag has not been implemented inhypernode-importer3 - The
--set-maintenance-on-sourceflag has not been implemented inhypernode-importer3 - The new importer does not try to automatically fix your
Cm_Cache_Backend_Redisconfig anymore - Our Hyperformance service can now handle Magento 2 applications
- The Byte Service Panel importer can now also handle Magento 2 imports (this will be rolled out later this week). Note that the Service Panel benchmark tool does not work for Magento 2 yet.
- We backported python3-paramiko 2.0.0-1 for various bugfixes and new features like ECDSA support and other newer ciphers
- Nonstandard ports using configured netcat proxycommands will now also be rewritten
Netcat proxycommands are now supported
Previously our SSH config ProxyCommand support in use with nonstandard SSH ports could only parse new style builtin SSH secure channel forwards like:
# .ssh/config example
Host tim.hypernode.io
ProxyCommand ssh -W %h:%p app@vdloo.hypernode.io
# Is translated to a nonstandard port if specified in the importer
ssh -W vdloo.hypernode.io:2222 app@tim.hypernode.ioNow the new importer also supports netcat proxycommands:
# .ssh/config example
Host vdloo.hypernode.io
ProxyCommand ssh tim.hypernode.io nc %h %p 2> /dev/null
# Is now also translated to a nonstandard port if specified in the importer
ssh tim.hypernode.io nc vdloo.hypernode.io 2222 2> /dev/nullNew importer, new approach
In the new importer the approach to connecting to the remote SSH server and the database port has been overhauled to perform all operations on the node on which the importer runs. Previously more commands were executed on the remote host, but that proved to be problematic in some cases because the environment on the remote end could vary greatly requiring a lot of special casing in the import logic. In the new importer direct operations on the remote server have been isolated and minimized to provide greater compatibility.
This is achieved by forwarding the remote services to the localhost on the importing server so that local tooling can be used to interact with the remote database directly as if it were connecting to it from the shop using the database itself. In the original hypernode-importer this type of functionality was also implemented but only served as a secondary fallback mechanism, in the new importer this is the primary means of importing.
Different scenarios
To illustrate, these are some of the scenarios the importer is now capable of operating on.
- Importing from a remote server with a local database that is only locally accessible on the remote server
Internet
|
.====. | .====.
| db | | | db |
'====' | '===='
|| | ||
.=====. | .=====.
| src |-->--| dst |
'=====' | '====='
|
- Importing from a remote server with a local database that is publicly accessible over the internet
Internet
|
.====. | .====.
| db |-->|---| db |
'====' | '===='
|| | ||
.=====. | .=====.
| src |-->--| dst |
'=====' | '====='
|
- Importing from a remote server with a remote database that is only locally accessible on the remote server
Intranet Internet
| |
| | .====.
| | | db |
| | '===='
| | ||
.====. | .=====. | .=====.
| db |-->--| src |-->--| dst |
'====' | '=====' | '====='
| |
- Importing from a remote server with a remote database that is publicly accessible over the internet
Internet Internet
| |
| | .====.
.---->----------->---| db |
|| | | '===='
|| | | ||
.====. | .=====. | .=====.
| db |-->--| src |-->--| dst |
'====' | '=====' | '====='
| |
- Importing from a remote server with a local database that has fire-walled access to a specific IP
Internet Internet
| |
.====. | | .====.
| db | | | | db |
'====' | | '===='
|| | | ||
.=====. | .======. | .=====.
| src |-->--| prox |-->--| dst |
'=====' | '======' | '====='
| |
- Importing from a remote server with a local database that is publicly accessible, but SSH access is fire-walled to a specific IP
Internet Internet
| |
.====. | | .====.
| db |--->------------>---| db |
'====' | | '===='
|| | | ||
.=====. | .======. | .=====.
| src |-->--| prox |-->--| dst |
'=====' | '======' | '====='
| |
- Importing from remote server with a remote database that is not publicly accessible, and SSH access is fire-walled to a specific IP
Intranet Internet Internet
| | |
| | | .====.
| | | | db |
| | | '===='
| | | ||
.====. | .=====. | .======. | .=====.
| db |-->--| src |-->--| prox |-->--| dst |
'====' | '=====' | '======' | '====='
| | |
- Importing from a remote server with a remote database that is publicly accessible, but SSH access is fire-walled to a specific IP
Internet Internet Internet
| | |
| | | .====.
.---->----------->------------>---| db |
|| | | | '===='
|| | | | ||
.====. | .=====. | .======. | .=====.
| db |-->--| src |-->--| prox |-->--| dst |
'====' | '=====' | '======' | '====='
| | |
The scenarios where the remote database port is forwarded require that the remote SSH server does not have TCP forwarding disabled. You can check your sshd config for AllowTcpForwarding no. For information about using the hypernode-importer through a proxy, check out this earlier changelog.
New return codes
If you are writing deployment scripts around the hypernode-importer, you might be interested in the newly added return codes. These signify a specific type of import failure. This is a complete list of available return codes, the bottom 9 have been newly added in hypernode-importer3.
NOERROR_RC = 0
CATCHALL_RC = 1
USER_ABORTED_RC = 150
CIRCULAR_IMPORT_RC = 151
SSH_AUTH_FAIL_RC = 152
SSH_SPECIAL_CHAR_RC = 153
SSH_NO_PASS_RC = 154
NO_MAG_RC = 155
NO_LOCAL_DB_ACCESS_RC = 156
DB_SPECIAL_CHAR_RC = 157
NO_FILE_COPYING_TOOL_RC = 158
MAG_2_NOT_SUPPORTED_RC = 159
SSH_TIMEOUT_RC = 160
REDIS_FLUSH_ERR_RC = 161
MAGE_FLUSH_ERR_RC = 162
NO_SUCH_DNS_RC = 163
NO_REMOTE_DB_ACCESS_RC = 164
SSH_NO_ROUTE_RC = 165
NO_LOCAL_DB_CONN_RC = 166
REMOTE_DB_CONF_NOT_FOUND_RC = 167
REMOTE_DB_CONF_PARSE_ERR_RC = 168
CAN_NOT_SYNC_FILES_RC = 169
NO_REMOTE_DB_CONN_RC = 170
DB_NAME_SPECIAL_CHAR_RC = 171
DB_TABLE_PREFIX_SPECIAL_CHAR_RC = 172
REMOTE_DB_M2_CONF_PARSE_ERR_RC = 173We will be deploying this new version of the importer later today.