In the past couple of months we have been very busy with overhauling our cloud provisioning automation and migration processes behind the scenes to provide a better experience when changing between Hypernode plans. While always being able to anticipate when you need to upgrade to a Hypernode with more resources to facilitate the growth of a shop and arranging it beforehand would be ideal, in practice it often turns out to be a different story.
Measures to minimize downtime during up – and downgrades
Unexpected peaks happen, shops go viral and even organic growth can cause a previously well-performing shop to hit an operational snag when some gradual tipping point is reached. This makes it very important that you can react swiftly and upgrade to a larger plan to accommodate the need for more computing power. In order to make the decision to upgrade as easy as it can be and leave everything except for ‘do I need more firepower yes or no’ out of consideration, the scaling process must be as seamless as possible.
Disk swap on larger Magento Excellence plans
Last month we vastly improved the downtime between migrations for larger Excellence plans by replacing the data synchronization step with cloud-level processes that detach and re-attach the external data-volume. By proverbially removing the hard-drive from a smaller Hypernode and shoving it directly into a larger Hypernode the variable migration time associated with having to copy the data from one node to the other was reduced to an almost constant amount of time.
Switch between development and production plan without downtime
Before that we made it so that you can switch between a development and production node of the same size in-place, without an actual migration being performed. The same goes for any product change that will result with the new node running on the same type of hardware as the original one. This also makes it so that you can convert a Magento Trial Hypernode to a Magento Grow without any downtime.
New: dedicated IP’s
Now as a third measure we have integrated Floating IPs at DigitalOcean and Elastic IP Addresses at Amazon EC2 into our cloud automation to provide a dedicated IP for all new Hypernodes. When you scale your Hypernode to a larger plan the IP address will be detached from the old instance and attached to the new instance. The result of this is that the IP of your node will not change anymore if you migrate to a new plan that is in the same datacenter as the old one.
Benefits of scaling without IP changes
Being able to upgrade a Hypernode without having to deal with an IP change has a couple of main advantages. The first is that the DNS will not have to be changed as often anymore. Previously we recommended a CNAME construction to prevent users from having to update their DNS every time they changed their product.
No DNS change needed (customer side)
Instead of setting an A record to point to the IP, using a CNAME to point to a domainname managed by us would make sure that your domain would point to the right server after a migration. But for various reasons this strategy can sometimes be undesirable, which caused people to maintain A records anyway.
In those situations an IP change could be a major hassle because the IP of the new node would not be known until the migration has been completed. Note that we still recommend the CNAME because under some circumstances the IP could still change, i.e. when you migrate from DigitalOcean to Amazon or between regions.
No DNS update needed (server side)
The second benefit is that during volume swap migrations or quick rsync migrations the DNS change could become the bottleneck. The new server would be up and running already but from the perspective of visitors on the website it would seem that the shop was down until the DNS update was propagated. For our larger Excellence plans this can bring the downtime during upgrades down to around one minute.
No need to whitelist new IP’s
Finally the most notable advantage is that having to whitelist the new IP in firewalls of external vendors is no longer an issue. Shops often make connections with external systems like ERP services. It is not uncommon for those services to be strictly firewalled requiring you to re-register the IP after a change. This scenario is particularly cumbersome because it might not be immediately apparent to the web developer if something broke. For example, without in-depth monitoring it might take a few days to notice that some external synchronization processes haven’t been happening.
The automated dedicated IP transfer
With this new functionality upgrading to a larger node becomes a trivial matter. When the downtime is minimal and the IP does not change there is not a lot left to worry about when deciding at a critical time if risking an upgrade would be worth it. There are some caveats though; it is not always possible to migrate without an IP change. There are some preconditions.
The product you are changing to must be in the same datacenter as the the your node is currently on. This means that upgrading between a DigitalOcean Hypernode to an Amazon Hypernode will still cause an IP change.
To illustrate, here are some diagrams to visualize how the process works. For instance, if you have an already existing Hypernode that was created before today it will not have a dedicated IP. It will have an IP that is bound to the instance and can not be attached to another larger Hypernode. Any new Hypernodes will automatically be provisioned with a dedicated IP. This means that if you now change the plan of an existing Hypernode from a DigitalOcean plan to any other DigitalOcean plan like Magento Professional the IP will still change.
DigitalOcean
|
.===========. | .===========. .====.
| hypernode |>-rsync->| hypernode |==| IP |
'===========' '===========' '===='
But then the second time the product is changed to a different in the same datacenter the migration will look something like this.
DigitalOcean
|
.====. .===========. | .===========. .====.
| IP |==| hypernode |>-rsync->| hypernode |==| IP |
'====' '===========' '===========' '===='
d DigitalOcean d
e | e
.====. t .===========. | .===========. t .====.
| IP | a | hypernode |>-rsync->| hypernode | a | IP |
'====' c '===========' '===========' c '===='
h h
DigitalOcean
---------------------->|<---------------------- .====. .===========. | .===========. .====. | IP | | hypernode |>-rsync->| hypernode | | IP |
'====' '===========' '===========' '===='
a DigitalOcean a
t | t
.====. t .===========. | .===========. t .====.
| IP | a | hypernode |>-rsync->| hypernode | a | IP |
'====' c '===========' '===========' c '===='
h h DigitalOcean
|
.====. .===========. | .===========. .====.
| IP |==| hypernode |>-rsync->| hypernode |==| IP |
'====' '===========' '===========' '===='
Because the node will now have a dedicated IP it can be transferred to a larger node during an upgrade. As mentioned, any new Hypernodes ordered after today will get a dedicated IP by default so those don’t need to be migrated once before they can migrate without an IP change.
Switch between providers and regions will lead to IP-change
Hypernode is a multi-cloud platform. Our products span multiple cloud providers and you can request your node to be moved to any DigitalOcean or Amazon region you like. Unfortunately dedicated IPs are not transferable between different datacenters. This means that when you upgrade your Hypernode to an Amazon plan like Excellence while you previously were on Professional at DigitalOcean a new IP will still be assigned.
In those cases the system will fall back to the regular migration process and proceed like before. Requesting to be switched to a different region will result in an IP change as well. Note that for example at DigitalOcean, ams2 and ams3 also count as different regions.
DigitalOcean Amazon
.====. .===========. .===========. .====.
| IP |==| hypernode |>-rsync->| hypernode |==| IP |
'====' '===========' '===========' '===='One minute downtime with volume swap migrations
In our previous post on volume swap migrations we’ve explained how detaching the volume from an instance and attaching it to a larger instance greatly sped up the migration process for our larger Amazon plans. While the volume swapping process can take less than a minute, the perceived downtime for customers was a bit longer.
Previously, after a volume swap the services would be up and running on the new machine in a very short amount of time (slightly dependent on how long it takes for the services to halt gracefully). There was one problem though. While processing could proceed on the new machine with very little interruption, the new bottle-neck for customer facing services was now the DNS TTL.
Visitors would be refreshing the web page after the crucial point in the migration and the DNS would still point them to the old instance. The new one would be up and running already and because the old instance is disabled after a migration, those visitors would end up getting a connection refused until the DNS expired and started pointing to the new node.
Because dedicated IPs completely cut out the DNS change phase in the migration process this is now no longer a problem and the downtime during a migration will be strictly from the moment the services stop on the old node and start on the new node. This makes the new migration process for the larger Excellence plans look something like this:
Amazon
|
.====. .==========. | .==========.
| IP |==| instance | | | instance |
'====' '==========' | '=========='
|| |
.========. |
| volume |
'========' d Amazon
e |
.====. t .==========. | .==========.
| IP | a | instance | | | instance |
'====' c '==========' | '=========='
h detach |
.========. |
| volume | --->
'========'
-----> Amazon a
| t
.==========. | .==========. t .====.
| instance | | | instance | a | IP |
'==========' | '==========' c '===='
| attach h
| .========.
| | volume |
| '========' Amazon
|
.==========. | .==========. .====.
| instance | | | instance |==| IP |
'==========' | '==========' '===='
| ||
| .========.
| | volume |
| '========'During our testing phase we have timed the downtime during these types of upgrades to be around one minute. The unavailability could take slightly longer depending on whether the services on the original node can be gracefully stopped in a timely manner. For example, if the server is very busy and MySQL is still processing a lot of transactions the downtime could be extended because of that.
Checking if your node has a dedicated IP
To make it more apparent whether or not your IP will change during an upgrade we have added a new Dedicated IP item on the plan change page in the service panel. This new entry will show a green checkmark if the node currently has a dedicated IP. If it does not have one yet, it will show a dash.
If both the current and the new plan have a checkmark and the new plan is at the same provider (in the same region) as the original plan, then you know the IP should not change.
This new feature will be available in the service panel next Tuesday. The dedicated IP transfer logic is already live in our backend systems.