In this release we have made a small change to our hypernode-log command-line tool (and livelog) which displays the tasks our automation system has and is performing on your Hypernode. We are changing the state reverted to background for the create_backup job.

The output of the hypernode-log command represents the status of Jobs on our internal JobBoard as they are queued, processed and completed. A job can have various states such as running, success and reverted. An example is:

Having this information accessible is a great way for Hypernode users to get insight into what our cloud automation is doing behind the scenes. You can for example use this information to track where your Hypernode is in the upgrade or downgrade process when you scale your plan to different size. To automatically increase or decrease the computational capacity of a Hypernode we have complicated workflows that deal with many steps such as creating cloud resources like instances, volumes and floating IPs and either syncing the data of your store to the new environment or performing cloud operations such as detach and attaching storage volumes depending on what plan or cloud you’re migrating to.

Being able to spectate the progress of these processes might give you a better insight in how longs take or coordinate with our support department more efficiently if there are any issues. While this sort of transparency is one of the features that makes Hypernode unique and especially interesting for power users, it can sometimes lead to confusion because information like this about internal processes can be hard to interpret.

A situation that sometimes happens is that when our automation creates an EBS snapshot at AWS or Cinder snapshot in our OpenStack it can take a while for this operation to complete. The exact mechanism for a snapshot like that depends on the cloud provider but mostly the creation instant because of how snaphots are implemented (point-in-time, copy-on-write). It commonly does take some time however before the snapshot is ready to be used as a source to create a new volume.

Our automation waits for the resource to reach the state available. If for whatever reason it takes too long for this backup to become available our job processing system will stop waiting for the backup and continue with other jobs. There can be many reasons why it might take a while before the backup to become ready for use. The size of the disk, the mutations on the storage, whether or not this is the first snapshot or the utilization of the storage backend in the cloud can all affect how long it takes.

None of this is problematic but because of how our create_backup job is implemented you might have seen the alarming reverted state for this job. This generally does not mean the backup failed but that the exclusive lock for your Hypernode has been released so that other jobs may be processed (setting changes, plan upgraded, etc) and that the creation of the backup will continue in the background asynchronously.

To make things a bit more clear and less confusing, in this release we have changed the state for this specific job from reverted to background to indicate that in some cases the backup creation job continued in the background and that it did not necessarily fail as might have been previously concluded by the reverted message.

To see which backups are actually available for your hypernode you can use the hypernode-systemctl list_backups command. This command will display the available snapshots (as either periodic or instant backup). These can be attached as a new volume to your node using the hypernode-systemctl attach_backup command.

Note that we have not changed the actual API output, if you have implemented an integration with our API directly your automation will not be affected. This change will be deployed to all Hypernodes over the course of the coming week.