In this release we have made a small change to our hypernode-log command-line tool (and livelog) which displays the tasks our automation system has and is performing on your Hypernode. We are changing the state reverted to background for the create_backup job.

The output of the hypernode-log command represents the status of Jobs on our internal JobBoard as they are queued, processed and completed. A job can have various states such as running, success and reverted. An example is:


app@uaikgo-hoithomastest-magweb-cmbl:~$ hypernode-log
ACTION                          START                   END                     STATE       TASKS   RUNNING
update_node                     2021-02-24T11:04:52Z    2021-02-24T11:04:53Z    running     2/3     all_update_node_to_update_flow
update_node                     2021-02-23T12:06:09Z    2021-02-23T12:11:07Z    success     3/3     finished
destroying_node                 2021-02-22T17:42:05Z    2021-02-22T17:43:25Z    success     7/7     finished
update_node                     2021-02-22T16:43:15Z    2021-02-22T16:48:23Z    success     3/3     finished
ensure_s3_backup_configured     2021-02-22T16:43:13Z    2021-02-22T16:43:51Z    success     5/5     finished
ensure_monitoring_for_app       2021-02-22T16:43:12Z    2021-02-22T16:43:42Z    success     6/6     finished
update_alerting                 2021-02-22T16:43:12Z    2021-02-22T16:43:39Z    success     2/2     finished
emergency_rescue                2021-02-22T16:41:49Z    2021-02-22T16:43:38Z    success     56/56   finished
xgrade_app                      2021-02-22T16:02:28Z    2021-02-22T16:43:13Z    success     33/33   finished
update_node                     2021-02-22T16:01:58Z    2021-02-22T16:03:59Z    success     3/3     finished
create_backup                   2021-02-21T18:00:40Z    2021-02-21T18:00:55Z    success     2/2     finished

Having this information accessible is a great way for Hypernode users to get insight into what our cloud automation is doing behind the scenes. You can for example use this information to track where your Hypernode is in the upgrade or downgrade process when you scale your plan to different size. To automatically increase or decrease the computational capacity of a Hypernode we have complicated workflows that deal with many steps such as creating cloud resources like instances, volumes and floating IPs and either syncing the data of your store to the new environment or performing cloud operations such as detach and attaching storage volumes depending on what plan or cloud you’re migrating to.

Being able to spectate the progress of these processes might give you a better insight in how longs take or coordinate with our support department more efficiently if there are any issues. While this sort of transparency is one of the features that makes Hypernode unique and especially interesting for power users, it can sometimes lead to confusion because information like this about internal processes can be hard to interpret.

A situation that sometimes happens is that when our automation creates an EBS snapshot at AWS or Cinder snapshot in our OpenStack it can take a while for this operation to complete. The exact mechanism for a snapshot like that depends on the cloud provider but mostly the creation instant because of how snaphots are implemented (point-in-time, copy-on-write). It commonly does take some time however before the snapshot is ready to be used as a source to create a new volume.

Our automation waits for the resource to reach the state available. If for whatever reason it takes too long for this backup to become available our job processing system will stop waiting for the backup and continue with other jobs. There can be many reasons why it might take a while before the backup to become ready for use. The size of the disk, the mutations on the storage, whether or not this is the first snapshot or the utilization of the storage backend in the cloud can all affect how long it takes.

None of this is problematic but because of how our create_backup job is implemented you might have seen the alarming reverted state for this job. This generally does not mean the backup failed but that the exclusive lock for your Hypernode has been released so that other jobs may be processed (setting changes, plan upgraded, etc) and that the creation of the backup will continue in the background asynchronously.


app@uaikgo-hoithomastest-magweb-cmbl:~$ hypernode-log
ACTION                          START                   END                     STATE       TASKS   RUNNING
update_node                     2021-02-24T11:04:52Z    2021-02-24T11:04:53Z    running     2/3     all_update_node_to_update_flow
create_backup                   2021-02-23T18:10:46Z    2021-02-23T18:11:47Z    reverted    1/2     finished

To make things a bit more clear and less confusing, in this release we have changed the state for this specific job from reverted to background to indicate that in some cases the backup creation job continued in the background and that it did not necessarily fail as might have been previously concluded by the reverted message.


app@uaikgo-hoithomastest-magweb-cmbl:~$ hypernode-log | grep background
create_backup                   2021-02-23T18:10:46Z    2021-02-23T18:11:47Z    background  1/2     finished
create_backup                   2021-02-22T18:11:35Z    2021-02-22T18:12:36Z    background  1/2     finished

To see which backups are actually available for your hypernode you can use the hypernode-systemctl list_backups command. This command will display the available snapshots (as either periodic or instant backup). These can be attached as a new volume to your node using the hypernode-systemctl attach_backup command.


app@uaikgo-hoithomastest-magweb-cmbl:~$ hypernode-systemctl list_backups
Backup ID                             Type      Created at                 Expires at                 
eae33123-a761-45e7-a1e1-e741b60f28c5  periodic  2021-02-18T19:00:40+01:00  2021-03-18T19:00:40+01:00  
f60ab7d8-7361-4683-97ee-0ad6a850b2b9  periodic  2021-02-19T19:00:43+01:00  2021-02-26T19:00:43+01:00  
dce77270-0dd2-4e41-913c-791604ce5930  periodic  2021-02-20T19:00:46+01:00  2021-02-27T19:00:46+01:00  
8501bd3d-8d0f-41c0-8a19-e914d0df447d  periodic  2021-02-21T19:00:40+01:00  2021-02-28T19:00:40+01:00  
b14a24e6-53d2-4565-8ac5-fcbca65b138f  instant   2021-02-22T17:39:40+01:00  2021-02-24T17:39:40+01:00  
f10a810f-48d6-489f-a159-cd80caf72cbf  periodic  2021-02-22T19:12:01+01:00  2021-03-01T19:12:01+01:00  
0f127905-115b-4635-833a-30a3fedc67db  periodic  2021-02-23T19:11:29+01:00  2021-03-02T19:11:29+01:00 

Note that we have not changed the actual API output, if you have implemented an integration with our API directly your automation will not be affected. This change will be deployed to all Hypernodes over the course of the coming week.