In this release we have made a small change to our hypernode-log command-line tool (and livelog) which displays the tasks our automation system has and is performing on your Hypernode. We are changing the state reverted to background for the create_backup job.
The output of the hypernode-log command represents the status of Jobs on our internal JobBoard as they are queued, processed and completed. A job can have various states such as running, success and reverted. An example is:
app@uaikgo-hoithomastest-magweb-cmbl:~$ hypernode-log
ACTION START END STATE TASKS RUNNING
update_node 2021-02-24T11:04:52Z 2021-02-24T11:04:53Z running 2/3 all_update_node_to_update_flow
update_node 2021-02-23T12:06:09Z 2021-02-23T12:11:07Z success 3/3 finished
destroying_node 2021-02-22T17:42:05Z 2021-02-22T17:43:25Z success 7/7 finished
update_node 2021-02-22T16:43:15Z 2021-02-22T16:48:23Z success 3/3 finished
ensure_s3_backup_configured 2021-02-22T16:43:13Z 2021-02-22T16:43:51Z success 5/5 finished
ensure_monitoring_for_app 2021-02-22T16:43:12Z 2021-02-22T16:43:42Z success 6/6 finished
update_alerting 2021-02-22T16:43:12Z 2021-02-22T16:43:39Z success 2/2 finished
emergency_rescue 2021-02-22T16:41:49Z 2021-02-22T16:43:38Z success 56/56 finished
xgrade_app 2021-02-22T16:02:28Z 2021-02-22T16:43:13Z success 33/33 finished
update_node 2021-02-22T16:01:58Z 2021-02-22T16:03:59Z success 3/3 finished
create_backup 2021-02-21T18:00:40Z 2021-02-21T18:00:55Z success 2/2 finished
Having this information accessible is a great way for Hypernode users to get insight into what our cloud automation is doing behind the scenes. You can for example use this information to track where your Hypernode is in the upgrade or downgrade process when you scale your plan to different size. To automatically increase or decrease the computational capacity of a Hypernode we have complicated workflows that deal with many steps such as creating cloud resources like instances, volumes and floating IPs and either syncing the data of your store to the new environment or performing cloud operations such as detach and attaching storage volumes depending on what plan or cloud you’re migrating to.
Being able to spectate the progress of these processes might give you a better insight in how longs take or coordinate with our support department more efficiently if there are any issues. While this sort of transparency is one of the features that makes Hypernode unique and especially interesting for power users, it can sometimes lead to confusion because information like this about internal processes can be hard to interpret.
A situation that sometimes happens is that when our automation creates an EBS snapshot at AWS or Cinder snapshot in our OpenStack it can take a while for this operation to complete. The exact mechanism for a snapshot like that depends on the cloud provider but mostly the creation instant because of how snaphots are implemented (point-in-time, copy-on-write). It commonly does take some time however before the snapshot is ready to be used as a source to create a new volume.
Our automation waits for the resource to reach the state available. If for whatever reason it takes too long for this backup to become available our job processing system will stop waiting for the backup and continue with other jobs. There can be many reasons why it might take a while before the backup to become ready for use. The size of the disk, the mutations on the storage, whether or not this is the first snapshot or the utilization of the storage backend in the cloud can all affect how long it takes.
None of this is problematic but because of how our create_backup job is implemented you might have seen the alarming reverted state for this job. This generally does not mean the backup failed but that the exclusive lock for your Hypernode has been released so that other jobs may be processed (setting changes, plan upgraded, etc) and that the creation of the backup will continue in the background asynchronously.
app@uaikgo-hoithomastest-magweb-cmbl:~$ hypernode-log
ACTION START END STATE TASKS RUNNING
update_node 2021-02-24T11:04:52Z 2021-02-24T11:04:53Z running 2/3 all_update_node_to_update_flow
create_backup 2021-02-23T18:10:46Z 2021-02-23T18:11:47Z reverted 1/2 finished
To make things a bit more clear and less confusing, in this release we have changed the state for this specific job from reverted to background to indicate that in some cases the backup creation job continued in the background and that it did not necessarily fail as might have been previously concluded by the reverted message.
app@uaikgo-hoithomastest-magweb-cmbl:~$ hypernode-log | grep background
create_backup 2021-02-23T18:10:46Z 2021-02-23T18:11:47Z background 1/2 finished
create_backup 2021-02-22T18:11:35Z 2021-02-22T18:12:36Z background 1/2 finished
To see which backups are actually available for your hypernode you can use the hypernode-systemctl list_backups command. This command will display the available snapshots (as either periodic or instant backup). These can be attached as a new volume to your node using the hypernode-systemctl attach_backup command.
app@uaikgo-hoithomastest-magweb-cmbl:~$ hypernode-systemctl list_backups
Backup ID Type Created at Expires at
eae33123-a761-45e7-a1e1-e741b60f28c5 periodic 2021-02-18T19:00:40+01:00 2021-03-18T19:00:40+01:00
f60ab7d8-7361-4683-97ee-0ad6a850b2b9 periodic 2021-02-19T19:00:43+01:00 2021-02-26T19:00:43+01:00
dce77270-0dd2-4e41-913c-791604ce5930 periodic 2021-02-20T19:00:46+01:00 2021-02-27T19:00:46+01:00
8501bd3d-8d0f-41c0-8a19-e914d0df447d periodic 2021-02-21T19:00:40+01:00 2021-02-28T19:00:40+01:00
b14a24e6-53d2-4565-8ac5-fcbca65b138f instant 2021-02-22T17:39:40+01:00 2021-02-24T17:39:40+01:00
f10a810f-48d6-489f-a159-cd80caf72cbf periodic 2021-02-22T19:12:01+01:00 2021-03-01T19:12:01+01:00
0f127905-115b-4635-833a-30a3fedc67db periodic 2021-02-23T19:11:29+01:00 2021-03-02T19:11:29+01:00
Note that we have not changed the actual API output, if you have implemented an integration with our API directly your automation will not be affected. This change will be deployed to all Hypernodes over the course of the coming week.