Whilst trying to restore a normal backup job from one node to another in another part of the cluster, the following is experienced:
Warning: volblocksize (4096) is less than the default minimum block size (8192). To reduce wasted space a volblocksize of 8192 is recommended. new volume ID is 'nas01:vm-113-disk-0' iscsiadm: No session found. rescan volumes... iscsiadm: No session found. VM is locked (create) starting VM for live-restore repository: 'pbs-backup@[email protected]:backup05', snapshot: 'vm/139/2024-06-07T11:46:25Z' restoring 'drive-scsi0' to 'nas01:vm-113-disk-0' iscsiadm: No session found. iscsiadm: No session found. kvm: -drive file.file.filename=iscsi://a.b.c.d/iqn.2005-10.org.freenas.ctl:target/8,if=none,id=drive-scsi0,format=alloc-track,file.driver=raw,cache=none,aio=io_uring,file.detect-zeroes=on,backing=drive-scsi0-pbs,auto-remove=on: Could not open 'iscsi://102.216.79.179/iqn.2005-10.org.freenas.ctl:target/8': No such file or directory An error occurred during live-restore: start failed: QEMU exited with code 1 error before or during data restore, some or all disks were not completely restored. VM 113 state is NOT cleaned up. TASK ERROR: live-restore failed
Let’s dissect this problem:
- “Live restore” was ticked.
- The actual error is
kvm: -drive file.file.filename=iscsi://a.b.c.d/iqn.2005-10.org.freenas.ctl:target-example-name/8,if=none,id=drive-scsi0,format=alloc-track,file.driver=raw,cache=none,aio=io_uring,file.detect-zeroes=on,backing=drive-scsi0-pbs,auto-remove=on: Could not open 'iscsi://a.b.c.d/iqn.2005-10.org.freenas.ctl:target/8': No such file or directory
If you want to narrow that down further, the actual error is “No such file or directory”
Here is the initial analysis based on the multiple `iscsiadm: No session found.` messages:
- The source host tries to connect to the remote host.
- To establish a live restore, first bring up the iscsiadm service to look for disk data
- Try again, a few times, eventually giving up.
In the end, bypassing live restore made it work, but unfortunately the whole idea is a live restore so we give up for now.
Documentation:
- https://pve.proxmox.com/pve-docs/chapter-vzdump.html#_live_restore
- https://pve.proxmox.com/wiki/Backup_and_Restore#vzdump_restore