Migrating VM instances fails using arm64 hosts

[yadvr]

problem

Migrating systemvms (CPVM/SSVM) from arm64 local-storage based host throws this error using 4.22/RC3:

2025-11-02 00:24:01,001 DEBUG [c.c.a.t.Request] (AgentManager-Handler-1:[]) (logid:) Seq 233-8103101629546364942: Processing:  { Ans: , MgmtId: 104339268210986, via: 233, Ver: v1, Flags: 110, [{"com.cloud.agent.api.Answer":{"result":"false","details":"com.cloud.utils.exception.CloudRuntimeException: Could not fetch storage pool 63229d40-f348-4636-8707-74ab46ab28d0 from libvirt due to org.libvirt.LibvirtException: Storage pool not found: no storage pool with matching uuid '63229d40-f348-4636-8707-74ab46ab28d0'
        at com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.getStoragePool(KVMStoragePoolManager.java:286)
        at com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.getStoragePool(KVMStoragePoolManager.java:272)
        at com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.disconnectPhysicalDisksViaVmSpec(KVMStoragePoolManager.java:247)
        at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtPrepareForMigrationCommandWrapper.handleRollback(LibvirtPrepareForMigrationCommandWrapper.java:160)
        at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtPrepareForMigrationCommandWrapper.execute(LibvirtPrepareForMigrationCommandWrapper.java:59)
        at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtPrepareForMigrationCommandWrapper.execute(LibvirtPrepareForMigrationCommandWrapper.java:50)
        at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)
        at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:2280)
        at com.cloud.agent.Agent.processRequest(Agent.java:813)
        at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1295)
        at com.cloud.utils.nio.Task.call(Task.java:83)
        at com.cloud.utils.nio.Task.call(Task.java:29)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:840)
","wait":"0","bypassHostMaintenance":"false"}}] }

Later also causes NPE:

2025-11-02 00:24:03,278 INFO  [o.a.c.s.m.KvmNonManagedStorageDataMotionStrategy] (Work-Job-Executor-19:[ctx-b7d81f9a, job-15822/job-15823, ctx-a65e4c89]) (logid:95b78c81) Expunging dest volume [id: 1547, state: Ready] as part of failed VM migration with volumes command for VM [1161].
2025-11-02 00:24:03,279 DEBUG [o.a.c.s.v.VolumeServiceImpl] (Work-Job-Executor-19:[ctx-b7d81f9a, job-15822/job-15823, ctx-a65e4c89]) (logid:95b78c81) Failed to copy volume java.lang.NullPointerException: Cannot invoke "com.cloud.storage.VolumeVO.getId()" because "this.volumeVO" is null
        at org.apache.cloudstack.storage.volume.VolumeObject.stateTransit(VolumeObject.java:243)
        at org.apache.cloudstack.storage.volume.VolumeObject.processEvent(VolumeObject.java:413)
        at org.apache.cloudstack.storage.motion.StorageSystemDataMotionStrategy.copyAsync(StorageSystemDataMotionStrategy.java:2199)
        at org.apache.cloudstack.storage.motion.DataMotionServiceImpl.copyAsync(DataMotionServiceImpl.java:136)
        at org.apache.cloudstack.storage.volume.VolumeServiceImpl.migrateVolumes(VolumeServiceImpl.java:2356)
        at org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.migrateVolumes(VolumeOrchestrator.java:1514)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:569)

2025-11-02 00:24:03,364 ERROR [c.c.v.VmWorkJobDispatcher] (Work-Job-Executor-19:[ctx-b7d81f9a, job-15822/job-15823]) (logid:95b78c81) Unable to complete AsyncJob {"accountId":2,"cmd":"com.cloud.vm.VmWorkMigrateWithStorage","cmdInfo":"rO0ABXNyACVjb20uY2xvdWQudm0uVm1Xb3JrTWlncmF0ZVdpdGhTdG9yYWdlsew9z6UxtXMCAANKAApkZXN0SG9zdElkSgAJc3JjSG9zdElkTAAMdm9sdW1lVG9Qb29sdAAPTGphdmEvdXRpbC9NYXA7eHIAE2NvbS5jbG91ZC52bS5WbVdvcmufmbZW8CVnawIABEoACWFjY291bnRJZEoABnVzZXJJZEoABHZtSWRMAAtoYW5kbGVyTmFtZXQAEkxqYXZhL2xhbmcvU3RyaW5nO3hwAAAAAAAAAAIAAAAAAAAAAgAAAAAAAASJdAAZVmlydHVhbE1hY2hpbmVNYW5hZ2VySW1wbAAAAAAAAADpAAAAAAAAALNzcgARamF2YS51dGlsLkhhc2hNYXAFB9rBwxZg0QMAAkYACmxvYWRGYWN0b3JJAAl0aHJlc2hvbGR4cD9AAAAAAAAAdwgAAAAQAAAAAHg","cmdVersion":0,"completeMsid":null,"created":"Sun Nov 02 00:23:46 IST 2025","id":15823,"initMsid":104339268210986,"instanceId":null,"instanceType":null,"lastPolled":null,"lastUpdated":null,"processStatus":0,"removed":null,"result":null,"resultCode":0,"status":"IN_PROGRESS","userId":2,"uuid":"f4950b59-4f53-47dc-856a-5abda1a15964"}, job origin: 15822 com.cloud.utils.exception.CloudRuntimeException: Failed to migrate VM [VM instance {"id":1161,"instanceName":"s-1161-VM","state":"Migrating","type":"SecondaryStorageVm","uuid":"ff0368d4-e71c-4e49-ad70-c7dab8ef7ea9"}] along with its volumes due to [java.lang.NullPointerException: Cannot invoke "com.cloud.storage.VolumeVO.getId()" because "this.volumeVO" is null].
        at org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.migrateVolumes(VolumeOrchestrator.java:1520)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:569)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)

On the target host, this exists:

root@cloudpi:/home/rohit# virsh pool-dumpxml cc155507-f2ba-4a96-a9b9-2dac9dc50a24
<pool type='dir'>
  <name>cc155507-f2ba-4a96-a9b9-2dac9dc50a24</name>
  <uuid>cc155507-f2ba-4a96-a9b9-2dac9dc50a24</uuid>
  <capacity unit='bytes'>1969186340864</capacity>
  <allocation unit='bytes'>1526301487104</allocation>
  <available unit='bytes'>442884853760</available>
  <source>
  </source>
  <target>
    <path>/var/lib/libvirt/images</path>
    <permissions>
      <mode>0711</mode>
      <owner>0</owner>
      <group>0</group>
    </permissions>
  </target>
</pool>

On the source host, this exists:

root@kvmpi:/home/rohit# virsh pool-dumpxml 63229d40-f348-4636-8707-74ab46ab28d0
<pool type='dir'>
  <name>63229d40-f348-4636-8707-74ab46ab28d0</name>
  <uuid>63229d40-f348-4636-8707-74ab46ab28d0</uuid>
  <capacity unit='bytes'>1969186340864</capacity>
  <allocation unit='bytes'>939565473792</allocation>
  <available unit='bytes'>1029620867072</available>
  <source>
  </source>
  <target>
    <path>/var/lib/libvirt/images</path>
    <permissions>
      <mode>0711</mode>
      <owner>0</owner>
      <group>0</group>
    </permissions>
  </target>
</pool>

In ACS DB: storage pool are defined (source is pikvm, dest is cloudpi)

*************************** 2. row ***************************
                   id: 44
                 name: local-pikvm-arm64
                 uuid: 63229d40-f348-4636-8707-74ab46ab28d0
            pool_type: Filesystem
                 port: 0
       data_center_id: 1
               pod_id: 1
           cluster_id: 13
           used_bytes: 939565735936
       capacity_bytes: 1969186340864
         host_address: 10.10.1.10
            user_info: NULL
                 path: /var/lib/libvirt/images
              created: 2024-07-29 15:40:15
              removed: NULL
          update_time: 2025-11-01 19:00:44
               status: Up
storage_provider_name: DefaultPrimary
                scope: HOST
           hypervisor: KVM
              managed: 0
        capacity_iops: NULL
               parent: 0
            used_iops: 24
*************************** 3. row ***************************
                   id: 47
                 name: local-cloudpi-arm64
                 uuid: cc155507-f2ba-4a96-a9b9-2dac9dc50a24
            pool_type: Filesystem
                 port: 0
       data_center_id: 1
               pod_id: 1
           cluster_id: 13
           used_bytes: 1527497064448
       capacity_bytes: 1969186340864
         host_address: 10.10.1.5
            user_info: NULL
                 path: /var/lib/libvirt/images
              created: 2025-08-10 11:13:27
              removed: NULL
          update_time: 2025-11-01 19:06:01
               status: Up
storage_provider_name: DefaultPrimary
                scope: HOST
           hypervisor: KVM
              managed: 0
        capacity_iops: NULL
               parent: 0
            used_iops: 49

versions

ACS 4.22 RC3, adv zone, arm64+x86 mix host env. Preferred arch set to arm64.

The steps to reproduce the bug

See description above

What to do about it?

Ideally the local storage pool of the target host should be used in the domain/xml

[yadvr]

@harikrishna-patnala I'm not sure if it's a blocker (yet), as it's a non-x86 env with local storage or it could be an env issue. But I've hit same issue with user VMs using local storage as well. Ideally it should rewrite pool uuid, as per the db entry in the migration domain xml. A workaround I could use with the user-vms was to power it off, migrate the volume and start it on the target kvm host.

cc @DaanHoogland @weizhouapache and others

[weizhouapache]

@harikrishna-patnala I'm not sure if it's a blocker (yet), as it's a non-x86 env with local storage or it could be an env issue. But I've hit same issue with user VMs using local storage as well. Ideally it should rewrite pool uuid, as per the db entry in the migration domain xml. A workaround I could use with the user-vms was to power it off, migrate the volume and start it on the target kvm host.

cc @DaanHoogland @weizhouapache and others

@rohityadavcloud
I tested system vm and user vm migration with local storage between 2 hosts in a same cluster, both worked
can you share the full log of job-15822 ?

[yadvr]

@weizhouapache logs attached:

job15822.log

It could be something specific to my env, or as the src/target hosts were arm64 (RPi5) based.

[kiranchavala]

Trying to reproduce the issue again in my lab will let you know the results shortly

[harikrishna-patnala]

Trying to reproduce the issue again in my lab will let you know the results shortly

Thank you @kiranchavala

[sureshanaparti]

I'm able to live migrate the user instances (arch: x86_64) on the local storage to another host in the same cluster, with 4.22 RC3. Wei confirmed systemvm migration also works. Maybe, arm64 or env issue.

[weizhouapache]

@weizhouapache logs attached:

job15822.log

It could be something specific to my env, or as the src/target hosts were arm64 (RPi5) based.

it looks the real issue is the migration failure

2025-11-02 00:24:00,987 DEBUG [c.c.a.t.Request] (AgentManager-Handler-2:[]) (logid:) Seq 179-5498895145019375634: Processing: { Ans: , MgmtId: 104339268210986, via: 179, Ver: v1, Flags: 110, [{"com.cloud.agent.api.MigrateAnswer":{"result":"false","details":"Exception during migrate: org.libvirt.LibvirtException: operation failed: job 'migration in' failed: load of migration failed: Invalid argument","wait":"0","bypassHostMaintenance":"false"}}] }

may check agent.log or libvirtd log

[kiranchavala]

@weizhouapache @sureshanaparti

Tried with nfs storage on a arm64 environment

agent.log

2025-11-03 13:33:00,884 INFO  [resource.wrapper.LibvirtMigrateCommandWrapper] (AgentRequest-Handler-2:[]) (logid:d9127b51) VM [v-6-VM] will have CPU shares altered from [695] to [521] as part of migration because the cgroups version differs between hosts.
2025-11-03 13:33:01,078 INFO  [resource.wrapper.LibvirtMigrateCommandWrapper] (AgentRequest-Handler-2:[]) (logid:d9127b51) Starting live migration of instance [v-6-VM] to destination host [qemu+tls://192.168.55.210/system] having the final XML configuration: [<?xml version="1.0" encoding="UTF-8"?><domain type="kvm">
2025-11-03 13:33:02,100 INFO  [resource.wrapper.LibvirtMigrateCommandWrapper] (AgentRequest-Handler-2:[]) (logid:d9127b51) Waiting for migration of v-6-VM to complete, waited 1000ms
2025-11-03 13:33:03,102 INFO  [resource.wrapper.LibvirtMigrateCommandWrapper] (AgentRequest-Handler-2:[]) (logid:d9127b51) Waiting for migration of v-6-VM to complete, waited 2000ms
2025-11-03 13:33:04,104 INFO  [resource.wrapper.LibvirtMigrateCommandWrapper] (AgentRequest-Handler-2:[]) (logid:d9127b51) Waiting for migration of v-6-VM to complete, waited 3000ms
2025-11-03 13:33:05,106 INFO  [resource.wrapper.LibvirtMigrateCommandWrapper] (AgentRequest-Handler-2:[]) (logid:d9127b51) Waiting for mi


2025-11-03 13:34:07,852 ERROR [resource.wrapper.LibvirtMigrateCommandWrapper] (AgentRequest-Handler-2:[]) (logid:d9127b51) Can't migrate domain [v-6-VM] due to: [org.libvirt.LibvirtException: operation failed: job 'migration out' failed: Unable to read from socket: Connection reset by peer]. java.util.concurrent.ExecutionException: org.libvirt.LibvirtException: operation failed: job 'migration out' failed: Unable to read from socket: Connection reset by peer
        at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122)
        at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:205)
        at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtMigrateCommandWrapper.execute(LibvirtMigrateCommandWrapper.java:326)
        at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtMigrateCommandWrapper.execute(LibvirtMigrateCommandWrapper.java:91)
        at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)
        at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:2280)
        at com.cloud.agent.Agent.processRequest(Agent.java:813)
        at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1295)
        at com.cloud.utils.nio.Task.call(Task.java:83)
        at com.cloud.utils.nio.Task.call(Task.java:29)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: org.libvirt.LibvirtException: operation failed: job 'migration out' failed: Unable to read from socket: Connection reset by peer
        at org.libvirt.ErrorHandler.processError(Unknown Source)
        at org.libvirt.ErrorHandler.processError(Unknown Source)
        at org.libvirt.Domain.migrate(Unknown Source)
        at com.cloud.hypervisor.kvm.resource.MigrateKVMAsync.call(MigrateKVMAsync.java:142)
        at com.cloud.hypervisor.kvm.resource.MigrateKVMAsync.call(MigrateKVMAsync.java:34)
        ... 4 more

journalctl -u libvirtd

Nov 03 13:34:07 raspberrypi4 libvirtd[20596]: operation failed: job 'migration out' failed: Unable to read from socket: Connection reset by peer

[kiranchavala]

Same issue with local storage tried migrating systemvm s-30 from raspi4 to raspi5

cloudstack logs

2025-11-03 13:57:29,157 DEBUG [c.c.a.t.Request] (Work-Job-Executor-11:[ctx-56cb586b, job-173/job-174, ctx-dc8e2acc]) (logid:50c31d05) Seq 5-8185855272699297828: Sending  { Cmd , MgmtId: 206863090450418, via: 5(raspberrypi5), Ver: v1, Flags: 100111, [{"com.cloud.agent.api.PrepareForMigrationCommand":{"vm":{"id":"30","name":"s-30-VM","state":"Migrating","type":"SecondaryStorageVm","cpus":"1","minSpeed":"500","maxSpeed":"500","minRam":"(512.00 MB) 536870912","maxRam":"(512.00 MB) 536870912","arch":"aarch64","os":"Debian GNU/Linux 12 (64-bit)","platformEmulator":"Debian GNU/Linux 12 (64-bit)","bootArgs":"","enableHA":"false","limitCpuUse":"false","enableDynamicallyScaleVm":"false","details":{},"uuid":"76f608cb-4ba5-4153-85d0-a2b160c9c3b9","enterHardwareSetup":"false","disks":[{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"16f3277c-ced8-4a0d-bae8-1f093de446f6","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"4f12a46e-6fad-47fe-a98e-b2fdcb990206","name":"raspberrypi4-local-4f12a46e","id":"4","poolType":"Filesystem","host":"192.168.55.211","path":"/var/lib/libvirt/images","port":"0","url":"Filesystem://192.168.55.211/var/lib/libvirt/images/?ROLE=Primary&STOREUUID=4f12a46e-6fad-47fe-a98e-b2fdcb990206","isManaged":"false"}},"name":"ROOT-30","size":"(4.88 GB) 5242880000","path":"16f3277c-ced8-4a0d-bae8-1f093de446f6","volumeId":"36","vmName":"s-30-VM","accountId":"1","format":"QCOW2","provisioningType":"THIN","poolId":"4","id":"36","deviceId":"0","bytesReadRate":"(0 bytes) 0","bytesWriteRate":"(0 bytes) 0","iopsReadRate":"(0 bytes) 0","iopsWriteRate":"(0 bytes) 0","hypervisorType":"KVM","directDownload":"false","deployAsIs":"false","checkpointPaths":[],"checkpointImageStoreUrls":[],"followRedirects":"false"}},"diskSeq":"0","path":"16f3277c-ced8-4a0d-bae8-1f093de446f6","type":"ROOT","details":{"storageHost":"192.168.55.211","managed":"false","storagePort":"0","storageMigrateSecretConsumer":"afd2231e-2dc6-4829-8ff4-c646750a9177","protocoltype":"Filesystem","storage.pool.disk.wait":"60","volumeSize":"(4.88 GB) 5242880000"}}],"nics":[{"deviceId":"2","networkRateMbps":"-1","defaultNic":"true","pxeDisable":"true","nicUuid":"6e3dcfa1-397a-4fe5-9f71-cd94ebcbb47e","details":{"MacAddressChanges":"true","ForgedTransmits":"true","PromiscuousMode":"false","MacLearning":"false"},"dpdkEnabled":"false","networkId":"200","networkSegmentName":"D1-A1-Z1-S200","uuid":"5830d80a-b2cf-4b69-a863-130a8f5a1890","ip":"192.168.55.50","netmask":"255.255.255.0","gateway":"192.168.55.1","mac":"1e:00:6b:00:00:16","dns1":"8.8.8.8","broadcastType":"Vlan","type":"Public","broadcastUri":"vlan://untagged","isolationUri":"vlan://untagged","securityGroupEnabled":"false"},{"deviceId":"0","networkRateMbps":"-1","defaultNic":"false","pxeDisable":"true","nicUuid":"432400b4-ba64-4d7d-9049-efb79b955466","details":{"MacAddressChanges":"true","ForgedTransmits":"true","PromiscuousMode":"false","MacLearning":"false"},"dpdkEnabled":"false","networkId":"202","networkSegmentName":"D1-A1-Z1-S202","uuid":"a0379ee9-d1a7-41e9-a111-90b116cd8439","ip":"169.254.24.249","netmask":"255.255.0.0","gateway":"169.254.0.1","mac":"0e:00:a9:fe:18:f9","broadcastType":"LinkLocal","type":"Control","securityGroupEnabled":"false"},{"deviceId":"1","networkRateMbps":"-1","defaultNic":"false","pxeDisable":"true","nicUuid":"2da02b0d-e4ca-4d09-a026-90dc5457ca4f","details":{"MacAddressChanges":"true","ForgedTransmits":"true","PromiscuousMode":"false","MacLearning":"false"},"dpdkEnabled":"false","networkId":"201","networkSegmentName":"D1-A1-Z1-S201","uuid":"c225423e-b1bc-45a7-89b4-d60ce848aaf7","ip":"192.168.55.80","netmask":"255.255.255.0","gateway":"192.168.55.1","mac":"1e:00:0e:00:00:0a","broadcastType":"Native","type":"Management","securityGroupEnabled":"false"}],"vcpuMaxLimit":"1","configDriveLocation":"SECONDARY","guestOsDetails":{},"extraConfig":{},"networkIdToNetworkNameMap":{}},"rollback":"true","wait":"0","bypassHostMaintenance":"false"}}] }


2025-11-03 13:57:29,165 DEBUG [c.c.a.t.Request] (AgentManager-Handler-13:[]) (logid:) Seq 5-8185855272699297828: Processing:  { Ans: , MgmtId: 206863090450418, via: 5, Ver: v1, Flags: 110, [{"com.cloud.agent.api.Answer":{"result":"false","details":"com.cloud.utils.exception.CloudRuntimeException: Could not fetch storage pool 4f12a46e-6fad-47fe-a98e-b2fdcb990206 from libvirt due to org.libvirt.LibvirtException: Storage pool not found: no storage pool with matching uuid '4f12a46e-6fad-47fe-a98e-b2fdcb990206'
        at com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.getStoragePool(KVMStoragePoolManager.java:286)
        at com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.getStoragePool(KVMStoragePoolManager.java:272)
        at com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.disconnectPhysicalDisksViaVmSpec(KVMStoragePoolManager.java:247)
        at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtPrepareForMigrationCommandWrapper.handleRollback(LibvirtPrepareForMigrationCommandWrapper.java:160)
        at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtPrepareForMigrationCommandWrapper.execute(LibvirtPrepareForMigrationCommandWrapper.java:59)
        at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtPrepareForMigrationCommandWrapper.execute(LibvirtPrepareForMigrationCommandWrapper.java:50)
        at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)
        at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:2280)
        at com.cloud.agent.Agent.processRequest(Agent.java:813)
        at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1295)
        at com.cloud.utils.nio.Task.call(Task.java:83)

root@raspberrypi4:/var/lib/libvirt/images# virsh pool-list
 Name                                   State    Autostart
------------------------------------------------------------
 4f12a46e-6fad-47fe-a98e-b2fdcb990206   active   no
 c7d94500-acf4-3ee0-9410-56cd2aa3fda9   active   no


root@raspberrypi5:~# virsh pool-list
 Name                                   State    Autostart
------------------------------------------------------------
 5996bfe3-686a-4543-982c-83a968a042bf   active   no
 c7d94500-acf4-3ee0-9410-56cd2aa3fda9   active   no

agent logs

2025-11-03 13:56:19,367 INFO  [resource.wrapper.LibvirtMigrateCommandWrapper] (AgentRequest-Handler-2:[]) (logid:50c31d05) VM [s-30-VM] will have CPU shares altered from [695] to [521] as part of migration because the cgroups version differs between hosts.
2025-11-03 13:56:19,426 INFO  [resource.wrapper.LibvirtMigrateCommandWrapper] (AgentRequest-Handler-2:[]) (logid:50c31d05) Starting live migration of instance [s-30-VM] to destination host [qemu+tls://192.168.55.210/system] having the final XML configuration: [<?xml version="1.0" encoding="UTF-8"?><domain type="kvm">
  <name>s-30-VM</name>
  <uuid>76f608cb-4ba5-4153-85d0-a2b160c9c3b9</uuid>
  <description>Debian GNU/Linux 12 (64-bit)</description>
  <memory unit="KiB">524288</memory>
  <currentMemory unit="KiB">524288</currentMemory>
  <vcpu placement="static">1</vcpu>
  <cputune>
    <shares>521</shares>
  </cputune>
  <resource>
    <partition>/machine</partition>
  </resource>
  <sysinfo type="smbios">
    <system>
      <entry name="manufacturer">Apache Software Foundation</entry>
      <entry name="product">CloudStack KVM Hypervisor</entry>
      <entry name="serial">76f608cb-4ba5-4153-85d0-a2b160c9c3b9</entry>
      <entry name="uuid">76f608cb-4ba5-4153-85d0-a2b160c9c3b9</entry>
    </system>
  </sysinfo>
  <os>
    <type arch="aarch64" machine="virt-8.2">hvm</type>
    <loader readonly="yes" type="pflash">/usr/share/AAVMF/AAVMF_CODE.fd</loader>
    <nvram template="/usr/share/AAVMF/AAVMF_VARS.fd">/var/lib/libvirt/qemu/nvram/s-30-VM_VARS.fd</nvram>
    <boot dev="cdrom"/>
    <boot dev="hd"/>
    <smbios mode="sysinfo"/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pae/>
    <gic version="2"/>
  </features>
  <cpu check="none" mode="host-passthrough">
    <topology cores="1" dies="1" sockets="1" threads="1"/>
  </cpu>
  <clock offset="utc">
    <timer name="kvmclock"/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <devices>
    <emulator>/usr/bin/qemu-system-aarch64</emulator>
    <disk device="disk" type="file">
      <driver cache="none" name="qemu" type="qcow2"/>

      <target bus="virtio" dev="vda"/>
      <serial>16f3277cced84a0dbae8</serial>
      <address bus="0x08" domain="0x0000" function="0x0" slot="0x00" type="pci"/>
    <source file="/var/lib/libvirt/images/afd2231e-2dc6-4829-8ff4-c646750a9177"/></disk>
    <disk device="cdrom" type="file">
      <driver name="qemu" type="raw"/>
      <target bus="scsi" dev="hdc"/>
      <readonly/>
      <address bus="0" controller="0" target="0" type="drive" unit="2"/>
    </disk>
    <controller index="0" model="qemu-xhci" type="usb">
      <address bus="0x05" domain="0x0000" function="0x0" slot="0x00" type="pci"/>
    </controller>
    <controller index="0" model="pcie-root" type="pci"/>
    <controller index="1" model="pcie-root-port" type="pci">
      <model name="pcie-root-port"/>
      <target chassis="1" port="0x8"/>
      <address bus="0x00" domain="0x0000" function="0x0" multifunction="on" slot="0x01" type="pci"/>
    </controller>
    <controller index="2" model="pcie-root-port" type="pci">
      <model name="pcie-root-port"/>
      <target chassis="2" port="0x9"/>
      <address bus="0x00" domain="0x0000" function="0x1" slot="0x01" type="pci"/>
    </controller>
    <controller index="3" model="pcie-root-port" type="pci">
      <model name="pcie-root-port"/>
      <target chassis="3" port="0xa"/>
      <address bus="0x00" domain="0x0000" function="0x2" slot="0x01" type="pci"/>
    </controller>
    <controller index="4" model="pcie-root-port" type="pci">
      <model name="pcie-root-port"/>
      <target chassis="4" port="0xb"/>
      <address bus="0x00" domain="0x0000" function="0x3" slot="0x01" type="pci"/>
    </controller>
    <controller index="5" model="pcie-root-port" type="pci">
      <model name="pcie-root-port"/>
      <target chassis="5" port="0xc"/>
      <address bus="0x00" domain="0x0000" function="0x4" slot="0x01" type="pci"/>
    </controller>
    <controller index="6" model="pcie-root-port" type="pci">
      <model name="pcie-root-port"/>
      <target chassis="6" port="0xd"/>
      <address bus="0x00" domain="0x0000" function="0x5" slot="0x01" type="pci"/>
    </controller>
    <controller index="7" model="pcie-to-pci-bridge" type="pci">
      <model name="pcie-pci-bridge"/>
      <address bus="0x01" domain="0x0000" function="0x0" slot="0x00" type="pci"/>
    </controller>
    <controller index="8" model="pcie-root-port" type="pci">
      <model name="pcie-root-port"/>
      <target chassis="8" port="0xe"/>
      <address bus="0x00" domain="0x0000" function="0x6" slot="0x01" type="pci"/>
    </controller>
    <controller index="9" model="pcie-root-port" type="pci">
      <model name="pcie-root-port"/>
      <target chassis="9" port="0xf"/>
      <address bus="0x00" domain="0x0000" function="0x7" slot="0x01" type="pci"/>
    </controller>
    <controller index="10" model="pcie-root-port" type="pci">
      <model name="pcie-root-port"/>
      <target chassis="10" port="0x10"/>
      <address bus="0x00" domain="0x0000" function="0x0" multifunction="on" slot="0x02" type="pci"/>
    </controller>
    <controller index="11" model="pcie-root-port" type="pci">
      <model name="pcie-root-port"/>
      <target chassis="11" port="0x11"/>
      <address bus="0x00" domain="0x0000" function="0x1" slot="0x02" type="pci"/>
    </controller>
    <controller index="0" model="lsilogic" type="scsi">
      <address bus="0x07" domain="0x0000" function="0x0" slot="0x01" type="pci"/>
    </controller>
    <controller index="0" type="virtio-serial">
      <address bus="0x06" domain="0x0000" function="0x0" slot="0x00" type="pci"/>
    </controller>
    <interface type="bridge">
      <mac address="0e:00:a9:fe:18:f9"/>
      <source bridge="cloud0"/>
      <model type="virtio"/>
      <link state="up"/>
      <rom bar="off" file=""/>
      <address bus="0x02" domain="0x0000" function="0x0" slot="0x00" type="pci"/>
    </interface>
    <interface type="bridge">
      <mac address="1e:00:0e:00:00:0a"/>
      <source bridge="cloudbr0"/>
      <model type="virtio"/>
      <link state="up"/>
      <rom bar="off" file=""/>
      <address bus="0x03" domain="0x0000" function="0x0" slot="0x00" type="pci"/>
    </interface>
    <interface type="bridge">
      <mac address="1e:00:6b:00:00:16"/>
      <source bridge="cloudbr0"/>
      <model type="virtio"/>
      <link state="up"/>
      <rom bar="off" file=""/>
      <address bus="0x04" domain="0x0000" function="0x0" slot="0x00" type="pci"/>
    </interface>
    <serial type="pty">
      <target port="0"/>
    </serial>
    <console type="pty">
      <target port="0" type="serial"/>
    </console>
    <channel type="unix">
      <source mode="bind" path="/var/lib/libvirt/qemu/s-30-VM.org.qemu.guest_agent.0"/>
      <target name="org.qemu.guest_agent.0" type="virtio"/>
      <address bus="0" controller="0" port="1" type="virtio-serial"/>
    </channel>
    <input bus="usb" type="tablet">
      <address bus="0" port="1" type="usb"/>
    </input>
    <input bus="usb" type="keyboard">
      <address bus="0" port="2" type="usb"/>
    </input>
    <input bus="usb" type="mouse">
      <address bus="0" port="3" type="usb"/>
    </input>
    <graphics autoport="yes" listen="192.168.55.210" passwd="TXui6rU0" port="-1" type="vnc">
      <listen address="192.168.55.210" type="address"/>
    </graphics>
    <video>
      <model heads="1" primary="yes" type="virtio"/>
      <address bus="0x0a" domain="0x0000" function="0x0" slot="0x00" type="pci"/>
    </video>
    <watchdog action="none" model="i6300esb">
      <address bus="0x07" domain="0x0000" function="0x0" slot="0x02" type="pci"/>
    </watchdog>
    <memballoon model="virtio">
      <address bus="0x09" domain="0x0000" function="0x0" slot="0x00" type="pci"/>
    </memballoon>
  </devices>
</domain>].
2025-11-03 13:56:20,447 INFO  [resource.wrapper.LibvirtMigrateCommandWrapper] (AgentRequest-Handler-2:[]) (logid:50c31d05) Waiting for migration of s-30-VM to complete, waited 1000ms
2025-11-03 13:56:21,449 INFO  [resource.wrapper.LibvirtMigrateCommandWrapper] (AgentRequest-Handler-2:[]) (logid:50c31d05) Waiting for migration of s-30-VM to complete, waited 2000ms
2025-11-03 13:56:22,452 INFO  [resource.wrapper.LibvirtMigrateCommandWrapper] (AgentRequest-Handler-2:[]) (logid:50c31d05) Waiting for migration of s-30-VM to complete, waited 3000ms
2025-11-03 13:56:23,455 INFO  [resource.wrapper.LibvirtMigrateCommandWrapper] (AgentRequest-Handler-2:[]) (logid:50c31d05) Waiting for migration of s-30-VM to complete, waited 4000ms
2025-11-03 13:56:24,458 INFO  [resource.wrapper.LibvirtMigrateCommandWrapper] (AgentRequest-Handler-2:[]) (logid:50c31d05) Waiting for migration of s-30-VM to complete, waited 5000ms
2025-11-03 13:56:25,538 INFO  [resource.wrapper.LibvirtMigrateCommandWrapper] (AgentRequest-Handler-2:[]) (logid:50c31d05) Waiting for migration of s-30-VM to complete, waited 6000ms
2025-11-03 13:56:26,541 INFO  [resource.wrapper.LibvirtMig


2025-11-03 13:57:29,093 INFO  [resource.wrapper.LibvirtMigrateCommandWrapper] (AgentRequest-Handler-2:[]) (logid:50c31d05) Migration thread of VM [s-30-VM] finished.
2025-11-03 13:57:29,095 ERROR [resource.wrapper.LibvirtMigrateCommandWrapper] (AgentRequest-Handler-2:[]) (logid:50c31d05) Can't migrate domain [s-30-VM] due to: [org.libvirt.LibvirtException: operation failed: job 'migration out' failed: Unable to read from socket: Connection reset by peer]. java.util.concurrent.ExecutionException: org.libvirt.LibvirtException: operation failed: job 'migration out' failed: Unable to read from socket: Connection reset by peer
	at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122)
	at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:205)
	at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtMigrateCommandWrapper.execute(LibvirtMigrateCommandWrapper.java:326)
	at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtMigrateCommandWrapper.execute(LibvirtMigrateCommandWrapper.java:91)
	at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)
	at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:2280)
	at com.cloud.agent.Agent.processRequest(Agent.java:813)
	at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1295)
	at com.cloud.utils.nio.Task.call(Task.java:83)
	at com.cloud.utils.nio.Task.call(Task.java:29)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: org.libvirt.LibvirtException: operation failed: job 'migration out' failed: Unable to read from socket: Connection reset by peer
	at org.libvirt.ErrorHandler.processError(Unknown Source)
	at org.libvirt.ErrorHandler.processError(Unknown Source)
	at org.libvirt.Domain.migrate(Unknown Source)
	at com.cloud.hypervisor.kvm.resource.MigrateKVMAsync.call(MigrateKVMAsync.java:142)
	at com.cloud.hypervisor.kvm.resource.MigrateKVMAsync.call(MigrateKVMAsync.java:34)
	... 4 more

[weizhouapache]

oh, good, so the issue is with the live volume migration between arm64 hosts.
thanks @kiranchavala for the testing

any errors in /var/log/libvirt/qemu/i-xx-xxx.log on both source and dest hosts ?

[harikrishna-patnala]

Thanks @kiranchavala for testing it. Do we know if this has worked before in 4.20 ? Asking this to understand if this is a regression or an existing issue.

[kiranchavala]

oh, good, so the issue is with the live volume migration between arm64 hosts. thanks @kiranchavala for the testing

any errors in /var/log/libvirt/qemu/i-xx-xxx.log on both source and dest hosts ?

@weizhouapache @harikrishna-patnala

even normal vm migration is failing on arm64

Please find the logs

Tried migrating s-77 from raspi4 to raspi5

cslogs

2025-11-04 06:21:52,419 ERROR [resource.wrapper.LibvirtMigrateCommandWrapper] (AgentRequest-Handler-5:[]) (logid:2c65c047) Can't migrate domain [s-77-VM] due to: [org.libvirt.LibvirtException: operation failed: job 'migration out' failed: Unable to read from socket: Connection reset by peer]. java.util.concurrent.ExecutionException: org.libvirt.LibvirtException: operation failed: job 'migration out' failed: Unable to read from socket: Connection reset by peer
	at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122)
	at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:205)
	at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtMigrateCommandWrapper.execute(LibvirtMigrateCommandWrapper.java:326)
	at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtMigrateCommandWrapper.execute(LibvirtMigrateCommandWrapper.java:91)
	at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)
	at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:2280)
	at com.cloud.agent.Agent.processRequest(Agent.java:813)
	at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1295)
	at com.cloud.utils.nio.Task.call(Task.java:83)
	at com.cloud.utils.nio.Task.call(Task.java:29)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: org.libvirt.LibvirtException: operation failed: job 'migration out' failed: Unable to read from socket: Connection reset by peer
	at org.libvirt.ErrorHandler.processError(Unknown Source)
	at org.libvirt.ErrorHandler.processError(Unknown Source)
	at org.libvirt.Domain.migrate(Unknown Source)
	at com.cloud.hypervisor.kvm.resource.MigrateKVMAsync.call(MigrateKVMAsync.java:142)
	at com.cloud.hypervisor.kvm.resource.MigrateKVMAsync.call(MigrateKVMAsync.java:34)
	... 4 more

qemu log from raspi4

root@raspberrypi4:/var/log/libvirt/qemu# cat s-77-VM.log
2025-11-04 06:16:26.490+0000: starting up libvirt version: 10.0.0, package: 10.0.0-2ubuntu8.9 (Ubuntu), qemu version: 8.2.2Debian 1:8.2.2+ds-0ubuntu1.10, kernel: 6.8.0-1040-raspi, hostname: raspberrypi4
LC_ALL=C \
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/snap/bin \
USER=root \
HOME=/var/lib/libvirt/qemu/domain-2-s-77-VM \
XDG_DATA_HOME=/var/lib/libvirt/qemu/domain-2-s-77-VM/.local/share \
XDG_CACHE_HOME=/var/lib/libvirt/qemu/domain-2-s-77-VM/.cache \
XDG_CONFIG_HOME=/var/lib/libvirt/qemu/domain-2-s-77-VM/.config \
/usr/bin/qemu-system-aarch64 \
-name guest=s-77-VM,debug-threads=on \
-S \
-object '{"qom-type":"secret","id":"masterKey0","format":"raw","file":"/var/lib/libvirt/qemu/domain-2-s-77-VM/master-key.aes"}' \
-blockdev '{"driver":"file","filename":"/usr/share/AAVMF/AAVMF_CODE.fd","node-name":"libvirt-pflash0-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-pflash0-format","read-only":true,"driver":"raw","file":"libvirt-pflash0-storage"}' \
-blockdev '{"driver":"file","filename":"/var/lib/libvirt/qemu/nvram/s-77-VM_VARS.fd","node-name":"libvirt-pflash1-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-pflash1-format","read-only":false,"driver":"raw","file":"libvirt-pflash1-storage"}' \
-machine virt-8.2,usb=off,gic-version=2,dump-guest-core=off,memory-backend=mach-virt.ram,pflash0=libvirt-pflash0-format,pflash1=libvirt-pflash1-format,acpi=on \
-accel kvm \
-cpu host \
-m size=524288k \
-object '{"qom-type":"memory-backend-ram","id":"mach-virt.ram","size":536870912}' \
-overcommit mem-lock=off \
-smp 1,sockets=1,dies=1,cores=1,threads=1 \
-uuid 8ff51165-a0c8-4482-9e85-cb54e8656c2b \
-smbios 'type=1,manufacturer=Apache Software Foundation,product=CloudStack KVM Hypervisor,serial=8ff51165-a0c8-4482-9e85-cb54e8656c2b,uuid=8ff51165-a0c8-4482-9e85-cb54e8656c2b' \
-no-user-config \
-nodefaults \
-chardev socket,id=charmonitor,fd=36,server=on,wait=off \
-mon chardev=charmonitor,id=monitor,mode=control \
-rtc base=utc \
-no-shutdown \
-boot strict=on \
-device '{"driver":"pcie-root-port","port":8,"chassis":1,"id":"pci.1","bus":"pcie.0","multifunction":true,"addr":"0x1"}' \
-device '{"driver":"pcie-root-port","port":9,"chassis":2,"id":"pci.2","bus":"pcie.0","addr":"0x1.0x1"}' \
-device '{"driver":"pcie-root-port","port":10,"chassis":3,"id":"pci.3","bus":"pcie.0","addr":"0x1.0x2"}' \
-device '{"driver":"pcie-root-port","port":11,"chassis":4,"id":"pci.4","bus":"pcie.0","addr":"0x1.0x3"}' \
-device '{"driver":"pcie-root-port","port":12,"chassis":5,"id":"pci.5","bus":"pcie.0","addr":"0x1.0x4"}' \
-device '{"driver":"pcie-root-port","port":13,"chassis":6,"id":"pci.6","bus":"pcie.0","addr":"0x1.0x5"}' \
-device '{"driver":"pcie-pci-bridge","id":"pci.7","bus":"pci.1","addr":"0x0"}' \
-device '{"driver":"pcie-root-port","port":14,"chassis":8,"id":"pci.8","bus":"pcie.0","addr":"0x1.0x6"}' \
-device '{"driver":"pcie-root-port","port":15,"chassis":9,"id":"pci.9","bus":"pcie.0","addr":"0x1.0x7"}' \
-device '{"driver":"pcie-root-port","port":16,"chassis":10,"id":"pci.10","bus":"pcie.0","multifunction":true,"addr":"0x2"}' \
-device '{"driver":"pcie-root-port","port":17,"chassis":11,"id":"pci.11","bus":"pcie.0","addr":"0x2.0x1"}' \
-device '{"driver":"qemu-xhci","id":"usb","bus":"pci.5","addr":"0x0"}' \
-device '{"driver":"lsi","id":"scsi0","bus":"pci.7","addr":"0x1"}' \
-device '{"driver":"virtio-serial-pci","id":"virtio-serial0","bus":"pci.6","addr":"0x0"}' \
-blockdev '{"driver":"file","filename":"/mnt/c7d94500-acf4-3ee0-9410-56cd2aa3fda9/bb80dd0f-05a8-4eb9-a525-59b15f0a77ac","node-name":"libvirt-3-storage","auto-read-only":true,"discard":"unmap","cache":{"direct":true,"no-flush":false}}' \
-blockdev '{"node-name":"libvirt-3-format","read-only":true,"cache":{"direct":true,"no-flush":false},"driver":"qcow2","file":"libvirt-3-storage","backing":null}' \
-blockdev '{"driver":"file","filename":"/mnt/c7d94500-acf4-3ee0-9410-56cd2aa3fda9/36d04d51-90cb-4248-9d15-fbb27f9bc0e0","node-name":"libvirt-2-storage","auto-read-only":true,"discard":"unmap","cache":{"direct":true,"no-flush":false}}' \
-blockdev '{"node-name":"libvirt-2-format","read-only":false,"cache":{"direct":true,"no-flush":false},"driver":"qcow2","file":"libvirt-2-storage","backing":"libvirt-3-format"}' \
-device '{"driver":"virtio-blk-pci","bus":"pci.8","addr":"0x0","drive":"libvirt-2-format","id":"virtio-disk0","bootindex":2,"write-cache":"on","serial":"36d04d5190cb42489d15"}' \
-device '{"driver":"scsi-cd","bus":"scsi0.0","scsi-id":2,"device_id":"drive-scsi0-0-2","id":"scsi0-0-2","bootindex":1}' \
-netdev '{"type":"tap","fd":"33","vhost":true,"vhostfd":"34","id":"hostnet0"}' \
-device '{"driver":"virtio-net-pci","netdev":"hostnet0","id":"net0","mac":"0e:00:a9:fe:88:39","bus":"pci.2","addr":"0x0","rombar":0,"romfile":""}' \
-netdev '{"type":"tap","fd":"41","vhost":true,"vhostfd":"46","id":"hostnet1"}' \
-device '{"driver":"virtio-net-pci","netdev":"hostnet1","id":"net1","mac":"1e:00:89:00:00:0c","bus":"pci.3","addr":"0x0","rombar":0,"romfile":""}' \
-netdev '{"type":"tap","fd":"48","vhost":true,"vhostfd":"50","id":"hostnet2"}' \
-device '{"driver":"virtio-net-pci","netdev":"hostnet2","id":"net2","mac":"1e:00:6d:00:00:17","bus":"pci.4","addr":"0x0","rombar":0,"romfile":""}' \
-chardev pty,id=charserial0 \
-serial chardev:charserial0 \
-chardev socket,id=charchannel0,fd=35,server=on,wait=off \
-device '{"driver":"virtserialport","bus":"virtio-serial0.0","nr":1,"chardev":"charchannel0","id":"channel0","name":"org.qemu.guest_agent.0"}' \
-device '{"driver":"usb-tablet","id":"input0","bus":"usb.0","port":"1"}' \
-device '{"driver":"usb-kbd","id":"input1","bus":"usb.0","port":"2"}' \
-device '{"driver":"usb-mouse","id":"input2","bus":"usb.0","port":"3"}' \
-audiodev '{"id":"audio1","driver":"none"}' \
-object '{"qom-type":"tls-creds-x509","id":"vnc-tls-creds0","dir":"/etc/pki/libvirt-vnc","endpoint":"server","verify-peer":true}' \
-vnc 192.168.55.211:1,password=on,tls-creds=vnc-tls-creds0,audiodev=audio1 \
-device '{"driver":"virtio-gpu-pci","id":"video0","max_outputs":1,"bus":"pci.10","addr":"0x0"}' \
-device '{"driver":"i6300esb","id":"watchdog0","bus":"pci.7","addr":"0x2"}' \
-watchdog-action none \
-device '{"driver":"virtio-balloon-pci","id":"balloon0","bus":"pci.9","addr":"0x0"}' \
-sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \
-msg timestamp=on
2025-11-04 06:16:26.491+0000: Domain id=2 is tainted: high-privileges
char device redirected to /dev/pts/1 (label charserial0)
2025-11-04 06:17:11.931+0000: Domain id=2 is tainted: custom-ga-command
2025-11-04 06:21:47.508+0000: initiating migration
2025-11-04T06:21:52.011225Z qemu-system-aarch64: Unable to read from socket: Connection reset by peer
warning: old compression is deprecated; use multifd compression methods instead
warning: old compression is deprecated; use multifd compression methods instead
warning: old compression is deprecated; use multifd compression methods instead
warning: block migration is deprecated; use blockdev-mirror with NBD instead

qemu log from raspi4

root@raspberrypi5:/var/log/libvirt/qemu# cat s-77-VM.log
2025-11-04 06:21:45.774+0000: starting up libvirt version: 10.0.0, package: 10.0.0-2ubuntu8.9 (Ubuntu), qemu version: 8.2.2Debian 1:8.2.2+ds-0ubuntu1.10, kernel: 6.8.0-1040-raspi, hostname: raspberrypi5
LC_ALL=C \
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/snap/bin \
USER=root \
HOME=/var/lib/libvirt/qemu/domain-2-s-77-VM \
XDG_DATA_HOME=/var/lib/libvirt/qemu/domain-2-s-77-VM/.local/share \
XDG_CACHE_HOME=/var/lib/libvirt/qemu/domain-2-s-77-VM/.cache \
XDG_CONFIG_HOME=/var/lib/libvirt/qemu/domain-2-s-77-VM/.config \
/usr/bin/qemu-system-aarch64 \
-name guest=s-77-VM,debug-threads=on \
-S \
-object '{"qom-type":"secret","id":"masterKey0","format":"raw","file":"/var/lib/libvirt/qemu/domain-2-s-77-VM/master-key.aes"}' \
-blockdev '{"driver":"file","filename":"/usr/share/AAVMF/AAVMF_CODE.fd","node-name":"libvirt-pflash0-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-pflash0-format","read-only":true,"driver":"raw","file":"libvirt-pflash0-storage"}' \
-blockdev '{"driver":"file","filename":"/var/lib/libvirt/qemu/nvram/s-77-VM_VARS.fd","node-name":"libvirt-pflash1-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-pflash1-format","read-only":false,"driver":"raw","file":"libvirt-pflash1-storage"}' \
-machine virt-8.2,usb=off,gic-version=2,dump-guest-core=off,memory-backend=mach-virt.ram,pflash0=libvirt-pflash0-format,pflash1=libvirt-pflash1-format,acpi=on \
-accel kvm \
-cpu host \
-m size=524288k \
-object '{"qom-type":"memory-backend-ram","id":"mach-virt.ram","size":536870912}' \
-overcommit mem-lock=off \
-smp 1,sockets=1,dies=1,cores=1,threads=1 \
-uuid 8ff51165-a0c8-4482-9e85-cb54e8656c2b \
-smbios 'type=1,manufacturer=Apache Software Foundation,product=CloudStack KVM Hypervisor,serial=8ff51165-a0c8-4482-9e85-cb54e8656c2b,uuid=8ff51165-a0c8-4482-9e85-cb54e8656c2b' \
-no-user-config \
-nodefaults \
-chardev socket,id=charmonitor,fd=39,server=on,wait=off \
-mon chardev=charmonitor,id=monitor,mode=control \
-rtc base=utc \
-no-shutdown \
-boot strict=on \
-device '{"driver":"pcie-root-port","port":8,"chassis":1,"id":"pci.1","bus":"pcie.0","multifunction":true,"addr":"0x1"}' \
-device '{"driver":"pcie-root-port","port":9,"chassis":2,"id":"pci.2","bus":"pcie.0","addr":"0x1.0x1"}' \
-device '{"driver":"pcie-root-port","port":10,"chassis":3,"id":"pci.3","bus":"pcie.0","addr":"0x1.0x2"}' \
-device '{"driver":"pcie-root-port","port":11,"chassis":4,"id":"pci.4","bus":"pcie.0","addr":"0x1.0x3"}' \
-device '{"driver":"pcie-root-port","port":12,"chassis":5,"id":"pci.5","bus":"pcie.0","addr":"0x1.0x4"}' \
-device '{"driver":"pcie-root-port","port":13,"chassis":6,"id":"pci.6","bus":"pcie.0","addr":"0x1.0x5"}' \
-device '{"driver":"pcie-pci-bridge","id":"pci.7","bus":"pci.1","addr":"0x0"}' \
-device '{"driver":"pcie-root-port","port":14,"chassis":8,"id":"pci.8","bus":"pcie.0","addr":"0x1.0x6"}' \
-device '{"driver":"pcie-root-port","port":15,"chassis":9,"id":"pci.9","bus":"pcie.0","addr":"0x1.0x7"}' \
-device '{"driver":"pcie-root-port","port":16,"chassis":10,"id":"pci.10","bus":"pcie.0","multifunction":true,"addr":"0x2"}' \
-device '{"driver":"pcie-root-port","port":17,"chassis":11,"id":"pci.11","bus":"pcie.0","addr":"0x2.0x1"}' \
-device '{"driver":"qemu-xhci","id":"usb","bus":"pci.5","addr":"0x0"}' \
-device '{"driver":"lsi","id":"scsi0","bus":"pci.7","addr":"0x1"}' \
-device '{"driver":"virtio-serial-pci","id":"virtio-serial0","bus":"pci.6","addr":"0x0"}' \
-blockdev '{"driver":"file","filename":"/mnt/c7d94500-acf4-3ee0-9410-56cd2aa3fda9/bb80dd0f-05a8-4eb9-a525-59b15f0a77ac","node-name":"libvirt-3-storage","auto-read-only":true,"discard":"unmap","cache":{"direct":true,"no-flush":false}}' \
-blockdev '{"node-name":"libvirt-3-format","read-only":true,"cache":{"direct":true,"no-flush":false},"driver":"qcow2","file":"libvirt-3-storage","backing":null}' \
-blockdev '{"driver":"file","filename":"/mnt/c7d94500-acf4-3ee0-9410-56cd2aa3fda9/36d04d51-90cb-4248-9d15-fbb27f9bc0e0","node-name":"libvirt-2-storage","auto-read-only":true,"discard":"unmap","cache":{"direct":true,"no-flush":false}}' \
-blockdev '{"node-name":"libvirt-2-format","read-only":false,"cache":{"direct":true,"no-flush":false},"driver":"qcow2","file":"libvirt-2-storage","backing":"libvirt-3-format"}' \
-device '{"driver":"virtio-blk-pci","bus":"pci.8","addr":"0x0","drive":"libvirt-2-format","id":"virtio-disk0","bootindex":2,"write-cache":"on","serial":"36d04d5190cb42489d15"}' \
-device '{"driver":"scsi-cd","bus":"scsi0.0","scsi-id":2,"device_id":"drive-scsi0-0-2","id":"scsi0-0-2","bootindex":1}' \
-netdev '{"type":"tap","fd":"40","vhost":true,"vhostfd":"42","id":"hostnet0"}' \
-device '{"driver":"virtio-net-pci","netdev":"hostnet0","id":"net0","mac":"0e:00:a9:fe:88:39","bus":"pci.2","addr":"0x0","rombar":0,"romfile":""}' \
-netdev '{"type":"tap","fd":"43","vhost":true,"vhostfd":"44","id":"hostnet1"}' \
-device '{"driver":"virtio-net-pci","netdev":"hostnet1","id":"net1","mac":"1e:00:89:00:00:0c","bus":"pci.3","addr":"0x0","rombar":0,"romfile":""}' \
-netdev '{"type":"tap","fd":"45","vhost":true,"vhostfd":"46","id":"hostnet2"}' \
-device '{"driver":"virtio-net-pci","netdev":"hostnet2","id":"net2","mac":"1e:00:6d:00:00:17","bus":"pci.4","addr":"0x0","rombar":0,"romfile":""}' \
-chardev pty,id=charserial0 \
-serial chardev:charserial0 \
-chardev socket,id=charchannel0,fd=38,server=on,wait=off \
-device '{"driver":"virtserialport","bus":"virtio-serial0.0","nr":1,"chardev":"charchannel0","id":"channel0","name":"org.qemu.guest_agent.0"}' \
-device '{"driver":"usb-tablet","id":"input0","bus":"usb.0","port":"1"}' \
-device '{"driver":"usb-kbd","id":"input1","bus":"usb.0","port":"2"}' \
-device '{"driver":"usb-mouse","id":"input2","bus":"usb.0","port":"3"}' \
-audiodev '{"id":"audio1","driver":"none"}' \
-object '{"qom-type":"tls-creds-x509","id":"vnc-tls-creds0","dir":"/etc/pki/libvirt-vnc","endpoint":"server","verify-peer":true}' \
-vnc 192.168.55.210:1,password=on,tls-creds=vnc-tls-creds0,audiodev=audio1 \
-device '{"driver":"virtio-gpu-pci","id":"video0","max_outputs":1,"bus":"pci.10","addr":"0x0"}' \
-device '{"driver":"i6300esb","id":"watchdog0","bus":"pci.7","addr":"0x2"}' \
-watchdog-action none \
-incoming defer \
-device '{"driver":"virtio-balloon-pci","id":"balloon0","bus":"pci.9","addr":"0x0"}' \
-sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \
-msg timestamp=on
2025-11-04 06:21:45.775+0000: Domain id=2 is tainted: high-privileges
2025-11-04 06:21:45.775+0000: Domain id=2 is tainted: hook-script
2025-11-04 06:21:45.775+0000: Domain id=2 is tainted: host-cpu
char device redirected to /dev/pts/2 (label charserial0)
2025-11-04T06:21:51.802223Z qemu-system-aarch64: error while loading state for instance 0x0 of device 'cpu'
2025-11-04T06:21:51.803803Z qemu-system-aarch64: Failed to put registers after init: Invalid argument
2025-11-04 06:21:52.090+0000: shutting down, reason=crashed

[weizhouapache]

thanks @kiranchavala

The error logs show

2025-11-04T06:21:51.802223Z qemu-system-aarch64: error while loading state for instance 0x0 of device 'cpu'
2025-11-04T06:21:51.803803Z qemu-system-aarch64: Failed to put registers after init: Invalid argument

Do the hosts have the same CPU ? @rohityadavcloud @kiranchavala
what CPU mode/model do you use ?

[kiranchavala]

Thanks @weizhouapache as discussed , i got a combination of raspi5 and rapsi4

root@raspberrypi5:~# cat /proc/cpuinfo
processor	: 0
BogoMIPS	: 108.00
Features	: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm lrcpc dcpop asimddp
CPU implementer	: 0x41
CPU architecture: 8
CPU variant	: 0x4
CPU part	: 0xd0b
CPU revision	: 1

processor	: 1
BogoMIPS	: 108.00
Features	: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm lrcpc dcpop asimddp
CPU implementer	: 0x41
CPU architecture: 8
CPU variant	: 0x4
CPU part	: 0xd0b
CPU revision	: 1

processor	: 2
BogoMIPS	: 108.00
Features	: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm lrcpc dcpop asimddp
CPU implementer	: 0x41
CPU architecture: 8
CPU variant	: 0x4
CPU part	: 0xd0b
CPU revision	: 1

processor	: 3
BogoMIPS	: 108.00
Features	: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm lrcpc dcpop asimddp
CPU implementer	: 0x41
CPU architecture: 8
CPU variant	: 0x4
CPU part	: 0xd0b
CPU revision	: 1

Revision	: d04170
Serial		: 4c893570359db976
Model		: Raspberry Pi 5 Model B Rev 1.0

root@raspberrypi4:~# cat /proc/cpuinfo
processor	: 0
BogoMIPS	: 108.00
Features	: fp asimd evtstrm crc32 cpuid
CPU implementer	: 0x41
CPU architecture: 8
CPU variant	: 0x0
CPU part	: 0xd08
CPU revision	: 3

processor	: 1
BogoMIPS	: 108.00
Features	: fp asimd evtstrm crc32 cpuid
CPU implementer	: 0x41
CPU architecture: 8
CPU variant	: 0x0
CPU part	: 0xd08
CPU revision	: 3

processor	: 2
BogoMIPS	: 108.00
Features	: fp asimd evtstrm crc32 cpuid
CPU implementer	: 0x41
CPU architecture: 8
CPU variant	: 0x0
CPU part	: 0xd08
CPU revision	: 3

processor	: 3
BogoMIPS	: 108.00
Features	: fp asimd evtstrm crc32 cpuid
CPU implementer	: 0x41
CPU architecture: 8
CPU variant	: 0x0
CPU part	: 0xd08
CPU revision	: 3

Revision	: d03114
Serial		: 1000000051ea32c3
Model		: Raspberry Pi 4 Model B Rev 1.4

Cloudstack error message

2025-11-04 08:55:33,444 ERROR [resource.wrapper.LibvirtMigrateCommandWrapper] (AgentRequest-Handler-1:[]) (logid:) Can't migrate domain [s-11-VM] due to: [org.libvirt.LibvirtException: internal error: QEMU unexpectedly closed the monitor (vm='s-11-VM'): 2025-11-04T08:55:33.039650Z qemu-system-aarch64: error while loading state for instance 0x0 of device 'cpu'
2025-11-04T08:55:33.040223Z qemu-system-aarch64: Failed to put registers after init: Invalid argument]. java.util.concurrent.ExecutionException: org.libvirt.LibvirtException: internal error: QEMU unexpectedly closed the monitor (vm='s-11-VM'): 2025-11-04T08:55:33.039650Z qemu-system-aarch64: error while loading state for instance 0x0 of device 'cpu'
2025-11-04T08:55:33.040223Z qemu-system-aarch64: Failed to put registers after init: Invalid argument
	at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122)
	at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:205)
	at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtMigrateCommandWrapper.execute(LibvirtMigrateCommandWrapper.java:311)
	at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtMigrateCommandWrapper.execute(LibvirtMigrateCommandWrapper.java:88)
	at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)
	at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1974)
	at com.cloud.agent.Agent.processRequest(Agent.java:778)
	at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1193)
	at com.cloud.utils.nio.Task.call(Task.java:83)
	at com.cloud.utils.nio.Task.call(Task.java:29)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: org.libvirt.LibvirtException: internal error: QEMU unexpectedly closed the monitor (vm='s-11-VM'): 2025-11-04T08:55:33.039650Z qemu-system-aarch64: error while loading state for instance 0x0 of device 'cpu'
2025-11-04T08:55:33.040223Z qemu-system-aarch64: Failed to put registers after init: Invalid argument
	at org.libvirt.ErrorHandler.processError(Unknown Source)
	at org.libvirt.ErrorHandler.processError(Unknown Source)
	at org.libvirt.Domain.migrate(Unknown Source)
	at com.cloud.hypervisor.kvm.resource.MigrateKVMAsync.call(MigrateKVMAsync.java:142)
	at com.cloud.hypervisor.kvm.resource.MigrateKVMAsync.call(MigrateKVMAsync.java:34)
	... 4 more

thanks @kiranchavala

The error logs show

2025-11-04T06:21:51.802223Z qemu-system-aarch64: error while loading state for instance 0x0 of device 'cpu'
2025-11-04T06:21:51.803803Z qemu-system-aarch64: Failed to put registers after init: Invalid argument

Do the hosts have the same CPU ? @rohityadavcloud @kiranchavala what CPU mode/model do you use ?