On Fri, Mar 11, 2016 at 01:56:05AM +0200, Michael S. Tsirkin wrote:
> On Thu, Mar 03, 2016 at 03:37:37PM +0000, Stefan Hajnoczi wrote:
> > Michael pointed out that the virtio-vsock draft specification does not
> > address live migration and in fact currently precludes migration.
> >
> > Migration is fundamental so the device specification at least mustn't
> > preclude it. Having brainstormed migration with Matthew Benjamin and
> > Michael Tsirkin, I am now summarizing the approach that I want to
> > include in the next draft specification.
> >
> > Feedback and comments welcome! In the meantime I will implement this in
> > code and update the draft specification.
> >
> > 1. Requirements
> >
> > Virtio-vsock is a new AF_VSOCK transport. As such, it should provide at
> > least the same guarantees as the existing AF_VSOCK VMCI transport. This
> > is for consistency and to allow code reuse across any AF_VSOCK
> > transport.
> >
> > Virtio-vsock aims to replace virtio-serial by providing the same
> > guest/host communication ability but with sockets API semantics that are
> > more popular and convenient for application developers. Therefore
> > virtio-vsock migration should provide at least the same level of
> > migration functionality as virtio-serial.
> >
> > Ideally it should be possible to migrate applications using AF_VSOCK
> > together with the virtual machine so that guest<->host communication is
> > interrupted. Neither AF_VSOCK VMCI nor virtio-serial support this
> > today.
>
> I'm not sure why do you say this about virtio serial.
> It appears that if host pre-connected to destination
> qemu before migration, backend reconnects transparently
> on destination.
You are right, virtio-serial supports keeping active ports open across
migration (as well as closing active ports across migration). In
virtio-vsock the equivalent would be setsockopt() CRIU-style socket
migration which is not implemented today.
> > 2. Basic disruptive migration flow
> >
> > When the virtual machine migrates from the source host to the
> > destination host, the guest's CID may change. The CID namespace is
> > host-wide
>
>
> BTW, I think CIDs would have to become per network namespace.
Yes, I agree.
> > so other hosts may have CID collisions and allocate a new CID
> > for incoming migration VMs.
>
> I guess all this is so that guest can retrieve its CID and
> send it to host using some side-channel?
Yes.
> > The device notifies the guest that the CID has changed. Guest sockets
> > are affected as follows:
> >
> > * Established connections are reset (ECONNRESET) and the guest
> > application will have to reconnect.
> >
> > * Listen sockets remain open. The only thing to note is that
> > connections from the host are now made to the new CID. This means
> > the local address of the listen socket is automatically updated to
> > the new CID.
> >
> > * Sockets in other states are unchanged.
> >
> > Applications must handle disruptive migration by reconnecting if
> > necessary after ECONNRESET.
> >
> > 3. Checkpoint/restore for seamless migration
> >
> > Applications that wish to communicate across live migration can do so
> > but this requires extra application-specific checkpoint/restore code.
> >
> > This is similar to the approach taken by the CRIU project where
> > getsockopt()/setsockopt() is used to migrate socket state. The
> > difference is that the application process is not automatically migrated
> > from the source host to the destination host. Therefore, the
> > application needs to migrate its own state somehow.
> >
> > The flow is as follows:
> >
> > The application on the source host must quiesce (stop sending/receiving)
> > and use getsockopt() to extract socket state information from the host
> > kernel.
> >
> > A new instance of the application is started on the destination host and
> > given the state so it can restore the connection. The setsockopt()
> > syscall is used to restore socket state information.
> >
> > The guest is given a list of <host_old_cid, host_new_cid, host_port,
> > guest_port> tuples for established connections that must not be reset
> > when the guest CID update notification is received. These connections
> > will carry on as if nothing changed.
> >
> > Note that the connection's remote address is updated from host_old_cid
> > to host_new_cid. This allows remapping of CIDs (if necessary).
> > Typically this will be unused because the host always has well-known CID
> > 2. In a guest<->guest scenario it may be used to remap CIDs.
> >
> >
> > For the time being I am focussing on the basic disruptive migration flow
> > only. Checkpoint/restore can be added with a feature bit in the future.
> > It is a lot more complex and I'm not sure whether there will be any
> > users yet.
> >
> > Stefan
>
> This makes some things harder. For example, imagine a guest
> reboot mixed with migration. We don't know why did the connection
> die, so we'll retry connections until - when?
>
> Could you please describe some user of vsock and show how
> it recovers from destructive migration?
qemu-guest-agent runs inside the guest with an AF_VSOCK listen socket.
libvirt arbitrates the qemu-guest-agent connection and provides an API
for applications to send commands.
When an application sends a command, libvirt checks if the connection to
qemu-guest-agent is established. If there is no connection libvirt will
attempt to connect.
The command is sent to qemu-guest-agent and the response is handed back
to the guest application. libvirt arbitrates access so commands from
multiple applications are serialized.
Live migration resets the established connection between
qemu-guest-agent and the source host's libvirt daemon. When an
application issues the next qemu-guest-agent command the libvirt daemon
on the destination host notices there is no established connection yet
and starts a new one.
Libvirt refuses to send qemu-guest-agent commands while live migration
is in progress.
Stefan
