virtio — archive

[Rusty Russell <rusty@au1.ibm.com>]

Signed-off-by: Rusty Russell <rusty@au1.ibm.com> --- content.tex 263 +++++++++++++++++++++++++++++++++++++ ------------------ introduction.tex 4 + 2 files changed, 182 insertions(+), 85 deletions(-) diff --git a/content.tex b/content.tex index e2cf1b7..4b00e6b 100644 --- a/content.tex +++ b/content.tex @@ -2716,9 +2716,10 @@ features. subsection{Feature bits}label{sec:Device Types / Network Device / Feature bits} egin{description} -item[VIRTIO_NET_F_CSUM (0)] Device handles packets with partial checksum +item[VIRTIO_NET_F_CSUM (0)] Device handles packets with partial checksum. This + “checksum offload” is a common feature on modern network cards. -item[VIRTIO_NET_F_GUEST_CSUM (1)] Driver handles packets with partial checksum +item[VIRTIO_NET_F_GUEST_CSUM (1)] Driver handles packets with partial checksum. item[VIRTIO_NET_F_CTRL_GUEST_OFFLOADS (2)] Control channel offloads reconfiguration support. @@ -2762,6 +2763,29 @@ features. channel. end{description} +subsubsection{Feature bit requirements}label{sec:Device Types / Network Device / Feature bits / Feature bit requirements} + +Some networking feature bits require other networking feature bits +(see ef{drivernormative:Basic Facilities of a Virtio Device / Feature Bits}): + +egin{description} +item[VIRTIO_NET_F_GUEST_TSO4] Requires VIRTIO_NET_F_GUEST_CSUM. +item[VIRTIO_NET_F_GUEST_TSO6] Requires VIRTIO_NET_F_GUEST_CSUM. +item[VIRTIO_NET_F_GUEST_ECN] Requires VIRTIO_NET_F_GUEST_TSO4 or VIRTIO_NET_F_GUEST_TSO6. +item[VIRTIO_NET_F_GUEST_UFO] Requires VIRTIO_NET_F_GUEST_CSUM. + +item[VIRTIO_NET_F_HOST_TSO4] Requires VIRTIO_NET_F_CSUM. +item[VIRTIO_NET_F_HOST_TSO6] Requires VIRTIO_NET_F_CSUM. +item[VIRTIO_NET_F_HOST_ECN] Requires VIRTIO_NET_F_HOST_TSO4 or VIRTIO_NET_F_HOST_TSO6. +item[VIRTIO_NET_F_HOST_UFO] Requires VIRTIO_NET_F_CSUM. + +item[VIRTIO_NET_F_CTRL_RX] Requires VIRTIO_NET_F_CTRL_VQ. +item[VIRTIO_NET_F_CTRL_VLAN] Requires VIRTIO_NET_F_CTRL_VQ. +item[VIRTIO_NET_F_GUEST_ANNOUNCE] Requires VIRTIO_NET_F_CTRL_VQ. +item[VIRTIO_NET_F_MQ] Requires VIRTIO_NET_F_CTRL_VQ. +item[VIRTIO_NET_F_CTRL_MAC_ADDR] Requires VIRTIO_NET_F_CTRL_VQ. +end{description} + subsubsection{Legacy Interface: Feature bits}label{sec:Device Types / Network Device / Feature bits / Legacy Interface: Feature bits} egin{description} item[VIRTIO_NET_F_GSO (6)] Device handles packets with any GSO type. @@ -2789,7 +2813,7 @@ VIRTIO_NET_F_MQ is set. This field specifies the maximum number of each of transmit and receive virtqueues (receiveq0..receiveqN and transmitq0..transmitqN respectively; N=field{max_virtqueue_pairs} - 1) that can be configured once VIRTIO_NET_F_MQ -is negotiated. Legal values for this field are 1 to 0x8000. +is negotiated. egin{lstlisting} /* Note: LEGACY version was not little endian! */ @@ -2800,6 +2824,23 @@ struct virtio_net_config { }; end{lstlisting} +devicenormative{Device Types / Network Device / Device configuration layout} + +The device MUST set field{max_virtqueue_pairs} to between 1 and 0x8000 inclusive, +if it offers VIRTIO_NET_F_MQ. + +drivernormative{Device Types / Network Device / Device configuration layout} + +A driver SHOULD negotiate VIRTIO_NET_F_MAC if the device offers it. +If the driver negotiates the VIRTIO_NET_F_MAC feature, the driver MUST set +the physical address of the NIC to field{mac}. Otherwise, it SHOULD +use a locally-administered MAC address (see hyperref[intro:IEEE 802], +"9.2 48-bit universal LAN MAC addresses"). + +If the driver does not negotiate the VIRTIO_NET_F_STATUS feature, it SHOULD +assume the link is active, otherwise it SHOULD read the link status from +the bottom bit of field{status}. + subsubsection{Legacy Interface: Device configuration layout}label{sec:Device Types / Network Device / Device configuration layout / Legacy Interface: Device configuration layout} For legacy devices, field{status} and field{max_virtqueue_pairs} in struct virtio_net_config are the native endian of the guest rather than (necessarily) little-endian. @@ -2807,56 +2848,40 @@ native endian of the guest rather than (necessarily) little-endian. subsection{Device Initialization}label{sec:Device Types / Network Device / Device Initialization} +A driver would perform a typical initialization routine like so: + egin{enumerate} -item The initialization routine should identify the receive and +item Identify and initialize the receive and transmission virtqueues, up to N+1 of each kind. If VIRTIO_NET_F_MQ feature bit is negotiated, N=field{max_virtqueue_pairs}-1, otherwise identify N=0. -item If the VIRTIO_NET_F_MAC feature bit is set, the configuration - space field{mac} entry indicates the “physical” address of the - network card, otherwise a private MAC address should be - assigned. All drivers are expected to negotiate this feature if - it is set. - item If the VIRTIO_NET_F_CTRL_VQ feature bit is negotiated, identify the control virtqueue. +item Fill the receive queues with buffers: see ef{sec:Device Types / Network Device / Device Operation / Setting Up Receive Buffers}. + +item Even with VIRTIO_NET_F_MQ, only receiveq0, transmitq0 and + controlq are used by default. The driver would send the + VIRTIO_NET_CTRL_MQ_VQ_PAIRS_SET command specifying the + number of the transmit and receive queues to use. + +item If the VIRTIO_NET_F_MAC feature bit is set, the configuration + space field{mac} entry indicates the “physical” address of the + network card, otherwise the driver would typically generate a random + local MAC address. + item If the VIRTIO_NET_F_STATUS feature bit is negotiated, the link - status can be read from the bottom bit of field{status}. - Otherwise, the link should be assumed active. - -item Only receiveq0, transmitq0 and controlq are used by default. - To use more queues driver must negotiate the VIRTIO_NET_F_MQ - feature; initialize up to field{max_virtqueue_pairs} of each of - transmit and receive queues; - execute VIRTIO_NET_CTRL_MQ_VQ_PAIRS_SET command specifying the - number of the transmit and receive queues that is going to be - used and wait until the device consumes the controlq buffer and - acks this command. - The receive virtqueue should be filled with receive buffers - before multiqueue is activated - (see ef{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Automatic receive steering in multiqueue mode}~ ameref{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Automatic receive steering in multiqueue mode}). - This is described in detail below in ameref{sec:Device Types / Network Device / Device Operation / Setting Up Receive Buffers}. - -item A driver can indicate that it will generate checksumless - packets by negotating the VIRTIO_NET_F_CSUM feature. This - “checksum offload” is a common feature on modern network cards. + status comes from the bottom bit of field{status}. + Otherwise, the driver assumes it's active. + +item A performant driver would indicate that it will generate checksumless + packets by negotating the VIRTIO_NET_F_CSUM feature. -item If that feature is negotiatedfootnote{ie. VIRTIO_NET_F_HOST_TSO* and VIRTIO_NET_F_HOST_UFO are -dependent on VIRTIO_NET_F_CSUM; a device which offers the offload -features must offer the checksum feature, and a driver which -accepts the offload features must accept the checksum feature. -Similar logic applies to the VIRTIO_NET_F_GUEST_TSO4 features -depending on VIRTIO_NET_F_GUEST_CSUM. -}, a driver can use TCP or UDP +item If that feature is negotiated, a driver can use TCP or UDP segmentation offload by negotiating the VIRTIO_NET_F_HOST_TSO4 (IPv4 TCP), VIRTIO_NET_F_HOST_TSO6 (IPv6 TCP) and VIRTIO_NET_F_HOST_UFO - (UDP fragmentation) features. It should not send TCP packets - requiring segmentation offload which have the Explicit Congestion - Notification bit set, unless the VIRTIO_NET_F_HOST_ECN feature is - negotiated.footnote{This is a common restriction in real, older network cards. -} + (UDP fragmentation) features. item The converse features are also available: a driver can save the virtual device some work by negotiating these features.footnote{For example, a network packet transported between two guests on @@ -2871,6 +2896,9 @@ if both guests are amenable. See ef{sec:Device Types / Network Device / Device Operation / Setting Up Receive Buffers}~ ameref{sec:Device Types / Network Device / Device Operation / Setting Up Receive Buffers} and ef{sec:Device Types / Network Device / Device Operation / Setting Up Receive Buffers}~ ameref{sec:Device Types / Network Device / Device Operation / Setting Up Receive Buffers} below. end{enumerate} +A truly minimal driver would only accept VIRTIO_NET_F_MAC and ignore +everything else. + subsection{Device Operation}label{sec:Device Types / Network Device / Device Operation} Packets are transmitted by placing them in the @@ -2911,10 +2939,11 @@ Transmitting a single packet is simple, but varies depending on the different features the driver negotiated. egin{enumerate} -item If the driver negotiated VIRTIO_NET_F_CSUM, and the packet has - not been fully checksummed, then the virtio_net_hdr's fields - are set as follows. Otherwise, the packet must be fully - checksummed, and flags is zero. +item The driver MAY send a completely checksummed packet. In this case, + field{flags} will be zero, and field{gso_type} will be VIRTIO_NET_HDR_GSO_NONE. + +item If the driver negotiated VIRTIO_NET_F_CSUM, it MAY skip + checksumming the packet: egin{itemize} item field{flags} has the VIRTIO_NET_HDR_F_NEEDS_CSUM set, @@ -2923,17 +2952,20 @@ the different features the driver negotiated. item field{csum_offset} indicates how many bytes after the csum_start the new (16 bit ones' complement) checksum should be placed. + + item The TCP checksum field in the packet is set to the sum + of the TCP pseudo header, so that replacing it by the ones' + complement checksum of the TCP header and body will give the + correct result. end{itemize} +egin{note} For example, consider a partially checksummed TCP (IPv4) packet. It will have a 14 byte ethernet header and 20 byte IP header followed by the TCP header (with the TCP checksum field 16 bytes into that header). field{csum_start} will be 14+20 = 34 (the TCP -checksum includes the header), and field{csum_offset} will be 16. The -value in the TCP checksum field should be initialized to the sum -of the TCP pseudo header, so that replacing it by the ones' -complement checksum of the TCP header and body will give the -correct result. +checksum includes the header), and field{csum_offset} will be 16. +end{note} item If the driver negotiated VIRTIO_NET_F_HOST_TSO4, TSO6 or UFO, and the packet requires @@ -2962,15 +2994,32 @@ specifically in the protocol. end{itemize} item If the driver negotiated the VIRTIO_NET_F_MRG_RXBUF feature, - field{num_buffers} is set to zero. + field{num_buffers} is set to zero. This field is unused on transmitted packets. -item The header and packet are added as one output buffer to the +item The header and packet are added as one output descriptor to the transmitq, and the device is notified of the new entry (see ef{sec:Device Types / Network Device / Device Initialization}~ ameref{sec:Device Types / Network Device / Device Initialization}).footnote{Note that the header will be two bytes longer for the VIRTIO_NET_F_MRG_RXBUF case. } end{enumerate} +drivernormative{Device Types / Network Device / Device Operation / Packet Transmission} + +If a driver has not negotiated VIRTIO_NET_F_CSUM, field{flags} MUST be zero and +the packet must be fully checksummed. + +If a driver negotiated the VIRTIO_NET_F_MRG_RXBUF feature, it MUST include +field{num_buffers} in the header, and it MUST set the value to zero. If a driver +did not negotiate VIRTIO_NET_F_MRG_RXBUF, it MUST NOT include field{num_buffers} in the header. +egin{note} + ie. With VIRTIO_NET_F_MRG_RXBUF, both receive and transmit headers + are 12 bytes. Without it, they're 10 bytes. +end{note} + +A driver SHOULD NOT send TCP packets requiring segmentation offload which have the Explicit Congestion Notification bit set, unless the VIRTIO_NET_F_HOST_ECN feature is +negotiatedfootnote{This is a common restriction in real, older network cards.}, in +which case it MUST set the VIRTIO_NET_HDR_GSO_ECN bit in field{gso_type}. + paragraph{Packet Transmission Interrupt}label{sec:Device Types / Network Device / Device Operation / Packet Transmission / Packet Transmission Interrupt} Often a driver will suppress transmission interrupts using the @@ -2990,19 +3039,33 @@ fully populated as possible: if it runs out, network performance will suffer. If the VIRTIO_NET_F_GUEST_TSO4, VIRTIO_NET_F_GUEST_TSO6 or -VIRTIO_NET_F_GUEST_UFO features are used, the Driver will need to -accept packets of up to 65550 bytes long (the maximum size of a +VIRTIO_NET_F_GUEST_UFO features are used, the maximum incoming packet +will be to 65550 bytes long (the maximum size of a TCP or UDP packet, plus the 14 byte ethernet header), otherwise -1514. bytes. So unless VIRTIO_NET_F_MRG_RXBUF is negotiated, every -buffer in the receive queue needs to be at least this length.footnote{Obviously each one can be split across multiple descriptor -elements. -} +1514 bytes. The 12-byte struct virtio_net_hdr is prepended to this, +making for 65562 or 1526 bytes. + +drivernormative{Device Types / Network Device / Device Operation / Setting Up Receive Buffers} -If VIRTIO_NET_F_MRG_RXBUF is negotiated, each buffer must be at -least the size of the struct virtio_net_hdr. +egin{itemize} +item If VIRTIO_NET_F_MRG_RXBUF is not negotiated: + egin{itemize} + item If VIRTIO_NET_F_GUEST_TSO4, VIRTIO_NET_F_GUEST_TSO6 or + VIRTIO_NET_F_GUEST_UFO are negotiated, the driver SHOULD populate + the receive queue(s) with buffers of at least 65562 bytes. + item Otherwise, the driver SHOULD populate the receive queue(s) + with buffers of at least 1526 bytes. + end{itemize} +item If VIRTIO_NET_F_MRG_RXBUF is negotiated, each buffer MUST be at + least the size of the struct virtio_net_hdr. +end{itemize} + +egin{note} +Obviously each buffer can be split across multiple descriptor elements. +end{note} If VIRTIO_NET_F_MQ is negotiated, each of receiveq0...receiveqN -that will be used should be populated with receive buffers. +that will be used SHOULD be populated with receive buffers. paragraph{Packet Receive Interrupt}label{sec:Device Types / Network Device / Device Operation / Setting Up Receive Buffers / Packet Receive Interrupt} @@ -3029,7 +3092,7 @@ Processing packet involves: virtio_net_hdr. item If the VIRTIO_NET_F_GUEST_CSUM feature was negotiated, the - VIRTIO_NET_HDR_F_NEEDS_CSUM bit in field{flags} may be + VIRTIO_NET_HDR_F_NEEDS_CSUM bit in field{flags} MAY be set: if so, the checksum on the packet is incomplete and field{csum_start} and field{csum_offset} indicate how to calculate it (see Packet Transmission point 1). @@ -3113,11 +3176,6 @@ command-specific-data is two variable length tables of 6-byte MAC addresses. The first table contains unicast addresses, and the second contains multicast addresses. -When VIRTIO_NET_F_MAC_ADDR is not negotiated, field{mac} in the -config space is writeable and is used to set the default MAC -address which rx filtering accepts. -When VIRTIO_NET_F_MAC_ADDR is negotiated, field{mac} in the -config space becomes read-only for the driver. The VIRTIO_NET_CTRL_MAC_ADDR_SET command is used to set the default MAC address which rx filtering accepts. @@ -3129,6 +3187,11 @@ accepts. The command-specific-data for VIRTIO_NET_CTRL_MAC_ADDR_SET is the 6-byte MAC address. +drivernormative{Device Types / Network Device / Device Operation / Control Virtqueue / Setting MAC Address Filtering} + +A driver MUST NOT write to the field{mac} if VIRTIO_NET_F_MAC_ADDR is +negotiated. + The VIRTIO_NET_CTRL_MAC_ADDR_SET command is atomic whereas field{mac} in config space is not, therefore drivers @@ -3180,17 +3243,16 @@ the guest in this way). #define VIRTIO_NET_CTRL_ANNOUNCE_ACK 0 end{lstlisting} -The Driver needs to check VIRTIO_NET_S_ANNOUNCE bit in status -field when it notices the changes of device configuration. The +The driver checks VIRTIO_NET_S_ANNOUNCE bit in the device configuration field{status} field +when it notices the changes of device configuration. The command VIRTIO_NET_CTRL_ANNOUNCE_ACK is used to indicate that -driver has received the notification and device would clear the -VIRTIO_NET_S_ANNOUNCE bit in the status filed after it received -this command. +driver has received the notification and device clears the +VIRTIO_NET_S_ANNOUNCE bit in field{status}. Processing this notification involves: egin{enumerate} -item Sending the gratuitous packets or marking there are pending +item Sending the gratuitous packets (eg. ARP) or marking there are pending gratuitous packets to be sent and letting deferred routine to send them. @@ -3198,6 +3260,20 @@ Processing this notification involves: vq. end{enumerate} +drivernormative{Device Types / Network Device / Device Operation / Control Virtqueue / Gratuitous Packet Sending} + +If the driver negotiates VIRTIO_NET_F_GUEST_ANNOUNCE, it SHOULD notify +network peers of its new location after it sees the VIRTIO_NET_S_ANNOUNCE bit +in field{status}. The driver MUST send a command on the command queue +with class VIRTIO_NET_CTRL_ANNOUNCE and command VIRTIO_NET_CTRL_ANNOUNCE_ACK. + +devicenormative{Device Types / Network Device / Device Operation / Control Virtqueue / Gratuitous Packet Sending} + +If VIRTIO_NET_F_GUEST_ANNOUNCE is negotiated, the device MUST clear the +VIRTIO_NET_S_ANNOUNCE bit in field{status} upon receipt of a command buffer +with class VIRTIO_NET_CTRL_ANNOUNCE and command VIRTIO_NET_CTRL_ANNOUNCE_ACK +before marking the buffer as used. + paragraph{Automatic receive steering in multiqueue mode}label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Automatic receive steering in multiqueue mode} If the driver negotiates the VIRTIO_NET_F_MQ feature bit (depends @@ -3220,11 +3296,10 @@ struct virtio_net_ctrl_mq { Multiqueue is disabled by default. The driver enables multiqueue by executing the VIRTIO_NET_CTRL_MQ_VQ_PAIRS_SET command, specifying -the number of the transmit and receive queues to be used; subsequently, +the number of the transmit and receive queues to be used up to +field{max_virtqueue_pairs}; subsequently, transmitq0..transmitqn and receiveq0..receiveqn where -n=virtqueue_pairs-1 MAY be used. All these virtqueues MUST have -been pre-configured in advance. The range of legal values for the -field{virtqueue_pairs} field is between 1 and field{max_virtqueue_pairs}. +n=virtqueue_pairs-1 MAY be used. When multiqueue is enabled, the device MUST use automatic receive steering based on packet flow. Programming of the receive steering @@ -3235,12 +3310,29 @@ no packets have been transmitted yet, the device MAY steer a packet to a random queue out of the specified receiveq0..receiveqn. Multiqueue is disabled by setting field{virtqueue_pairs} to 1 (this is -the default). After the command has been consumed by the device, the -device MUST NOT steer new packets to virtqueues -receveq1..receiveqN (i.e. other than receiveq0) and MUST NOT read from -transmitq1..transmitqN (i.e. other than transmitq0); accordingly, -the driver MUST NOT transmit new packets on virtqueues other than -transmitq0. +the default) and waiting for the device to use the command buffer. + +drivernormative{Device Types / Network Device / Device Operation / Control Virtqueue / Automatic receive steering in multiqueue mode} + +The driver MUST configure the virtqueues before enabling them with the +VIRTIO_NET_CTRL_MQ_VQ_PAIRS_SET command. + +The driver MUST NOT request a field{virtqueue_pairs} of 0 or +greater than field{max_virtqueue_pairs} in the device configuration space. + +The driver MUST queue packets only on any transmitq0 before the +VIRTIO_NET_CTRL_MQ_VQ_PAIRS_SET command. + +The driver MUST NOT queue packets on transmit queues greater than +field{virtqueue_pairs} once it has placed the VIRTIO_NET_CTRL_MQ_VQ_PAIRS_SET command in the available ring. + +devicenormative{Device Types / Network Device / Device Operation / Control Virtqueue / Automatic receive steering in multiqueue mode} + +The device MUST queue packets only on any receiveq0 before the +VIRTIO_NET_CTRL_MQ_VQ_PAIRS_SET command. + +The device MUST NOT queue packets on receive queues greater than +field{virtqueue_pairs} once it has placed the VIRTIO_NET_CTRL_MQ_VQ_PAIRS_SET command in the used ring. subparagraph{Legacy Interface: Automatic receive steering in multiqueue mode}label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Automatic receive steering in multiqueue mode / Legacy Interface: Automatic receive steering in multiqueue mode} For legacy devices, field{virtqueue_pairs} is in the @@ -3276,9 +3368,10 @@ There is a corresponding device feature for each offload. Upon feature negotiation corresponding offload gets enabled to preserve backward compartibility. -Corresponding feature must be negotiated at startup in order to allow dynamic -change of specific offload state. +drivernormative{Device Types / Network Device / Device Operation / Control Virtqueue / Offloads State Configuration / Setting Offloads State} +A driver MUST NOT enable a offload for which the appropriate feature +has not been negotiated. subparagraph{Legacy Interface: Setting Offloads State}label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Offloads State Configuration / Setting Offloads State / Legacy Interface: Setting Offloads State} For legacy devices, field{offloads} is the diff --git a/introduction.tex b/introduction.tex index 65392d9..d098718 100644 --- a/introduction.tex +++ b/introduction.tex @@ -82,6 +82,10 @@ To simplify transition and note differences, the following terms are used: phantomsectionlabel{intro:Virtio PCI Draft} extbf{[Virtio PCI Draft]} & Virtio PCI Draft Specification ewlineurl{ http://ozlabs.org/~rusty/virtio-spec/virtio-0.9.5.pdf}\ + phantomsectionlabel{intro:IEEE 802} extbf{[IEEE 802]} & + IEEE Standard for Local and Metropolitan Area Networks: Overview and Architecture, + ewlineurl{ http://standards.ieee.org/about/get/802/802.html} , + IEEE\ end{longtable} section{Structure Specifications} -- 1.8.3.2