On Sun, Jul 31, 2022 at 06:43:54PM +0300, Max Gurtovoy wrote:
> Introduce the concept of a management and a managed device and add
> example of using this concept to manage resources.
>
> A management device (from any type) supports the
> VIRTIO_ADMIN_DEVICE_MGMT and VIRTIO_ADMIN_DEVICE_MGMT_ATTRS admin
> commands to manage some resources of a managed device.
>
> Reviewed-by: Parav Pandit <>
> Signed-off-by: Max Gurtovoy <>
Actually this does two things:
- more SRIOV group definitions
- new MGMT commands
> ---
> admin.tex
| 96 +++++++++++++++++++++++++++++++++++++++++++++++-
> content.tex
| 50 +++++++++++++++++++++++++
> introduction.tex | 30 +++++++++++++++
> 3 files changed, 175 insertions(+), 1 deletion(-)
>
> diff --git a/admin.tex b/admin.tex
> index 26ac60e..73bc411 100644
> --- a/admin.tex
> +++ b/admin.tex
> @@ -75,12 +75,106 @@ \section{Administration command set}\label{sec:Basic Facilities of a Virtio Devi
> \hline
> Opcode & Command \\
> \hline \hline
> -0000h - 7FFFh
& Generic admin cmds
\\
> +0000h
& VIRTIO_ADMIN_DEVICE_MGMT
\\
> +\hline
> +0001h
& VIRTIO_ADMIN_DEVICE_MGMT_ATTRS
\\
> +\hline
> +0002h - 7FFFh
& Generic admin cmds
\\
> \hline
> 8000h - FFFFh
& Reserved
\\
> \hline
> \end{tabular}
>
> +\begin{note}
> +{The following commands are mandatory for management devices: VIRTIO_ADMIN_DEVICE_MGMT and VIRTIO_ADMIN_DEVICE_MGMT_ATTRS.}
> +\end{note}
this is the first time we mention management devices. We should not use
terms before they are defined.
> +
> +\subsection{VIRTIO ADMIN DEVICE MGMT command}\label{sec:Basic Facilities of a Virtio Device / Admin command set / VIRTIO ADMIN DEVICE MGMT command}
> +
> +The VIRTIO_ADMIN_DEVICE_MGMT command is used by a management device to manage resources of managed virtio devices.
Please rephrase and change terminology to avoid saying manage 3 times in
a single sentence :)
In this case we have concepts of a group already.
Can these commands ever work for the self type?
I don't think they can, right?
> +The \field{command} is set to VIRTIO_ADMIN_DEVICE_MGMT by the driver.
> +
> +The command specific data set by the driver is of form:
> +\begin{lstlisting}
> +struct virtio_admin_device_mgmt_data {
> +
/*
> +
* 0 - reserved
> +
* 1 - assign resource to the designated device
> +
* 2 - query resource of the designated device
> +
* 3 - 255 are reserved for future operations
> +
*/
> +
u8 operation;
> +
/*
> +
* 0 - Device feature bits 0 to 63
> +
* 1 - 65535 are reserved
> +
*/
> +
le16 resource;
I think we need to move padding around here to align the next field.
> +
/*
> +
* The value to the given resource.
> +
* Valid only if operation == 1 (assign).
> +
* resource = 0 indicates device feature bits 0 to 63.
> +
*/
> +
le64 resource_val;
> +
u8 reserved[5];
> +};
> +\end{lstlisting}
> +
> +The following table describes the command specific error codes codes:
> +
> +\begin{tabular}{|l|l|l|}
> +\hline
> +Opcode & Status & Description \\
> +\hline \hline
> +00h
& VIRTIO_ADMIN_CS_ERR_DEV_IN_USE
& designated device is in use, operation failed
\\
> +\hline
> +01h
& VIRTIO_ADMIN_CS_RSC_VAL_INVALID
& resource value is invalid \\
> +\hline
> +02h
& VIRTIO_ADMIN_CS_RSC_UNSUPPORTED
& unsupported or invalid resource \\
> +\hline
> +03h
& VIRTIO_ADMIN_CS_OP_UNSUPPORTED
& unsupported or invalid operation \\
> +\hline
> +04h - FFh
& Reserved
& - \\
> +\hline
> +\end{tabular}
> +
> +The device, upon success, returns a result that describes the information according to the requested operation.
> +This result is of form:
> +\begin{lstlisting}
> +struct virtio_admin_device_mgmt_result {
> +
le64 resource_val;
> +
u8 reserved[8];
> +};
> +\end{lstlisting}
> +
> +If the \field{operation} was set to 1 ("assign resource to the designated device") and the command succeeded, the device successfully assigned
> +resources to the designated device. Also, upon success, the assigned value should returned in the \field{resource_val} of the virtio_admin_device_mgmt_result
> +structure and should be equal to the \field{resource_val} of the virtio_admin_device_mgmt_data structure set by the driver.
> +In case of a failure, the value of this field is undefined and will be ignored by the driver.
> +
> +If the \field{operation} was set to 2 ("query resource of the designated device") and the command succeeded, the device will return the requested
> +value in the \field{resource_val} of the virtio_admin_device_mgmt_result structure.
> +In case of a failure, the value of this field is undefined and will be ignored by the driver.
> +
> +\subsection{VIRTIO ADMIN DEVICE MGMT ATTRS command}\label{sec:Basic Facilities of a Virtio Device / Admin command set / VIRTIO ADMIN DEVICE MGMT ATTRS command}
> +
> +The VIRTIO_ADMIN_DEVICE_MGMT_ATTRS command has no command specific data set by the driver.
> +The \field{command} is set to VIRTIO_ADMIN_DEVICE_MGMT_ATTRS.
> +
> +The device, upon success, returns a result that describes the management device attributes.
> +This result is of form:
> +\begin{lstlisting}
> +struct virtio_admin_device_mgmt_attrs_result {
> +
/* Indicates which of the below fields are valid.
> +
* Bits 0 - managed_device_features_63_0
> +
* Bits 1 - 63 - reserved for future fields
> +
*/
> +
le64 attrs_mask;
> +
> +
le64 managed_device_features_63_0;
> +
u8 reserved[48];
why are we reserving these?
> +};
> +\end{lstlisting}
> +
> \section{Admin Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues}
>
> An admin virtqueue is a management interface of a device that can be used to send administrative
> diff --git a/content.tex b/content.tex
> index 5fda1a0..b15a887 100644
> --- a/content.tex
> +++ b/content.tex
> @@ -451,6 +451,48 @@ \section{Exporting Objects}\label{sec:Basic Facilities of a Virtio Device / Expo
>
> \input{admin.tex}
>
> +\section{Device management}\label{sec:Basic Facilities of a Virtio Device / Device management}
> +
> +A device group might consist of one or more virtio devices. For example, virtio PCI SR-IOV PF and its VFs compose a SR-IOV type device group.
> +A capable PCI SR-IOV PF virtio device might act as the management device in this group, and its PCI SR-IOV VFs are the managed devices.
> +A management device might have various management capabilities and attributes to manage its managed devices. These capabilities and attributes exposed
> +via feature bits and in the result of VIRTIO_ADMIN_DEVICE_MGMT_ATTRS command (see section \ref{sec:Basic Facilities of a Virtio Device / Admin command set / VIRTIO ADMIN DEVICE MGMT ATTRS command} for more details).
> +
> +The management device will use the VIRTIO_ADMIN_DEVICE_MGMT admin command to manage its managed devices (see section
> +\ref{sec:Basic Facilities of a Virtio Device / Admin command set / VIRTIO ADMIN DEVICE MGMT command} for more details).
> +
> +\subsection{Device features management}\label{sec:Basic Facilities of a Virtio Device / Device management / Device features management}
> +
> +A virtio management device that negotiated VIRTIO_F_FEATURE_63_0_MGMT can control the values of the first 64 feature bits for its managed devices.
> +In the result of VIRTIO_ADMIN_DEVICE_MGMT_ATTRS admin command, a capable management device will return the feature bits that can be enabled for its managed devices in \field{managed_device_features_63_0}
> +(see section \ref{sec:Basic Facilities of a Virtio Device / Admin command set / VIRTIO ADMIN DEVICE MGMT ATTRS command} for more details).
> +A management device driver, using VIRTIO_ADMIN_DEVICE_MGMT can update the feature bits assignment for a specific managed device.
> +A management device driver can only assign a subset of the bits returned in \field{managed_device_features_63_0}.
> +A management device driver can update the managed device feature bits only before the managed device driver set the ACKNOWLEDGE status bit.
> +
> +\begin{note}
> +{It's recommended to manipulate the managed device feature bits before loading the managed device driver.}
> +\end{note}
> +
> +A typical sequence for configuring feature bits is following:
> +
> +\begin{enumerate}
> +\item Ensure that the driver of the managed device doesn't run on the device
> +
> +\item Load the driver of the management device
> +
> +\item Query the management device capabilities using commands VIRTIO_ADMIN_DEVICE_MGMT_ATTRS and look for \field{managed_device_features_63_0}
> +
> +\item Find the managed device group_id and group_member_id
> +
> +\item Query the current feature bits configuration using command VIRTIO_ADMIN_DEVICE_MGMT (query operation)
> +
> +\item Assign desired feature bits for the managed device using command VIRTIO_ADMIN_DEVICE_MGMT (assign operation)
> +
> +\item After successful completion of the assignment, load the managed device driver
> +
> +\end{enumerate}
> +
> \chapter{General Initialization And Device Operation}\label{sec:General Initialization And Device Operation}
>
> We start with an overview of device initialization, then expand on the
> @@ -6853,6 +6895,14 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}
>
>
\item[VIRTIO_F_ADMIN_VQ (41)] This feature indicates that an administration virtqueue is supported.
>
> + \item[VIRTIO_F_SR_IOV_MGMT_DEVICE (42)] This feature indicates that the device is a SR-IOV management device.
> + SR-IOV management device is a device that can manage other devices within the SR-IOV device group.
> + For more details see \ref{sec:Introduction / Terminology / Virtio SR-IOV type management device}.
> +
> + \item[VIRTIO_F_FEATURE_63_0_MGMT (43)] This feature indicates that the device is a management device that supports
> + feature mgmt of bits 0 to 63 for its managed devices.
eschew abbreviation in text please.
why limit this to bits 0 to 63? why not make the command cover any number of bits?
> + For more details see \ref{sec:Basic Facilities of a Virtio Device / Device management / Device features management}.
> +
> \end{description}
>
> \drivernormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}
> diff --git a/introduction.tex b/introduction.tex
> index e8bde45..e20df24 100644
> --- a/introduction.tex
> +++ b/introduction.tex
> @@ -173,6 +173,9 @@ \subsection{Device group}\label{sec:Introduction / Terminology / Device group}
> \item SR-IOV type (group identifier = 1) - this group includes a virtio PCI SR-IOV physical function (PF) and all its virtual functions (VFs).
> For this group type, the PF device has group member identifier of 0. Each VF group member identifier equals the PCI VF number according to the PCI Express Base Specification
> (Single Root I/O Virtualization and Sharing chapter). Devices that are members in this group use the Virtio PCI transport (for more details see \ref{sec:Virtio Transport Options / Virtio Over PCI Bus}).
> +A PCI SR-IOV PF device can act as a management device for SR-IOV type device group. A PCI SR-IOV VF device can act as a managed device for SR-IOV type device group
> +(see \ref{sec:Introduction / Terminology / Virtio management device} and \ref{sec:Introduction / Terminology / Virtio managed device} for more information).
> +SR-IOV type device group may contain zero or one management devices.
> \end{enumerate}
>
> \begin{note}
> @@ -180,6 +183,33 @@ \subsection{Device group}\label{sec:Introduction / Terminology / Device group}
>
member identifier equals to 0 within Self type group and a group member identifier equals to VF number (e.g 4) within SR-IOV type group.
> \end{note}
>
> +\subsection{Virtio management device}\label{sec:Introduction / Terminology / Virtio management device}
> +
> +A virtio device that supports VIRTIO_ADMIN_DEVICE_MGMT and VIRTIO_ADMIN_DEVICE_MGMT_ATTRS admin commands (see
> +\ref{sec:Basic Facilities of a Virtio Device / Admin command set / VIRTIO ADMIN DEVICE MGMT command} and
> +\ref{sec:Basic Facilities of a Virtio Device / Admin command set / VIRTIO ADMIN DEVICE MGMT ATTRS command} for more information).
> +This device can manage a virtio managed device. A device group may contain zero or more management devices.
> +
> +A PCI SR-IOV Physical Function based virtio device is an example of a possible virtio management device (for SR-IOV type device group).
> +
> +\subsection{Virtio SR-IOV type management device}\label{sec:Introduction / Terminology / Virtio SR-IOV type management device}
> +
> +A virtio management device for SR-IOV type device group. A driver can issue the management admin commands to this device with \field{group_id} set to 1
> +and \field{group_member_id} set to an identifier that corresponds with one of its managed virtio devices (PCI SR-IOV VFs).
> +A Virtio SR-IOV type management device must expose VIRTIO_F_SR_IOV_MGMT_DEVICE feature bit.
> +
> +\subsection{virtio managed device}\label{sec:Introduction / Terminology / Virtio managed device}
> +
> +A virtio device that can be managed by a virtio management device.
> +A device group may contain zero or more managed devices.
> +
> +A PCI SR-IOV Virtual Function based virtio device is an example of a possible virtio managed device (for SR-IOV type device group).
> +
> +\subsection{virtio SR-IOV type managed device}\label{sec:Introduction / Terminology / Virtio SR-IOV type managed device}
> +
> +A virtio managed device for SR-IOV type device group. This device is a PCI SR-IOV VF and is managed by a virtio SR-IOV type management device (virtio PCI SR-IOV PF).
> +It is implied that all the virtio PCI SR-IOV VFs related to a virtio PCI SR-IOV PF that is virtio SR-IOV type management device are SR-IOV type managed devices.
> +
> \section{Structure Specifications}\label{sec:Structure Specifications}
>
> Many device and driver in-memory structure layouts are documented using
> --
> 2.21.0
