path: root/virtio-video.tex

\section{Video Device}\label{sec:Device Types / Video Device}

The virtio video encoder device and decoder device are virtual devices
that supports encoding and decoding respectively. While the encoder
and the decoder are different devices, they use the same protocol.

\subsection{Device ID}
\label{sec:Device Types / Video Device / Device ID}

\begin{description}
\item[30] encoder device
\item[31] decoder device
\end{description}

\subsection{Virtqueues}
\label{sec:Device Types / Video Device / Virtqueues}

\begin{description}
\item[0] commandq - queue for sending commands.
\item[1] eventq - queue for sending events happened in the device.
\end{description}

\subsection{Feature bits}
\label{sec:Device Types / Video Device / Feature bits}

\begin{description}
\item[VIRTIO_VIDEO_F_RESOURCE_GUEST_PAGES (0)] Guest pages can be used
  for video buffers.
\item[VIRTIO_VIDEO_F_RESOURCE_NON_CONTIG (1)] The device can use
  non-contiguous memories for video buffers. Without this flag, the
  driver and device MUST use video buffers that are contiguous in the
  device-side.
\item[VIRTIO_VIDEO_F_RESOURCE_VIRTIO_OBJECT (2)] Object exported by
  another virtio device can be used for video buffers.
\end{description}

\devicenormative{\subsubsection}{Feature bits}{Device Types / Video
  Device / Feature bits}

The device MUST offer at least either
VIRTIO_VIDEO_F_RESOURCE_GUEST_PAGES or
VIRTIO_VIDEO_F_RESOURCE_VIRTIO_OBJECT.

\subsection{Device configuration layout}
\label{sec:Device Types / Video Device / Device configuration layout}

Video device configuration uses the following layout structure:

\begin{lstlisting}
struct virtio_video_config {
        le32 version;
        le32 max_caps_length;
        le32 max_resp_length;
};
\end{lstlisting}

\begin{description}
\item[\field{version}] is the protocol version that the device talks.
  The device MUST set this to 0.
\item[\field{max_caps_length}] defines the maximum length of a
  descriptor required to call VIRTIO_VIDEO_CMD_QUERY_CAPABILITY in
  bytes. The device MUST set this value.
\item[\field{max_resp_length}] defines the maximum length of a
  descriptor required to call a command other than
  VIRTIO_VIDEO_CMD_QUERY_CAPABILITY in bytes. The device MUST set this
  value.
\end{description}

\subsection{Device Initialization}
\label{sec:Device Types / Video Device / Device Initialization}

\devicenormative{\subsubsection}{Device Initialization}{Device Types /
  Video Device / Device Initialization}

The driver SHOULD query device capability by using the
VIRTIO_VIDEO_CMD_QUERY_CAPABILITY and use that information for the
initial setup.

\subsection{Device Operation}
\label{sec:Device Types / Video Device / Device Operation}

The driver allocates input and output buffers and queues the buffers
to the device. The device performs operations on the buffers according
to the function in question.

\subsubsection{Command Virtqueue}

\paragraph{Device Operation: Command header}

All commands and responses on the command virtqueue have a fixed
header using the following layout structure and definitions:

\begin{lstlisting}
  enum virtio_video_cmd_type {
        /* Command */
        VIRTIO_VIDEO_CMD_QUERY_CAPABILITY = 0x0100,
        VIRTIO_VIDEO_CMD_STREAM_CREATE,
        VIRTIO_VIDEO_CMD_STREAM_DESTROY,
        VIRTIO_VIDEO_CMD_STREAM_DRAIN,
        VIRTIO_VIDEO_CMD_RESOURCE_CREATE,
        VIRTIO_VIDEO_CMD_RESOURCE_QUEUE,
        VIRTIO_VIDEO_CMD_RESOURCE_DESTROY_ALL,
        VIRTIO_VIDEO_CMD_QUEUE_CLEAR,
        VIRTIO_VIDEO_CMD_GET_PARAMS,
        VIRTIO_VIDEO_CMD_SET_PARAMS,
        VIRTIO_VIDEO_CMD_QUERY_CONTROL,
        VIRTIO_VIDEO_CMD_GET_CONTROL,
        VIRTIO_VIDEO_CMD_SET_CONTROL,

        /* Response */
        VIRTIO_VIDEO_RESP_OK_NODATA = 0x0200,
        VIRTIO_VIDEO_RESP_OK_QUERY_CAPABILITY,
        VIRTIO_VIDEO_RESP_OK_RESOURCE_QUEUE,
        VIRTIO_VIDEO_RESP_OK_GET_PARAMS,
        VIRTIO_VIDEO_RESP_OK_QUERY_CONTROL,
        VIRTIO_VIDEO_RESP_OK_GET_CONTROL,

        VIRTIO_VIDEO_RESP_ERR_INVALID_OPERATION = 0x0300,
        VIRTIO_VIDEO_RESP_ERR_OUT_OF_MEMORY,
        VIRTIO_VIDEO_RESP_ERR_INVALID_STREAM_ID,
        VIRTIO_VIDEO_RESP_ERR_INVALID_RESOURCE_ID,
        VIRTIO_VIDEO_RESP_ERR_INVALID_PARAMETER,
        VIRTIO_VIDEO_RESP_ERR_UNSUPPORTED_CONTROL,
};

struct virtio_video_cmd_hdr {
        le32 type;
        le32 stream_id;
};
\end{lstlisting}

The fixed header \field{virtio_video_cmd_hdr} in each message includes
the following field:
\begin{description}
\item[\field{type}] specifies the type of the driver command
  (VIRTIO_VIDEO_CMD_*) or the device response (VIRTIO_VIDEO_RESP_*).
\item[\field{stream_id}] specifies a target stream if needed.
\end{description}

On success the device will return VIRTIO_VIDEO_RESP_OK_NODATA in case
there is no payload. Otherwise the \field{type} field will indicate
the kind of payload.

On error the device will return one of the VIRTIO_VIDEO_RESP_ERR_*
error codes.

\paragraph{Device Operation: Query device capability}

\begin{description}
\item[VIRTIO_VIDEO_CMD_QUERY_CAPABILITY] Retrieve information about
supported formats.

The driver uses \field{virtio_video_query_capability} to send a
query request.

\begin{lstlisting}
enum virtio_video_queue_type {
        VIRTIO_VIDEO_QUEUE_TYPE_INPUT = 0x100,
        VIRTIO_VIDEO_QUEUE_TYPE_OUTPUT,
};

struct virtio_video_query_capability {
        struct virtio_video_ctrl_hdr hdr;
        le32 queue_type;
        u8 padding[4];
};
\end{lstlisting}
\begin{description}
\item[\field{queue_type}] is the queue type that the driver asks
information about. The driver MUST set either
\field{VIRTIO_VIDEO_QUEUE_TYPE_INPUT} or
\field{VIRTIO_VIDEO_QUEUE_TYPE_OUTPUT}.
\end{description}

The device responds to VIRTIO_VIDEO_CMD_QUERY_CAPABILITY with
\field{virtio_video_query_capability_resp}.
\begin{lstlisting}
struct virtio_video_query_capability_resp {
        struct virtio_video_cmd_hdr hdr;
        le32 num_descs;
        u8 padding[4];
        /* Followed by struct virtio_video_format_desc descs[] */
};
\end{lstlisting}

The device MUST return its capability with \field{
  virtio_video_capability_resp} that includes the following fields:
\begin{description}
\item[\field{num_descs}] is a number of
  \field{virtio_video_format_desc} that follow. The value MUST not
  exceed 64.
\end{description}

The format description \field{virtio_video_format_desc} is defined as
follows:
\begin{lstlisting}
enum virtio_video_format {
        /* Raw formats */
        VIRTIO_VIDEO_FORMAT_RAW_MIN = 1,
        VIRTIO_VIDEO_FORMAT_ARGB8888 = VIRTIO_VIDEO_FORMAT_RAW_MIN,
        VIRTIO_VIDEO_FORMAT_BGRA8888,
        VIRTIO_VIDEO_FORMAT_NV12, /* 12  Y/CbCr 4:2:0  */
        VIRTIO_VIDEO_FORMAT_YUV420, /* 12  YUV 4:2:0     */
        VIRTIO_VIDEO_FORMAT_YVU420, /* 12  YVU 4:2:0     */
        VIRTIO_VIDEO_FORMAT_RAW_MAX = VIRTIO_VIDEO_FORMAT_YVU420,

        /* Coded formats */
        VIRTIO_VIDEO_FORMAT_CODED_MIN = 0x1000,
        VIRTIO_VIDEO_FORMAT_MPEG2 = VIRTIO_VIDEO_FORMAT_CODED_MIN, /* MPEG-2 Part 2 */
        VIRTIO_VIDEO_FORMAT_MPEG4, /* MPEG-4 Part 2 */
        VIRTIO_VIDEO_FORMAT_H264, /* H.264 */
        VIRTIO_VIDEO_FORMAT_HEVC, /* HEVC aka H.265*/
        VIRTIO_VIDEO_FORMAT_VP8, /* VP8 */
        VIRTIO_VIDEO_FORMAT_VP9, /* VP9 */
        VIRTIO_VIDEO_FORMAT_CODED_MAX = VIRTIO_VIDEO_FORMAT_VP9,
};

enum virtio_video_planes_layout_flag {
        VIRTIO_VIDEO_PLANES_LAYOUT_SINGLE_BUFFER = 1 << 0,
        VIRTIO_VIDEO_PLANES_LAYOUT_PER_PLANE = 1 << 1,
};

struct virtio_video_format_desc {
        le64 mask;
        le32 format; /* One of VIRTIO_VIDEO_FORMAT_* types */
        le32 planes_layout; /* Bitmask with VIRTIO_VIDEO_PLANES_LAYOUT_* */
        le32 plane_align;
        le32 num_frames;
        /* Followed by struct virtio_video_format_frame frames[] */
};
\end{lstlisting}
\begin{description}
\item[\field{mask}] is a bitset that represents the supported
  combination of input and output format. If \textit{i}-th bit is set
  in \field{mask} of \textit{j}-th \field{virtio_video_format_desc}
  for input, the device supports encoding or decoding from the
  \textit{j}-th input format to \textit{i}-th output format.
\item[\field{format}] specifies an image format. The device MUST set
  one of \field{enum virtio_video_format}.
\item[\field{planes_layout}] is a bitmask representing a set of plane
  layout types the device supports. This driver MUST ignore this field
  for encoded formats.
  \begin{description}
  \item[\field{VIRTIO_VIDEO_PLANES_LAYOUT_SINGLE_BUFFER}] The device
    expects planes in one buffer laid out one after another.
  \item[\field{VIRTIO_VIDEO_PLANES_LAYOUT_PER_PLANE}] The device
    expects planes to be located in separate buffers.
  \end{description}
\item[\field{plane_align}] is a plane alignment the device require
  when multiple planes are located in one buffer. The driver MUST
  ignore this field if \field{format} is a bitstream format or
  \field{planes_layout} doesn't have
  \field{VIRTIO_VIDEO_PLANES_LAYOUT_SINGLE_BUFFER} bit.
\item[\field{num_frames}] is the number of
  \field{virtio_video_format_frame} that follows.
\end{description}

The frame information \field{virtio_video_format_frame} is defined as
follows:
\begin{lstlisting}
struct virtio_video_format_range {
        le32 min;
        le32 max;
        le32 step;
        u8 padding[4];
};

struct virtio_video_format_frame {
        struct virtio_video_format_range width;
        struct virtio_video_format_range height;
        le32 num_rates;
        u8 padding[4];
        /* Followed by struct virtio_video_format_range frame_rates[] */
};
\end{lstlisting}
\begin{description}
\item[\field{width, height}] represents a range of resolutions
  supported by the device. If its \field{step} is not applicable, its
  \field{min} is equal to its \field{max}.
\item[\field{num_rates}] is the number of
  \field{virtio_video_format_ranges} that follows to represent
  supported frame rates.
\end{description}
\end{description}

\devicenormative{\subparagraph}{Device Operation: Query device
  capability}{Device Types / Video Device / Device Operation / Device
  Operation: Query device capability}

The total size of the device response MUST not exceed
\field{max_cap_length} in bytes reported in the device configuration.

\paragraph{Device Operation: Create streams}

To process buffers, the device needs to associate them with a certain
video stream (essentially, a context). Streams are created by
VIRTIO_VIDEO_CMD_STREAM_CREATE with a default set of parameters
determined by the device.

\begin{description}
\item[VIRTIO_VIDEO_CMD_STREAM_CREATE] Create a video stream (context)
  within the device.

\begin{lstlisting}
enum virtio_video_mem_type {
        VIRTIO_VIDEO_MEM_TYPE_GUEST_PAGES,
        VIRTIO_VIDEO_MEM_TYPE_VIRTIO_OBJECT,
};

struct virtio_video_stream_create {
        struct virtio_video_cmd_hdr hdr;
        le32 in_mem_type; /* One of VIRTIO_VIDEO_MEM_TYPE_* types */
        le32 out_mem_type; /* One of VIRTIO_VIDEO_MEM_TYPE_* types */
        le32 coded_format; /* One of VIRTIO_VIDEO_FORMAT_* types */
        u8 padding[4];
        u8 tag[64];
};
\end{lstlisting}

\begin{description}
\item[\field{in_mem_type, out_mem_type}] is a type of buffer
  management for input/output buffers. The driver MUST set a value in
  \field{enum virtio_video_mem_type} that the device reported a
  corresponding feature bit.
\begin{description}
\item[\field{VIRTIO_VIDEO_MEM_TYPE_GUEST_PAGES}] Use guest pages. The
  driver MUST not set this value if a feature bit
  VIRTIO_VIDEO_F_RESOURCE_GUEST_PAGES is not set.
\item[\field{VIRTIO_VIDEO_MEM_TYPE_VIRTIO_OBJECT}] Use object exported
  by another virtio device. The driver MUST not set this value if a
  feature bit VIRTIO_VIDEO_F_RESOURCE_VIRTIO_OBJECT is not set.
\end{description}
\item[\field{coded_format}] is the encoded format that will be
  processed.
\item[\field{tag}] is the name associated with this stream. The tag
  MUST be encoded in UTF-8 and NUL-terminated.
\end{description}

The driver MUST set \field{stream_id} in \field{virtio_video_cmd_hdr}
to an integer that is not used before. If a used value is passed as
\field{stream_id}, the device MUST reports an error with
VIRTIO_VIDEO_RESP_ERR_INVALID_STREAM_ID.

\item[VIRTIO_VIDEO_CMD_STREAM_DESTROY] Destroy a video stream
  (context) within the device.

\begin{lstlisting}
struct virtio_video_stream_destroy {
        struct virtio_video_cmd_hdr hdr;
};
\end{lstlisting}
\end{description}

\paragraph{Device Operation: Import resources}

\begin{itemize*}
\item Use VIRTIO_VIDEO_CMD_RESOURCE_CREATE to import a resource to use
  it as a video buffer.
\item Use VIRTIO_VIDEO_CMD_RESOURCE_DESTROY_ALL to invalidate all the
  resources imported by VIRTIO_VIDEO_CMD_RESOURCE_CREATE so far.
\end{itemize*}

\begin{description}
\item[VIRTIO_VIDEO_CMD_RESOURCE_CREATE] Create a resource descriptor
  within the device.

The driver sends \field{virtio_video_resource_create} defined as
follows:
\begin{lstlisting}
#define VIRTIO_VIDEO_MAX_PLANES 8

struct virtio_video_resource_create {
        struct virtio_video_cmd_hdr hdr;
        le32 queue_type; /* One of VIRTIO_VIDEO_QUEUE_TYPE_* types */
        le32 resource_id;
        le32 planes_layout;
        le32 num_planes;
        le32 plane_offsets[VIRTIO_VIDEO_MAX_PLANES];
        le32 num_entries[VIRTIO_VIDEO_MAX_PLANES];
        /*
         * Followed by either
         * - struct virtio_video_mem_entry entries[]
         *   for VIRTIO_VIDEO_MEM_TYPE_GUEST_PAGES
         * - struct virtio_video_object_entry entries[]
         *   for VIRTIO_VIDEO_MEM_TYPE_VIRTIO_OBJECT
         */
};
\end{lstlisting}
\begin{description}
\item[\field{queue_type}] specifies a direction of queue which the
  resource will be used for.
\item[\field{resource_id}] is an identifier of the resource. The
  driver MUST set it to an integer that is not used before for the
  given \field{queue_type}. If the specified \field{resource_id} is
  already in use, the device MUST report an error with
  VIRTIO_VIDEO_RESP_ERR_INVALID_RESOURCE_ID.
\item[\field{planes_layout}] specifies a plane layout. The driver MUST
  set only one bit of \field{virtio_video_planes_layout_flag} that is
  supported for a current raw format.
\item[\field{num_planes}] specifies a number of planes.
\item[\field{plane_offsets}] specifies offsets for each plane.
\item[\field{num_entries}] is an array of numbers of \field{entries}
  memory entries for each plane. If \field{planes_layout} is
  VIRTIO_VIDEO_PLANES_LAYOUT_SINGLE_BUFFER, only the first element is
  used.
\end{description}

The data following \field{virtio_video_resource_create} depend on a
type of \field{virtio_video_mem_type} specified in
\field{virtio_video_stream_create}.
\begin{itemize*}
\item When \field{VIRTIO_VIDEO_MEM_TYPE_GUEST_PAGES} was specified,
  \field{virtio_video_resource_create} is followed by an array of
  \field{virtio_video_mem_entry} defined as follows:
\begin{lstlisting}
struct virtio_video_mem_entry {
        le64 addr;
        le32 length;
        u8 padding[4];
};
\end{lstlisting}
\begin{description}
\item[\field{addr}] is the physical guest address.
\item[\field{length}] is the length of the resource.
\end{description}
The number of \field{virtio_video_mem_entry} MUST be equal to the sum
of integers in the array \field{num_entries}.

\item When \field{VIRTIO_VIDEO_MEM_TYPE_VIRTIO_OBJECT} was specified,
  \field{virtio_video_resource_create} is followed by an array of
  \field{virtio_video_object_entry} defined as follows:

\begin{lstlisting}
struct virtio_video_object_entry {
        le64 uuid_low;
        le64 uuid_high;
};
\end{lstlisting}
\begin{description}
\item[\field{uuid_low, uuid_high}] is a UUID of the resource.
\end{description}
If \field{planes_layout} is VIRTIO_VIDEO_PLANES_LAYOUT_SINGLE_BUFFER,
the number of \field{virtio_video_object_entry} MUST be one. If it is
VIRTIO_VIDEO_PLANES_LAYOUT_PER_PLANE, the number MUST be equal to
\field{num_planes}.
\end{itemize*}

\item[VIRTIO_VIDEO_CMD_RESOURCE_DESTROY_ALL] Invalidate all the
  resource descriptor created so far.
\begin{lstlisting}
struct virtio_video_resource_destroy_all {
        struct virtio_video_cmd_hdr hdr;
        le32 queue_type; /* One of VIRTIO_VIDEO_QUEUE_TYPE_* types */
        u8 padding[4];
};
\end{lstlisting}
\begin{description}
\item[\field{queue_type}] is a type of queue.
\end{description}
\end{description}

\paragraph{Device Operation: Process buffers}

\begin{itemize*}
\item Use VIRTIO_VIDEO_CMD_RESOURCE_QUEUE to queue the resource for
  processing in the device. The request completes asynchronously and
  out-of-order when the device has finished with the buffer.
\item Use VIRTIO_VIDEO_CMD_STREAM_DRAIN to ask the device to process
  and return all of the already queued buffers.
\item Use VIRTIO_VIDEO_CMD_QUEUE_CLEAR to ask the device to return
  back already queued buffers from the input or the output queue. This
  also includes input or output buffers that can be currently owned by
  the device's processing pipeline.
\end{itemize*}

\begin{description}
\item[VIRTIO_VIDEO_CMD_RESOURCE_QUEUE] Add a buffer to the device's
queue.

\begin{lstlisting}
struct virtio_video_resource_queue {
        struct virtio_video_cmd_hdr hdr;
        le32 queue_type;
        le32 resource_id;
        le64 timestamp;
        le32 num_data_sizes;
        le32 data_sizes[VIRTIO_VIDEO_MAX_PLANES];
        u8 padding[4];
};
\end{lstlisting}

\begin{description}
\item[\field{resource_id}] is the ID of the resource to be queued.
\item[\field{timestamp}] is an abstract sequence counter that can be
  used for synchronization. For an input buffer, the driver MUST set a
  \field{timestamp}.
\item[\field{num_data_sizes}] is the number of \field{data_sizes}
  entries in use.
\item[\field{data_sizes}] number of data bytes within a plane.
\end{description}

The device returns \field{virtio_video_resource_queue_resp} defined as
follows:
\begin{lstlisting}
enum virtio_video_buffer_flag {
        VIRTIO_VIDEO_BUFFER_FLAG_ERR = 0x0001,
        VIRTIO_VIDEO_BUFFER_FLAG_EOS = 0x0002,

        /* Encoder only */
        VIRTIO_VIDEO_BUFFER_FLAG_IFRAME = 0x0004,
        VIRTIO_VIDEO_BUFFER_FLAG_PFRAME = 0x0008,
        VIRTIO_VIDEO_BUFFER_FLAG_BFRAME = 0x0010,
};

struct virtio_video_resource_queue_resp {
        struct virtio_video_cmd_hdr hdr;
        le64 timestamp;
        le32 flags;
        le32 size; /* Encoded size */
};
\end{lstlisting}
\begin{description}
\item[\field{timestamp}] is an abstract sequence counter that can be
  used for synchronization. For an output buffer, the device MUST copy
  the \field{timestamp} of the input buffer this output buffer was
  produced from.
\item[\field{flags}] marks specific buffers in the sequence with
  VIRTIO_VIDEO_BUFFER_FLAG_* flags.
\item[\field{size}] is the data size in the buffer (encoder only).
\end{description}

\begin{itemize*}
\item For each VIRTIO_VIDEO_CMD_RESOURCE_QUEUE request, the device
  MUST send a response to the queue request with
  VIRTIO_VIDEO_OK_NODATA when it has finished processing the buffer
  successfully.
\item The device MUST mark a buffer that triggered a processing error
  with the VIRTIO_VIDEO_BUFFER_F_ERR flag.
\item The device MUST mark the last buffer with the
  VIRTIO_VIDEO_BUFFER_F_EOS flag to denote completion of the drain
  sequence.
\item In case of encoder, to denote a particular frame type the device
  MUST mark the respective buffer with VIRTIO_VIDEO_BUFFER_IFRAME,
  VIRTIO_VIDEO_BUFFER_PFRAME, or VIRTIO_VIDEO_BUFFER_BFRAME.
\item If the processing was stopped due to
  VIRTIO_VIDEO_CMD_QUEUE_CLEAR, the device MUST respond with
  VIRTIO_VIDEO_RESP_OK_NODATA as a response type and
  VIRTIO_VIDEO_BUFFER_FLAG_ERR in \field{flags}.
\item The driver and device MUST follow requirements about buffer
  ownership explained in \ref{sec:Device Types / Video Device / Device
    Operation / Buffer lifecycle}.
\end{itemize*}

\item[VIRTIO_VIDEO_CMD_STREAM_DRAIN] Ask the device to push all the
  queued buffers through the pipeline.

\begin{lstlisting}
struct virtio_video_stream_drain {
        struct virtio_video_cmd_hdr hdr;
};
\end{lstlisting}

\begin{itemize*}
\item While the device is processing a VIRTIO_VIDEO_CMD_STREAM_DRAIN
  command, the device MUST return
  VIRTIO_VIDEO_RESP_ERR_INVALID_OPERATION for incoming commands of
  VIRTIO_VIDEO_CMD_RESOURCE_QUEUE for input buffers and
  VIRTIO_VIDEO_CMD_STREAM_DRAIN.
\item If the processing was stopped due to
  VIRTIO_VIDEO_CMD_QUEUE_CLEAR, the device MUST respond with
  VIRTIO_VIDEO_RESP_OK_NODATA as a response type and
  VIRTIO_VIDEO_BUFFER_FLAG_ERR in \field{flags}.
\end{itemize*}

\item[VIRTIO_VIDEO_CMD_QUEUE_CLEAR] Return already queued
  buffers back from the input or the output queue of the device. The
  device SHOULD return all of the buffers from the respective queue as
  soon as possible without pushing the buffers through the processing
  pipeline.

\begin{lstlisting}
struct virtio_video_resource_queue_clear {
        struct virtio_video_cmd_hdr hdr;
        le32 queue_type; /* One of VIRTIO_VIDEO_QUEUE_TYPE_* types */
        u8 padding[4];
};
\end{lstlisting}
\begin{description}
\item[\field{queue_type}] queue type.
\end{description}

While the device is processing a VIRTIO_VIDEO_CMD_QUEUE_CLEAR command,
the device MUST return VIRTIO_VIDEO_RESP_ERR_INVALID_OPERATION for
incoming commands of VIRTIO_VIDEO_CMD_RESOURCE_QUEUE,
VIRTIO_VIDEO_CMD_STREAM_DRAIN, and VIRTIO_VIDEO_CMD_QUEUE_CLEAR.

\end{description}

\paragraph{Device Operation: Handle stream parameters}

\begin{itemize*}
\item Use VIRTIO_VIDEO_CMD_GET_PARAMS to get the current stream
  parameters for input and output streams from the device.
\item Use VIRTIO_VIDEO_CMD_SET_PARAMS to provide new stream parameters
  to the device.
\item After setting stream parameters, the driver may issue
  VIRTIO_VIDEO_CMD_GET_PARAMS as some parameters of both input and
  output can be changed implicitly by the device during the set
  operation.
\end{itemize*}

\begin{description}
\item[VIRTIO_VIDEO_CMD_GET_PARAMS] Get parameters of the input or the
  output of a stream.

\begin{lstlisting}
struct virtio_video_plane_format {
        le32 plane_size;
        le32 stride;
};
\end{lstlisting}

\begin{description}
\item[\field{plane_size}] size of the plane in bytes.
\item[\field{stride}] stride used for the plane in bytes.
\end{description}

\begin{lstlisting}
struct virtio_video_crop {
        le32 left;
        le32 top;
        le32 width;
        le32 height;
};

struct virtio_video_params {
        le32 queue_type; /* One of VIRTIO_VIDEO_QUEUE_TYPE_* types */
        le32 format; /* One of VIRTIO_VIDEO_FORMAT_* types */
        le32 frame_width;
        le32 frame_height;
        le32 min_buffers;
        le32 max_buffers;
        struct virtio_video_crop crop;
        le32 frame_rate;
        le32 num_planes;
        struct virtio_video_plane_format plane_formats[VIRTIO_VIDEO_MAX_PLANES];
};
\end{lstlisting}

\begin{description}
\item[\field{frame_width}] the value to get/set.
\item[\field{frame_height}] the value to get/set.
\item[\field{pixel_format}] the value to get/set.
\item[\field{min_buffers}] minimum buffers required to handle the
  format (r/o).
\item[\field{max_buffers}] maximum buffers required to handle the
  format (r/o).
\item[\field{crop}] cropping (composing) rectangle.
\item[\field{frame_rate}] the value to get/set.
\item[\field{num_planes}] number of planes used to store pixel data
(r/o).
\item[\field{plane_formats}] description of each plane.
\end{description}

\begin{lstlisting}
struct virtio_video_get_params {
        struct virtio_video_cmd_hdr hdr;
        le32 queue_type; /* One of VIRTIO_VIDEO_QUEUE_TYPE_* types */
        u8 padding[4];
};

struct virtio_video_get_params_resp {
        struct virtio_video_cmd_hdr hdr;
        struct virtio_video_params params;
};
\end{lstlisting}

\begin{description}
\item[\field{queue_type}] queue type.
\item[\field{params}] parameter values.
\end{description}

\item[VIRTIO_VIDEO_CMD_SET_PARAMS] Change parameters of a stream.

\begin{lstlisting}
struct virtio_video_set_params {
        struct virtio_video_cmd_hdr hdr;
        struct virtio_video_params params;
};
\end{lstlisting}

\begin{description}
\item[\field{params}] parameters to set.
\end{description}

Setting stream parameters might have side effects within the device.
For example, the device MAY perform alignment of width and height,
change the number of planes it uses for the format, or do whatever
changes that are required to continue normal operation using the
updated parameters. It is up to the driver to check the parameter set
after the VIRTIO_VIDEO_CMD_SET_PARAMS request has been issued.
\end{description}

\paragraph{Device Operation: Handle control values}

The driver can query, get and set control values. Though control
values are associated with a video stream like stream parameters,
supported values differ depending on the device capabilities and
formats of videos being processed.

\subparagraph{Commands for controls}

\begin{description}
\item[VIRTIO_VIDEO_CMD_QUERY_CONTROL] Query supported control values.

\begin{lstlisting}
struct virtio_video_query_control {
        struct virtio_video_cmd_hdr hdr;
        le32 control; /* One of VIRTIO_VIDEO_CONTROL_* types */
        u8 padding[4];
};

struct virtio_video_query_control_resp {
        struct virtio_video_cmd_hdr hdr;
        /* Followed by one of struct virtio_video_query_control_resp_* */
};
\end{lstlisting}

If the device supports a given control, the device MUST return a
struct that indicates supported control values after sending
\field{virtio_video_query_control_resp}. The struct used as the
response differs depending on the value of requested \field{control}.
Otherwise, the device MUST report an error with
VIRTIO_VIDEO_RESP_UNSUPPORTED_CONTROL.

\item[VIRTIO_VIDEO_CMD_GET_CONTROL] Get a control value.

\begin{lstlisting}
struct virtio_video_get_control {
        struct virtio_video_cmd_hdr hdr;
        le32 control; /* One of VIRTIO_VIDEO_CONTROL_* types */
        u8 padding[4];
};

struct virtio_video_get_control_resp {
        struct virtio_video_cmd_hdr hdr;
        /* Followed by one of struct virtio_video_control_val_* */
};
\end{lstlisting}

If the device supports a given control and a control value is
available, the device MUST return a control value after
\field{virtio_video_get_control_resp}.
The struct used as the response differs depending on the value of
requested as \field{control}. If the given control is unsupported, the
device MUST report an error with
VIRTIO_VIDEO_RESP_UNSUPPORTED_CONTROL.

\item[VIRTIO_VIDEO_CMD_SET_CONTROL] Set a control value.

\begin{lstlisting}
struct virtio_video_set_control {
        struct virtio_video_cmd_hdr hdr;
        le32 control; /* One of VIRTIO_VIDEO_CONTROL_* types */
        u8 padding[4];
        /* Followed by one of struct virtio_video_control_val_* */
};

struct virtio_video_set_control_resp {
        struct virtio_video_cmd_hdr hdr;
};
\end{lstlisting}

The driver MUST set \field{control} in
\field{virtio_video_set_control} and send a corresponding struct after
it. If the given control is unsupported, the device MUST report an
error with VIRTIO_VIDEO_RESP_UNSUPPORTED_CONTROL.
\end{description}

\subparagraph{Types of controls}

The types of controls are defined as follows:
\begin{lstlisting}
enum virtio_video_control_type {
        VIRTIO_VIDEO_CONTROL_BITRATE = 1,
        VIRTIO_VIDEO_CONTROL_PROFILE,
        VIRTIO_VIDEO_CONTROL_LEVEL,
};
\end{lstlisting}

\begin{description}
\item[VIRTIO_VIDEO_CONTROL_BITRATE] Bitrate of video. (Only for the
  encoder device.)

The driver can VIRTIO_VIDEO_CMD_GET_CONTROL and
VIRTIO_VIDEO_CMD_SET_CONTROL to get and set a bitrate for encoding
respectively.
The following \field{virtio_video_control_val_bitrate} is used when
the device returns a value as a response of
VIRTIO_VIDEO_CMD_GET_CONTROL and when the driver specifies a value to
be set by VIRTIO_VIDEO_CMD_SET_CONTROL.
\begin{lstlisting}
struct virtio_video_control_val_bitrate {
        le32 bitrate;
        u8 padding[4];
};
\end{lstlisting}

\item[VIRTIO_VIDEO_CONTROL_PROFILE] Profile of a compressed video
  stream.

VIRTIO_VIDEO_CONTROL_PROFILE is used to handle profiles, which
represent sets of cabalities for several compressed formats such as
H.264 and VP9. The enum representing profiles are defined as follows:
\begin{lstlisting}
enum virtio_video_profile {
        /* H.264 */
        VIRTIO_VIDEO_PROFILE_H264_MIN = 0x100,
        VIRTIO_VIDEO_PROFILE_H264_BASELINE = VIRTIO_VIDEO_PROFILE_H264_MIN,
        VIRTIO_VIDEO_PROFILE_H264_MAIN,
        VIRTIO_VIDEO_PROFILE_H264_EXTENDED,
        VIRTIO_VIDEO_PROFILE_H264_HIGH,
        VIRTIO_VIDEO_PROFILE_H264_HIGH10PROFILE,
        VIRTIO_VIDEO_PROFILE_H264_HIGH422PROFILE,
        VIRTIO_VIDEO_PROFILE_H264_HIGH444PREDICTIVEPROFILE,
        VIRTIO_VIDEO_PROFILE_H264_SCALABLEBASELINE,
        VIRTIO_VIDEO_PROFILE_H264_SCALABLEHIGH,
        VIRTIO_VIDEO_PROFILE_H264_STEREOHIGH,
        VIRTIO_VIDEO_PROFILE_H264_MULTIVIEWHIGH,
        VIRTIO_VIDEO_PROFILE_H264_MAX = VIRTIO_VIDEO_PROFILE_H264_MULTIVIEWHIGH,

        /* HEVC */
        VIRTIO_VIDEO_PROFILE_HEVC_MIN = 0x200,
        VIRTIO_VIDEO_PROFILE_HEVC_MAIN = VIRTIO_VIDEO_PROFILE_HEVC_MIN,
        VIRTIO_VIDEO_PROFILE_HEVC_MAIN10,
        VIRTIO_VIDEO_PROFILE_HEVC_MAIN_STILL_PICTURE,
        VIRTIO_VIDEO_PROFILE_HEVC_MAX =
                VIRTIO_VIDEO_PROFILE_HEVC_MAIN_STILL_PICTURE,

        /* VP8 */
        VIRTIO_VIDEO_PROFILE_VP8_MIN = 0x300,
        VIRTIO_VIDEO_PROFILE_VP8_PROFILE0 = VIRTIO_VIDEO_PROFILE_VP8_MIN,
        VIRTIO_VIDEO_PROFILE_VP8_PROFILE1,
        VIRTIO_VIDEO_PROFILE_VP8_PROFILE2,
        VIRTIO_VIDEO_PROFILE_VP8_PROFILE3,
        VIRTIO_VIDEO_PROFILE_VP8_MAX = VIRTIO_VIDEO_PROFILE_VP8_PROFILE3,

        /* VP9 */
        VIRTIO_VIDEO_PROFILE_VP9_MIN = 0x400,
        VIRTIO_VIDEO_PROFILE_VP9_PROFILE0 = VIRTIO_VIDEO_PROFILE_VP9_MIN,
        VIRTIO_VIDEO_PROFILE_VP9_PROFILE1,
        VIRTIO_VIDEO_PROFILE_VP9_PROFILE2,
        VIRTIO_VIDEO_PROFILE_VP9_PROFILE3,
        VIRTIO_VIDEO_PROFILE_VP9_MAX = VIRTIO_VIDEO_PROFILE_VP9_PROFILE3,
};
\end{lstlisting}

The driver can query supported profiles for a compressed format by
VIRTIO_VIDEO_CMD_QUERY_CONTROL command with
\field{virtio_video_query_control_profile}. If the device supports the
given format and the format's profiles are listed in \field{enum
  virtio_video_profile}, the device MUST returns a list of supported
profiles as a response
\field{virtio_video_query_control_resp_profile}.
\begin{lstlisting}
struct virtio_video_query_control_profile {
        le32 format; /* One of VIRTIO_VIDEO_FORMAT_* */
        u8 padding[4];
};

struct virtio_video_query_control_resp_profile {
        le32 num;
        u8 padding[4];
        /* Followed by an array le32 profiles[] */
};
\end{lstlisting}

The device and the driver use \field{virtio_video_control_val_profile}
for VIRTIO_VIDEO_CMD_GET_CONTROL and VIRTIO_VIDEO_CMD_SET_CONTROL.
\begin{lstlisting}
struct virtio_video_control_val_profile {
        le32 profile;
        u8 padding[4];
};
\end{lstlisting}

\item[VIRTIO_VIDEO_CONTROL_LEVEL] Level of a compressed video stream.

VIRTIO_VIDEO_CONTROL_LEVEL is used to handle levels of video stream.
Levels are defined for some compressed video formats to specify sets
of constraints that indicate a degree of required decoder performance.
The enum representing levels are defined as follows:
\begin{lstlisting}
enum virtio_video_level {
        /* H.264 */
        VIRTIO_VIDEO_LEVEL_H264_MIN = 0x100,
        VIRTIO_VIDEO_LEVEL_H264_1_0 = VIRTIO_VIDEO_LEVEL_H264_MIN,
        VIRTIO_VIDEO_LEVEL_H264_1_1,
        VIRTIO_VIDEO_LEVEL_H264_1_2,
        VIRTIO_VIDEO_LEVEL_H264_1_3,
        VIRTIO_VIDEO_LEVEL_H264_2_0,
        VIRTIO_VIDEO_LEVEL_H264_2_1,
        VIRTIO_VIDEO_LEVEL_H264_2_2,
        VIRTIO_VIDEO_LEVEL_H264_3_0,
        VIRTIO_VIDEO_LEVEL_H264_3_1,
        VIRTIO_VIDEO_LEVEL_H264_3_2,
        VIRTIO_VIDEO_LEVEL_H264_4_0,
        VIRTIO_VIDEO_LEVEL_H264_4_1,
        VIRTIO_VIDEO_LEVEL_H264_4_2,
        VIRTIO_VIDEO_LEVEL_H264_5_0,
        VIRTIO_VIDEO_LEVEL_H264_5_1,
        VIRTIO_VIDEO_LEVEL_H264_MAX = VIRTIO_VIDEO_LEVEL_H264_5_1,
};
\end{lstlisting}

The driver can query supported levels for a compressed format by
VIRTIO_VIDEO_CMD_QUERY_CONTROL command with
\field{virtio_video_query_control_level}. If the device supports the
given format and the format's levels are listed in \field{enum
  virtio_video_level}, the device MUST return a list of supported
levels as a response \field{virtio_video_query_control_resp_level}.
\begin{lstlisting}
struct virtio_video_query_control_level {
        le32 profile; /* One of VIRTIO_VIDEO_PROFILE_* */
        u8 padding[4];
};

struct virtio_video_query_control_resp_level {
        le32 num;
        u8 padding[4];
        /* Followed by an array le32 level[] */
};
\end{lstlisting}

The device and the driver use \field{virtio_video_control_val_level}
for VIRTIO_VIDEO_CMD_GET_CONTROL and VIRTIO_VIDEO_CMD_SET_CONTROL.
\begin{lstlisting}
struct virtio_video_control_val_level {
        le32 level;
        u8 padding[4];
};
\end{lstlisting}
\end{description}

\subsubsection{Buffer lifecycle}
\label{sec:Device Types / Video Device / Device Operation / Buffer
  lifecycle}

The state machine in Figure~\ref{fig:buffer-lifecycle} shows the life
cycle of a video buffer.

\begin{figure}[h]
  \centering
  \includegraphics[width=\textwidth]{images/generated/video-buffer-lifecycle.png}
  \caption{Lifecycle of a buffer}
  \label{fig:buffer-lifecycle}
\end{figure}

\drivernormative{\subparagraph}{Buffer lifecycle}{Device Types / Video
  Device / Device Operation / Device Operation: Buffer lifecycle}

The following table shows whether the driver can read or write each
buffer in each state in Figure~\ref{fig:buffer-lifecycle}. The driver
MUST not read or write buffers in the state that doesn't permit.
\begin{center}
  \begin{tabular}{|c|c|c|}
    \hline
    State & Input buffers & Output buffers \\
    \hline
    Created  & Read / Write & Read \\
    Queued   & -            & -    \\
    Dequeued & Read / Write & Read \\
    \hline
  \end{tabular}
\end{center}

\devicenormative{\subparagraph}{Buffer lifecycle}{Device Types / Video
  Device / Device Operation / Device Operation: Buffer lifecycle}

The following table shows whether the device can read or write each
buffer in each state in Figure~\ref{fig:buffer-lifecycle}. The device
MUST not read or write buffers in the state that doesn't permit.
\begin{center}
  \begin{tabular}{ |c|c|c| }
    \hline
    State & Input buffers & Output buffers \\
    \hline
    Created  & -    & - \\
    Queued   & Read & Read / Write \\
    Dequeued & -    & Read \\
    \hline
  \end{tabular}
\end{center}

\subsubsection{Event Virtqueue}

While processing buffers, the device can send asynchronous event
notifications to the driver. The behaviour depends on the exact
stream. For example, the decoder device sends a resolution change
event when it encounters new resolution metadata in the stream.

The device can report events on the event queue. The driver initially
populates the queue with device-writeable buffers. When the device
needs to report an event, it fills a buffer and notifies the driver.
The driver consumes the report and adds a new buffer to the virtqueue.

\begin{lstlisting}
enum virtio_video_event_type {
        /* For all devices */
        VIRTIO_VIDEO_EVENT_ERROR = 0x0100,

        /* For decoder only */
        VIRTIO_VIDEO_EVENT_DECODER_RESOLUTION_CHANGED = 0x0200,
};

struct virtio_video_event {
        le32 event_type; /* One of VIRTIO_VIDEO_EVENT_* types */
        le32 stream_id;
};
\end{lstlisting}

\begin{description}
\item[\field{event_type}] type of the triggered event .
\item[\field{stream_id}] id of the source stream.
\end{description}

The device MUST send VIRTIO_VIDEO_EVENT_DECODER_RESOLUTION_CHANGED
whenever it encounters new resolution data in the stream. This
includes the case of the initial device configuration after metadata
has been parsed and the case of dynamic resolution change.