Internet DRAFT - draft-wang-avt-rfc3984bis
draft-wang-avt-rfc3984bis
Audio/Video Transport WG Y.-K. Wang
Internet Draft S. Wenger
Intended status: Standards track M.M. Hanuksela
Expires: January 2009 Nokia
T. Stockhammer
Nomor Research
M. Westerlund
Ericsson
D. Singer
Apple
July 14, 2008
RTP Payload Format for H.264 Video
draft-wang-avt-rfc3984bis-01.txt
Status of this Memo
By submitting this Internet-Draft, each author represents that any
applicable patent or other IPR claims of which he or she is aware
have been or will be disclosed, and any of which he or she becomes
aware will be disclosed, in accordance with Section 6 of BCP 79.
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF), its areas, and its working groups. Note that
other groups may also distribute working documents as Internet-
Drafts.
Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress."
The list of current Internet-Drafts can be accessed at
http://www.ietf.org/ietf/1id-abstracts.txt
The list of Internet-Draft Shadow Directories can be accessed at
http://www.ietf.org/shadow.html
This Internet-Draft will expire on January 14, 2009.
Copyright Notice
Copyright (C) The IETF Trust (2008).
Wang, et al Expires January 14, 2009 [Page 1]
Internet-Draft RFC3984bis July 2008
Abstract
This memo describes an RTP Payload format for the ITU-T
Recommendation H.264 video codec and the technically identical
ISO/IEC International Standard 14496-10 video codec. The RTP payload
format allows for packetization of one or more Network Abstraction
Layer Units (NALUs), produced by an H.264 video encoder, in each RTP
payload. The payload format has wide applicability, as it supports
applications from simple low bit-rate conversational usage, to
Internet video streaming with interleaved transmission, to high bit-
rate video-on-demand.
This memo intends to obsolete RFC 3984.
Table of Contents
1. Introduction...................................................4
1.1. The H.264 Codec...........................................4
1.2. Parameter Set Concept.....................................5
1.3. Network Abstraction Layer Unit Types......................6
2. Conventions....................................................7
3. Scope..........................................................7
4. Definitions and Abbreviations..................................7
4.1. Definitions...............................................7
4.2. Abbreviations.............................................9
5. RTP Payload Format.............................................9
5.1. RTP Header Usage..........................................9
5.2. Common Structure of the RTP Payload Format...............12
5.3. NAL Unit Octet Usage.....................................13
5.4. Packetization Modes......................................15
5.5. Decoding Order Number (DON)..............................16
5.6. Single NAL Unit Packet...................................19
5.7. Aggregation Packets......................................20
5.7.1. Single-Time Aggregation Packet......................22
5.7.2. Multi-Time Aggregation Packets (MTAPs)..............24
5.7.3. Fragmentation Units (FUs)...........................28
6. Packetization Rules...........................................32
6.1. Common Packetization Rules...............................32
6.2. Single NAL Unit Mode.....................................33
6.3. Non-Interleaved Mode.....................................33
6.4. Interleaved Mode.........................................33
7. De-Packetization Process......................................34
7.1. Single NAL Unit and Non-Interleaved Mode.................34
7.2. Interleaved Mode.........................................34
Wang, et al Expires January 14, 2009 [Page 2]
Internet-Draft RFC3984bis July 2008
7.2.1. Size of the Deinterleaving Buffer...................35
7.2.2. Deinterleaving Process..............................35
7.3. Additional De-Packetization Guidelines...................37
8. Payload Format Parameters.....................................38
8.1. MIME Registration........................................38
8.2. SDP Parameters...........................................50
8.2.1. Mapping of MIME Parameters to SDP...................50
8.2.2. Usage with the SDP Offer/Answer Model...............50
8.2.3. Usage in Declarative Session Descriptions...........55
8.3. Examples.................................................56
8.4. Parameter Set Considerations.............................61
9. Security Considerations.......................................63
10. Congestion Control...........................................64
11. IANA Consideration...........................................65
12. Informative Appendix: Application Examples...................65
12.1. Video Telephony according to ITU-T Recommendation H.241
Annex A.......................................................65
12.2. Video Telephony, No Slice Data Partitioning, No NAL Unit
Aggregation...................................................65
12.3. Video Telephony, Interleaved Packetization Using NAL Unit
Aggregation...................................................66
12.4. Video Telephony with Data Partitioning..................66
12.5. Video Telephony or Streaming with FUs and Forward Error
Correction....................................................67
12.6. Low Bit-Rate Streaming..................................69
12.7. Robust Packet Scheduling in Video Streaming.............70
13. Informative Appendix: Rationale for Decoding Order Number....71
13.1. Introduction............................................71
13.2. Example of Multi-Picture Slice Interleaving.............71
13.3. Example of Robust Packet Scheduling.....................73
13.4. Robust Transmission Scheduling of Redundant Coded Slices77
13.5. Remarks on Other Design Possibilities...................77
14. Acknowledgements.............................................78
15. References...................................................78
15.1. Normative References....................................78
15.2. Informative References..................................79
Authors' Addresses...............................................80
Intellectual Property Statement..................................82
Disclaimer of Validity...........................................82
Acknowledgement..................................................83
16. Backward compatibility to RFC 3984...........................83
17. Changes from RFC 3984........................................83
17.1. Technical changes.......................................83
17.2. Editorial changes.......................................86
18. Open issues..................................................97
Wang, et al Expires January 14, 2009 [Page 3]
Internet-Draft RFC3984bis July 2008
1. Introduction
This memo intends to obsolete RFC 3984.
1.1. The H.264 Codec
This memo specifies an RTP payload specification for the video coding
standard known as ITU-T Recommendation H.264 [1] and ISO/IEC
International Standard 14496 Part 10 [2] (both also known as Advanced
Video Coding, or AVC). Recommendation H.264 was approved by ITU-T on
May 2003, and the approved draft specification is available for
public review [8]. In this memo the H.264 acronym is used for the
codec and the standard, but the memo is equally applicable to the
ISO/IEC counterpart of the coding standard.
The H.264 video codec has a very broad application range that covers
all forms of digital compressed video from, low bit-rate Internet
streaming applications to HDTV broadcast and Digital Cinema
applications with nearly lossless coding. Compared to the current
state of technology, the overall performance of H.264 is such that
bit rate savings of 50% or more are reported. Digital Satellite TV
quality, for example, was reported to be achievable at 1.5 Mbit/s,
compared to the current operation point of MPEG 2 video at around 3.5
Mbit/s [9].
The codec specification [1] itself distinguishes conceptually between
a video coding layer (VCL) and a network abstraction layer (NAL).
The VCL contains the signal processing functionality of the codec;
mechanisms such as transform, quantization, and motion compensated
prediction; and a loop filter. It follows the general concept of
most of today's video codecs, a macroblock-based coder that uses
inter picture prediction with motion compensation and transform
coding of the residual signal. The VCL encoder outputs slices: a bit
string that contains the macroblock data of an integer number of
macroblocks, and the information of the slice header (containing the
spatial address of the first macroblock in the slice, the initial
quantization parameter, and similar information). Macroblocks in
slices are arranged in scan order unless a different macroblock
allocation is specified, by using the so-called Flexible Macroblock
Ordering syntax. In-picture prediction is used only within a slice.
More information is provided in [9].
The Network Abstraction Layer (NAL) encoder encapsulates the slice
output of the VCL encoder into Network Abstraction Layer Units (NAL
units), which are suitable for transmission over packet networks or
use in packet oriented multiplex environments. Annex B of H.264
defines an encapsulation process to transmit such NAL units over
Wang, et al Expires January 14, 2009 [Page 4]
Internet-Draft RFC3984bis July 2008
byte-stream oriented networks. In the scope of this memo, Annex B is
not relevant.
Internally, the NAL uses NAL units. A NAL unit consists of a one-
byte header and the payload byte string. The header indicates the
type of the NAL unit, the (potential) presence of bit errors or
syntax violations in the NAL unit payload, and information regarding
the relative importance of the NAL unit for the decoding process.
This RTP payload specification is designed to be unaware of the bit
string in the NAL unit payload.
One of the main properties of H.264 is the complete decoupling of the
transmission time, the decoding time, and the sampling or
presentation time of slices and pictures. The decoding process
specified in H.264 is unaware of time, and the H.264 syntax does not
carry information such as the number of skipped frames (as is common
in the form of the Temporal Reference in earlier video compression
standards). Also, there are NAL units that affect many pictures and
that are, therefore, inherently timeless. For this reason, the
handling of the RTP timestamp requires some special considerations
for NAL units for which the sampling or presentation time is not
defined or, at transmission time, unknown.
1.2. Parameter Set Concept
One very fundamental design concept of H.264 is to generate self-
contained packets, to make mechanisms such as the header duplication
of RFC 2429 [10] or MPEG-4's Header Extension Code (HEC) [11]
unnecessary. This was achieved by decoupling information relevant to
more than one slice from the media stream. This higher layer meta
information should be sent reliably, asynchronously, and in advance
from the RTP packet stream that contains the slice packets.
(Provisions for sending this information in-band are also available
for applications that do not have an out-of-band transport channel
appropriate for the purpose.) The combination of the higher-level
parameters is called a parameter set. The H.264 specification
includes two types of parameter sets: sequence parameter set and
picture parameter set. An active sequence parameter set remains
unchanged throughout a coded video sequence, and an active picture
parameter set remains unchanged within a coded picture. The sequence
and picture parameter set structures contain information such as
picture size, optional coding modes employed, and macroblock to slice
group map.
To be able to change picture parameters (such as the picture size)
without having to transmit parameter set updates synchronously to the
slice packet stream, the encoder and decoder can maintain a list of
Wang, et al Expires January 14, 2009 [Page 5]
Internet-Draft RFC3984bis July 2008
more than one sequence and picture parameter set. Each slice header
contains a codeword that indicates the sequence and picture parameter
set to be used.
This mechanism allows the decoupling of the transmission of parameter
sets from the packet stream, and the transmission of them by external
means (e.g., as a side effect of the capability exchange), or through
a (reliable or unreliable) control protocol. It may even be possible
that they are never transmitted but are fixed by an application
design specification.
1.3. Network Abstraction Layer Unit Types
Tutorial information on the NAL design can be found in [12], [13],
and [14].
All NAL units consist of a single NAL unit type octet, which also co-
serves as the payload header of this RTP payload format. The payload
of a NAL unit follows immediately.
The syntax and semantics of the NAL unit type octet are specified in
[1], but the essential properties of the NAL unit type octet are
summarized below. The NAL unit type octet has the following format:
+---------------+
|0|1|2|3|4|5|6|7|
+-+-+-+-+-+-+-+-+
|F|NRI| Type |
+---------------+
The semantics of the components of the NAL unit type octet, as
specified in the H.264 specification, are described briefly below.
F: 1 bit
forbidden_zero_bit. The H.264 specification declares a value of
1 as a syntax violation.
NRI: 2 bits
nal_ref_idc. A value of 00 indicates that the content of the NAL
unit is not used to reconstruct reference pictures for inter
picture prediction. Such NAL units can be discarded without
risking the integrity of the reference pictures. Values greater
than 00 indicate that the decoding of the NAL unit is required to
maintain the integrity of the reference pictures.
Type: 5 bits
nal_unit_type. This component specifies the NAL unit payload
Wang, et al Expires January 14, 2009 [Page 6]
Internet-Draft RFC3984bis July 2008
type as defined in table 7-1 of [1], and later within this memo.
For a reference of all currently defined NAL unit types and their
semantics, please refer to section 7.4.1 in [1].
This memo introduces new NAL unit types, which are presented in
section 5.2. The NAL unit types defined in this memo are marked as
unspecified in [1]. Moreover, this specification extends the
semantics of F and NRI as described in section 5.3.
2. Conventions
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
document are to be interpreted as described in RFC-2119 [3].
This specification uses the notion of setting and clearing a bit when
bit fields are handled. Setting a bit is the same as assigning that
bit the value of 1 (On). Clearing a bit is the same as assigning
that bit the value of 0 (Off).
3. Scope
This payload specification can only be used to carry the "naked"
H.264 NAL unit stream over RTP, and not the bitstream format
discussed in Annex B of H.264. Likely, the first applications of
this specification will be in the conversational multimedia field,
video telephony or video conferencing, but the payload format also
covers other applications, such as Internet streaming and TV over IP.
4. Definitions and Abbreviations
4.1. Definitions
This document uses the definitions of [1]. The following terms,
defined in [1], are summed up for convenience:
access unit: A set of NAL units always containing a primary coded
picture. In addition to the primary coded picture, an access
unit may also contain one or more redundant coded pictures or
other NAL units not containing slices or slice data partitions of
a coded picture. The decoding of an access unit always results
in a decoded picture.
coded video sequence: A sequence of access units that consists,
in decoding order, of an instantaneous decoding refresh (IDR)
access unit followed by zero or more non-IDR access units
Wang, et al Expires January 14, 2009 [Page 7]
Internet-Draft RFC3984bis July 2008
including all subsequent access units up to but not including any
subsequent IDR access unit.
IDR access unit: An access unit in which the primary coded
picture is an IDR picture.
IDR picture: A coded picture containing only slices with I or SI
slice types that causes a "reset" in the decoding process. After
the decoding of an IDR picture, all following coded pictures in
decoding order can be decoded without inter prediction from any
picture decoded prior to the IDR picture.
primary coded picture: The coded representation of a picture to
be used by the decoding process for a bitstream conforming to
H.264. The primary coded picture contains all macroblocks of the
picture.
redundant coded picture: A coded representation of a picture or a
part of a picture. The content of a redundant coded picture
shall not be used by the decoding process for a bitstream
conforming to H.264. The content of a redundant coded picture
may be used by the decoding process for a bitstream that contains
errors or losses.
VCL NAL unit: A collective term used to refer to coded slice and
coded data partition NAL units.
In addition, the following definitions apply:
decoding order number (DON): A field in the payload structure, or
a derived variable indicating NAL unit decoding order. Values of
DON are in the range of 0 to 65535, inclusive. After reaching
the maximum value, the value of DON wraps around to 0.
NAL unit decoding order: A NAL unit order that conforms to the
constraints on NAL unit order given in section 7.4.1.2 in [1].
NALU-time: The value that the RTP timestamp would have if the NAL
unit would be transported in its own RTP packet.
transmission order: The order of packets in ascending RTP
sequence number order (in modulo arithmetic). Within an
aggregation packet, the NAL unit transmission order is the same
as the order of appearance of NAL units in the packet.
media aware network element (MANE): A network element, such as a
middlebox or application layer gateway that is capable of parsing
Wang, et al Expires January 14, 2009 [Page 8]
Internet-Draft RFC3984bis July 2008
certain aspects of the RTP payload headers or the RTP payload and
reacting to the contents.
Informative note: The concept of a MANE goes beyond normal
routers or gateways in that a MANE has to be aware of the
signaling (e.g., to learn about the payload type mappings of
the media streams), and in that it has to be trusted when
working with SRTP. The advantage of using MANEs is that they
allow packets to be dropped according to the needs of the
media coding. For example, if a MANE has to drop packets due
to congestion on a certain link, it can identify those packets
whose dropping has the smallest negative impact on the user
experience and remove them in order to remove the congestion
and/or keep the delay low.
4.2. Abbreviations
DON: Decoding Order Number
DONB: Decoding Order Number Base
DOND: Decoding Order Number Difference
FEC: Forward Error Correction
FU: Fragmentation Unit
IDR: Instantaneous Decoding Refresh
IEC: International Electrotechnical Commission
ISO: International Organization for Standardization
ITU-T: International Telecommunication Union,
Telecommunication Standardization Sector
MANE: Media Aware Network Element
MTAP: Multi-Time Aggregation Packet
MTAP16: MTAP with 16-bit timestamp offset
MTAP24: MTAP with 24-bit timestamp offset
NAL: Network Abstraction Layer
NALU: NAL Unit
SEI: Supplemental Enhancement Information
STAP: Single-Time Aggregation Packet
STAP-A: STAP type A
STAP-B: STAP type B
TS: Timestamp
VCL: Video Coding Layer
5. RTP Payload Format
5.1. RTP Header Usage
The format of the RTP header is specified in RFC 3550 [4] and
reprinted in Figure 1 for convenience. This payload format uses the
fields of the header in a manner consistent with that specification.
Wang, et al Expires January 14, 2009 [Page 9]
Internet-Draft RFC3984bis July 2008
When one NAL unit is encapsulated per RTP packet, the RECOMMENDED RTP
payload format is specified in section 5.6. The RTP payload (and the
settings for some RTP header bits) for aggregation packets and
fragmentation units are specified in sections 5.7 and 5.8,
respectively.
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|V=2|P|X| CC |M| PT | sequence number |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| timestamp |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| synchronization source (SSRC) identifier |
+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+
| contributing source (CSRC) identifiers |
| .... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Figure 1 RTP header according to RFC 3550
The RTP header information to be set according to this RTP payload
format is set as follows:
Marker bit (M): 1 bit
Set for the very last packet of the access unit indicated by the
RTP timestamp, in line with the normal use of the M bit in video
formats, to allow an efficient playout buffer handling. For
aggregation packets (STAP and MTAP), the marker bit in the RTP
header MUST be set to the value that the marker bit of the last
NAL unit of the aggregation packet would have been if it were
transported in its own RTP packet. Decoders MAY use this bit as
an early indication of the last packet of an access unit, but
MUST NOT rely on this property.
Informative note: Only one M bit is associated with an
aggregation packet carrying multiple NAL units. Thus, if a
gateway has re-packetized an aggregation packet into several
packets, it cannot reliably set the M bit of those packets.
Payload type (PT): 7 bits
The assignment of an RTP payload type for this new packet format
is outside the scope of this document and will not be specified
here. The assignment of a payload type has to be performed
either through the profile used or in a dynamic way.
Wang, et al Expires January 14, 2009 [Page 10]
Internet-Draft RFC3984bis July 2008
Sequence number (SN): 16 bits
Set and used in accordance with RFC 3550. For the single NALU
and non-interleaved packetization mode, the sequence number is
used to determine decoding order for the NALU.
Timestamp: 32 bits
The RTP timestamp is set to the sampling timestamp of the
content. A 90 kHz clock rate MUST be used.
If the NAL unit has no timing properties of its own (e.g.,
parameter set and SEI NAL units), the RTP timestamp is set to the
RTP timestamp of the primary coded picture of the access unit in
which the NAL unit is included, according to section 7.4.1.2 of
[1].
The setting of the RTP Timestamp for MTAPs is defined in section
5.7.2.
Receivers SHOULD ignore any picture timing SEI messages included
in access units that have only one display timestamp. Instead,
receivers SHOULD use the RTP timestamp for synchronizing the
display process.
RTP senders SHOULD NOT transmit picture timing SEI messages for
pictures that are not supposed to be displayed as multiple
fields.
If one access unit has more than one display timestamp carried in
a picture timing SEI message, then the information in the SEI
message SHOULD be treated as relative to the RTP timestamp, with
the earliest event occurring at the time given by the RTP
timestamp, and subsequent events later, as given by the
difference in SEI message picture timing values. Let tSEI1,
tSEI2, ..., tSEIn be the display timestamps carried in the SEI
message of an access unit, where tSEI1 is the earliest of all
such timestamps. Let tmadjst() be a function that adjusts the
SEI messages time scale to a 90-kHz time scale. Let TS be the
RTP timestamp. Then, the display time for the event associated
with tSEI1 is TS. The display time for the event with tSEIx,
where x is [2..n] is TS + tmadjst (tSEIx - tSEI1).
Informative note: Displaying coded frames as fields is needed
commonly in an operation known as 3:2 pulldown, in which film
content that consists of coded frames is displayed on a
display using interlaced scanning. The picture timing SEI
message enables carriage of multiple timestamps for the same
coded picture, and therefore the 3:2 pulldown process is
Wang, et al Expires January 14, 2009 [Page 11]
Internet-Draft RFC3984bis July 2008
perfectly controlled. The picture timing SEI message
mechanism is necessary because only one timestamp per coded
frame can be conveyed in the RTP timestamp.
Informative note: Because H.264 allows the decoding order to
be different from the display order, values of RTP timestamps
may not be monotonically non-decreasing as a function of RTP
sequence numbers. Furthermore, the value for interarrival
jitter reported in the RTCP reports may not be a trustworthy
indication of the network performance, as the calculation
rules for interarrival jitter (section 6.4.1 of RFC 3550)
assume that the RTP timestamp of a packet is directly
proportional to its transmission time.
5.2. Common Structure of the RTP Payload Format
The payload format defines three different basic payload structures.
A receiver can identify the payload structure by the first byte of
the RTP payload, which co-serves as the RTP payload header and, in
some cases, as the first byte of the payload. This byte is always
structured as a NAL unit header. The NAL unit type field indicates
which structure is present. The possible structures are as follows:
Single NAL Unit Packet: Contains only a single NAL unit in the
payload. The NAL header type field will be equal to the original NAL
unit type; i.e., in the range of 1 to 23, inclusive. Specified in
section 5.6.
Aggregation packet: Packet type used to aggregate multiple NAL units
into a single RTP payload. This packet exists in four versions, the
Single-Time Aggregation Packet type A (STAP-A), the Single-Time
Aggregation Packet type B (STAP-B), Multi-Time Aggregation Packet
(MTAP) with 16-bit offset (MTAP16), and Multi-Time Aggregation Packet
(MTAP) with 24-bit offset (MTAP24). The NAL unit type numbers
assigned for STAP-A, STAP-B, MTAP16, and MTAP24 are 24, 25, 26, and
27, respectively. Specified in section 5.7.
Fragmentation unit: Used to fragment a single NAL unit over multiple
RTP packets. Exists with two versions, FU-A and FU-B, identified
with the NAL unit type numbers 28 and 29, respectively. Specified in
section 5.8.
Informative note: This specification does not limit the size of
NAL units encapsulated in single NAL unit packets and
fragmentation units. The maximum size of a NAL unit encapsulated
in any aggregation packet is 65535 bytes.
Wang, et al Expires January 14, 2009 [Page 12]
Internet-Draft RFC3984bis July 2008
Table 1 summarizes NAL unit types and the corresponding RTP packet
types when each of these NAL units is directly used a packet payload,
and where the types are described in this memo.
Table 1. Summary of NAL unit types and the corresponding packet
types
NAL Unit Packet Packet Type Name Section
Type Type
---------------------------------------------------------
0 undefined -
1-23 NAL unit Single NAL unit packet 5.6
24 STAP-A Single-time aggregation packet 5.7.1
25 STAP-B Single-time aggregation packet 5.7.1
26 MTAP16 Multi-time aggregation packet 5.7.2
27 MTAP24 Multi-time aggregation packet 5.7.2
28 FU-A Fragmentation unit 5.8
29 FU-B Fragmentation unit 5.8
30-31 undefined -
5.3. NAL Unit Octet Usage
The structure and semantics of the NAL unit octet were introduced in
section 1.3. For convenience, the format of the NAL unit type octet
is reprinted below:
+---------------+
|0|1|2|3|4|5|6|7|
+-+-+-+-+-+-+-+-+
|F|NRI| Type |
+---------------+
This section specifies the semantics of F and NRI according to this
specification.
F: 1 bit
forbidden_zero_bit. A value of 0 indicates that the NAL unit
type octet and payload should not contain bit errors or other
syntax violations. A value of 1 indicates that the NAL unit type
octet and payload may contain bit errors or other syntax
violations.
MANEs SHOULD set the F bit to indicate detected bit errors in the
NAL unit. The H.264 specification requires that the F bit is
equal to 0. When the F bit is set, the decoder is advised that
bit errors or any other syntax violations may be present in the
payload or in the NAL unit type octet. The simplest decoder
Wang, et al Expires January 14, 2009 [Page 13]
Internet-Draft RFC3984bis July 2008
reaction to a NAL unit in which the F bit is equal to 1 is to
discard such a NAL unit and to conceal the lost data in the
discarded NAL unit.
NRI: 2 bits
nal_ref_idc. The semantics of value 00 and a non-zero value
remain unchanged from the H.264 specification. In other words, a
value of 00 indicates that the content of the NAL unit is not
used to reconstruct reference pictures for inter picture
prediction. Such NAL units can be discarded without risking the
integrity of the reference pictures. Values greater than 00
indicate that the decoding of the NAL unit is required to
maintain the integrity of the reference pictures.
In addition to the specification above, according to this RTP
payload specification, values of NRI indicate the relative
transport priority, as determined by the encoder. MANEs can use
this information to protect more important NAL units better than
they do less important NAL units. The highest transport priority
is 11, followed by 10, and then by 01; finally, 00 is the lowest.
Informative note: Any non-zero value of NRI is handled
identically in H.264 decoders. Therefore, receivers need not
manipulate the value of NRI when passing NAL units to the
decoder.
An H.264 encoder MUST set the value of NRI according to the H.264
specification (subclause 7.4.1) when the value of nal_unit_type
is in the range of 1 to 12, inclusive. In particular, the H.264
specification requires that the value of NRI SHALL be equal to 0
for all NAL units having nal_unit_type equal to 6, 9, 10, 11, or
12.
For NAL units having nal_unit_type equal to 7 or 8 (indicating a
sequence parameter set or a picture parameter set, respectively),
an H.264 encoder SHOULD set the value of NRI to 11 (in binary
format). For coded slice NAL units of a primary coded picture
having nal_unit_type equal to 5 (indicating a coded slice
belonging to an IDR picture), an H.264 encoder SHOULD set the
value of NRI to 11 (in binary format).
For a mapping of the remaining nal_unit_types to NRI values, the
following example MAY be used and has been shown to be efficient
in a certain environment [13]. Other mappings MAY also be
desirable, depending on the application and the H.264/AVC Annex A
profile in use.
Wang, et al Expires January 14, 2009 [Page 14]
Internet-Draft RFC3984bis July 2008
Informative note: Data Partitioning is not available in
certain profiles; e.g., in the Main or Baseline profiles.
Consequently, the NAL unit types 2, 3, and 4 can occur only if
the video bitstream conforms to a profile in which data
partitioning is allowed and not in streams that conform to the
Main or Baseline profiles.
Table 2. Example of NRI values for coded slices and coded slice data
partitions of primary coded reference pictures
NAL Unit Type Content of NAL unit NRI (binary)
----------------------------------------------------------------
1 non-IDR coded slice 10
2 Coded slice data partition A 10
3 Coded slice data partition B 01
4 Coded slice data partition C 01
Informative note: As mentioned before, the NRI value of non-
reference pictures is 00 as mandated by H.264/AVC.
An H.264 encoder SHOULD set the value of NRI for coded slice and
coded slice data partition NAL units of redundant coded reference
pictures equal to 01 (in binary format).
Definitions of the values for NRI for NAL unit types 24 to 29,
inclusive, are given in sections 5.7 and 5.8 of this memo.
No recommendation for the value of NRI is given for NAL units
having nal_unit_type in the range of 13 to 23, inclusive, because
these values are reserved for ITU-T and ISO/IEC. No
recommendation for the value of NRI is given for NAL units having
nal_unit_type equal to 0 or in the range of 30 to 31, inclusive,
as the semantics of these values are not specified in this memo.
5.4. Packetization Modes
This memo specifies three cases of packetization modes:
o Single NAL unit mode
o Non-interleaved mode
o Interleaved mode
The single NAL unit mode is targeted for conversational systems that
comply with ITU-T Recommendation H.241 [15] (see section 12.1). The
non-interleaved mode is targeted for conversational systems that may
Wang, et al Expires January 14, 2009 [Page 15]
Internet-Draft RFC3984bis July 2008
not comply with ITU-T Recommendation H.241. In the non-interleaved
mode, NAL units are transmitted in NAL unit decoding order. The
interleaved mode is targeted for systems that do not require very low
end-to-end latency. The interleaved mode allows transmission of NAL
units out of NAL unit decoding order.
The packetization mode in use MAY be signaled by the value of the
OPTIONAL packetization-mode MIME parameter. The used packetization
mode governs which NAL unit types are allowed in RTP payloads. Table
3 summarizes the allowed packet payload types for each packetization
mode. Packetization modes are explained in more detail in section 6.
Table 3. Summary of allowed NAL unit types for each packetization
mode (yes = allowed, no = disallowed, ig = ignore)
Payload Packet Single NAL Non-Interleaved Interleaved
Type Type Unit Mode Mode Mode
-------------------------------------------------------------
0 undefined ig ig ig
1-23 NAL unit yes yes no
24 STAP-A no yes no
25 STAP-B no no yes
26 MTAP16 no no yes
27 MTAP24 no no yes
28 FU-A no yes yes
29 FU-B no no yes
30-31 undefined ig ig ig
Some UAL unit or payload type values (indicated as undefined in Table
3) are reserved for future extensions. NAL units of those types
SHOULD NOT be sent by a sender (direct as packet payloads, or as
aggregation units in aggregation packets, or as fragmented units in
FU packets) and MUST be ignored by a receiver. For example, the
payload types 1-23, with the associated packet type "NAL unit", are
allowed in "Single NAL Unit Mode" and in "Non-Interleaved Mode", but
disallowed in "Interleaved Mode". However, NAL units of NAL unit
types 1-23 can be used in "Interleaved Mode" as aggregation units in
STAP-B, MTAP16 and MTAP14 packets as well as fragmented units in FU-A
and FU-B packets. Similarly, NAL units of NAL unit types 1-23 can
also be used in the "Non-Interleaved Mode" as aggregation units in
STAP-A packets or fragmented units in FU-A packets, in addition to
being directly used as packet payloads.
5.5. Decoding Order Number (DON)
In the interleaved packetization mode, the transmission order of NAL
units is allowed to differ from the decoding order of the NAL units.
Wang, et al Expires January 14, 2009 [Page 16]
Internet-Draft RFC3984bis July 2008
Decoding order number (DON) is a field in the payload structure or a
derived variable that indicates the NAL unit decoding order.
Rationale and examples of use cases for transmission out of decoding
order and for the use of DON are given in section 13.
The coupling of transmission and decoding order is controlled by the
OPTIONAL sprop-interleaving-depth MIME parameter as follows. When
the value of the OPTIONAL sprop-interleaving-depth MIME parameter is
equal to 0 (explicitly or per default), the transmission order of NAL
units MUST conform to the NAL unit decoding order. When the value of
the OPTIONAL sprop-interleaving-depth MIME parameter is greater than
0,
o the order of NAL units in an MTAP16 and an MTAP24 is NOT REQUIRED
to be the NAL unit decoding order, and
o the order of NAL units generated by decapsulating STAP-Bs, MTAPs,
and FUs in two consecutive packets is NOT REQUIRED to be the NAL
unit decoding order.
The RTP payload structures for a single NAL unit packet, an STAP-A,
and an FU-A do not include DON. STAP-B and FU-B structures include
DON, and the structure of MTAPs enables derivation of DON as
specified in section 5.7.2.
Informative note: When an FU-A occurs in interleaved mode, it
always follows an FU-B, which sets its DON.
Informative note: If a transmitter wants to encapsulate a single
NAL unit per packet and transmit packets out of their decoding
order, STAP-B packet type can be used.
In the single NAL unit packetization mode, the transmission order of
NAL units, determined by the RTP sequence number, MUST be the same as
their NAL unit decoding order. In the non-interleaved packetization
mode, the transmission order of NAL units in single NAL unit packets,
STAP-As, and FU-As MUST be the same as their NAL unit decoding order.
The NAL units within an STAP MUST appear in the NAL unit decoding
order. Thus, the decoding order is first provided through the
implicit order within a STAP, and second provided through the RTP
sequence number for the order between STAPs, FUs, and single NAL unit
packets.
Signaling of the value of DON for NAL units carried in STAP-B, MTAP,
and a series of fragmentation units starting with an FU-B is
specified in sections 5.7.1, 5.7.2, and 5.8, respectively. The DON
value of the first NAL unit in transmission order MAY be set to any
Wang, et al Expires January 14, 2009 [Page 17]
Internet-Draft RFC3984bis July 2008
value. Values of DON are in the range of 0 to 65535, inclusive.
After reaching the maximum value, the value of DON wraps around to 0.
The decoding order of two NAL units contained in any STAP-B, MTAP, or
a series of fragmentation units starting with an FU-B is determined
as follows. Let DON(i) be the decoding order number of the NAL unit
having index i in the transmission order. Function don_diff(m,n) is
specified as follows:
If DON(m) == DON(n), don_diff(m,n) = 0
If (DON(m) < DON(n) and DON(n) - DON(m) < 32768),
don_diff(m,n) = DON(n) - DON(m)
If (DON(m) > DON(n) and DON(m) - DON(n) >= 32768),
don_diff(m,n) = 65536 - DON(m) + DON(n)
If (DON(m) < DON(n) and DON(n) - DON(m) >= 32768),
don_diff(m,n) = - (DON(m) + 65536 - DON(n))
If (DON(m) > DON(n) and DON(m) - DON(n) < 32768),
don_diff(m,n) = - (DON(m) - DON(n))
A positive value of don_diff(m,n) indicates that the NAL unit having
transmission order index n follows, in decoding order, the NAL unit
having transmission order index m. When don_diff(m,n) is equal to 0,
then the NAL unit decoding order of the two NAL units can be in
either order. A negative value of don_diff(m,n) indicates that the
NAL unit having transmission order index n precedes, in decoding
order, the NAL unit having transmission order index m.
Values of DON related fields (DON, DONB, and DOND; see section 5.7)
MUST be such that the decoding order determined by the values of DON,
as specified above, conforms to the NAL unit decoding order. If the
order of two NAL units in NAL unit decoding order is switched and the
new order does not conform to the NAL unit decoding order, the NAL
units MUST NOT have the same value of DON. If the order of two
consecutive NAL units in the NAL unit stream is switched and the new
order still conforms to the NAL unit decoding order, the NAL units
MAY have the same value of DON. For example, when arbitrary slice
order is allowed by the video coding profile in use, all the coded
slice NAL units of a coded picture are allowed to have the same value
of DON. Consequently, NAL units having the same value of DON can be
decoded in any order, and two NAL units having a different value of
DON should be passed to the decoder in the order specified above.
When two consecutive NAL units in the NAL unit decoding order have a
different value of DON, the value of DON for the second NAL unit in
Wang, et al Expires January 14, 2009 [Page 18]
Internet-Draft RFC3984bis July 2008
decoding order SHOULD be the value of DON for the first, incremented
by one.
An example of the decapsulation process to recover the NAL unit
decoding order is given in section 7.
Informative note: Receivers should not expect that the absolute
difference of values of DON for two consecutive NAL units in the
NAL unit decoding order will be equal to one, even in error-free
transmission. An increment by one is not required, as at the
time of associating values of DON to NAL units, it may not be
known whether all NAL units are delivered to the receiver. For
example, a gateway may not forward coded slice NAL units of non-
reference pictures or SEI NAL units when there is a shortage of
bit rate in the network to which the packets are forwarded. In
another example, a live broadcast is interrupted by pre-encoded
content, such as commercials, from time to time. The first intra
picture of a pre-encoded clip is transmitted in advance to ensure
that it is readily available in the receiver. When transmitting
the first intra picture, the originator does not exactly know how
many NAL units will be encoded before the first intra picture of
the pre-encoded clip follows in decoding order. Thus, the values
of DON for the NAL units of the first intra picture of the pre-
encoded clip have to be estimated when they are transmitted, and
gaps in values of DON may occur.
5.6. Single NAL Unit Packet
The single NAL unit packet defined here MUST contain only one NAL
unit, of the types defined in [1]. This means that neither an
aggregation packet nor a fragmentation unit can be used within a
single NAL unit packet. A NAL unit stream composed by decapsulating
single NAL unit packets in RTP sequence number order MUST conform to
the NAL unit decoding order. The structure of the single NAL unit
packet is shown in Figure 2.
Informative note: The first byte of a NAL unit co-serves as the
RTP payload header.
Wang, et al Expires January 14, 2009 [Page 19]
Internet-Draft RFC3984bis July 2008
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|F|NRI| Type | |
+-+-+-+-+-+-+-+-+ |
| |
| Bytes 2..n of a Single NAL unit |
| |
| +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| :...OPTIONAL RTP padding |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Figure 2 RTP payload format for single NAL unit packet
5.7. Aggregation Packets
Aggregation packets are the NAL unit aggregation scheme of this
payload specification. The scheme is introduced to reflect the
dramatically different MTU sizes of two key target networks: wireline
IP networks (with an MTU size that is often limited by the Ethernet
MTU size; roughly 1500 bytes), and IP or non-IP (e.g., ITU-T H.324/M)
based wireless communication systems with preferred transmission unit
sizes of 254 bytes or less. To prevent media transcoding between the
two worlds, and to avoid undesirable packetization overhead, a NAL
unit aggregation scheme is introduced.
Two types of aggregation packets are defined by this specification:
o Single-time aggregation packet (STAP): aggregates NAL units with
identical NALU-time. Two types of STAPs are defined, one without
DON (STAP-A) and another including DON (STAP-B).
o Multi-time aggregation packet (MTAP): aggregates NAL units with
potentially differing NALU-time. Two different MTAPs are defined,
differing in the length of the NAL unit timestamp offset.
Each NAL unit to be carried in an aggregation packet is encapsulated
in an aggregation unit. Please see below for the four different
aggregation units and their characteristics.
The structure of the RTP payload format for aggregation packets is
presented in Figure 3.
Wang, et al Expires January 14, 2009 [Page 20]
Internet-Draft RFC3984bis July 2008
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|F|NRI| Type | |
+-+-+-+-+-+-+-+-+ |
| |
| one or more aggregation units |
| |
| +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| :...OPTIONAL RTP padding |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Figure 3 RTP payload format for aggregation packets
MTAPs and STAPs share the following packetization rules: The RTP
timestamp MUST be set to the earliest of the NALU-times of all the
NAL units to be aggregated. The type field of the NAL unit type
octet MUST be set to the appropriate value, as indicated in Table 4.
The F bit MUST be cleared if all F bits of the aggregated NAL units
are zero; otherwise, it MUST be set. The value of NRI MUST be the
maximum of all the NAL units carried in the aggregation packet.
Table 4. Type field for STAPs and MTAPs
Type Packet Timestamp offset DON related fields
field length (DON, DONB, DOND)
(in bits) present
--------------------------------------------------------
24 STAP-A 0 no
25 STAP-B 0 yes
26 MTAP16 16 yes
27 MTAP24 24 yes
The marker bit in the RTP header is set to the value that the marker
bit of the last NAL unit of the aggregated packet would have if it
were transported in its own RTP packet.
The payload of an aggregation packet consists of one or more
aggregation units. See sections 5.7.1 and 5.7.2 for the four
different types of aggregation units. An aggregation packet can
carry as many aggregation units as necessary; however, the total
amount of data in an aggregation packet obviously MUST fit into an IP
packet, and the size SHOULD be chosen so that the resulting IP packet
is smaller than the MTU size. An aggregation packet MUST NOT contain
fragmentation units specified in section 5.8. Aggregation packets
MUST NOT be nested; i.e., an aggregation packet MUST NOT contain
another aggregation packet.
Wang, et al Expires January 14, 2009 [Page 21]
Internet-Draft RFC3984bis July 2008
5.7.1. Single-Time Aggregation Packet
Single-time aggregation packet (STAP) SHOULD be used whenever NAL
units are aggregated that all share the same NALU-time. The payload
of an STAP-A does not include DON and consists of at least one
single-time aggregation unit, as presented in Figure 4. The payload
of an STAP-B consists of a 16-bit unsigned decoding order number
(DON) (in network byte order) followed by at least one single-time
aggregation unit, as presented in Figure 5.
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
: |
+-+-+-+-+-+-+-+-+ |
| |
| single-time aggregation units |
| |
| +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| :
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Figure 4 Payload format for STAP-A
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
: decoding order number (DON) | |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |
| |
| single-time aggregation units |
| |
| +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| :
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Figure 5 Payload format for STAP-B
The DON field specifies the value of DON for the first NAL unit in an
STAP-B in transmission order. For each successive NAL unit in
appearance order in an STAP-B, the value of DON is equal to (the
value of DON of the previous NAL unit in the STAP-B + 1) % 65536, in
which '%' stands for the modulo operation.
A single-time aggregation unit consists of 16-bit unsigned size
information (in network byte order) that indicates the size of the
following NAL unit in bytes (excluding these two octets, but
Wang, et al Expires January 14, 2009 [Page 22]
Internet-Draft RFC3984bis July 2008
including the NAL unit type octet of the NAL unit), followed by the
NAL unit itself, including its NAL unit type byte. A single-time
aggregation unit is byte aligned within the RTP payload, but it may
not be aligned on a 32-bit word boundary. Figure 6 presents the
structure of the single-time aggregation unit.
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
: NAL unit size | |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |
| |
| NAL unit |
| |
| +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| :
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Figure 6 Structure for single-time aggregation unit
Figure 7 presents an example of an RTP packet that contains an STAP-
A. The STAP contains two single-time aggregation units, labeled as 1
and 2 in the figure.
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| RTP Header |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|STAP-A NAL HDR | NALU 1 Size | NALU 1 HDR |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| NALU 1 Data |
: :
+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| | NALU 2 Size | NALU 2 HDR |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| NALU 2 Data |
: :
| +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| :...OPTIONAL RTP padding |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Figure 7 An example of an RTP packet including an STAP-A containing
two single-time aggregation units
Wang, et al Expires January 14, 2009 [Page 23]
Internet-Draft RFC3984bis July 2008
Figure 8 presents an example of an RTP packet that contains an STAP-
B. The STAP contains two single-time aggregation units, labeled as 1
and 2 in the figure.
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| RTP Header |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|STAP-B NAL HDR | DON | NALU 1 Size |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| NALU 1 Size | NALU 1 HDR | NALU 1 Data |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +
: :
+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| | NALU 2 Size | NALU 2 HDR |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| NALU 2 Data |
: :
| +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| :...OPTIONAL RTP padding |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Figure 8 An example of an RTP packet including an STAP-B containing
two single-time aggregation units
5.7.2. Multi-Time Aggregation Packets (MTAPs)
The NAL unit payload of MTAPs consists of a 16-bit unsigned decoding
order number base (DONB) (in network byte order) and one or more
multi-time aggregation units, as presented in Figure 9. DONB MUST
contain the value of DON for the first NAL unit in the NAL unit
decoding order among the NAL units of the MTAP.
Informative note: The first NAL unit in the NAL unit decoding
order is not necessarily the first NAL unit in the order in which
the NAL units are encapsulated in an MTAP.
Wang, et al Expires January 14, 2009 [Page 24]
Internet-Draft RFC3984bis July 2008
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
: decoding order number base | |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |
| |
| multi-time aggregation units |
| |
| +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| :
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Figure 9 NAL unit payload format for MTAPs
Two different multi-time aggregation units are defined in this
specification. Both of them consist of 16 bits unsigned size
information of the following NAL unit (in network byte order), an 8-
bit unsigned decoding order number difference (DOND), and n bits (in
network byte order) of timestamp offset (TS offset) for this NAL
unit, whereby n can be 16 or 24. The choice between the different
MTAP types (MTAP16 and MTAP24) is application dependent: the larger
the timestamp offset is, the higher the flexibility of the MTAP, but
the overhead is also higher.
The structure of the multi-time aggregation units for MTAP16 and
MTAP24 are presented in Figures 10 and 11, respectively. The
starting or ending position of an aggregation unit within a packet is
NOT REQUIRED to be on a 32-bit word boundary. The DON of the NAL
unit contained in a multi-time aggregation unit is equal to (DONB +
DOND) % 65536, in which % denotes the modulo operation. This memo
does not specify how the NAL units within an MTAP are ordered, but,
in most cases, NAL unit decoding order SHOULD be used.
The timestamp offset field MUST be set to a value equal to the value
of the following formula: If the NALU-time is larger than or equal to
the RTP timestamp of the packet, then the timestamp offset equals
(the NALU-time of the NAL unit - the RTP timestamp of the packet).
If the NALU-time is smaller than the RTP timestamp of the packet,
then the timestamp offset is equal to the NALU-time + (2^32 - the RTP
timestamp of the packet).
Wang, et al Expires January 14, 2009 [Page 25]
Internet-Draft RFC3984bis July 2008
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
: NAL unit size | DOND | TS offset |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| TS offset | |
+-+-+-+-+-+-+-+-+ NAL unit |
| |
| +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| :
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Figure 10 Multi-time aggregation unit for MTAP16
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
: NAL unit size | DOND | TS offset |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| TS offset | |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |
| NAL unit |
| +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| :
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Figure 11 Multi-time aggregation unit for MTAP24
For the "earliest" multi-time aggregation unit in an MTAP the
timestamp offset MUST be zero. Hence, the RTP timestamp of the MTAP
itself is identical to the earliest NALU-time.
Informative note: The "earliest" multi-time aggregation unit is
the one that would have the smallest extended RTP timestamp among
all the aggregation units of an MTAP if the NAL units contained
in the aggregation units were encapsulated in single NAL unit
packets. An extended timestamp is a timestamp that has more than
32 bits and is capable of counting the wraparound of the
timestamp field, thus enabling one to determine the smallest
value if the timestamp wraps. Such an "earliest" aggregation
unit may not be the first one in the order in which the
aggregation units are encapsulated in an MTAP. The "earliest"
NAL unit need not be the same as the first NAL unit in the NAL
unit decoding order either.
Wang, et al Expires January 14, 2009 [Page 26]
Internet-Draft RFC3984bis July 2008
Figure 12 presents an example of an RTP packet that contains a multi-
time aggregation packet of type MTAP16 that contains two multi-time
aggregation units, labeled as 1 and 2 in the figure.
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| RTP Header |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|MTAP16 NAL HDR | decoding order number base | NALU 1 Size |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| NALU 1 Size | NALU 1 DOND | NALU 1 TS offset |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| NALU 1 HDR | NALU 1 DATA |
+-+-+-+-+-+-+-+-+ +
: :
+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| | NALU 2 SIZE | NALU 2 DOND |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| NALU 2 TS offset | NALU 2 HDR | NALU 2 DATA |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |
: :
| +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| :...OPTIONAL RTP padding |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Figure 12 An RTP packet including a multi-time aggregation packet of
type MTAP16 containing two multi-time aggregation units
Figure 13 presents an example of an RTP packet that contains a multi-
time aggregation packet of type MTAP24 that contains two multi-time
aggregation units, labeled as 1 and 2 in the figure.
Wang, et al Expires January 14, 2009 [Page 27]
Internet-Draft RFC3984bis July 2008
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| RTP Header |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|MTAP24 NAL HDR | decoding order number base | NALU 1 Size |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| NALU 1 Size | NALU 1 DOND | NALU 1 TS offs |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|NALU 1 TS offs | NALU 1 HDR | NALU 1 DATA |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +
: :
+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| | NALU 2 SIZE | NALU 2 DOND |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| NALU 2 TS offset | NALU 2 HDR |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| NALU 2 DATA |
: :
| +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| :...OPTIONAL RTP padding |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Figure 13 An RTP packet including a multi-time aggregation packet of
type MTAP24 containing two multi-time aggregation units
5.7.3. Fragmentation Units (FUs)
This payload type allows fragmenting a NAL unit into several RTP
packets. Doing so on the application layer instead of relying on
lower layer fragmentation (e.g., by IP) has the following advantages:
o The payload format is capable of transporting NAL units bigger
than 64 kbytes over an IPv4 network that may be present in pre-
recorded video, particularly in High Definition formats (there is
a limit of the number of slices per picture, which results in a
limit of NAL units per picture, which may result in big NAL
units).
o The fragmentation mechanism allows fragmenting a single NAL unit
and applying generic forward error correction as described in
section 12.5.
Fragmentation is defined only for a single NAL unit and not for any
aggregation packets. A fragment of a NAL unit consists of an integer
number of consecutive octets of that NAL unit. Each octet of the NAL
unit MUST be part of exactly one fragment of that NAL unit.
Wang, et al Expires January 14, 2009 [Page 28]
Internet-Draft RFC3984bis July 2008
Fragments of the same NAL unit MUST be sent in consecutive order with
ascending RTP sequence numbers (with no other RTP packets within the
same RTP packet stream being sent between the first and last
fragment). Similarly, a NAL unit MUST be reassembled in RTP sequence
number order.
When a NAL unit is fragmented and conveyed within fragmentation units
(FUs), it is referred to as a fragmented NAL unit. STAPs and MTAPs
MUST NOT be fragmented. FUs MUST NOT be nested; i.e., an FU MUST NOT
contain another FU.
The RTP timestamp of an RTP packet carrying an FU is set to the NALU-
time of the fragmented NAL unit.
Figure 14 presents the RTP payload format for FU-As. An FU-A
consists of a fragmentation unit indicator of one octet, a
fragmentation unit header of one octet, and a fragmentation unit
payload.
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| FU indicator | FU header | |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |
| |
| FU payload |
| |
| +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| :...OPTIONAL RTP padding |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Figure 14 RTP payload format for FU-A
Figure 15 presents the RTP payload format for FU-Bs. An FU-B
consists of a fragmentation unit indicator of one octet, a
fragmentation unit header of one octet, a decoding order number (DON)
(in network byte order), and a fragmentation unit payload. In other
words, the structure of FU-B is the same as the structure of FU-A,
except for the additional DON field.
Wang, et al Expires January 14, 2009 [Page 29]
Internet-Draft RFC3984bis July 2008
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| FU indicator | FU header | DON |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-|
| |
| FU payload |
| |
| +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| :...OPTIONAL RTP padding |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Figure 15 RTP payload format for FU-B
NAL unit type FU-B MUST be used in the interleaved packetization mode
for the first fragmentation unit of a fragmented NAL unit. NAL unit
type FU-B MUST NOT be used in any other case. In other words, in the
interleaved packetization mode, each NALU that is fragmented has an
FU-B as the first fragment, followed by one or more FU-A fragments.
The FU indicator octet has the following format:
+---------------+
|0|1|2|3|4|5|6|7|
+-+-+-+-+-+-+-+-+
|F|NRI| Type |
+---------------+
Values equal to 28 and 29 in the Type field of the FU indicator octet
identify an FU-A and an FU-B, respectively. The use of the F bit is
described in section 5.3. The value of the NRI field MUST be set
according to the value of the NRI field in the fragmented NAL unit.
The FU header has the following format:
+---------------+
|0|1|2|3|4|5|6|7|
+-+-+-+-+-+-+-+-+
|S|E|R| Type |
+---------------+
S: 1 bit
When set to one, the Start bit indicates the start of a
fragmented NAL unit. When the following FU payload is not the
start of a fragmented NAL unit payload, the Start bit is set to
zero.
Wang, et al Expires January 14, 2009 [Page 30]
Internet-Draft RFC3984bis July 2008
E: 1 bit
When set to one, the End bit indicates the end of a fragmented
NAL unit, i.e., the last byte of the payload is also the last
byte of the fragmented NAL unit. When the following FU payload
is not the last fragment of a fragmented NAL unit, the End bit is
set to zero.
R: 1 bit
The Reserved bit MUST be equal to 0 and MUST be ignored by the
receiver.
Type: 5 bits
The NAL unit payload type as defined in table 7-1 of [1].
The value of DON in FU-Bs is selected as described in section 5.5.
Informative note: The DON field in FU-Bs allows gateways to
fragment NAL units to FU-Bs without organizing the incoming NAL
units to the NAL unit decoding order.
A fragmented NAL unit MUST NOT be transmitted in one FU; i.e., the
Start bit and End bit MUST NOT both be set to one in the same FU
header.
The FU payload consists of fragments of the payload of the fragmented
NAL unit so that if the fragmentation unit payloads of consecutive
FUs are sequentially concatenated, the payload of the fragmented NAL
unit can be reconstructed. The NAL unit type octet of the fragmented
NAL unit is not included as such in the fragmentation unit payload,
but rather the information of the NAL unit type octet of the
fragmented NAL unit is conveyed in F and NRI fields of the FU
indicator octet of the fragmentation unit and in the type field of
the FU header. An FU payload MAY have any number of octets and MAY
be empty.
Informative note: Empty FUs are allowed to reduce the latency of
a certain class of senders in nearly lossless environments.
These senders can be characterized in that they packetize NALU
fragments before the NALU is completely generated and, hence,
before the NALU size is known. If zero-length NALU fragments
were not allowed, the sender would have to generate at least one
bit of data of the following fragment before the current fragment
could be sent. Due to the characteristics of H.264, where
sometimes several macroblocks occupy zero bits, this is
undesirable and can add delay. However, the (potential) use of
zero-length NALU fragments should be carefully weighed against
Wang, et al Expires January 14, 2009 [Page 31]
Internet-Draft RFC3984bis July 2008
the increased risk of the loss of at least a part of the NALU
because of the additional packets employed for its transmission.
If a fragmentation unit is lost, the receiver SHOULD discard all
following fragmentation units in transmission order corresponding to
the same fragmented NAL unit.
A receiver in an endpoint or in a MANE MAY aggregate the first n-1
fragments of a NAL unit to an (incomplete) NAL unit, even if fragment
n of that NAL unit is not received. In this case, the
forbidden_zero_bit of the NAL unit MUST be set to one to indicate a
syntax violation.
6. Packetization Rules
The packetization modes are introduced in section 5.2. The
packetization rules common to more than one of the packetization
modes are specified in section 6.1. The packetization rules for the
single NAL unit mode, the non-interleaved mode, and the interleaved
mode are specified in sections 6.2, 6.3, and 6.4, respectively.
6.1. Common Packetization Rules
All senders MUST enforce the following packetization rules regardless
of the packetization mode in use:
o Coded slice NAL units or coded slice data partition NAL units
belonging to the same coded picture (and thus sharing the same RTP
timestamp value) MAY be sent in any order; however, for delay-
critical systems, they SHOULD be sent in their original decoding
order to minimize the delay. Note that the decoding order is the
order of the NAL units in the bitstream.
o Parameter sets are handled in accordance with the rules and
recommendations given in section 8.4.
o MANEs MUST NOT duplicate any NAL unit except for sequence or
picture parameter set NAL units, as neither this memo nor the
H.264 specification provides means to identify duplicated NAL
units. Sequence and picture parameter set NAL units MAY be
duplicated to make their correct reception more probable, but any
such duplication MUST NOT affect the contents of any active
sequence or picture parameter set. Duplication SHOULD be
performed on the application layer and not by duplicating RTP
packets (with identical sequence numbers).
Wang, et al Expires January 14, 2009 [Page 32]
Internet-Draft RFC3984bis July 2008
Senders using the non-interleaved mode and the interleaved mode MUST
enforce the following packetization rule:
o MANEs MAY convert single NAL unit packets into one aggregation
packet, convert an aggregation packet into several single NAL unit
packets, or mix both concepts, in an RTP translator. The RTP
translator SHOULD take into account at least the following
parameters: path MTU size, unequal protection mechanisms (e.g.,
through packet-based FEC according to RFC 2733 [18], especially
for sequence and picture parameter set NAL units and coded slice
data partition A NAL units), bearable latency of the system, and
buffering capabilities of the receiver.
Informative note: An RTP translator is required to handle RTCP
as per RFC 3550.
6.2. Single NAL Unit Mode
This mode is in use when the value of the OPTIONAL packetization-mode
MIME parameter is equal to 0 or the packetization-mode is not
present. All receivers MUST support this mode. It is primarily
intended for low-delay applications that are compatible with systems
using ITU-T Recommendation H.241 [15] (see section 12.1). Only
single NAL unit packets MAY be used in this mode. STAPs, MTAPs, and
FUs MUST NOT be used. The transmission order of single NAL unit
packets MUST comply with the NAL unit decoding order.
6.3. Non-Interleaved Mode
This mode is in use when the value of the OPTIONAL packetization-mode
MIME parameter is equal to 1. This mode SHOULD be supported. It is
primarily intended for low-delay applications. Only single NAL unit
packets, STAP-As, and FU-As MAY be used in this mode. STAP-Bs,
MTAPs, and FU-Bs MUST NOT be used. The transmission order of NAL
units MUST comply with the NAL unit decoding order.
6.4. Interleaved Mode
This mode is in use when the value of the OPTIONAL packetization-mode
MIME parameter is equal to 2. Some receivers MAY support this mode.
STAP-Bs, MTAPs, FU-As, and FU-Bs MAY be used. STAP-As and single NAL
unit packets MUST NOT be used. The transmission order of packets and
NAL units is constrained as specified in section 5.5.
Wang, et al Expires January 14, 2009 [Page 33]
Internet-Draft RFC3984bis July 2008
7. De-Packetization Process
The de-packetization process is implementation dependent. Therefore,
the following description should be seen as an example of a suitable
implementation. Other schemes may be used as well as long as the
output for the same input is the same as the process described below.
The output is the same meaning that the number of NAL units and their
order are both the identical. Optimizations relative to the
described algorithms are likely possible. Section 7.1 presents the
de-packetization process for the single NAL unit and non-interleaved
packetization modes, whereas section 7.2 describes the process for
the interleaved mode. Section 7.3 includes additional decapsulation
guidelines for intelligent receivers.
All normal RTP mechanisms related to buffer management apply. In
particular, duplicated or outdated RTP packets (as indicated by the
RTP sequences number and the RTP timestamp) are removed. To
determine the exact time for decoding, factors such as a possible
intentional delay to allow for proper inter-stream synchronization
must be factored in.
7.1. Single NAL Unit and Non-Interleaved Mode
The receiver includes a receiver buffer to compensate for
transmission delay jitter. The receiver stores incoming packets in
reception order into the receiver buffer. Packets are decapsulated
in RTP sequence number order. If a decapsulated packet is a single
NAL unit packet, the NAL unit contained in the packet is passed
directly to the decoder. If a decapsulated packet is an STAP-A, the
NAL units contained in the packet are passed to the decoder in the
order in which they are encapsulated in the packet. For all the FU-A
packets containing fragments of a single NAL unit, the decapsulated
fragments are concatenated in their sending order to recover the NAL
unit, which is then passed to the decoder.
Informative note: If the decoder supports Arbitrary Slice Order,
coded slices of a picture can be passed to the decoder in any
order regardless of their reception and transmission order.
7.2. Interleaved Mode
The general concept behind these de-packetization rules is to reorder
NAL units from transmission order to the NAL unit decoding order.
The receiver includes a receiver buffer, which is used to compensate
for transmission delay jitter and to reorder NAL units from
transmission order to the NAL unit decoding order. In this section,
Wang, et al Expires January 14, 2009 [Page 34]
Internet-Draft RFC3984bis July 2008
the receiver operation is described under the assumption that there
is no transmission delay jitter. To make a difference from a
practical receiver buffer that is also used for compensation of
transmission delay jitter, the receiver buffer is here after called
the deinterleaving buffer in this section. Receivers SHOULD also
prepare for transmission delay jitter; i.e., either reserve separate
buffers for transmission delay jitter buffering and deinterleaving
buffering or use a receiver buffer for both transmission delay jitter
and deinterleaving. Moreover, receivers SHOULD take transmission
delay jitter into account in the buffering operation; e.g., by
additional initial buffering before starting of decoding and
playback.
This section is organized as follows: subsection 7.2.1 presents how o
calculate the size of the deinterleaving buffer. Subsection 7.2.2
specifies the receiver process how to organize received NAL units to
the NAL unit decoding order.
7.2.1. Size of the Deinterleaving Buffer
When SDP Offer/Answer model or any other capability exchange
procedure is used in session setup, the properties of the received
stream SHOULD be such that the receiver capabilities are not
exceeded. In the SDP Offer/Answer model, the receiver can indicate
its capabilities to allocate a deinterleaving buffer with the deint-
buf-cap MIME parameter. The sender indicates the requirement for the
deinterleaving buffer size with the sprop-deint-buf-req MIME
parameter. It is therefore RECOMMENDED to set the deinterleaving
buffer size, in terms of number of bytes, equal to or greater than
the value of sprop-deint-buf-req MIME parameter. See section 8.1 for
further information on deint-buf-cap and sprop-deint-buf-req MIME
parameters and section 8.2.2 for further information on their use in
SDP Offer/Answer model.
When a declarative session description is used in session setup, the
sprop-deint-buf-req MIME parameter signals the requirement for the
deinterleaving buffer size. It is therefore RECOMMENDED to set the
deinterleaving buffer size, in terms of number of bytes, equal to or
greater than the value of sprop-deint-buf-req MIME parameter.
7.2.2. Deinterleaving Process
There are two buffering states in the receiver: initial buffering and
buffering while playing. Initial buffering occurs when the RTP
session is initialized. After initial buffering, decoding and
playback are started, and the buffering-while-playing mode is used.
Wang, et al Expires January 14, 2009 [Page 35]
Internet-Draft RFC3984bis July 2008
Regardless of the buffering state, the receiver stores incoming NAL
units, in reception order, in the deinterleaving buffer as follows.
NAL units of aggregation packets are stored in the deinterleaving
buffer individually. The value of DON is calculated and stored for
each NAL unit.
The receiver operation is described below with the help of the
following functions and constants:
o Function AbsDON is specified in section 8.1.
o Function don_diff is specified in section 5.5.
o Constant N is the value of the OPTIONAL sprop-interleaving-depth
MIME type parameter (see section 8.1) incremented by 1.
Initial buffering lasts until one of the following conditions is
fulfilled:
o There are N or more VCL NAL units in the deinterleaving buffer.
o If sprop-max-don-diff is present, don_diff(m,n) is greater than
the value of sprop-max-don-diff, in which n corresponds to the NAL
unit having the greatest value of AbsDON among the received NAL
units and m corresponds to the NAL unit having the smallest value
of AbsDON among the received NAL units.
o Initial buffering has lasted for the duration equal to or greater
than the value of the OPTIONAL sprop-init-buf-time MIME parameter.
The NAL units to be removed from the deinterleaving buffer are
determined as follows:
o If the deinterleaving buffer contains at least N VCL NAL units,
NAL units are removed from the deinterleaving buffer and passed to
the decoder in the order specified below until the buffer contains
N-1 VCL NAL units.
o If sprop-max-don-diff is present, all NAL units m for which
don_diff(m,n) is greater than sprop-max-don-diff are removed from
the deinterleaving buffer and passed to the decoder in the order
specified below. Herein, n corresponds to the NAL unit having the
greatest value of AbsDON among the NAL units in the deinterleaving
buffer.
The order in which NAL units are passed to the decoder is specified
as follows:
Wang, et al Expires January 14, 2009 [Page 36]
Internet-Draft RFC3984bis July 2008
o Let PDON be a variable that is initialized to 0 at the beginning
of the RTP session.
o For each NAL unit associated with a value of DON, a DON distance
is calculated as follows. If the value of DON of the NAL unit is
larger than the value of PDON, the DON distance is equal to DON -
PDON. Otherwise, the DON distance is equal to 65535 - PDON + DON
+ 1.
o NAL units are delivered to the decoder in ascending order of DON
distance. If several NAL units share the same value of DON
distance, they can be passed to the decoder in any order.
o When a desired number of NAL units have been passed to the
decoder, the value of PDON is set to the value of DON for the last
NAL unit passed to the decoder.
7.3. Additional De-Packetization Guidelines
The following additional de-packetization rules may be used to
implement an operational H.264 de-packetizer:
o Intelligent RTP receivers (e.g., in gateways) may identify lost
coded slice data partitions A (DPAs). If a lost DPA is found, a
gateway may decide not to send the corresponding coded slice data
partitions B and C, as their information is meaningless for H.264
decoders. In this way a MANE can reduce network load by
discarding useless packets without parsing a complex bitstream.
o Intelligent RTP receivers (e.g., in gateways) may identify lost
FUs. If a lost FU is found, a gateway may decide not to send the
following FUs of the same fragmented NAL unit, as their
information is meaningless for H.264 decoders. In this way a MANE
can reduce network load by discarding useless packets without
parsing a complex bitstream.
o Intelligent receivers having to discard packets or NALUs should
first discard all packets/NALUs in which the value of the NRI
field of the NAL unit type octet is equal to 0. This will
minimize the impact on user experience and keep the reference
pictures intact. If more packets have to be discarded, then
packets with a numerically lower NRI value should be discarded
before packets with a numerically higher NRI value. However,
discarding any packets with an NRI bigger than 0 very likely leads
to decoder drift and SHOULD be avoided.
Wang, et al Expires January 14, 2009 [Page 37]
Internet-Draft RFC3984bis July 2008
8. Payload Format Parameters
This section specifies the parameters that MAY be used to select
optional features of the payload format and certain features of the
bitstream. The parameters are specified here as part of the MIME
subtype registration for the ITU-T H.264 | ISO/IEC 14496-10 codec. A
mapping of the parameters into the Session Description Protocol (SDP)
[5] is also provided for applications that use SDP. Equivalent
parameters could be defined elsewhere for use with control protocols
that do not use MIME or SDP.
Some parameters provide a receiver with the properties of the stream
that will be sent. The names of all these parameters start with
"sprop" for stream properties. Some of these "sprop" parameters are
limited by other payload or codec configuration parameters. For
example, the sprop-parameter-sets parameter is constrained by the
profile-level-id parameter. The media sender selects all "sprop"
parameters rather than the receiver. This uncommon characteristic of
the "sprop" parameters may not be compatible with some signaling
protocol concepts, in which case the use of these parameters SHOULD
be avoided.
8.1. MIME Registration
The MIME subtype for the ITU-T H.264 | ISO/IEC 14496-10 codec is
allocated from the IETF tree.
The receiver MUST ignore any unspecified parameter.
Media Type name: video
Media subtype name: H264
Required parameters: none
OPTIONAL parameters:
profile-level-id:
A base16 [6] (hexadecimal) representation of the following
three bytes in the sequence parameter set NAL unit specified
in [1]: 1) profile_idc, 2) a byte herein referred to as
profile-iop, composed of the values of constraint_set0_flag,
constraint_set1_flag,constraint_set2_flag, and
reserved_zero_5bits in bit-significance order, starting from
the most significant bit, and 3) level_idc. Note that
reserved_zero_5bits is required to be equal to 0 in [1], but
Wang, et al Expires January 14, 2009 [Page 38]
Internet-Draft RFC3984bis July 2008
other values for it may be specified in the future by ITU-T or
ISO/IEC.
If the profile-level-id parameter is used to indicate
properties of a NAL unit stream, it indicates the profile and
level that a decoder has to support in order to comply with
[1] when it decodes the stream. The profile-iop byte
indicates whether the NAL unit stream also obeys all
constraints of the indicated profiles as follows. If bit 7
(the most significant bit), bit 6, or bit 5 of profile-iop is
equal to 1, all constraints of the Baseline profile, the Main
profile, or the Extended profile, respectively, are obeyed in
the NAL unit stream. If the profile-level-id parameter is
used for capability exchange or session setup procedure, it
indicates the profile that the codec supports and the highest
level supported for the signaled profile. The profile-iop
byte indicates whether the codec has additional limitations
whereby only the common subset of the algorithmic features and
limitations of the profiles signaled with the profile-iop byte
and of the profile indicated by profile_idc is supported by
the codec. For example, if a codec supports only the common
subset of the coding tools of the Baseline profile and the
Main profile at level 2.1 and below, the profile-level-id
becomes 42E015, in which 42 stands for the Baseline profile,
E0 indicates that only the common subset for all profiles is
supported, and 15 indicates level 2.1.
Informative note: Capability exchange and session setup
procedures should provide means to list the capabilities
for each supported codec profile separately. For example,
the one-of-N codec selection procedure of the SDP
Offer/Answer model can be used (section 10.2 of [7]).
If no profile-level-id is present, the Baseline Profile
without additional constraints at Level 1 MUST be implied.
max-mbps, max-fs, max-cpb, max-dpb, and max-br:
These parameters MAY be used to signal the capabilities of a
receiver implementation. These parameters MUST NOT be used for
any other purpose. The profile-level-id parameter MUST be
present in the same receiver capability description that
contains any of these parameters. The level conveyed in the
value of the profile-level-id parameter MUST be such that the
receiver is fully capable of supporting. max-mbps, max-fs,
max-cpb, max-dpb, and max-br MAY be used to indicate
capabilities of the receiver that extend the required
capabilities of the signaled level, as specified below.
Wang, et al Expires January 14, 2009 [Page 39]
Internet-Draft RFC3984bis July 2008
When more than one parameter from the set (max-mbps, max-fs,
max-cpb, max-dpb, max-br) is present, the receiver MUST
support all signaled capabilities simultaneously. For
example, if both max-mbps and max-br are present, the signaled
level with the extension of both the frame rate and bit rate
is supported. That is, the receiver is able to decode NAL
unit streams in which the macroblock processing rate is up to
max-mbps (inclusive), the bit rate is up to max-br
(inclusive), the coded picture buffer size is derived as
specified in the semantics of the max-br parameter below, and
other properties comply with the level specified in the value
of the profile-level-id parameter.
If a receiver can support all the properties of level A, the
level specified in the value of the profile-level-id MUST be
level A (i.e. MUST NOT be lower than level A). In other
words, a sender or receiver MUST NOT signal values of max-
mbps, max-fs, max-cpb, max-dpb, and max-br that meet the
requirements of a higher level compared to the level specified
in the value of the profile-level-id parameter.
Informative note: When the OPTIONAL MIME type parameters
are used to signal the properties of a NAL unit stream,
max-mbps, max-fs, max-cpb, max-dpb, and max-br are not
present, and the value of profile-level-id must always be
such that the NAL unit stream complies fully with the
specified profile and level.
max-mbps: The value of max-mbps is an integer indicating the
maximum macroblock processing rate in units of macroblocks per
second. The max-mbps parameter signals that the receiver is
capable of decoding video at a higher rate than is required by
the signaled level conveyed in the value of the profile-level-
id parameter. When max-mbps is signaled, the receiver MUST be
able to decode NAL unit streams that conform to the signaled
level, with the exception that the MaxMBPS value in Table A-1
of [1] for the signaled level is replaced with the value of
max-mbps. The value of max-mbps MUST be greater than or equal
to the value of MaxMBPS for the level given in Table A-1 of
[1]. Senders MAY use this knowledge to send pictures of a
given size at a higher picture rate than is indicated in the
signaled level.
max-fs: The value of max-fs is an integer indicating the maximum
frame size in units of macroblocks. The max-fs parameter
signals that the receiver is capable of decoding larger
picture sizes than are required by the signaled level conveyed
Wang, et al Expires January 14, 2009 [Page 40]
Internet-Draft RFC3984bis July 2008
in the value of the profile-level-id parameter. When max-fs
is signaled, the receiver MUST be able to decode NAL unit
streams that conform to the signaled level, with the exception
that the MaxFS value in Table A-1 of [1] for the signaled
level is replaced with the value of max-fs. The value of max-
fs MUST be greater than or equal to the value of MaxFS for the
level given in Table A-1 of [1]. Senders MAY use this
knowledge to send larger pictures at a proportionally lower
frame rate than is indicated in the signaled level.
max-cpb: The value of max-cpb is an integer indicating the
maximum coded picture buffer size in units of 1000 bits for
the VCL HRD parameters (see A.3.1 item i of [1]) and in units
of 1200 bits for the NAL HRD parameters (see A.3.1 item j of
[1]). The max-cpb parameter signals that the receiver has
more memory than the minimum amount of coded picture buffer
memory required by the signaled level conveyed in the value of
the profile-level-id parameter. When max-cpb is signaled, the
receiver MUST be able to decode NAL unit streams that conform
to the signaled level, with the exception that the MaxCPB
value in Table A-1 of [1] for the signaled level is replaced
with the value of max-cpb. The value of max-cpb MUST be
greater than or equal to the value of MaxCPB for the level
given in Table A-1 of [1]. Senders MAY use this knowledge to
construct coded video streams with greater variation of bit
rate than can be achieved with the MaxCPB value in Table A-1
of [1].
Informative note: The coded picture buffer is used in the
hypothetical reference decoder (Annex C) of H.264. The use
of the hypothetical reference decoder is recommended in
H.264 encoders to verify that the produced bitstream
conforms to the standard and to control the output bitrate.
Thus, the coded picture buffer is conceptually independent
of any other potential buffers in the receiver, including
de-interleaving and de-jitter buffers. The coded picture
buffer need not be implemented in decoders as specified in
Annex C of H.264, but rather standard-compliant decoders
can have any buffering arrangements provided that they can
decode standard-compliant bitstreams. Thus, in practice,
the input buffer for video decoder can be integrated with
de-interleaving and de-jitter buffers of the receiver.
max-dpb: The value of max-dpb is an integer indicating the
maximum decoded picture buffer size in units of 1024 bytes.
The max-dpb parameter signals that the receiver has more
memory than the minimum amount of decoded picture buffer
Wang, et al Expires January 14, 2009 [Page 41]
Internet-Draft RFC3984bis July 2008
memory required by the signaled level conveyed in the value of
the profile-level-id parameter. When max-dpb is signaled, the
receiver MUST be able to decode NAL unit streams that conform
to the signaled level, with the exception that the MaxDPB
value in Table A-1 of [1] for the signaled level is replaced
with the value of max-dpb. Consequently, a receiver that
signals max-dpb MUST be capable of storing the following
number of decoded frames, complementary field pairs, and non-
paired fields in its decoded picture buffer:
Min(1024 * max-dpb / ( PicWidthInMbs * FrameHeightInMbs *
256 * ChromaFormatFactor ), 16)
PicWidthInMbs, FrameHeightInMbs, and ChromaFormatFactor are
defined in [1].
The value of max-dpb MUST be greater than or equal to the
value of MaxDPB for the level given in Table A-1 of [1].
Senders MAY use this knowledge to construct coded video
streams with improved compression.
Informative note: This parameter was added primarily to
complement a similar codepoint in the ITU-T Recommendation
H.245, so as to facilitate signaling gateway designs. The
decoded picture buffer stores reconstructed samples. There
is no relationship between the size of the decoded picture
buffer and the buffers used in RTP, especially de-
interleaving and de-jitter buffers.
max-br: The value of max-br is an integer indicating the maximum
video bit rate in units of 1000 bits per second for the VCL
HRD parameters (see A.3.1 item i of [1]) and in units of 1200
bits per second for the NAL HRD parameters (see A.3.1 item j
of [1]).
The max-br parameter signals that the video decoder of the
receiver is capable of decoding video at a higher bit rate
than is required by the signaled level conveyed in the value
of the profile-level-id parameter.
When max-br is signaled, the video codec of the receiver MUST
be able to decode NAL unit streams that conform to the
signaled level, conveyed in the profile-level-id parameter,
with the following exceptions in the limits specified by the
level:
Wang, et al Expires January 14, 2009 [Page 42]
Internet-Draft RFC3984bis July 2008
o The value of max-br replaces the MaxBR value of the signaled
level (in Table A-1 of [1]).
o When the max-cpb parameter is not present, the result of the
following formula replaces the value of MaxCPB in Table A-1
of [1]: (MaxCPB of the signaled level) * max-br / (MaxBR of
the signaled level).
For example, if a receiver signals capability for Level 1.2
with max-br equal to 1550, this indicates a maximum video
bitrate of 1550 kbits/sec for VCL HRD parameters, a maximum
video bitrate of 1860 kbits/sec for NAL HRD parameters, and a
CPB size of 4036458 bits (1550000 / 384000 * 1000 * 1000).
The value of max-br MUST be greater than or equal to the value
MaxBR for the signaled level given in Table A-1 of [1].
Senders MAY use this knowledge to send higher bitrate video as
allowed in the level definition of Annex A of H.264, to
achieve improved video quality.
Informative note: This parameter was added primarily to
complement a similar codepoint in the ITU-T Recommendation
H.245, so as to facilitate signaling gateway designs. No
assumption can be made from the value of this parameter
that the network is capable of handling such bit rates at
any given time. In particular, no conclusion can be drawn
that the signaled bit rate is possible under congestion
control constraints.
redundant-pic-cap:
This parameter signals the capabilities of a receiver
implementation. When equal to 0, the parameter indicates that
the receiver makes no attempt to use redundant coded pictures
to correct incorrectly decoded primary coded pictures. When
equal to 0, the receiver is not capable of using redundant
slices; therefore, a sender SHOULD avoid sending redundant
slices to save bandwidth. When equal to 1, the receiver is
capable of decoding any such redundant slice that covers a
corrupted area in a primary decoded picture (at least partly),
and therefore a sender MAY send redundant slices. When the
parameter is not present, then a value of 0 MUST be used for
redundant-pic-cap. When present, the value of redundant-pic-
cap MUST be either 0 or 1.
When the profile-level-id parameter is present in the same
capability signaling as the redundant-pic-cap parameter, and
Wang, et al Expires January 14, 2009 [Page 43]
Internet-Draft RFC3984bis July 2008
the profile indicated in profile-level-id is such that it
disallows the use of redundant coded pictures (e.g., Main
Profile), the value of redundant-pic-cap MUST be equal to 0.
When a receiver indicates redundant-pic-cap equal to 0, the
received stream SHOULD NOT contain redundant coded pictures.
Informative note: Even if redundant-pic-cap is equal to 0,
the decoder is able to ignore redundant codec pictures
provided that the decoder supports such a profile
(Baseline, Extended) in which redundant coded pictures are
allowed.
Informative note: Even if redundant-pic-cap is equal to 1,
the receiver may also choose other error concealment
strategies to replace or complement decoding of redundant
slices.
sprop-parameter-sets:
This parameter MAY be used to convey any sequence and picture
parameter set NAL units (herein referred to as the initial
parameter set NAL units) that MUST precede any other NAL units
in decoding order. The parameter MUST NOT be used to indicate
codec capability in any capability exchange procedure. The
value of the parameter is the base64 [6] representation of the
initial parameter set NAL units as specified in sections
7.3.2.1 and 7.3.2.2 of [1]. The parameter sets are conveyed
in decoding order, and no framing of the parameter set NAL
units takes place. A comma is used to separate any pair of
parameter sets in the list. Note that the number of bytes in
a parameter set NAL unit is typically less than 10, but a
picture parameter set NAL unit can contain several hundreds of
bytes.
Informative note: When several payload types are offered in
the SDP Offer/Answer model, each with its own sprop-
parameter-sets parameter, then the receiver cannot assume
that those parameter sets do not use conflicting storage
locations (i.e., identical values of parameter set
identifiers). Therefore, a receiver should double-buffer
all sprop-parameter-sets and make them available to the
decoder instance that decodes a certain payload type.
packetization-mode:
This parameter signals the properties of an RTP payload type
or the capabilities of a receiver implementation. Only a
single configuration point can be indicated; thus, when
capabilities to support more than one packetization-mode are
Wang, et al Expires January 14, 2009 [Page 44]
Internet-Draft RFC3984bis July 2008
declared, multiple configuration points (RTP payload types)
must be used.
When the value of packetization-mode is equal to 0 or
packetization-mode is not present, the single NAL mode, as
defined in section 6.2 of RFC 3984, MUST be used. This mode
is in use in standards using ITU-T Recommendation H.241 [15]
(see section 12.1). When the value of packetization-mode is
equal to 1, the non-interleaved mode, as defined in section
6.3 of RFC 3984, MUST be used. When the value of
packetization-mode is equal to 2, the interleaved mode, as
defined in section 6.4 of RFC 3984, MUST be used. The value
of packetization-mode MUST be an integer in the range of 0 to
2, inclusive.
sprop-interleaving-depth:
This parameter MUST NOT be present when packetization-mode is
not present or the value of packetization-mode is equal to 0
or 1. This parameter MUST be present when the value of
packetization-mode is equal to 2.
This parameter signals the properties of an RTP packet stream.
It specifies the maximum number of VCL NAL units that precede
any VCL NAL unit in the RTP packet stream in transmission
order and follow the VCL NAL unit in decoding order.
Consequently, it is guaranteed that receivers can reconstruct
NAL unit decoding order when the buffer size for NAL unit
decoding order recovery is at least the value of sprop-
interleaving-depth + 1 in terms of VCL NAL units.
The value of sprop-interleaving-depth MUST be an integer in
the range of 0 to 32767, inclusive.
sprop-deint-buf-req:
This parameter MUST NOT be present when packetization-mode is
not present or the value of packetization-mode is equal to 0
or 1. It MUST be present when the value of packetization-mode
is equal to 2.
sprop-deint-buf-req signals the required size of the
deinterleaving buffer for the RTP packet stream. The value of
the parameter MUST be greater than or equal to the maximum
buffer occupancy (in units of bytes) required in such a
deinterleaving buffer that is specified in section 7.2 of RFC
3984. It is guaranteed that receivers can perform the
deinterleaving of interleaved NAL units into NAL unit decoding
Wang, et al Expires January 14, 2009 [Page 45]
Internet-Draft RFC3984bis July 2008
order, when the deinterleaving buffer size is at least the
value of sprop-deint-buf-req in terms of bytes.
The value of sprop-deint-buf-req MUST be an integer in the
range of 0 to 4294967295, inclusive.
Informative note: sprop-deint-buf-req indicates the
required size of the deinterleaving buffer only. When
network jitter can occur, an appropriately sized jitter
buffer has to be provisioned for as well.
deint-buf-cap:
This parameter signals the capabilities of a receiver
implementation and indicates the amount of deinterleaving
buffer space in units of bytes that the receiver has available
for reconstructing the NAL unit decoding order. A receiver is
able to handle any stream for which the value of the sprop-
deint-buf-req parameter is smaller than or equal to this
parameter.
If the parameter is not present, then a value of 0 MUST be
used for deint-buf-cap. The value of deint-buf-cap MUST be an
integer in the range of 0 to 4294967295, inclusive.
Informative note: deint-buf-cap indicates the maximum
possible size of the deinterleaving buffer of the receiver
only. When network jitter can occur, an appropriately
sized jitter buffer has to be provisioned for as well.
sprop-init-buf-time:
This parameter MAY be used to signal the properties of an RTP
packet stream. The parameter MUST NOT be present, if the
value of packetization-mode is equal to 0 or 1.
The parameter signals the initial buffering time that a
receiver MUST wait before starting decoding to recover the NAL
unit decoding order from the transmission order. The
parameter is the maximum value of (decoding time of the NAL
unit - transmission time of a NAL unit), assuming reliable and
instantaneous transmission, the same timeline for transmission
and decoding, and that decoding starts when the first packet
arrives.
An example of specifying the value of sprop-init-buf-time
follows. A NAL unit stream is sent in the following
interleaved order, in which the value corresponds to the
Wang, et al Expires January 14, 2009 [Page 46]
Internet-Draft RFC3984bis July 2008
decoding time and the transmission order is from left to
right:
0 2 1 3 5 4 6 8 7 ...
Assuming a steady transmission rate of NAL units, the
transmission times are:
0 1 2 3 4 5 6 7 8 ...
Subtracting the decoding time from the transmission time
column-wise results in the following series:
0 -1 1 0 -1 1 0 -1 1 ...
Thus, in terms of intervals of NAL unit transmission times,
the value of sprop-init-buf-time in this example is 1. The
parameter is coded as a non-negative base10 integer
representation in clock ticks of a 90-kHz clock. If the
parameter is not present, then no initial buffering time value
is defined. Otherwise the value of sprop-init-buf-time MUST
be an integer in the range of 0 to 4294967295, inclusive.
In addition to the signaled sprop-init-buf-time, receivers
SHOULD take into account the transmission delay jitter
buffering, including buffering for the delay jitter caused by
mixers, translators, gateways, proxies, traffic-shapers, and
other network elements.
sprop-max-don-diff:
This parameter MAY be used to signal the properties of an RTP
packet stream. It MUST NOT be used to signal transmitter or
receiver or codec capabilities. The parameter MUST NOT be
present if the value of packetization-mode is equal to 0 or 1.
sprop-max-don-diff is an integer in the range of 0 to 32767,
inclusive. If sprop-max-don-diff is not present, the value of
the parameter is unspecified. sprop-max-don-diff is
calculated as follows:
sprop-max-don-diff = max{AbsDON(i) - AbsDON(j)},
for any i and any j>i,
