H.264 stream structure

What you call “H.264 stream” is probably not the real raw H.264 stream – it’s probably “H.264 byte stream”. It’s important to use the proper terminology to discuss stream validity or multiplexing matters.

SODB

Let’s start from the SODB (String Of Data Bits). This is the real raw H.264 stream. The syntax is specified ISO/IEC 14496-10 in a form of bit string syntax.

image

RBSP

SODB is not very convenient because it’s a bit stream, not a byte stream. It means the number of bits for one syntax may not be byte-aligned and therefore difficult to process. This is why the standard introduces RBSP (Raw Byte Sequence Payload). RBSP stores SODB in a byte stream so that the first bit of each syntax is always aligned at the first bit of a byte.

image

NAL Unit

The problem of RBSP is that RBSP may contain any byte pattern. Since it doesn’t have special synchronization byte sequence, to find the synchronization point in RBSP (For example, to find the first byte of IDR picture), you may have to parse every single bit syntax from the beginning of the file.

NAL Unit is another wrapper layer to prevent certain byte pattern from occurring in the stream. When RBSP has any of 0x000000, 0x000001, 0x000002, and 0x000003, they are converted to 0x00000300, 0x00000301, 0x00000302, and 0x00000303 respectively in NAL unit. Therefore, we can use any of 0x000000, 0x000001, 0x000002, or 0x000003 as a special synchronization byte sequence.

NAL Unit also adds one byte header to indicate the type of the NAL Unit.

image

Byte Stream Format

Although NAL Unit allows to put a synchronization byte sequence, it doesn’t have any yet. The standard defines another wrapper to add three or four bytes synchronization byte pattern: the Byte Stream Format. The byte stream format puts a synchronization byte sequence (0x000001 or 0x00000001) before every NAL Unit. The byte stream format is used as the elementary stream of H.264 in transport stream.

image

The original intention of the standard is to let applications to pick a suitable format. For example, MPEG Transport Stream uses byte stream format to allow decoder to find the  NAL Unit easily. Other container format such as AVI where the length of the header is stored in a packet would use NAL Unit or even RBSP to reduce the overhead of synchronization bytes. However, in real world, almost all storage and delivery formats use byte stream format. It’s due to practical reasons such as to simplify the re-wrapping process.

Nonetheless, to avoid any confusion, it’s better idea to use the term “H.264 Byte Stream Format” instead of “H.264 raw stream” or “H.264 stream”.

Advertisements

About Moto

Engineer who likes coding
This entry was posted in Video. Bookmark the permalink.

8 Responses to H.264 stream structure

  1. Andrey says:

    Thanks a lot for explanation!

  2. ks says:

    Strong article, everything is clear written! thank you

  3. theambient says:

    Good but this is the most easy to understand piece of H.264 bitstream. Did you write another h.264 stream related posts? After brief overview i did not find so.

  4. Pingback: Video/Audio Codec | canlinflexray

  5. koraxkorakos says:

    Your article is the concise description I have been looking for. Thanks!

  6. quyendv says:

    How to know the size of NAL Unit?

    • Moto says:

      If you have a byte stream, you can scan the start codes and find the number of bytes between two successive start codes. Don’t forget to subtract the size of start code itself (It can be 4 or 5 bytes).

      If you have MP4 (ISO/IEC 14496-12) stream, “mdat” payload carries NAL units in length-data format, i.e. [LengthOfNalUnit1][NalUnit1][LengthOfNalUnit2][NalUnit2]… You can find the length of a NAL unit by looking at the length field. The field is coded in big-endian format. The size of the field is signaled in AVCDecoderConfigurationRecord::lengthSizeMinusOne syntax and it can be 1, 2, or 4 bytes. See ISO/IEC 14496-15 section 5.2.3 for the detail.

      Hope it helps!

  7. Pranab says:

    Nice article.
    Can we paly H.264 Byte Stream using MSE? or is there any other way around?

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s