Native MP4 Video Contract v1

Scope

AuraJS native hosts can load and play codec-backed .mp4 files through the existing aura.video surface.

This contract is intentionally narrow:

native desktop only
visual playback into a runtime texture
existing play / pause / stop / seek / getTexture / getInfo flow
existing JS helpers such as createVideoBillboardSurface(...) and createMediaPresentationController(...)
automatic audio demux from the MP4 container (if an audio track is present)
separate authored audio assets layered through the native media-presentation contract (still supported)

This contract does not claim:

browser/web .mp4 parity
sample-accurate audio/video lockstep timing
subtitle rendering
editor-authored cinematic timelines
universal codec/container support beyond .mp4

Runtime Environment

MP4 decode is handled in-process via the embedded video codec library. No external tools are required for video playback.

Supported Codecs

The in-process decoder supports the following video codecs inside .mp4 containers:

H.264 (AVC) -- most common desktop/web video codec, full support
H.265 (HEVC) -- higher compression, full support
VP9 -- royalty-free, supported
MPEG-4 Part 2 -- legacy codec, supported for backward compatibility

Container format is MP4 (ISOBMFF), auto-detected by file extension.

Supported Audio Codecs

When an audio track is present in the MP4 container, it is automatically decoded and played back through the engine's miniaudio mixer. Supported audio codecs:

AAC -- most common MP4 audio codec, full support
MP3 -- supported inside MP4 containers
Opus -- royalty-free, supported
FLAC -- lossless, supported
PCM -- uncompressed, supported

Audio is resampled to the engine format (44100 Hz, stereo, F32 interleaved) regardless of the source format.

Audio Sync Strategy

Audio and video run free on the real-time clock. Both start on play() and pause/stop together. On seek, audio is stopped and restarts on the next play() call. This matches the existing media-presentation contract where JS authors own fine-grained timing control.

Supported Behavior

Native .mp4 playback supports:

aura.video.load('assets/intro.mp4')
explicit type: 'mp4' or extension-driven auto-detection
texture-backed playback for draw3d.billboard(...)
getInfo(handle).sourceKind === 'mp4'
use through higher-level helpers that accept a normal source path

Minimal example:

const handle = aura.video.load('assets/intro.mp4', { type: 'mp4', looping: true });
const textureHandle = aura.video.getTexture(handle);
aura.video.play(handle);

Packaging Contract

For native builds and npm-first package/install flows:

.mp4 assets are preserved
asset packaging classifies .mp4 as video
the public docs still describe frame sequences or sprite sheets as the portable fallback
higher-level separate-audio presentation semantics are documented in docs/native-media-presentation-contract-v1.md

Non-Goals

Current native .mp4 support does not guarantee:

sample-accurate audio/video sync (free-running is sufficient for cutscenes)
browser support
.webm, .mov, .mkv, or .avi
zero-copy or streaming decode internals as part of the public contract
hardware-accelerated audio or video decode

AURA

JSGG