Apple Immersive Video

Written By Ben Gunsberger

What is Apple Immersive Video?

Apple Immersive Video, introduced with the Apple Vision Pro, refers to a method of recording and displaying video that gives a sense of depth and space by utilising stereoscopic techniques. This type of video captures and presents content in a way that mimics how human eyes perceive the real world, offering a 180-degree field of view that can make the viewer feel as if they are physically present in the video. Essentially, it uses two slightly offset images to create a 3D effect, enhancing the realism of the video experience

Apple’s Immersive Video Streams

Apple's Immersive Video streams are designed to deliver high-quality, immersive content that takes full advantage of the capabilities of the Apple Vision Pro headset. These streams have specific technical characteristics that enable an immersive viewing experience.

Fom Mike Swanson’s excellent research.

  • Resolution: 4320×4320

  • Frame Rate: 90fps

  • Colour: HDR10

  • Codec: MV-HEVC

  • FOV180° H/V

  • Max Bitrate: 50Mbps

  • Projection: It’s Complicated

Gear List

After extensive research and testing, here’s a list of gear (and daily rental costs in AUD) for capturing high-quality Apple Immersive Video. This setup can produce professional-grade 180-degree stereoscopic content compatible with the Apple Vision Pro:

Workflow

A well-defined workflow is crucial when using prosumer gear, as it helps maximize the potential of your equipment and compensate for any limitations compared to professional-grade tools.

This workflow can delivery an excellent quality 2x4096x4096x59.94FPS Immersive Video (what Apple calls “8K”)

  1. Shoot 8192 x 4096 59.94 fps Cinema RAW Light LT

  2. Use EOS VR Utility to Convert to Equirectangular SBS Canon Log3 ProRes 4444 (1gb/sec)

  3. Sharpen/Denoise in Topaz Video AI & export as ProRes 4444

  4. Import into Premiere Pro

  5. Set Premiere colour to Rec2100 PQ / 203 Nits

  6. Set Laptop Display Profile to HDR Video

  7. Edit/Grade

  8. Export Rec2100 PQ / 203 Nits Canon ProRes 4444

  9. Convert to Spatial video using Spatial Video Tool

  10. Playback in Spatial Gen AVP App

Spatial Video Tool Code Example

After exporting your edited video from Premiere Pro, the next crucial step is converting it into Apple's immersive video format using Mike Swanson’s Spatial Video Tool, which prepares the content for optimal viewing on devices like the Apple Vision Pro.

spatial make -i input.mpv -f sbs --cdist 63.0 \\
  --hfov 180.0 --hadjust 0.0 \\
  --projection halfEquirect --hero right \\
  --primary right --quality 0.4 -o output.mov

Spatial Video Quality/Bitrate Settings

During my testing with the Spatial Video Tool, I observed the following bitrates corresponding to different quality settings:

  • 0.6: 255Mbps

  • 0.5: 78Mbps

  • 0.4: 27Mbps

Exporting For YouTube

When uploading immersive video content to YouTube, it's crucial to use specific formats and settings to ensure proper playback and metadata recognition. Here's an overview of the key requirements:

  1. Export in HEVC in HDR

2. Include the VR metadata

Note: After uploading, it takes some time for the 4K version to become available.

Distribution

Distribution for Apple Immersive Video is still complicated, due to the infancy of the format and the large file sizes involved.

For small-scale distribution, the following works:

  1. Create your MV-HEVC as above

  2. Upload to Dropbox, GDrive etc

  3. Share the link

And then the recipient needs to:

  1. Install the ‘spatialgen’ app for the Apple Vision Pro

  2. Download the shared file

  3. Open the local file in spatialgen

  4. Select ‘180’ and play

Camera Preflight List

A comprehensive checklist is crucial, especially when working with unfamiliar equipment, as it helps ensure all essential steps are followed, minimizing the risk of errors and oversights that could compromise the quality of the work.

  • Check Focus (Still Mode)

  • Clean Lens

  • Tighten Tripod

  • Level Tripod

  • Connect Battery

  • Check Tripod Legs in Shot

  • Get Out of Frame!

Lessons

After my initial foray into Apple Immersive Video production, I’m eager to refine my workflow for even better results in future projects. Here are some key lessons learned:

  • Need Sturdier Tripod & Head

  • Need Better Levelling Head/Plate

  • Use Histogram/Zebra to check exposure

  • Lower ISO - Noise is BAD

  • Wind Noise is BAD

  • Darks seem lighter in VR, so grade them down

  • 500gb card is not enough

Still To Work Out

While I made significant progress in establishing an Apple Immersive Video workflow, there are still numerous aspects that I need to confirmed or optimise to ensure the highest quality output:

  • Match Export Colour to Edit Better

  • Better Sharpen/Denoise

  • Optimise Bitrate/Quality

  • Confirm Best Colour Gamut for RAW conversion

  • Confirm Best Colour Settings for Premiere

  • Confirm Best Colour Settings for Export

  • Confirm Best Colour Profile for Laptop

  • Is 60fps needed for landscapes?

  • Better way to check focus/exposure

  • What is the minimum battery requirement?

  • Streaming support

  • Better Monitoring

  • Why does the final Spatial encoding bitrate increase over time?

  • How to share at full resolution/dynamic range on YouTube

Other Links

Here are some additional resources and references that provide valuable insights into Apple Immersive Video production and related technologies:

Ben Gunsberger
Previous

AI, Spatial Experiences, and the Creative Industry
Next

Apple Vision Pro Feedback