Spatial audio support is currently available using Facebook Audio 360 on Windows desktop, UWP and Android. On Windows, only Windows 10 and above is supported and the Media Foundation video API must be selected. On Android the ExoPlayer video API must be selected. The video files must be using a MKV file container and audio must be using the Opus codec encoded with Facebook Audio 360 tools.
The best way to encode the video is to use the FB360 Encoder tool which comes as part of the FB360 Spatial Workstation. Set Output Format to “FB360 Matroska (Spatial Workstaton 8 channel)” and then set your video and audio source files and encode your video. This should create a MKV file with 10 channels of Opus audio.
The settings are located under the Audio section of the MediaPlayer component. The
“Head Transform” field must be set to the transform that represents the player's head so that rotation and positional changes affect the audio rendering. Usually this is the main camera.
“Enable Focus” can be enabled when a specific region of audio in the 360 field needs to be given focus. The rest of the audio has its volume reduced.
Next the Facebook Audio 360 support must be enabled for each platform that needs it via the
“Platform Specific” panel. Currently it is only available on Windows desktop and Android.
The “Channel Mode” must be set to the channel encoding mode used when creating the video. Currently this can not be determined automatically. The default is “TBE_8_2” which means 8 channels of hybrid ambisonics and 2 channels of head-locked stereo audio.
// Set the head transform for audio mediaPlayer.AudioHeadTransform = Camera.main; // Set the MediaPlayer for spatial audio on Android - ExoPlayer API only mediaPlayer.PlatformOptionsAndroid.videoApi = Android.VideoApi.ExoPlayer; mediaPlayer.PlatformOptionsAndroid.audioOutput = Android.AudioOutput.FacebookAudio360; mediaPlayer.PlatformOptionsAndroid.audio360ChannelMode = Audio360ChannelMode.TBE_8_2; // Set the MediaPlayer for spatial audio on Windows - Media Foundation API only mediaPlayer.PlatformOptionsWindows.videoApi = Windows.VideoApi.MediaFoundation; mediaPlayer.PlatformOptionsWindows.audioOutput = Windows.AudioOutput.FacebookAudio360; mediaPlayer.PlatformOptionsWindows.audio360ChannelMode = Audio360ChannelMode.TBE_8_2;
Encoding with Facebook 360 Spatial WorkStation
More information on encoding etc can be found on the Facebook Audio 360 website at: https://facebook360.fb.com/spatial-workstation/
Alternative steps for encoding manually
Create a WAV file with the audio format they need (Eg 9 channels ambisonics with 2 channels of head-locked audio will require a 11 channel WAV file with the 2 head-locked channels at the end)
Use Opus tools, as described here to convert the WAV file to Opus: https://opus-codec.org/downloads/ https://facebookincubator.github.io/facebook-360-spatial-workstation/Documentation/SDK/Audio360_SDK_GettingStarted.html#encoding-opus-files
Use ffmpeg to mux this opus file into the video container (ensure that the video file doesn’t have any audio first):
ffmpeg -i audio.opus video.mp4 -c:a copy -c:v copy audio_video.mkv
In AVPro Video specify the required channel map
Converting existing ambisonic videos
It is also possible to convert existing ambisonic videos so they are compatible. For example if you have an existing MP4 file with 4-channel 1st order ambisonic audio, then it is possible to convert this into the above format (MKV container with Opus audio) using a tool like FFMPEG. Simply put the following command in a .BAT file and then drag your MP4 into the batch file:
ffmpeg -y -i input.mp4 -c:v copy -acodec libopus -mapping_family 255 output.mkv
This should then generate an MKV file that you can play with AVPro Video. All that remains is to set the channel mapping in the MediaPlayer component to AMBIX_4.