Skip to content

Remove Unity resampling#271

Open
MaxHeimbrock wants to merge 2 commits intomax/audiostream-replay-on-device-changefrom
max/audiostream-resample-rust
Open

Remove Unity resampling#271
MaxHeimbrock wants to merge 2 commits intomax/audiostream-replay-on-device-changefrom
max/audiostream-resample-rust

Conversation

@MaxHeimbrock
Copy link
Copy Markdown
Contributor

@MaxHeimbrock MaxHeimbrock commented May 5, 2026

Summary

Remove second resampling in Unity, by configuring the Rust audio stream with correct (channels, sampleRate) to begin with. The problem is, that in Unity you don't know the channel number of each audio source until it runs, so we need to delay the initialization of the native audio stream until audio is read in Unity. The same lazy-create path is reused to recreate the native audio stream when Unity's delivered format changes mid-stream — for example after a system audio device switch.

Changes

  • Drop C# RemixAndResample per-frame path. OnAudioStreamEvent now just memcpys the inbound AudioFrame into the ring buffer — no FFI roundtrip on the OS-audio-scheduled callback thread. Rust's NativeAudioStream does the rate/channel conversion internally, once, with its existing pipeline.
  • Lazily create the FFI native stream from the first OnAudioRead. AudioSettings.GetConfiguration().speakerMode disagrees with what OnAudioFilterRead actually delivers when the AudioSource is routed differently (per-source mono routing, spatializers, mixer chains, headphone profiles) — observed on macOS Editor + Sony headphones (system Stereo, source mono → low-pitched playback). The first callback is the authoritative source for (channels, sampleRate); we post a one-shot CreateOrRecreateFfiStream(...) to the main thread via FfiClient._context, build the NewAudioStreamRequest with the observed values, and only then subscribe to AudioStreamEventReceived.
  • Recreate the native stream when Unity's delivered format changes. A single gate in OnAudioRead covers both first-create and runtime mismatch (channels != _ffiNumChannels || sampleRate != _ffiSampleRate). On mismatch we post the same CreateOrRecreateFfiStream to the main thread; the new stream is built outside the lock, then under the lock we swap _handle, update the _ffi* trackers, clear the buffer, reset priming, and clear the _pendingFfiRequest flag. Old handle is disposed AFTER the swap so its Rust task exits cleanly. OnAudioStreamEvent drops frames while _pendingFfiRequest is set so old-format bytes don't land in the new buffer; the existing handle-id filter catches any in-flight stragglers from the old stream after the swap. This pairs with Recover AudioStream when system audio output device changes #274 (re-attach AudioProbe and re-Play() the AudioSource on device change) so audio resumes seamlessly when the user plugs/unplugs headphones, switches Bluetooth devices, etc.

Files

  • Runtime/Scripts/AudioStream.cs_resampler field gone; Handle is now _handle behind a property; CreateOrRecreateFfiStream added; _ffiNumChannels/_ffiSampleRate/_pendingFfiRequest track FFI-side configuration; constructor only sets up AudioSource/AudioProbe/pause hook.
  • Tests/EditMode/MediaStreamLifetimeTests.cs — assertions updated for the renamed/removed members.

Profiler

Before change, after 5 minutes and multiple audio sources: Total Audio CPU reaches ~100%
Screenshot 2026-05-06 at 14 47 09


With change, after 5 minutes and multiple audio sources: Total Audio CPU stays below ~10%
Screenshot 2026-05-06 at 14 46 48

Comment thread Runtime/Scripts/AudioStream.cs Outdated
@MaxHeimbrock MaxHeimbrock force-pushed the max/audiostream-resample-rust branch from 90d0b04 to 3b570fc Compare May 6, 2026 06:38
@MaxHeimbrock MaxHeimbrock changed the title Max/audiostream resample rust Remove Unity resampling by configuring native audio source correctly May 6, 2026
@MaxHeimbrock MaxHeimbrock force-pushed the max/audiostream-resample-rust branch from 3b570fc to 337c769 Compare May 6, 2026 12:25
@MaxHeimbrock MaxHeimbrock marked this pull request as ready for review May 6, 2026 12:54
MaxHeimbrock and others added 2 commits May 7, 2026 10:59
Configuring the native stream from AudioSettings.speakerMode at construction
time can disagree with what OnAudioFilterRead actually delivers (per-source
mono routing, spatializers, mixer chains, headphone profiles), and the C#
RemixAndResample path it covered for is heavy work on the OS-audio-scheduled
FFI callback thread. Defer the NewAudioStreamRequest until the first audio
callback so we configure with Unity's authoritative (channels, sampleRate)
and let Rust's NativeAudioStream do the rate/channel conversion once.
OnAudioStreamEvent becomes a memcpy into the ring buffer.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
When the system output device switches mid-call (e.g. headphones unplugged),
Unity tears down its audio engine and OnAudioFilterRead resumes with a
(channels, sampleRate) that may differ from what Rust was configured for.
Detect the mismatch in OnAudioRead, post a CreateOrRecreateFfiStream to the
main thread, and gate OnAudioStreamEvent on a _pendingFfiRequest flag so
we drop in-flight frames at the old format. After the new stream is built,
swap _handle, clear the buffer, reset priming, and dispose the old handle
so its Rust task exits. The handle-id filter remains as a second line of
defense for old-stream stragglers arriving during the brief overlap.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@MaxHeimbrock MaxHeimbrock force-pushed the max/audiostream-resample-rust branch from 337c769 to 433ff53 Compare May 7, 2026 09:06
@MaxHeimbrock MaxHeimbrock changed the title Remove Unity resampling by configuring native audio source correctly Remove Unity resampling May 7, 2026
@MaxHeimbrock MaxHeimbrock changed the base branch from main to max/audiostream-replay-on-device-change May 7, 2026 09:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant