Stereo2Spatial: Convert Stereo Music Tracks to Spatialized Binaural Mixes [P]
Our take
The emergence of AI-powered tools for spatial audio creation marks a fascinating shift in how we experience music, and the release of Stereo2Spatial by francislabounty represents a compelling leap forward. The desire to transform existing stereo tracks into immersive binaural mixes is increasingly relevant, particularly as spatial audio formats gain traction across streaming platforms and consumer devices. As discussed in The real AI race may no longer be at the frontier, the growing demand for open models and greater control over AI technology is driving innovation across various domains, and this project exemplifies that trend. The initial approach using a flow-matching diffusion model within a VAE’s latent space, while encountering challenges, laid crucial groundwork that ultimately paved the way for a more robust waveform-based solution. This iterative process, documented with impressive transparency, highlights the realities of AI model development – the need for experimentation, adaptation, and persistence in overcoming technical hurdles.
The journey detailed by the creator, from latent space modeling to waveform processing and the eventual stabilization achieved through amplitude lifting, is a valuable lesson in the practical application of AI. The implementation of amplitude lifting, inspired by the WavFlow paper, effectively addressed training instability, a common issue when working with raw audio waveforms. This is a significant contribution, demonstrating a deep understanding of the underlying technical complexities. Further, the project’s commitment to openness, releasing both the waveform and latent models under the Apache 2.0 license, alongside a user-friendly Windows desktop application, fosters accessibility and encourages community engagement. This aligns with the broader movement towards democratizing AI tools, as explored in Thinking Machines open sources first multimodal language model, Inkling, focused on low cost and 'resistance to censorship'. The choice to release everything, including the codebase and a playlist of generated tracks, allows others to readily experiment and build upon this work, accelerating progress in spatial audio processing.
The fact that Stereo2Spatial effectively leverages existing technologies—building upon the EAR-VAE and drawing inspiration from WavFlow—underscores the power of collaborative innovation within the AI space. It’s not about inventing entirely new concepts, but about intelligently combining and adapting existing approaches to solve specific problems. The decision to focus initially on binaural output, rather than a full 7.1.4 mix, is a pragmatic one, proving the core concept before scaling up to more complex configurations. The use of mix-style conditioning, available in the waveform version, adds another layer of control and creative potential for users. The dedicated case studies provide valuable insights into the model's capabilities and limitations, further enhancing its utility for both researchers and audio enthusiasts. We can see the parallels of this approach in A Gentle Introduction to Autoencoders & Latent Space, which clarifies how latent space manipulation can unlock new creative possibilities.
Looking ahead, the potential for Stereo2Spatial to evolve into a full 7.1.4 spatializer, as the creator envisions, is particularly exciting. The codebase and data already developed provide a strong foundation for future expansion. Furthermore, the availability of open-source spatial audio tools like this one empowers independent creators and smaller studios, leveling the playing field in a space traditionally dominated by expensive commercial solutions. The question now becomes: how will the increasing accessibility of AI-powered spatial audio tools reshape the landscape of music production and consumption, and what new creative avenues will emerge as a result?
I have released a model that I have been working on for ~6 months off and on. I've been enjoying listening to spatial music, but there is a lot of music out there with no real quality spatial mix. So I decided to make a model to convert stereo to spatial.
I started by making a flow-matching diffusion model that operated in the latent space of a separately trained VAE (EAR-VAE). I used the VAE to encode the stereo input in to 1 latent and then separately on each channel of the 7.1.4 output. This design is similar to a paper I found about another stereo -> spatial project (ImmersiveFlow (they have code links in the paper but they 404 for me so I made the codebase from scratch)). One key addition I made over the ImmersiveFlow paper was a way for the model to carry state (memory tokens) across windows to enable stable long context generations.
Since the VAE was out of distribution with the output (was trained to encode\decode stereo tracks directly, not individual channels and certainly not individual channels of a 7.1.4 mix), the results hit real quality bottlenecks. But it showed that the mapping was possible. Rather than stick to modeling in latent space, I decided to pivot to modeling raw waveforms. This ended up fixing all of the quality issues I was seeing with the latent version, but at the cost of more training compute and instability.
The waveform version of the model would train to around 60K-80K steps with loss going down like normal, validation generations looking good\improving, and then it would quickly become unstable and the loss would shoot up. I tried direct waveforms, scaling the waveforms by a multiplier, aggressive grad clipping, lower learning rates, and all of those experiments failed the same way. Luckily, I came across a recent paper called WavFlow. The authors of this project also had issues with modeling raw waveforms with flow-matching diffusion, and they solved it by using amplitude lifting (scale each audio track to an rms of 0.33 then multiply by 3) (the WavFlow paper used a clip of 1.0 before the scale which leads to a model space of -3 - 3, but I used a clip of 4.0 which leads to a model space of -12 - 12 (from my testing this led to better binaural outputs)). Once I implemented amplitude lifting like the paper, the training stability issues vanished.
The waveform model was trained on 7,669 tracks for ~20 days on 2x A6000 gpus
- 10 days stage one, effective batch of 16, 10- 18- 26- and 34-second training sequences
- 10 days stage two, effective batch of 16, 122-second training sequences
- There is optional mix-style conditioning for controllable outputs (waveform version only)
- Direct binaural output (waveform training is more compute expensive than latent training so binaural proves the idea\theory and the same codebase and data can be used in the future to train a 7.1.4 version (once I have access to the compute))
I also made a Windows desktop app for inferencing with the model and consuming the results.
Everything has been released apache 2.0.
Huggingface link for waveform model: https://huggingface.co/francislabounty/stereo2spatial-v2-binaural
Huggingface link for latent model: https://huggingface.co/francislabounty/stereo2spatial-v1
Github repo for training\inferencing: https://github.com/francislabountyjr/stereo2spatial
Github repo for the Windows app: https://github.com/francislabountyjr/stereo2spatial-app
App Homepage: https://stereo2spatial.francislabounty.com
Case study for the waveform model: https://francislabounty.com/blog/stereo2spatial-v2
Case study for the latent model: https://francislabounty.com/blog/stereo2spatial
Playlist of generations made from the model (will be updated with more tracks over time): https://www.youtube.com/playlist?list=PLQ-HHjPijrAg
If you have any questions\etc. leave a comment and I will do my best to get them answered!
Disclaimer: I know AI written posts are generally frowned upon so this post was 100% written by me (mistakes and all). But the case studies and READMEs from the above links were made with AI assistance.
[link] [comments]
Read on the original site
Open the publisher's page for the full experience