Meta has introduced SAM Audio, the first unified multimodal model for audio separation, enabling users to isolate sounds using text, visual, or time-based prompts. Powered by the new Perception Encoder Audiovisual engine, the model simplifies audio editing and opens new possibilities for creators and researchers alike.
Meta has taken a decisive step toward reshaping how people interact with sound by introducing SAM Audio, the first unified multimodal model designed specifically for audio separation. Building on the success of the Segment Anything Model (SAM), which changed the way people think about segmenting objects in images and videos, SAM Audio extends the same philosophy to sound. The ambition is clear: make isolating, cleaning, and understanding audio as intuitive as clicking on an object or typing a simple phrase.
For years, audio separation has remained a fragmented and technically demanding space. Musicians, podcasters, filmmakers, and everyday users have relied on specialised tools, each built for narrow tasks like noise reduction, vocal isolation, or instrument separation. These tools often require technical expertise, manual tweaking, or rigid workflows. Meta’s introduction of SAM Audio signals a shift away from this complexity toward a unified, prompt-based system that mirrors how people naturally think about sound.
At the centre of this breakthrough is a simple but powerful idea: people should be able to isolate any sound using intuitive prompts, whether through text, visuals, or time-based cues. Want to remove a barking dog from a podcast? Mark the time span where it appears. Want to isolate a guitar solo from a band performance? Click on the guitar in the video. Want to extract a voice or filter out traffic noise? Type it in. SAM Audio brings all of these interactions together in a single model, delivering state-of-the-art performance across a wide range of real-world scenarios.
Driving SAM Audio’s capabilities is Perception Encoder Audiovisual, or PE-AV, the technical engine that functions as the model’s sensory backbone. Built on the open-source Perception Encoder that Meta shared earlier this year, PE-AV integrates audio and visual understanding into a shared representation. Meta describes PE-AV as “the ears,” enabling SAM Audio, “the brain,” to reason about and segment sound with remarkable precision. Together, they allow the system to understand not just what sound is present, but where it comes from and how it relates to what is happening visually.
The implications of this pairing are far-reaching. In a video recording of a live band, for example, a user can simply click on the guitarist to isolate the instrument’s audio track. In a busy street interview, background noise can be filtered out by typing a short text prompt like “traffic.” For longer recordings, such as podcasts or lectures, industry-first span prompting allows users to mark entire time segments and resolve recurring audio issues in one action. This combination of modalities gives users unprecedented control without requiring deep technical knowledge.
Meta’s release of SAM Audio is not just about introducing a new model, but about opening an ecosystem. Alongside SAM Audio and PE-AV, the company is also releasing SAM Audio-Bench, the first in-the-wild benchmark for audio separation, and SAM Audio Judge, the first automatic judge model designed to evaluate audio separation quality. These tools address a long-standing challenge in the field: the lack of standardised, real-world evaluation methods. By providing both a benchmark and an automated judge, Meta aims to accelerate research, improve transparency, and encourage the development of better audio systems across the industry.
The technical foundation of SAM Audio reflects the scale of the challenge it is tackling. At its core, the model uses a generative framework built on a flow-matching diffusion transformer. This architecture takes an audio mixture along with one or more prompts, encodes them into a shared multimodal space, and generates both the target audio and the residual sound. To support this, Meta developed a comprehensive data engine capable of producing large-scale, high-quality training data. This engine combines advanced audio mixing techniques, automated multimodal prompt generation, and a robust pseudo-labeling pipeline, ensuring the model can perform reliably in complex, real-world conditions.
Yet for all its technical sophistication, Meta’s emphasis remains on accessibility. SAM Audio is designed to lower the barrier to entry for audio creation and editing, much like SAM did for computer vision. The goal is not just to serve audio professionals, but to empower everyday creators—people making videos for social platforms, students recording lectures, or families capturing moments on their phones. By aligning interaction methods with natural human behaviour, Meta hopes to make sound segmentation feel less like engineering and more like expression.
This philosophy is embodied in the Segment Anything Playground, where SAM Audio is now available for anyone to try. The platform allows users to experiment with audio and video assets provided by Meta or upload their own content. Alongside SAM Audio, users can also explore SAM 3 and SAM 3D, bringing together Meta’s most recent advances in multimodal understanding. The Playground functions as both a demonstration space and an invitation—an open door for creators, researchers, and developers to see what these models can do and imagine what might come next.
From Meta’s perspective, SAM Audio is a foundational step toward the next generation of creative media tools. Audio clean-up, background noise removal, and advanced editing are only the beginning. As models like SAM Audio mature, they could enable entirely new workflows for storytelling, collaboration, and accessibility. For example, isolating specific sounds could help people with hearing impairments focus on voices, or allow journalists to quickly clean up field recordings without specialised software.
There is also a broader strategic narrative at play. By releasing SAM Audio and PE-AV to the community, along with detailed research papers, Meta is reinforcing its commitment to open research and shared progress. The company is positioning itself not just as a product builder, but as a platform for innovation—one that provides the building blocks for others to create new applications, tools, and experiences.
Just as the original Segment Anything Model redefined what was possible in computer vision, SAM Audio aims to do the same for sound. It reframes audio separation from a technical task into an intuitive interaction, grounded in the ways people naturally perceive and describe what they hear. In doing so, Meta is not just introducing a new model, but proposing a new mental model for working with audio.
As SAM Audio becomes available starting today, Meta is eager to see what people create, explore, and improve with it. For the first time, segmenting sound is no longer confined to specialists or single-purpose tools. It is becoming something anyone can do—by clicking, typing, or simply marking a moment in time. In that shift lies the true significance of SAM Audio: not just hearing sound differently, but giving people the power to shape it with ease.
Discover more from Creative Brands
Subscribe to get the latest posts sent to your email.
