Riffusion

Introduction

Riffusion offers a practical and straightforward way to convert text into short audio clips by deriving spectrogram images and turning them into actual sound. It is a useful tool for quickly creating musical ideas and background clips without needing deep knowledge of audio production.

What is Riffusion?

Riffusion is an open-source project and web service that showcases a model for audio generation based on image-generation techniques built on diffusion models. The core technical idea behind Riffusion is to turn the problem of audio generation into the problem of generating a mel-spectrogram image.

A diffusion model (based on Stable Diffusion or similar models) is trained or fine-tuned on spectrogram images. A text prompt is used as input to a text-to-image model to produce a new spectrogram image. This image is then converted into audio signals using spectrogram inversion algorithms, such as inverse STFT with phase-retrieval algorithms like Griffin-Lim, or using a vocoder like MelGAN or HiFi-GAN if available.

The practical result follows this pipeline: text → spectrogram generator (Diffusion) → 512×512 spectrogram image → converting the spectrogram back into an audio waveform (WAV/OGG). The project is available as a repository on GitHub for local running, and it has illustrative web interfaces (Gradio-based demos) and cloud services that provide a ready-to-use generation experience.

Key Features

• Text to Spectrogram (Text → Spectrogram): Uses a model derived from Stable Diffusion techniques to generate spectrogram images based on textual prompts.
• Spectrogram to Audio: The processing pipeline includes algorithms to reconstruct the time-domain signal from the spectrogram, typically via inverse-STFT with Griffin-Lim or via a trained vocoder that converts a mel-spectrogram into a high-quality audio waveform.
• Spectrogram Editing (spectrogram inpainting): The ability to draw or shade parts of the spectrogram in the image interface to regenerate targeted audio segments (example: removing or changing the intensity of a specific melodic phrase).
• Input an existing spectrogram image (image→audio): Upload an existing spectrogram image to obtain an audio file, enabling the conversion of visual examples or visible snippets into sound.
• Fine-grained control parameters: Settings such as seed to reproduce results, guidance scale (strength of text guidance), number of diffusion steps, spectrogram resolution, and sample rate to control the clip’s length and quality.
• Available as an open-source project and a web service: A version that can be run locally via GitHub in addition to experimental web interfaces for quick use.

Practical Use Cases

• Text-to-Music (text to a music clip):

  You write a description such as “lo-fi hip hop, mellow piano loop with vinyl crackle, 90 BPM” and the system produces a spectrogram that reflects the music’s frequency composition. Practical benefit: you can try several text combinations within minutes to get raw musical ideas. Practical experience recommends using guided descriptions (instruments, mood, BPM, effects) to obtain predictable results.

• Spectrogram Inpainting (spectrogram editing):

  You can select a time–frequency region in the spectrogram image (for example, the high-frequency portion during the fifth second) and regenerate it with a different text prompt, allowing you to modify timbre or intensity without regenerating the entire clip — useful for tweaking drum hits or a specific melodic line.

• Seed and Reproducibility (seed and repetition):

  By setting a seed, you can reproduce the same spectrogram repeatedly, a