DMAD — 4-Step Audio-Video Generation Demo

Prompt (Scene & Soundscape) 1344×768 · 24fps

Example Prompts Click to apply

Connected directly to the official ZeroGPU inference engine (ZhengmingYu/dmad). ZeroGPU executes the 4 forward passes on NVIDIA RTX Pro 6000 hardware.

Generated Video & Synchronized Stereo Audio

Ready

Format: 1344×768 H.264 Audio: Stereo AAC (32kHz) Length: ~5.2s (124 frames)

Download MP4

Sampling Rule

Re-noise Multistep

Denoise to x₀, then re-noise to next σ with fresh noise

Sigma Shift

Video 12.0 / Audio 2.0

Aligned exponential grid for modality packing

Critic Overhead

0% (Inference Free)

Adversarial discriminator is discarded post-training

How DMAD Works

Distribution Matching as Adversarial Distillation for Fast Visual Generation (Yu et al., arXiv:2610.02188)

Distillation Pipeline Overview

1. Teacher Model

50-step MiniMax-H3 (33B omni-modal DiT). Processes joint packed sequence tokens of umT5 text tokens, 3D video latents, and 1D audio latents with 3D rotary embeddings.

2. Adversarial Critic

A critic discriminator matches the distribution of student samples directly to teacher generations. Student gradients represent the exact difference of log-likelihood ratios without ODE inversion.

3. Few-Step Student

Rank-128 LoRA applied to 312 DiT modules. Evaluated in only 4 or 8 denoising steps using a shifted linear sigma grid with re-noise stochasticity. Zero inference penalty.

Model / Setup	Steps	Resolution	AVGen-Bench Score	Sampling Time	License
MiniMax-H3 Teacher	50 steps	1344×768	Baseline	~600 s	MiniMax H3 License
DMAD H3 4-step (lora_critic)	4 steps	1344×768	Competitive with Teacher	~120 s (12.5× faster)	MiniMax H3 License
DMAD H3 8-step (full_critic)	8 steps	1344×768	Exceeds 4-step on audio-video	~210 s	MiniMax H3 License
DMAD Wan2.1 1.3B & 14B	4 steps	480p (81 frames)	High Fidelity T2V	Fast	Apache 2.0
DMAD SDXL UNet	1 & 4 steps	1024×1024	Zero guidance needed	< 1 s	Open RAIL++-M

ComfyUI Workflows & Local Deployment

Run the DMAD students on consumer hardware (8 GB VRAM and up) using ComfyUI or Python diffusers.

8 GB GPU Optimized

5s Video Workflow (4 Steps)

Peaks under 7.7 GiB VRAM with ComfyUI dynamic memory offload. Produces 124 frames (5.2s) at 1344×768.

Download JSON ➔

16 GB+ GPU

15s Full Podcast Workflow

Extended 15-second generation with full stereo audio track. Fits in 14 GiB GPU memory with --low-vram.

Download JSON ➔

8-Step High Quality

8-Step 15s Wok Workflow

Uses the newly released 8step_full_critic LoRA checkpoint for highest visual texture and crisp audio.

Download JSON ➔

Python Inference: DMAD SDXL (4 Steps)

import torch
from diffusers import DiffusionPipeline, LCMScheduler, UNet2DConditionModel
from huggingface_hub import hf_hub_download

base_model_id = "stabilityai/stable-diffusion-xl-base-1.0"
unet = UNet2DConditionModel.from_config(base_model_id, subfolder="unet").to("cuda", torch.float16)
unet.load_state_dict(torch.load(hf_hub_download("ZhengmingYu/DMAD", "sdxl/dmad_sdxl_4step_unet_fp16.bin"), map_location="cuda"))

pipe = DiffusionPipeline.from_pretrained(base_model_id, unet=unet, torch_dtype=torch.float16, variant="fp16").to("cuda")
pipe.scheduler = LCMScheduler.from_config(pipe.scheduler.config)

# 4-step sampling:
image = pipe(prompt="a cinematic shot of a futuristic neon city", num_inference_steps=4, guidance_scale=0, timesteps=[999, 749, 499, 249]).images[0]
image.save("dmad_output.png")

Instant Joint Audio & Video Generation

Pre-Computed DMAD Showcases

How DMAD Works

Distillation Pipeline Overview

ComfyUI Workflows & Local Deployment

Python Inference: DMAD SDXL (4 Steps)