Sol-Attn2

Abstract

Attention dominates the cost of video diffusion. Block-sparse attention skips most key blocks, but whatever it skips is probability mass the softmax expected, and that error grows with sparsity. Sol-Attn2 keeps the skipped part as a compressed tail. A threshold router chooses key blocks with a dynamic budget per query block. Selected blocks are computed exactly by an FP8 tensor-core kernel. Every unselected block is compressed on the fly into a fixed set of 64 surrogate key/value tokens, which join the same softmax as the exact tokens. The compensation costs one extra 64-token block per query block, whatever the sequence length. Routing and the tail work training-free; FP8 attention with an exact backward makes the op both fast and trainable, and a short distillation (QAD) adapts the model to it.

The forward and backward kernels form FP8 probabilities with no exponential instruction (ExpCast), and the backward replays exactly the bytes the forward produced. On MiniMax H3, Sol-Attn2 works training-free and improves further after lightweight quantization-aware distillation (QAD). With the LoRA merged into the weights (Sol-H3 engine) it speeds up MiniMax H3 denoising of a 12-second video by 1.97× on one GB200, and by 1.78× in the HyperFlow pipeline with the LoRA kept as separate branches. On a 1080p, 15-second video (219K tokens) at matched 10% density it denoises 2.8× faster than dense, 1.39× faster than Sol-Attn 1.0 and 1.26× faster than cuDNN block-sparse attention.

Dense vs. Sol-Attn2

Same prompt and seed, 8-step HyperFlow sampling, 5-second clips. Left: dense attention (the teacher). Right: Sol-Attn2 in every denoising call after 400 steps of QAD, at about 11% of block pairs. Use Play both to watch the two in lockstep. Three prompts come from our held-out set; three are long multi-shot prompts from the VDN blog, used verbatim (the clips are 5 s, so later shots of those prompts may not appear).

Method

Starting training-free

Everything in this section works on a released model with no training. Video tokens are reordered into 4×4×4 spatio-temporal cubes, so every 64 consecutive tokens form a compact patch of the video. Attention is organised over 64×64 tiles: for a query block i and a key block j, the tile is either computed exactly, or the key block is folded into the compressed tail of query block i. Text and audio queries stay dense.

Attention as a sum over key blocks

Fix one query row q with softmax scale s. Each key token t has a score. Each key block j (token set B_j, n_j=|B_j| tokens) has a weight sum and a value-weighted sum:

The attention output is a ratio of two sums over blocks, and the log of the denominator is the row's log-sum-exp:

Block sparsity computes Z_j, A_j exactly for a selected set E and skips the rest. Sparse-only attention silently replaces the denominator by \sum_{j\in E}Z_j, so the skipped mass is not just lost: the remaining weights are inflated to fill it.

Count bias in block summaries

The router and the tail both need Z_j without visiting every token, so they summarise a block by its mean key \bar k_j and mean value \bar v_j. Jensen's inequality bounds the block mass by its summary:

Two consequences. The count term \log n_j must be kept: 64 tokens with the same score carry 64 times the mass of one, and without the term a partial block or a small pooled unit looks as heavy as a full block. The Jensen gap, the amount by which the summary underestimates the mass, grows with the spread of scores inside the block; pooling at a finer granularity shrinks it. The router therefore pools queries and keys into groups of 8 and uses the log-mean-exp of the 8×8 pooled scores s_{ab}, plus the count term, as the block mass:

Threshold router with a dynamic budget

Block j holds a share m_{ij}-L_i of row i's mass, with L_i=\log\sum_j e^{m_{ij}} computed from the same estimates. One global constant ε compares that share with the uniform share 1/N_B:

A block is kept when it holds more than e^{\varepsilon}\approx1.6 times the uniform share (ε = 0.47). The constant yields about 10% density, while each query block keeps as many blocks as its attention needs: peaked rows keep a few, diffuse rows keep more. The \log N_B term lets the same ε hold across video lengths.

Complementary approximation: shared normalisation

Instead of skipping the unselected blocks T_i, Sol-Attn2 estimates their sums (\hat Z_T,\hat A_T) and keeps them in the same normaliser:

The exact part and the tail share one denominator, so neither is renormalised on its own; in the kernel this is simply one more key block in the online softmax. With \hat Z_T=0 the formula reduces to sparse-only attention.

Building the tail. Per query block, every unselected unit u (an 8-token chunk) contributes its count, mean key and mean value. The units are split into 64 slots c, and each slot is weighted at an anchor query \bar q with the count-corrected mass from above:

The 64 surrogates (k_F^c, v_F^c, b_F^c) enter the kernel as one more key block, and every query row r reads them with its own query: s\,q_r^\top k_F^c + b_F^c = \log S_c + s\,(q_r-\bar q)^\top k_F^c. At the anchor this is the exact slot mass, count term included; away from it, it is a first-order expansion of \log Z. The tail costs one extra 64-token block per query block, whatever the video length, and in training it receives gradients like any other key block.

What changed from Sol-Attn 1.0

Sol-Attn 1.0 also selected blocks by score above the row mean + τ·std. That rule standardises each row's scores, which removes the row's temperature: peaked and diffuse rows get similar block counts, the opposite of what their mass needs (correlation between block count and blocks needed for 70% of the mass: −0.37, against +0.57 for the share threshold). The share rule keeps the temperature because it normalises by the row's own log-sum-exp.

FP8 kernels without exponentials

QK and PV run on FP8 E4M3 tensor cores. The probabilities P that feed PV must themselves be FP8. A standard kernel computes 2^{u} on the special-function unit and then converts it. We skip the exponential: read as an integer, an E4M3 byte is already a piecewise-linear base-2 logarithm, so one affine map of the score, rounded to the nearest integer, is the FP8 code of 2^{u}.

Spending FP8 headroom on the small probabilities

A softmax kernel subtracts the running row maximum, so every probability is at most 1. E4M3, however, reaches 448 ≈ 2^8.8, and with ExpCast every value below 2^−6.9 rounds to code 0. Anchored at the maximum, the format wastes almost nine octaves of headroom above 1 while flushing the long tail of small attention weights to zero, and with ~10% of tiles selected that tail carries real mass. We place the ExpCast basis a few octaves below the running maximum instead (offset 4): the largest probabilities become values up to 2⁴ = 16, still far below 448, and four more octaves of small weights survive. Numerator and denominator are both computed from the same bytes, so the shift cancels exactly in the normalisation.

Measured on the deployed route: the FP8 error of dense attention against FP64 drops from 9.1% to 4.7%, and step-0 validation loss improves by 2.8% (paired ratio 0.972 [0.950, 0.995], 24 of 32 samples better), for about 3% more kernel time because fewer tiles qualify for the fastest rounding path. Offsets 4–7 give similar accuracy; 4 is the cheapest.

Trainable: FP8 backward for QAD

Training-free Sol-Attn2 already beats the baselines, but sparsity and FP8 still leave an error that a few hundred steps of distillation can repair. Two pieces make that possible: a backward pass that differentiates the FP8 forward as it actually ran, and a light quantization-aware distillation (QAD) recipe.

Backward: replay the bytes, differentiate the shadow

A standard FP8 attention backward recomputes the probabilities from the saved row statistics, P = \exp(S - L), in high precision. That is the gradient of a softmax the forward never computed: our forward fed PV with ExpCast bytes, sticky per-group bases and its own row sums. Training against a different function lets the student exploit the mismatch. Our backward differentiates the forward as it actually ran:

Concretely, for every tile the backward re-runs the forward's own rounding sequence to regenerate the exact FP8 bytes of P, so dV = P^\top dO uses what the forward used. For the score gradient it differentiates the continuous curve through those bytes, holding quantisation scales and bases fixed:

The tail participates like any other key block: gradients of its 64 surrogates flow back through the slot compression to the unselected keys and values. The backward runs as two deterministic passes:

The replayed probabilities are bit-identical to the forward's on every gate capture. On the deployed route the full forward + backward takes 42.3 ms (5 s) and 153.8 ms (12 s), against 43.4 / 197.4 ms for FastVideo's BF16 block-sparse Triton kernels and 96.6 / 539.8 ms for dense BF16 SDPA.

QAD: a light distillation

Our quantization-aware distillation (QAD) is deliberately light. The teacher is the same 8-step HyperFlow model running dense BF16 attention, live, in the same job. The student is that model with Sol-Attn2 (routing, tail and FP8 attention with its exact backward) in every attention call, plus a rank-256 LoRA as the only trainable weights. Both see the same noisy latents at exactly the eight timesteps the sampler uses, and the student matches the teacher's video and audio predictions under a relative MSE. The clean latents are clips generated by MiniMax H3 itself, so no real video data is needed. 400 steps take about 48 B200 GPU-hours.

Speed

Attention kernels

Real Sol-Attn2 routes from MiniMax H3 (56 heads; 12-second and 5-second videos, 83k and 35k tokens) on one GB200, all measured in one job. Complete call starts from BF16 Q/K/V and includes everything: quantisation, routing and the tail. Kernel only times the attention kernel on prepared operands. Sparse baselines execute the same 64×64 tiles as our route.

Sparse baselines run behind our router (BF16 routing, index build and layout conversion), so the complete call compares whole attention ops on the same route. cuDNN FA is PyTorch SDPA with the cuDNN backend. All rows from one HSG job (7892561) plus the complete-call job 7893191: same q/k/v and route, CUDA-event median of 40. In kernel-only mode the cuDNN kernels execute exactly our selected tiles; cuDNN's SM100 FP8 path needs a fixed number of key blocks per query block, so it gets the mean count (equal work).

Why ExpCast matters. Both kernels below run exactly the same selected tiles; Nsight Compute reports how busy each hardware unit of the SM is, as a share of its peak.

The special-function unit (MUFU) is the small per-SM unit that evaluates transcendental functions such as exp2, at a fraction of the rate of ordinary arithmetic. cuDNN issues one exp2 per score, which keeps that unit busy 59% of the time; it becomes the bottleneck, and the tensor cores wait for P. ExpCast turns the same work into a fused multiply-add and an integer conversion on the regular ALU and FMA pipes, so the tensor cores stay busy 71% of the time.

Where the time goes in one call

One Sol-Attn2 call is five steps. The figure marks which ones already run fused and which still run as separate kernels, with measured times for one 12-second call (56 heads, 83k tokens).

Why it saves compute. Dense attention multiplies every query block with every key block: N_B^2 tiles of QK and PV. Counted in tile-equivalents, Sol-Attn2 runs three much smaller pieces:

The router works on 8-token means, so it does 1/64 of the dense score work (an 8×8 instead of a 64×64 product per tile) in pooled precision. At density ρ ≈ 0.1 the exact part is a tenth of the dense tiles, and those run on FP8 tensor cores at twice the BF16 rate without any exponential instruction. The tail adds one tile per query block, so its cost grows linearly with length while everything else grows quadratically. Against Sol-Attn 1.0 the same tiles get cheaper (FP8 instead of BF16, no exp2 bottleneck) and the router gets more accurate for about the same cost.

Speed across configurations

Denoiser time, dense versus Sol-Attn2 on the same pipeline, sampler, GPU count and video; Sol-Attn2 runs in every denoising call.

One GB200. HyperFlow: MiniMax H3 with the HyperFlow 8-step adapter applied as LoRA branches, 736×1312, HSG job 7894980. Sol-H3: NVlabs/Sana sol-engine with the FastH3 4-step adapter merged into the weights, 1344×768, HSG job 7895280. Medians of three warm requests.

Against Sol-Attn 1.0, each row below times dense, Sol-Attn 1.0 and Sol-Attn2 in the same job, so all three share one GPU, prompt and seed.

Denoiser time on one GB200, medians of three warm requests, all calls sparse; speedups in parentheses are against dense. Sol-Attn 1.0 is its BF16 kernel with the mean + τ·std threshold plus the diagonal band (τ calibrated to about 10% density), on the same 4×4×4 cube layout, layers and dense text/audio query rows as Sol-Attn2. HSG jobs 7918238 (HyperFlow) and 7918503 (Sol-H3). Per attention call the gap is 2.4× (47.0 vs 19.4 ms at 12 s); end to end it shrinks to 1.1–1.3× because attention is only part of each call.

Sol-H3 ships its own Sol-Attn 1.0 backends: sol_bsa (its default: Sol-Attn 1.0 routing, cuDNN block-sparse attention for the selected blocks, then the mean-atom residual) and sol (its CuTe kernel). Both run here with Sol-H3's own settings (raster order, prefix sink, diagonal threshold) on the same layers and call schedule as Sol-Attn2. At Sol-H3's default τ = 1.0 they keep about twice our density, so the table also lists τ = 1.6, where the routed density matches ours.

One GB200, medians of three warm requests, all calls sparse, blocks 3–46. Each speedup divides by the dense run in the same process (dense 12 s: 46.0–47.5 s). Density for Sol-H3 is its routed fraction before the prefix sink is added (the sink adds 0.01–0.03); for Sol-Attn2 it is selected over eligible blocks outside the dense text/audio rows. cuDNN frontend 29106622 (Sol-H3's pin), block-sparse attention only. HSG jobs 7919812 and 7920385. Sol-H3's quality at τ = 1.6 was not evaluated.

Long video: 1080p, 15 seconds

At 1088×1920 and 15 seconds the packed sequence is 219,524 tokens (3,431 KV blocks of 64), and dense attention takes 87% of the denoiser. Every sparse arm below is set to the same video density, 0.100, and runs on the Sol-H3 engine with all of its optimizations except the attention backend: LoRA merged, kernel fusions and AdaLN precompute on. Sol-Attn2 denoises 2.8× faster than dense, 1.39× faster than Sol-Attn 1.0 and 1.26× faster than Sol-H3's cuDNN block-sparse backend.

One GB200 per run, FastH3 4-step adapter, one prompt, seed 1000; each arm runs in a fresh process, medians of two timed requests after warm-up. Video density is the same for every arm: selected over eligible video (query block, KV block) pairs, band included, text/audio rows and columns excluded, measured from each arm's own route on every sparse call; each arm's threshold was solved to hit 0.100 (Sol-Attn2 ε 0.2127, Sol-Attn 1.0 τ 1.3047, sol_bsa τ 1.6243). Text/audio sink off. Per transformer call, attention takes 12–13 s with Sol-Attn2, 17.6 s with sol_bsa, 19.7 s with Sol-Attn 1.0 and about 46 s dense. Supporting this length needed one change to the tail compression kernel (64 lanes per slot, inference only); its output is bitwise unchanged up to 2,048 KV blocks. HSG jobs 7930955 and 7930958.

Multi-GPU runs use the Sol-H3 engine with the LoRA merged and its fusions on (norms, modulation, SwiGLU), with Ulysses sequence parallelism. The first table compares dense and Sol-Attn2 at each GPU count; the second shows how each scales from one GPU.

Denoiser time, 1344×768, FastH3 4-step adapter, HSG jobs 7895280, 7895601, 7895602. Speedups in the second table divide the one-GPU denoiser time by the multi-GPU time of the same arm. Both arms use the engine's BF16 all-to-all, 15–16% of denoiser time on 4 and 8 GPUs, which is why Sol-Attn2 scales slightly below dense: its compute shrinks while the communication stays the same.

Video generation

Panel (b) and the table below break down the HyperFlow runs: the full MiniMax H3 pipeline on one GB200 with the deployed Sol-Attn2 op (quantisation, routing, tail and FP8 attention), 8-step HyperFlow with all eight calls sparse, as deployed after QAD. Median of three warm requests on the same prompt and seed.

The gain grows with video length because attention grows quadratically while everything else grows linearly. In one sparse call, attention in the 44 sparse blocks is 33% of the dense call at 5 s and 50% at 12 s; Sol-Attn2 cuts it from 986 to 251 ms (5 s) and from 5337 to 844 ms (12 s). The rest of the call (projections, FFN, the six dense blocks) stays the same and caps the per-call speedup.

The HyperFlow pipeline applies LoRA as separate branches with no kernel fusion. In the Sol-H3 engine the LoRA is merged and norms, modulation and SwiGLU are fused, so everything outside attention is faster and the same Sol-Attn2 op removes more of the total: per call at 12 s, attention drops from 7879 to 2194 ms and the rest from 2895 to 2643 ms (with fusions on, the 12 s denoiser speedup is 2.22×; on 4 and 8 GPUs with Ulysses it is 2.12× and 2.09×).

After Sol-Attn2, attention is no longer the largest cost. The table splits the 4-step denoiser on the Sol-H3 engine (merged LoRA) by module. At 12 s, attention falls from 62% to 31% of the time; the QKV and output projections plus the FFN are now 64%.

CUDA events around each block's attention module and FFN in one extra request per arm (HSG job 7919812, one GB200, fusions off). Attention op = the attention kernel call (for Sol-Attn2: quantisation, routing, tail and FP8 core). QKV + out proj. = the attention module minus the op (projections, qk-norm, RoPE). FFN = up projection, SwiGLU and down projection. Other = norms, modulation, residuals, embeddings, refiner and output head. The FFN and projection rows differ by 5–10% between arms on the same GPU; treat that as run-to-run variation.

Quality

Training-free

Released HyperFlow adapter, no training. 64 held-out prompts, 8-step sampling with the first two calls dense, PSNR against the dense teacher's video, paired over the same prompts and seeds.

After QAD

Recipe in Training.

Validation loss on 32 fixed samples:

"Tie" means the 95% interval contains 1. After QAD we judge by validation loss and samples, not PSNR.

What's next

Tail. An equal-cost tail with 8 anchors × 8 surrogates in the same single extra tile (each anchor serves its 8 query rows) gains +0.13 to +0.16 dB training-free PSNR over the deployed 64-slot form at the same density; it is now in QAD. Grouping surrogates by key content instead of token order makes the tail independent of token order at no quality cost.

Training signal for the tail. QAD trains the model only through its final output, so the tail never gets a direct signal. We are testing a local loss that pulls each layer's Sol-Attn2 output toward its own dense attention, and a SAS-style learnable selector whose normalised gate also weights the tail, so every skipped block gets its own gradient.

One fused routing kernel. Routing and tail construction are 5.1 of the 19.4 ms of one 12 s call. Fusing them into one scan, and the tail tile into the new FP8 attention kernel, attacks exactly that remainder.

Acknowledgements

We thank the teams and open projects that Sol-Attn2 builds on.

• Sol-EngineInference stack that hosts Sol-Attn2 and its end-to-end acceleration.
• Sol-AttnOn-the-fly attention sparsification that Sol-Attn2 extends.
• SoL-PiCoding agent used throughout the development of the kernels and experiments.
• MiniMaxMiniMax-H3 open weights and code, the base model of all experiments.
• Humanize + KDAAgentic kernel-design workflow for the FP8 kernels.
• VDNVideo DeltaNet; its long multi-shot prompts are used in our samples.
• VC-AttentionLow-bit attention with value smoothing and softmax casting; our dense FP8 baseline.
• HyperFlow8-step distilled MiniMax-H3 adapter used for sampling and QAD.

Citations

If Sol-Attn2 helps your work, please cite this project page together with Sol-Engine and Sol-Attn.

@misc{solattn2_2026,
  title        = {Sol-Attn2: Sparse Attention with Complementary Approximation},
  year         = {2026},
  howpublished = {\url{https://hp-l33-sol-attn2.static.hf.space/}},
  note         = {Project page}
}

@misc{li2026solvideoinferenceengine,
  title         = {Sol Video Inference Engine: Agent-Native Full-Stack Acceleration Framework for Efficient Video Generation},
  author        = {Yitong Li and Junsong Chen and Haopeng Li and Haozhe Liu and Jincheng Yu and Ligeng Zhu and Ping Luo and Song Han and Enze Xie},
  year          = {2026},
  eprint        = {2606.23743},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CV},
  url           = {https://arxiv.org/abs/2606.23743}
}

@misc{li2026solattn,
  title         = {Sol-Attn: Accelerating Video Generation Inference via On-the-Fly Attention Sparsification},
  author        = {Haopeng Li and Yitong Li and Junsong Chen and Tian Ye and Haozhe Liu and Jincheng Yu and Duomin Wang and Ruihua Zhang and Zeke Xie and Enze Xie and Song Han},
  year          = {2026},
  eprint        = {2607.24027},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CV},
  url           = {https://arxiv.org/abs/2607.24027}
}