Asuka-FM: 206M Dual-Stream SPRINT DiT (Open-Domain Foundation)

An ultra-efficient, 206M-parameter Dual-Stream Diffusion Transformer (DiT) trained from scratch on a single consumer GPU (1x NVIDIA RTX 4090 24GB) in **149.0 GPU hours ($49.17)**—or 161.2 hours ($53.20) including one-time offline VAE pre-encoding—using Rectified Flow, SPRINT token-dropping, Calibrated Continuous RoPE, and Latent-to-Pixel Adaptation (DiP).

Evaluated strictly as an open-domain foundation model without benchmark-specific fine-tuning (avoiding BLIP3o-60K overfitting), Asuka-FM achieves 0.588 GenEval / 78.96 DPG-Bench at 256px and 0.609 GenEval / 78.51 DPG-Bench at 512px, outperforming SDXL on both metrics at a fraction of the compute.

Full Technical Report: This repository hosts the complete open-domain model weights (all training checkpoints across 256px, 512px, experimental 1MP, and Pixel DiT adaptation). For architectural derivations, hardware profiling (RTX 4090 vs. A40), and full benchmark ablations, please refer to our paper:
Asuka-FM: A System-Architectural Co-Design Framework for Efficient Text-to-Image Pretraining on Consumer Hardware (arXiv:2611.26437).
Looking for the anime domain-specialized checkpoints? See the companion anime release and Zenn technical deep-dive (Japanese).

  • Developed by: aipracticecafe / UC3M & DTU
  • Repository: deeplearningcafe/asuka-fm
  • Architecture: AdaLN-Free Dual-Stream DiT (SPRINT, SwiGLU, Sandwich RMSNorm)
  • Parameters: ~206.12M (200.03M Backbone + 6.09M Text Adapter) + 10.09M PDH (Pixel Mode)
  • Objective: Conditional Flow Matching ($v$-prediction, Logit-Normal sampling)
  • License: OpenRAIL++

Asuka-FM Open-Domain Showcase Figure: Zero-shot open-domain generations from the 206M Asuka-512 foundation model trained from scratch on a single RTX 4090 in 149.0 GPU hours.


Key Architecture & Features

1. SPRINT Dual-Stream Transformer Backbone

The base architecture is based on the results from (i1: A Simple and Fully Open Recipe for Strong Text-to-Image Models)[https://arxiv.org/abs/2606.11289]

  • Minimalist Conditioning: Uses prepended time tokens rather than heavy per-block AdaLN modulation, reducing parameter overhead.
  • Sandwich Normalization & SwiGLU: RMSNorm placed before and after attention and FFN operations ensures training stability in bfloat16.
  • SPRINT Acceleration: Employs structured group-wise 2x2 patch dropping (75% drop ratio during early pretraining), cutting compute demands by half while retaining global semantic context.

2. HDM Continuous Aspect-Normalized RoPE (Camera Control)

Based on the Patch Diffusion: Faster and More Data-Efficient implementation proposed in the HDM paper. Early experiments with discrete integer position IDs caused severe multi-head replication when deviating from trained aspect ratios. We resolve this by adopting continuous aspect-normalized coordinate mapping which was the original idea proposed in HDM:

rH=HW,rW=WHr_H = \sqrt{\frac{H}{W}}, \quad r_W = \sqrt{\frac{W}{H}}

ypos=(ycenterH)⋅(2rH)−rH,xpos=(xcenterW)⋅(2rW)−rWy_{\text{pos}} = \left(\frac{y_{\text{center}}}{H}\right) \cdot (2 r_H) - r_H, \quad x_{\text{pos}} = \left(\frac{x_{\text{center}}}{W}\right) \cdot (2 r_W) - r_W

Manipulating $(x_{\text{pos}}, y_{\text{pos}})$ directly controls the visual framing at inference time:

  • X-Shift / Y-Shift: Translates the virtual camera viewport horizontally or vertically.
  • Zoom: Scales spatial coordinates around the focal center.

3. Components

  • Text Encoder: LiquidAI/LFM2.5-230M (unidirectional causal LM hidden states passed via a 2-layer Transformer Text Adapter).
  • Autoencoder: kaiyuyue/FLUX.2-dev-vae (C=32, f=8, patch size p=2, effective image token size 16x16).

4. Curated Open-Domain Corpus (625K) & Multi-Captioning

  • Open-Domain General Corpus (625K): Following empirical findings from i1, training on balanced mixtures with 100k–400k high-quality samples per source yields competitive generative performance without massive web scraping. We curate an open-domain dataset of 625,000 samples by extracting exactly 125,000 samples across five sources including synthetic and real images (all $\min(H, W) \ge 512\text{ px}$):
    1. FLUX-Reason-6M (complex spatial composition)
    2. GPT-Edit-Simpler (paired instructional clarity)
    3. Midjourney-v6-Recap (artistic lighting and textures)
    4. Pexels (natural photography and human anatomy)
    5. TextAtlas5M (text-rich scenes)
  • VLM Multi-Captioning: We maintain five synthetic captions per image generated via Qwen3-VL-30B-A3B from the i1 dataset. At each epoch, a caption is sampled dynamically, providing continuous semantic regularization that prevents text memorization. Training context is partitioned into three discrete tiers ([77, 152, 227] tokens), reducing padding overhead by up to 66% and triggering TorchInductor recompilation exactly twice during warmup.
  • Pixel SFT Corpus: Real-data pixel fine-tuning (Stage 3) uses 396,635 curated high-aesthetic samples ($\min(H, W) \ge 1024\text{ px}$) from i1.

Training Recipe & Multi-Stage Progression (1x RTX 4090, 24GB)

Pretraining advances across two core latent resolution tiers ($256^2 \to 512^2$) using shifted square cropping, in-RAM caching at $256^2$, and async network streaming at $512^2$, followed by experimental 1MP ($1024^2$) scaling and 512px Latent-to-Pixel adaptation.

  • Optimizer: 8-bit AdamW (bitsandbytes), $\beta=(0.9, 0.95)$, WD=0.01
  • Effective Batch Size: 256 (Latent Stages 1–2) / 128 (Pixel Adaptation)
  • Precision: Mixed Precision (torch.bfloat16 with torch.compile)

Latent Pretraining Progression (Stages 1 – 2 + Experimental 1MP)

Stage / Phase Res. Epochs Drop ($r$) Samples Throughput MFU (Theo/Act) Compute Time Cost ($0.33/h)
Offline VAE Encode $256^2$ — — 0.625M 45.2 img/s — — 3.8 h $1.27
1A: SPRINT Sparse $256^2$ 0–74 0.75 46.88M 282.6 img/s 74.4% / 86.8% 23.74 EFLOPs 47.0 h $15.51
1B/1C: Dense + WSD $256^2$ 75–99 0.00 12.50M 172.5 img/s 62.9% / 73.4% 8.73 EFLOPs 21.0 h $6.93
Offline VAE Encode $512^2$ — — 0.625M 20.6 img/s — — 8.4 h $2.78
2A: High-Res Sparse $512^2$ 100–124 0.75 15.63M 108.5 img/s 58.5% / 68.3% 16.24 EFLOPs 40.0 h $13.20
2B/2C: Dense + WSD $512^2$ 125–139 0.00 9.38M 64.8 img/s 61.0% / 71.2% 16.94 EFLOPs 41.0 h $13.53
DiT Subtotal (1+2) — 140 — 84.38M — 64.8% / 75.6% 65.65 EFLOPs 149.0 h $49.17
Grand Total (w/ VAE) — — — — — — 65.65 EFLOPs 161.2 h $53.20

Latent-to-Pixel Adaptation (512px DiP Head, Zero-VAE Decode)

Stage / Phase Patch ($P$) Drop ($r$) Target Samples Throughput MFU (Act) Time Cost
Phase 0: Offline Synth (A40) — — — 0.10M 1.85 img/s — 15.0 h $6.00
Stage 1: PDH Warmup (Frozen DiT) 16 0.75 $v$-pred 0.50M 61.8 img/s 72.8% 2.25 h $0.74
Stage 2A: Joint SPRINT (Synth) 16 0.75 $v$-pred 2.50M 57.7 img/s 67.8% 12.0 h $3.98
Stage 2B: Dense Refine (Synth) 16 0.00 $v$-pred 1.50M 44.4 img/s 69.8% 9.4 h $3.10
Stage 3: Real-Data SFT (396k) 16 0.00 $v$-pred 5.16M 42.1 img/s 66.2% 34.0 h $11.22
Pixel DiT Total (Stages 1–3) 16 — — 9.66M 46.5 img/s 67.5% 57.65 h $19.04
Large Patch (ps32-adapt16) 32 0.00 $v$-pred 9.92M 38.3 img/s 63.3% 72.0 h $23.76

Quantitative Benchmarks (GenEval & DPG-Bench, No BLIP3o Fine-Tuning)

Model / Checkpoint Res. Patch ($P$) GenEval ↑ DPG-Bench ↑ Single Two-Obj Color-Attr
Asuka-256 (Dense Final, Ep 99) $256^2$ 16 0.588 78.96 0.950 0.750 0.418
Asuka-512 (75% Sparse, Ep 124) $512^2$ 16 0.480 68.40 0.963 0.639 0.163
Asuka-512 (Dense Final, Ep 139) $512^2$ 16 0.609 78.51 0.950 0.811 0.450
Pixel DiT ($v$-pred Final, Step 70k) $512^2$ 16 0.568 75.31 0.950 0.780 0.420
Pixel DiT ($x_0$-pred Abl., Step 70k) $512^2$ 16 0.545 76.32 0.950 0.699 0.348
Pixel DiT Large Patch (ps32, 150k) $512^2$ 32 0.504 69.87 0.938 0.576 0.250

Checkpoint Guide

All intermediate and final training checkpoints are published in this repository for full reproducibility. The primary milestones and ablation checkpoints are summarized below:

Latent Foundation Checkpoints (Requires FLUX.2 VAE)

Checkpoint Path Stage / Phase Description & Recommended Usage
epoch_74_step_183075 Stage 1A Final Last 256px SPRINT 75%-sparse checkpoint.
epoch_99_step_246541 Stage 1 Final Final 256px dense checkpoint (0% drop). GenEval 0.588 / DPG 78.96.
epoch_124_step_311210 Stage 2A Final Last 512px SPRINT 75%-sparse checkpoint.
epoch_139_step_391917 Stage 2 Final Recommended default for 512px inference (0% drop). GenEval 0.609 / DPG 78.51.
epoch_147_step_420000 Stage 3A (Exp.) Experimental (not in paper): Last 1MP ($1024^2$) checkpoint at 75% token drop.
epoch_149_step_430000 Stage 3B (Exp.) Experimental (not in paper): Last 1MP ($1024^2$) dense fine-tuned checkpoint (0% drop).

Pixel-Space Adaptation Checkpoints (Zero-VAE Decode via 10M PDH)

(Note: Both Stage 2B synthetic refinement and Stage 3 real-data SFT checkpoints reside under the stage2/ directory on disk.)

Checkpoint Path Objective Patch ($P$) Description & Recommended Usage
512px/pixel_adaptation/stage2/epoch_37_step_30000 $v$-pred 16 Last $v$-pred checkpoint trained on 100k synthetic data.
512px/pixel_adaptation/stage2/epoch_49_step_70000 $v$-pred 16 Recommended Pixel DiT (real-data SFT). GenEval 0.568 / DPG 75.31. Optimal for few-step ($N \le 8$) & RL rollouts.
512px/pixel_adaptation/stage2/large_ps/epoch_73_step_150000 $v$-pred 32 Speed Tradeoff: $4\times$ token compression ($1.78\times$ faster sampling, reduced compositional binding).
512px/pixel_adaptation/x_pred/stage2/epoch_30_step_25000 $x_0$-pred 16 Ablation Only: Last $x_0$-pred checkpoint on synthetic data.
512px/pixel_adaptation/x_pred/stage2/epoch_49_step_70000 $x_0$-pred 16 Ablation Only: Real-data SFT $x_0$-pred (prone to $t \to 0$ clipping/color desaturation).
512px/pixel_adaptation/x_pred/stage2/large_ps/epoch_74_step_150000 $x_0$-pred 32 Ablation Only: Large-patch ($P=32$) $x_0$-pred variant.

Choosing Between Latent and Pixel Checkpoints (Amortization Asymmetry): Pixel DiT eliminates the terminal VAE decode ($4.82\text{ s}$ at 1MP), but evaluates the convolutional PDH head at every ODE step ($+27.8%$ step cost). With a break-even threshold of $N^* \approx 12.25$ steps, Latent DiT (epoch_139_step_391917) is 9%–31% faster for standard 20–30 step sampling, whereas Pixel DiT (epoch_49_step_70000) is up to $3.34\times$ faster in few-step regimes ($N \le 8$) and avoids VAE decode memory bottlenecks during online RL (Flow-GRPO).


Inference and WebUI

The training repository has a simple Gradio UI to perform sampling with camera controls. ui_app.py

Recommended Settings

  • Sampling Algorithm: Forward Euler ODE Solver (Rectified Flow)
  • Inference Steps: 75
  • CFG Scale: 9.0
  • Time Shift: 3.0

Prompt Formatting Guidelines

Unlike tag-based anime models, this open-domain foundation model was trained exclusively on long, descriptive natural-language paragraphs generated by Qwen3-VL-30B-A3B.

  • Short keywords underperform unless expanded into rich visual sentences specifying subjects, materials, lighting, spatial relations, and background.
  • Leave negative_prompt empty ("") for standard evaluation matching the paper's benchmark protocol.

Python Sampling Script

from dataclasses import dataclass
import torch
from omegaconf import OmegaConf
from src.diffusion.sampling import generate_samples
from src.diffusion.schedules import LinearSchedule
from src.models.factory import load_trainable_model


@dataclass(frozen=True)
class InferencePreset:
    config_path: str
    checkpoint: str
    model_type: str
    pixel_sampling: bool
    default_steps: int
    default_cfg: float


PRESETS = {
    "512_latent": InferencePreset(
        config_path="configs/latent_512.yaml",
        checkpoint="epoch_139_step_391917",
        model_type="sprint_dual",
        pixel_sampling=False,
        default_steps=30,
        default_cfg=6.0,
    ),
    "512_pixel_vpred": InferencePreset(
        config_path="configs/pixel_512_p16.yaml",
        checkpoint="512px/pixel_adaptation/stage2/epoch_49_step_70000",
        model_type="sprint_dual_pixel",
        pixel_sampling=True,
        default_steps=4,
        default_cfg=6.0,
    ),
    "512_pixel_ps32": InferencePreset(
        config_path="configs/pixel_512_p32.yaml",
        checkpoint=(
            "512px/pixel_adaptation/stage2/large_ps/"
            "epoch_73_step_150000"
        ),
        model_type="sprint_dual_pixel",
        pixel_sampling=True,
        default_steps=20,
        default_cfg=6.0,
    ),
}

# Select mode: '512_latent', '512_pixel_vpred', or '512_pixel_ps32'
preset = PRESETS["512_latent"]
cfg = OmegaConf.load(preset.config_path)
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
dtype = torch.bfloat16

model, text_encoder, vae, tokenizer, _ = load_trainable_model(
    models_path=cfg.paths.models,
    device=device,
    dtype=dtype,
    resume_from_checkpoint=preset.checkpoint,
    model_type=preset.model_type,
    model_cfg=cfg.models,
    autocast_dtype=dtype,
)

schedule = LinearSchedule(device=device)
sample_configs = [
    {
        "prompt": (
            "An oil painting portrait of a distinguished raccoon wearing "
            "an ornate crimson Renaissance doublet and a white pleated "
            "ruff collar, illuminated by warm studio lighting against a "
            "dark museum backdrop with visible canvas brushwork."
        ),
        "negative_prompt": "",
        "height": 512,
        "width": 512,
        "sample_steps": preset.default_steps,
        "cfg_scale": preset.default_cfg,
        "shift": 1.0,
        "zoom": 1.0,
        "x_shift": 0.0,
        "y_shift": 0.0,
        "seed": 42,
    }
]

# Pixel DiT outputs RGB tensors directly; bypass terminal VAE decoding
images = generate_samples(
    unet=model,
    text_encoder=text_encoder,
    tokenizer=tokenizer,
    vae=None if preset.pixel_sampling else vae,
    schedule=schedule,
    sample_configs=sample_configs,
    global_batch_size=1,
    diffusion_type="flow_matching",
    device=device,
    dtype=dtype,
    autocast_dtype=dtype,
    use_unet_mult=False,
    vae_mean=cfg.models.vae_mean,
    vae_std=cfg.models.vae_std,
    in_channels=cfg.models.in_channels,
    coord_system="aspect_norm",
    pixel_sampling=preset.pixel_sampling,
)

images[0].save("output.png")

Content Advisory & Limitations

  • Dataset Scale & Crowd Scenes: Pretrained on 625,000 samples with 206M parameters; multi-billion parameter models trained on hundreds of millions of images remain superior on complex crowd scenes and obscure long-tail entities.
  • Multi-Word Text Rendering: Although TextAtlas5M is included in the training mixture, the compact 206M capacity cannot reliably render dense multi-word typography.
  • Camera Control & Square-Synthetic Warping: Because synthetic training sources (FLUX-Reason-6M, Midjourney-v6-Recap) are natively square, shifted square cropping can teach off-center coordinates to correlate with clipped focal points. Consequently, large zoom ($>1.0\times$) or pan offsets may cause high-frequency texture smearing, semantic color bleeding, or geometric warping on rigid structures.
  • Experimental 1MP & Pixel Ablations: The $1024^2$ checkpoints (epoch_147_*, epoch_149_*) are experimental and unbenchmarked. Under pixel adaptation, prefer native $v$-prediction over the $x_0$-prediction ablation checkpoints, which exhibit dynamic range clipping and chromatic desaturation.
  • No Built-In Safety Filters: These are raw research weights without safety classifiers or output filters; users and deployers are solely responsible for implementing content moderation.

References

License

This model is licensed under the CreativeML Open RAIL++-M License. You may use, modify, and distribute the weights provided you include the full LICENSE text and enforce Attachment A restrictions.

  • No Illegal Use: You must not use this model or its derivatives to generate illegal content or for any restricted uses under Attachment A (including harming or exploiting minors, defamation, harassment, or unlawful acts).
  • No Responsibility / Liability: The model is provided "AS IS" without warranty. You are solely responsible for your usage and the outputs you generate; to the fullest extent permitted by applicable law, the authors disclaim all liability for any misuse or damages arising from the model.
Downloads last month
-
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Dataset used to train aipracticecafe/asuka-fm-air

Papers for aipracticecafe/asuka-fm-air