Instructions to use aipracticecafe/asuka-fm-air with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Diffusers
How to use aipracticecafe/asuka-fm-air with Diffusers:
pip install -U diffusers transformers accelerate
import torch from diffusers import DiffusionPipeline # switch to "mps" for apple devices pipe = DiffusionPipeline.from_pretrained("aipracticecafe/asuka-fm-air", dtype=torch.bfloat16, device_map="cuda") prompt = "Astronaut in a jungle, cold color palette, muted colors, detailed, 8k" image = pipe(prompt).images[0] - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- Draw Things
- DiffusionBee
Asuka-FM: 206M Dual-Stream SPRINT DiT (Open-Domain Foundation)
An ultra-efficient, 206M-parameter Dual-Stream Diffusion Transformer (DiT) trained from scratch on a single consumer GPU (1x NVIDIA RTX 4090 24GB) in **149.0 GPU hours ($49.17)**—or 161.2 hours ($53.20) including one-time offline VAE pre-encoding—using Rectified Flow, SPRINT token-dropping, Calibrated Continuous RoPE, and Latent-to-Pixel Adaptation (DiP).
Evaluated strictly as an open-domain foundation model without benchmark-specific
fine-tuning (avoiding BLIP3o-60K overfitting), Asuka-FM achieves 0.588
GenEval / 78.96 DPG-Bench at 256px and 0.609 GenEval / 78.51 DPG-Bench
at 512px, outperforming SDXL on both metrics at a fraction of the compute.
Full Technical Report: This repository hosts the complete open-domain model weights (all training checkpoints across 256px, 512px, experimental 1MP, and Pixel DiT adaptation). For architectural derivations, hardware profiling (RTX 4090 vs. A40), and full benchmark ablations, please refer to our paper:
Asuka-FM: A System-Architectural Co-Design Framework for Efficient Text-to-Image Pretraining on Consumer Hardware (arXiv:2611.26437).
Looking for the anime domain-specialized checkpoints? See the companion anime release and Zenn technical deep-dive (Japanese).
- Developed by: aipracticecafe / UC3M & DTU
- Repository: deeplearningcafe/asuka-fm
- Architecture: AdaLN-Free Dual-Stream DiT (SPRINT, SwiGLU, Sandwich RMSNorm)
- Parameters: ~206.12M (200.03M Backbone + 6.09M Text Adapter) + 10.09M PDH (Pixel Mode)
- Objective: Conditional Flow Matching ($v$-prediction, Logit-Normal sampling)
- License: OpenRAIL++
Figure: Zero-shot open-domain generations from the 206M Asuka-512 foundation
model trained from scratch on a single RTX 4090 in 149.0 GPU hours.
Key Architecture & Features
1. SPRINT Dual-Stream Transformer Backbone
The base architecture is based on the results from (i1: A Simple and Fully Open Recipe for Strong Text-to-Image Models)[https://arxiv.org/abs/2606.11289]
- Minimalist Conditioning: Uses prepended time tokens rather than heavy per-block AdaLN modulation, reducing parameter overhead.
- Sandwich Normalization & SwiGLU: RMSNorm placed before and after attention and FFN operations ensures training stability in bfloat16.
- SPRINT Acceleration: Employs structured group-wise 2x2 patch dropping (75% drop ratio during early pretraining), cutting compute demands by half while retaining global semantic context.
2. HDM Continuous Aspect-Normalized RoPE (Camera Control)
Based on the Patch Diffusion: Faster and More Data-Efficient implementation proposed in the HDM paper. Early experiments with discrete integer position IDs caused severe multi-head replication when deviating from trained aspect ratios. We resolve this by adopting continuous aspect-normalized coordinate mapping which was the original idea proposed in HDM:
Manipulating $(x_{\text{pos}}, y_{\text{pos}})$ directly controls the visual framing at inference time:
- X-Shift / Y-Shift: Translates the virtual camera viewport horizontally or vertically.
- Zoom: Scales spatial coordinates around the focal center.
3. Components
- Text Encoder:
LiquidAI/LFM2.5-230M(unidirectional causal LM hidden states passed via a 2-layer Transformer Text Adapter). - Autoencoder:
kaiyuyue/FLUX.2-dev-vae(C=32, f=8, patch size p=2, effective image token size 16x16).
4. Curated Open-Domain Corpus (625K) & Multi-Captioning
- Open-Domain General Corpus (625K): Following empirical findings from
i1, training on balanced mixtures with
100k–400k high-quality samples per source yields competitive generative
performance without massive web scraping. We curate an open-domain dataset
of 625,000 samples by extracting exactly 125,000 samples across five
sources including synthetic and real images (all $\min(H, W) \ge 512\text{ px}$):
FLUX-Reason-6M(complex spatial composition)GPT-Edit-Simpler(paired instructional clarity)Midjourney-v6-Recap(artistic lighting and textures)Pexels(natural photography and human anatomy)TextAtlas5M(text-rich scenes)
- VLM Multi-Captioning: We maintain five synthetic captions per image
generated via
Qwen3-VL-30B-A3Bfrom thei1dataset. At each epoch, a caption is sampled dynamically, providing continuous semantic regularization that prevents text memorization. Training context is partitioned into three discrete tiers ([77, 152, 227]tokens), reducing padding overhead by up to 66% and triggering TorchInductor recompilation exactly twice during warmup. - Pixel SFT Corpus: Real-data pixel fine-tuning (Stage 3) uses 396,635
curated high-aesthetic samples ($\min(H, W) \ge 1024\text{ px}$) from
i1.
Training Recipe & Multi-Stage Progression (1x RTX 4090, 24GB)
Pretraining advances across two core latent resolution tiers ($256^2 \to 512^2$) using shifted square cropping, in-RAM caching at $256^2$, and async network streaming at $512^2$, followed by experimental 1MP ($1024^2$) scaling and 512px Latent-to-Pixel adaptation.
- Optimizer: 8-bit AdamW (
bitsandbytes), $\beta=(0.9, 0.95)$, WD=0.01 - Effective Batch Size: 256 (Latent Stages 1–2) / 128 (Pixel Adaptation)
- Precision: Mixed Precision (
torch.bfloat16withtorch.compile)
Latent Pretraining Progression (Stages 1 – 2 + Experimental 1MP)
| Stage / Phase | Res. | Epochs | Drop ($r$) | Samples | Throughput | MFU (Theo/Act) | Compute | Time | Cost ($0.33/h) |
|---|---|---|---|---|---|---|---|---|---|
| Offline VAE Encode | $256^2$ | — | — | 0.625M | 45.2 img/s | — | — | 3.8 h | $1.27 |
| 1A: SPRINT Sparse | $256^2$ | 0–74 | 0.75 | 46.88M | 282.6 img/s | 74.4% / 86.8% | 23.74 EFLOPs | 47.0 h | $15.51 |
| 1B/1C: Dense + WSD | $256^2$ | 75–99 | 0.00 | 12.50M | 172.5 img/s | 62.9% / 73.4% | 8.73 EFLOPs | 21.0 h | $6.93 |
| Offline VAE Encode | $512^2$ | — | — | 0.625M | 20.6 img/s | — | — | 8.4 h | $2.78 |
| 2A: High-Res Sparse | $512^2$ | 100–124 | 0.75 | 15.63M | 108.5 img/s | 58.5% / 68.3% | 16.24 EFLOPs | 40.0 h | $13.20 |
| 2B/2C: Dense + WSD | $512^2$ | 125–139 | 0.00 | 9.38M | 64.8 img/s | 61.0% / 71.2% | 16.94 EFLOPs | 41.0 h | $13.53 |
| DiT Subtotal (1+2) | — | 140 | — | 84.38M | — | 64.8% / 75.6% | 65.65 EFLOPs | 149.0 h | $49.17 |
| Grand Total (w/ VAE) | — | — | — | — | — | — | 65.65 EFLOPs | 161.2 h | $53.20 |
Latent-to-Pixel Adaptation (512px DiP Head, Zero-VAE Decode)
| Stage / Phase | Patch ($P$) | Drop ($r$) | Target | Samples | Throughput | MFU (Act) | Time | Cost |
|---|---|---|---|---|---|---|---|---|
| Phase 0: Offline Synth (A40) | — | — | — | 0.10M | 1.85 img/s | — | 15.0 h | $6.00 |
| Stage 1: PDH Warmup (Frozen DiT) | 16 | 0.75 | $v$-pred | 0.50M | 61.8 img/s | 72.8% | 2.25 h | $0.74 |
| Stage 2A: Joint SPRINT (Synth) | 16 | 0.75 | $v$-pred | 2.50M | 57.7 img/s | 67.8% | 12.0 h | $3.98 |
| Stage 2B: Dense Refine (Synth) | 16 | 0.00 | $v$-pred | 1.50M | 44.4 img/s | 69.8% | 9.4 h | $3.10 |
| Stage 3: Real-Data SFT (396k) | 16 | 0.00 | $v$-pred | 5.16M | 42.1 img/s | 66.2% | 34.0 h | $11.22 |
| Pixel DiT Total (Stages 1–3) | 16 | — | — | 9.66M | 46.5 img/s | 67.5% | 57.65 h | $19.04 |
Large Patch (ps32-adapt16) |
32 | 0.00 | $v$-pred | 9.92M | 38.3 img/s | 63.3% | 72.0 h | $23.76 |
Quantitative Benchmarks (GenEval & DPG-Bench, No BLIP3o Fine-Tuning)
| Model / Checkpoint | Res. | Patch ($P$) | GenEval ↑ | DPG-Bench ↑ | Single | Two-Obj | Color-Attr |
|---|---|---|---|---|---|---|---|
| Asuka-256 (Dense Final, Ep 99) | $256^2$ | 16 | 0.588 | 78.96 | 0.950 | 0.750 | 0.418 |
| Asuka-512 (75% Sparse, Ep 124) | $512^2$ | 16 | 0.480 | 68.40 | 0.963 | 0.639 | 0.163 |
| Asuka-512 (Dense Final, Ep 139) | $512^2$ | 16 | 0.609 | 78.51 | 0.950 | 0.811 | 0.450 |
| Pixel DiT ($v$-pred Final, Step 70k) | $512^2$ | 16 | 0.568 | 75.31 | 0.950 | 0.780 | 0.420 |
| Pixel DiT ($x_0$-pred Abl., Step 70k) | $512^2$ | 16 | 0.545 | 76.32 | 0.950 | 0.699 | 0.348 |
Pixel DiT Large Patch (ps32, 150k) |
$512^2$ | 32 | 0.504 | 69.87 | 0.938 | 0.576 | 0.250 |
Checkpoint Guide
All intermediate and final training checkpoints are published in this repository for full reproducibility. The primary milestones and ablation checkpoints are summarized below:
Latent Foundation Checkpoints (Requires FLUX.2 VAE)
| Checkpoint Path | Stage / Phase | Description & Recommended Usage |
|---|---|---|
epoch_74_step_183075 |
Stage 1A Final | Last 256px SPRINT 75%-sparse checkpoint. |
epoch_99_step_246541 |
Stage 1 Final | Final 256px dense checkpoint (0% drop). GenEval 0.588 / DPG 78.96. |
epoch_124_step_311210 |
Stage 2A Final | Last 512px SPRINT 75%-sparse checkpoint. |
epoch_139_step_391917 |
Stage 2 Final | Recommended default for 512px inference (0% drop). GenEval 0.609 / DPG 78.51. |
epoch_147_step_420000 |
Stage 3A (Exp.) | Experimental (not in paper): Last 1MP ($1024^2$) checkpoint at 75% token drop. |
epoch_149_step_430000 |
Stage 3B (Exp.) | Experimental (not in paper): Last 1MP ($1024^2$) dense fine-tuned checkpoint (0% drop). |
Pixel-Space Adaptation Checkpoints (Zero-VAE Decode via 10M PDH)
(Note: Both Stage 2B synthetic refinement and Stage 3 real-data SFT checkpoints
reside under the stage2/ directory on disk.)
| Checkpoint Path | Objective | Patch ($P$) | Description & Recommended Usage |
|---|---|---|---|
512px/pixel_adaptation/stage2/epoch_37_step_30000 |
$v$-pred | 16 | Last $v$-pred checkpoint trained on 100k synthetic data. |
512px/pixel_adaptation/stage2/epoch_49_step_70000 |
$v$-pred | 16 | Recommended Pixel DiT (real-data SFT). GenEval 0.568 / DPG 75.31. Optimal for few-step ($N \le 8$) & RL rollouts. |
512px/pixel_adaptation/stage2/large_ps/epoch_73_step_150000 |
$v$-pred | 32 | Speed Tradeoff: $4\times$ token compression ($1.78\times$ faster sampling, reduced compositional binding). |
512px/pixel_adaptation/x_pred/stage2/epoch_30_step_25000 |
$x_0$-pred | 16 | Ablation Only: Last $x_0$-pred checkpoint on synthetic data. |
512px/pixel_adaptation/x_pred/stage2/epoch_49_step_70000 |
$x_0$-pred | 16 | Ablation Only: Real-data SFT $x_0$-pred (prone to $t \to 0$ clipping/color desaturation). |
512px/pixel_adaptation/x_pred/stage2/large_ps/epoch_74_step_150000 |
$x_0$-pred | 32 | Ablation Only: Large-patch ($P=32$) $x_0$-pred variant. |
Choosing Between Latent and Pixel Checkpoints (Amortization Asymmetry): Pixel DiT eliminates the terminal VAE decode ($4.82\text{ s}$ at 1MP), but evaluates the convolutional PDH head at every ODE step ($+27.8%$ step cost). With a break-even threshold of $N^* \approx 12.25$ steps, Latent DiT (
epoch_139_step_391917) is 9%–31% faster for standard 20–30 step sampling, whereas Pixel DiT (epoch_49_step_70000) is up to $3.34\times$ faster in few-step regimes ($N \le 8$) and avoids VAE decode memory bottlenecks during online RL (Flow-GRPO).
Inference and WebUI
The training repository has a simple Gradio UI to perform sampling with camera controls. ui_app.py
Recommended Settings
- Sampling Algorithm: Forward Euler ODE Solver (Rectified Flow)
- Inference Steps: 75
- CFG Scale: 9.0
- Time Shift: 3.0
Prompt Formatting Guidelines
Unlike tag-based anime models, this open-domain foundation model was trained
exclusively on long, descriptive natural-language paragraphs generated by
Qwen3-VL-30B-A3B.
- Short keywords underperform unless expanded into rich visual sentences specifying subjects, materials, lighting, spatial relations, and background.
- Leave
negative_promptempty ("") for standard evaluation matching the paper's benchmark protocol.
Python Sampling Script
from dataclasses import dataclass
import torch
from omegaconf import OmegaConf
from src.diffusion.sampling import generate_samples
from src.diffusion.schedules import LinearSchedule
from src.models.factory import load_trainable_model
@dataclass(frozen=True)
class InferencePreset:
config_path: str
checkpoint: str
model_type: str
pixel_sampling: bool
default_steps: int
default_cfg: float
PRESETS = {
"512_latent": InferencePreset(
config_path="configs/latent_512.yaml",
checkpoint="epoch_139_step_391917",
model_type="sprint_dual",
pixel_sampling=False,
default_steps=30,
default_cfg=6.0,
),
"512_pixel_vpred": InferencePreset(
config_path="configs/pixel_512_p16.yaml",
checkpoint="512px/pixel_adaptation/stage2/epoch_49_step_70000",
model_type="sprint_dual_pixel",
pixel_sampling=True,
default_steps=4,
default_cfg=6.0,
),
"512_pixel_ps32": InferencePreset(
config_path="configs/pixel_512_p32.yaml",
checkpoint=(
"512px/pixel_adaptation/stage2/large_ps/"
"epoch_73_step_150000"
),
model_type="sprint_dual_pixel",
pixel_sampling=True,
default_steps=20,
default_cfg=6.0,
),
}
# Select mode: '512_latent', '512_pixel_vpred', or '512_pixel_ps32'
preset = PRESETS["512_latent"]
cfg = OmegaConf.load(preset.config_path)
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
dtype = torch.bfloat16
model, text_encoder, vae, tokenizer, _ = load_trainable_model(
models_path=cfg.paths.models,
device=device,
dtype=dtype,
resume_from_checkpoint=preset.checkpoint,
model_type=preset.model_type,
model_cfg=cfg.models,
autocast_dtype=dtype,
)
schedule = LinearSchedule(device=device)
sample_configs = [
{
"prompt": (
"An oil painting portrait of a distinguished raccoon wearing "
"an ornate crimson Renaissance doublet and a white pleated "
"ruff collar, illuminated by warm studio lighting against a "
"dark museum backdrop with visible canvas brushwork."
),
"negative_prompt": "",
"height": 512,
"width": 512,
"sample_steps": preset.default_steps,
"cfg_scale": preset.default_cfg,
"shift": 1.0,
"zoom": 1.0,
"x_shift": 0.0,
"y_shift": 0.0,
"seed": 42,
}
]
# Pixel DiT outputs RGB tensors directly; bypass terminal VAE decoding
images = generate_samples(
unet=model,
text_encoder=text_encoder,
tokenizer=tokenizer,
vae=None if preset.pixel_sampling else vae,
schedule=schedule,
sample_configs=sample_configs,
global_batch_size=1,
diffusion_type="flow_matching",
device=device,
dtype=dtype,
autocast_dtype=dtype,
use_unet_mult=False,
vae_mean=cfg.models.vae_mean,
vae_std=cfg.models.vae_std,
in_channels=cfg.models.in_channels,
coord_system="aspect_norm",
pixel_sampling=preset.pixel_sampling,
)
images[0].save("output.png")
Content Advisory & Limitations
- Dataset Scale & Crowd Scenes: Pretrained on 625,000 samples with 206M parameters; multi-billion parameter models trained on hundreds of millions of images remain superior on complex crowd scenes and obscure long-tail entities.
- Multi-Word Text Rendering: Although
TextAtlas5Mis included in the training mixture, the compact 206M capacity cannot reliably render dense multi-word typography. - Camera Control & Square-Synthetic Warping: Because synthetic training
sources (
FLUX-Reason-6M,Midjourney-v6-Recap) are natively square, shifted square cropping can teach off-center coordinates to correlate with clipped focal points. Consequently, largezoom($>1.0\times$) or pan offsets may cause high-frequency texture smearing, semantic color bleeding, or geometric warping on rigid structures. - Experimental 1MP & Pixel Ablations: The $1024^2$ checkpoints
(
epoch_147_*,epoch_149_*) are experimental and unbenchmarked. Under pixel adaptation, prefer native $v$-prediction over the $x_0$-prediction ablation checkpoints, which exhibit dynamic range clipping and chromatic desaturation. - No Built-In Safety Filters: These are raw research weights without safety classifiers or output filters; users and deployers are solely responsible for implementing content moderation.
References
- i1: i1: A Simple and Fully Open Recipe for Strong Text-to-Image Models
- SPRINT: Sparse-Dense Residual Fusion for Efficient Diffusion Transformers
- DiP: DiP: Taming Diffusion Models in Pixel Space
- Patch Diffusion: Patch Diffusion: Faster and More Data-Efficient
- HDM: Home-made Diffusion Model from Scratch to Hatch
- Benchmarks: GenEval | DPG-Bench
- Asuka-FM: Asuka-FM: A System-Architectural Co-Design Framework for Efficient Text-to-Image Pretraining on Consumer Hardware
License
This model is licensed under the CreativeML Open RAIL++-M License. You may use, modify, and distribute the weights provided you include the full LICENSE text and enforce Attachment A restrictions.
- No Illegal Use: You must not use this model or its derivatives to generate illegal content or for any restricted uses under Attachment A (including harming or exploiting minors, defamation, harassment, or unlawful acts).
- No Responsibility / Liability: The model is provided "AS IS" without warranty. You are solely responsible for your usage and the outputs you generate; to the fullest extent permitted by applicable law, the authors disclaim all liability for any misuse or damages arising from the model.
- Downloads last month
- -