AToken - A Unified Tokenizer for Vision

Executive Summary

This document provides a comprehensive analysis of ATOKEN, a novel visual tokenizer developed by researchers at Apple. ATOKEN represents a significant advancement in artificial intelligence by being the first unified framework capable of processing images, videos, and 3D assets for both high-fidelity reconstruction (generation) and high-level semantic understanding. Current visual AI systems are fragmented, employing specialized models for either reconstruction or understanding and for specific modalities, which has limited the generalization and transfer learning capabilities seen in modern language models.

ATOKEN overcomes these limitations through several key innovations. It introduces a sparse 4D latent space that represents all visual modalities within a single, shared framework. This is processed by a pure transformer architecture with 4D Rotary Position Embeddings, enabling the model to handle inputs of arbitrary resolution and duration. To ensure stable and scalable training, ATOKEN utilizes a novel adversarial-free training objective that combines perceptual and Gram matrix losses, achieving state-of-the-art reconstruction quality without the instabilities of GAN-based methods. The model is trained using a four-stage progressive curriculum, incrementally adding capabilities for images, videos, and 3D assets, along with an optional final stage for discrete tokenization.

Empirically, ATOKEN demonstrates state-of-the-art or competitive performance across all modalities. It achieves an rFID of 0.21 for image reconstruction with 82.2% ImageNet accuracy; an rFVD of 3.01 for video reconstruction with 40.2% MSRVTT retrieval; and a PSNR of 28.28 for 3D reconstruction with 90.9% classification accuracy. A key finding is that multimodal training enhances single-modality performance, with image reconstruction improving as video and 3D capabilities are added. The model's effectiveness is further validated across a range of downstream applications, including multimodal LLMs and generative tasks like text-to-video and image-to-3D synthesis, establishing ATOKEN as a versatile foundation for the next generation of multimodal AI systems.

The Challenge of Fragmented Visual AI

The versatility of Large Language Models (LLMs) stems from a unified approach where simple tokenizers convert diverse text types into a shared token space, enabling massive scaling and knowledge transfer. In contrast, the visual domain remains fragmented, facing three fundamental challenges that have prevented similar progress.

• Task Specialization: Existing visual models are optimized for either reconstruction or understanding, but not both.
  • Reconstruction Methods (e.g., SD-VAE, VQGAN) excel at compressing visual data for generation but lack semantic features for understanding tasks.
  • Understanding Encoders (e.g., CLIP, SigLIP2) produce rich semantic representations for tasks like classification and retrieval but cannot reconstruct the original visual content.
• Modality Fragmentation: Tokenizers are typically designed for a single modality. Image and video tokenizers cannot process 3D data, while 3D tokenizers are restricted to their domain and cannot leverage vast image and video datasets for pretraining. This prevents seamless knowledge transfer across visual formats.
• Architectural Trade-offs: Different architectures present their own limitations. Convolutional models show diminishing returns when scaled, while transformer-based tokenizers often suffer from training instabilities when using adversarial objectives (GANs). Furthermore, models must choose between continuous tokens (for reconstruction quality) or discrete tokens (for LLM compatibility), with few supporting both.

ATOKEN is designed to address all these limitations simultaneously, offering the first unified solution as detailed in the comparison below.

Method	Reconstruction	Understanding	Modalities	Token Types	GAN Free
Reconstruction-Only (e.g., SD-VAE, Hunyuan)	✓	✗	Image/Video	Continuous/Discrete	✗
Understanding-Only (e.g., SigLIP2, VideoPrism)	✗	✓	Image/Video	N/A	✓
Unified (Image-Only) (e.g., VILA-U, UniTok)	✓	✓	Image	Discrete	✗
ATOKEN (This work)	✓	✓	Image, Video, & 3D	Continuous & Discrete	✓

ATOKEN's Unified Framework

ATOKEN unifies visual modalities and tasks through a novel architecture and data representation strategy. It leverages a sparse 4D representation, a pure transformer backbone, and a dual-projection system to handle both generation and understanding.

Core Innovation: The Sparse 4D Latent Space

The central insight behind ATOKEN is that all visual modalities can be represented within a shared, sparse 4D space defined by temporal (t) and spatial (x, y, z) coordinates.

• Unified Representation: Visual inputs are converted into a set of feature-coordinate pairs {(zi, pi)}.
• Modality as a Subspace: Each modality naturally occupies a different subspace of this 4D representation:
  • Images: 2D slices on the (x, y) plane (t=0, z=0).
  • Videos: Temporal stacks extending along the t axis (z=0).
  • 3D Assets: Surface voxels occupying the (x, y, z) space (t=0).
• 3D Processing: For 3D assets, multi-view images are rendered from spherically sampled cameras. ATOKEN's standard patchification is applied, and features are aggregated back into the voxel space.

Pure Transformer Architecture

ATOKEN employs a unified transformer architecture for both its encoder and decoder, which processes the sparse 4D representations efficiently.

• Encoder: The encoder extends a pretrained SigLIP2 vision tower. This is achieved by generalizing its 2D patch embeddings to 4D space-time blocks and augmenting its 2D position embeddings with 4D Rotary Position Embeddings (RoPE). This modification preserves the strong semantic priors of SigLIP2 while enabling it to process all modalities and handle arbitrary resolutions and temporal lengths.
• Decoder: The decoder shares the same transformer architecture but is trained from scratch for reconstruction. It maps the structured latents back to visual outputs using task-specific heads:
  • Images/Videos: Decodes directly to pixel space.
  • 3D Assets: Decodes to pixel-space features and then to Gaussian Splatting parameters for efficient rendering.

Dual-Task Representation

From the unified 4D latent space, ATOKEN extracts representations for both reconstruction and understanding through complementary projections, all derived from the same encoded features.

• For Reconstruction: Latents are projected to a lower-dimensional space, supporting both continuous latents and discrete tokens (via optional FSQ quantization). These are fed to the decoder to reconstruct the visual input.
• For Understanding: Latents are aggregated via attention pooling into a single global representation, which is then projected for alignment with text embeddings.

This dual-path design allows the model to be jointly optimized for both tasks without architectural duplication.

A Novel Training Methodology

ATOKEN's success relies on an innovative training strategy that ensures stability, scalability, and cross-modal learning.

Adversarial-Free Training with Gram Matrix Loss

Transformer-based visual tokenizers often face severe training instability when using GANs. ATOKEN's developers found that in their setup, the discriminator quickly overpowered the generator, leading to mode collapse. Their analysis revealed that reconstruction error (rFID) is dominated by the covariance component (~86.6%), which captures texture and style, rather than the mean component (~13.4%).

This motivated the adoption of an adversarial-free loss that combines:

• Perceptual Losses: L1, LPIPS, and CLIP losses to ensure pixel-level and semantic similarity.
• Gram Matrix Loss: This loss directly optimizes the covariance of feature maps, effectively capturing style and texture without the instability of adversarial training. This approach proved to achieve superior and stable reconstruction quality throughout training.

Progressive Multimodal Curriculum

ATOKEN is trained using a four-stage progressive curriculum, starting from the pretrained SigLIP2 encoder and incrementally building capabilities.

Stage	Name	New Capabilities & Details
Stage 1	Image Foundation	Adds image reconstruction capabilities to the pretrained SigLIP2 encoder. Trains on images with resolutions from 64px to 512px.
Stage 2	Video Dynamics	Extends to temporal sequences by adding video reconstruction and understanding. Increases image resolution up to 1024px and video up to 512px. Uses temporal tiling with KV-caching for efficiency.
Stage 3	3D Geometry	Incorporates 3D assets (64³ voxel grids) for both reconstruction (via Gaussian splatting) and understanding. Further increases image/video resolution to 2048px/1024px.
Stage 4	Discrete Tokenization	Optionally fine-tunes the entire model to add FSQ quantization, enabling the generation of discrete tokens compatible with autoregressive models.

Key Finding: Cross-Modal Enhancement

A surprising and significant finding from the progressive curriculum is that multimodal training enhances single-modality performance. The final ATOKEN model achieves better image reconstruction quality than the earlier, image-only versions. For example, the image rFID score improved by 19% from Stage 1 (0.258) to Stage 3 (0.209). This suggests that learning from the temporal dynamics of videos and the geometric structures of 3D assets provides complementary signals that benefit image representation learning.

Performance and Capabilities

ATOKEN was comprehensively evaluated on reconstruction quality, semantic understanding, and downstream applications, demonstrating competitive or state-of-the-art performance across all modalities.

Unified Performance Comparison

ATOKEN is the first tokenizer to achieve strong performance across all three modalities for both reconstruction and understanding.

Modality	Task	Metric	ATOKEN-So/C (Continuous)	ATOKEN-So/D (Discrete)
Image	Reconstruction	rFID↓	0.21	0.38
	Understanding	ImageNet Acc.↑	82.2%	82.2%
Video	Reconstruction	rFVD↓	3.01	22.16
	Understanding	MSR-VTT R@1↑	40.2%	40.3%
3D	Reconstruction	PSNR↑	28.28	28.17
	Understanding	Toys4k Acc.↑	90.9%	91.3%

Scaling Analysis

A scaling analysis was conducted by comparing the main So400m model (~800M parameters) with a smaller Base variant (~192M parameters).

• Small Models Suffer Interference: The Base model performed reasonably in Stage 1 (image-only) but suffered severe performance degradation when video capabilities were added. Its ImageNet rFID degraded by 49%.
• Large Models Benefit from Synergy: The So400m model showed continuous improvement. Its ImageNet rFID improved by 19% and its video PSNR also increased after 3D data was introduced.
• Conclusion: This indicates that a sufficient model capacity is critical for successful multimodal tokenization, allowing larger models to benefit from cross-modal learning while smaller models suffer from task interference.

Downstream Application Viability

ATOKEN's effectiveness as a universal visual foundation was tested in a variety of downstream tasks, demonstrating its versatility without compromising task-specific performance.

Vision-Language Understanding (Multimodal LLMs)

ATOKEN was integrated as a frozen vision encoder into the SlowFast-LLaVA-1.5 framework and compared against models using the specialized Oryx-ViT encoder.

• Image Understanding: SlowFast-LLaVA with ATOKEN showed overall better performance than the same model with Oryx-ViT across 1B, 3B, and 7B scales on benchmarks like RW-QA, SQA, and TextVQA.
• Video Understanding: ATOKEN excelled at smaller model scales, achieving state-of-the-art results on several benchmarks. At the 1.5B scale, it outperformed Oryx-ViT by 0.8% on LongVideoBench and 1.4% on LVBench.

These results validate ATOKEN's ability to serve as a powerful and general-purpose vision backbone for multimodal LLMs.

Multimodal Generation

ATOKEN's generative latents (both continuous and discrete) were successfully used to power various generative models.

• Image Generation (Continuous): When used with the Lightning-DiT framework, ATOKEN achieved a gFID of 1.56, competitive with specialized reconstruction-focused tokenizers like VAVAE (1.35 gFID).
• Image Generation (Discrete): Integrated into the TokenBridge autoregressive framework, ATOKEN achieved a gFID of 2.23, outperforming the other unified tokenizer UniTok (2.51 gFID) despite using a more challenging, larger vocabulary space.
• Text-to-Video Generation: In a resource-constrained setup, ATOKEN achieved results comparable to specialized video tokenizers like Hunyuan and Wan across multiple text-to-image and text-to-video benchmarks.
• Image-to-3D Synthesis: ATOKEN's tokens were used to train an image-to-3D diffusion model. It successfully generated 3D assets from single images, though performance did not match the specialized Trellis-SLAT model. This is hypothesized to be due to ATOKEN's much larger latent channel dimension (48 vs. 8), which likely requires further hyperparameter optimization in the downstream model.

Future Implications

ATOKEN marks a critical step toward unifying visual AI in a manner analogous to how tokenization unified language modeling. By creating a single framework that handles high-fidelity reconstruction and rich semantic understanding across images, videos, and 3D assets, it breaks down the fragmentation that has long limited the field. The combination of a sparse 4D representation, a pure transformer architecture, an adversarial-free training strategy, and a progressive curriculum has proven highly effective.

The results not only demonstrate state-of-the-art performance but also reveal that learning across modalities can be synergistic, improving capabilities rather than compromising them. While this work successfully validates ATOKEN's utility across numerous separate downstream tasks, the ultimate goal—building a comprehensive omnimodel that leverages its full potential—remains a direction for future research. This work provides a powerful foundation and a clear path toward the next generation of truly general-purpose multimodal AI systems.