HMA (Heterogeneous Masked Autoregression) is a robot action-video dynamic modeling method jointly proposed by MIT, Meta FAIR and UIUC. Through heterogeneous mask autoregressive pre-training, cross-ontology and cross-domain action-video joint distribution is learned on 40+ data sets and 3 million+ trajectories to achieve real-time, high-fidelity robot video simulation. Compared with previous SOTA, the inference speed is increased by 15 times and can be widely used in strategy evaluation, synthetic data generation and interactive video simulation.

HMA: Robot Action for the Real World - Video Dynamic Modeling

Core parameters and statistics

Project	Details
Product name	HMA (Heterogeneous Masked Autoregression)
Product Type	Open Source Research Model / Robot World Model
Delivery form	Open source code (GitHub) + pre-trained weights (HuggingFace)
Model scale	HMA-MagVit: 362M parameters; HMA-MAR: 1B parameters
Training data	40+ datasets, 3 million+ trajectories
Inference speed	15 times faster than SOTA (real-time operation)
Open Source License	Apache-2.0
Supported languages	English
Target users	Robot researchers, embodied intelligence engineers, AI simulation developers
Core Paper	arXiv 2502.04296 (February 2025)
Project homepage	https://liruiw.github.io/hma

Parameter Interpretation: The parameter size of HMA is in the "medium to large" range - the MagVit version (362M) is suitable for single-GPU inference and rapid iteration, and the MAR version (1B) has further improved video fidelity. Compared with previous mainstream diffusion model solutions (such as UniPi and VideoPoet), HMA adopts the masked autoregressive route, which has an order of magnitude advantage in generation speed (15 times acceleration). This is its core engineering value: for the first time, it pushes robot video simulation to the threshold of "interactive real-time".

User and market recognition

Academic Adoption Signal: HMA quickly gained attention from the robotics learning community after being published on arXiv in February 2025. The authors of the paper come from MIT CSAIL, Meta AI (FAIR) and UIUC. This combination itself provides strong academic credibility. The project GitHub repository (github.com/liruiw/HMA) currently shows 41 stars. Although it is not a large-scale community project, considering its professionalism (robot video dynamic modeling), this data is in line with expectations for early research projects.

Third-party communication: HMA has been included in the AI ​​tool set navigation station (ai-bot.cn), and multiple technical interpretation videos have appeared on Bilibili and other platforms (such as "MIT's latest research! HMA: 15 times faster, high-quality robot action data generation!"), indicating that it has also gained attention in the Chinese technology community. The paper is classified under cs.RO (Robotics) on arXiv, cross-referenced to cs.CV (Computer Vision) and cs.LG (Machine Learning), reflecting interdisciplinary influence.

Benchmark performance: In comparison with multiple SOTA video generation models, HMA takes the lead in both visual fidelity (FVD indicator) and controllability (action tracking accuracy). The paper shows qualitative and quantitative results on multiple real robot data sets such as Bridge, Dobb-E, Kaist, etc. The video generated by HMA is significantly better than the baseline method in terms of physical rationality (object interaction, shadow, occlusion processing).

Current Limitations: As an academic research project, HMA does not yet have a commercial version or enterprise-level support. Its user base is mainly limited to research teams with deep learning training and robot deployment capabilities. The number of GitHub stars and community activity are at an early stage, and a complete documentation ecosystem and user community have not yet been formed.

Cost advantage

HMA’s fully open source model puts its cost structure in stark contrast to commercial robot simulation tools:

Cost dimension	HMA (open source)	Commercial robot simulators (such as NVIDIA Isaac Sim)	Cloud video generation API
C-side/researcher	Completely free, code + weight open source	Free community version has limited functions, full version requires enterprise license	Billed by Token/second, long videos are expensive
API / Developer	No API layer, need to deploy inference by yourself	Python SDK provided, GPU cluster required	Billed by request, typical $0.01-0.10/second
Enterprise/Private	Self-hosted, no licensing fees, only GPU hardware	Isaac Sim Enterprise Edition costs $10,000+ per year	No support for privatization

C-side cost: Researchers can clone the repository directly through GitHub, download the pre-trained weights (0.4B/1B) on HuggingFace, and run inference and interactive demo on a single RTX 3090/4090. Hardware cost is the only real expense.

Developer Cost: HMA does not provide a commercial API. To integrate into the workflow, the team needs to complete the environment configuration (Python 3.10+, PyTorch, Accelerate and other dependencies), model deployment and inference pipeline construction by themselves. The initial deployment cost is about 2-5 man-days, suitable for teams with DL Ops capabilities.

Enterprise Cost: Compared with commercial simulators (NVIDIA Isaac Sim Enterprise Edition $10K+/year, Amazon RoboMaker billed by the hour), HMA has a significant TCO advantage - zero software license, but requires a team with model deployment and tuning capabilities. Hidden costs are mainly reflected in: data preparation (robot trajectory data needs to be converted into RLDS format), model post-training adaptation (fine-tuning experiments for a specific robot ontology), and lack of commercial support.

The free truth: HMA is completely free and open source, but the premise of "free" is that the user has the ability to build a Python deep learning development environment and the ability to obtain robot operation data. For teams without robot data accumulation, the cost of collecting trajectories from scratch may exceed the annual fee for commercial tools.

Main functions

• Heterogeneous Masked Autoregressive Video Prediction: Given an initial frame and action sequence, HMA autoregressively generates subsequent video frames. Unlike the diffusion model, which is generated independently frame by frame, the masked autoregressive mechanism models timing dependencies at the token level and can predict multiple future tokens in a single step, significantly improving inference efficiency. Applicable tasks: Rapid video rollback evaluation of robot strategies and visualization of planning results.

• Cross-ontology, cross-domain heterogeneous pre-training: HMA is jointly trained on 40+ data sets, and the data sources include real robot teleoperation (Bridge, Dobb-E, Kaist), human operation videos (EpicKitchens), simulation data (RLBench, MetaWorld), etc. The model does not need to be trained separately for each robot ontology, and the pre-trained weights can be directly used for unseen robot forms. Use value: Lower the threshold for model deployment in new robot scenarios, compressing data collection from weeks to training and adaptation after hours.

• Dual-modal generation - discrete Token and continuous Token: HMA-MagVit uses VQ-VAE discretization (362M parameters) to generate discrete video token sequences, suitable for precise control and structured evaluation; HMA-MAR uses masked autoregressive continuous representation (1B parameters) to generate soft tokens, which is better in visual quality. Both share the same training framework and can be switched according to task requirements. Use value: Choose the discrete model for accuracy-sensitive scenarios (strategy evaluation), and choose the continuous model for visual quality-prioritized scenarios (demo generation).

• Interactive real-time video simulation Demo: The project provides an interactive Demo based on Pygame (python -m sim.app). Users can select the initial image from the gallery, use the keyboard arrow keys to control the robot's movements, and observe the video generation results in real time. Use Value: Intuitively understand model behavior and quickly verify the causal correspondence between actions and visual feedback.

• Policy evaluation and synthetic data generation pipeline: HMA has a built-in complete evaluation pipeline that can calculate FVD, PSNR, SSIM and other indicators of generated videos and real videos; it also supports the generation of a large number of synthetic trajectories from trained model sampling for data enhancement or policy pre-training. Use value: Compress the policy iteration cycle from "real environment rollout → collect data → retrain" to "simulation rollout → synthetic data → policy update", reducing dependence on real robot hardware and manual annotation.

• Post-Training mechanism: HMA supports post-training fine-tuning of pre-trained models on specific data sets, and a fine-tuning script (run_langtable_finetuning.sh) for the Language Table data set has been provided. Use Value: Quickly adapt to specific robot platforms and specific tasks (such as pushing blocks and grabbing) without training from scratch.

Model and version evolution

The version context of HMA is based on paper preprints and code release:

Version	Date	Core Changes
arXiv preprint	2025-02-06	The paper is released publicly for the first time, synchronizing the open source GitHub code repository and HuggingFace model weights
Initial code submission	2025-02 (approx.)	Contains complete training, generation, evaluation, and visualization pipelines, supporting 40+ data sets
HMA-MagVit (362M)	Same as initial code	Discrete token model, based on VQ-VAE + masked autoregression
HMA-MAR (1B)	Same as the initial code	Continuous token model, based on masked autoregression, with larger number of parameters

Version evolution description: HMA is currently in the "paper open source" stage. The code repository is a one-time initial submission (first commit), and there is no version label or official release yet. The author noted "Code quality: tired grad student" in the README, indicating that the code takes reproducibility as its primary goal and does not carry out engineering packaging. Subsequent evolution directions are expected to include: more data set adaptation, model quantification acceleration, and more friendly packaging interfaces (such as ROS integration, Gym environment packaging).

Technical advantages

Dismantling of the causal chain of Mechanism → Effect → Scenario:

1. Masked Autoregression replaces the diffusion model: Traditional robot video generation methods (such as UniPi) use the diffusion model to generate frame by frame. Each frame requires multi-step denoising, and the inference delay is high. HMA adopts a masked autoregressive strategy—randomly masking a portion of video tokens during training, allowing the model to predict the masked content; predicting all masked tokens at once during inference, and then iteratively refine autoregressively. Effect: Reduce the number of inference steps from 50-1000 steps in the diffusion model to 8-16 steps, achieving 15 times end-to-end acceleration. Applicable scenarios: Interactive simulation and online strategy evaluation that require real-time feedback.

2. Heterogeneous Training overcomes data scarcity: The robotics field has long faced the problems of scattered data sources, different ontologies, and diverse tasks. HMA allows models to be jointly pre-trained on 40+ heterogeneous data sets through a unified tokenization interface (unified encoding of videos of different resolutions and action sequences of different frequencies into fixed-length token sequences). Effect: The pre-trained model demonstrates zero-sample scheduling capabilities on unseen robot bodies and tasks, and post-training can generate more than 100 frames of effective data on new scenes with only 200 trajectories. Applicable scenarios: Rapid deployment of new robot platforms and cross-ontology knowledge migration.

3. Action Conditioning achieves precise controllability: HMA not only splices action tokens in the input, but also uses a modulation mechanism to allow the action signal to affect the intermediate representation of each layer of Transformer, rather than just injecting it at the input end. Effect: The generated video strictly follows the input action sequence - the direction and strength of the push block, the trajectory of the robotic arm, etc. are accurately reflected in the generated results. Compared with the unconditional video generation model, the controllability of HMA is significantly improved. Applicable scenarios: Strategy verification that requires precise motion constraints (such as "If a force in the x direction is applied, how will the object move?").

4. VQ-VAE + MAR dual pipeline takes into account accuracy and quality: the discrete branch (MagVit) uses VQ-VAE to compress video frames into discrete tokens, ensuring the controllability of the generation process and the repeatability of evaluation; the continuous branch (MAR) directly operates the continuous latent space to retain more detailed information. Both share the same Transformer backbone and have low switching costs. Effect: The discrete model is better in quantitative indicators such as FVD and is suitable for benchmark comparison; the continuous model is more natural in human visual evaluation and suitable for demonstration and visualization. Applicable scenarios: Choose discrete for academic paper evaluation, and choose continuous for product prototype demonstration.

How to use

Environment setup

HMA requires Python 3.10+ and CUDA environment. It is recommended to use Conda or venv to manage dependencies:

# Clone repository
git clone https://github.com/liruiw/HMA.git
cd HMA

#Install dependencies and download data (venv will be automatically created)
./build.sh

# Activate Python environment
source venv/bin/activate

Interactive Demo

# Start interactive GUI Demo
python -m sim.app

After startup, select the initial image from the gallery, use the keyboard arrow keys to control the actions, and observe the video frames generated by HMA in real time. This is the fastest way to experience HMA capabilities without pre-training the model.

Pre-trained model inference

# Download the pre-trained model (needs to be obtained manually from HuggingFace)
#Single data set inference to generate video
python hma/generate.py \
  --checkpoint_dir data/model/step_100/ \
  --val_data_dir data/kaist_nonprehensile_converted_externally_to_rlds_magvit_max1000000_val

Model evaluation

# Discrete model evaluation
accelerate launch hma/evaluate.py \
  --checkpoint_dir "data/${RUN_NAME}/final_checkpt" \
  --val_data_dir "data/${dataset}_magvit_traj1000000_val" \
  --wandb_run_name "${RUN_NAME}"

# Continuous model evaluation
accelerate launch hma/evaluate_feature.py \
  --checkpoint_dir "data/${RUN_NAME}/final_checkpt" \
  --val_data_dir "data/${dataset}_magvit_traj1000000_val"

Pre-training (multiple data sets)

# VQ tokens model (discrete)
bash experiments/scripts/discrete_model/run_40datasets_waction.sh

# Soft tokens model (continuous)
bash experiments/scripts/continuous_model/run_30datasets_mar_waction.sh

Entry summary

Entrance	Purpose	Technical Requirements
GitHub repository	Source code, training scripts, evaluation pipeline	Python 3.10+, CUDA, PyTorch
HuggingFace model library	Pre-training weight download	None (download can be used for inference)
Project homepage (Demo)	Online interactive demonstration	Browser access, no local environment required
arXiv paper	Technical principles and experimental details	None
Interactive Demo (local)	Local real-time simulation experience	Python 3.10+, Pygame, GPU recommended

Product Pricing

HMA is a purely open source project and the pricing structure is extremely simple:

• Model Weights: Free, Apache-2.0 License under the MIT License
• Source Code: Free, Apache-2.0 License
• Papers: Free, open access on arXiv
• Online Demo: Free, the project homepage provides HuggingFace Spaces online experience

There are currently no paid tiers. In terms of commercial licensing, Apache-2.0 allows free use, modification, and redistribution. However, whether the model weights include additional usage restrictions on third-party data sets requires verification of the original license agreement of each data set (such as the Bridge data set terms of use).

Pricing comparison with competitive products:

Projects	HMA	NVIDIA Isaac Sim	Google Genie	Physical Intelligence π0
Software Pricing	Free	Community Edition is free, Enterprise Edition is $10K+/year	Undisclosed	Undisclosed
Inference cost	Self-hosted, single GPU	Cloud GPU on demand	Cloud API speculative billing	Undisclosed
Data requirements	Self-prepared robot trajectory	Built-in simulation environment	No external data required	Commercial cooperation required
Customizability	Fully open	Modular, partially closed source	Black box API	Limited to cooperative customers

Application scenarios

• Rapid Robot Strategy Evaluation and Iteration: Traditional strategy evaluation requires repeated rollouts on actual robots, which is time-consuming and involves the risk of hardware wear. HMA can be used as a "video world model" to receive the action sequence output by the strategy, generate the corresponding video prediction results, and quickly verify the qualitative performance of the strategy (whether the goal is achieved, whether the action is smooth, and whether the interaction is physically reasonable). Deduction benefits: From "each evaluation requires a 30-minute rollout of the real robot" to "2 minutes to complete the HMA simulation evaluation", the single iteration cycle is shortened by more than 90%, which is especially suitable for the early preliminary screening stage of the strategy.

• Synthetic data generation for policy pre-training: When real data is limited, HMA is used to start from a small number of real trajectories, adapt to the target scene through post-training, and then generate synthetic video-action pairs through large-scale sampling. These synthetic data can be used to pretrain the backend visual encoder of the policy model or as a means of data augmentation. Implementation Tip: The HMA paper shows that using only 200 real trajectories for post-training can generate more than 100 frames of effective synthetic data, covering edge cases that are not fully sampled in the original data distribution (such as different occlusion modes, lighting changes).

• Interactive robot behavior analysis and teaching: In scientific research or teaching, researchers can use HMA's interactive demo to intuitively demonstrate the visual feedback corresponding to different action inputs (such as push, pull, and rotation) to understand the physical laws of robot-environment interaction. No actual robotics hardware required, just a laptop with a GPU. Deduction benefits: Lower the entry barrier for robots - from "needing to purchase and maintain a hardware platform" to "only a software environment", reducing the cost of robot teaching in university courses and remote collaboration by an order of magnitude.

• Cross-ontology knowledge transfer pre-research: When the team needs to switch from an existing robot platform (such as Franka) to a new platform (such as UR5), they can use HMA's heterogeneous pre-training capabilities to do post-training with a small amount of trajectory data from the new platform to quickly evaluate whether the migration solution is feasible without investing in complete hardware deployment costs. Deduction benefits: Compress the preliminary feasibility verification on the new platform from weeks of hardware + data preparation to several days of simulation experiments.

• Multi-robot collaborative simulation: Since the action conditional design of HMA supports multi-entity interaction scenarios (such as multiple robot arms collaborating to push boxes), it can be used for multi-robot collaborative strategy verification beyond line of sight. The current demo has demonstrated the interactive simulation capability of controlling multiple objects at the same time.

Applicable people

• Robot Algorithm Researchers: HMA’s core user group. Teams that need to evaluate new strategies, generate visualizations of comparative experiments, or conduct ablation studies. HMA provides a path to "replace part of the real rollout with simulation video", directly reducing experimental costs. Prerequisites: Familiar with PyTorch and deep learning training process; have the ability to read and process robot trajectory data (RLDS format).

• Graduate students in Embodied Intelligence/Robotics: In paper reproductions, course projects, or open source contributions, HMA provides ready-to-use pre-trained models and complete training pipelines that can be used as research baselines or comparison methods. Interactive demos are also suitable as teaching demonstration tools. Prerequisites: Have basic Python and basic CUDA environment configuration capabilities.

• Algorithm Team of Robotics Startup Company: In the early product development stage, HMA can be used as a "free world model" for rapid prototyping of strategies, reducing dependence on physical robot hardware. Recommended usage: First use the public data set to load the pre-trained model and run through the demo, and then collect your own robot data for post-training adaptation.

• AI simulation platform builder: A technical team that needs to provide a diverse visual simulation environment for robots. HMA’s heterogeneous pre-training features and Apache-2.0 license make it suitable for integration into higher-level workflow platforms.

• Not suitable for the crowd:

  • Pure business users (no programming background): HMA has no GUI application or no-code interface, all operations are based on command line and Python scripts.
  • Industrial users who require stable commercial support: HMA has no SLA, no technical support, no enterprise-level documentation, and is not suitable for critical paths in production environments.
  • General developers and designers in non-robotic fields: HMA focuses on robot action-video modeling and cannot be used for general video generation, content creation and other scenarios.

Summary and Outlook

Core Competencies: HMA's core contribution lies in "for the first time introducing the masked autoregressive paradigm into robot video dynamic modeling, and achieving breakthroughs in the scale of heterogeneous data pre-training and real-time inference speed." The 15x acceleration advances it from "offline generation tool" to "interactive real-time simulation" category, which is an essential difference from previous diffusion model solutions. The fully open source (Apache-2.0) licensing model gives it a natural dissemination advantage in the academic community.

Current limitations: (1) Low degree of code engineering - the author self-assessed "Code quality: tired grad student", no release version, no CI/CD, no Docker image, deployment requires certain technical experience; (2) Complex data set preparation - video-action data sets in RLDS format require specific tokenization pipelines; (3) Model scale limitations - 1B parameters can be run on consumer-grade GPUs, but complete pre-training of 40+ data sets requires multiple GPUs Cluster; (4) The community ecology has not yet been formed - the documents are mainly papers + README, there is no forum, and there is no known third-party integration.

Follow-up observation points: (1) Whether the warehouse will release an official release version and the frequency of versioned checkpoint updates; (2) Whether there is integrated support for robot standard interfaces such as ROS/Gym; (3) whether the author team will release a larger model version or fine-tuning weights for specific scenarios (such as dexterous operation); (4) whether third-party packages (such as Docker images, HuggingFace inference APIs, Colab notebooks) will appear in the community to lower the threshold of use.

Procurement/Adoption Risk Assessment: HMA is suitable as a "technology pre-research and evaluation acceleration tool" and is not recommended as the only simulation dependency for production environments at this stage. It is recommended to adopt strategies: (1) First run through the entire process with interactive demo and preset data sets to evaluate the actual fidelity of the model in its own scenario; (2) Start with 200 own trajectories to conduct post-training experiments to quantify the data flywheel effect; (3) Before fully verifying, maintain a backup channel for the real robot rollout. For enterprises requiring commercial-grade support, it is recommended to wait for the community version to mature or evaluate commercial alternatives (such as NVIDIA Isaac Sim, Physical Intelligence's π0).

Related tools: CrewAI, langchain

Version Info

• HMA initial version ：It includes two model specifications: HMA-MagVit (362M parameters) and HMA-MAR (1B parameters), supports two generation modes of discrete token and continuous token, and provides full process code of pre-training, post-training, evaluation and visualization. （2025-02-06）
• arXiv preprints ：The paper was first published on arXiv (2502.04296), synchronizing the open source code and model weights. （2025-02-06）