ForgeTrain is an open source large model pre-training framework released by Wallface Intelligence and Tsinghua University's OpenBMB community. The framework code is 100% self-written by AI Agent Loop, using Forge Engineering's three-stage methodology (standard establishment → bit-by-bit alignment → performance overshoot). The training speed on NVIDIA H100 exceeds that of Megatron-LM by about 10%. It has completely run through the pre-training process on Huawei's Ascend chip and successfully trained the MiniCPM5-1B model.

ForgeTrain — AI Agent’s self-written large model pre-training framework

Core parameters and statistics

ForgeTrain is a production-level large model pre-training framework written entirely by AI Agents with zero human code intervention. It was jointly released by Face Wall Intelligence and the OpenBMB open source community of Tsinghua University's Natural Language Processing Laboratory, and represents an important milestone in the journey of "AI for AI" from concept verification to production implementation.

Parameter item	Value
Developer	Wall-Facing Intelligence (ModelBest) and Tsinghua University OpenBMB Community
Target scenario	Large-scale distributed LLM pre-training (hundred-calorie to kilo-calorie level)
Supported hardware	NVIDIA H100 (SM90)/Huawei Ascend series
Verified model	MiniCPM4-0.5B, MiniCPM4-8B, MiniCPM5-1B
Training framework scale	Supports DP-only (0.5B) and TP=2 / DP=4 (8B)
Code writing method	100% AI Agent Loop self-generated, zero human manual editing
Open Source License	Apache 2.0
Training throughput	MFU on H100 reaches 44.13%, surpassing Megatron-LM by about 10%
Measured cluster	64 H100 GPUs, BF16 accuracy, Distributed Data Parallel
Harness component	Will be open source soon (scaffolding that drives Agent Loop’s automatic output training framework)

ForgeTrain's MFU (Model FLOPS Utilization) 44.13% was measured by training MiniCPM4-0.5B in 64× H100, BF16, pure DP mode. For comparison, the MFU of Megatron-LM v0.15 under the same hardware configuration is about 40%. This means that with the same investment in computing power, ForgeTrain can save about 10% of GPU computing costs, or train a larger model under the same budget. For a 64-card cluster, a 10% computing power savings translates into tens of thousands of dollars in power and machine time costs.

User and market recognition

ForgeTrain has received widespread attention from academia and industry since its release. Its core value is that it has verified for the first time the proposition that "AI Agent can independently write a production-level training framework."

• Academic Approval: The project was incubated by the Natural Language Processing Laboratory of Tsinghua University, and the relevant methodology has been published in the submission stage of the top conference. The preprint of the paper quotes Forge Engineering's three-stage methodology, which provides a reproducible theoretical framework for AI autonomous writing infrastructure.
• Industrial Implementation: Wall-Facing Intelligence has used ForgeTrain to completely run through the pre-training process of MiniCPM5-1B on the Huawei Ascend chip. This model ranks first in the world in the AA list below 2B in size. This means that for the first time, domestic chips have completed production-level pre-training of large models through a self-written AI training stack, breaking the traditional path dependence of "the training framework must be optimized by human handwriting".
• Open Source Community: The GitHub repository (OpenBMB/ForgeTrain) is open source under the Apache 2.0 protocol. Community contributors include the Wallface core team and external developers. The v0.1.0 training engine is currently released, and the Harness orchestration framework (Agent Loop automated scaffolding) is in the Coming Soon state.
• Industry Benchmarking: Compared with competing products such as NVIDIA VibeTensor (marked not for production), Anthropic C compiler OpenAI Harness, ForgeTrain is the only AI generation framework that is simultaneously usable at the production level, has excellent performance, and is completely open source. This has important reference value and engineering demonstration significance in the context of the industry where major manufacturers such as Meta are exploring "AI writing AI".

Cost advantage

The cost advantage of ForgeTrain needs to be understood from three levels:

Comparative dimensions	ForgeTrain	Megatron-LM (NVIDIA)	Handwriting training framework
Development cost	AI Agent is written independently, human investment is close to zero	Requires long-term maintenance by a team of senior system engineers	Team development cycle of months to years
Computing efficiency	44.13% MFU, about 10% higher than Megatron	~40% MFU (same hardware)	Depends on team optimization level, usually lower than mature frameworks
Hardware adaptation	H100 + Ascend dual platform, Harness can automatically adapt	NVIDIA GPU only	Redevelopment required for each new hardware adaptation
License	Apache 2.0 is completely open source and free	Open source but ecologically locked CUDA	Self-research costs are high
Training output	MiniCPM5-1B production-level model has been produced	Additional manpower is required for development and maintenance	Full link support is required

C-side users: ForgeTrain is not directly targeted at individual users. Its usage threshold is for research institutions or enterprises with multi-card GPU clusters. However, through open source and free methods, any team with computing resources can obtain and use it at zero cost.

Developers and Research Institutions: The open source license (Apache 2.0) allows free use, modification and commercial use. Compared to closed-source training frameworks that require the purchase of a commercial license, the cost of adoption of ForgeTrain is only the cost of cluster hardware and electricity. For academic research teams, this significantly lowers the threshold for large model pre-training experiments.

Enterprise Users: ForgeTrain’s breakthrough in domestic chip adaptation has important commercial value. Enterprises do not need to wait for N years to develop a self-developed CUDA compatible stack. They can directly use ForgeTrain to start large model training on domestic chips. For medium and large enterprises that are building AI infrastructure, adopting ForgeTrain can avoid being locked into a single GPU vendor while saving more than 10% of computing power costs.

Main functions

• Distributed pre-training engine: supports distributed collaborative training of hundreds to thousands of GPUs, and has completed production-level verification on a 64× H100 cluster. The training engine has five built-in CUDA Graph capture granularities (forward / step / step_full / step_optimizer / step_nccl_opt), which can be freely combined with BucketedGradReducer and sharding optimizer wgrad-overlap to provide flexible optimization space for training tasks of different sizes.
• Cross-hardware platform adaptation: Supports both NVIDIA H100 (SM90) and Huawei Ascend series chips. The MFU on the H100 reaches 44.13%, and the MiniCPM5-1B pre-training process has been completely passed on the Shengteng. This dual-platform support is of strategic significance for the construction of domestic computing power ecosystem - the model team can quickly iterate on H100 and complete production deployment on Ascend.
• AI Agent Loop independent development: The framework code is completely generated independently by the Coding Agent in the automatic loop mode, and humans only provide Harness specifications and context. Agent independently completes the complete development process of "reading reference implementation → writing code → starting training → parsing logs → root cause location → fixing patches → passing access control → submitting code". Each operator has been verified by real distributed training.
• Self-developed high-performance operator library: Contains 5 custom GEMM operators based on CuTeDSL, with a single operator MFU of up to 90%; self-developed FlashAttention implementation, the performance surpasses Transformer Engine / FlashAttention 3, and is the same as FlashAttention 4. The operator is compiled into a persistent cache through AOT C-export, which only incurs a few seconds of dlopen overhead in subsequent training.
• Bit-by-bit consistency verification: The AI-generated framework and the reference implementation (Megatron-LM) produce exactly the same numerical results under the same input. This is achieved through the second stage of "bit-by-bit alignment" of Forge Engineering - the Agent accurately reproduces the forward/backward propagation results of the reference implementation under Harness constraints, ensuring correctness before entering the performance optimization stage.
• Automated Evaluation Harness (coming soon): Built-in automated testing and performance evaluation

The system converts "Running right" and "Running fast" into criteria that the machine can automatically judge. Harness also includes an Agent Loop orchestration solution, allowing any team to reproduce the complete process from bit-by-bit alignment to performance overshoot.

Model and version evolution

ForgeTrain is currently in the initial release stage of v0.1.0, and the project roadmap is clear:

Version	Release Date	Content Covered
v0.1.0 (current)	2026-05	The training engine is publicly released, supporting NVIDIA H100 · MiniCPM4-0.5B (DP-only) and MiniCPM4-8B (TP=2). Contains training engine source code, 5 CuTeDSL custom GEMMs, and self-developed FlashAttention.
Harness (coming soon)	To be determined	Agent Loop orchestration scaffolding allows Coding Agent to automatically generate the core specifications of the training framework. It is a key component to achieve the reproducibility of "AI writing AI".
Huawei Ascend version	On the roadmap	MiniCPM5-1B Ascend training framework has been put into practice and the official version is to be released.
Self-generated Harness	On the roadmap	The training framework self-generates Harness scaffolding to achieve recursive self-improvement.

v0.1.0 is the first milestone of the project - verifying the feasibility of independently writing a production-level training framework for AI Agent. The next milestone is to release the Harness component to enable third-party teams to reproduce the training framework production process of Agent Loop instead of just using the already produced training engine.

Technical advantages

ForgeTrain's technical architecture revolves around a core innovation link: Forge Engineering methodology → Independent development of Agent Loop → High-performance training engine → Cross-hardware adaptation.

Forge Engineering three-stage methodology

Forge Engineering is the underlying methodology of ForgeTrain, which is divided into three stages:

1. Establishing standards (Harnessing): Collect key operating data from the reference training stack (Megatron-LM), build automated evaluation Harness, and define correctness and performance benchmarks. Harness is not a static document, but a set of executable specifications - including data stream truncation, numerical comparison, performance thresholds and other machine-decidable standards.
2. Bit-for-Bit Replication: The AI ​​Agent generates a training framework that is consistent bit by bit with the reference implementation under Harness constraints. This stage does not pursue performance, only correctness - if the output of the code produced by the Agent is not completely consistent with the reference implementation under the same input, it will be automatically rolled back and repaired.
3. Performance Surpassing: Lift the binary consistency restriction and switch to performance-oriented Harness, allowing the AI ​​Agent to independently iteratively optimize in a larger operator combination space, and ultimately surpass the reference implementation written by humans in speed. This step is key - the same Harness protocol can forge completely different proprietary optimization implementations on different hardware.

Agent Loop independently developed

Different from the traditional "humans write code → compile → debug" cycle, ForgeTrain's code is completed by Coding Agent in a fully automated cycle:

LLM (Coding Agent) → Read Harness protocol → Generate operator code → torchrun starts training → Parse log → Root cause location → Modify code → Pass numerical gate → Submit to exports/

The biggest feature of this cycle is that the generation of each operator has been verified by real distributed training. During the development process, Agent independently enumerated and benchmarked multiple operator implementation variants such as CuTeDSL / cuBLAS / Triton / TransformerEngine and dozens of communication + CUDA-Graph capture combinations, and finally selected the combination with the best performance as the default configuration for production.

High performance and engineering optimization

• 44.13% MFU implementation mechanism: CuTeDSL custom GEMM through CUDA Graph five capture granularities (single operator MFU reaches 90%), Triton fusion kernel (CE fwd+bwd / SwiGLU / RMSNorm+residual / RoPE / fusion Adam + parameter synchronization), and communication-compute overlap (comm-compute overlap), in pure DP mode, approaching the theoretical limit of the hardware.
• Zero abstraction loss: Unlike traditional training frameworks that maintain versatility through abstraction layers, ForgeTrain retains versatility in the Harness protocol and high performance in each forge. This means that the code generated for a specific hardware and model size does not contain any unnecessary abstraction overhead.
• Consumer Development Concept: Unlike traditional software that treats code as an "asset" for long-term maintenance, Forge Engineering unbundles code into deeply customized products forged on demand. When the hardware is replaced or the model is upgraded, there is no need to modify the old code, but directly run Harness re-forging.

How to use

The use of ForgeTrain is divided into two levels: using the produced training engine (currently available) and using the Harness autonomous forging training framework (to be released soon).

Using the training engine (v0.1.0)

Entrance	Address	Description
GitHub repository	https://github.com/OpenBMB/ForgeTrain	Main repository, containing training engine source code and complete documentation
Training engine documentation	exports/train_engine_0.5B/README.md	Complete CLI documentation, configuration reference, performance baseline of 0.5B model training engine
Training engine documentation	exports/train_engine_8b/README.md	Complete documentation for 8B model training engine

Requirements: Python ≥ 3.11 · CUDA 12.x · PyTorch ≥ 2.4 · NVIDIA H100 80GB (SM90). Full pre-training requires 8× H100, and early alignment stages can be run on a single card.

Typical steps:

1. Clone the repository and install dependencies:

   git clone https://github.com/OpenBMB/ForgeTrain.git
   cd ForgeTrain/exports/train_engine_0.5B
   pip install -e .
   pip install datasets transformers

2. Verify installation:

   PYTHONPATH=src python -c "from training_engine_tensor import config; print('OK')"

   Expected output: OK

3. Precompiled operator (first run):

   PYTHONPATH=src CUSTOM_GEMM=1 OP_ATTENTION=v1 python scripts/precompile_ops.py

   This command warms up the AOT export and cpp_extension build, compiling the 5 CuTeDSL GEMMs into a persistent cache.

4. Single node 8× H100 training:

   torchrun --standalone --nproc-per-node=8 \
    -m training_engine_tensor pretrain \
    --num-steps 200 \
    --global-batch-size 1280 --micro-batch-size 10 \
    --seq-length 4096 \
    --hf-dataset openai/gsm8k \
    --hf-dataset-config main \
    --hf-text-template "Question: {question}\nAnswer: {answer}" \
    --tokenizer-path openbmb/MiniCPM4-0.5B \
    --save-dir ./checkpoints/run1

Forged with Harness (coming soon)

After the Harness component is released, the team can start independent forging in the following ways:

cd ForgeTrain/harness
bash agent-loop.sh # Start Agent Loop with zero manual intervention

This script will drive a Coding Agent to run in a loop, starting from reading the reference training stack, through bit-by-bit alignment and performance optimization, and finally producing a customized training framework.

Product Pricing

ForgeTrain is currently completely free and open source, licensed under the Apache 2.0 license.

• Open source and free: The complete source code and documentation of the training engine have been publicly released on GitHub, and can be obtained, used and modified without any charge. No additional authorization is required for commercial use.
• Harness component: Coming soon, also expected to be open source under the Apache 2.0 license.
• Enterprise Support: Facewall Intelligence provides ForgeTrain’s enterprise-level deployment and technical support services. Please contact the business owner for specific pricing. For enterprises that have large-scale GPU clusters and want to customize their training stack, they can negotiate enterprise-level services including Harness customization, chip adaptation optimization, and expert onsite services.
• Cloud/Hosted Service: Currently there is no hosted training service, users need to manage the GPU infrastructure by themselves. The possibility of Facewall Intelligence launching a model training platform based on ForgeTrain cannot be ruled out in the future.

Overall, ForgeTrain’s pricing is extremely friendly to the developer community—you get a production-grade training framework at zero purchase cost. The hidden costs for enterprises mainly include hardware investment and operation and maintenance manpower for GPU clusters, but these costs cannot be avoided when using any pre-training framework.

Application scenarios

• Large model pre-training and fine-tuning: Directly replaces human-written training frameworks such as Megatron-LM and DeepSpeed, and is used for full pre-training and instruction fine-tuning of production-level large models. ForgeTrain provides about 10% MFU improvement under the same hardware, which is a significant computing power saving for large model training that often requires hundreds of cards and takes weeks. The expected benefits are training a larger model with the same budget, or shortening the training cycle with the same model size.
• Domestic Computing Power Adaptation: ForgeTrain is the first self-written AI training framework that runs through the complete pre-training process on Huawei's Ascend chip. For domestic chip manufacturers and companies that use domestic computing power, ForgeTrain can be used to quickly forge exclusive high-performance training stacks for Ascend and other chips without investing ten years in building another CUDA ecosystem. The expected benefit is that the time from "chip tape-out" to "software availability" is greatly shortened.
• AI Research Acceleration: ForgeTrain makes it possible to "shift the annual capacity growth of large models from a function of human scale to a function of computing power scale". The research team uses Harness to iterate independently in a larger search space and quickly experiment with new training methods, model architectures, and optimization strategies. The expected benefits will shift the team’s energy from training framework maintenance to model algorithm innovation.
• Software Engineering Paradigm Experiment: As the first industrial-level instance of Forge Engineering, ForgeTrain provides a reference methodology and architectural reference for the automatic generation of AI for other complex systems (compilers, databases, operating systems, deep learning runtimes). Research teams can explore autonomous development of AI in their own fields based on ForgeTrain’s Harness model.
• End-side model development: Wall-face Intelligence has used ForgeTrain to train MiniCPM5-1B, ranking first in the world in scale below 2B on the AA list. This proves that ForgeTrain is also suitable for the development of high-efficiency models on the end-side - teams that have needs for end-side deployment scenarios such as mobile IoT can

Take this training path directly.

Applicable people

• Large Model Training Engineers and Researchers: This is the core user group of ForgeTrain. They can directly use the training engine for large model pre-training, or they can delve into the operator code generated by Agent Loop to learn the implementation of a high-performance training framework. ForgeTrain's open source code base itself is an executable textbook on "H100 High Performance Training Project".
• AI Infrastructure Team: Technical teams responsible for enterprise AI training infrastructure can use ForgeTrain to reduce computing power costs (about 10% MFU improvement compared to Megatron-LM), while avoiding being locked into a single chip vendor through cross-hardware support. For teams investigating training framework options, ForgeTrain’s Apache 2.0 license and open source community ecosystem make it a low-risk, long-term option.
• Domestic chip ecosystem developers: Developers who write training stacks for domestic chips such as Ascend and Cambrian can use ForgeTrain's Harness methodology to quickly create exclusive training frameworks for domestic chips instead of starting from scratch. Harness' "protocol-driven, automatic forging" model reduces the labor cost and time cycle of chip adaptation.
• AI Agent and automation technology enthusiasts: ForgeTrain is the cutting-edge practice of "AI making AI". Developers interested in Agent Loop, automatic code generation, AI self-improvement and other technical directions can delve into its implementation mechanism. Agent-Friendly Quick Deploy is designed so that Coding Agent can complete the complete deployment and acceptance process independently.

Not suitable for the crowd: ForgeTrain is not suitable for the following scenarios - individual developers who do not have high-performance GPU clusters (single card or even no-card users cannot realize its value); teams that need ready-to-use SaaS training services (ForgeTrain is a self-service open source framework and does not provide hosting services); teams that have no understanding of the underlying CUDA / PyTorch and are unwilling to invest in engineering operation and maintenance; need complete MLops Enterprises that use the platform (dataset management, experiment tracking, model registration, etc.) should use it in conjunction with the traditional MLops platform, not as a replacement.

Summary and Outlook

ForgeTrain’s core competitiveness lies in its ability to realize the complete transition of “AI-made AI” from concept to production-level implementation for the first time. It is not a proof-of-concept demo, but a production-grade training framework that has produced real model weights on a 64-card cluster. Its Forge Engineering methodology provides a reproducible path for AI to independently write infrastructure software - from standard establishment to bit-by-bit alignment to performance overshoot, each step has clear passing standards.

Current Limitations:

• Limited range of supported models: v0.1.0 only verified two configurations: MiniCPM4-0.5B (DP-only) and MiniCPM4-8B (TP=2). Models with larger scale or different architectures require additional adaptation work.
• The Harness component is not yet open source: currently users can only use the training engine that has been produced, and are not yet able to experience the full process of "independent forging". The release of the Agent Loop orchestration framework is the most anticipated feature by the community.
• High hardware threshold: complete training requires H100 80GB or above GPU, and a single card cannot complete production-level pre-training, limiting the participation of individual developers.
• The community ecosystem is in its early stages: Compared with Megatron-LM and DeepSpeed, which have mature communities and rich documentation, ForgeTrain’s tutorials, cases and third-party contributions are still being accumulated.
• Lack of MLops integration: ForgeTrain focuses on the training engine itself and does not include MLops capabilities such as experiment management, model registration, and deployment, and needs to be used in conjunction with other tools.

Procurement and Adoption Risk Assessment: For companies planning to purchase ForgeTrain, it is recommended to adopt the strategy of "pilot first, small-scale verification, and then expansion". First, verify whether the stability and performance data of the framework are consistent with the official on a small training task (such as a single node 8-card MiniCPM4-0.5B); after confirmation, expand to larger-scale model training. Special attention needs to be paid to the following aspects: the official release time of Harness (which will determine whether an autonomous forging workflow can be implemented), Wallface Intelligence’s charging model and service SLA for enterprise-level support, and the sustainability of the long-term maintenance of the project. From a technical risk perspective, ForgeTrain is based on mature open source components (PyTorch, CUTLASS, FlashAttention), and the core training engine has high credibility; the main risk lies in the delivery quality of the Harness component and whether the community ecosystem can continue to grow. Overall, ForgeTrain is a project with great potential and has initially proven its technical feasibility, but it still needs a long time to go through ecological construction before it becomes a mainstream training framework.

Related tools: hugging-face, replicate

Version Info

• Initial Release ：The first publicly released version supports distributed pre-training of MiniCPM4-0.5B and MiniCPM4-8B on NVIDIA H100; includes the training engine source code, 5 CuTeDSL custom GEMM operators, and self-developed FlashAttention implementation; the Harness component (Agent Loop orchestration framework) is to be released. （2026-05）
• Initial Release ：The first publicly released version supports distributed pre-training of MiniCPM4-0.5B and MiniCPM4-8B on NVIDIA H100. （2026-05）