LLaMA-Factory is a unified LLM/VLM efficient fine-tuning framework that supports LoRA, QLoRA, and full-parameter training and evaluation processes, and is suitable for rapid iteration by model teams.

LLaMA-Factory

Core parameters and statistics

LLaMA-Factory is a unified and efficient fine-tuning framework, officially positioned as "Unified Efficient Fine-Tuning of 100+ LLMs & VLMs", covering the complete training link from instruction fine-tuning to preference alignment.

Projects	Public Information
Official positioning	Unified Efficient Fine-Tuning of 100+ LLMs & VLMs
Open Source License	Apache-2.0
GitHub Stars	~73,400
GitHub Forks	~9,000
Open Issues	~980
Contributors	290+
Number of releases	36 Releases
Latest version	v0.9.5 (2026-05-30)
First created	2023-05-28
Official documentation	llamafactory.readthedocs.io
Supported models	100+ LLM and VLM
Training strategy	Pre-training, SFT, Reward Modeling, PPO, DPO, KTO, ORPO, SimPO
Quantization bits	2/3/4/5/6/8-bit QLoRA

Product Boundary: LLaMA-Factory focuses on training and evaluation links and does not replace online inference gateways, enterprise application layer workflow platforms, or data annotation tools. If all you need is to call off-the-shelf inference APIs without training or fine-tuning, LLaMA-Factory will significantly increase implementation complexity.

Market position: In the open source training framework, LLaMA-Factory, axolotl, Hugging Face TRL, and Unsloth form the main competitive landscape. The core difference of LLaMA-Factory lies in "zero code entry + widest model coverage", rather than training speed or ultimate optimization of a single model. The following table shows the key differences from similar frameworks:

Dimensions	LLaMA-Factory	axolotl	Hugging Face TRL	Unsloth
Code threshold	Zero code (Web UI + YAML)	YAML configuration required	Python script required	Python script required
Model coverage	100+ (including VLM/multimodal)	~50	Unlimited (depends on Transformers)	~20
Training strategies	8 types (including RLHF)	~5 types	6 types	3 types
Quantitative support	2-8bit full series	4/8-bit	4/8-bit	4-bit
Web UI	Built-in LLaMA Board	None	None	None
Distributed training	DeepSpeed + Megatron	DeepSpeed	DeepSpeed	DeepSpeed
The lowest hardware threshold	QLoRA 2bit only requires 4GB	8GB+	8GB+	6GB+
Open Source License	Apache-2.0	Apache-2.0	Apache-2.0	Apache-2.0

User and market recognition of LLaMA-Factory

Community size: 73,400+ stars and 9,000+ forks on GitHub, making it one of the most watched projects in the field of open source fine-tuning. 290+ contributors and 36 Releases indicate that the project has passed the early experimental stage and formed a relatively stable community collaboration mechanism.

Corporate Endorsement: The official README clearly lists being adopted by Amazon (used on SageMaker HyperPod for multi-modal financial document information extraction), NVIDIA (RTX AI Toolkit integration), Alibaba Cloud (PAI-DSW image integration), etc. These cases show that LLaMA-Factory has been included in the AI ​​infrastructure ecosystem of leading cloud vendors.

Academic citation: The paper "LLaMAFactory: Unified Efficient Fine-Tuning of 100+ Language Models" was published in ACL 2024 (System Demonstrations) and has academic recognition.

Undisclosed items: Officially undisclosed commercial customer scale, paid conversion indicators and revenue data. Market-based data shall be subject to official subsequent disclosures.

Cost advantage: Using open source to lower the computing power threshold for model customization

LLaMA-Factory itself is completely open source and free, and the core cost of use comes from computing resources and human investment. The following starts with the three-tier cost structure.

C-side/Individual: Individual researchers can run QLoRA fine-tuning on a single-card GPU (such as RTX 4090 24GB), with video memory requirements as low as 6GB (2-bit QLoRA). Paired with open source data sets on Hugging Face or ModelScope, the computing power cost of a single fine-tuning experiment can be controlled at the level of tens of yuan. The free version of Google Colab lets you run small-scale experiments.

Developers/API Team: The obvious cost is GPU instance leasing. LoRA/QLoRA can reduce the cost of single fine-tuning to 10%-30% of full parameter training. The following table shows the estimated hardware costs under different configurations (taking mainstream cloud GPU on-demand prices as an example):

Fine-tuning strategy	Video memory requirements (7B model)	Video memory requirements (70B model)	Single training time estimate (7B)	Cloud GPU cost estimate (7B)
Full parameters (bf16)	~60GB	~600GB	4-8 hours	$10-30
LoRA (16bit)	~16GB	~160GB	2-4 hours	$5-15
QLoRA (8bit)	~10GB	~80GB	2-4 hours	$3-10
QLoRA (4bit)	~6GB	~48GB	3-5 hours	$3-8
QLoRA (2bit)	~4GB	~24GB	4-6 hours	$2-6

Note: The above table is an estimated value. The actual cost varies depending on the model architecture, sequence length, data set size, GPU model (A100/H100/4090) and cloud vendor pricing strategy. The real-time billing provided by the cloud provider shall prevail.

Enterprise/privatization: Enterprises can deploy private computing power, but they need to bear additional costs:

• Data governance cost: The human investment in training data cleaning, annotation quality review, and copyright compliance review usually exceeds the cost of computing power.
• Experiment Management Cost: Multi-version experiment comparison and super-parameter search Checkpoint management require supporting experiment tracking infrastructure (such as W&B, MLflow).
• Cluster Scheduling Cost: Multi-card/multi-node training requires the configuration of DeepSpeed ​​or Megatron, which involves operation and maintenance capabilities.
• Model Governance Cost: Fine-tuned model version management A/B evaluation, security alignment review.

Hidden benefits: The unified training entrance can reduce the maintenance cost of "one training script for each model" by about 60%-80% (estimated). The team does not need to rewrite the data loading, training cycle and evaluation logic when switching models.

Main functions of LLaMA-Factory

• Unified fine-tuning portal for 100+ models: Covering mainstream model families such as LLaMA, Qwen, DeepSeek, Gemma, Mistral, GLM, and Phi, you can switch models with a single set of configuration portals. In actual use, switching models only requires modifying the model_name_or_path field in YAML, which greatly reduces adaptation script writing.
• 8 training strategies: from pre-training (Pre-training), supervised fine-tuning (SFT), reward modeling (Reward Modeling) to PPO/DPO/KTO/ORPO/SimPO preference alignment, covering the entire link from instruction fine-tuning to value alignment. Teams can cascade multi-stage training pipelines within the same framework.
• Full coverage of parameter efficient fine-tuning: Supports 10+ parameter efficient methods such as LoRA, QLoRA (2/3/4/5/6/8-bit), DoRA, LongLoRA, LoRA+, LoftQ, PiSSA, etc. LoRA can fine-tune 7B models on 16GB of video memory, and 2-bit QLoRA can even run on 4GB of video memory, greatly lowering the experimental threshold.
• Advanced optimization algorithm integration: Integrate cutting-edge training optimizers such as GaLore, BAdam, APOLLO, Adam-mini, Muon, OFT, Mixture-of-Depths, etc. These algorithms reduce training costs from dimensions such as gradient compression (GaLore), memory optimization (APOLLO), and parameter grouping (BAdam).
• Built-in engineering acceleration technology: Native integration of FlashAttention-2, Unsloth, Liger Kernel, KTransformers and other acceleration libraries can be automatically enabled without changing the training script. FlashAttention-2 can speed up long sequence training by 2-4 times.
• Experiment monitoring and visualization: The built-in LLaMA Board (Gradio Web UI) provides real-time training curve, loss/learning rate monitoring; it also supports external experiment management tools such as WandB, TensorBoard, MLflow, and SwanLab.
• Multi-modal training support: Not only supports text LLM, but also supports fine-tuning of visual-language models (VLM) such as LLaVA, Qwen2.5-VL, InternVL, MiniCPM-V, and audio understanding models such as Qwen2-Audio and Qwen2.5-Omni.
• Integrated inference deployment: After training is completed, you can directly start the OpenAI style API service (supports vLLM or SGLang backend), and deploy the LoRA weights produced by the training or the merged full model as an inference endpoint with one click, reducing the work from training to inference.

process switching costs.

Synergy: The above features do not exist in isolation. "Multi-model coverage + unified configuration" means that the team does not need to rewrite the data loading and training cycles when switching models; "Efficient parameter fine-tuning + acceleration library" allows low-resource teams to fine-tune 70B+ models; "Training + API deployment integration" compresses the engineering links from experiment to online in one framework, avoiding weight conversion and adaptation work between training scripts and inference services.

Model and version evolution of LLaMA-Factory

LLaMA-Factory has experienced rapid iterations since its creation on 2023-05-28. The main line of version evolution revolves around the three directions of "model coverage expansion, training strategy enrichment, and engineering infrastructure upgrades."

Mainline release

Version	Date	Key Changes
v0.9.5	2026-05-30	Added Qwen3.5/3.6, Gemma 4, Transformers v5 adaptation; the latest of 36 Releases
v0.9.4	2025-12-31	Annual mainline version, stability and capability integration, covering multi-model fine-tuning unified entrance
v0.9.3	2025-06-16	Extended model support for Llama 4, Gemma 3, Qwen3, InternVL3, Qwen2.5-Omni and other models
v0.9.2	~2025-03	Introducing Megatron-core training backend integrated OFT/OFTv2 algorithm support
v0.9.1	~2024-12	Continuously expand new models such as DeepSeek V3, optimize Web UI and experiment management
v0.8.x	2024-06/09	Establish a multi-model and multi-strategy infrastructure to support LLaMA 3, Qwen2, GLM-4, etc.

Early Milestones

• 2023-05-28: The project was created for the first time, with the initial focus on fine-tuning the LLaMA series.
• 2024-03: The paper was accepted by ACL 2024, and the academic influence was significantly improved.
• 2024-06: v0.8.0 was released, the multi-model architecture became stable, and the community stars exceeded 10,000.
• 2025-01: DeepSeek R1 was quickly adapted after its release, reflecting day-0 model follow-up capabilities.

Version evolution rules

The version rhythm of LLaMA-Factory shows the characteristics of "adaptation is completed within 1-7 days after the release of a new model". For example, DeepSeek R1 was released on 2025-01-20, and the project completed support within a few days. This speed of update is due to its plug-in model registration architecture - adding a model family usually only requires the provision of configuration files and templates, without modifying the core training code. For purchasers, this means that adopting LLaMA-Factory reduces the risk of tool chain changes due to model switching.

Technical advantages of LLaMA-Factory

Architecture design

LLaMA-Factory adopts a "unified configuration abstraction layer + plug-in model registration" architecture. The core training engine is encapsulated through PEFT (Parameter-Efficient Fine-Tuning) and Transformers libraries, and the upper layer describes the model path, data set, training strategy and hyperparameters through YAML configuration. The core value of this design is:

• Model-independent training interface: The training script is decoupled from the model. To add a new model, you only need to register the configuration file and dialogue template, without modifying the training cycle code.
• Strategies are pluggable: LoRA, QLoRA, full-parameter fine-tuning and other strategies are managed uniformly through the peft library, and the training strategy can be changed by switching configuration items without changing the data loading and evaluation process.
• Multiple backend support: In addition to native PyTorch DDP, it supports distributed strategies such as DeepSpeed ​​ZeRO (Stage 1-3) and Megatron-core, which is suitable for single-machine multi-card to multi-machine multi-card scenarios.

Why can LLaMA-Factory achieve "low-resource fine-tuning of large models"?

The core mechanism has three layers of superposition:

1. Quantization compression layer: Through quantization libraries such as bitsandbytes, AQLM, AWQ, GPTQ, HQQ, EETQ, etc., the model weight is compressed from 16-bit to 2-8-bit, significantly reducing the memory usage. Taking 4-bit QLoRA as an example, the video memory requirement of the 70B model is reduced from ~600GB to ~80GB, allowing a single 4×A100 to run.
2. Parameter efficient layer: LoRA/DoRA and other methods only train 0.1%-1% of the original parameters. The gradient calculation amount of back propagation is greatly reduced, and the training speed is increased by 2-5 times compared with full parameter fine-tuning.
3. Computing acceleration layer: FlashAttention-2 increases the training speed of the attention mechanism by 2-4 times through block calculation and IO-aware optimization; Liger Kernel fuses multiple cores to reduce video memory reading and writing; Unsloth further improves LoRA training throughput through manually optimized CUDA kernels.

Actual effect: These three layers of superposition compress the full-parameter 70B fine-tuning that originally required 8×A100 (~$100/hour) to a QLoRA experiment of 1×RTX 4090 (~$1/hour). The training cost is reduced by about two orders of magnitude, allowing individual developers and small and medium-sized teams to participate in large model customization.

Adaptation boundary

Scenario	LLaMA-Factory Advantages	Unfit Boundary
Instruction fine-tuning (SFT)	Zero threshold, configure and run	Not applicable when in-depth customization of training cycles (such as custom loss functions) is required
Preference alignment (DPO/PPO)	Complete built-in process	When large-scale RLHF (tens of thousands of prompt sampling) is required, an external RL framework needs to be used
Multi-modal (VLM) fine-tuning	Covers mainstream VLM architecture	Limited video/3D modality training support
Distributed training	DeepSpeed/Megatron integration	Large-scale pre-training (training hundreds of billions of parameter models from scratch) is not its design goal
Production inference deployment	Built-in vLLM/SGLang API	For high-concurrency production inference, it is recommended to use a dedicated inference engine (such as TGI, vLLM independent deployment)

Usage path of LLaMA-Factory

LLaMA-Factory provides four entrances, suitable for teams with different technical backgrounds and scenarios:

Entry method	Suitable for the crowd	Start command	Prerequisites
YAML command line	Developers with ML engineering experience	llamafactory-cli train examples/train_lora/qwen3_lora_sft.yaml	Python 3.11+, PyTorch 2.0+, GPU
Web UI (LLaMA Board)	Algorithm researcher, novice	llamafactory-cli webui	Same as above, automatically open the Gradio interface
Docker Containers	DevOps/Platform Team	docker run -it --gpus=all hiyouga/llamafactory:latest	Docker + NVIDIA Container Toolkit
OpenAI style API	Inference deployment scenario	API_PORT=8000 llamafactory-cli api examples/inference/qwen3.yaml	Weight file after training is completed

Typical steps: 3 commands to complete a fine-tuning

The core workflow of LLaMA-Factory can be condensed into 3 commands:

# 1. Training: LoRA fine-tuning Qwen3-4B
llamafactory-cli train examples/train_lora/qwen3_lora_sft.yaml

# 2. Dialogue test: load the trained LoRA weights for interaction
llamafactory-cli chat examples/inference/qwen3_lora_sft.yaml

# 3. Export merge: merge LoRA weights into the base model
llamafactory-cli export examples/merge_lora/qwen3_lora_sft.yaml

Typical YAML configuration example

# qwen3_lora_sft.yaml
model_name_or_path: Qwen/Qwen3-4B-Instruct
template: qwen
stage: sft
finetuning_type: lora
lora_target: all
dataset: identity,alpaca_en_demo
dataset_dir:data
cutoff_len: 1024
learning_rate: 1.0e-4
num_train_epochs: 3.0
per_device_train_batch_size: 4
gradient_accumulation_steps: 4
fp16: true

Configuration parameter description: finetuning_type controls the fine-tuning strategy (lora/freeze/full), lora_target specifies the module injected by LoRA, dataset uses the dataset name registered in data/dataset_info.json.

Suggestions for landing path

When actually introducing the LLaMA-Factory team, it is recommended to proceed in three stages:

1. Pilot (1-2 weeks): Select a specific business task (such as customer service intent classification, code completion), use the official template + public data set to run through the entire SFT process, and verify the training benefits.
2. Contrast (2-4 weeks): Make an A/B comparison with the existing solution (or Prompt engineering solution) on the pilot task. The quantitative indicators include accuracy improvement, delay change and GPU cost. Key acceptance point: Whether the performance of the trained model on the validation set is improved by more than 5%, and the computing power cost is within an acceptable range.
3. Extension (January-March): Cover more business scenarios, establish a standardized data preparation → training → evaluation → deployment pipeline, and introduce experiment management and model version control.

Product Pricing

LLaMA-Factory itself is completely open source and free (Apache-2.0 license), but the costs involved in the use process are concentrated on the following levels:

Cost Type	Individual/Researcher	Development Team	Enterprise
Software License	Free	Free	Free
GPU computing power	On-demand $0.5-5/hour	Monthly $500-5000	Private cluster + operation and maintenance
Data preparation	Using public data sets	Annotation + cleaning manpower	Annotation platform + compliance review
Experiment management	Colab/W&B free version	W&B/MLflow hosting	Self-built experiment platform
Deployment and operation	Docker stand-alone	K8s + CI/CD	Inference gateway + monitoring
Model evaluation	Manual + simple script	Automated evaluation pipeline	Multi-dimensional evaluation + red team testing

Open source license restrictions: The Apache-2.0 license allows commercial use, but you need to pay attention to the license restrictions of the fine-tuned base model itself (such as LLaMA's Meta license, Gemma's Google license, Qwen's Alibaba license). The base model license may contain additional terms (such as the need to apply for commercial authorization if monthly active users exceed the threshold, prohibition of use in specific industries, etc.), and these restrictions are independent of the LLaMA-Factory license.

Hidden Cost Tip: Although LLaMA-Factory lowers the technical threshold for single fine-tuning, data quality management is often the largest cost item in actual projects - models trained with low-quality data require repeated iterative fine-tuning. Although the cost of a single experiment is low, the cumulative GPU and labor costs may be far higher than expected.

Application scenarios of LLaMA-Factory

• Internal model customization: Inject industry corpus (finance, law, medical, manufacturing) into general models to improve the accuracy of understanding vertical scenarios. For example, financial institutions need to make models understand financial reporting terminology and regulatory compliance requirements. Injecting 500-5000 field conversations through SFT can significantly improve this. In such scenarios, LLaMA-Factory’s QLoRA strategy can reduce customization costs from hundreds of thousands of yuan to thousands of yuan.
• Knowledge distillation and small model enhancement: Use large models (such as DeepSeek-R1, GPT-4o) to generate high-quality instruction data, and then use LLaMA-Factory to fine-tune small models (such as Qwen3-4B, Gemma-2B) to improve the performance of small models on specific tasks while maintaining inference speed. This is especially valuable in offline or edge deployment scenarios where calls to external APIs are not allowed.
• Multimodal model customization: Domain multimodal fine-tuning of visual-language models such as Qwen2.5-VL and InternVL, suitable for tasks such as chart understanding, OCR post-processing, and medical imaging report generation. LLaMA-Factory uniformly covers text and multi-modal training entrances, eliminating the need to switch between multiple frameworks.
• Preference Alignment and Security Control: Use preference alignment strategies such as DPO/PPO/KTO to tune model output to comply with business specifications and security requirements. It is suitable for scenarios such as content review, customer service language compliance, and unified brand tone. Preference alignment requires paired preference data sets (good/bad answers), and LLaMA-Factory's built-in KTO method can complete the alignment with only "good answer" data.
• Research and Baseline Construction: Academic teams need to run unified evaluations on multiple models. LLaMA-Factory's unified configuration system can standardize experimental variables (models, data sets, hyperparameters) and improve the reproducibility of experimental results.

Key points of scene verification: Before choosing LLaMA-Factory, be sure to confirm "whether fine-tuning is really needed". If Prompt engineering + RAG suffices, the cost-output ratio of fine-tuning may be negative. The window of value for fine-tuning is in scenarios where "a specific format/style/knowledge needs to be learned and cannot be accomplished through contextual prompts."

Applicable groups of LLaMA-Factory

• Algorithm Engineers and ML Researchers: Need to frequently compare the performance differences of different models on the same task, or verify the effect of new training strategies. The unified entrance to LLaMA-Factory can standardize experiments and reduce bias in conclusions caused by differences in scripts. For this type of user, the YAML command line is more efficient than the web UI.
• MLOps and platform team: The training process needs to be standardized, automated, and integrated with platform tools such as CI/CD, experiment tracking, and model registration. LLaMA-Factory’s API deployment capabilities and Docker images make it suitable for embedding into MLOps pipelines.
• Independent developers and small teams: Scenarios with limited budget but need for model customization. With QLoRA + cloud GPU, a domain fine-tuning experiment can be completed for a few dollars per day. The combination of Google Colab Free Edition + LLaMA-Factory makes zero-budget startup possible.
• Academic Research Team: A reproducible open source training framework is needed to support thesis experiments. The LLaMA-Factory paper was published in ACL 2024. The community is active and it is suitable as a teaching and experimental platform.

Does not fit boundaries:

• If the team's goal is just to call an off-the-shelf model API, without training or fine-tuning, LLaMA-Factory is not suitable.
• If the team has needs for deeply customized training cycles (such as custom loss functions, special scheduling strategies), the abstraction layer of LLaMA-Factory may become a limitation, and it is better to use PyTorch + Transformers directly.
• If the team needs to pre-train a 100 billion parameter model from scratch, it is recommended to use Megatron-LM or NeMo directly. The design goal of LLaMA-Factory is fine-tuning rather than large-scale pre-training.
• If the team does not have GPU resources or cloud budget, even if QLoRA lowers the threshold, a single experiment still requires at least 4GB of video memory, making pure CPU training unrealistic.

Summary and Outlook

The core value of LLaMA-Factory is to unify the training requirements of "multi-model, multi-strategy, and multi-modality" into one framework, replace script coding with YAML configuration, and reduce model customization from "requiring handwritten training instructions" to "modifying configuration files". It has obvious advantages in the ecological follow-up speed (day-0 adaptation of new models) and community scale in the open source training framework. The support of multiple hardware backends such as Huawei Ascend NPU and AMD ROCm also expands its deployment boundaries.

Current Limitations:

• Training is deeply dependent on the Hugging Face ecosystem, and has limited support for models with non-Transformers architecture.
• The implementation of preference alignment (PPO/DPO) is suitable for small and medium-scale experiments, and large-scale production-level RLHF still needs to be supplemented by external reinforcement learning frameworks.
• The official documentation site is still marked as WIP (Work in Progress), and some advanced functions are not well documented. When encountering problems, you need to rely on GitHub Issues and community discussions.
• Commercial support and SLA are not disclosed, and enterprise-level procurement needs to evaluate whether community support meets production requirements.

Procurement/Adoption Risk Assessment:

• If the team already has MLOps infrastructure (model registration, experiment tracking, GPU cluster scheduling), LLaMA-Factory can be quickly integrated as a lightweight training layer. It is recommended to use 1-2 business scenarios as a pilot for 2-4 weeks, focusing on verifying the three indicators of training effect improvement, computing power consumption and manual intervention frequency.
• If the team builds training capabilities from scratch, the hidden costs (data governance, experiment management, model evaluation) may far exceed the learning cost of the tool itself. It is recommended to first confirm whether the data quality and evaluation standards are in place before selecting a framework.
• Before deploying in production, be sure to check whether the base model license used for fine-tuning covers the expected usage scenarios, especially the monthly active user limit and commercial licensing terms. This part of the compliance risk is independent of LLaMA-Factory, but directly affects whether the training output can go online.

Related tools: Hugging Face, replicate

Version Info

• v0.9.5 ：The official release notes include Qwen3.5/3.6, Gemma 4 and Transformers v5 related adaptation updates. （2026-05-30）
• v0.9.4 ：The annual mainline version undertakes the stable iteration of multi-model fine-tuning capabilities. （2025-12-31）
• v0.9.3 ：The release notes cover Llama4, Gemma3, Qwen3, InternVL3, Qwen2.5-Omni and other model support. （2025-06-16）