Framepainter
Free
FramePainter is an open source image editing tool included in ICCV 2025. It uses video diffusion prior (Stable Video Diffusion) to implement sketch-based interactive image editing. Users only need to draw sketches on the image, and AI can understand the editing intention and generate professional-level effects, performing well in both common editing scenarios and out-of-domain generalization scenarios.
FramePainter
Core parameters and statistics
| Project | Specifications |
|---|---|
| Product Name | FramePainter |
| Category | AI Visual Design / Image Editing |
| Delivery form | Desktop tool (Python/PyTorch) |
| Support Platform | Desktop |
| Supported languages | zh-CN, en-US |
| Target users | AI researchers, designers, creative workers |
| User scale | GitHub 406 Stars (academic project) |
| Pricing Model | Free (MIT Open Source) |
Platform coverage and user scale data are based on the official real-time page and third-party statistics.
User and market recognition
The FramePainter paper was accepted by ICCV 2025 and achieved leading SOTA results in the direction of interactive image editing. Compared to previous methods, FramePainter achieves better editing quality and generalization capabilities with significantly less training data. 406 Stars on GitHub, HuggingFace provides full pre-trained weights. It is included in many AI tool aggregation sites such as Art Weekly and KDJingpai. The proposed technical idea of "reconstructing image editing into image-to-video generation" provides a new paradigm reference for subsequent interactive editing research.
Cost advantage
| Cost Dimension | Description |
|---|---|
| Software licensing | MIT open source, completely free |
| Pre-trained weights | Free download from HuggingFace |
| Commercial use | MIT License, commercial use permitted |
| Hardware cost | Need to bring your own NVIDIA GPU (recommended ≥8GB video memory) |
The paper points out that due to the use of Stable Video Diffusion's video diffusion prior, the training data volume and computational cost are much lower than previous methods based on text-to-image diffusion models.
Main functions
- Sketch-based Editing: The user draws a sketch on the image to specify the editing intention, and AI automatically understands the semantics of the sketch and generates an image that conforms to the instructions. The thickness, position, and shape of the sketch directly affect the editing results without the need for precise masking.
- Click-based Editing: Specify the editing area and target through click operations, suitable for quickly adjusting local content of the image.
- Drag-based Editing: Drag the key points in the image to control changes in object posture, shape or position. Through video diffusion, you can understand the movement rules a priori and make the deformation more natural.
- Out-of-domain generalized editing: Core highlight - Using video diffusion prior, reasonable results can still be generated in editing scenarios not covered by the training data (such as changing a clown fish into a shark shape).
- Matching Attention Mechanism: Expand the receptive field and enhance the dense correspondence between the edited image and the source image token, ensuring content coherence after substantial editing.
Model and version evolution
| Version | Date | Key Changes |
|---|---|---|
| v1.0 (ICCV 2025) | ~2025-01 | Complete inference demo, HuggingFace weight, supports all editing modes |
| arXiv v1 | 2025-01-14 | The paper was released for the first time (2501.08225), and the technical solution was made public |
Technical advantages
- Problem Reconstruction Paradigm: Redefine interactive image editing as an "image-to-video generation" problem. By inheriting the video diffusion prior of Stable Video Diffusion, you can naturally understand the changing laws of object motion and significantly reduce training costs.
- Lightweight Sparse Control Encoder: Only a lightweight encoder is introduced to inject editing signals, the amount of new parameters is minimal, and training and inference are highly efficient.
- Matching Attention: Expand the cross-attention search range, establish dense point-by-point correspondence, and maintain content coherence even if posture or shape is greatly changed.
- Multi-modal editing signal unification: The same architecture supports multiple interaction methods such as sketching, clicking, dragging, etc., without the need to train different models separately.
How to use
| Entrance | Installation/Usage |
|---|---|
| GitHub source code | git clone → conda create → pip install → download weight → python app.py |
Download the pre-trained weights from HuggingFace (Yabo/FramePainter) into the checkpoints/ directory. During inference, stabilityai/stable-video-diffusion-img2vid-xt-1-1 is automatically downloaded as the base model.
Product Pricing
| Project | Cost |
|---|---|
| Software Licensing | Free (MIT License) |
| Pre-trained weights | Free (HuggingFace) |
| Commercial use | Permitted (MIT License) |
| Cloud service | No official cloud service, you need to deploy it locally |
Application scenarios
- Interactive Image Design and Creative Editing: Designers use sketches to quickly adjust image composition, replace objects, and change styles, eliminating complex masking and layer operations in traditional software.
- Advertising and marketing visual material production: The marketing team quickly generates multiple versions of advertising materials through drag and drop and sketches, shortening the process from requirement to drawing to minutes.
- Film and TV post-production and concept design: The ability to generalize outside the domain allows it to play a role in creative scenarios not covered by training data, making it suitable for movie storyboards and game concept design.
Applicable people
- Individual users: AI researchers and developers, providing complete training and inference codes, suitable as a baseline system for interactive image editing research.
- SME Team: Designers and creative workers need to quickly iterate on visual solutions.
- Large Enterprises: Not suitable - purely academic projects, without commercial-grade UI/UX polishing and customer service support.
- Unfit Boundary: Production environments that require pixel-level precise control (such as print finishing); batch processing scenarios that require high real-time performance; users without NVIDIA GPU.
Comparison of competing products
| Compare Dimensions | FramePainter | ControlNet | DragGAN |
|---|---|---|---|
| Core differences | Video diffusion prior, out-of-domain generalization | Conditional control generation | Drag and drop editing |
| Price | Free (MIT Open Source) | Free (Open Source) | Free (Open Source) |
| Cover scene | Interactive editing, creative design | Conditional generation | Posture editing |
| User evaluation | Strong generalization ability | Precise control | Intuitive interaction |
| Technical threshold | Medium (requires GPU + installation) | Medium (requires GPU) | Medium (requires GPU) |
Summary and Outlook
FramePainter takes the paradigm innovation of "reconstructing image editing into image-to-video generation" as its core, and uses video diffusion priors to significantly reduce the training cost of interactive image editing. The ability to generalize outside the domain is its key highlight.
Procurement/Adoption Risk Assessment: FramePainter is a purely academic research project with no commercialization team or enterprise-level support. Code updates and maintenance rely on the original author's personal time, and there is no SLA commitment. It is recommended to evaluate the risks of use in production projects, or wait for a third party to package it into a commercializable product/service. |---|---| | Product type | Sketch-based interactive image editing tool (academic open source project) | | Platform Support | Desktop (Python/PyTorch) | | Paper publication time | January 2025 (arXiv), included in ICCV 2025 | | Current version | 1.0 (ICCV 2025 corresponding version) | | Basic model | Stable Video Diffusion (SVD) | | Core Architecture | Sparse Control Encoder + Matching Attention | | Input method | Image + Sketch | | Edit signals | Draw, click, drag and other visual interactive operations | | License | MIT License | | GitHub Stars | 406 (as of July 2026) | | HuggingFace weight | Yabo/FramePainter |
FramePainter is the official implementation of papers included in ICCV 2025 jointly proposed by Harbin Institute of Technology (HIT) and Huawei's Noah's Ark Laboratory. It is not a commercial SaaS product, but an open source academic research project - redefining interactive image editing as an "image to video generation" problem, thereby inheriting the powerful temporal prior capabilities of the video diffusion model. The core insight is that video data naturally contains the changing process of objects under physical interaction. Modeling image editing as "pseudo video generation" between two frames can significantly reduce training costs and ensure timing consistency.
User and market recognition
FramePainter has gained widespread attention in the academic community and in the field of AI image editing:
- Academic Recognition: The paper was accepted by ICCV 2025 and achieved leading SOTA results in the direction of interactive image editing. Compared to previous methods, FramePainter achieves better editing quality and generalization capabilities with significantly less training data.
- Open Source Community: GitHub received 406 Stars (as of July 2026), complete pre-trained weights are provided on HuggingFace, and community developers can freely download and deploy.
- Media Coverage: Included and reported by multiple AI tool aggregation sites such as Art Weekly and KDJingpai.
- Technical influence: The proposed technical idea of "reconstructing image editing into image-to-video generation" provides a new paradigm reference for subsequent interactive editing research.
Cost advantage
FramePainter is a completely open source project (MIT license) with zero cost to use.
- Software Cost: No subscription or licensing fees required. The code is freely cloned from GitHub and the pre-trained weights are freely downloaded from HuggingFace.
- Hardware Cost: A computer with an NVIDIA GPU (recommended video memory ≥ 8GB) is required to run inference. Compared with cloud service pay-per-use commercial image editing tools, the marginal cost of local deployment approaches zero when used heavily.
- Training cost: The paper points out that due to the use of Stable Video Diffusion's video diffusion prior, FramePainter's training data volume and computational cost are much lower than previous methods based on text-to-image diffusion models - this is the source of its core cost advantage.
Main functions
- Sketch-based Editing: The user draws a sketch on the image to specify the editing intention (such as changing the shape of the object, adjusting the posture, replacing local content), and FramePainter automatically understands the semantics of the sketch and generates an image that conforms to the editing instructions. The thickness, position, and shape of the sketch directly affect the editing results, and the interaction is intuitive and does not require precise masking.
- Click-based Editing: Supports specifying the editing area and target through click operations, suitable for quickly adjusting local content of the image. Point-and-click operations are faster than sketching and are suitable for simple replacement and repair tasks.
- Drag-based Editing: Users drag key points in the image to control changes in object posture, shape or position. Different from drag-and-drop editing based on optical flow, FramePainter understands the movement patterns of objects a priori through video diffusion, and the generated deformation is more natural.
- Out-of-Domain Generalization: This is the core highlight of FramePainter - using video diffusion prior, the model can still generate reasonable results in editing scenarios that have not appeared in the training data. For example, turning a clownfish into a shark shape, adding reflective light effects that did not exist originally to a cup, etc. demonstrate creativity beyond the boundaries of conventional editing.
- Matching Attention: In view of the limitations of temporal attention in processing large movements, a matching attention layer is proposed to expand the receptive field and enhance the dense correspondence between the edited image and the source image token, ensuring the coherence of the editing results in content and structure.
Model and version evolution
| Version | Time | Core Changes |
|---|---|---|
| arXiv v1 | 2025-01-14 | The paper was released for the first time (2501.08225), making the technical plan, experimental results and part of the code public |
| v1.0 (ICCV 2025) | ~2025-01 | Complete inference demo released, pre-trained weights uploaded to HuggingFace, supporting complete editing process |
After the paper was first made public in January 2025, the complete inference code and pre-training weights were released in the same month. No new code version will be released after the official acceptance of ICCV 2025. The current warehouse is the stable version officially implemented by the paper.
Technical advantages
- Problem Reconstruction Paradigm: The core technical innovation of FramePainter is to redefine interactive image editing as an "Image-to-Video Generation" problem. Traditional methods regard editing as a conditional image generation task, which requires a large amount of paired training data and additional reference encoders to learn physical dynamics; FramePainter, by inheriting the existing video diffusion prior of Stable Video Diffusion, naturally understands the changing patterns of objects in motion, greatly reducing training costs and dependence on the amount of paired data.
- Lightweight Sparse Control Encoder: Introduce only a lightweight sparse control encoder to inject editing signals (sketch, click, drag, etc.) instead of duplicating the entire U-Net encoder like ControlNet. This allows the model to maintain the basic capabilities of SVD while adding minimal new parameters and making training and inference more efficient.
- Matching Attention: The standard temporal attention mechanism has a limited receptive field when dealing with large movements between two frames, which can easily lead to inconsistent editing. Matching attention establishes dense point-by-point correspondences between edited images and source image tokens by expanding the search scope in cross-attention, maintaining content coherence even when object poses or shapes are drastically changed.
- Unification of multi-modal editing signals: The same architecture supports multiple interaction methods such as sketching, clicking, and dragging at the same time, eliminating the need to train different models for different interaction signals. This unified framework makes it possible to combine different editing methods (such as sketching the outline first and then dragging to adjust the position).
- Out-of-domain generalization ability: Thanks to the rich physical world knowledge contained in the video diffusion prior, FramePainter can still generate visually reasonable results in editing scenarios not covered by the training data (such as turning fish into shark shapes), which was previously difficult to achieve with pure image diffusion models.
How to use
FramePainter is an open source project that runs locally and requires no account registration or paid subscription.
Preparation
# Clone repository
git clone https://github.com/YBYBZhang/FramePainter.git
cd FramePainter
#Create Conda context
conda create -n framepainter python=3.10
conda activate framepainter
# Install dependencies
pip install -r requirements.txtDownload weight
Download the pre-trained weights from HuggingFace and put them in the checkpoints/ directory. During inference, app.py will automatically download stabilityai/stable-video-diffusion-img2vid-xt-1-1 as the base model.
Run inference
pythonapp.pyAfter startup, you can upload images, draw sketches, and preview the editing effects in real time in the interactive interface. For detailed instructions, please refer to the GitHub warehouse README and the original text of the paper.
Product Pricing
FramePainter is fully open source (MIT License), without any payment.
| Project | Cost |
|---|---|
| Software License | Free (MIT Open Source) |
| Pre-trained weights | Free (HuggingFace download) |
| Commercial Use | Permitted (MIT License) |
| Subscription/Cloud Service | No official cloud service, need to deploy it locally |
In terms of hardware, you need to bring your own NVIDIA GPU. If you want to experience it online, you can pay attention to the third-party demo deployed by the community (unofficial maintenance).
Application scenarios
- Interactive Image Design and Creative Editing: Designers use sketches to quickly adjust image composition, replace objects, and change styles. FramePainter's intuitive sketch interaction eliminates the complex masking and layer manipulation found in traditional software, making creative iteration more efficient. The matching attention mechanism ensures that image content remains coherent even after extensive editing.
- Advertising and Marketing Visual Material Production: The marketing team quickly generates multiple versions of advertising materials through drag and drop (such as replacing product packaging, adjusting model postures, changing backgrounds). Since there is no need for training data, you can operate directly on the finished map, and the time from request to map production is shortened to minutes.
- Film and television post-production and concept design: Film and television post-production personnel quickly generate previews of lens effects during the conceptual design stage. FramePainter's out-of-domain generalization ability allows it to play a role in creative scenarios not covered by training data (such as adding non-realistic elements to historical scenes), making it suitable for movie storyboards and game concept design.
- Academic Research and Algorithm Validation: As the official open source implementation of the ICCV 2025 paper, FramePainter provides computer vision researchers with a directly runnable interactive editing baseline system. Subsequent research can carry out improvements and comparative experiments on its basis.
Applicable people
- AI Image Editing Researchers and Developers: FramePainter provides complete training and inference code and is suitable as a baseline system for interactive image editing research. The method design and ablation experiments in the paper provide a clear reference for subsequent improvements.
- Professional designers and creative workers: Designers who need to quickly iterate on visual solutions without being too demanding on editing accuracy. Sketching and dragging interactions are faster and more direct than the precise masking operations of traditional tools, making them suitable for concept exploration in the creative stage.
- AI painting and technology enthusiasts: AI enthusiasts who want to understand and use the latest ICCV paper results. The local deployment process is clear (Conda + pip + weight download), and you can experience the SOTA interactive editing effect with an NVIDIA GPU.
- Not suitable for the crowd: Production situations that require pixel-level precise control (such as print finishing) are not suitable for using editing tools based on the diffusion model; scenarios that have strict real-time requirements (such as batch automated processing pipelines) need to consider the time-consuming inference; users without NVIDIA GPU hardware cannot run it locally. Additionally, as a research project, FramePainter does not come with commercial-level UI/UX polish and customer support.
Summary and Outlook
With the paradigm innovation of "reconstructing image editing into image-to-video generation" as its core, FramePainter leverages Stable Video Diffusion's video diffusion prior to significantly reduce the training cost of interactive image editing, and achieves high-quality multi-modal editing effects through sparse control encoders and matching attention mechanisms. Its out-of-domain generalization ability is a key highlight that distinguishes it from previous methods - it can still generate reasonable visual results in editing scenarios not covered by training data.
Unfit boundary: Production scenarios that require pixel-level precise control, batch processing scenarios with high real-time requirements, and user groups without NVIDIA GPUs. Procurement/Adoption Risk: FramePainter is a purely academic research project with no commercialization team or enterprise-level support; code updates and maintenance rely on the original author's personal time, and there is no SLA commitment; it is recommended to evaluate the use risks in production projects, or wait for a third party to package it into a commercializable product/service.
Related tools: midjourney, stable-diffusion
Version Info
- ICCV 2025 official version ：ICCV 2025 includes the corresponding version of the paper, supports sketch-based interactive image editing, and is based on Stable Video Diffusion and lightweight sparse control encoder. There is no official precise release date yet.
- arXiv preprints ：The paper was first published on arXiv (2501.08225), disclosing the technical solution and experimental results.
User Reviews