Skip to content
,

ComfyUI ControlNet Guide (2026): Canny, Depth, Pose for SDXL, Flux, Qwen-Image and Z-Image

7–10 minutes
Smiling winking woman in a denim jacket in a library holding a card reading CONTROLNET, with the text Control Every Pose: ComfyUI ControlNet

ControlNet is how you tell an image model exactly where things should go. You give it a guide image, such as edge lines, a depth map or a stick-figure pose, and the model follows that structure while your prompt decides the style.

It is the difference between “a woman in a café” and “a woman in a café, standing in this exact pose, with the window right here”.

In ComfyUI, ControlNet is built into the core nodes. The extra parts you need are a ControlNet model that matches your base model and, usually, a preprocessor node pack to make the guide images.

This guide covers the whole setup. You will learn which nodes to use, where every file goes, which preprocessor fits which job, and the right ControlNet for SD 1.5, SDXL, Flux, Qwen-Image and Z-Image Turbo.

It also explains the strength and timing settings, how to stack several ControlNets, and how to fix the usual errors. It is checked against the ComfyUI docs and official model pages as of November 2026.

TL;DR

  • Core nodes: Load ControlNet Model + Apply ControlNet (positive, negative, control_net, image, vae, strength, start_percent, end_percent).
  • Preprocessors: install comfyui_controlnet_aux (Fannovel16) for Canny, Lineart, Depth Anything V2, DWPose and more.
  • Folders: classic ControlNets go in models/controlnet; Qwen-Image DiffSynth patches and Z-Image Fun ControlNet go in models/model_patches.
  • Match the model family: an SD 1.5 ControlNet will not work with SDXL, Flux or Qwen-Image.
  • Start with strength 0.6-0.8 and end_percent 0.8, then adjust.
  • Fastest start: open ComfyUI’s built-in templates (Workflow > Browse Templates) and search “ControlNet”.

What ControlNet Does

A normal text-to-image model only has your prompt to go on. It chooses the pose, layout and camera angle by itself. ControlNet adds a second input: a control image that describes structure.

The model then tries to match that structure while it generates. Common uses include:

  • Copying a pose from a photo onto a different character.
  • Keeping the layout of a room or product shot while changing the style.
  • Turning a rough sketch or line drawing into a finished image.
  • Keeping depth and perspective from a reference photo.

If you are new to ComfyUI itself, start with our ComfyUI beginner’s guide first, then come back here.

What You Need

  • An up-to-date ComfyUI. The core ControlNet nodes are built in.
  • ComfyUI Manager, to install the preprocessor pack. See how to install ComfyUI Manager.
  • A base model (SD 1.5, SDXL, Flux, Qwen-Image or Z-Image Turbo).
  • A ControlNet model made for that base model.
  • The preprocessor pack comfyui_controlnet_aux.

Install the preprocessors

  1. Open ComfyUI and click Manager > Custom Nodes Manager.
  2. Search for comfyui_controlnet_aux (author Fannovel16) and click Install.
  3. Restart ComfyUI.

Manual install, from your ComfyUI folder:

cd custom_nodes
git clone https://github.com/Fannovel16/comfyui_controlnet_aux
cd comfyui_controlnet_aux
pip install -r requirements.txt

On the portable Windows build, use the bundled Python, for example ..\..\..\python_embeded\python.exe -m pip install -r requirements.txt. Preprocessor weights download automatically the first time you use each node.

ControlNet Types and When to Use Each

Type Preprocessor node Best for
Canny Canny Edge Hard edges, product shots, logos, architecture
Lineart / Soft edge Lineart, AnyLine, HED, PiDiNet Sketches, illustrations, softer outlines
Depth Depth Anything V2, MiDaS, Zoe, Metric3D Keeping perspective and 3D layout
Pose DWPose, OpenPose Copying body, hand and face pose
Scribble Scribble, or your own drawing Turning rough doodles into images
MLSD M-LSD lines Straight lines, rooms, interiors
Tile None (use the image) Upscaling while adding detail

Tip: DWPose is usually more accurate than classic OpenPose, especially for hands. Depth Anything V2 is a good default for depth.

Which ControlNet Model for Which Base Model

A ControlNet only works with the model family it was trained for. This is the most common mistake beginners make.

Base model Example ControlNet Folder
SD 1.5 ControlNet v1.1 files such as control_v11p_sd15_canny or control_v11p_sd15_scribble_fp16 models/controlnet
SDXL Union models such as xinsir’s ControlNet Union for SDXL models/controlnet
Flux.1 dev Union models such as Shakker-Labs ControlNet Union Pro 2.0 models/controlnet
Qwen-Image InstantX ControlNet Union, or DiffSynth model patches models/controlnet or models/model_patches
Z-Image Turbo Z-Image-Turbo-Fun-Controlnet-Union models/model_patches

“Union” models handle several control types in one file, which saves disk space. Always check the model card for the exact control types supported and any recommended strength. For Flux setup itself, see how to use Flux.

Build a Basic ControlNet Workflow

The quickest way is to load a built-in template: Workflow > Browse Templates, then search “ControlNet”. If you prefer to build it yourself, start from the default text-to-image workflow and add these nodes.

  1. Load Image: load your reference photo or sketch.
  2. Preprocessor: add the matching node (for example Canny Edge or DWPose Estimator) and connect the image to it.
  3. Preview Image: connect the preprocessor output so you can check the control map looks right.
  4. Load ControlNet Model: pick your ControlNet file from models/controlnet.
  5. Apply ControlNet: connect positive and negative from your two CLIP Text Encode nodes, control_net from the loader, image from the preprocessor, and vae from your checkpoint loader.
  6. Connect the positive and negative outputs of Apply ControlNet into the KSampler, replacing the direct links from the text encoders.
  7. Set the Empty Latent Image size to the same aspect ratio as your control image, then click Queue.

Note: older workflows may use the deprecated “Apply ControlNet (OLD)” node, which only takes conditioning. It still loads, but swap it for the current node when you can.

Want ready-made graphs to learn from? Our ComfyUI workflows collection has several you can load and study.

Strength, Start and End Explained

Setting What it does Good starting value
strength How strongly the image follows the control map 0.6-0.8 (1.0 for strict pose)
start_percent When control begins (0 = from the first step) 0.0
end_percent When control stops (1 = until the last step) 0.8-1.0

The early steps set the layout, and the later steps add detail. So ending control at around 0.8 keeps the structure but lets the model clean up fine details freely. If results look stiff or have edge artefacts, lower the strength or end earlier. If the model ignores your pose, raise the strength.

Stack Multiple ControlNets

You can chain several Apply ControlNet nodes. Connect the positive and negative outputs of the first into the inputs of the second, then into the KSampler.

A popular combination is pose plus depth: DWPose fixes the body, while depth keeps the scene’s perspective. When stacking, lower each strength (for example 0.6 and 0.4) so they do not fight each other. With a Union model you can load the same file twice and feed it different control maps.

ControlNet for Qwen-Image

Qwen-Image has three ControlNet routes, each with a built-in ComfyUI template. They are made for the original Qwen-Image family; check the model card before using them with newer releases. For installing the base model, see our Qwen-Image local guide.

1. InstantX ControlNet Union

  • File: Qwen-Image-InstantX-ControlNet-Union.safetensors in models/controlnet.
  • Supports canny, soft edge, depth and pose in one model.
  • Uses the normal Load ControlNet Model and Apply ControlNet nodes.
  • Template: image_qwen_image_instantx_controlnet.

2. DiffSynth model patches

  • Files: qwen_image_canny_diffsynth_controlnet, qwen_image_depth_diffsynth_controlnet and qwen_image_inpaint_diffsynth_controlnet in models/model_patches.
  • Load with ModelPatchLoader and apply with QwenImageDiffsynthControlnet, which has strength and an optional mask.
  • Template: image_qwen_image_controlnet_patch.

3. Union control LoRA

  • File: qwen_image_union_diffsynth_lora.safetensors in models/loras.
  • Load with LoraLoaderModelOnly and feed your control image as the reference.
  • Template: image_qwen_image_union_control_lora.

ControlNet for Z-Image Turbo

Z-Image Turbo uses a single Fun ControlNet Union file that supports Canny, HED, Depth, Pose and MLSD.

  • File: Z-Image-Turbo-Fun-Controlnet-Union.safetensors in models/model_patches.
  • Template: image_z_image_turbo_fun_union_controlnet.
  • Because Z-Image Turbo runs in very few steps, keep control strength moderate and test a few values.

For more on fast Z-Image setups, see our NitroFusion2 Z-Image Turbo guide. If you use MOSS-Image instead, it has its own control LoRAs, covered in MOSS-Image control LoRAs in ComfyUI.

Troubleshooting

Problem Likely cause Fix
ControlNet has no effect Apply ControlNet outputs not wired into the KSampler Connect its positive and negative outputs to the sampler
Error about shapes or mat1 and mat2 ControlNet is for a different base model Use a ControlNet trained for your model family
Model not listed in the loader Wrong folder, or ComfyUI not refreshed Check models/controlnet vs models/model_patches, then press R to refresh
Preprocessor nodes missing (red) comfyui_controlnet_aux not installed Install via Manager and restart
Preprocessor download fails Hugging Face blocked or offline Download the weights manually into the pack’s ckpts folder
Stiff, over-traced images Strength too high Lower strength to 0.5-0.7 or set end_percent to 0.7
Stretched or cropped result Latent size does not match control image ratio Match the aspect ratio, or resize the control image
Out of memory Large ControlNet plus big base model Use fp16 or smaller files, lower resolution, or try a GGUF base

Inpainting with control maps not behaving? Our ComfyUI inpainting fix guide covers the masking side.

FAQ

Do I need a custom node to use ControlNet in ComfyUI?

No. The loader and apply nodes are built in. You only need the comfyui_controlnet_aux pack if you want ComfyUI to generate the control maps for you.

Can I use my own sketch without a preprocessor?

Yes. A clean black-and-white drawing can go straight into Apply ControlNet with a scribble or lineart ControlNet.

What is a Union ControlNet?

A single model trained on several control types, such as canny, depth and pose. You pick the type by the control image you feed it.

Why does my SD 1.5 ControlNet fail with SDXL?

The internal sizes are different. Each base model family needs its own ControlNet.

How much VRAM does ControlNet add?

It depends on the model. SD 1.5 ControlNets add little; Flux and Qwen-Image ControlNets are larger and can add several gigabytes.

Can ControlNet copy a face?

It copies structure, not identity. Pose and depth will match, but the face comes from your prompt and model. Do not use it to recreate real people without their permission.

Sources

Free weekly email

Get new AI guides every week

One short weekly email with our newest local AI, ComfyUI and AI tool guides. No spam, no pop-ups, just the good stuff.

Free. Unsubscribe anytime in one click. We only use your email to send this newsletter. See our privacy policy.

Smiling winking woman in a denim jacket in a library holding a card reading CONTROLNET, with the text Control Every Pose: ComfyUI ControlNet