🟢 ControlNet

ControlNet is a plugin for Stable Diffusion that allows the incorporation of a predefined shape into the initial image, which AI then completes.

Plugin Installation

For Windows users who followed the installation and deployment section, the plugin is already included in the downloaded package, so no manual installation is necessary. Others can install the sd-webui-controlnet plugin from the extensions section.

Model Installation

In addition to the plugin, the ControlNet model needs to be downloaded from the following links:

https://huggingface.co/webui/ControlNet-modules-safetensors/tree/main

https://huggingface.co/lllyasviel/ControlNet/tree/main/models

Place the downloaded models in the following path:

For some versions, place the applications in the /extensions/sd-webui-controlnet/models directory.

Model Overview

Model	Effect
Canny	Uses algorithms to intricately capture the edges of the image as references for generation.
mlsd	Attempts to capture prominent straight lines in the image for reference.
hed	Attempts to capture image features for reference.
Scribbles	Uses provided lines as structural references for generating images.
openpose	Generates a skeleton from the human figures in the image for reference.
seg	Replaces the image with broad color blocks for reference.
depth	Tries to capture the depth and distance within the image for reference.
normal	Similar to depth, tries to capture the depth and distance within the image for reference.

Examples

For instance, using Canny for coloring line art:

Enter the prompt words, select the main model, sampler, and adjust the height as needed.

Then click on the ControlNet module, upload your line art, check the enable box and invert input colors (since our image has a white background), adjust the canvas height accordingly. If your GPU performance is limited, consider selecting the low memory mode.