What is ControlNet?

Roughly speaking, a diffusion model learns the relationship between noise and images, so it can recover an image from noise.

So what happens if, along with the noise, you also give it "another piece of information that corresponds to the image"?

  • Train it on how line art relates to finished, colored illustrations
    • → Give it line art, and it colors it for you
  • Train it on how stick figures (pose images) relate to photos of people
    • → Give it a stick figure, and it makes an image of a person in that pose

You can build AI like that.

ControlNet is one such mechanism: it controls the output using "additional image conditions" (pose, line art, depth, etc.) as guides.

Typical Types of ControlNet

There's no real limit to the kinds of "additional information" ControlNet can use, as long as someone has the idea, but the commonly used ones fall into a few patterns. Here are just the main ones.

openpose (Pose / Stick Figure)

Sets the pose of a person or character with a stick figure or skeleton.

depth (Depth Map)

Locks in the composition and depth using a depth map.

scribble (Doodle)

Give it just a rough doodle and it generates an image from that.

lineart / anime (Line Art)

Give it line art and it generates the coloring.

inpaint (For Inpainting)

Naturally fills in masked areas.

There are many other variations too, such as edge detection (Canny), segmentation, and QR codes. In fact, you can build any kind of ControlNet as long as you can prepare images and their corresponding representations.

Instruction-Based Image Editing

With recent image editing models, more and more of what used to be done with ControlNet is now handled as "instruction-based image editing."

Instruction-based image editing lets you edit an image just by giving instructions like "zoom out" or "make it night."

That means ControlNet-style tasks can be treated as "image editing" too.

  • Pose image + "Draw a character in black clothes with this pose"
  • Depth map + "Keep the same composition and make it a night-time photo"
  • Rough sketch + "Turn this rough sketch into a clean illustration"