What is Differential Diffusion?

In normal inpainting, the mask is treated as a binary choice of "white or black". If it becomes even slightly gray, it is considered "not masked" and only pure white parts are inpainted.

Differential Diffusion is a mechanism to continuously change the strength of denoise according to the density of the mask. Thanks to this, it is possible to perform inpainting with different amounts of change for each location in a single sampling.

It is assumed that you have read inpainting first. For how to create masks, please refer to Mask Operations and AI Mask Generation.

Usage

Just prepare a gradient mask and add the Differential Diffusion node to the inpainting workflow.

workflow

SD1.5_Differential_Diffusion.json
  • The base is a workflow using the Set Latent Noise Mask node.
    • Of course, it can also be used in workflows using inpainting models or ControlNet models.
  • 🟩 Add the Differential Diffusion node

The whiter part of the mask relies more on the prompt, and the darker part leaves more of the "original picture".

Interesting Uses

Changing the Amount of Change for Each Part

The mask does not have to be a gradient. By changing the density for each location within a single mask, you can specify different amounts of change for each part in a single sampling.

SD1.5_Differential_Diffusion_multi-obj.json
  • Draw different mask densities for each part you want to change (e.g., light gray for the face, white for the background, etc.)

Blending Mask Boundaries

A common problem with inpainting is that the boundaries of the mask appear clearly. Let's blend this boundary naturally by combining Differential Diffusion and a blurred mask.

SD1.5_Differential_Diffusion_blur.json
  • 🟪 This time, incorporate it into a workflow using an inpainting model.
  • Blur the boundaries of the mask with the Gaussian Blur Mask node (ComfyUI-Impact-Pack).
    • Since blurring effectively shrinks the mask, make the mask slightly larger as a pre-processing step.

Using Depth Map as Mask

A depth map is represented by a black and white gradient. In other words, it can be used as a mask compatible with Differential Diffusion.

SD1.5_Differential_Diffusion_depthmap.json
  • 🟦 Create a depth map with Depth Anything V2.
    • Since this is an IMAGE, convert it to a mask with the Convert Image to Mask node.

Honestly, performance is lacking with SD1.5, but using a depth map as a mask itself is a favorite method.

Sample Images