What is IP-Adapter?

When looking at a picture or photo and thinking "I want it drawn with this same vibe," it is almost impossible to explain every detail using only text.

Therefore, several mechanisms have been proposed to "let the AI look directly at the image without going through text." Among them, IP-Adapter is one of the classic methods used for "transferring" style and subject.

Think of it as the pioneer of "reference2image" and "Subject Transfer".

Required Custom Nodes

SD1.5 × IP-Adapter

There are several types of IP-Adapter, but let's start by trying the most standard one.

Download Models

  • IP-Adapter Main Model (for SD1.5)
  • CLIP Vision Model 
  • 📂ComfyUI/
  └── 📂models/
      ├── 📂clip_vision/
      │   └── OpenCLIP-ViT-H-14.safetensors
      └── 📂ip_adapter/
          └── ip-adapter_sd15.safetensors

workflow

ip_adapter_sd15.json
  • 🟩 Connect various models and the reference image you want to use to the IPAdapter Advanced node.
  • 🟦 Crop the reference image with the Prep Image For ClipVision node.
    • Details below.

What is it "Looking" At?

The CLIP ViT-H-14, which acts as the "eye" for IP-Adapter, basically only sees a 224 × 224 square area. Therefore, if you pass a portrait photo as is, the face or feet might be cut off, or it might only pick up features from the middle of the body.

If you want to decide which part should be the standard, resize and crop it first as shown in the workflow above.

Main IP-Adapter Models

There are several derivative models, and "what and how much they borrow" from the reference image varies greatly from model to model.

ip-adapter-plus_sd15

A model that strongly transfers composition and object positions.

ip_adapter_plus_sd15.json

ip-adapter_sd15_light

A model that prioritizes text prompts.

ip_adapter_sd15_light.json

ip-adapter-plus-face_sd15

An IP-Adapter specialized for faces (head).

ip_adapter_plus_face_sd15.json

SDXL Model Links

For those who want to try SDXL as well, here is a list of model links for SDXL.