[Bug]: Ip-Adapter ControlNet, RuntimeError: Cannot set version_counter for inference tensor

[MythicalChu]

Checklist

• The issue exists after disabling all extensions
• The issue exists on a clean installation of webui
• The issue is caused by an extension, but I believe it is caused by a bug in the webui
• The issue exists in the current version of the webui
• The issue has not been reported before recently
• The issue has been reported before but has not been fixed yet

What happened?

ControlNet errors out, output image doesn't use it.

Steps to reproduce the problem

When using IpAdapter with or without InsightFace

What should have happened?

ControlNet use.

What browsers do you use to access the UI ?

Other

Sysinfo

sysinfo-2024-02-06-18-46.json

Console logs

venv "C:\AI\Webui-Forge\stable-diffusion-webui-forge\venv\Scripts\Python.exe"
Python 3.10.10 (tags/v3.10.10:aad5f6a, Feb  7 2023, 17:20:36) [MSC v.1929 64 bit (AMD64)]
Version: f0.0.7-latest-44-g7359740f
Commit hash: 7359740f36e9f59d8c358cbb5b07a4fde85900c3
Launching Web UI with arguments: --ckpt-dir C:/AI/Makeayo/Models --hypernetwork-dir /models/hypernetworks --embeddings-dir /models/embeddings --lora-dir C:/AI/Makeayo/Extras --directml --skip-torch-cuda-test --always-normal-vram
Using directml with device:
Total VRAM 1024 MB, total RAM 32662 MB
Set vram state to: NORMAL_VRAM
Device: privateuseone
VAE dtype: torch.float32
Warning: caught exception 'Torch not compiled with CUDA enabled', memory monitor disabled
Using sub quadratic optimization for cross attention, if you have memory or speed issues try using: --attention-split
==============================================================================
You are running torch 2.0.0+cpu.
The program is tested to work with torch 2.1.2.
To reinstall the desired version, run with commandline flag --reinstall-torch.
Beware that this will cause a lot of large files to be downloaded, as well as
there are reports of issues with training tab on the latest version.

Use --skip-version-check commandline argument to disable this check.
==============================================================================
ControlNet preprocessor location: C:\AI\Webui-Forge\stable-diffusion-webui-forge\models\ControlNetPreprocessor
Loading weights [71e14760e2] from C:/AI/Makeayo/Models\lazymixRealAmateur_v30b.safetensors
2024-02-06 13:33:59,339 - ControlNet - INFO - ControlNet UI callback registered.
model_type EPS
UNet ADM Dimension 0
Running on local URL:  http://127.0.0.1:7860

To create a public link, set `share=True` in `launch()`.
Startup time: 9.3s (prepare environment: 0.9s, import torch: 3.2s, import gradio: 1.0s, setup paths: 0.8s, initialize shared: 0.1s, other imports: 0.5s, list SD models: 0.1s, load scripts: 1.8s, create ui: 0.6s, gradio launch: 0.2s).
Using split attention in VAE
Working with z of shape (1, 4, 32, 32) = 4096 dimensions.
Using split attention in VAE
extra {'cond_stage_model.clip_l.logit_scale', 'cond_stage_model.clip_l.text_projection'}
left over keys: dict_keys(['alphas_cumprod', 'alphas_cumprod_prev', 'betas', 'log_one_minus_alphas_cumprod', 'posterior_log_variance_clipped', 'posterior_mean_coef1', 'posterior_mean_coef2', 'posterior_variance', 'sqrt_alphas_cumprod', 'sqrt_one_minus_alphas_cumprod', 'sqrt_recip_alphas_cumprod', 'sqrt_recipm1_alphas_cumprod'])
To load target model SD1ClipModel
Begin to load 1 model
Model loaded in 1.9s (load weights from disk: 0.6s, forge load real models: 0.9s, load VAE: 0.1s, calculate empty prompt: 0.2s).
Token merging is under construction now and the setting will not take effect.
2024-02-06 13:42:33,775 - ControlNet - INFO - ControlNet Input Mode: InputMode.SIMPLE
2024-02-06 13:42:33,778 - ControlNet - INFO - Using preprocessor: InsightFace+CLIP-H (IPAdapter)
2024-02-06 13:42:33,778 - ControlNet - INFO - preprocessor resolution = 0.5
Applied providers: ['CPUExecutionProvider'], with options: {'CPUExecutionProvider': {}}
find model: C:\AI\Webui-Forge\stable-diffusion-webui-forge\models\insightface\models\buffalo_l\1k3d68.onnx landmark_3d_68 ['None', 3, 192, 192] 0.0 1.0
Applied providers: ['CPUExecutionProvider'], with options: {'CPUExecutionProvider': {}}
find model: C:\AI\Webui-Forge\stable-diffusion-webui-forge\models\insightface\models\buffalo_l\2d106det.onnx landmark_2d_106 ['None', 3, 192, 192] 0.0 1.0
Applied providers: ['CPUExecutionProvider'], with options: {'CPUExecutionProvider': {}}
find model: C:\AI\Webui-Forge\stable-diffusion-webui-forge\models\insightface\models\buffalo_l\det_10g.onnx detection [1, 3, '?', '?'] 127.5 128.0
Applied providers: ['CPUExecutionProvider'], with options: {'CPUExecutionProvider': {}}
find model: C:\AI\Webui-Forge\stable-diffusion-webui-forge\models\insightface\models\buffalo_l\genderage.onnx genderage ['None', 3, 96, 96] 0.0 1.0
Applied providers: ['CPUExecutionProvider'], with options: {'CPUExecutionProvider': {}}
find model: C:\AI\Webui-Forge\stable-diffusion-webui-forge\models\insightface\models\buffalo_l\w600k_r50.onnx recognition ['None', 3, 112, 112] 127.5 127.5
set det-size: (640, 640)
Warning torch.load doesn't support weights_only on this pytorch version, loading unsafely.
2024-02-06 13:42:35,485 - ControlNet - INFO - Current ControlNet IPAdapterPatcher: C:\AI\Webui-Forge\stable-diffusion-webui-forge\models\ControlNet\ip-adapter-faceid-plusv2_sd15.bin
C:\AI\Webui-Forge\stable-diffusion-webui-forge\venv\lib\site-packages\insightface\utils\transform.py:68: FutureWarning: `rcond` parameter will change to the default of machine precision times ``max(M, N)`` where M and N are the input matrix dimensions.
To use the future default and silence this warning we advise to pass `rcond=None`, to keep using the old, explicitly pass `rcond=-1`.
  P = np.linalg.lstsq(X_homo, Y)[0].T # Affine matrix. 3 x 4
To load target model CLIPVisionModelProjection
Begin to load 1 model
*** Error running process_before_every_sampling: C:\AI\Webui-Forge\stable-diffusion-webui-forge\extensions-builtin\sd_forge_controlnet\scripts\controlnet.py
    Traceback (most recent call last):
      File "C:\AI\Webui-Forge\stable-diffusion-webui-forge\modules\scripts.py", line 830, in process_before_every_sampling
        script.process_before_every_sampling(p, *script_args, **kwargs)
      File "C:\AI\Webui-Forge\stable-diffusion-webui-forge\venv\lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
        return func(*args, **kwargs)
      File "C:\AI\Webui-Forge\stable-diffusion-webui-forge\extensions-builtin\sd_forge_controlnet\scripts\controlnet.py", line 457, in process_before_every_sampling
        self.process_unit_before_every_sampling(p, unit, self.current_params[i], *args, **kwargs)
      File "C:\AI\Webui-Forge\stable-diffusion-webui-forge\venv\lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
        return func(*args, **kwargs)
      File "C:\AI\Webui-Forge\stable-diffusion-webui-forge\extensions-builtin\sd_forge_controlnet\scripts\controlnet.py", line 403, in process_unit_before_every_sampling
        params.model.process_before_every_sampling(p, cond, mask, *args, **kwargs)
      File "C:\AI\Webui-Forge\stable-diffusion-webui-forge\extensions-builtin\sd_forge_ipadapter\scripts\forge_ipadapter.py", line 147, in process_before_every_sampling
        unet = opIPAdapterApply(
      File "C:\AI\Webui-Forge\stable-diffusion-webui-forge\extensions-builtin\sd_forge_ipadapter\lib_ipadapter\IPAdapterPlus.py", line 743, in apply_ipadapter
        image_prompt_embeds = self.ipadapter.get_image_embeds_faceid_plus(face_embed.to(self.device, dtype=self.dtype), clip_embed.to(self.device, dtype=self.dtype), weight_v2, faceid_v2)
      File "C:\AI\Webui-Forge\stable-diffusion-webui-forge\venv\lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
        return func(*args, **kwargs)
      File "C:\AI\Webui-Forge\stable-diffusion-webui-forge\extensions-builtin\sd_forge_ipadapter\lib_ipadapter\IPAdapterPlus.py", line 354, in get_image_embeds_faceid_plus
        embeds = self.image_proj_model(face_embed, clip_embed, scale=s_scale, shortcut=shortcut)
      File "C:\AI\Webui-Forge\stable-diffusion-webui-forge\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
        return forward_call(*args, **kwargs)
      File "C:\AI\Webui-Forge\stable-diffusion-webui-forge\extensions-builtin\sd_forge_ipadapter\lib_ipadapter\IPAdapterPlus.py", line 128, in forward
        x = self.proj(id_embeds)
      File "C:\AI\Webui-Forge\stable-diffusion-webui-forge\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
        return forward_call(*args, **kwargs)
      File "C:\AI\Webui-Forge\stable-diffusion-webui-forge\venv\lib\site-packages\torch\nn\modules\container.py", line 217, in forward
        input = module(input)
      File "C:\AI\Webui-Forge\stable-diffusion-webui-forge\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
        return forward_call(*args, **kwargs)
      File "C:\AI\Webui-Forge\stable-diffusion-webui-forge\venv\lib\site-packages\torch\nn\modules\linear.py", line 114, in forward
        return F.linear(input, self.weight, self.bias)
    RuntimeError: Cannot set version_counter for inference tensor

---

Additional information

No response

[lllyasviel]

update and try again?

[MythicalChu]

Generation is frozen on the/before the 1st step now. Maybe it's the same (old torch) bug that ComfyUI's Ip-adapter has: sigma_t = sigmas[index].item() , the var can't be read.

here's the thread: cubiq/ComfyUI_IPAdapter_plus#109

here's our manual fix: cubiq/ComfyUI_IPAdapter_plus#109 (comment)

I see someone posted a new fix 4 days ago, I haven't tested it. cubiq/ComfyUI_IPAdapter_plus#109 (comment)

edit:
following that suggestion from 4 days ago, changing this:

sigma = extra_options["sigmas"][0].item() if 'sigmas' in extra_options else 999999999.9

to this, fixes the issue.

sigma = extra_options["sigmas"][0] if 'sigmas' in extra_options else None
sigma = sigma.item() if sigma else 999999999.9

(on \extensions-builtin\sd_forge_ipadapter\lib_ipadapter\IPAdapterPlus.py)

[MythicalChu]

btw, the same problem happens on Kohya Deep Shrink, so I noticed that the file \ldm_patched\contrib\external_model_downscale.py has an almost identical line:

sigma = transformer_options["sigmas"][0].item()

treating it the same way as above:

sigma = transformer_options["sigmas"][0] if 'sigmas' in transformer_options else None
sigma = sigma.item() if sigma else 999999999.9

also stops the freezing on directml on Deep Shrink's use.

Seems everywhere with a ["sigmas"][0].item() will freeze Directml

Since I have your attention, Tiled Vae Decoding isn't working and it seems to be the origin of the "RuntimeError: Cannot set version_counter for inference tensor" error message. But commenting out the line (407):

@torch.inference_mode()

of \ldm_patched\modules\utils.py solves the problem. It decodes on the gpu whereas it just wasn't possible to decode even moderate sized images before this change.

[MythicalChu]

The recent changes to the Ip-adapter code still freeze Directml. I'm not sure if it was intentional or a typo, but the current line is different from the discussed one:

sigma = sigma.item() if sigma is not None else 999999999.9
the code discussed:
sigma = sigma.item() if sigma else 999999999.9

(unrelated, but I'll also mention that the added @torch.inference_mode() lines on forge_loader.py (ln 236, ln 242) and the "with torch.inference_mode()" in processing.py (ln 817) all break Directml, with the same "RuntimeError: Cannot set version_counter for inference tensor" as for the Tiled Vae Decoding, but this time it happens as early as on the prompt processing. Commenting out these lines and changing processing.py back to "with torch.no_grad():" eliminates the issue.)

torch.inference_mode() is completely broken for Directml, it seems.

[Maveyyl]

@MythicalChu All your fixes allowed forge to work for me! Much thanks! I still have infinite freezes when using textual inversions though.