Fixed LayerStyle Advance – Transformers 5.0 Compatibility Fix

Summary

The ComfyUI LayerStyle Advance custom node (SmolVLM / SmolLM2) failed to load on Hugging Face Transformers 5.0 with:

ImportError: cannot import name 'AutoModelForVision2Seq' from 'transformers'

This document explains the cause, the fix, and the meaning of the changes.

1. Error cause

What broke

The node’s `smol.py` imports:

from transformers import AutoProcessor, AutoModelForVision2Seq, AutoTokenizer, AutoModelForCausalLM

and uses `AutoModelForVision2Seq` to load SmolVLM (image + text → text).

In Transformers 5.0, the name `AutoModelForVision2Seq` was removed from the public API.

Why it was removed

Transformers unified “vision-to-sequence” (image + text → text) under a single auto class:

• Old (pre‑5.0): `AutoModelForVision2Seq` for models like SmolVLM, BLIP-2, etc.

• New (5.0+): `AutoModelForImageTextToText` for all such models.

So the failure is due to an API rename, not a change in model behavior. SmolVLM is still supported; it’s just loaded via `AutoModelForImageTextToText` in 5.0.

Where it failed

• File: `ComfyUI_LayerStyle_Advance/py/smol.py`

• Trigger: Import at top of `smol.py` when the node is loaded.

• Impact: The whole LayerStyle Advance node fails to load, so all its nodes (including SmolVLM) become unavailable.

2. Countermeasure (approach)

• Goal: Keep the node working on Transformers 5.0 without breaking older environments.

• Idea: Use the new API name that exists in 5.0 and is the documented way to load image‑to‑text models.

• Concrete change:

  • Use `AutoModelForImageTextToText` instead of `AutoModelForVision2Seq` in both the import and in every `from_pretrained` call for the VLM.

No compatibility layer (e.g. try/except with two class names) is needed if you only care about 5.0: the new name is the correct one there.
If you must support both old and new transformers, you can do:

try:
    from transformers import AutoModelForVision2Seq
except ImportError:
    from transformers import AutoModelForImageTextToText as AutoModelForVision2Seq

For a “Transformers 5.0 only” fix, a direct replacement is enough.

3. Modified file

4. Modified code

4.1 Import (line 7)

Before:

from transformers import AutoProcessor, AutoModelForVision2Seq, AutoTokenizer, AutoModelForCausalLM

After:

from transformers import AutoProcessor, AutoModelForImageTextToText, AutoTokenizer, AutoModelForCausalLM

4.2 SmolVLM model loading – flash attention path (lines 112–116)

Before:

model = AutoModelForVision2Seq.from_pretrained(
    model_path,
    torch_dtype=torch_dtype,
    _attn_implementation="flash_attention_2" if use_flash_attention else "eager",
).to(device)

After:

model = AutoModelForImageTextToText.from_pretrained(
    model_path,
    torch_dtype=torch_dtype,
    _attn_implementation="flash_attention_2" if use_flash_attention else "eager",
).to(device)

4.3 SmolVLM model loading – fallback (lines 118–122)

Before:

model = AutoModelForVision2Seq.from_pretrained(
    model_path,
    torch_dtype=torch_dtype,
    _attn_implementation="eager",
).to(device)

After:

model = AutoModelForImageTextToText.from_pretrained(
    model_path,
    torch_dtype=torch_dtype,
    _attn_implementation="eager",
).to(device)

So: one import change and two `from_pretrained` call changes; all three replace `AutoModelForVision2Seq` with `AutoModelForImageTextToText`.

5. Meaning of the changes

• Semantic: The node still loads the same kind of model (image + text → text). Only the class name used to load it is updated to the Transformers 5.0 API.

• Behavior:

  • `AutoModelForImageTextToText.from_pretrained(...)` still resolves SmolVLM via the same config/model type and gives the same forward signature and usage.

  • Flash vs eager attention and device/dtype handling are unchanged.

• Compatibility:

  • Transformers 5.0: Works.

  • Older transformers (e.g. 4.x) that still export `AutoModelForVision2Seq`: Would require the compatibility import above if you need to support them from this file.

So the “meaning” is: update the official class name for image‑to‑text models so the node runs on Transformers 5.0, without changing what the node does logically.

6. Complete explanation

6.1 Role of the file

`smol.py` provides four node classes:

1. LS_Load_SmolLM2_Model – loads SmolLM2 (text-only), uses `AutoModelForCausalLM` (unchanged).

2. LS_Load_SmolVLM_Model – loads SmolVLM (image+text), uses `AutoProcessor` and the image‑to‑text model class (this is what we changed).

3. LS_SmolLM2 – runs inference with the loaded SmolLM2 model.

4. LS_SmolVLM – runs inference with the loaded SmolVLM model (processor + model from (2)).

Only the SmolVLM loader relied on `AutoModelForVision2Seq`; that is the only part affected by the API rename.

6.2 Why the new name is correct

In Transformers:

• AutoModelForVision2Seq was a legacy name for “vision (and text) in → text out” models.

• AutoModelForImageTextToText is the current, single entry point for that in 5.0.
  The library maps model types (including SmolVLM) to this class in `models/auto/modeling_auto.py`.

So for SmolVLM on 5.0, `AutoModelForImageTextToText` is the intended and supported API.

6.3 What was not changed

• SmolLM2: Still uses `AutoModelForCausalLM` and `AutoTokenizer`; no change.

• SmolVLM: Still uses `AutoProcessor` for images + text; only the model class is switched to `AutoModelForImageTextToText`.

• Inference: `processor`, `model.generate`, and post-processing (e.g. `split_vlm_content`) are unchanged.

6.4 How to confirm the fix

1. Use an environment with Transformers 5.0.

2. Start ComfyUI and ensure the LayerStyle Advance custom node loads without import errors.

3. Add “Load SmolVLM Model (Advance)” and “SmolVLM (Advance)” to a workflow, load a model, run on an image + prompt, and check that output text is produced as before.

If you need to support both old and new transformers from the same `smol.py`, use the try/except import in Section 2 and keep using a single name (e.g. `AutoModelForVision2Seq`) in the rest of the file; otherwise the direct replacement above is enough for a clean “Transformers 5.0” setup.

7. References

• Hugging Face Transformers: image‑to‑text and `AutoModelForImageTextToText` in the 5.0 API.

• SmolVLM: HuggingFaceTB/SmolVLM-Instruct; in 5.0 it is loaded via `AutoModelForImageTextToText`.