Anima HiresFix Bug

[ThesiiNCey]

HiresFix isn't working properly with Anima, images are noisy and blurry. I know this is a known problem, also in Comfy and other UIs, but perhaps you have a solution for how to fix it?

Unfortunately, I'm not very familiar with anything related to coding.

[ThesiiNCey]

Perhaps you could also explain why you're giving it a thumbs down? @BagronkeN

[Phenrei]

I was fighting this issue last night, generations after the first would come out with visible noise patterns, and I eventually had to swap the hi-res step to simple scheduler (or at least not beta) to get decent results, or use alternate ESRGAN models more designed around cleaning images or keeping them like the source than upscaling them (an old favorite model I had called Falcoon worked okay). Also need to make sure you're using resolutions that scale up cleanly with the changes made to the hr pass last night (if you're on newest code), I was using a resolution and upscaling by 1.5x and if the ratio changed the results were quite bad.

[Haoming02]

• Anima Base v1.0 can directly generate up to 1.5 Megapixels

1024x1536

• Anima Base v1.0 can also generate up to 4 Megapixels without artifacts (unlike Previews)

896x1152 x 2 = 1792x2304

• If you need higher than that, use MultiDiffusion in img2img

1024x1536 x 2 = 2048x3072

[TohnoSeika]

• Anima Base v1.0 can also generate up to 4 Megapixels without artifacts (unlike Previews)

896x1152 x 2 = 1792x2304

Thank you ,and where to find your "[Nvidia]Ultra" upscaler pth.

[Haoming02]

https://github.com/Haoming02/sd-forge-nvidia-vfx

[ThesiiNCey]

https://civitai.red/models/934764/miaomiao-harem?modelVersionId=2967371 has also a info because hiresfix in the model info

[hirorohi03]

• If you need higher than that, use MultiDiffusion in img2img

That's a pretty cool img2img UI—different from the default settings.
I wonder what's inside the 2.0x button?

I'm upscaling the image before running Multidiffusion by adding upscaler_for_img2img in the Settings for img2img.

• Settings

• Img2Img

[Haoming02]

I wonder what's inside the 2.0x button?

Just a button that sets Resize by to 2.0 from https://github.com/Haoming02/sd-webui-yapping

[Phenrei]

So, after much debugging and fighting with this, I have worked out that while the Lora config json file for the VAE for Anima and WAN are mostly the same, just using is_wan flag for Anima's VAE processing causes multiple issues once you push beyond 1536x1536 or modify some other values (any shift above 1 and higher denoising on the HR pass is causing things to "cook" during VAE encode/decode), but they can be mitigated by other changes.

I'm still working deep into the code to figure out exactly what's at fault for some of the more extreme noise issues, but currently I have a copy of the wan_vae in backend/nn (and one for the patched 2D VAE that dynamically strips out the time tensor on demand) specifically retooled towards Anima's expected 96 dim value, because while the config is being loaded I'm still running into "screen door" or vertical line issues at higher dimensions due to what appears to be some issues in the HR pass logic I've yet to completely work out. In loader, putting AutoencoderKLQwenImage on it own case and using them there is working better than falling back on the WanVAE class, though not much else may have changed in it yet, and I'm not yet sure how best to handle existing Qwen2DVAE use.

The bigger change though was splitting Anima onto its own is_anima flag instead of piggybacking on is_wan when patching vae logic that allows branching to set some other values in the patcher (flat 1,8,8 and a,8,8 seem to work better so far in testing) and handle the differences in behavior with the 5D tensor arrays, especially in regards to the needs of tiled processing. A method to squeeze out the extra time value and run a 2D-like anti-aliased tile mode is working far better at extreme upscales. I'm getting acceptable noise levels after some other tweaks going 2x up from a 1536x1536 or even a 1920x1600 image (so generating a 4k+ image without any upscale-only or img2img pass, just base + HR + one 1x clean pass after with a dedithering model if needed) for reference as to how far that is allowing me to push it thanks to the low steps required when using the RES4LYFE-ported samplers.

I also ended up having to code a small plugin and modifications into the base diffusion_engine/anima.py functions for encode/decode to allow me to tweak the internal multipliers on its math and put a final outer adjustment and scale factor to work around the silly noise that base VAE adds (and/or use a community one that's come out with the effect lessened due to other color changes they made), but that was really only for the extreme cases and I wouldn't expect that to be merged, though if it's something I could hook without rewriting that inner piece of python it would be sharable as-is.

Either way, many of these things are just sort of messy hacks and I'm sure with more internal knowledge of the code you could rework them better, but just wanted to mention that I have identified it's both possible and just treating anima exactly like wan seems to have introduced some bugs locking out otherwise working options. If you'd like to see any of the changes I've made directly (once I can clean them up a bit), let me know, I can put in a PR or new issue.

Thanks.

[Haoming02]

This has nothing to do with the VAE, this is just the inherit problem with Anima

Anima-Preview-0.1 would generate artifacts at just 2 MP while Anima-Base-1.0 can generate up to 4 MP just fine

This is because Preview was trained on 512x512 while Base was trained on up to 1536x1536

Infotext

masterpiece, best quality, good quality, absurdres, newest, anime coloring, anime screenshot. safe.
1girl, solo, hatsune miku, vocaloid, casual, clothed, looking at viewer, smile
Negative prompt: score_1, score_2, score_3, blurry, jpeg artifacts, sepia, watermark, worst quality, low quality, large breasts, muscular, deformed hands, bad anatomy, extra limbs, poorly drawn face, mutated, extra eyes, bad proportions, character doll, chibi, old, early, censored, 3d, high contrast, ai-generated
Steps: 24, Sampler: Euler a, Schedule type: Normal, CFG scale: 4, Shift: 3, Seed: 1234, Size: 896x1152, Model: anima-preview, Model hash: 41fa7b7861, Module 1: qwen_3_06b, Module 2: qwen_image_vae, Denoising strength: 0.4, RNG: CPU, Hires Module 1: Use same choices, Hires upscale: 2, Hires Shift: 3, Hires upscaler: [Nvidia] Ultra, Hires steps: 16, Hires CFG Scale: 4, Script: X/Y/Z plot, X Type: Checkpoint name, X Values: "ani\\anima-preview.safetensors,ani\\anima-base-v1.0.safetensors [bd43b7cffe]", Version: neo-2.24

---

If you want even higher resolution, go finetune it further with larger dataset...

[Phenrei]

My findings are showing it's not a hard and fast limit, and it's mostly coming from the very noisy VAE they use and some silly defaults. If you try that same prompt with a shift 1 on the HR pass you should see far less noise even without any other changes, something about shift values + HR is just fundamentally broken and causes that pixelation pattern. Updates just to VAE processing logic to avoid some of the weirdness of using a 5D latent and VAE for purely image data not being put in a video are a big part of how I've gotten around what seemed like these limits, even without any tuning on the model itself. I get that you can take finished high res images into the img2img flow or just through normal ESRGAN to get similar results, but the things I've gotten out of this updated code are doing 3840 x 3200 (so over 12MP) purely in the txt2img flow already.

[Phenrei]

Oh, was doing additional research on the new Anima controlnet cleaner and saw a method to naturally convert the 3D VAE into 2D on the fly in those scripts (for apparently a 2x speed and 1/3rd ram), would probably make this even easier to test and write fixes for, and give more of a reason to actually add a unique anima logic flow for VAE: kohya-ss/sd-scripts#2382

I'll test an implementation of that in forge while I'm digging on this if possible, that that might save me some of the conversion headaches I've been having.

[Phenrei]

Sorry, bit slow on the uptake that I should have just run that sample prompt on my 5090 to show you (with my current settings it nearly runs OOM but I specifically turned up tile sizes aggressively and the thing I linked above will hopefully help that). Only ran 8 HR steps because they took like 9 seconds each, at slightly different settings I know are working, namely 1 shift, 0.2 denoise, and a higher CFG than the non-HD pass to avoid some of the sepia that bleeds in from the terrible VAE, though I think I went a little higher than was optimal. You can still see the sort of "screen door" effect at the edge of the 16x16 grid the VAE runs at are the remaining artifact on the image, which I'm still trying to work out exactly how to avoid fully:

With a custom VAE that was trying to fix some color issues the effect is lessened, as well as adjustments through my plugin to add slight boosts to the decode values (an outer 0.05 add and then a 1.05 multiply) and a slight reduction on the encode side of the exact opposite amounts (-0.05 removed and .95 multiply). Directly multiplying the inner tensors or other variable had mostly worse effects, but the plugin was built to support them to test. It's a bit hard to immediately tell the difference on such a bright image, but the file size is nearly 2.5mb smaller due to the reduction in noise: