2x performance increase with new nvidia drivers

[ghost]

Found this today: https://www.reddit.com/r/StableDiffusion/comments/13q4ku4/nvidia_2x_performance_improvement_for_stable/

This reads like its tensorrt but its coming straight from Nvidia. Models will need to be converted just like with tensorrt. Release tomorrow

Apparently AMD cards support it too: https://gpuopen.com/amd-microsoft-directml-stable-diffusion/

[tkiikt]

https://blogs.nvidia.com/blog/2023/05/23/microsoft-build-nvidia-ai-windows-rtx/

On May 24, we’ll release our latest optimizations in Release 532.03 drivers that combine with Olive-optimized models to deliver big boosts in AI performance. Using an Olive-optimized version of the Stable Diffusion text-to-image generator with the popular Automatic1111 distribution, performance is improved over 2x with the new driver.

Optimize DirectML performance with Olive
Stable Diffusion Optimization with DirectML

[Tillerz]

:-o Does stable_diffusion.py --optimize work on windows?

[Centurion-Rome]

"The stable diffusion models are large, and the optimization process is resource intensive. It is recommended to run optimization on a system with a minimum of 16GB of memory (preferably 32GB). Expect optimization to take several minutes (especially the U-Net model)."

[ghost]

I also have only 8 but I think renting a GPU will be feasible for conversion. You could convert your favourite models for a few dollars and then be done with it.

But as with all speed booster news in SD I dont want to be too optimistic about this yet.

[Silanda]

I also have only 8 but I think renting a GPU will be feasible for conversion. You could convert your favourite models for a few dollars and then be done with it.

That says RAM, not VRAM.

[mark-wd]

Let's see how it fares in real world usage.

[vladmandic]

Yeah, I don't have much faith in this.

"Olive" optimized means compiling model from Torch to ONNX and then using new ONNX backend with built-in tensor cores support.
Which would mean that entire SD ecosystem should stop using Torch/CUDA/ROCm/etc and switch to ONNX - highly unlikely.

And if going down that path, I'd much rather support TensorRT directly instead of ONNX unless someone says that ONNX is next-best-thing and will completely surpass Torch in the near-future.

Microsoft uses ONNX internally in Azure for ML-as-a-Service, so this makes nVidida a much more interesting choice for MS and I get why nVidida would do it (why wouldn't they?).

[Tillerz]

Ah poo, I had high hopes for a speed bump. :( I mean, with an 4090 I cannot complain, but if there's MORE, why not have it. :-p

[maikelsz]

Yeah, I don't have much faith in this.

"Olive" optimized means compiling model from Torch to ONNX and then using new ONNX backend with built-in tensor cores support. Which would mean that entire SD ecosystem should stop using Torch/CUDA/ROCm/etc and switch to ONNX - highly unlikely.

And if going down that path, I'd much rather support TensorRT directly instead of ONNX unless someone says that ONNX is next-best-thing and will completely surpass Torch in the near-future.

Microsoft uses ONNX internally in Azure for ML-as-a-Service, so this makes nVidida a much more interesting choice for MS and I get why nVidida would do it (why wouldn't they?).

my GPUs are AMD and I work on Windows. I am not going to use Linux or spend on NVIDIA. If this gives me more performance in my configurations, fine, I'll use it, others can continue using CUDA/ROCm

[lshqqytiger]

I had a look this yesterday.
It generated a 512x512 image with 2.03it/s on my RX 5700 XT. (1.5it/s on vladmandic's)
And it will be faster if you are using NVIDIA card and on the latest driver.
But it is for ONNX Runtime, so it is difficult to apply it right now.

[aifartist]

I never got the Olive stuff to work on native Ubuntu.
I have been doing my own conversions of my models from PYT/DiffUsers -> onnx -> optimized onnx -> TRT using my own pipeline.
The speed is great 90+ it/s but if I compare side by side the quality does suffer. AITemplate is slower than TRT but the quality seems good but I haven't done an exhaustive comparison.

I have a theory that it is the ONNX intermediate stage that is messing up the quality during the conversion. I see many large values clamped to INT32 and subnormal FP16 value warnings. A1111 does crisp images using fp16. I don't see why a compiled TRT model using fp16 should be worse unless during certain intermediate processing some weights were screwed up. I can't get confirmation on this theory although I did post an issue on the onnx github.

[Tillerz]

Is SD way faster on Linux than windows with the same quality?

[mark-wd]

I never got the Olive stuff to work on native Ubuntu. I have been doing my own conversions of my models from PYT/DiffUsers -> onnx -> optimized onnx -> TRT using my own pipeline. The speed is great 90+ it/s but if I compare side by side the quality does suffer. AITemplate is slower than TRT but the quality seems good but I haven't done an exhaustive comparison.

I have a theory that it is the ONNX intermediate stage that is messing up the quality during the conversion. I see many large values clamped to INT32 and subnormal FP16 value warnings. A1111 does crisp images using fp16. I don't see why a compiled TRT model using fp16 should be worse unless during certain intermediate processing some weights were screwed up. I can't get confirmation on this theory although I did post an issue on the onnx github.

90 it/s is near real time rendering...

[vladmandic]

Is SD way faster on Linux than windows with the same quality?

No. Its the same, there is a slight difference in interrupt handling as well as vram overhead due to how drivers work, but at the end no performance difference.

[aifartist]

I never got the Olive stuff to work on native Ubuntu. I have been doing my own conversions of my models from PYT/DiffUsers -> onnx -> optimized onnx -> TRT using my own pipeline. The speed is great 90+ it/s but if I compare side by side the quality does suffer. AITemplate is slower than TRT but the quality seems good but I haven't done an exhaustive comparison.
I have a theory that it is the ONNX intermediate stage that is messing up the quality during the conversion. I see many large values clamped to INT32 and subnormal FP16 value warnings. A1111 does crisp images using fp16. I don't see why a compiled TRT model using fp16 should be worse unless during certain intermediate processing some weights were screwed up. I can't get confirmation on this theory although I did post an issue on the onnx github.

90 it/s is near real time rendering...

That is what I'm working towards.

[tkiikt]

a test run on Windws: https://youtu.be/r-ZYOlZTLDA

[aifartist]

i think even to hit 70it/s cores were used for some ops, i don't think tensorrt is that much faster than torch on its own (it is faster, but not 50%). but then stars aligned and you get tensor cores getting fully utilized and hit 96 which is a near-magic number for 4090.

i'm really curios how does aitemplate perform?

VoltaML does 40.9 it/s on models in general.
With AIT 512x512 Euler_a 25 steps I get:
100%|████████████| 25/25 [00:00<00:00, 70.70it/s]
100%|████████████| 25/25 [00:00<00:00, 72.25it/s]
100%|████████████| 25/25 [00:00<00:00, 72.22it/s]
100%|████████████| 25/25 [00:00<00:00, 72.00it/s]
which is as good as I'm seeing right now with my messed up TRT env.

When I first used VML it was quite hacky and experimental. They've come a long ways.
The AIT path seems sound. If you wanted to try it I could walk you through install, setup, and use on Linux if you have Linux.

[vladmandic]

The AIT path seems sound. If you wanted to try it I could walk you through install, setup, and use on Linux if you have Linux.

sure!

[vladmandic]

btw, one concern is purely teorhetical - model needs to be "baked in" for all those optimizations to work, which means that lora's cannot be applied on top of it. if you want to use lora/lyco, you'd need to merge it with model (and you get sort-of dreambooth) and then bake that. not really feasible in production if you want to use any kind of trained stuff.

[aifartist]

The AIT path seems sound. If you wanted to try it I could walk you through install, setup, and use on Linux if you have Linux.

sure!

First you need AIT
git clone --recursive https://github.com/facebookincubator/AITemplate
cd AITemplate
python3 -m venv `pwd`/venv
source venv/bin/activate
cd python
python3 setup.py build # The instructions say use "bdist_wheel" but that doesn't work.
cd ../..

git clone https://github.com/voltaML/voltaML-fast-stable-diffusion
cd voltaML-fast-stable-diffusion
python3 -m venv `pwd`/venv
source venv/bin/activate
pip3 install ../AITemplate/python/build # main installs everything but this.
python3 main.py

Wait till the install finishes and connect browser to http://127.0.0.1:5003/
Find tiny DOWNLOAD icon on left edge, choose model and download.
Click tiny left hand icon for "txt2img" can click the "Load Model" button. Click "Load" on the model you want. Note the 3 tabs "PyTorch", "AITemplate" and "Extra", Right now you want a PyTorch model.
Return to txt2img and generate an image changing standard setting like sampler or batch count. On a fast system you show get around 40.

AITemplate:
Click the small Accelerate icon on the left edge.
Select the model at the bottom.
Perhaps raise the number of cpu threads to do the compile from 8 to 24 depending on your cpu and how idle your system is. This should just affect the compile time.
Click "Accelerate" and wait.
When done return to "txt2img" and click Load Model and click on the AITemplate tab.
You should see a model now there. Load it and generate an image.

If anything goes wrong describe it to me and wait till I see your message and respond.
I'd like to see what kind of speed you see on your setup.

[lshqqytiger]

btw, one concern is purely teorhetical - model needs to be "baked in" for all those optimizations to work, which means that lora's cannot be applied on top of it. if you want to use lora/lyco, you'd need to merge it with model (and you get sort-of dreambooth) and then bake that. not really feasible in production if you want to use any kind of trained stuff.

I agree. There will be many things which cannot be migrated to ONNX ecosystem.

[mark-wd]

That's what I was going to ask... if the underlying technology can't just be harnessed without needing to go about this whole route.

[mark-wd]

https://www.reddit.com/r/StableDiffusion/comments/13u0kha/optimization_comparison_in_a1111_13_tensorrt_vs/