Fooocus 2.1.0 Image Prompts (Midjourney Image Prompts)

[lllyasviel]

Fooocus 2.1.0 has completed the implementation of image prompts. Because after this version, almost all features of Midjourney are included, the version directly jump to 2.1.0.

Image Prompt is one of the most important feature of Midjourney. Below is the banner from Midjourney:

In Fooocus, it looks like this:

Technically, this feature is based on a mixture of IP-Adapter, and a pre-computed negative embedding from Fooocus team, an attention hacking algorithm from Fooocus team, and an adaptive balancing/weighting algorithm from Fooocus team.

The motivation of these efforts is to achieve a best match to the Midjourney Image Prompt. In other software like A1111/ComfyUI/InvokeAI, the IP-Adapter still has some open problems like ignoring text prompts, or over-burned results when multiple images are used. These problems are solved in Fooocus and users can enjoy Midjourney-like experience of Image Prompt.

The detailed differences are in the below table:

	Midjourney Image Pompt	IP-Adapter + A1111/ComfyUI/InvokeAI	Fooocus Image Prompt
Work together with text prompts	Text prompts and image prompts will be mixed	Tends to ignore text prompts	Text prompts and image prompts will be mixed
Multiple images as input	Result quality does not decrease for multiple image inputs	Using more images leads to worse result quality	Result quality does not decrease for multiple image inputs
When the method fails (single image inputs)	Give unrelated but still high-quality image	Give related but low-quality and over-burned image	Give unrelated but still high-quality image
When the method fails (multiple image inputs)	Partially ignore some images that it cannot understand, still give high-quality results	Give related but low-quality and over-burned image	Partially ignore some images that it cannot understand, still give high-quality results
Quality Influence	Using image prompt does not influence the output quality	Using image prompt influences the quality of base model	Using image prompt does not influence the output quality, almostly
Result diversity	Results are still diverse after using image prompts	Results tend to have small and minimized variations	Results are still diverse after using image prompts

Using this method will download 2.5GB files at the first time!

Example: Single Image Prompt without Text Prompts

(Non-cherrypicked random batch, default parameters, real results should be better if tuned)

(seed 1234, here is the image)

(this example uses default style and Fooocus V2 style)

Example: Single Image Prompt with Text Prompts

Note that mixing text and IP-Adapter is extremely difficult in ComfyUI/A1111. Fooocus does not have this problem.

(Non-cherrypicked random batch, default parameters, real results should be better if tuned)

(this example uses default style and Fooocus V2 style)

Example: Multiple Images without Text Prompts

Note that mixing multiple IP-Adapters is likely to cause lower result quality in ComfyUI/A1111. Using Fooocus can resolve this to some extents.

(Non-cherrypicked random batch, default parameters, real results should be better if tuned)

(this example uses default style and Fooocus V2 style)

Example: Multiple Images with Text Prompts and Even Multiple Styles

This is almost impossible in A1111/ComfyUI since mixing text and IP-Adapter is extremely difficult in ComfyUI/A1111, and mixing multiple IP-Adapters is likely to cause lower result quality in ComfyUI/A1111.

(Non-cherrypicked random batch, default parameters, real results should be better if tuned)

This image is too complicated to understand so I annotated here:

So mixing too many things make it hard to recognize but everything is there and it does not fail or causing quality decerase, unlike ComfyUI/A1111/InvokeAI.

Fooocus Image Prompt (Advanced)

If you check “advanced”, you will be able to use two structure controls:

PyraCanny: A pyramid-based Canny edge control. The reason is that SDXL uses 1024px images and standard Canny tends to miss some image details from time to time, at such a high resolution. This method uses multiple resolutions to detect canny edges and then combine them softly, so that more structures are captured (than canny). The pyramid part is from “Edge Drawing: A combined real-time edge and segment detector”. You will download 350MB control models when using it.

CPDS: A structure extraction algorithm from “Contrast Preserving Decolorization (CPD)”. The “CPDS” means CPD Structure. The control model is modified by Fooocus team – it starts from SAI’s depth control-lora. The reason for using this method is for the fast speed and download-free preprocessor. Note that we only use the structure part of images, and it is not really “decolorization”. You will download 350MB control models when using it.

(Non-cherrypicked random batch, default parameters, real results should be better if tuned)

(this example uses default style and Fooocus V2 style)

(this example uses default style and Fooocus V2 style)

(this example uses default style and Fooocus V2 style)

For developers:

In Developer Debug Mode, you can mix the upscale/vary/inpaint with all above features if you know what you are doing and REALLY need it (the denoising strength can also be set in Developer mode). You can also get the preprocessor result by checking the “debug preprocessor”.

But keep in mind:

If you accidentally get satisfying results in Fooocus by tuning a lot of advanced parameters, you should try to copy your positive prompt, reopen Fooocus, do not change anything, and paste the prompt. You will find that results are even better, and all those tunings are unnecessary. (The only exception is probably changing base model in “Advanced”.)

[eric23]

Awesome!

[YANGlattez]

cool！

[woshitoutouge]

OMG!

[sherozbek1706]

Using more images leads to worse result quality

[lllyasviel]

Different from A1111 or ComfyUI, using more images does not lead to worse result quality in Fooocus.
If you find unexpected examples, may directly open an issue.

[mirek190]

wow .... fooocus is getting better and better

[3Diva]

This looks like an incredible update! Could someone help me use it please? When I try to put an image in the image prompt and render I just get an error message:

ValueError: Query/Key/Value should all have the same dtype
query.dtype: torch.float16
key.dtype: torch.float32
key.dtype: torch.float32

EDIT: It works fine if I select "PyraCanny" or "CPDS" but if I try to use "Image Prompt" is gives me that ValueError.

[lllyasviel]

fixed in 2.1.19

[3Diva]

@lllyasviel That works great now! Thank you so much!

[ayushyadav547]

Mine still shows error in FaceSwap

[felipesmaia]

Me too.. Error in any function in input image

[puntomaupunto]

same for me. In the output of the script there is a ctl-C.

[mirek190]

hard to say ...for me works ok

[lllyasviel]

2.1.19:

PyraCanny improved a bit
Note that CPDS may be changed in the future.

[lanyusan]

@lllyasviel the new release is awesome.

Is there anyways to do style blending like this?

Or like what is done here:

https://www.tensorflow.org/tutorials/generative/style_transfer

I have tried the latest image prompt but couldn't get similar results.

[lanyusan]

So I have tried a few more times and manged to get some thing like this. I have to uncheck all styles.

[lllyasviel]

you can get this if you do not use bad parameters and just keep everything as default, but not sure if this is what is expected because the style itself is messy

(seed 710945904

(v2.1.22

[lanyusan]

I have tried a few more times with different samples. It seems for text prompt free style transfer, PyraCanny works the best.

[wajihassan001001-dot]

i tried so many times but idont get same results of my model

[wajihassan001001-dot]

@lllyasviel please let me what i am doing wrong ?

[lllyasviel]

fixed some errors in CPDS in 2.1.24

[3Diva]

The new features are so cool. The "Image Prompt" one seams a bit like "Revision". I'm enjoying the pose control we get from "PyraCanny" too! Very nice!

Thank you for the sweet new features!

[lllyasviel]

2.1.24:

You can now drag here to resize the area:

for example

[lslsl3q]

you are my hero

[barepixels]

my try, wish there is a way to strengthen his face likeness, no style selected

here are the source images. maybe someone can do better

I think this is a better version. Wish I can just load the image with metadata and resume where I left off.

[lanyusan]

This s great. Would you please share a screen shot of your configuration for generating it?

Thanks.

[barepixels]

There is no metadata embedded in the png. Unlike Foocus-MRE one can not restore with all the settings to continue where left off. This is all I have in my log. Log did not record new Image Prompt settings.

[lanyusan]

Thanks. Did you use PyraCanny or CPDS or just plain image input?

[barepixels]

I did a whole bunch of testing I don't remember

[Marcelx8]

I'm very new to this, but I'm playing around with Fooocus daily since about 2 weeks ago. You mentioned metadata - would it be possible to first do an upscale version of the image, or subtle variant, and then only try and combine the 2 styles with the output images of the variants/upscales? Would that process generate the meta required? If you respond with "nope" 😅, then I'll first need to learn more about these things, hehe.

The reason I asked is because I think that is how I got an image to create a better variant from the original. Again, maybe that was just by coincidence on my end.

[lllyasviel]

Not an update but another message

This is a bird image

(generated by prompt "bird" all default parameters)

turn it into a "dragon", all default parameters, seed 123

You can see that the mixture between text and image is extremely robust and works in 100% cases.

Some users may think that they can just change the weight of IP-Adapter in other software like A1111/ComfyUI/InvokeAI or even diffusers to get similar performance and similar robustness. Unfortunatly, this is not possible, to the best of my knowledge.

If you know how to achieve such robustness in other software, please let me know.

[lllyasviel]

interesting.

Although we only want to compare open-source software, comparing with commercial online services is also possible.

[lanyusan]

Promeai follows the first image's face more faithfully. It might be more preferable for ordinary users to have fun with their photos.

Would be great if Fooocus can detect faces and keep faces more faithfully.

[lllyasviel]

Promeai's results are a typical case of "fail to make variations".

I do not think their generated face is the uploaded face. It is just a random face. But their model fail to produce enough variations so that users have less choices.

Fooocus also generate in a random way. But fooocus succeeded in making variations at least.

[lanyusan]

Yes I understand.

I am thinking about ordinary users' cases. One common case is to restyle a portrait but keep face faithfully.

It will make Foocus a lot more appealing for those people if the img2img or image prompt functions have an option to preserve human faces more faithfully.

Just an idea.

[lllyasviel]

You can try Anvanced -> Advanced -> Developer Debug Mode -> Softness of ControlNet -> set to lower value or 0 if really need this.

[lllyasviel]

Hint:

you do not need to turn off "Fooocus V2" in most cases.

"Fooocus V2" is handled in a different way than text prompts. You do not need to worry about unwanted texts are added to your prompts.

[Greshnic07]

Помогите плиз, никто не может ответить на мой вопрос: как например готовое фото девушки (после удаления BG ) поставить на готовый фон (например поле кукурузы) чтобы она гармонично вписалась туда ?
Всегда сохраняется только поза(

[valtezar]

Вероятно всего, придется, поместить ее в нужное место в фотошопе, а затем вернутся в фукус, и обработать края.

[martinenkoEduard]

What is the best model or method to make faces similar but not identical, like siblings (brother/sister)? Also, how can I maintain the same style and posture?
Similar to face-swapping, but not exactly like that. I want to create something similar, like a brother or sister.

[Yogesh-DevHub]

Describe the gender and age in prompt, use "image prompt / face swap" option by decreasing the "weight / stop-at", after few attempts you should get the idea for weight and stop-at values to get somewhat similar face image, like siblings.
For poses upload one more image select PyraCanny.

A video with quick explanation about what I said above.
https://youtu.be/fnhjSC4T69o?si=xA2kl45SIF1Cfh3v

[arii-jain]

As a beginner, I have a simple photoshop use case, I want to change the fabric of mannequin model. Tried several PnCs of inpaint and image prompt, but to no avail. Can someone please guide the correct steps?

[FrankEnglish]

please can you explain me the different use of "stop at" and "weight"? whic is the result on the final immage created? it's not clear, tnx

[Yogesh-DevHub]

The process of image-to-image generation needs an image as a reference to generate a final result.
If this process takes 10 seconds to generate a result, then 'stop at' defines when fooocus should stop taking reference from the given image and start using his own creative skill to generate within that 10 second.

This image-to-image generation option takes multiple images at a time. While taking reference from multiple images, which image should be considered more for taking reference is managed by weight. More weight will include more elements from the respective image in the final result.

A YT video with same examples: https://youtu.be/x9QSBKOo9Zo

I am not a tech person aware how things work inside fooocus but this are my findings after using the application for months.

[puntomaupunto]

sorry for the dumb question, but is there a way to run Fooocus 2.5 from colab? I noticed the warning "You are using gradio version 3.41.2, however version 4.44.1 is available, please upgrade." and I think that maybe I am using an old notebook.

[Yogesh-DevHub]

I don't think there is an issue with the old notebook, please check #2911 (comment)

You can ignore that warning.

[puntomaupunto]

thank you!

[maochoa31415]

How I can configure (via Docker) fooocus to use the CPU?

[plasma800]

Broken image links in this guide. Guide still useful, but though you might want to know.

[YavuzErgul40]

Hello. I would like to thank each and every one of them for their work. He does great work. I have a request from you, please provide the possibility of outputting as vector or SVG. (If you have, can you tell me how to do it? I searched but I could not find it). Thank you very much again

[Siddharth97D]

Hello,

Great work, team! I would like to integrate this using an API—is this possible and permitted? I reviewed the Gradio API documentation included in the UI, but it seems a bit confusing. Could you please assist me? I also want to contribute in terms of development if possible.

Thank you in advance!

[vincentansonfrancis]

sir it shows this when i tried to collab RuntimeError: Found no NVIDIA driver on your system. Please check that you have an NVIDIA GPU and installed a driver from http://www.nvidia.com/Download/index.aspx.

am using macmini

[Yogesh-DevHub]

Select a GPU

[gkcre8tive]

Is there a way to swap out faces within characters generated out based on various movie scenes and characters, but using a photo of a real person in place of the one within the character generated, what would be the best way to describe such a prompt within Foocus?

[Shekhouse]

So I am trying to get faceswap these images. I am trying to get the woman with the orange glasses her face onto the sleeping one. now its not working for me, I am getting the most random outcomes. I have tried alot of different settings and ways but I hope there is someone who knows how to do it and can explain it

[kouroshmohseni]

If I want to recreate these sculptures, how can I do it? I have 3 photos as a reference.

[bhadani2578]

I have a fabric design image (shown above). Using Fooocus, how can I create a men's kurta with the same fabric design and color applied to it?

[joseph-fuzzylogix]

What should be the stop at and weight values if I want my 1st image to make the same pose as what is shown in 2nd image. Keeping the rest same. Only the pose of the model needs to be changed.

[mashb1t]

You probably want to apply the ControlNet for pose all the way, so stop at should be 1.

Feel free to open a new discussion instead of posting comments in an old release post though.

[bilalhassankhan68-svg]

hey how we can create uncenssored images in foocus colab
i am giving but foocus colab not creating whole images