You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Currently, it only supports running on the CPU. The CPU performance on Colab is not very strong, which results in slower processing. I'm currently working on optimizing its CPU performance and adding support for GPU acceleration.
Reacted by juniofaathir, Liu Xiaopai, Luigi Maselli and JohnClaw
Reacted by juniofaathir, Erik Scholz, Nobody and lin72h
@czkoko
i was able to convert "reliberate" but not realisticvision5.1 with baked VAE. if the civit ai model has a vae free version, you should be able to convert any of them. all major models have a huggingface link you should prefer over civit ai.
@czkoko you can use that model?? I've been trying some civitai model and converting it, but it didn't work like at #8
@juniofaathir Most of the SD 1.x models from Civitai are working fine, except for a few that include control model weights. I'm currently researching how to adapt these models.
Here's something interesting: I almost got a full generation on a 2gb 32bit mobile phone, before running out of ram. If someone has a better 32bit arm device, please see if the generation is successful.
This is determined by the characteristics of the ggml library, quantization can only be for the weight of the full connection layer, and the weight of the convolutional layer can only be f16.
Here's something interesting: I almost got a full generation on a 2gb 32bit mobile phone, before running out of ram. If someone has a better 32bit arm device, please see if the generation is successful.
@ClashSAN
I used Stable Diffusion v1.5 but in half precision mode (fp16) only. It took around 55 minutes to generate a 512x512 image on my phone (Snapdragon 888 chipset with 8GB RAM).
By the way, I've made a small optimization to make inference faster. I've tested it and it provides a ~10% speed improvement. Feel free to pull the latest code and give it a try. Just a reminder, don't forget to run the following code to update the submodule:
I've created a new benchmark category in the discussion forum and posted some benchmark information. You can also share your benchmark information there if you'd like.
I'm really digging this project. It's pretty interesting how the timing and memory usage don't really change based on the precision - unlike llama.cpp where speed scales linearly with precision (so q8 is twice as fast as f16). Whether it's f32, f16, q8, q4, they all take about the same time and memory. Also want to say it's noticeably slower than the OpenVINO version of Stable Diffusion. So, there's definitely room for improvement.
Here's something interesting: I almost got a full generation on a 2gb 32bit mobile phone, before running out of ram. If someone has a better 32bit arm device, please see if the generation is successful.
@ClashSAN I used Stable Diffusion v1.5 but in half precision mode (fp16) only. It took around 55 minutes to generate a 512x512 image on my phone (Snapdragon 888 chipset with 8GB RAM).
Activity
grigio commentedon Aug 20, 2023
ah ok, maybe #1
mjkrakowski commentedon Aug 20, 2023
it is rather slow, q8 is the fastest i guess
sdcpptest.ipynb.txt
mjkrakowski commentedon Aug 20, 2023
with 256x256px image size the q4_1 took about 8-9 minutes.
leejet commentedon Aug 21, 2023
Currently, it only supports running on the CPU. The CPU performance on Colab is not very strong, which results in slower processing. I'm currently working on optimizing its CPU performance and adding support for GPU acceleration.
h3ndrik commentedon Aug 21, 2023
My old Skylake PC takes about 38s per step for the 8bit model. (OpenBLAS doesn't seem to make a difference)
the f32 model about 40s per step.
My old Laptop from 2016 needs 90s per step with the 8bit model.
czkoko commentedon Aug 21, 2023
Sample Picture test on M1 16G, 5-bit, 512x768, 15 steps, euler a
The picture quality of q5_1 is quite good.
16-bit: memory < 3GB , 23 s/step
5-bit: memory < 2GB , 22.5 s/step
juniofaathir commentedon Aug 21, 2023
@czkoko are you using sd 1.5 ggml base model? I think your result is just too good for just an base model
czkoko commentedon Aug 21, 2023
@juniofaathir SD 1.5 base model can't generate such portrait, i use epicrealism
juniofaathir commentedon Aug 21, 2023
@czkoko you can use that model?? I've been trying some civitai model and converting it, but it didn't work like at #8
czkoko commentedon Aug 21, 2023
@juniofaathir There is no problem for me to use it. You can try the model I mentioned, or other training models, and filter merge models.
mjkrakowski commentedon Aug 21, 2023
@czkoko
i was able to convert "reliberate" but not realisticvision5.1 with baked VAE. if the civit ai model has a vae free version, you should be able to convert any of them. all major models have a huggingface link you should prefer over civit ai.
klosax commentedon Aug 21, 2023
Linking my tests using cuda acceleration (cublas) here #6 (comment)
leejet commentedon Aug 22, 2023
@juniofaathir Most of the SD 1.x models from Civitai are working fine, except for a few that include control model weights. I'm currently researching how to adapt these models.
ClashSAN commentedon Aug 22, 2023
@leejet hey, this implementation seems to use a very low amount of ram, lower and faster than using onnx f16 models. Thank you for your efforts!
It seems like the peak RAM usage stays at the minimum 1.4gb, when doing 256×384 images, using the current "q4_0" method!
Are you choosing a specific "recipe"?
like explained here: https://huggingface.co/blog/stable-diffusion-xl-coreml
The current composition of the model:
using these mixed quantization methods seems better than creating distilled models, they can be tailored and optimized for individual models..
ClashSAN commentedon Aug 22, 2023
Here's something interesting: I almost got a full generation on a 2gb 32bit mobile phone, before running out of ram. If someone has a better 32bit arm device, please see if the generation is successful.
leejet commentedon Aug 22, 2023
This is determined by the characteristics of the ggml library, quantization can only be for the weight of the full connection layer, and the weight of the convolutional layer can only be f16.
aparatext commentedon Aug 24, 2023
Tested with q4_0 of default v1.4 checkpoint.
nviet commentedon Aug 24, 2023
@ClashSAN
I used Stable Diffusion v1.5 but in half precision mode (fp16) only. It took around 55 minutes to generate a 512x512 image on my phone (Snapdragon 888 chipset with 8GB RAM).
The project works well on Android so maybe @leejet wants to update the supported platform list.
grigio commentedon Aug 24, 2023
leejet commentedon Aug 24, 2023
Glad to hear that. I'll update the documentation later.
leejet commentedon Aug 24, 2023
By the way, I've made a small optimization to make inference faster. I've tested it and it provides a
~10%speed improvement. Feel free to pull the latest code and give it a try. Just a reminder, don't forget to run the following code to update the submodule:juniofaathir commentedon Aug 24, 2023
@leejet do I need to
makeagain?leejet commentedon Aug 25, 2023
Yes, you need to make again
mjkrakowski commentedon Aug 25, 2023
fyi, GGML is deprecated and replaced by GGUF, people might like to slow down in creating ggml's in advance :>
https://github.com/ggerganov/llama.cpp
leejet commentedon Aug 26, 2023
I've created a new benchmark category in the discussion forum and posted some benchmark information. You can also share your benchmark information there if you'd like.
https://github.com/leejet/stable-diffusion.cpp/discussions/categories/benchmark
RedAndr commentedon Sep 27, 2023
I'm really digging this project. It's pretty interesting how the timing and memory usage don't really change based on the precision - unlike llama.cpp where speed scales linearly with precision (so q8 is twice as fast as f16). Whether it's f32, f16, q8, q4, they all take about the same time and memory. Also want to say it's noticeably slower than the OpenVINO version of Stable Diffusion. So, there's definitely room for improvement.
bitxsw93 commentedon Oct 10, 2024
you can try mnn-diffusion, on andriod 8Gen3, it can reach 2s/iter, and 1s/iter on Apple M3, with 512x512 images.
reference: https://zhuanlan.zhihu.com/p/721798565
source code: https://github.com/alibaba/MNN/tree/master
cmake : configure CMAKE_C_FLAGS and target_link_libraries for MSVC (l…