From the course: Programming Generative AI: From Variational Autoencoders to Stable Diffusion with PyTorch and Hugging Face

Unlock this course with a free trial

Join today to access over 24,800 courses taught by industry experts.

Adding conditional control to text-to-image diffusion models

Adding conditional control to text-to-image diffusion models

From the course: Programming Generative AI: From Variational Autoencoders to Stable Diffusion with PyTorch and Hugging Face

Start my 1-month free trial Buy for my team

Adding conditional control to text-to-image diffusion models

“

- So, when I presented conditional generative models and when I presented the initial latent diffusion model paper, I presented it and mentioned that in theory, you can use any conditioning you want, it doesn't have to be text, it could be an image, it could be a class of something you want to generate, the conditioning, as long as you've trained the latent diffusion model with the conditioning, you can use that conditioning to guide the generation process. But, the downside to this approach is that if you want to do a new task, let's say you have a latent diffusion model and you want to do something like in painting or do something like a pose-generation model where you pass in a kind of skeleton pose and it generates a person in the same pose, if the model isn't trained for that conditioning, you'll have to retrain everything from scratch. Now, one approach that basically researchers have developed is ControlNet, which is a really clever way, similar to how LoRA works, where you…

Contents