ControlNet

ControlNet can best be described by the the very scientists who developed it: "We present a neural network structure, ControlNet, to control pretrained large diffusion models to support additional input conditions. The ControlNet learns task-specific conditions in an end-to-end way, and the learning is robust even when the training dataset is small (< 50k)." https://arxiv.org/abs/2302.05543

In other words, in addition to the word prompts and other numeric parametric inputs, the user can introduce an additional model to further indicate the desired output. There are many different ways this is done.