@royalaid
Just to give a direction, when I played around with Stable Diffusion, the best and most consistent results were based around LORA fine tunes applied to specific SD models. I am sure the state has moved on from that but it might be what the end result looks like, a set of LORAs targeting a specific model that are referenced by a DB lookup.
This would probably require some bespoke architecture to dynamically load the LORAs into the pipeline, but I think it is tractable if a little cumbersome.
Cc @gm8xx8 and @july , they might have better info than me but it might also be completely outside of their wheelhouse.
Also consider maybe throwing it on bountycaster