Control lips, teeth and tongue for lip sync task.
#20
by Temir - opened
Is it possible to force sd model output precise lips, teeth and tongue position for stylize video? CrucibleAI trained control net model for face position. Maybe there is a way to train mouth as well?
Not with this control setup. :( We don't have a good pipeline for detecting teeth and tongues, as we lean on MediaPipe face. In the future, if another network comes along with this capability, we can try and train with that network.
Could you use MediaPipe's FaceMesh as control to provide much more detailed landmarks, perhaps?