If by latency you’re referring to the inference FPS not being able to keep up with stream FPS, leading to desynchronization, then you can just add max-buffers=1 drop=True to appsink:
! appsink max-buffers=1 drop=True
This will drop old frames automatically and the inference will always use the newest frame.