YOLO keypoint detection

I trained a YOLO v8 model (using approximately 4000 images) to detect keypoints around the object. My object is cuboid shaped but it has features on every faces that are distinct. When I ran the prediction, 5-8% of the time keypoints are flipped (appears as if they are rotated by 180 degrees about the Y-axis of the 2D image plane) and my rotational error for the object goes to 180 degrees as well. I am curious if this has to do with my training, the shape of the object or something else. Any help is appreciated.

Did you set flip_idx in your dataset YAML?