YOLOv8 Multiple Object Tracking file into MOT Challenge format and access to Kalman Filter outputs

Support
, ,
1

Hi there, my name is Matheus.

  • Ultralytics version: 8.3.93
  • Python version: 3.10.12

I’m trying to do multiple object tracking (MOT) using a pretrained Ultralytics YOLO model and the ByteTrack algorithm on MOT20 Dataset. Furthermore, I also would like to evaluate the performance of this MOT task and predict the future positions of the tracked objects.

In a first approach, I used the model.track() function as described on this official Ultralytics page. I was able to do the MOT task and write informations about the tracked objects into a MOT Challenge formatted file. However, I could not find a way to get the Kalman Filter predictions. How could I do this? Is it possible? See the code below:

import ultralytics
from ultralytics import YOLO
from pathlib import Path

print('Running Ultralytics YOLO tracking...')
results = model.track(source=source,             # 'source' is a string containing the path for a directory of images
                      conf=confidence_threshold,
                      iou=iou_threshold,
                      imgsz=imgsz,
                      device=device,
                      stream_buffer=True,
                      stream=True,
                      persist=True,
                      classes=[1],
                      batch=32,
                      agnostic_nms=True,
                      tracker='bytetrack.yaml'
)

print('Writing MOT Challenge tracking file...')
with open('output.txt'), "w") as f:
        for result in results:
                result_to_device = result.to(device)
                if result_to_device.boxes.id is not None:
                      boxes = result_to_device.boxes.xyxy 
                      track_ids = result_to_device.boxes.id.int().tolist()
                      frame_id = int(Path(result_to_device.path).stem)

                      for box, track_id in zip(boxes, track_ids):
                          x1, y1, x2, y2 = box

                          bbox_left = x1
                          bbox_top = y1
                          width = x2 - x1
                          height = y2 - y1
                          
                          f.write(f"{frame_id},{track_id},{bbox_left},{bbox_top},{width},{height},1,-1,-1,-1\n")

Then, aiming to access the Kalman Filter predictions used internally by the ByteTrack algorithm, I tried to use the BYTETracker class as described in detail on this official Ultralytics page. However, I encountered many problems (KeyError, AttributeError, and others) with this second approach and I wasn’t able to execute the code correctly. See the code below:

import ultralytics
from ultralytics import YOLO
from pathlib import Path
from types import SimpleNamespace

args = SimpleNamespace(track_buffer=30)
tracker = BYTETracker(args, frame_rate=25)

print('Running Ultralytics YOLO tracking...')
results = model(source=source,                 # 'source' is a string containing the path for a directory of images
                conf=confidence_threshold,
                iou=iou_threshold,
                imgsz=imgsz,
                device=device,
                stream_buffer=True,
                stream=True,
                classes=[1],
                batch=32,
                agnostic_nms=True
)

for result in results:
        result_to_device = result.to(device)

        objects_in_this_frame = tracker.update(result_to_device.boxes)
        if len(objects_in_this_frame) > 0 and objects_in_this_frame is not None:
                tracked_objects.append(tracker.frame_id, objects_in_this_frame)

            
        print('Writing MOT Challenge tracking file...')
                with open(os.path.join(output, sequence_name+'.txt'), "w") as f:
                        for frame_id, track in tracked_objects:
                                min_x = track[0]
                                min_y = track[1]
                                max_x = track[2]
                                max_y = track[3]

                                track_id = track[4]
                                confidence = track[5]
                                class_label = track[6]
                                object_id = track[7]
                    
                                width = max_x - min_x
                                height = max_y - min_y

                                f.write(f"{frame_id},{track_id},{min_x},{min_y},{width},{height},1,-1,-1,-1\n")

Could someone help me, please?
Thank you in advance.

2

I will direct you to the ultralytics/trackers/track.py file, as it might be another good reference point if you have not reviewed it yet. Additionally, a few things of note:

  1. args = SimpleNamespace(track_buffer=30) does not contain all of the arguments passed to the tracker, so this could be part of the issue. You’ll want to load the ultralytics/cfg/trackers/bytetrack.yaml, like:
  1. In the ultralytics/trackers/track.py, there are a few functions defined, these are callbacks used by the model to execute object tracking during a call to the model.predict method. I can’t give you a guide on how to do it and don’t have sufficient time to investigate, but you might want to see if you can modify these callbacks to save or output (to the ultralytics/engine/results.py:Results object) directly.
  2. I suspect that you will likely be able to access the “raw” data from the tracks object returned when the update() method is called:
3

You can also check this where I use Kalman Filter predictions to interpolate detections in Ultralytics.

4

Thank you for the replies.
I’ll take a look in all the given referencies.

1 Like
5

Hello Matheus, thanks for reaching out and providing the details.

The model.track() function is indeed a high-level interface and doesn’t directly expose the internal Kalman Filter states or predictions.

To access the Kalman Filter predictions, using the BYTETracker class directly is the appropriate approach. The errors you encountered likely stem from how the tracker.update() method is called. It expects the full results object from the model’s prediction, which includes bounding boxes, confidence scores, and class labels, rather than just the .boxes attribute. You can find the details on the expected input and internal logic within the Reference for ultralytics/trackers/byte_tracker.py documentation.

After successfully calling tracker.update(result_to_device), you can access the state of active tracks through the tracker.tracked_stracks list. Each element in this list is an STrack object, containing attributes like mean and covariance which represent the Kalman Filter’s current state estimate (including position and velocity). You can inspect the mean attribute of these STrack objects for the updated state. If you need the predicted state for the next frame based on the current state, you could potentially call the predict() method on individual STrack objects after the update step.

Also, ensure the args object passed during BYTETracker initialization includes all necessary parameters expected by the tracker, such as thresholds, not just track_buffer. You might need to define attributes like track_high_thresh, track_low_thresh, etc., within your SimpleNamespace.

Let us know if adjusting the input to tracker.update() resolves the errors.

6

Thank you for your detailed response, @pderrenger.

Following your suggestions, I was able to correctly create the BYTETracker instance. Now, the new version of my code is showed below:

import ultralytics
from ultralytics import YOLO
from pathlib import Path
from types import SimpleNamespace
from ultralytics.trackers.byte_tracker import BYTETracker

args = SimpleNamespace(
               tracker_type="bytetrack",  # tracker type, ['botsort', 'bytetrack']
               track_high_thresh=0.25,    # threshold for the first association
               track_low_thresh=0.1,      # threshold for the second association
               new_track_thresh=0.25,     # threshold for init new track if the detection does not match any tracks
               track_buffer=30,           # buffer to calculate the time when to remove tracks
               match_thresh=0.8,          # threshold for matching tracks
               fuse_score=True            # Whether to fuse confidence scores with the iou distances before matching
)

tracker = BYTETracker(args, frame_rate=frame_rate)

print('Running Ultralytics YOLO tracking...')
results = model(source=source,
               conf=confidence_threshold,
               iou=iou_threshold,
               imgsz=imgsz,
               device=device,
               stream_buffer=True,
               stream=True,
               classes=[1],
               batch=32,
               agnostic_nms=True
)

objects_in_this_frame = tracker.update(results)
if len(objects_in_this_frame) > 0 and objects_in_this_frame is not None:
               tracked_objects.append(tracker.frame_id, objects_in_this_frame)

However, I’m still facing an error, which is:

AttributeError: 'Results' object has no attribute 'conf'.

I believe that this error occurs beacuse the tracker.update() method tries to access the conf attribute directly from results, i.e., results.conf. How would I overcome this issue? Should I pass one result instance at a time to the tracker.update() method? In other words:

for result in results:
         ... = tracker.update(result)

Or should I use the model.detect() method – as indicated in the Reference for ultralytics/trackers/byte_tracker.py – instead of just model() to create the detections before updating the tracker?

7

Yes, they are updated one at a time, notice the for-loop of predictor.results:

I also believe you need to extract the Results object like shown in the track.py code:

So before using objects_in_this_frame = tracker.update(results) you would add:

results = model(source=source,
               conf=confidence_threshold,
               iou=iou_threshold,
               imgsz=imgsz,
               device=device,
               stream_buffer=True,
               stream=True,
               classes=[1],
               batch=32,
               agnostic_nms=True
)

- objects_in_this_frame = tracker.update(results)
+ for i, result in enumeration(results):
+     detection = result.boxes.cpu().numpy()
+     objects_in_this_frame = tracker.update(detection, result.orig_img)
      # additional logic as needed
8

Thank you, @BurhanQ.
I was able to correctly execute the tracker.update() function. This is the final version of my code:

args = SimpleNamespace(
                tracker_type="bytetrack",  # tracker type, ['botsort', 'bytetrack']
                track_high_thresh=0.25,    # threshold for the first association
                track_low_thresh=0.1,      # threshold for the second association
                new_track_thresh=0.25,     # threshold for init new track if the detection does not match any tracks
                track_buffer=30,           # buffer to calculate the time when to remove tracks
                match_thresh=0.8,          # threshold for matching tracks
                fuse_score=True            # Whether to fuse confidence scores with the iou distances before matching
)

tracker = BYTETracker(args, frame_rate=frame_rate)

print('Running Ultralytics YOLO tracking...')
results = model(source=source,
                conf=confidence_threshold,
                iou=iou_threshold,
                imgsz=imgsz,
                device=device,
                stream_buffer=True,
                stream=False,
                classes=[1],
                batch=32,
                agnostic_nms=True,
                max_det=max_det,
)

tracked_objects = []
for result in results:
               result_to_device = result.to(device)

               det = result_to_device.boxes.cpu().numpy()
               if len(det) == 0:
                    continue

               objects_in_this_frame = []
               objects_in_this_frame = tracker.update(det, result_to_device.orig_img)
               if len(objects_in_this_frame) == 0:
                    continue
                
               tracked_objects.append([tracker.frame_id, objects_in_this_frame])

Now I can move on to the next step of my project, which is to use the Kalman filter information from the tracked objects to predict their future position.

1 Like