Does YOLOE leverage prior knowledge when fine-tuning?

Discussion
1

When fine-tuning YOLOE (either full fine-tuning or linear probing), does it initialize with it’s already existing embedding to detection knowledge of the
for example, if I am fine-tuning a model to detect “person” and “flame” does it start with its existing knowledge of those classes

In essence, I’m trying to understand if the model starts with a “head start” on the classes it already knows.

simple code example:

from ultralytics import YOLOE
from ultralytics.models.yolo.yoloe import YOLOEPESegTrainer

model = YOLOE("yoloe-11s-seg.pt")

results = model.train(
    data="flame_and_person.yaml", epochs=100, trainer=YOLOEPESegTrainer,
)

If it is the case that fine-tuning on YOLOE leverages prior knowledge, is there a downside to fine-tuning YOLOE11 instead of YOLO11 considering that on exported inference the runtime performance should be identical

Note: I would assume “Catastrophic Forgetting” for all other classes, similarly to closed-set object detector such as YOLO11

Thanks!

3

Yes, the embeddings are pre-calculated for the classes in your dataset and used as initialization to get a head start.

There are no downsides other than slower training compared to regular YOLO. YOLOE probably works better for fine-tuning because it has undergone pretraining on a much larger dataset.

1 Like
4

To Quantify the “slower training” on my GTX 1650 Ti Mobile (4 GB), no training overhead was measured

Model Time VRAM Used
YOLO11m 102 minutes and 1.8 seconds 2.46G
YOLOE11m 102 minutes and 26.2 seconds 2.45G
Note: time is total script execution time for 5 epochs including initialization evaluations, etc, and VRAM usage as reported by Ultralytics

Based on these measurements, I would conclude that YOLOE fine-tuning has no downsides, other the lack of “nano” and “extra-large” models

Scripts:

# Train Yolo
from ultralytics import YOLO


model = YOLO("yolo11m.pt")

results = model.train(
    project="fire-test-runs",
    name="yolo11m-od-flame-tuning-test",
    data="./fire-flame-1_u_seg/data.yaml",
    epochs=5,
    batch=2,
    device=0,
    close_mosaic=0,
    plots=True,
    save_period=10,
    resume=False,
    exist_ok=False,
    multi_scale=False,
)

# Train YOLOE Object Detector
from ultralytics import YOLOE
from ultralytics.models.yolo.yoloe import YOLOEPETrainer as Trainer

model = YOLOE("yoloe-11m.yaml").load("yoloe-11m-seg.pt")

results = model.train(
    project="fire-test-runs",
    name="yoloe11m-od-flame-full-tuning-test",
    data="./fire-flame-1_u_seg/data.yaml",
    epochs=5,
    batch=2,
    device=0,
    trainer=Trainer,
    close_mosaic=0,
    plots=True,
    save_period=10,
    resume=False,
    exist_ok=False,
    multi_scale=False,
)
1 Like
5

Thanks for sharing your findings!

6

Hi, I have question if anyone could help with. I would like to do linear probing yoloe, for some custom classes for my retail project. For example, customer taking an item, customer watching, etc. Is it only enough to train the Classification head, with my custom dataset, wich contain only the related images? or, I should train both coco and then my custom dataset for the additional classes? I have confused that if I train the custom classes, the model may forget the pre-trained coco.

7

You would have to train on combined dataset of COCO and your custom classes if you don’t want the model to forget existing classes.

And you’re trying to perform action recognition which is not suitable with object detection alone. You will have lots of false positives. Object detection only uses single frame for prediction. It doesn’t have temporal context.

1 Like
8

Thanks for that. I will try this. I am trying to first extract the action by frame and then analyse with LSTM. I will try and comment back.

9

That sounds like a better direction :+1:. I’d use YOLOE/YOLO as the per-frame spatial extractor, then let the LSTM handle the temporal action logic. For retail actions like “taking an item” or “watching,” the useful sequence features are usually track_id, person/item boxes, class confidences, box motion, hand/object distance if available, and optionally pose keypoints.

Ultralytics tracking can help keep the same person/item identity across frames using model.track(..., persist=True), and you can try botsort.yaml or bytetrack.yaml depending on the scene. The tracking docs have examples for this. (docs.ultralytics.com)

A simple starting point would be:

results = model.track(frame, persist=True, tracker="botsort.yaml")

for box in results[0].boxes:
    cls = int(box.cls)
    conf = float(box.conf)
    track_id = int(box.id) if box.id is not None else -1
    xyxy = box.xyxy[0].tolist()

Then build short windows per track_id and train your LSTM/action classifier on those windows. I’d validate the frame-level detector/tracker first, because most downstream action errors usually come from missed detections, ID switches, or ambiguous labels rather than the L