Modifying yolo11 architecture to have one backbone and 2 necks and heads

Discussion
, , , ,
1

Hi,
I am interested in modifying the architecture of yolo11. I want to keep the backbone as it is, and attach it to 2 necks and heads. My goal is to freeze the pretrained backbone and train the two necks to detect different objects. Can anyone explain in detail how this can be done. So far I haven’t managed to find too many discussions on modifying yolo11.

Thanks

2

Yeah, you won’t because the number of ways people want to modify a model are so varied that there could never be a way to write a “guide” how to do it. Most models are developed by testing various configurations, so your best best is to test things out yourself.

The big question to consider is why have two heads? You can train a single head model that can detect multiple classes.

3

Thanks for your response. I am not looking for a “guide” for this. However, it would be great if you could share something general about how modifications can be mode to the architecture. For instance, do I need to play around with certain files for this?

4

Hello big_tree,

To modify the YOLO11 architecture, you’ll primarily be working with the model configuration YAML file. This file defines the structure of the model, including the backbone, neck, and head. You can create a custom YAML file that specifies one backbone and two necks and heads.

The parse_model function, found in ultralytics/nn/tasks.py, processes this YAML file to construct the PyTorch model. You can examine the logic in parse_model to understand how the different modules defined in your configuration file are created and connected. The function iterates through the layers defined in the "backbone" and "head" sections of your YAML, instantiating the corresponding PyTorch modules.

You can define two separate heads and necks in your configuration file by adding entries in the head portion of the YAML, each with its own structure. Refer to the tasks - Ultralytics YOLO Docs for how to do this.

Remember to adjust input and output channels appropriately to ensure compatibility between the layers.

5

Might be good to reference some of the examples modifications in this repo.

There’s a lot to consider when it comes to modifying the model layers. The simplest way would be to modify the YAML config for each model without changing a layer type (unless it exists in the nn/modules already). You can try changing the parameters or remove layers, but it’s likely to break things more often than not.