Adding hooks to models

Discussion
1

Hello,

I was trying to debug a segmentation model by adding hooks to its hidden layers to track activations.

However, because Yolo models always pass true to the key word “fuse” in the autobackend.py
the hooks end up being removed upon inference.

I have created a pull request with a short fix that involves adding a new key word argument when creating a yolo model and a field called self.fuse_layers which controls whether or not batch normalization will be fused with adjacent convolutional 2d blocks. I do not know if pull requests are reviewed by ultralytics team very often, so I just wanted to make a post here asking if this might be a reasonable addition to the library.

Thanks.

2

There are folks reviewing the PRs. I don’t know much about the section of code you’re proposing to change (meaning it’s my ignorance), but it sounds like this might be a change that has a very limited use case. I can tell you from experience that usually those aren’t accepted, just because it’s difficult to justify a change that would not be beneficial to many other users that we’re aware of. If it’s a reasonably small fix and doesn’t have larger implications, then it’s much more likely to be accepted.

3

The changes are minimal, and local to Yolo. It allows users to choose to fuse modules during inference, which was not available before. This allows debugging of activations. Which has many uses from interpretability to collecting data in general. It is a simple change, but it allows convenience. So far all tests have passed.

4

I think you can achieve this by just doing

model.model.is_fused = lambda: True

which will prevent it from being fused.

5

I do not think so. The call of the Yolo model calls/creates a new predictor during inference. I believe this is done every time. It automatically passes true when setup model is called to auto backend.

6

If the is_fused attribute returns True, then the model will not be fused, even during prediction:

In [1]: from ultralytics import YOLO
In [2]: model = YOLO("yolo11n.pt")

In [3]: model.model.is_fused = lambda: True

In [4]: results = model(verbose=False)
WARNING ⚠️ 'source' is missing. Using 'source=/ultralytics/ultralytics/assets'.

In [5]: model.model.model[0]  # BatchNorm layer still present after prediction
Out[5]:
Conv(
  (conv): Conv2d(3, 16, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
  (bn): BatchNorm2d(16, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
  (act): SiLU(inplace=True)
)
7

You are correct. model.model.is_fused does now allow hooks to be added. I will remove the pull request then.

2 Likes
8

Excellent, glad to hear you found a working solution! We appreciate you taking the time to investigate this and for your initiative in creating a pull request. The community’s engagement is what makes this project thrive.

Please don’t hesitate to reach out if anything else comes up. Happy coding