Ultralytics with custom module not working in other envs

Support
, ,
1

I have forked the ultralytics repo and cloned to my local env. Then I followed below steps to setup.

git clone https://github.com/ultralytics/ultralytics.git
cd ultralytics
pip install -e .

Then I have added my custom module it worked. But when I try running in Kaggle it is not working and gives me below error message.

/usr/local/lib/python3.11/dist-packages/torch/nn/init.py:511: UserWarning: Initializing zero-element tensors is a no-op
  warnings.warn("Initializing zero-element tensors is a no-op")
Traceback (most recent call last):
  File "/kaggle/working/ultralytics/tests/model_validity2.py", line 6, in <module>
    model = YOLO("/kaggle/working/ultralytics/ultralytics/cfg/models/12/swin-t-12.yaml",task='detect')
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/kaggle/working/ultralytics/ultralytics/models/yolo/model.py", line 81, in __init__
    super().__init__(model=model, task=task, verbose=verbose)
  File "/kaggle/working/ultralytics/ultralytics/engine/model.py", line 147, in __init__
    self._new(model, task=task, verbose=verbose)
  File "/kaggle/working/ultralytics/ultralytics/engine/model.py", line 255, in _new
    self.model = (model or self._smart_load("model"))(cfg_dict, verbose=verbose and RANK == -1)  # build model
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/kaggle/working/ultralytics/ultralytics/nn/tasks.py", line 387, in __init__
    self.model, self.save = parse_model(deepcopy(self.yaml), ch=ch, verbose=verbose)  # model, savelist
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/kaggle/working/ultralytics/ultralytics/nn/tasks.py", line 1650, in parse_model
    m_ = torch.nn.Sequential(*(m(*args) for _ in range(n))) if n > 1 else m(*args)  # module
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/kaggle/working/ultralytics/ultralytics/nn/tasks.py", line 1650, in <genexpr>
    m_ = torch.nn.Sequential(*(m(*args) for _ in range(n))) if n > 1 else m(*args)  # module
                               ^^^^^^^^
  File "/kaggle/working/ultralytics/ultralytics/nn/modules/swin_transformers.py", line 88, in __init__
    self.cv3 = Conv(2 * c_, c2, 1, 1)
               ^^^^^^^^^^^^^^^^^^^^^^
  File "/kaggle/working/ultralytics/ultralytics/nn/modules/conv.py", line 65, in __init__
    self.conv = nn.Conv2d(c1, c2, k, s, autopad(k, p, d), groups=g, dilation=d, bias=False)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/torch/nn/modules/conv.py", line 521, in __init__
    super().__init__(
  File "/usr/local/lib/python3.11/dist-packages/torch/nn/modules/conv.py", line 166, in __init__
    torch.empty(
TypeError: empty() received an invalid combination of arguments - got (tuple, dtype=NoneType, device=NoneType), but expected one of:
 * (tuple of ints size, *, tuple of names names, torch.memory_format memory_format = None, torch.dtype dtype = None, torch.layout layout = None, torch.device device = None, bool pin_memory = False, bool requires_grad = False)
 * (tuple of ints size, *, torch.memory_format memory_format = None, Tensor out = None, torch.dtype dtype = None, torch.layout layout = None, torch.device device = None, bool pin_memory = False, bool requires_grad = False)

What I have done wrong in the setting up?

forked repo : GitHub - pawara101/ultralytics: Ultralytics YOLO 🚀

when I run in kaggle I have done below steps

!git clone https://github.com/pawara101/ultralytics.git
%cd ultralytics
!pip install -e .
2

Restart Kaggle runtime after installing

3

hi @Toxite I have restared the kernel but the issue still remains.

m = YOLO("ultralytics/cfg/models/12/swin-t-12.yaml",verbose=True)

error log


                   from  n    params  module                                       arguments                     
  0                  -1  1      1856  ultralytics.nn.modules.conv.Conv             [3, 64, 3, 2]                 
  1                  -1  1     73984  ultralytics.nn.modules.conv.Conv             [64, 128, 3, 2]               
  2                  -1  3    279808  ultralytics.nn.modules.block.C2f             [128, 128, 3, True]           
  3                  -1  1    295424  ultralytics.nn.modules.conv.Conv             [128, 256, 3, 2]              
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
/tmp/ipykernel_125/4158778360.py in <cell line: 0>()
----> 1 m = YOLO("ultralytics/cfg/models/12/swin-t-12.yaml",verbose=True)

/kaggle/working/ultralytics/ultralytics/models/yolo/model.py in __init__(self, model, task, verbose)
     79         else:
     80             # Continue with default YOLO initialization
---> 81             super().__init__(model=model, task=task, verbose=verbose)
     82             if hasattr(self.model, "model") and "RTDETR" in self.model.model[-1]._get_name():  # if RTDETR head
     83                 from ultralytics import RTDETR

/kaggle/working/ultralytics/ultralytics/engine/model.py in __init__(self, model, task, verbose)
    145         __import__("os").environ["CUBLAS_WORKSPACE_CONFIG"] = ":4096:8"  # to avoid deterministic warnings
    146         if str(model).endswith((".yaml", ".yml")):
--> 147             self._new(model, task=task, verbose=verbose)
    148         else:
    149             self._load(model, task=task)

/kaggle/working/ultralytics/ultralytics/engine/model.py in _new(self, cfg, task, model, verbose)
    253         self.cfg = cfg
    254         self.task = task or guess_model_task(cfg_dict)
--> 255         self.model = (model or self._smart_load("model"))(cfg_dict, verbose=verbose and RANK == -1)  # build model
    256         self.overrides["model"] = self.cfg
    257         self.overrides["task"] = self.task

/kaggle/working/ultralytics/ultralytics/nn/tasks.py in __init__(self, cfg, ch, nc, verbose)
    385             LOGGER.info(f"Overriding model.yaml nc={self.yaml['nc']} with nc={nc}")
    386             self.yaml["nc"] = nc  # override YAML value
--> 387         self.model, self.save = parse_model(deepcopy(self.yaml), ch=ch, verbose=verbose)  # model, savelist
    388         self.names = {i: f"{i}" for i in range(self.yaml["nc"])}  # default names dict
    389         self.inplace = self.yaml.get("inplace", True)

/kaggle/working/ultralytics/ultralytics/nn/tasks.py in parse_model(d, ch, verbose)
   1648             c2 = ch[f]
   1649 
-> 1650         m_ = torch.nn.Sequential(*(m(*args) for _ in range(n))) if n > 1 else m(*args)  # module
   1651         t = str(m)[8:-2].replace("__main__.", "")  # module type
   1652         m_.np = sum(x.numel() for x in m_.parameters())  # number params

/kaggle/working/ultralytics/ultralytics/nn/tasks.py in <genexpr>(.0)
   1648             c2 = ch[f]
   1649 
-> 1650         m_ = torch.nn.Sequential(*(m(*args) for _ in range(n))) if n > 1 else m(*args)  # module
   1651         t = str(m)[8:-2].replace("__main__.", "")  # module type
   1652         m_.np = sum(x.numel() for x in m_.parameters())  # number params

/kaggle/working/ultralytics/ultralytics/nn/modules/swin_transformers.py in __init__(self, c1, c2, n, shortcut, g, e)
     86         self.cv1 = Conv(c1, c_, 1, 1)
     87         self.cv2 = Conv(c1, c_, 1, 1)
---> 88         self.cv3 = Conv(2 * c_, c2, 1, 1)
     89         num_heads = c_ // 32
     90         self.m = SwinTransformerBlock(c_, c_, num_heads, n)

/kaggle/working/ultralytics/ultralytics/nn/modules/conv.py in __init__(self, c1, c2, k, s, p, g, d, act)
     63         """
     64         super().__init__()
---> 65         self.conv = nn.Conv2d(c1, c2, k, s, autopad(k, p, d), groups=g, dilation=d, bias=False)
     66         self.bn = nn.BatchNorm2d(c2)
     67         self.act = self.default_act if act is True else act if isinstance(act, nn.Module) else nn.Identity()

/usr/local/lib/python3.11/dist-packages/torch/nn/modules/conv.py in __init__(self, in_channels, out_channels, kernel_size, stride, padding, dilation, groups, bias, padding_mode, device, dtype)
    519         padding_ = padding if isinstance(padding, str) else _pair(padding)
    520         dilation_ = _pair(dilation)
--> 521         super().__init__(
    522             in_channels,
    523             out_channels,

/usr/local/lib/python3.11/dist-packages/torch/nn/modules/conv.py in __init__(self, in_channels, out_channels, kernel_size, stride, padding, dilation, transposed, output_padding, groups, bias, padding_mode, device, dtype)
    164         else:
    165             self.weight = Parameter(
--> 166                 torch.empty(
    167                     (out_channels, in_channels // groups, *kernel_size),
    168                     **factory_kwargs,

TypeError: empty() received an invalid combination of arguments - got (tuple, dtype=NoneType, device=NoneType), but expected one of:
 * (tuple of ints size, *, tuple of names names, torch.memory_format memory_format = None, torch.dtype dtype = None, torch.layout layout = None, torch.device device = None, bool pin_memory = False, bool requires_grad = False)
 * (tuple of ints size, *, torch.memory_format memory_format = None, Tensor out = None, torch.dtype dtype = None, torch.layout layout = None, torch.device device = None, bool pin_memory = False, bool requires_grad = False)
4

This isn’t a Kaggle setup issue, it’s a channel‑shapes issue inside your custom Swin module.

From the traceback, the failure happens when PyTorch tries to build this layer in your swin_transformers.py:

self.cv3 = Conv(2 * c_, c2, 1, 1)

nn.Conv2d is complaining because one of the arguments that should be an int (typically in_channels or out_channels) is actually a non‑int (tuple/float/etc.), so torch.empty((out_channels, in_channels // groups, *kernel_size)) gets an invalid size.

That means in Kaggle at least, either c_ or c2 is not a plain integer when you hit that line.

A few concrete things to try:

  1. In SwinTransformer.__init__ (or whatever class owns that line), add a quick debug print before self.cv3:
print("Swin debug:", c1, c2, type(c1), type(c2))
self.cv3 = Conv(2 * c_, c2, 1, 1)

and run the same YAML in Kaggle. You’ll see exactly what types are coming in.

  1. Also check your parse_model changes in ultralytics/nn/tasks.py for this module. You must ensure c1/c2 are set to integers from ch[f] and args[0], similar to the pattern shown in the model YAML configuration guide:
if m is CustomBlock:  # your custom module
    c1, c2 = ch[f], args[0]
    args = [c1, c2, *args[1:]]

Do something analogous for your Swin block if it needs special handling.

  1. Run the exact same command and YAML locally:
from ultralytics import YOLO
m = YOLO("ultralytics/cfg/models/12/swin-t-12.yaml", verbose=True)

If you add the same debug print, you’ll likely see the same bad type locally as well; you probably just hadn’t exercised this YAML there yet.

Your git clone + pip install -e . flow on Kaggle is fine; the error is coming purely from how the custom module’s channels are being set up, not from the environment.

5

Seems your approach is working fine. Thanks