How to Combine Weights to Detect from Multiple Datasets?

dhruvildave22 · April 1, 2022, 10:28am

I am trying this.

train: # train images
  - images/coco_dataset
  - images/my_custom_dataset

but i have one question here, I am using this coco dataset from roboflow https://public.roboflow.com/object-detection/microsoft-coco-subset. it already gives all the train and valid datasets in images-labels format.
This dataset has at-least 118k(5000 images for valid and rest for train) images and my custom dataset has only 3000(1900 train, 800 valid and 300 test ) images with labels.
My only question here:
will this going to affect the prediction rate for my custom dataset since the number of images in my dataset is much lower than the coco dataset which i will be using?

Here is my data.yaml


train: # train images
  - /content/yolov5/datasets/animal_dataset/train/images
  - /content/yolov5/datasets/coco/train/images
val: # validation images
  - /content/yolov5/datasets/animal_dataset/valid/images
  - /content/yolov5/datasets/coco/valid/images

nc: 83

names: ['aeroplane', 'apple', 'backpack', 'banana', 'baseball bat', 'baseball glove', 'bear', 'bed', 
        'bench', 'bicycle', 'bird', 'boat', 'book', 'bottle', 'bowl', 'broccoli', 'bus', 'cake', 'car', 'carrot', 
        'cat', 'cell phone', 'chair', 'clock', 'cow', 'cup', 'diningtable', 'dog', 'donut', 'elephant', 'fire hydrant', 
        'fork', 'frisbee', 'giraffe', 'hair drier', 'handbag', 'horse', 'hot dog', 'keyboard', 'kite', 'knife', 
        'laptop', 'microwave', 'motorbike', 'mouse', 'orange', 'oven', 'parking meter', 'person', 'pizza', 
        'pottedplant', 'refrigerator', 'remote', 'sandwich', 'scissors', 'sheep', 'sink', 'skateboard', 'skis', 
        'snowboard', 'sofa', 'spoon', 'sports ball', 'stop sign', 'suitcase', 'surfboard', 'teddy bear', 
        'tennis racket', 'tie', 'toaster', 'toilet', 'toothbrush', 'traffic light', 'train', 'truck', 'tvmonitor', 
        'umbrella', 'vase', 'wine glass', 'zebra', 'lion', 'frog', 'tiger']

Thanks
Dhruvil Dave

Glenn · April 1, 2022, 7:52pm

@dhruvildave22 yes, just as you say one dominant dataset will receive more attention than a smaller dataset if you train both at the same time.

We have a parameter called --image-weights in train.py which will sample images based on their performance, so poorly performing images should be then overly represented and the problem you describe may be mitigated. This is highly experimental, but you can try it out and see if it helps your situation.

Alternatively you can create a subset of COCO, i.e. using only 10% of the COCO dataset and train on this dataset along with your dataset. This would manually help rebalance the two datasets to be closer in magnitude to each other.

github.com

ultralytics/yolov5/blob/71621df87589faea19ba4c4098bb68e73201f30c/train.py#L487

      
        
            parser.add_argument('--batch-size', type=int, default=16, help='total batch size for all GPUs, -1 for autobatch')
            parser.add_argument('--imgsz', '--img', '--img-size', type=int, default=640, help='train, val image size (pixels)')
            parser.add_argument('--rect', action='store_true', help='rectangular training')
            parser.add_argument('--resume', nargs='?', const=True, default=False, help='resume most recent training')
            parser.add_argument('--nosave', action='store_true', help='only save final checkpoint')
            parser.add_argument('--noval', action='store_true', help='only validate final epoch')
            parser.add_argument('--noautoanchor', action='store_true', help='disable AutoAnchor')
            parser.add_argument('--evolve', type=int, nargs='?', const=300, help='evolve hyperparameters for x generations')
            parser.add_argument('--bucket', type=str, default='', help='gsutil bucket')
            parser.add_argument('--cache', type=str, nargs='?', const='ram', help='--cache images in "ram" (default) or "disk"')
            parser.add_argument('--image-weights', action='store_true', help='use weighted image selection for training')
            parser.add_argument('--device', default='', help='cuda device, i.e. 0 or 0,1,2,3 or cpu')
            parser.add_argument('--multi-scale', action='store_true', help='vary img-size +/- 50%%')
            parser.add_argument('--single-cls', action='store_true', help='train multi-class data as single-class')
            parser.add_argument('--optimizer', type=str, choices=['SGD', 'Adam', 'AdamW'], default='SGD', help='optimizer')
            parser.add_argument('--sync-bn', action='store_true', help='use SyncBatchNorm, only available in DDP mode')
            parser.add_argument('--workers', type=int, default=8, help='max dataloader workers (per RANK in DDP mode)')
            parser.add_argument('--project', default=ROOT / 'runs/train', help='save to project/name')
            parser.add_argument('--name', default='exp', help='save to project/name')
            parser.add_argument('--exist-ok', action='store_true', help='existing project/name ok, do not increment')
            parser.add_argument('--quad', action='store_true', help='quad dataloader')

YYeDoudou · April 11, 2022, 3:52pm

I have a question, how exactly do you change the ‘class’ index without manually editing them? I created and downloaded my custom dataset from Roboflow.

Glenn · April 12, 2022, 9:28am

You could write a script to read your labels.txt files, do a string replace on class indices, then write back to the original file the updated text.

Guttappa_Sajjan · April 13, 2022, 6:23am

Use this script

import os, fnmatch

def findReplace(directory, find, replace, filePattern):
    for path, dirs, files in os.walk(os.path.abspath(directory)):
        for filename in fnmatch.filter(files, filePattern):
            filepath = os.path.join(path, filename)
            with open(filepath) as f:
                s = f.read()
            s = s.replace(find, replace)
            with open(filepath, "w") as f:
                f.write(s)


findReplace('<Path to labels>','<Find this>','<Replace This>',"*.txt")

Powercube7 · April 23, 2022, 1:52pm

Something that I feel that needs to be added about creating multiple models:

You can use the model ensembling found in the YOLOv5 wiki to run inference on different sources, leading to results equivalent as using one model with all of the classes

Glenn · April 23, 2022, 6:29pm

@Powercube7 yes that’s true! As long as your classes don’t conflict you can ensemble different trained models together to achieve a similar effect.

vanloctc · June 1, 2022, 9:14am

I also have a similar question with @ [khandriod]
And my answer is the Transfer learning method.
About my problem, I have a dataset (A) and a pre-trained model (preA) after training with the dataset (A). and then I want to detect a new object (B) that doesn’t train before but that object (B) is similar (eg: cats and dogs) to the dataset (A). My method is similar with @ [Glenn]
With pre-trained model data.YAML => yoloOriginal.pt

path: ../datasets/coco128 
train: 
  - images/train2017 
val: 
  - images/train2017
nc: 80
name : [ 80 classes ]

With new pre-trained model data_new.YAML => yoloOriginal_and_newClasses.pt

path: ../datasets/coco128   
train: 
  - images/train2017  
  - images/train # new class
val:  
  - images/train2017  
  - images/train # new class
nc: 81
name : [ 81 classes ]

After using Transfer learning, My model can inference both A and B object.
I’m also a beginner at Yolo/Deep learning. So How about if I want to continue using Transfer Learning for C and D and E objects, Is there any problem such as accuracy with my model of Transfer Learning method?
Could you help me some ideal, @ Glenn and @ KalenMichael ?

Glenn · June 1, 2022, 3:00pm

@vanloctc sure, you can transfer learn iteratively as many times as you want. There’s no limit, though of course over time the model will lose the original classes data in favor of the new classes and data.

Symbadian · June 13, 2022, 11:26am

hi @Glenn, I am Matt and I am working on using yolov5 for activity as per a paper that I am following. I have been trying your suggestions concerning your solution above and have been experiencing an error that is challenging my understanding and my progress moving forward.

The issues:
I am targeting the location concerning the path to the data and I placed my custom data in the same path as per the original dataset.

But I am constantly getting a persistent error that is challenging to decipher, can assist me to understand this, please?

Thank you in advance for acknowledging my digital presence!

Traceback (most recent call last):
  File "/Users/symbadian/Desktop/LIRIS_new_dataset/yolov5/train.py", line 667, in <module>
    main(opt)
  File "/Users/symbadian/Desktop/LIRIS_new_dataset/yolov5/train.py", line 562, in main
    train(opt.hyp, opt, device, callbacks)
  File "/Users/symbadian/Desktop/LIRIS_new_dataset/yolov5/train.py", line 92, in train
    loggers = Loggers(save_dir, weights, opt, hyp, LOGGER)  # loggers instance
  File "/Users/symbadian/Desktop/LIRIS_new_dataset/yolov5/utils/loggers/__init__.py", line 83, in __init__
    self.wandb = WandbLogger(self.opt, run_id)
  File "/Users/symbadian/Desktop/LIRIS_new_dataset/yolov5/utils/loggers/wandb/wandb_utils.py", line 180, in __init__
    self.data_dict = check_wandb_dataset(opt.data)
  File "/Users/symbadian/Desktop/LIRIS_new_dataset/yolov5/utils/loggers/wandb/wandb_utils.py", line 48, in check_wandb_dataset
    data_dict = yaml.safe_load(f)
  File "/Users/symbadian/miniforge3/lib/python3.9/site-packages/yaml/__init__.py", line 125, in safe_load
    return load(stream, SafeLoader)
  File "/Users/symbadian/miniforge3/lib/python3.9/site-packages/yaml/__init__.py", line 81, in load
    return loader.get_single_data()
  File "/Users/symbadian/miniforge3/lib/python3.9/site-packages/yaml/constructor.py", line 49, in get_single_data
    node = self.get_single_node()
  File "/Users/symbadian/miniforge3/lib/python3.9/site-packages/yaml/composer.py", line 36, in get_single_node
    document = self.compose_document()
  File "/Users/symbadian/miniforge3/lib/python3.9/site-packages/yaml/composer.py", line 55, in compose_document
    node = self.compose_node(None, None)
  File "/Users/symbadian/miniforge3/lib/python3.9/site-packages/yaml/composer.py", line 84, in compose_node
    node = self.compose_mapping_node(anchor)
  File "/Users/symbadian/miniforge3/lib/python3.9/site-packages/yaml/composer.py", line 127, in compose_mapping_node
    while not self.check_event(MappingEndEvent):
  File "/Users/symbadian/miniforge3/lib/python3.9/site-packages/yaml/parser.py", line 98, in check_event
    self.current_event = self.state()
  File "/Users/symbadian/miniforge3/lib/python3.9/site-packages/yaml/parser.py", line 438, in parse_block_mapping_key
    raise ParserError("while parsing a block mapping", self.marks[-1],
yaml.parser.ParserError: while parsing a block mapping
  in "/Users/symbadian/Desktop/LIRIS_new_dataset/yolov5/data/coco128.yaml", line 11, column 1
expected <block end>, but found '<block sequence start>'
  in "/Users/symbadian/Desktop/LIRIS_new_dataset/yolov5/data/coco128.yaml", line 13, column 3
wandb: Waiting for W&B process to finish... (failed 1). Press Control-C to abort syncing.

I also changed the number of classes and added the class type as

Symbadian · June 13, 2022, 11:36am

Hi @Glenn I am matt working on using yolov5 to detect activity as per a paper I am following.

I have my custom dataset and I find it challenging to understand the errors that have been dispensed in the development stages.

The issues:
I implemented your processes above but the same procedure is throwing an error that I cannot seem to bypass as a newbie to yolo and its operations. Can you guide me here, please?

classes:

errors:

Traceback (most recent call last):
  File "/Users/symbadian/Desktop/LIRIS_new_dataset/yolov5/train.py", line 667, in <module>
    main(opt)
  File "/Users/symbadian/Desktop/LIRIS_new_dataset/yolov5/train.py", line 562, in main
    train(opt.hyp, opt, device, callbacks)
  File "/Users/symbadian/Desktop/LIRIS_new_dataset/yolov5/train.py", line 92, in train
    loggers = Loggers(save_dir, weights, opt, hyp, LOGGER)  # loggers instance
  File "/Users/symbadian/Desktop/LIRIS_new_dataset/yolov5/utils/loggers/__init__.py", line 83, in __init__
    self.wandb = WandbLogger(self.opt, run_id)
  File "/Users/symbadian/Desktop/LIRIS_new_dataset/yolov5/utils/loggers/wandb/wandb_utils.py", line 180, in __init__
    self.data_dict = check_wandb_dataset(opt.data)
  File "/Users/symbadian/Desktop/LIRIS_new_dataset/yolov5/utils/loggers/wandb/wandb_utils.py", line 48, in check_wandb_dataset
    data_dict = yaml.safe_load(f)
  File "/Users/symbadian/miniforge3/lib/python3.9/site-packages/yaml/__init__.py", line 125, in safe_load
    return load(stream, SafeLoader)
  File "/Users/symbadian/miniforge3/lib/python3.9/site-packages/yaml/__init__.py", line 81, in load
    return loader.get_single_data()
  File "/Users/symbadian/miniforge3/lib/python3.9/site-packages/yaml/constructor.py", line 49, in get_single_data
    node = self.get_single_node()
  File "/Users/symbadian/miniforge3/lib/python3.9/site-packages/yaml/composer.py", line 36, in get_single_node
    document = self.compose_document()
  File "/Users/symbadian/miniforge3/lib/python3.9/site-packages/yaml/composer.py", line 55, in compose_document
    node = self.compose_node(None, None)
  File "/Users/symbadian/miniforge3/lib/python3.9/site-packages/yaml/composer.py", line 84, in compose_node
    node = self.compose_mapping_node(anchor)
  File "/Users/symbadian/miniforge3/lib/python3.9/site-packages/yaml/composer.py", line 127, in compose_mapping_node
    while not self.check_event(MappingEndEvent):
  File "/Users/symbadian/miniforge3/lib/python3.9/site-packages/yaml/parser.py", line 98, in check_event
    self.current_event = self.state()
  File "/Users/symbadian/miniforge3/lib/python3.9/site-packages/yaml/parser.py", line 438, in parse_block_mapping_key
    raise ParserError("while parsing a block mapping", self.marks[-1],
yaml.parser.ParserError: while parsing a block mapping
  in "/Users/symbadian/Desktop/LIRIS_new_dataset/yolov5/data/coco128.yaml", line 11, column 1
expected <block end>, but found '<block sequence start>'
  in "/Users/symbadian/Desktop/LIRIS_new_dataset/yolov5/data/coco128.yaml", line 13, column 3
wandb: Waiting for W&B process to finish... (failed 1). Press Control-C to abort syncing.

AM I missing an edit somewhere? is there something else that I should edit to coordinate with the custom configurations? let me know please… and thanx in advance!

Symbadian · June 13, 2022, 11:39am

@Glenn this is the class as per the above and this is in the format of yolov5 as it was processed in roboflow. Please bear with my limited understanding, I am new to yolo and it’s operations.
AND thank you in advance for acknowledging my digital presence…

Matt…

Glenn · June 13, 2022, 2:13pm

@Symbadian your YAML may be incorrectly formatted. For a multi-directory training yaml example see GlobalWheat2020.yaml:

github.com

ultralytics/yolov5/blob/6a67594cbfe6599aab7006e33948f073e14246a8/data/GlobalWheat2020.yaml#L1-L27

      
        
            # YOLOv5 🚀 by Ultralytics, GPL-3.0 license
            # Global Wheat 2020 dataset http://www.global-wheat.com/ by University of Saskatchewan
            # Example usage: python train.py --data GlobalWheat2020.yaml
            # parent
            # ├── yolov5
            # └── datasets
            #     └── GlobalWheat2020  ← downloads here (7.0 GB)
            
            

            
# Train/val/test sets as 1) dir: path/to/imgs, 2) file: path/to/imgs.txt, or 3) list: [path/to/imgs1, path/to/imgs2, ..]
            path: ../datasets/GlobalWheat2020  # dataset root dir
            train: # train images (relative to 'path') 3422 images
              - images/arvalis_1
              - images/arvalis_2
              - images/arvalis_3
              - images/ethz_1
              - images/rres_1
              - images/inrae_1
              - images/usask_1
            val: # val images (relative to 'path') 748 images (WARNING: train set contains ethz_1)

This file has been truncated. show original

Symbadian · June 13, 2022, 3:29pm

That’s the thing @Glenn, the entire process is running but I cannot see the results of the BeatingpyT class nor its output, and some other classes are missing as per the confusion matrix. Check out my output details and you would see what I mean. I thought that the files or the process were not working but when I check the confusion matrix the class is being integrated into the training process. Am I reading this incorrectly here? is this a data issue as in having a small volume of data? But at least if this was the issue a poor result would be demonstrated on the confusion matrix? I am no pro at this just trying to use logic. Some of the other classes are missing in the confusion matrix.

Symbadian · June 13, 2022, 3:34pm

I think I corrected the YAM file after carefully following your lead,

# Train/val/test sets as 1) dir: path/to/imgs, 2) file: path/to/imgs.txt, or 3) list: [path/to/imgs1, path/to/imgs2, ..]
path: ../datasets/coco128  # dataset root dir
train:   # train images (relative to 'path') 128 images
  - images/train2017
  - images/BeatingpyTrain


val:   # val images (relative to 'path') 128 images
  - images/train2017
  - images/BeatingpyTval
test:  # test images (optional)
  - images/BeatingpyTest
# Classes
nc: 81  # number of classes
names: ['person', 'bicycle', 'car', 'motorcycle', 'airplane', 'bus', 'train', 'truck', 'boat', 'traffic light',
        'fire hydrant', 'stop sign', 'parking meter', 'bench', 'bird', 'cat', 'dog', 'horse', 'sheep', 'cow',
        'elephant', 'bear', 'zebra', 'giraffe', 'backpack', 'umbrella', 'handbag', 'tie', 'suitcase', 'frisbee',
        'skis', 'snowboard', 'sports ball', 'kite', 'baseball bat', 'baseball glove', 'skateboard', 'surfboard',
        'tennis racket', 'bottle', 'wine glass', 'cup', 'fork', 'knife', 'spoon', 'bowl', 'banana', 'apple',
        'sandwich', 'orange', 'broccoli', 'carrot', 'hot dog', 'pizza', 'donut', 'cake', 'chair', 'couch',
        'potted plant', 'bed', 'dining table', 'toilet', 'tv', 'laptop', 'mouse', 'remote', 'keyboard', 'cell phone',
        'microwave', 'oven', 'toaster', 'sink', 'refrigerator', 'book', 'clock', 'vase', 'scissors', 'teddy bear',
        'hair drier', 'toothbrush','BeatingpyT']  # class names

I added the amount of data that is in the BeatingpyT class, that is just a few files therein. Functionality is what I am seeking for now and then I would add the correct dataset volume. I’m trying to get it to work to reduce the time it takes to process all the data as this would be shorter when using a small set for training operations.

Please guide me here, I am sure a step is missing, and thanks in advance for acknowledging me, really appreciate this…

Symbadian · June 13, 2022, 4:35pm

Hey @Glenn, So I tried adding more data to see if that was the issue but it was not. Clearly, something is off but I lack some experience to see where I must amend to move forward. Would really appreciate your guidance as my logic ran out, not sure where to go from here…

I sent over previously details of the storage formats and how the data is presented in the datastore… really would like to see where I am going wrong here at this stage…

thanx loads once more…

Glenn · June 13, 2022, 10:34pm

@Symbadian only classes that are present are displayed by val.py. If your class is not displayed as a row there are no labelled examples of it in your val set.

Symbadian · June 14, 2022, 6:12am

@Glenn top of the morning to you pal, I am confused…
Should I run a validation script of something? let me know please. As per the previous conversation. I decided to run more data as an imbalanced dataset may cause issues based on my understanding. With that said, the validation set has a significant amount of data that can be evaluated and processed by the model. Is it possible for you to assist me in understanding a bit further, please? What I’ve done incorrectly at this stage is puzzling me.

my dataset as per the images

Glenn · June 15, 2022, 9:46am

@Symbadian your dataset looks fine. Missing classes in a validation set will not cause any errors but also no metrics will be output for classes that have no labels.

If you want a histogram of your val labels you could switch your val and train sets in your data yaml, start training and then take a look at your labels.png file

Symbadian · June 15, 2022, 10:12am

Hey @Glenn, top of the morning pal, can you help me to understand what my confusion is with this aspect, please?

Isn’t the validation coming from the specified location?
for example when I investigate the

train: images/train2017

the data inside of that folder has no separation between what is set for training and validation.
it is one folder of data…

So I followed the same perspective with my data for ( - images/Knife_DeployTr ) here

train: images/train2017
  - images/Knife_DeployTr

What I am saying is, that I am not seeing a validation separation of the data for the original data in the images/train2017 folder.

Is this specified somewhere? and if yes can you guide me, Please??!!??
I have been searching all of the .py and configuration files for such a statement but thus far, my attempts have been futile.

I am sure that I am doing something wrong and my limited knowledge prevents me from detecting what that solution is. Please assist, really willing to learn about these operations.