Speed up inference time for the model trained with YOLO12x

Support YOLO
21

You can try installing onnx manually:

pip install onnx onnxruntime-gpu

then run export.

22

@Toxite But when I started the export process, I got the following message:

requirements: Ultralytics requirement [ā€˜onnx>=1.12.0,<1.18.0ā€™] not found, attempting AutoUpdateā€¦

Then, Ultralytics tried to install the version 1.17 of the onnx package.

(Note: When I installed the onnx package manually, the version to be installed was 1.18 by default.)

23

Does it continue after failing?

Can you post the full logs? These small snippets lack context

24

@Toxite The process still continued with the following message:

ONNX: starting export with onnx 1.18.0 opset 19ā€¦
ONNX: slimming with onnxslim 0.1.64ā€¦
ONNX: export success  701.3s, saved as 'best.onnx' (225.9 MB)
TensorRT: starting export with TensorRT 10.13.2.6...
25

Then itā€™s fine

26

@Toxite Regarding to the Live Inference with Streamlit Application using Ultralytics YOLO11 through the CLI command yolo solutions inference model="path/to/model.pt", can I pass two models simultaneously to the command?

27

No, you canā€™t

28

@Toxite I have two concerns which need to be assisted:

  1. If I do not use the TensorRTā€™s INT8 and FP16 quantizations (which means that I still keep the FP32 quantization by default), will the inference time for each frame in the videos be below 200 ms?
  2. If I use the TensorRTā€™s INT8 quantization, what should I do to keep (nearly) the same accuracy metric as the YOLO12x model? (Since when I used the TensorRTā€™s INT8 quantization and kept other parameters by default, the accuracy metric was extremely low.)
29
  1. You should use FP16 at least, if not INT8, if you want speedup. FP16 hardly changes the accuracy.
  2. INT8 requires passing a calibration dataaet. You need to pass your training dataset using data argument. This would have already been shown as a warning in the logs. You should read the logs closely.
30

@Toxite

  1. Regarding to my second concern above, I used my training dataset for calibration (passing my training datasetā€™s yaml file to the data argument), but the accuracy metric was extremely low.
  2. Should I pass my training dataset for calibration when using the TensorRTā€™s FP16 quantization?
31
  1. If thatā€™s the case, then itā€™s probably due to using YOLO12, which is not recommended.
  2. FP16 doesnā€™t require calibration dataset