Cudnn_status_not_supported

Support
, ,
1

Hello,

I have a Jetson ORIN NX from Seeed Studio. I followed the tutorial on the Ultralytics site to get Yolov8 to start. I installed Jetpack 6 with CUDA 12.2 and CuDNN 8.9.4 as in the tutorial.
The problem is that when I run Yolo I get the message “CuDNN Status not Supported”. Pytorch is installed from the version from the introduction: 2.3.0. OpenCV is installed in version 4.10 with CUDA. Below you can see a small report of what is on the Jetson.
I urgently need help. I hope you can help me to solve the problem.

General configuration for OpenCV 4.10.0 =====================================
  Version control:               unknown

  Extra modules:
    Location (extra):            /home/orin-nx/Downloads/workspace/opencv_contrib-4.10.0/modules
    Version control (extra):     unknown

  Platform:
    Timestamp:                   2024-09-13T07:04:37Z
    Host:                        Linux 5.15.136-tegra aarch64
    CMake:                       3.22.1
    CMake generator:             Unix Makefiles
    CMake build tool:            /usr/bin/gmake
    Configuration:               RELEASE

  CPU/HW features:
    Baseline:                    NEON FP16
    Dispatched code generation:  NEON_DOTPROD NEON_FP16 NEON_BF16
      requested:                 NEON_FP16 NEON_BF16 NEON_DOTPROD
      NEON_DOTPROD (1 files):    + NEON_DOTPROD
      NEON_FP16 (2 files):       + NEON_FP16
      NEON_BF16 (0 files):       + NEON_BF16

  C/C++:
    Built as dynamic libs?:      YES
    C++ standard:                11
    C++ Compiler:                /usr/bin/c++  (ver 11.4.0)
    C++ flags (Release):         -fsigned-char -W -Wall -Wreturn-type -Wnon-virtual-dtor -Waddress -Wsequence-point -Wformat -Wformat-security -Wmissing-declarations -Wundef -Winit-self -Wpointer-arith -Wshadow -Wsign-promo -Wuninitialized -Wsuggest-override -Wno-delete-non-virtual-dtor -Wno-comment -Wimplicit-fallthrough=3 -Wno-strict-overflow -fdiagnostics-show-option -pthread -fomit-frame-pointer -ffunction-sections -fdata-sections    -fvisibility=hidden -fvisibility-inlines-hidden -O3 -DNDEBUG  -DNDEBUG
    C++ flags (Debug):           -fsigned-char -W -Wall -Wreturn-type -Wnon-virtual-dtor -Waddress -Wsequence-point -Wformat -Wformat-security -Wmissing-declarations -Wundef -Winit-self -Wpointer-arith -Wshadow -Wsign-promo -Wuninitialized -Wsuggest-override -Wno-delete-non-virtual-dtor -Wno-comment -Wimplicit-fallthrough=3 -Wno-strict-overflow -fdiagnostics-show-option -pthread -fomit-frame-pointer -ffunction-sections -fdata-sections    -fvisibility=hidden -fvisibility-inlines-hidden -g  -O0 -DDEBUG -D_DEBUG
    C Compiler:                  /usr/bin/cc
    C flags (Release):           -fsigned-char -W -Wall -Wreturn-type -Waddress -Wsequence-point -Wformat -Wformat-security -Wmissing-declarations -Wmissing-prototypes -Wstrict-prototypes -Wundef -Winit-self -Wpointer-arith -Wshadow -Wuninitialized -Wno-comment -Wimplicit-fallthrough=3 -Wno-strict-overflow -fdiagnostics-show-option -pthread -fomit-frame-pointer -ffunction-sections -fdata-sections    -fvisibility=hidden -O3 -DNDEBUG  -DNDEBUG
    C flags (Debug):             -fsigned-char -W -Wall -Wreturn-type -Waddress -Wsequence-point -Wformat -Wformat-security -Wmissing-declarations -Wmissing-prototypes -Wstrict-prototypes -Wundef -Winit-self -Wpointer-arith -Wshadow -Wuninitialized -Wno-comment -Wimplicit-fallthrough=3 -Wno-strict-overflow -fdiagnostics-show-option -pthread -fomit-frame-pointer -ffunction-sections -fdata-sections    -fvisibility=hidden -g  -O0 -DDEBUG -D_DEBUG
    Linker flags (Release):      -Wl,--gc-sections -Wl,--as-needed -Wl,--no-undefined  
    Linker flags (Debug):        -Wl,--gc-sections -Wl,--as-needed -Wl,--no-undefined  
    ccache:                      NO
    Precompiled headers:         NO
    Extra dependencies:          m pthread cudart_static dl rt nppc nppial nppicc nppidei nppif nppig nppim nppist nppisu nppitc npps cublas cudnn cufft -L/usr/local/cuda/lib64 -L/usr/lib/aarch64-linux-gnu
    3rdparty dependencies:

  OpenCV modules:
    To be built:                 aruco bgsegm bioinspired calib3d ccalib core cudaarithm cudabgsegm cudacodec cudafeatures2d cudafilters cudaimgproc cudalegacy cudaobjdetect cudaoptflow cudastereo cudawarping cudev datasets dnn dnn_objdetect dnn_superres dpm face features2d flann freetype fuzzy gapi hfs highgui img_hash imgcodecs imgproc intensity_transform line_descriptor mcc ml objdetect optflow phase_unwrapping photo plot python3 quality rapid reg rgbd saliency shape signal stereo stitching structured_light superres surface_matching text tracking video videoio videostab wechat_qrcode xfeatures2d ximgproc xobjdetect xphoto
    Disabled:                    world
    Disabled by dependency:      -
    Unavailable:                 alphamat cannops cvv hdf java julia matlab ovis python2 sfm ts viz
    Applications:                apps
    Documentation:               NO
    Non-free algorithms:         NO

  GUI:                           GTK2
    GTK+:                        YES (ver 2.24.33)
      GThread :                  YES (ver 2.72.4)
      GtkGlExt:                  NO
    VTK support:                 NO

  Media I/O: 
    ZLib:                        /usr/lib/aarch64-linux-gnu/libz.so (ver 1.2.11)
    JPEG:                        /usr/lib/aarch64-linux-gnu/libjpeg.so (ver 80)
    WEBP:                        build (ver encoder: 0x020f)
    PNG:                         /usr/lib/aarch64-linux-gnu/libpng.so (ver 1.6.37)
    TIFF:                        /usr/lib/aarch64-linux-gnu/libtiff.so (ver 42 / 4.3.0)
    JPEG 2000:                   build (ver 2.5.0)
    OpenEXR:                     build (ver 2.3.0)
    HDR:                         YES
    SUNRASTER:                   YES
    PXM:                         YES
    PFM:                         YES

  Video I/O:
    DC1394:                      NO
    FFMPEG:                      YES
      avcodec:                   YES (58.134.100)
      avformat:                  YES (58.76.100)
      avutil:                    YES (56.70.100)
      swscale:                   YES (5.9.100)
      avresample:                NO
    GStreamer:                   YES (1.20.3)
    v4l/v4l2:                    YES (linux/videodev2.h)

  Parallel framework:            pthreads

  Trace:                         YES (with Intel ITT)

  Other third-party libraries:
    Lapack:                      NO
    Eigen:                       NO
    Custom HAL:                  YES (carotene (ver 0.0.1, Auto detected))
    Protobuf:                    build (3.19.1)
    Flatbuffers:                 builtin/3rdparty (23.5.9)

  NVIDIA CUDA:                   YES (ver 12.2, CUFFT CUBLAS)
    NVIDIA GPU arch:             87
    NVIDIA PTX archs:

  cuDNN:                         YES (ver 8.9.4)

  OpenCL:                        YES (no extra features)
    Include path:                /home/orin-nx/Downloads/workspace/opencv-4.10.0/3rdparty/include/opencl/1.2
    Link libraries:              Dynamic load

  Python 3:
    Interpreter:                 /usr/bin/python3 (ver 3.10.12)
    Libraries:                   /usr/lib/aarch64-linux-gnu/libpython3.10.so (ver 3.10.12)
    Limited API:                 NO
    numpy:                       /home/orin-nx/.local/lib/python3.10/site-packages/numpy/core/include (ver 1.23.5)
    install path:                lib/python3.10/dist-packages/cv2/python-3.10

  Python (for build):            /usr/bin/python3

  Java:                          
    ant:                         NO
    Java:                        NO
    JNI:                         NO
    Java wrappers:               NO
    Java tests:                  NO

  Install to:                    /usr/local
-----------------------------------------------------------------


Torch Version CUDA: 12.2
Torch CuDNN Version: 8904
Using Device: cuda
Loading Model: models/yolov8n.pt
/home/orin-nx/.local/lib/python3.10/site-packages/torch/nn/modules/conv.py:456: UserWarning: Plan failed with a cudnnException: CUDNN_BACKEND_EXECUTION_PLAN_DESCRIPTOR: cudnnFinalize Descriptor Failed cudnn_status: CUDNN_STATUS_NOT_SUPPORTED (Triggered internally at /opt/pytorch/aten/src/ATen/native/cudnn/Conv_v8.cpp:919.)
  return F.conv2d(input, weight, bias, self.stride,
2

@Bongy88 I’m not an expert with Jetson devices and don’t have one myself, I’ll ping @lakshanthad later to see if he might be able to help out too. First thing I would suggest is trying to use one of the the Ultralytics Docker images for Jetson, there are different ones for the various Jetpack versions. Testing with the Docker image is a good first step, as these have been verified to work.

The details you shared should be a great help in figuring out the source of the issue, so thanks for providing those! It would also help to share the code you ran that generated the error.

One final note on installing CUDA and CuDNN. I know that for x86 devices, these installs don’t need to happen to use PyTorch, as when torch installs, it includes precompiled binaries. Like I mentioned earlier, I’m not familiar with working on Jetson devices, but I figured I would mention this as it could be useful to try setup without installing these (if you don’t know if they’re required or not).

3

@Bongy88 Thanks for sharing a detailed output. As @BurhanQ mentioned, it is better to start by running a Docker container and based on your configuration, you can use our Jetpack 6 Docker container.

In the meantime, could you share with me how you installed CUDA and CuDNN on the Jetson? Did you install them using sudo apt install nvidia-jetpack? Because this method of installation contains CUDA and CuDNN already and it is the recommended way from NVIDIA to make sure all package versions work with each other. It would be better if you explain in detail how you setup a working environment starting all the way from unboxing the device, to flashing the device and so on. This would help me to understand whether you followed the correct steps to prepare the environment.

1 Like
4

@lakshanthad I used the instructions provided by Seeed Studio for flashing. Here Jetpack 6 with Cuda version 12.2 and cuDNN 8.9.4 is supplied directly. I then installed Ultralytics. The matching Torch version to the Cuda version and then onnxruntime-gpu with the corresponding numpy version. As described in the instructions. Then I installed OpenCV 4.9 with Cuda support and deleted the OpenCV version 4.10 that is installed by Ultralytics. The Python version used here is 3.10.

Translated with DeepL.com (free version)

5

Hello @Bongy88 . Thank you for the details.

Could you please tell me the exact commands that you executed to install everything? In this way, I can have a better understanding of your environment.

Also, please share the example code you ran that generated the error.

In the meantime, could you not install OpenCV with CUDA support, but keep the OpenCV version that comes with JetPack 6 and try again?

Thank you.

1 Like