Error: problem obtaining number of CUDA devices: 205

[crazynds]

The command:

mlcr run-mlperf,inference,_find-performance,_full,_r5.1-dev \
   --model=llama3_1-8b \
   --implementation=reference \
   --framework=vllm \
   --category=datacenter \
   --scenario=Offline \
   --execution_mode=test \
   --device=cuda  \
   --docker --quiet \
   --test_query_count=50 --rerun

I entered in the container with the error:

[2026-03-05 18:38:35,933 repo_action.py : 352 INFO ] - Repository mlperf-automations already exists at /home/mlcuser/MLC/repos/mlcommons@mlperf-automations. Checking for local changes...
[2026-03-05 18:38:35,939 repo_action.py : 363 INFO ] - No local changes detected. Pulling latest changes...
Already up to date.
[2026-03-05 18:38:36,375 repo_action.py : 365 INFO ] - Repository successfully pulled.
[2026-03-05 18:38:36,375 repo_action.py : 379 INFO ] - Registering the repo in repos.json
[2026-03-05 18:38:36,583 script_utils.py:  88 INFO ] - * mlcr app,mlperf,inference,generic,_reference,_llama3_1-8b,_vllm,_cuda,_test,_r5.1-dev_default,_offline
[2026-03-05 18:38:36,588 script_utils.py:  88 INFO ] -   * mlcr detect,os
[2026-03-05 18:38:36,594 module.py      :1018 INFO ] -        ! load /home/mlcuser/MLC/repos/local/cache/detect-os_2aa9b536/mlc-cached-state.json
[2026-03-05 18:38:36,597 script_utils.py:  88 INFO ] -   * mlcr get,sys-utils-mlc
[2026-03-05 18:38:36,598 module.py      :1018 INFO ] -        ! load /home/mlcuser/MLC/repos/local/cache/get-sys-utils-mlc_497501b4/mlc-cached-state.json
[2026-03-05 18:38:36,601 script_utils.py:  88 INFO ] -   * mlcr get,python
[2026-03-05 18:38:36,602 module.py      :1018 INFO ] -        ! load /home/mlcuser/MLC/repos/local/cache/get-python3_22454046/mlc-cached-state.json
[2026-03-05 18:38:36,612 script_utils.py:  88 INFO ] -   * mlcr get,mlcommons,inference,src
[2026-03-05 18:38:36,614 module.py      :1018 INFO ] -        ! load /home/mlcuser/MLC/repos/local/cache/get-mlperf-inference-src_8ebdbf7c/mlc-cached-state.json
[2026-03-05 18:38:36,616 script_utils.py:  88 INFO ] -   * mlcr get,mlperf,inference,utils
[2026-03-05 18:38:36,626 script_utils.py:  88 INFO ] -     * mlcr get,mlperf,inference,src
[2026-03-05 18:38:36,627 module.py      :1018 INFO ] -          ! load /home/mlcuser/MLC/repos/local/cache/get-mlperf-inference-src_8ebdbf7c/mlc-cached-state.json
[2026-03-05 18:38:36,638 module.py      :5081 INFO ] -          ! call "postprocess" from /home/mlcuser/MLC/repos/mlcommons@mlperf-automations/script/get-mlperf-inference-utils/customize.py
[2026-03-05 18:38:36,650 script_utils.py:  88 INFO ] -   * mlcr get,cuda-devices
[2026-03-05 18:38:36,669 script_utils.py:  88 INFO ] -     * mlcr get,cuda,_toolkit
[2026-03-05 18:38:36,687 script_utils.py:  88 INFO ] -       * mlcr detect,os
[2026-03-05 18:38:36,688 module.py      :1018 INFO ] -            ! load /home/mlcuser/MLC/repos/local/cache/detect-os_2aa9b536/mlc-cached-state.json
[2026-03-05 18:38:36,701 module.py      :4064 INFO ] -           # Requested paths: /home/mlcuser/venv/mlcflow/bin:/usr/local/lib/python3.10/dist-packages/torch_tensorrt/bin:/usr/local/mpi/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/ucx/bin:/opt/tensorrt/bin:/home/mlcuser/.local/bin:/usr/local/cuda/bin:/usr/cuda/bin:/usr/local/cuda-11/bin:/usr/cuda-11/bin:/usr/local/cuda-12/bin:/usr/cuda-12/bin:/usr/local/packages/cuda
[2026-03-05 18:38:36,717 module.py      :3786 INFO ] -           * /usr/local/cuda/bin/nvcc
[2026-03-05 18:38:36,717 module.py      :4936 INFO ] -                  ! cd /home/mlcuser/MLC/repos/local/cache/get-cuda_d20b3288
[2026-03-05 18:38:36,717 module.py      :4937 INFO ] -                  ! call /home/mlcuser/MLC/repos/mlcommons@mlperf-automations/script/get-cuda/run.sh from tmp-run.sh
[2026-03-05 18:38:36,740 module.py      :5081 INFO ] -                  ! call "detect_version" from /home/mlcuser/MLC/repos/mlcommons@mlperf-automations/script/get-cuda/customize.py
[2026-03-05 18:38:36,750 customize.py   : 107 INFO ] -           Detected version: 12.6
[2026-03-05 18:38:36,750 module.py      :4130 INFO ] -           # Found artifact in /usr/local/cuda/bin/nvcc
[2026-03-05 18:38:36,751 module.py      :4936 INFO ] -              ! cd /home/mlcuser/MLC/repos/local/cache/get-cuda_d20b3288
[2026-03-05 18:38:36,751 module.py      :4937 INFO ] -              ! call /home/mlcuser/MLC/repos/mlcommons@mlperf-automations/script/get-cuda/run.sh from tmp-run.sh
[2026-03-05 18:38:36,773 module.py      :5081 INFO ] -              ! call "postprocess" from /home/mlcuser/MLC/repos/mlcommons@mlperf-automations/script/get-cuda/customize.py
[2026-03-05 18:38:36,782 customize.py   : 107 INFO ] -           Detected version: 12.6
[2026-03-05 18:38:36,792 module.py      :1844 INFO ] -         - cache UID: d20b328851f4435e
[2026-03-05 18:38:36,792 module.py      :1911 INFO ] - ENV[CUDA_HOME]: /usr/local/cuda
[2026-03-05 18:38:36,792 module.py      :1911 INFO ] - ENV[MLC_CUDA_PATH_LIB_CUDNN_EXISTS]: no
[2026-03-05 18:38:36,792 module.py      :1911 INFO ] - ENV[MLC_CUDA_VERSION]: 12.6
[2026-03-05 18:38:36,792 module.py      :1911 INFO ] - ENV[MLC_CUDA_VERSION_STRING]: cu126
[2026-03-05 18:38:36,792 module.py      :1911 INFO ] - ENV[MLC_NVCC_BIN_WITH_PATH]: /usr/local/cuda/bin/nvcc
[2026-03-05 18:38:36,803 module.py      :4936 INFO ] -          ! cd /home/mlcuser/MLC/repos/local/cache/get-cuda-devices_103b073c
[2026-03-05 18:38:36,803 module.py      :4937 INFO ] -          ! call /home/mlcuser/MLC/repos/mlcommons@mlperf-automations/script/get-cuda-devices/run.sh from tmp-run.sh
rm: cannot remove 'a.out': No such file or directory

NVCC path: /usr/local/cuda/bin/nvcc


Checking compiler version ...

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2024 NVIDIA Corporation
Built on Fri_Jun_14_16:34:21_PDT_2024
Cuda compilation tools, release 12.6, V12.6.20
Build cuda_12.6.r12.6/compiler.34431801_0

Compiling program ...


Running program ...

[2026-03-05 18:38:37,438 module.py      :5011 INFO ] - ========================================================
[2026-03-05 18:38:37,439 module.py      :5013 INFO ] - Print file tmp-run.out:
[2026-03-05 18:38:37,439 module.py      :5014 INFO ] - 
[2026-03-05 18:38:37,439 module.py      :5015 INFO ] - Error: problem obtaining number of CUDA devices: 205

[2026-03-05 18:38:37,439 module.py      :5016 INFO ] - 
Traceback (most recent call last):
  File "/home/mlcuser/venv/mlcflow/bin/mlcr", line 8, in <module>
    sys.exit(mlcr())
  File "/home/mlcuser/venv/mlcflow/lib/python3.10/site-packages/mlc/main.py", line 91, in mlcr
    mlc_expand_short("run")
  File "/home/mlcuser/venv/mlcflow/lib/python3.10/site-packages/mlc/main.py", line 88, in mlc_expand_short
    main()
  File "/home/mlcuser/venv/mlcflow/lib/python3.10/site-packages/mlc/main.py", line 380, in main
    res = method(run_args)
  File "/home/mlcuser/venv/mlcflow/lib/python3.10/site-packages/mlc/script_action.py", line 386, in run
    return self.call_script_module_function("run", run_args)
  File "/home/mlcuser/venv/mlcflow/lib/python3.10/site-packages/mlc/script_action.py", line 282, in call_script_module_function
    raise ScriptExecutionError(f"Script {function_name} execution failed in {module_path}. \nError : {error}")
mlc.script_action.ScriptExecutionError: Script run execution failed in /home/mlcuser/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py. 
Error : Native run script failed inside MLC script (name = get-cuda-devices, return code = 256)


^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Please file an issue at https://github.com/mlcommons/mlperf-automations/issues along with the full MLC command being run and the relevant
or full console log.

And after that I ran:

mlcr run-mlperf,inference,_full,_r5.1-dev,_performance-only \
   --model=llama3_1-8b \
   --implementation=reference \
   --framework=vllm \
   --category=datacenter \
   --scenario=Offline \
   --execution_mode=valid \
   --device=cuda \
   --quiet

And got the same error:

[2026-03-05 18:42:51,125 script_utils.py:  88 INFO ] - * mlcr run-mlperf,inference,_full,_r5.1-dev,_performance-only
[2026-03-05 18:42:51,127 script_utils.py:  88 INFO ] -   * mlcr detect,os
[2026-03-05 18:42:51,134 module.py      :1018 INFO ] -        ! load /home/mlcuser/MLC/repos/local/cache/detect-os_2aa9b536/mlc-cached-state.json
[2026-03-05 18:42:51,136 script_utils.py:  88 INFO ] -   * mlcr detect,cpu
[2026-03-05 18:42:51,137 module.py      :1018 INFO ] -        ! load /home/mlcuser/MLC/repos/local/cache/detect-cpu_c14c2896/mlc-cached-state.json
[2026-03-05 18:42:51,140 script_utils.py:  88 INFO ] -   * mlcr get,python3
[2026-03-05 18:42:51,142 module.py      :1018 INFO ] -        ! load /home/mlcuser/MLC/repos/local/cache/get-python3_22454046/mlc-cached-state.json
[2026-03-05 18:42:51,152 script_utils.py:  88 INFO ] -   * mlcr get,mlcommons,inference,src
[2026-03-05 18:42:51,153 module.py      :1018 INFO ] -        ! load /home/mlcuser/MLC/repos/local/cache/get-mlperf-inference-src_8ebdbf7c/mlc-cached-state.json
[2026-03-05 18:42:51,157 script_utils.py:  88 INFO ] -   * mlcr get,sut,description
[2026-03-05 18:42:51,162 script_utils.py:  88 INFO ] -     * mlcr detect,os
[2026-03-05 18:42:51,163 module.py      :1018 INFO ] -          ! load /home/mlcuser/MLC/repos/local/cache/detect-os_2aa9b536/mlc-cached-state.json
[2026-03-05 18:42:51,165 script_utils.py:  88 INFO ] -     * mlcr detect,cpu
[2026-03-05 18:42:51,166 module.py      :1018 INFO ] -          ! load /home/mlcuser/MLC/repos/local/cache/detect-cpu_c14c2896/mlc-cached-state.json
[2026-03-05 18:42:51,169 script_utils.py:  88 INFO ] -     * mlcr get,python3
[2026-03-05 18:42:51,171 module.py      :1018 INFO ] -          ! load /home/mlcuser/MLC/repos/local/cache/get-python3_22454046/mlc-cached-state.json
[2026-03-05 18:42:51,186 script_utils.py:  88 INFO ] -     * mlcr get,compiler
[2026-03-05 18:42:51,187 module.py      :1018 INFO ] -          ! load /home/mlcuser/MLC/repos/local/cache/get-llvm_8c45579b/mlc-cached-state.json
[2026-03-05 18:42:51,191 script_utils.py:  88 INFO ] -     * mlcr get,cuda-devices
[2026-03-05 18:42:51,204 script_utils.py:  88 INFO ] -       * mlcr get,cuda,_toolkit
[2026-03-05 18:42:51,205 module.py      :1018 INFO ] -            ! load /home/mlcuser/MLC/repos/local/cache/get-cuda_d20b3288/mlc-cached-state.json
[2026-03-05 18:42:51,206 module.py      :1911 INFO ] - ENV[CUDA_HOME]: /usr/local/cuda
[2026-03-05 18:42:51,206 module.py      :1911 INFO ] - ENV[MLC_CUDA_PATH_LIB_CUDNN_EXISTS]: no
[2026-03-05 18:42:51,206 module.py      :1911 INFO ] - ENV[MLC_CUDA_VERSION]: 12.6
[2026-03-05 18:42:51,206 module.py      :1911 INFO ] - ENV[MLC_CUDA_VERSION_STRING]: cu126
[2026-03-05 18:42:51,206 module.py      :1911 INFO ] - ENV[MLC_NVCC_BIN_WITH_PATH]: /usr/local/cuda/bin/nvcc
[2026-03-05 18:42:51,217 module.py      :4936 INFO ] -            ! cd /home/mlcuser/MLC/repos/local/cache/get-cuda-devices_103b073c
[2026-03-05 18:42:51,217 module.py      :4937 INFO ] -            ! call /home/mlcuser/MLC/repos/mlcommons@mlperf-automations/script/get-cuda-devices/run.sh from tmp-run.sh

NVCC path: /usr/local/cuda/bin/nvcc


Checking compiler version ...

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2024 NVIDIA Corporation
Built on Fri_Jun_14_16:34:21_PDT_2024
Cuda compilation tools, release 12.6, V12.6.20
Build cuda_12.6.r12.6/compiler.34431801_0

Compiling program ...


Running program ...

[2026-03-05 18:42:51,857 module.py      :5011 INFO ] - ========================================================
[2026-03-05 18:42:51,857 module.py      :5013 INFO ] - Print file tmp-run.out:
[2026-03-05 18:42:51,857 module.py      :5014 INFO ] - 
[2026-03-05 18:42:51,857 module.py      :5015 INFO ] - Error: problem obtaining number of CUDA devices: 205

[2026-03-05 18:42:51,857 module.py      :5016 INFO ] - 
Traceback (most recent call last):
  File "/home/mlcuser/venv/mlcflow/bin/mlcr", line 8, in <module>
    sys.exit(mlcr())
  File "/home/mlcuser/venv/mlcflow/lib/python3.10/site-packages/mlc/main.py", line 91, in mlcr
    mlc_expand_short("run")
  File "/home/mlcuser/venv/mlcflow/lib/python3.10/site-packages/mlc/main.py", line 88, in mlc_expand_short
    main()
  File "/home/mlcuser/venv/mlcflow/lib/python3.10/site-packages/mlc/main.py", line 380, in main
    res = method(run_args)
  File "/home/mlcuser/venv/mlcflow/lib/python3.10/site-packages/mlc/script_action.py", line 386, in run
    return self.call_script_module_function("run", run_args)
  File "/home/mlcuser/venv/mlcflow/lib/python3.10/site-packages/mlc/script_action.py", line 282, in call_script_module_function
    raise ScriptExecutionError(f"Script {function_name} execution failed in {module_path}. \nError : {error}")
mlc.script_action.ScriptExecutionError: Script run execution failed in /home/mlcuser/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py. 
Error : Native run script failed inside MLC script (name = get-cuda-devices, return code = 256)


^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Please file an issue at https://github.com/mlcommons/mlperf-automations/issues along with the full MLC command being run and the relevant
or full console log.

[amd-arsuresh]

Is nvidia-smi working for you?

[crazynds]

Yes, it output normally outside container. I will try again inside the container.

[crazynds]

The nvidia-smi works normally inside too. But now the error changed:

[2026-03-06 16:12:41,660 repo_action.py : 352 INFO ] - Repository mlperf-automations already exists at /home/mlcuser/MLC/repos/mlcommons@mlperf-automations. Checking for local changes...
[2026-03-06 16:12:41,710 repo_action.py : 363 INFO ] - No local changes detected. Pulling latest changes...
Already up to date.
[2026-03-06 16:12:42,371 repo_action.py : 365 INFO ] - Repository successfully pulled.
[2026-03-06 16:12:42,372 repo_action.py : 379 INFO ] - Registering the repo in repos.json
[2026-03-06 16:12:45,947 script_utils.py:  88 INFO ] - * mlcr app,mlperf,inference,generic,_reference,_llama3_1-8b,_vllm,_cuda,_test,_r5.1-dev_default,_offline
[2026-03-06 16:12:45,955 script_utils.py:  88 INFO ] -   * mlcr detect,os
[2026-03-06 16:12:45,970 module.py      :1018 INFO ] -        ! load /home/mlcuser/MLC/repos/local/cache/detect-os_c47811bb/mlc-cached-state.json
[2026-03-06 16:12:45,975 script_utils.py:  88 INFO ] -   * mlcr get,sys-utils-mlc
[2026-03-06 16:12:45,976 module.py      :1018 INFO ] -        ! load /home/mlcuser/MLC/repos/local/cache/get-sys-utils-mlc_47b9604f/mlc-cached-state.json
[2026-03-06 16:12:45,981 script_utils.py:  88 INFO ] -   * mlcr get,python
[2026-03-06 16:12:45,982 module.py      :1018 INFO ] -        ! load /home/mlcuser/MLC/repos/local/cache/get-python3_57eb933f/mlc-cached-state.json
[2026-03-06 16:12:46,000 script_utils.py:  88 INFO ] -   * mlcr get,mlcommons,inference,src
[2026-03-06 16:12:46,002 module.py      :1018 INFO ] -        ! load /home/mlcuser/MLC/repos/local/cache/get-mlperf-inference-src_72fbe699/mlc-cached-state.json
[2026-03-06 16:12:46,004 script_utils.py:  88 INFO ] -   * mlcr get,mlperf,inference,utils
[2026-03-06 16:12:46,023 script_utils.py:  88 INFO ] -     * mlcr get,mlperf,inference,src
[2026-03-06 16:12:46,025 module.py      :1018 INFO ] -          ! load /home/mlcuser/MLC/repos/local/cache/get-mlperf-inference-src_72fbe699/mlc-cached-state.json
[2026-03-06 16:12:46,039 module.py      :5081 INFO ] -          ! call "postprocess" from /home/mlcuser/MLC/repos/mlcommons@mlperf-automations/script/get-mlperf-inference-utils/customize.py
[2026-03-06 16:12:46,056 script_utils.py:  88 INFO ] -   * mlcr get,cuda-devices
[2026-03-06 16:12:46,101 script_utils.py:  88 INFO ] -     * mlcr get,cuda,_toolkit
[2026-03-06 16:12:46,133 script_utils.py:  88 INFO ] -       * mlcr detect,os
[2026-03-06 16:12:46,134 module.py      :1018 INFO ] -            ! load /home/mlcuser/MLC/repos/local/cache/detect-os_c47811bb/mlc-cached-state.json
[2026-03-06 16:12:46,149 module.py      :4064 INFO ] -           # Requested paths: /home/mlcuser/venv/mlcflow/bin:/usr/local/lib/python3.10/dist-packages/torch_tensorrt/bin:/usr/local/mpi/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/ucx/bin:/opt/tensorrt/bin:/home/mlcuser/.local/bin:/usr/local/cuda/bin:/usr/cuda/bin:/usr/local/cuda-11/bin:/usr/cuda-11/bin:/usr/local/cuda-12/bin:/usr/cuda-12/bin:/usr/local/packages/cuda
[2026-03-06 16:12:46,178 module.py      :3786 INFO ] -           * /usr/local/cuda/bin/nvcc
[2026-03-06 16:12:46,178 module.py      :4936 INFO ] -                  ! cd /home/mlcuser/MLC/repos/local/cache/get-cuda_afbe5a59
[2026-03-06 16:12:46,178 module.py      :4937 INFO ] -                  ! call /home/mlcuser/MLC/repos/mlcommons@mlperf-automations/script/get-cuda/run.sh from tmp-run.sh
[2026-03-06 16:12:46,221 module.py      :5081 INFO ] -                  ! call "detect_version" from /home/mlcuser/MLC/repos/mlcommons@mlperf-automations/script/get-cuda/customize.py
[2026-03-06 16:12:46,233 customize.py   : 107 INFO ] -           Detected version: 12.6
[2026-03-06 16:12:46,233 module.py      :4130 INFO ] -           # Found artifact in /usr/local/cuda/bin/nvcc
[2026-03-06 16:12:46,234 module.py      :4936 INFO ] -              ! cd /home/mlcuser/MLC/repos/local/cache/get-cuda_afbe5a59
[2026-03-06 16:12:46,234 module.py      :4937 INFO ] -              ! call /home/mlcuser/MLC/repos/mlcommons@mlperf-automations/script/get-cuda/run.sh from tmp-run.sh
[2026-03-06 16:12:46,272 module.py      :5081 INFO ] -              ! call "postprocess" from /home/mlcuser/MLC/repos/mlcommons@mlperf-automations/script/get-cuda/customize.py
[2026-03-06 16:12:46,284 customize.py   : 107 INFO ] -           Detected version: 12.6
[2026-03-06 16:12:46,301 module.py      :1844 INFO ] -         - cache UID: afbe5a597c1b47fa
[2026-03-06 16:12:46,301 module.py      :1911 INFO ] - ENV[CUDA_HOME]: /usr/local/cuda
[2026-03-06 16:12:46,301 module.py      :1911 INFO ] - ENV[MLC_CUDA_PATH_LIB_CUDNN_EXISTS]: no
[2026-03-06 16:12:46,301 module.py      :1911 INFO ] - ENV[MLC_CUDA_VERSION]: 12.6
[2026-03-06 16:12:46,301 module.py      :1911 INFO ] - ENV[MLC_CUDA_VERSION_STRING]: cu126
[2026-03-06 16:12:46,301 module.py      :1911 INFO ] - ENV[MLC_NVCC_BIN_WITH_PATH]: /usr/local/cuda/bin/nvcc
[2026-03-06 16:12:46,314 module.py      :4936 INFO ] -          ! cd /home/mlcuser/MLC/repos/local/cache/get-cuda-devices_d76774cc
[2026-03-06 16:12:46,314 module.py      :4937 INFO ] -          ! call /home/mlcuser/MLC/repos/mlcommons@mlperf-automations/script/get-cuda-devices/run.sh from tmp-run.sh
rm: cannot remove 'a.out': No such file or directory

NVCC path: /usr/local/cuda/bin/nvcc


Checking compiler version ...

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2024 NVIDIA Corporation
Built on Fri_Jun_14_16:34:21_PDT_2024
Cuda compilation tools, release 12.6, V12.6.20
Build cuda_12.6.r12.6/compiler.34431801_0

Compiling program ...


Running program ...

[2026-03-06 16:12:48,141 module.py      :5081 INFO ] -          ! call "postprocess" from /home/mlcuser/MLC/repos/mlcommons@mlperf-automations/script/get-cuda-devices/customize.py
[2026-03-06 16:12:48,175 module.py      :1844 INFO ] -         - cache UID: d76774cc4a734a02
[2026-03-06 16:12:48,347 script_utils.py:  88 INFO ] -         * mlcr app,mlperf,reference,inference,_vllm,_llama3_1-8b,_offline,_cuda,_fp32
[2026-03-06 16:12:48,360 script_utils.py:  88 INFO ] -           * mlcr detect,os
[2026-03-06 16:12:48,361 module.py      :1018 INFO ] -                ! load /home/mlcuser/MLC/repos/local/cache/detect-os_c47811bb/mlc-cached-state.json
[2026-03-06 16:12:48,365 script_utils.py:  88 INFO ] -           * mlcr detect,cpu
[2026-03-06 16:12:48,367 module.py      :1018 INFO ] -                ! load /home/mlcuser/MLC/repos/local/cache/detect-cpu_8632bcab/mlc-cached-state.json
[2026-03-06 16:12:48,370 script_utils.py:  88 INFO ] -           * mlcr get,sys-utils-mlc
[2026-03-06 16:12:48,371 module.py      :1018 INFO ] -                ! load /home/mlcuser/MLC/repos/local/cache/get-sys-utils-mlc_47b9604f/mlc-cached-state.json
[2026-03-06 16:12:48,376 script_utils.py:  88 INFO ] -           * mlcr get,python
[2026-03-06 16:12:48,378 module.py      :1018 INFO ] -                ! load /home/mlcuser/MLC/repos/local/cache/get-python3_57eb933f/mlc-cached-state.json
[2026-03-06 16:12:48,449 script_utils.py:  88 INFO ] -           * mlcr get,generic-python-lib,_package.vllm
[2026-03-06 16:12:48,456 script_utils.py:  88 INFO ] -             * mlcr get,python3
[2026-03-06 16:12:48,458 module.py      :1018 INFO ] -                  ! load /home/mlcuser/MLC/repos/local/cache/get-python3_57eb933f/mlc-cached-state.json
[2026-03-06 16:12:48,459 module.py      :4936 INFO ] -                  ! cd /home/mlcuser
[2026-03-06 16:12:48,459 module.py      :4937 INFO ] -                  ! call /home/mlcuser/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/validate_cache.sh from tmp-run.sh
/home/mlcuser/venv/mlcflow/bin/python3 /home/mlcuser/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/detect-version.py
[2026-03-06 16:12:48,562 module.py      :5081 INFO ] -                  ! call "detect_version" from /home/mlcuser/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/customize.py
[2026-03-06 16:12:48,575 customize.py   : 152 INFO ] -                 Detected version: 0.16.0
[2026-03-06 16:12:48,580 script_utils.py:  88 INFO ] -               * mlcr get,python3
[2026-03-06 16:12:48,582 module.py      :1018 INFO ] -                    ! load /home/mlcuser/MLC/repos/local/cache/get-python3_57eb933f/mlc-cached-state.json
[2026-03-06 16:12:48,583 module.py      :1018 INFO ] -                    ! load /home/mlcuser/MLC/repos/local/cache/get-generic-python-lib_81076080/mlc-cached-state.json
[2026-03-06 16:12:48,600 script_utils.py:  88 INFO ] -           * mlcr get,ml-model,llama3,_8b
[2026-03-06 16:12:48,651 script_utils.py:  88 INFO ] -             * mlcr download-and-extract,_r2-downloader,_url.https://llama3-1.mlcommons-storage.org/metadata/llama3-1-8b-instruct.uri
[2026-03-06 16:12:48,696 script_utils.py:  88 INFO ] -               * mlcr download,file,_r2-downloader,_url.https://llama3-1.mlcommons-storage.org/metadata/llama3-1-8b-instruct.uri
[2026-03-06 16:12:48,732 script_utils.py:  88 INFO ] -                 * mlcr detect,os
[2026-03-06 16:12:48,734 module.py      :1018 INFO ] -                      ! load /home/mlcuser/MLC/repos/local/cache/detect-os_c47811bb/mlc-cached-state.json
[2026-03-06 16:12:48,751 customize.py   :  78 INFO ] - 
[2026-03-06 16:12:48,751 customize.py   :  79 INFO ] - Downloading from https://llama3-1.mlcommons-storage.org/metadata/llama3-1-8b-instruct.uri
[2026-03-06 16:12:48,751 module.py      :4936 INFO ] -                      ! cd /home/mlcuser/MLC/repos/local/cache/download-file_llama3-dataset-_f4c2d9a3
[2026-03-06 16:12:48,751 module.py      :4937 INFO ] -                      ! call /home/mlcuser/MLC/repos/mlcommons@mlperf-automations/script/download-file/run.sh from tmp-run.sh

Downloading: bash <(curl -s https://raw.githubusercontent.com/mlcommons/r2-downloader/refs/heads/main/mlc-r2-downloader.sh)  -d '/home/mlcuser/MLC/repos/local/cache/download-file_llama3-dataset-_f4c2d9a3/llama3-1-8b-instruct.uri'  https://llama3-1.mlcommons-storage.org/metadata/llama3-1-8b-instruct.uri
Checking if authentication is required...
Authentication required
Checking dependencies...
cloudflared not found. Attempting to install...
Installing cloudflared...
Downloading cloudflared...
cloudflared installed successfully to /usr/local/bin/cloudflared
Logging in to Cloudflare Access at https://llama3-1.mlcommons-storage.org/metadata/llama3-1-8b-instruct.uri...
2026-03-06T16:12:52Z ERR Could not verify token error="Could not create access request: failed to generate app token file path: mkdir /home/users/lhblago/.cloudflared: no such file or directory"
Could not create access request: failed to generate app token file path: mkdir /home/users/lhblago/.cloudflared: no such file or directory
Unable to find token for provided application. Please run login command to generate token.
mkdir /home/users/lhblago/.cloudflared: no such file or directory
Error: Failed to get access token.
Please re-run the script to re-authenticate.
Download failed. Retrying...
Retry attempt 2 of 2
Downloading: bash <(curl -s https://raw.githubusercontent.com/mlcommons/r2-downloader/refs/heads/main/mlc-r2-downloader.sh)  -d '/home/mlcuser/MLC/repos/local/cache/download-file_llama3-dataset-_f4c2d9a3/llama3-1-8b-instruct.uri'  https://llama3-1.mlcommons-storage.org/metadata/llama3-1-8b-instruct.uri
Checking if authentication is required...
Authentication required
Checking dependencies...
Logging in to Cloudflare Access at https://llama3-1.mlcommons-storage.org/metadata/llama3-1-8b-instruct.uri...
2026-03-06T16:12:53Z ERR Could not verify token error="Could not create access request: failed to generate app token file path: mkdir /home/users/lhblago/.cloudflared: no such file or directory"
Could not create access request: failed to generate app token file path: mkdir /home/users/lhblago/.cloudflared: no such file or directory
Unable to find token for provided application. Please run login command to generate token.
mkdir /home/users/lhblago/.cloudflared: no such file or directory
Error: Failed to get access token.
Please re-run the script to re-authenticate.
Download failed after 2 attempt(s).
Traceback (most recent call last):
  File "/home/mlcuser/venv/mlcflow/bin/mlcr", line 8, in <module>
    sys.exit(mlcr())
  File "/home/mlcuser/venv/mlcflow/lib/python3.10/site-packages/mlc/main.py", line 91, in mlcr
    mlc_expand_short("run")
  File "/home/mlcuser/venv/mlcflow/lib/python3.10/site-packages/mlc/main.py", line 88, in mlc_expand_short
    main()
  File "/home/mlcuser/venv/mlcflow/lib/python3.10/site-packages/mlc/main.py", line 380, in main
    res = method(run_args)
  File "/home/mlcuser/venv/mlcflow/lib/python3.10/site-packages/mlc/script_action.py", line 386, in run
    return self.call_script_module_function("run", run_args)
  File "/home/mlcuser/venv/mlcflow/lib/python3.10/site-packages/mlc/script_action.py", line 282, in call_script_module_function
    raise ScriptExecutionError(f"Script {function_name} execution failed in {module_path}. \nError : {error}")
mlc.script_action.ScriptExecutionError: Script run execution failed in /home/mlcuser/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py. 
Error : Native run script failed inside MLC script (name = download-file, return code = 256)


^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Please file an issue at https://github.com/mlcommons/mlperf-automations/issues along with the full MLC command being run and the relevant
or full console log.

[crazynds]

And running the command inside the container tellme to login in cloudflare:

Downloading: bash <(curl -s https://raw.githubusercontent.com/mlcommons/r2-downloader/refs/heads/main/mlc-r2-downloader.sh)  -d '/home/mlcuser/MLC/repos/local/cache/download-file_llama3-dataset-_f4c2d9a3/llama3-1-8b-instruct.uri'  https://llama3-1.mlcommons-storage.org/metadata/llama3-1-8b-instruct.uri
Checking if authentication is required...
Authentication required
Checking dependencies...
Logging in to Cloudflare Access at https://llama3-1.mlcommons-storage.org/metadata/llama3-1-8b-instruct.uri...
Please open the following URL and log in with your Cloudflare account:

https://llama3-1.mlcommons-storage.org/cdn-cgi/access/cli?aud=892c2283ed5ea77ad3dc38590b0684f177421d8347c8a13835848c1bfd08cb23&edge_token_transfer=true&redirect_url=https%3A%2F%2Fllama3-1.mlcommons-storage.org%2Fmetadata%2Fllama3-1-8b-instruct.uri%3Faud%3D892c2283ed5ea77ad3dc38590b0684f177421d8347c8a13835848c1bfd08cb23%26token%3Dq-qgf5Hb0_goLTkeeHKVQhnKMrCPcgbpcOOuDPdhEwQ%253D&send_org_token=true&token=q-qgf5Hb0_goLTkeeHKVQhnKMrCPcgbpcOOuDPdhEwQ%3D

Leave cloudflared running to download the token automatically.

[anandhu-eng]

Hi @crazynds , could you try again after adding --env.MLC_USE_ML_MODEL_FROM_HOST=yes to the run command from host? This would mount the path to the downloaded model in host to the container