-
Notifications
You must be signed in to change notification settings - Fork 559
Labels
plannedPlanned for the short termPlanned for the short termsnackx-squadIssues that are being handled by the x-squadIssues that are being handled by the x-squad
Description
Contact Details [Optional]
No response
System Information
This pipeline failed with cryptic errors while every other stack with the same mlflow component works
What happened?
MLflow fails on Azure
Reproduction steps
...
Relevant log output
INFO
2025-11-05 20:25:01
mlflow.exceptions.RestException: RESOURCE_DOES_NOT_EXIST: Run with id=2bf891a5-5cb5-4e0e-8615-5b6ad4bd3504 not found
Copy to Clipboard
ERROR
2025-11-05 20:25:01
An exception has occurred.
Traceback (most recent call last):
Traceback (most recent call last):
File "/opt/venv/lib/python3.11/site-packages/zenml/orchestrators/step_launcher.py", line 352, in launch
self._run_step(
File "/opt/venv/lib/python3.11/site-packages/zenml/orchestrators/step_launcher.py", line 352, in launch
self._run_step(
File "/opt/venv/lib/python3.11/site-packages/zenml/orchestrators/step_launcher.py", line 457, in _run_step
self._run_step_without_step_operator(
File "/opt/venv/lib/python3.11/site-packages/zenml/orchestrators/step_launcher.py", line 457, in _run_step
self._run_step_without_step_operator(
File "/opt/venv/lib/python3.11/site-packages/zenml/orchestrators/step_launcher.py", line 543, in _run_step_without_step_operator
runner.run(
File "/opt/venv/lib/python3.11/site-packages/zenml/orchestrators/step_launcher.py", line 543, in _run_step_without_step_operator
runner.run(
File "/opt/venv/lib/python3.11/site-packages/zenml/orchestrators/step_runner.py", line 185, in run
self._stack.prepare_step_run(info=step_run_info)
File "/opt/venv/lib/python3.11/site-packages/zenml/orchestrators/step_runner.py", line 185, in run
self._stack.prepare_step_run(info=step_run_info)
File "/opt/venv/lib/python3.11/site-packages/zenml/stack/stack.py", line 962, in prepare_step_run
component.prepare_step_run(info=info)
File "/opt/venv/lib/python3.11/site-packages/zenml/stack/stack.py", line 962, in prepare_step_run
component.prepare_step_run(info=info)
File "/opt/venv/lib/python3.11/site-packages/zenml/integrations/mlflow/experiment_trackers/mlflow_experiment_tracker.py", line 201, in prepare_step_run
mlflow.start_run(
File "/opt/venv/lib/python3.11/site-packages/zenml/integrations/mlflow/experiment_trackers/mlflow_experiment_tracker.py", line 201, in prepare_step_run
mlflow.start_run(
File "/opt/venv/lib/python3.11/site-packages/mlflow/tracking/fluent.py", line 368, in start_run
active_run_obj = client.get_run(existing_run_id)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/venv/lib/python3.11/site-packages/mlflow/tracking/fluent.py", line 368, in start_run
active_run_obj = client.get_run(existing_run_id)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/venv/lib/python3.11/site-packages/mlflow/tracking/client.py", line 251, in get_run
return self._tracking_client.get_run(run_id)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/venv/lib/python3.11/site-packages/mlflow/tracking/client.py", line 251, in get_run
return self._tracking_client.get_run(run_id)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/venv/lib/python3.11/site-packages/mlflow/tracking/_tracking_service/client.py", line 108, in get_run
return self.store.get_run(run_id)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/venv/lib/python3.11/site-packages/mlflow/tracking/_tracking_service/client.py", line 108, in get_run
return self.store.get_run(run_id)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/venv/lib/python3.11/site-packages/mlflow/store/tracking/rest_store.py", line 177, in get_run
response_proto = self._call_endpoint(GetRun, req_body)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/venv/lib/python3.11/site-packages/mlflow/store/tracking/rest_store.py", line 177, in get_run
response_proto = self._call_endpoint(GetRun, req_body)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/venv/lib/python3.11/site-packages/mlflow/store/tracking/rest_store.py", line 90, in _call_endpoint
return call_endpoint(self.get_host_creds(), endpoint, method, json_body, response_proto)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/venv/lib/python3.11/site-packages/mlflow/store/tracking/rest_store.py", line 90, in _call_endpoint
return call_endpoint(self.get_host_creds(), endpoint, method, json_body, response_proto)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/venv/lib/python3.11/site-packages/mlflow/utils/rest_utils.py", line 402, in call_endpoint
response = verify_rest_response(response, endpoint)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/venv/lib/python3.11/site-packages/mlflow/utils/rest_utils.py", line 402, in call_endpoint
response = verify_rest_response(response, endpoint)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/venv/lib/python3.11/site-packages/mlflow/utils/rest_utils.py", line 259, in verify_rest_response
raise RestException(json.loads(response.text))
File "/opt/venv/lib/python3.11/site-packages/mlflow/utils/rest_utils.py", line 259, in verify_rest_response
raise RestException(json.loads(response.text))
mlflow.exceptions.RestException: RESOURCE_DOES_NOT_EXIST: Run with id=2bf891a5-5cb5-4e0e-8615-5b6ad4bd3504 not found
mlflow.exceptions.RestException: RESOURCE_DOES_NOT_EXIST: Run with id=2bf891a5-5cb5-4e0e-8615-5b6ad4bd3504 not foundCode of Conduct
- I agree to follow this project's Code of Conduct
andras-diffusedrive
Metadata
Metadata
Assignees
Labels
plannedPlanned for the short termPlanned for the short termsnackx-squadIssues that are being handled by the x-squadIssues that are being handled by the x-squad
Type
Projects
Status
In Review