Skip to content

MLflow is broken on AzureML #4207

@AlexejPenner

Description

@AlexejPenner

Contact Details [Optional]

No response

System Information

https://cloud.zenml.io/workspaces/zenml-projects/projects/productprice/runs/9303a467-b097-48b5-b53e-1e00f7d78ecf?tab=overview

This pipeline failed with cryptic errors while every other stack with the same mlflow component works

What happened?

MLflow fails on Azure

Reproduction steps

...

Relevant log output

INFO
2025-11-05 20:25:01
mlflow.exceptions.RestException: RESOURCE_DOES_NOT_EXIST: Run with id=2bf891a5-5cb5-4e0e-8615-5b6ad4bd3504 not found
Copy to Clipboard
ERROR
2025-11-05 20:25:01
An exception has occurred.
Traceback (most recent call last):
Traceback (most recent call last):
  File "/opt/venv/lib/python3.11/site-packages/zenml/orchestrators/step_launcher.py", line 352, in launch
    self._run_step(
  File "/opt/venv/lib/python3.11/site-packages/zenml/orchestrators/step_launcher.py", line 352, in launch
    self._run_step(
  File "/opt/venv/lib/python3.11/site-packages/zenml/orchestrators/step_launcher.py", line 457, in _run_step
    self._run_step_without_step_operator(
  File "/opt/venv/lib/python3.11/site-packages/zenml/orchestrators/step_launcher.py", line 457, in _run_step
    self._run_step_without_step_operator(
  File "/opt/venv/lib/python3.11/site-packages/zenml/orchestrators/step_launcher.py", line 543, in _run_step_without_step_operator
    runner.run(
  File "/opt/venv/lib/python3.11/site-packages/zenml/orchestrators/step_launcher.py", line 543, in _run_step_without_step_operator
    runner.run(
  File "/opt/venv/lib/python3.11/site-packages/zenml/orchestrators/step_runner.py", line 185, in run
    self._stack.prepare_step_run(info=step_run_info)
  File "/opt/venv/lib/python3.11/site-packages/zenml/orchestrators/step_runner.py", line 185, in run
    self._stack.prepare_step_run(info=step_run_info)
  File "/opt/venv/lib/python3.11/site-packages/zenml/stack/stack.py", line 962, in prepare_step_run
    component.prepare_step_run(info=info)
  File "/opt/venv/lib/python3.11/site-packages/zenml/stack/stack.py", line 962, in prepare_step_run
    component.prepare_step_run(info=info)
  File "/opt/venv/lib/python3.11/site-packages/zenml/integrations/mlflow/experiment_trackers/mlflow_experiment_tracker.py", line 201, in prepare_step_run
    mlflow.start_run(
  File "/opt/venv/lib/python3.11/site-packages/zenml/integrations/mlflow/experiment_trackers/mlflow_experiment_tracker.py", line 201, in prepare_step_run
    mlflow.start_run(
  File "/opt/venv/lib/python3.11/site-packages/mlflow/tracking/fluent.py", line 368, in start_run
    active_run_obj = client.get_run(existing_run_id)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/mlflow/tracking/fluent.py", line 368, in start_run
    active_run_obj = client.get_run(existing_run_id)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/mlflow/tracking/client.py", line 251, in get_run
    return self._tracking_client.get_run(run_id)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/mlflow/tracking/client.py", line 251, in get_run
    return self._tracking_client.get_run(run_id)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/mlflow/tracking/_tracking_service/client.py", line 108, in get_run
    return self.store.get_run(run_id)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/mlflow/tracking/_tracking_service/client.py", line 108, in get_run
    return self.store.get_run(run_id)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/mlflow/store/tracking/rest_store.py", line 177, in get_run
    response_proto = self._call_endpoint(GetRun, req_body)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/mlflow/store/tracking/rest_store.py", line 177, in get_run
    response_proto = self._call_endpoint(GetRun, req_body)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/mlflow/store/tracking/rest_store.py", line 90, in _call_endpoint
    return call_endpoint(self.get_host_creds(), endpoint, method, json_body, response_proto)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/mlflow/store/tracking/rest_store.py", line 90, in _call_endpoint
    return call_endpoint(self.get_host_creds(), endpoint, method, json_body, response_proto)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/mlflow/utils/rest_utils.py", line 402, in call_endpoint
    response = verify_rest_response(response, endpoint)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/mlflow/utils/rest_utils.py", line 402, in call_endpoint
    response = verify_rest_response(response, endpoint)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/mlflow/utils/rest_utils.py", line 259, in verify_rest_response
    raise RestException(json.loads(response.text))
  File "/opt/venv/lib/python3.11/site-packages/mlflow/utils/rest_utils.py", line 259, in verify_rest_response
    raise RestException(json.loads(response.text))
mlflow.exceptions.RestException: RESOURCE_DOES_NOT_EXIST: Run with id=2bf891a5-5cb5-4e0e-8615-5b6ad4bd3504 not found
mlflow.exceptions.RestException: RESOURCE_DOES_NOT_EXIST: Run with id=2bf891a5-5cb5-4e0e-8615-5b6ad4bd3504 not found

Code of Conduct

  • I agree to follow this project's Code of Conduct

Metadata

Metadata

Assignees

Labels

plannedPlanned for the short termsnackx-squadIssues that are being handled by the x-squad

Type

Projects

Status

In Review

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions