feat(swarm): record provider/model on SwarmRun for cost audit + debug#92
Merged
warren618 merged 1 commit intoMay 10, 2026
Merged
Conversation
run.json currently has no field describing which LLM provider/model the
swarm was actually launched against — SwarmAgentSpec.model_name carries
per-agent overrides but defaults to None ("use the global config"), and
that resolved config is never captured anywhere. Cost auditing,
side-by-side provider experiments, and post-hoc bug triage all need this
information; today it has to be reconstructed from .env timestamps.
This adds two optional fields to SwarmRun:
- provider: lowercase form of LANGCHAIN_PROVIDER, the same env var
src/providers/llm.py:136,195 reads when constructing ChatLLM.
- model: LANGCHAIN_MODEL_NAME (raw, since model strings are
case-sensitive — "gpt-4o" vs "openai/gpt-5").
Both default to None and are populated in SwarmRuntime.start_run before
the run is persisted (runtime.py:start_run), so every new run.json from
this point forward records the active provider/model. Per-agent
overrides remain visible on SwarmAgentSpec.model_name; this field is the
run-level default.
Backward compat: existing run.json files lack the keys entirely. Pydantic
deserializes missing optional fields to their default (None), so
SwarmStore.get_run / list_runs continue to work on legacy files. A
parametrized regression test covers this in
agent/tests/test_swarm_run_metadata.py along with construction,
serialization round-trip, and acceptance of multiple provider names.
This was referenced May 10, 2026
Collaborator
|
Thanks for adding the run-level provider/model metadata. I verified the new serialization/backward-compatibility tests in a combined merge gate, and the change is small and useful for later cost/debug auditing. Merged. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
SwarmRun—providerandmodel— and populates them from the activeLANGCHAIN_PROVIDER/LANGCHAIN_MODEL_NAMEenv vars at run-creation time..swarm/runs/<id>/run.jsonwritten from this point forward records which LLM the run was actually executed against.run.jsonfiles lack the keys and continue to deserialise (defaultNone).Why
run.jsoncurrently has no field describing which provider / model the swarm was launched against.SwarmAgentSpec.model_namecarries per-agent overrides but defaults toNone("use the global config"), and that resolved config is never captured anywhere.This makes three things harder than they should be:
total_input_tokens/total_output_tokensare meaningless without knowing the model's price-per-token.kimi-k2.5vsgpt-4oon the same preset requires manually correlating timestamps with.envhistory.Changes
agent/src/swarm/models.pyTwo
Optional[str]fields onSwarmRun, both defaulting toNone:The docstring spells out the scoping rule: this is the run-level default; per-agent overrides remain visible on
SwarmAgentSpec.model_name.agent/src/swarm/runtime.pyIn
SwarmRuntime.start_run, read the same env varssrc/providers/llm.py:136,195reads when constructingChatLLM, and assign them on the run before_store.create_runpersistsrun.json. Empty / unset env producesNone, not the empty string.agent/tests/test_swarm_run_metadata.py(new)Six unit tests covering the new behaviour:
Nonebehaviourrun.json(noprovider/modelkeys) still parses, untouched fields preservedanthropic,deepseek,openrouter)Out of scope
SwarmAgentSpec.model_name. That field is unchanged.worker.py:_estimate_tokensis still a character-count heuristic. That's a separate gap worth addressing but it doesn't depend on this PR.run.json; surfacing them in the UI is a follow-up.Test plan
pytest agent/tests/test_swarm_run_metadata.py -v— 6/6 pass.pytest --ignore=agent/tests/e2e_backtest --tb=line -q— full sweep: same baseline pass count + 6 new tests, no regressions.pytest agent/tests/test_swarm_preset_inspect.py agent/tests/test_swarm_presets_packaging.py agent/tests/test_models.py -v— adjacent swarm tests still green (17/17).Checklist
agent/src/agent/,agent/src/session/,agent/src/providers/).LANGCHAIN_*env-var contract.CONTRIBUTING.md(Conventional Commit prefix, Google-style docstrings, regression test added).run.jsonparses without modification.