Hints retrieval in tool use agent #277
Conversation
![korbit-ai[bot]](https://cdn.statically.io/img/avatars.githubusercontent.com/in/322216?s=60&v=4){kind=small}
There was a problem hiding this comment.
Review by Korbit AI
Korbit automatically attempts to detect when you fix issues in new commits.
| Category | Issue | Status |
|---|---|---|
 |
Missing Embedding Cache ▹ view | ✅ Fix detected |
 |
Incomplete Error Logging ▹ view | |
 |
Incorrect TaskHint.choose_hints() docstring ▹ view | |
 |
Missing Goal Validation in Embedding-based Hint Selection ▹ view | ✅ Fix detected |
Files scanned
| File Path | Reviewed |
|---|---|
| src/agentlab/llm/tracking.py | ✅ |
| src/agentlab/agents/tool_use_agent/tool_use_agent.py | ✅ |
| src/agentlab/analyze/agent_xray.py | ✅ |
Explore our documentation to understand the languages and file types we support and the files we ignore.
Check out our docs on how you can make Korbit work best for you and your team.
| try: | ||
| hint_topic_idx = json.loads(response.think) | ||
| if hint_topic_idx < 0 or hint_topic_idx >= len(hint_topics): | ||
| logger.error(f"Wrong LLM hint id response: {response.think}, no hints") | ||
| return [] | ||
| hint_topic = hint_topics[hint_topic_idx] | ||
| hint_indices = topic_to_hints[hint_topic] | ||
| df = self.hint_db.iloc[hint_indices].copy() | ||
| df = df.drop_duplicates(subset=["hint"], keep="first") # leave only unique hints | ||
| hints = df["hint"].tolist() | ||
| logger.debug(f"LLM hint topic {hint_topic_idx}, chosen hints: {df['hint'].tolist()}") | ||
| except json.JSONDecodeError: | ||
| logger.error(f"Failed to parse LLM hint id response: {response.think}, no hints") | ||
| hints = [] |
There was a problem hiding this comment.
Incomplete Error Logging
Tell me more
What is the issue?
The error handling in choose_hints_llm() loses potentially useful error context by only logging the error message without including the original exception details.
Why this matters
Without the original exception details in the logs, debugging production issues will be more difficult as developers won't have access to the full stack trace and error context.
Suggested change ∙ Feature Preview
Include the exception details in the error logging using exc_info=True:
try:
hint_topic_idx = json.loads(response.think)
if hint_topic_idx < 0 or hint_topic_idx >= len(hint_topics):
logger.error(f"Wrong LLM hint id response: {response.think}, no hints")
return []
hint_topic = hint_topics[hint_topic_idx]
hint_indices = topic_to_hints[hint_topic]
df = self.hint_db.iloc[hint_indices].copy()
df = df.drop_duplicates(subset=["hint"], keep="first") # leave only unique hints
hints = df["hint"].tolist()
logger.debug(f"LLM hint topic {hint_topic_idx}, chosen hints: {df['hint'].tolist()}")
except json.JSONDecodeError as e:
logger.error(f"Failed to parse LLM hint id response: {response.think}, no hints", exc_info=True)
hints = []Provide feedback to improve future suggestions
💬 Looking for more details? Reply to this comment to chat with Korbit.
| def choose_hints(self, llm, task_name: str, goal: str) -> list[str]: | ||
| """Choose hints based on the task name.""" |
There was a problem hiding this comment.
Incorrect TaskHint.choose_hints() docstring
Tell me more
What is the issue?
The docstring is inaccurate as the method chooses hints based on task_name OR goal depending on hint_retrieval_mode, not just task_name.
Why this matters
Misleading docstring could cause confusion about the method's behavior when using different hint retrieval modes.
Suggested change ∙ Feature Preview
def choose_hints(self, llm, task_name: str, goal: str) -> list[str]:
"""Choose hints based on hint_retrieval_mode using task name or goal text."""
Provide feedback to improve future suggestions
💬 Looking for more details? Reply to this comment to chat with Korbit.
| def encode_hints(self): | ||
| self.uniq_hints = self.hint_db.drop_duplicates(subset=["hint"], keep="first") | ||
| logger.info( | ||
| f"Encoding {len(self.uniq_hints)} unique hints using {self.embedder_model} model." | ||
| ) | ||
| self.hint_embeddings = self.emb_model.encode( | ||
| self.uniq_hints["hint"].tolist(), prompt="task hint" | ||
| ) |
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
| def choose_hints_emb(self, goal: str) -> list[str]: | ||
| """Choose hints using embeddings to filter the hints.""" | ||
| goal_embeddings = self.emb_model.encode([goal], prompt="task description") | ||
| similarities = self.emb_model.similarity(goal_embeddings, self.hint_embeddings) |
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
amanjaiswal73892
left a comment
amanjaiswal73892
left a comment
There was a problem hiding this comment.
Everything looks good to me.
Description by Korbit AI
What change is being made?
Implement a task hint retrieval system in the tool use agent, enabling dynamic selection of hints through direct, LLM, or embedding-based methods, alongside integrating new models and embedding resources.
Why are these changes being made?
These changes are made to enhance the tool use agent's capability to provide contextual hints for tasks, leveraging modern AI techniques like language models and embeddings, thereby improving task performance and user support without manual hint selection while updating available model configurations and ensuring semantic richness in task-oriented assistance.