Human in the Loop Agent UI and Agent interface

[amanjaiswal73892]

Introduces a new agent interface to make any agent human-in-the-loop. At each step, the agent proposes the next action, and humans either select one of the actions or provide a natural-language hint to guide the generation of proposed actions.

• Includes a companion UI to enable human intervention, by adding one or more hints to get candidate actions and see past history of interaction.

• Adds a stable human_guided_generic_agent.

• Adds a draft human-in-the-loop (HITL) agent interface: Multi-candidate Generic Agent.

• Updates Xray to view any added in hints in agent_info tab.

• Registered the new script entry point agentlab-mentor for launching the HITL agent UI.

• You can run the the UI and the stable human guided generic agent as follows:

agentlab-mentor  --benchmark miniwob --task-name "miniwob.book-flight" --seed 7

• Download the human hints:

agentlab-mentor  --download-hints

Run agentlab-mentor Generic Agent UI on a benchmark task

Run HITL Generic Agent UI on a benchmark task

options:
  -h, --help            show this help message and exit
  --benchmark BENCHMARK
                        Benchmark name as registered in BrowserGym, e.g., miniwob, workarena_l1, webarena,
                        visualwebarena
  --task-name TASK_NAME
                        Exact task name within the benchmark (e.g., 'miniwob.book-flight')
  --seed SEED           Task seed to use for the selected task.
  --headless, --no-headless
                        Run the browser headless (default: True). Use --no-headless to show the browser.
  --download-hints [[OUTPUT_CSV]]
                        Extract hints from the default study directory and save to OUTPUT_CSV. If OUTPUT_CSV is
                        omitted, saves to 'extracted_hints.csv'. When provided, other args are ignored.

---

This pull request introduces a new "Human-in-the-Loop" (HILT) agent architecture for web automation tasks, enabling a human operator to select among multiple candidate actions proposed by an underlying agent. The changes modularize the agent design, add protocol definitions for multi-candidate agents.
Human-in-the-loop agent workflow and UI integration:

Generic multi-candidate agent implementation:

• [Stable] Added MultipleProposalGenericAgent and its argument class in generic_human_guided_agent.py, providing a concrete agent that generates multiple candidate actions using LLM prompts, parses structured responses, and integrates with a hint-labeling UI for human selection.
  Human-in-the-Loop agent architecture and protocol:

• [Draft] Added the MultiCandidateAgent protocol in base_multi_candidate_agent.py, defining a standard interface for agents that generate multiple candidate actions and update their internal state based on the selected candidate. Also introduced MultiCandidateAgentArgs for agent argument handling and naming conventions.

• [Draft] Implemented the HumanInTheLoopAgent class in hilt_agent.py, which wraps any multi-candidate agent and presents candidate actions to a human via a UI, allowing hints and selection, and updating agent state accordingly. Includes error handling and UI integration.

User interface and action visualization:

• Introduced utilities (overlay_action, img_to_base_64) in both agent files to overlay proposed actions on screenshots and encode images for the UI, enhancing human interpretability of agent suggestions. [1] [2]

---

Description by Korbit AI

What change is being made?

Introduce Human in the Loop (HILT) Agent UI and interfaces for agent implementations that support multiple candidate actions and user guidance.

Why are these changes being made?

These changes are being made to facilitate a more interactive approach where a human can guide the decision-making process of an AI agent by selecting among multiple proposed actions. The implementation allows agents to propose several viable actions in complex environments, with a user interface enabling human users to provide hints and select preferred actions. This setup enhances the adaptability and effectiveness of agents in dynamic environments by leveraging human intuition and expertise.

Is this description stale? Ask me to generate a new description by commenting /korbit-generate-pr-description

[korbit-ai]

Review by Korbit AI

Korbit automatically attempts to detect when you fix issues in new commits.

Category	Issue	Status
	Missing action match validation ▹ view	🧠 Incorrect
	Missing configuration interface definition ▹ view	🧠 Not in standard
	Inefficient Image Processing Loop ▹ view	✅ Fix detected
	Unclear State Update Parameter Specification ▹ view	✅ Fix detected
	Document factory function purpose and types ▹ view	✅ Fix detected
	Unused intervention round counter ▹ view	✅ Fix detected
	Optional Agent Name ▹ view	🧠 Not in scope
	Document complex interaction flow ▹ view	🧠 Not in standard
	Inconsistent Function Name Case ▹ view	✅ Fix detected
	Improper Type Hint Comment ▹ view	🧠 Incorrect

Files scanned

​

File Path	Reviewed
src/agentlab/agents/hilt_agent/base_multi_candidate_agent.py	✅
src/agentlab/agents/hilt_agent/hint_labelling.py	✅
src/agentlab/agents/hilt_agent/multi_candidate_generic_agent.py	✅
src/agentlab/agents/hilt_agent/hilt_agent.py	✅
src/agentlab/agents/hilt_agent/generic_human_guided_agent.py	✅
src/agentlab/agents/hilt_agent/hint_labelling_ui_files/hint_labeling_ui.html	✅

Explore our documentation to understand the languages and file types we support and the files we ignore.

Check out our docs on how you can make Korbit work best for you and your team.

Loving Korbit!? Share us on LinkedIn Reddit and X

[recursix]

maybe this could be in a utile file somehwere where we reconcile to avoid duplicates. e.g. the _url version would call this one.

[recursix]

linter doesn't resolve, are we missing init.py ?