fix: clean internal format markers from agent output when parsing fails

[majiayu000]

Summary

• Add _clean_raw_output() function to strip internal ReAct format markers (Thought:, Action:, etc.) from agent output when parsing fails
• Extract Final Answer content if present in the raw response
• Preserve original text in text field for debugging while cleaning output field
• Add comprehensive unit tests for the cleaning function

Test plan

• Added unit tests for _clean_raw_output() function
• Added tests for format_answer() with cleaning behavior
• CI should pass all existing tests
• Manual verification: hierarchical crew outputs should not contain internal markers

Fixes #3873

---

Note

Introduce output cleaning for failed parses in format_answer via _clean_raw_output and add comprehensive unit tests.

• Utilities (lib/crewai/src/crewai/utilities/agent_utils.py):
  • Add _clean_raw_output to strip internal ReAct markers (Thought:, Action:, Action Input:, Observation:) and extract content after Final Answer:.
  • Update format_answer fallback to use cleaned output in AgentFinish.output while preserving original text in AgentFinish.text.
• Tests (lib/crewai/tests/utilities/test_agent_utils.py):
  • Add unit tests covering cleaning behavior, multiline final answers, passthrough of plain text, and format_answer behavior on parse failure.

^{Written by Cursor Bugbot for commit 6038dbb. This will update automatically on new commits. Configure here.}

[cursor]

This PR is being reviewed by Cursor Bugbot

Details

Your team is on the Bugbot Free tier. On this plan, Bugbot will review limited PRs each billing cycle for each member of your team.

To receive Bugbot reviews on all of your PRs, visit the Cursor dashboard to activate Pro and start your 14-day free trial.

[cursor]

Numbered action markers not cleaned from output

The _clean_raw_output function uses exact prefix checks like startswith("Action:") and startswith("Action Input:"), but the parser accepts numbered variants via regex patterns that include \d* (e.g., Action\s*\d*\s*:). This means markers like Action 1:, Action1:, or Action 1 Input: are recognized by the parser but won't be cleaned when parsing fails. If an LLM outputs a numbered action marker and parsing fails, that internal format marker will appear in the user-facing output field.