Skip to content

[FR]: Testing agent should handle AppFunctionUriGrant results and pass returned images to the model #63

Open

Description

@PujieWear

Description

A function can return an image the standard way, as an AppFunctionUriGrant (a content URI plus FLAG_GRANT_READ_URI_PERMISSION). The Testing Agent doesn't recognize this:

  1. Not shown: AgentOrchestrator.handleLlmResponse (around line 301) only picks up images from a result field named imageUri, the agent's own convention from generateImage (see [FR]: Render image URIs inline within the Test/Sample Agent Chat UI #12). A standard AppFunctionUriGrant result is passed to the model as plain text, and no image appears in the chat.
  2. Not seen by the model: even with imageUri, the image is only attached to the chat message for the user. GeminiProviderImpl never sends image content to the model, so a function's returned image never reaches the LLM.

Why it matters

Visual apps rely on returning renders so the agent can check its own work. For example, a design app can expose renderDesign(): AppFunctionUriGrant and instruct the agent to render after changes and correct what it sees. With the Testing Agent, the model works blind, so this loop can't be tested at all, even though it is exactly the kind of flow a system agent would need to support.

Suggested behavior

  • Recognize AppFunctionUriGrant (or any result whose URI points to an image/* type) as an image result, alongside the imageUri convention.
  • Read it through the granted permission and attach it to the next model turn as image input (Gemini inline_data), so the model can see it.
  • Optionally show it inline in the chat, as [FR]: Render image URIs inline within the Test/Sample Agent Chat UI #12 did for imageUri.

Environment

  • Testing Agent at a3a2a89 (main, Oct 23, 2026)
  • androidx.appfunctions 1.0.0-beta01
  • Android 17 (API 37) emulator

Related: #12 (inline image URIs in the chat), #62.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions