Skip to content

feat: Add bring-your-own ElevenLabs voice agent support #258

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
rrnewton wants to merge 6 commits into
base: main
Choose a base branch
from
Open

feat: Add bring-your-own ElevenLabs voice agent support #258

rrnewton wants to merge 6 commits into from
+1,261 −142

Conversation

@rrnewton
Copy link

@rrnewton rrnewton commented Dec 3, 2025

Summary

Adds support for users to configure their own ElevenLabs API key and voice agent instead of using the default Happy voice agent. This enables self-hosted deployments and users who want to customize their voice
assistant experience.

Changes

  • Voice Settings UI: New configuration section in Settings > Voice to toggle between default and custom agent
  • Custom Credentials: Input fields for ElevenLabs API key and Agent ID with secure storage
  • Agent Management: "Find Agent" button to locate existing "Happy Coding Assistant" agent, and "Create/Update Agent" button to provision one with default configuration
  • API Integration: New apiVoice.ts module for fetching voice tokens with custom credentials and direct ElevenLabs API calls
  • Mute Button: Added mute toggle to the voice assistant status bar
  • Debug Logging: Voice session logging gated by EXPO_PUBLIC_DEBUG_VOICE env var
  • i18n: Added translation strings for all new UI elements across all supported languages

How it works

  1. User enables "Use Custom Agent" toggle in voice settings
  2. User enters their ElevenLabs API key
  3. User either:
    • Clicks "Find Agent" to locate an existing agent named "Happy Coding Assistant"
    • Clicks "Create Agent" to provision a new agent with the default Happy configuration
  4. Credentials are saved locally and passed to the server when requesting voice tokens
  5. Server uses provided credentials instead of its own defaults

Testing

  • Toggle custom agent on/off and verify voice sessions work in both modes
  • Test Find Agent with valid/invalid API keys
  • Test Create Agent flow
  • Verify mute button functionality
  • Check debug logging when env var is set

Manual Test plan

  • Enable custom agent toggle, enter valid API key
  • Click "Find Agent" - should find or report not found
  • Click "Create Agent" - should create new agent
  • Start voice session with custom agent - should work
  • Disable custom agent - should fall back to default
  • Test mute button during active voice session
rrnewton and others added 6 commits December 2, 2025 15:30
@rrnewton @claude
Add private voice agent support with server-side token generation
87edc9e 
- Create apiVoice.ts client to fetch conversation tokens from server
- Update RealtimeVoiceSession.tsx and .web.tsx to use conversationToken
  instead of hardcoded agentId for self-hosted deployments
- Enables users to configure their own ElevenLabs agent

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
@rrnewton @claude
Add ElevenLabs custom agent configuration UI in voice settings
dd7da16 
- Add settings fields for elevenLabsUseCustomAgent, elevenLabsAgentId, elevenLabsApiKey
- Create UI with toggle to enable custom agent and input fields for credentials
- Update apiVoice.ts to pass custom credentials to server when enabled
- Add translations for all supported languages (en, es, pl, ru, zh-Hans, ca, pt)
- Update settings tests with new fields

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
@rrnewton @claude
Enhance ElevenLabs voice settings with Find/Create agent buttons
8f0e95d 
- Add Find Agent and Create/Update Agent buttons to voice settings
- Add show/hide toggle for API key field
- Add Save Credentials button
- Add ElevenLabs agent management API functions (findHappyAgent, createOrUpdateHappyAgent)
- System prompt is now embedded in code as single source of truth
- Add translations for all new UI elements

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
@rrnewton @claude
Add 'Get API Key' help link to voice settings
107a3ea 
- Add help link that opens ElevenLabs API keys page
- Update description to include required permissions:
  - "ElevenLabs Agents" (Write) for agent management
  - "Text to Speech" (Access) for TTS
- Add getApiKey translation to all language files

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
@rrnewton @claude
Enable debug logging for voice session gated by env var
9e137d1 
Logs are now enabled when PUBLIC_EXPO_DANGEROUSLY_LOG_TO_SERVER_FOR_AI_AUTO_DEBUGGING
is set. All voice session callbacks now log with [Voice] prefix.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
@rrnewton @claude
Add mute button to voice assistant status bar
90bc2a5 
Allows muting the microphone during voice sessions without ending the session.
When muted, audio is not sent to the LLM but the session stays active.
Note: Mute functionality only works on web (native SDK limitation).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
@rrnewton rrnewton mentioned this pull request Dec 3, 2025
Open
@rrnewton
Copy link
Author

rrnewton commented Dec 3, 2025

(Update: ran some limited PR tests via the happy-devbox repo that simulates a monorepo linking the commits in these PRs together.)

E2E Tests

This PR includes working E2E tests using Playwright:

  • Create Account flow: Tests loading the webapp, creating an account, and verifying logged-in state
  • Error handling: Verifies no critical errors during normal operation

CI validation runs ./scripts/validate.sh which includes:

  • TypeScript builds for happy-cli, happy-server, and webapp
  • Unit tests for happy-cli and happy-server
  • E2E tests with browser automation

Green CI Run: https://github.com/rrnewton/happy-devbox/actions/runs/19894554194/job/57021496616
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

1 participant

@rrnewton