Add Ollama API integration

[ezcat207]

Summary

This PR adds comprehensive Ollama integration to the MIRIX desktop agent, enabling users to run local Large Language Models (LLMs) via Ollama as a privacy-focused, cost-effective alternative to cloud-based models. The implementation includes full backend support, real image recognition for vision models, bug fixes for multimodal content handling, frontend UI updates, and flexible configuration options.

What is Ollama?

Ollama is a tool that allows you to run LLMs locally on your machine. This PR enables MIRIX to use Ollama-hosted models (like Qwen, DeepSeek, Llama, etc.) instead of relying solely on cloud APIs.

🎯 Key Highlight: Vision Model Support

Ollama vision models can now see and analyze images! The implementation converts images from the database to base64 format and sends them to Ollama, enabling models like qwen3-vl:235b-cloud to perform real image analysis, OCR, and visual understanding tasks.

Key Changes

🔧 Backend Integration

1. Core Ollama Provider Support (mirix/llm_api/llm_api_tools.py)

• Added ollama as a new LLM provider option
• Implemented Ollama request handling using OpenAI-compatible API endpoints
• Ollama exposes an OpenAI-compatible API at /v1, allowing us to reuse existing OpenAI client code
• Handles endpoint configuration with default fallback to http://localhost:11434/v1

2. Vision Model Image Support (mirix/llm_api/ollama_helpers.py - NEW FILE)

• Created dedicated helper module to enable real image recognition for Ollama vision models
• Converts image_id to base64-encoded images that Ollama can actually see and analyze
• Uses FileManager to retrieve image file paths from the database
• Reads image files and converts them to base64 format (data:image/png;base64,<data>)
• Supports both source_url (direct URLs) and local file paths
• Includes comprehensive error handling with fallback to text placeholders if image loading fails
• Enables vision models like qwen3-vl:235b-cloud to perform real image analysis

3. Model Provider Detection (mirix/agent/agent_wrapper.py)

• Updated _determine_model_provider() to recognize Ollama models
• Enhanced _create_llm_config_for_provider() to generate proper Ollama configurations
• Supports custom Ollama endpoints via model_settings.ollama_base_url

4. Model Constants (mirix/agent/app_constants.py)

• Added OLLAMA_MODELS list containing supported models:
  • qwen3-vl:235b-cloud (multimodal vision model)
  • deepseek-v3.1:671b-cloud (large reasoning model)
  • llama3.2, mistral, and other popular Ollama models

🐛 Critical Bug Fixes

1. Fixed UnboundLocalError in Retry Logic (mirix/agent/agent.py)

• Issue: When an HTTP error occurred during LLM API calls, the retry loop would fail with UnboundLocalError: cannot access local variable 'response'
• Root Cause: Missing continue statement after handling HTTP errors
• Fix: Added continue to properly restart the retry loop
• Impact: Prevents crashes when API calls fail and need to be retried

2. Fixed ValueError in Multimodal Token Counting (mirix/utils/common.py)

• Issue: num_tokens_from_messages() crashed with ValueError: Message has non-string value when processing multimodal messages
• Root Cause: Function expected string content but received list-based multimodal content (text + images)
• Fix: Added logic to handle list-based content and estimate tokens for image content
• Impact: Enables proper token counting for messages containing images

3. Fixed 400 Bad Request Errors from Ollama

• Issue: Ollama returned 400 Bad Request when receiving messages with Gemini-specific image formats
• Root Cause: Ollama doesn't support CloudFileContent or image_id fields in image content
• Fix: Created ollama_helpers.py to preprocess messages and convert invalid formats to text placeholders
• Impact: Ensures all messages sent to Ollama are in valid OpenAI-compatible format

4. Improved Multimodal Content Handling (mirix/schemas/message.py)

• Enhanced to_openai_dict() method to properly handle various image content types
• Ensures compatibility across different LLM providers (OpenAI, Gemini, Ollama)

🎨 Frontend Updates

Settings Panel (frontend/src/components/SettingsPanel.js)

• Added Ollama models to both baseModels and baseMemoryModels arrays
• Users can now select Ollama models from the settings dropdown
• Models appear alongside existing OpenAI and Gemini options

App Initialization (frontend/src/App.js)

• Preserved desktop-agent's default model (gpt-4o-mini)
• Ensures backward compatibility - existing users won't see any changes unless they opt-in to Ollama

⚙️ Configuration Files

1. New Dedicated Config (mirix/configs/mirix_ollama.yaml - NEW FILE)

agent_name: mirix
model_name: qwen3-vl:235b-cloud
model_endpoint_type: ollama
model_endpoint: http://localhost:11434/v1
context_window: 16384
put_inner_thoughts_in_kwargs: true

• Ready-to-use configuration for Ollama
• Can be used by specifying this config file when starting MIRIX

2. Updated Existing Configs (mirix.yaml and mirix_monitor.yaml)

• Added commented-out Ollama configuration options
• Users can easily enable Ollama by uncommenting these lines
• Default configurations remain unchanged (Gemini models)

How to Use Ollama with MIRIX

Prerequisites

1. Install Ollama: https://ollama.ai/
2. Pull a model: ollama pull qwen3-vl:235b-cloud
3. Verify Ollama is running: curl http://localhost:11434/api/tags

Option 1: Uncomment in Existing Config

Edit mirix/configs/mirix_monitor.yaml (or mirix.yaml):

# Uncomment these lines:
model_name: qwen3-vl:235b-cloud
model_endpoint_type: ollama
model_endpoint: http://localhost:11434/v1
context_window: 16384
put_inner_thoughts_in_kwargs: true

Option 2: Use Dedicated Ollama Config

The mirix_ollama.yaml file is already configured - just use it when starting MIRIX.

Option 3: Change via Frontend Settings

1. Start MIRIX
2. Open Settings panel
3. Select an Ollama model from the dropdown (e.g., qwen3-vl:235b-cloud)
4. Click "Save"

Testing & Validation

✅ Functional Testing

• Tested with qwen3-vl:235b-cloud multimodal model
• Verified text-only conversations work correctly
• Verified multimodal conversations (text + images) work correctly
• Tested model switching between Gemini, OpenAI, and Ollama

✅ Bug Fix Validation

• Confirmed UnboundLocalError no longer occurs during API retries
• Confirmed ValueError no longer occurs with multimodal messages
• Confirmed 400 Bad Request errors are resolved with proper message preprocessing
• Tested token counting works correctly for both text and multimodal content

✅ Backward Compatibility

• Existing configurations continue to work without changes
• Default model remains gemini-2.0-flash for desktop-agent
• No breaking changes to existing functionality

✅ Configuration Testing

• Verified all three configuration methods work correctly
• Tested custom Ollama endpoints
• Validated configuration file parsing

✅ Vision Model Testing

• Created test image with geometric shapes (blue rectangle, red circle) and text
• Tested with qwen3-vl:235b-cloud vision model
• Successfully verified image recognition:
  • ✅ Correctly identified blue color and rectangle shape
  • ✅ Correctly identified red color and circle shape
  • ✅ Detected black borders and white text
  • ✅ Read and recognized text content ("BLUE BOX", "RED")
  • ✅ Identified white background
• Response time: ~9 seconds for detailed image analysis
• Tokens used: 142 prompt + 500 completion = 642 total

Important Notes

Backward Compatibility

• No breaking changes: All existing functionality remains intact
• Default behavior unchanged: Desktop-agent continues to use Gemini by default
• Opt-in feature: Users must explicitly enable Ollama via configuration

Design Decisions

1. Commented configs: Ollama options are commented out by default to avoid surprising existing users
2. OpenAI compatibility: Leverages Ollama's OpenAI-compatible API to minimize code changes
3. Message preprocessing: Handles provider-specific quirks transparently without affecting other providers
4. Flexible configuration: Supports multiple ways to enable Ollama (config files, frontend settings)

Known Limitations

• Ollama must be running locally (or accessible via network)
• Some Gemini-specific features (like CloudFileContent) are converted to text placeholders for Ollama
• Images are converted to base64 format, which increases message size (trade-off for local processing)

Files Changed

New Files:

• mirix/llm_api/ollama_helpers.py (91 lines - includes real image conversion to base64)
• mirix/configs/mirix_ollama.yaml (7 lines)

Modified Files:

• mirix/llm_api/llm_api_tools.py - Added Ollama provider handling
• mirix/agent/agent_wrapper.py - Updated model provider detection
• mirix/agent/app_constants.py - Added Ollama model constants
• mirix/agent/agent.py - Fixed retry logic bug
• mirix/utils/common.py - Fixed multimodal token counting
• mirix/schemas/message.py - Improved multimodal handling
• frontend/src/components/SettingsPanel.js - Added Ollama models to UI
• frontend/src/App.js - Preserved default settings
• mirix/configs/mirix.yaml - Added commented Ollama options
• mirix/configs/mirix_monitor.yaml - Added commented Ollama options

Total Changes: 12 files changed, 271 insertions(+), 40 deletions(-)

Related Issues

This PR addresses the need for:

• Local LLM inference capability
• Privacy-focused alternative to cloud APIs
• Cost reduction for high-volume usage
• Support for custom/fine-tuned models via Ollama

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}