How to Run Local LLMs with OpenAI Codex

This step-by-step guide shows you how to connect open LLMs and APIs to OpenAI Codex entirely locally, complete with screenshots. Codex only needs a local endpoint that speaks the OpenAI Responses API. Run using any open model like Qwen, DeepSeek, Gemma, and more.

For this tutorial, we’ll use the open models: Gemma 4 and Qwen3.5 which are strong agentic & coding models (works on 24GB RAM/unified mem device). For inference, we'll use Unsloth Studio and llama.cpp enables you to run/serve LLMs on macOS, Linux, and Windows. You can swap in any other model, just update the model names in your scripts and Codex config.

Setup Codex📖 Setup Local Model Tutorial

For model quants, we'll use Unsloth Dynamic GGUFs so you can run quantized GGUF models while retaining as much accuracy as possible.

Setup Codex

Codex is OpenAI's official coding agent that runs locally. While designed for ChatGPT, it supports custom API endpoints, which makes it work for local LLMs. We'll later point it to Unsloth Studio's /v1/responses endpoint once Unsloth is up.

apt update
sudo apt install nodejs npm -y
npm install -g @openai/codex

winget install --id OpenAI.Codex

winget install --id 9PLM9XGG6VKS --source msstore

bash brew install --cask codex

That's it for the install - don't run codex yet. Running it bare drops you into OpenAI's "Sign in with ChatGPT" picker (which is modal - there's no escape hatch). Once we wire up a local profile, codex --oss --profile unsloth_api or codex --oss --profile llama_cpp skips that screen entirely because custom providers default to requires_openai_auth = false. Start the local model server first, then launch Codex against it.

📖 Quickstart Tutorials

Before we begin, we firstly need to complete setup for the specific model you're going to use. We use Unsloth (a web UI) and llama.cpp which are open-source frameworks for running and serving LLMs on your Mac, Linux, Windows devices.

Before we begin, we firstly need to complete setup for the specific model you're going to use. We use Unsloth (a web UI) and llama.cpp which are open-source frameworks for running and serving LLMs on your Mac, Linux, Windows devices.

Unsloth also has unique self-healing tool-calling and web search capabilities. See right for Claude Code connected to Unsloth:

🦥 Unsloth Tutorial🦙 llama.cpp Tutorial

🦥 Unsloth Tutorial

For this tutorial, we will serve/connect local models to Claude Code via a UI by using Unsloth. Unsloth works on Windows, WSL, Linux and MacOS.

• Search, download, run GGUFs and safetensor models

• Self-healing tool calling + web search

• Code execution (Python, Bash)

• Automatic inference parameter tuning (temp, top-p, etc.)

• Fast CPU + GPU inference via llama.cpp

• Train LLMs 2x faster with 70% less VRAM

See below for install instructions:

1

Download Unsloth

The easiest way to get started is by installing the Unsloth Desktop app. It supports MacOS, Linux, Windows, NVIDIA, AMD, Intel and CPU setups.

Download Unsloth

• Download for macOS

• Download for Windows

• Download for Linux

Or, if you prefer manual installation:

MacOS, Linux, WSL:

Windows PowerShell:

2

Install

1. Open the Unsloth installer (.dmg, .exe files)

2. Drag Unsloth to Applications for Mac or complete setup for Windows.

3. Launch the app and wait for installation to complete

3

Choose a model

Open 'Select model' dropdown ontop or 'Model hub' tab, choose a model and a quantization that fits your device, then download it. Once it finishes, start chatting - no setup required.

4

Unsloth is now ready

To start chatting, type a message and press Enter.

• Connect tools: Claude Code, Codex, web search, MCP and more

• Train models: Fine-tune text, diffusion, embedding, and more

• Generate media: Create and train images, video, TTS locally

Model Loading + API Guide

1

Select Model

Before using the API, load a model from the Select model dropdown in the top-left corner of the Chat page.

In this guide, we’ll use: unsloth/gemma-4-26B-A4B-it-GGUF with the recommended UD-Q4_K_XL quantization.

2

Test the Model

Before using the Client, send a quick message:

This confirms that the model loaded correctly and is ready to respond.

3

Unsloth API key

In Unsloth, open Settings → API to view or create your API key. If you start Unsloth headless with unsloth run, the key is also printed to the console as sk-unsloth-....

Treat your API key like a password and avoid exposing it in screenshots or repositories.

Optional: adjust runtime settings

You can pass additional runtime options when starting a model with unsloth run.

Reasoning-capable models can be started with --reasoning on or --reasoning off. The -c flag controls the available context window.

Use -p if you need the API to run on a different port.

For more advanced runtime configuration, see the main API tuning section.

⚙️ Connect Codex

Now that we have setup the local LLM for Codex, we now configure Codex to work with your tool. You can either connect easily with unsloth start below or do it manually.

⚡ Run OpenAI Codex with unsloth start

To launch Codex directly with a model, run:

With a GGUF model loaded in Unsloth Studio, open your project folder and run:

Unsloth creates an isolated Codex home and a Unsloth backed Responses provider for the launch. Your normal ~/.codex configuration is left alone.

Codex state is temporary by default. Use --persist when you want to keep its Unsloth-managed configuration and sessions:

Codex currently requires a GGUF model served through the llama-server backend.

See the complete unsloth start reference for model loading, persistence and all wrapper options.

The rest of this guide covers a fully manual Codex provider setup.

🔌 Connect manually

This section is for manual setting up, it is the same whether you used Unsloth Studio, llama.cpp, or another OpenAI-compatible local server. Codex needs three values: the API key, the base URL, and the model name. The example below uses Unsloth Studio; for llama.cpp, use the same shape with the llama_cpp profile in the llama.cpp section.

1

Configure the Unsloth provider

Codex looks for ~/.codex/config.toml on macOS/Linux/WSL or %USERPROFILE%\.codex\config.toml on Windows. Create or edit it:

Next, create a Codex profile for Unsloth:

model must match the id your server reports at GET http://localhost:8888/v1/models. Unsloth Studio exposes the full repo id (for example unsloth/gemma-4-26B-A4B-it-GGUF). The llama.cpp section below uses --alias "unsloth/gemma-4-26B-A4B", so use that shorter id when pointing Codex at llama-server instead.

This config registers an unsloth_api Codex model provider, points it at Unsloth Studio, and sets unsloth_api as the default local provider for codex --oss. The separate unsloth_api profile selects the Unsloth provider and model only when you launch Codex with --profile unsloth_api, so your normal Codex setup is not changed. Codex reads the API key from an environment variable named UNSLOTH_STUDIO_AUTH_TOKEN. You'll set the real key in the next step.

Field

What it does

base_url

Your local server endpoint + /v1

env_key

Name of the env var Codex reads your API key from. This is not the key itself.

wire_api

responses. Codex now exclusively uses OpenAI's Responses API.

requires_openai_auth

false makes Codex skip the "Sign in with ChatGPT" screen for this provider. Default is already false, but be explicit.

model

The model ID your server exposes. Hit GET <base_url>/models to confirm the exact string.

oss_provider

Sets unsloth_api as the default local provider when launching Codex with --oss.

requires_openai_auth

false makes Codex skip the "Sign in with ChatGPT" screen for this provider.

OpenAI removed wire_api = "chat" support. Always use wire_api = "responses". If you set wire_api = "chat", Codex refuses to start with `wire_api = "chat"` is no longer supported. How to fix: set `wire_api = "responses"` in your provider config.

You can create multiple profile files, one for each Unsloth model you swap between. Launch the one you want with codex --profile <profile-name>.

2

Set the API key env var

Use the same env var name you wrote in env_key. In the Unsloth Studio example above, env_key = "UNSLOTH_STUDIO_AUTH_TOKEN", so set UNSLOTH_STUDIO_AUTH_TOKEN in the same terminal you will run Codex from:

If you renamed env_key, rename the variable in the commands too. For example, a llama.cpp profile that uses env_key = "LLAMA_CPP_API_KEY" needs LLAMA_CPP_API_KEY, not UNSLOTH_STUDIO_AUTH_TOKEN.

Session vs Persistent: the commands above apply to the current terminal only. To persist:

• MacOS / Linux / WSL: add the export line to ~/.bashrc (bash) or ~/.zshrc (zsh).

• Windows: run setx UNSLOTH_STUDIO_AUTH_TOKEN "YOUR_TOKEN" once, or add the $env: line to your PowerShell $PROFILE.

Running Codex inside WSL with Unsloth on Windows? WSL is a separate network namespace, so localhost from inside WSL doesn't reach Unsloth. Edit your config.toml to use the Windows host IP instead:

Then set base_url = "http://<that-ip>:8888/v1". If you have WSL2 mirrored networking enabled (.wslconfig → networkingMode=mirrored), localhost works as on native Windows.

3

Launch Codex

First launch in a new directory Codex asks "Do you trust the contents of this directory?" - pick Yes, continue. This is the per-cwd trust prompt, not the ChatGPT login (that one is skipped because of `requires_openai_auth = false`). Subsequent launches in the same directory skip this prompt.

Seeing Model metadata for unsloth/gemma-4-26B-A4B not found. Defaulting to fallback metadata? Codex ships with a built-in table of context windows, tool support, and input modalities for OpenAI's own models. For anything else - it falls back to safe defaults. The warning fires once per session for every non-OpenAI slug. Everything still works, you can ignore it.

To fix it: add model_context_window = 131072 to the top of ~/.codex/config.toml so Codex uses Gemma 4's real 128K context instead of its fallback guess. For full control over tool support and input modalities too, point model_catalog_json inside [profiles.unsloth_api] at a JSON file containing a custom ModelInfo entry for your slug.

The --profile unsloth_api flag tells Codex to load ~/.codex/unsloth_api.config.toml, which selects the Unsloth Studio provider and model. Add --oss to run through Codex's local OSS provider flow. The model name appears in Codex's status bar.

Add --search to enable web search:

To bypass all approval prompts (BEWARE this will make Codex do and execute code however it likes without any approvals!):

Try a real task

Try this prompt to install and run a simple Unsloth finetune:

and if we wait a little longer, you will see a successfully fine-tuned model with Unsloth!

Disconnect or revert

Launch Codex without -p unsloth_api and it'll use its default provider. Or delete the [profiles.unsloth_api] and [model_providers.unsloth_api] blocks from ~/.codex/config.toml.

You can leave Unsloth Studio running or shut it down. It doesn't intercept anything when stopped.

Troubleshooting

🦙 Llama.cpp Tutorial

We can also use llama.cpp directly. We need to deploy llama-server which is an open-source framework for running and serving LLMs efficiently on Mac, Linux and Windows devices. The model will be served on port 8001 with all agent tool calls routed through that single OpenAI-compatible endpoint.