MAX container

Get started​

First, make sure you're on a system with the following requirements:

Then start an endpoint with the MAX container:

1. Make sure you have Docker installed.

2. Agree to the Gemma 3 license on Hugging Face and set the HF_TOKEN environment variable:

   export HF_TOKEN="hf_..."

3. Start the container and an endpoint for Gemma 3:

   • NVIDIA
   • AMD

   docker run --gpus=1 \
     -v ~/.cache/huggingface:/root/.cache/huggingface \
     -v ~/.cache/max_cache:/opt/venv/share/max/.max_cache \
     -p 8000:8000 \
     modular/max-nvidia-full:latest \
     --model google/gemma-3-27b-it

   It can take a few minutes to pull the container and then download and compile the model.

   When the endpoint is ready, you'll see a message that says this:

   🚀 Server ready on http://0.0.0.0:8000 (Press CTRL+C to quit)

4. Open a new terminal and send a request using the openai Python API or curl:

   • Python
   • cURL

   1. Create a new virtual environment:

      mkdir quickstart && cd quickstart

      python3 -m venv .venv/quickstart \
        && source .venv/quickstart/bin/activate

   2. Install the OpenAI Python API:

      pip install openai

   3. Create the following file to send an inference request:

      generate-text.py

      from openai import OpenAI
      
      client = OpenAI(
          base_url="http://0.0.0.0:8000/v1",
          api_key="EMPTY",
      )
      
      completion = client.chat.completions.create(
          model="google/gemma-3-27b-it",
          messages=[
              {
                "role": "user",
                "content": "Who won the world series in 2020?"
              },
          ],
      )
      
      print(completion.choices[0].message.content)

   4. Run it and you should see results like this:

      python generate-text.py

      The **Los Angeles Dodgers** won the World Series in 2020!
      
      They defeated the Tampa Bay Rays 4 games to 2. It was their first World Series title since 1988.
      
      It was a unique World Series as it was played in a neutral site (Globe Life Field in Arlington, Texas) due to the COVID-19 pandemic.

For details about the OpenAI-compatible endpoint, see our Serve API docs.

To run a different model, change the --model to something else from our supported models.

For information about the available containers, see the Modular Docker Hub repositories.

Container options​

The docker run command above includes the bare minimum commands and options, but there are other docker options you might consider, plus several options to control features of the endpoint.

Container contents​

There are multiple MAX container options, including:

• max-full
• max-amd
• max-amd-base
• max-nvidia-full
• max-nvidia-base

docker run --gpus=1 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -v ~/.cache/max_cache:/opt/venv/share/max/.max_cache \
  --env "HF_TOKEN=${HF_TOKEN}" \
  -p 8000:8000 \
  modular/max-full:latest \
  --model google/gemma-3-27b-it

docker run \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -v ~/.cache/max_cache:/opt/venv/share/max/.max_cache \
  --env "HF_TOKEN=${HF_TOKEN}" \
  --group-add keep-groups \
  --device /dev/kfd \
  --device /dev/dri \
  -p 8000:8000 \
  modular/max-full:latest \
  --model google/gemma-3-27b-it

docker run \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -v ~/.cache/max_cache:/opt/venv/share/max/.max_cache \
  --env "HF_TOKEN=${HF_TOKEN}" \
  --group-add keep-groups \
  --device /dev/kfd \
  --device /dev/dri \
  -p 8000:8000 \
  modular/max-amd:latest \
  --model google/gemma-3-27b-it

docker run --gpus=1 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -v ~/.cache/max_cache:/opt/venv/share/max/.max_cache \
  --env "HF_TOKEN=${HF_TOKEN}" \
  -p 8000:8000 \
  modular/max-nvidia-full:latest \
  --model google/gemma-3-27b-it

Recommended cloud instances​

For best performance and compatibility with our supported models, we recommend that you deploy the MAX container on a cloud instance with a GPU that meets the MAX system requirements.

The Modular Platform is hardware-agnostic and optimized for both the latest NVIDIA and AMD GPUs. To take full advantage of this flexibility, Modular partners with compute providers that prioritize diverse hardware optionality. For enterprise-grade hardware flexibility, see our available editions.

If you're running on AWS, GCP, or Azure and want to test MAX with cloud GPUs, we recommend the following instances:

AWS instances:

• P6 instance family (B200 GPU)
• P5 instance family (H100 GPU)
• P4d instance family (A100 GPU)

GCP instances:

• A4 machine series (B200 GPU)
• A3 machine series (H100 GPU)
• A2 machine series (A100 GPU)

Azure instances:

• ND_GB200_v6-series virtual machine
• NCads_H100_v5-series virtual machine
• NCCads_H100_v5-series virtual machine
• ND_H100_v5-series virtual machine
• NC_A100_v4-series virtual machine
• NDm_A100_v4-series virtual machine
• ND_A100_v4-series virtual machine
• ND_MI300X_v5-series virtual machine (AMD GPU)

Logs​

The MAX container writes logs to stdout in JSON format, which you can consume and view via your cloud provider's platform (for example, with AWS CloudWatch).

Console log level is INFO by default. You can modify the log level using the MAX_SERVE_LOGS_CONSOLE_LEVEL environment variable. It accepts the following log levels (in order of increasing verbosity): CRITICAL, ERROR, WARNING, INFO, DEBUG. For example:

docker run modular/max-nvidia-full:latest \
    -env MAX_SERVE_LOGS_CONSOLE_LEVEL=DEBUG \
    ...

Logs default to structured JSON, but if you'd like a more readable format in your console, you can disable structured logs by adding the MODULAR_STRUCTURED_LOGGING=0 environment variable. For example:

docker run --gpus=1 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -v ~/.cache/max_cache:/opt/venv/share/max/.max_cache \
  -p 8000:8000 \
  --env "MODULAR_STRUCTURED_LOGGING=0" \
  modular/max-nvidia-full:latest \
  --model google/gemma-3-27b-it

Metrics​

MAX exposes a Prometheus-compatible /metrics endpoint for monitoring performance. For the full metrics reference and telemetry configuration, see Serving metrics.

License​

The NVIDIA MAX container uses the NVIDIA Deep Learning Container license.

Next steps​

You can get started with container-based deployments or read more about our supported models.

• Deploy MAX on GPU with self-hosted endpoints: Learn how to deploy MAX pipelines to cloud.
• Benchmark MAX on NVIDIA or AMD GPUs: Learn how to use our benchmarking script to measure the performance of MAX.
• Supported models: See all the model architectures supported by MAX.