Skip to main content

OpenAI-Compatible Endpoints

To call models hosted behind an openai proxy, make 2 changes:

  1. Put openai/ in front of your model name, so litellm knows you're trying to call an openai-compatible endpoint.

  2. Do NOT add anything additional to the base url e.g. /v1/embedding. LiteLLM uses the openai-client to make these calls, and that automatically adds the relevant endpoints.

Usage - completion​

import litellm
import os

response = litellm.completion(
    model="openai/mistral,               # add `openai/` prefix to model so litellm knows to route to OpenAI
    api_key="sk-1234",                  # api key to your openai compatible endpoint
    api_base="http://0.0.0.0:8000",     # set API Base of your Custom OpenAI Endpoint
    messages=[
                {
                    "role": "user",
                    "content": "Hey, how's it going?",
                }
    ],
)
print(response)

Usage - embedding​

import litellm
import os

response = litellm.embedding(
    model="openai/GPT-J",               # add `openai/` prefix to model so litellm knows to route to OpenAI
    api_key="sk-1234",                  # api key to your openai compatible endpoint
    api_base="http://0.0.0.0:8000",     # set API Base of your Custom OpenAI Endpoint
    input=["good morning from litellm"]
)
print(response)