OpenAI-Compatible API Tutorial: Connect SDKs, Local Models, and Cloud Providers Without Rewriting Your App

Overview

An OpenAI-compatible API usually exposes familiar request and response shapes, often including a chat or responses-style operation, model selection, message arrays, token limits, temperature controls, and optional streaming. The purpose is practical: an application already built around a common SDK may be able to use another model provider by changing configuration rather than rewriting every integration layer.

Compatibility is not the same as feature parity. Providers can differ in supported model names, authentication headers, base URL paths, context limits, tool-calling behavior, structured output support, streaming events, embeddings, rate limits, and error formats. A safe migration treats the API contract as a starting point, then verifies each capability your application actually uses.

Keep provider-specific settings outside business logic. A small configuration layer makes it easier to switch between a hosted endpoint, a local model API, and a cloud gateway while keeping prompts, routing, logging, and application behavior consistent.

LLM_BASE_URL=https://provider.example/v1
LLM_API_KEY=replace-with-secret
LLM_MODEL=provider-model-name
LLM_EMBEDDING_MODEL=embedding-model-name
LLM_TIMEOUT_SECONDS=60

Use environment variables or a secrets manager for credentials. Do not commit keys to source control, place them in client-side code, or print them in request logs. For broader deployment guidance, see the production checklist for deploying an LLM API on the cloud.

Checklist by scenario

Scenario 1: Connect an existing SDK to another hosted provider

1. Identify the SDK’s configuration points. Find the constructor or client settings for the API key and base URL. Some libraries expect a versioned base URL; others append the path themselves. Confirm whether the value should end with /v1, a provider-specific path, or no path suffix.
2. Replace the model identifier. Model names are provider-specific. Do not assume that a familiar model name exists at the new endpoint. Store the name in configuration and verify it with the provider’s model list or deployment settings.
3. Send a minimal request first. Test one short user message before enabling streaming, tools, JSON output, long context, or multimodal inputs. This isolates authentication and routing problems.
4. Compare the response shape. Confirm where the generated text, finish reason, usage information, request ID, and safety or moderation metadata appear. A response that looks similar may still omit fields your application expects.
5. Run application-level tests. Test representative prompts, not only a successful “hello” request. Include empty inputs, long inputs, malformed tool arguments, timeouts, and provider errors.

Scenario 2: Connect a local model API

Local model servers commonly expose an OpenAI-compatible endpoint, but the supported surface depends on the server configuration and selected model. Start with text-only chat and a small context window. Check whether the server requires a placeholder API key, a specific model alias, or an explicit host and port.

• Confirm that the local server is reachable from the application container or host.
• Use a stable model alias so application configuration does not depend on a changing file name.
• Set a realistic timeout for model loading and first-token latency.
• Check whether streaming is enabled and whether partial events use the format expected by your SDK.
• Measure memory, concurrency, and queue behavior before exposing the endpoint to other users.

Scenario 3: Add streaming

Streaming changes the application contract. Instead of waiting for one completed response, the client receives incremental events and must assemble text, handle disconnects, and decide what to do when a stream ends without a normal completion signal.

Test these cases explicitly: a normal stream, an upstream timeout, a user cancellation, a proxy that buffers events, and a provider error after several tokens have already been displayed. Never assume that every provider uses identical event names or includes usage data in the final event.

Scenario 4: Use tool calls or structured output

Tool calling is one of the least portable parts of an LLM integration. Compare the accepted schema, required fields, parallel-call behavior, argument serialization, and finish or stop indicators. Validate tool arguments with your own schema before execution, and treat model-generated arguments as untrusted input.

For structured output, verify whether the provider guarantees a schema, merely encourages valid JSON, or supports neither. Use a parser, validation step, retry policy, and a safe fallback. Prompt engineering can improve consistency, but it does not replace validation.

Scenario 5: Add embeddings for search or RAG

Embeddings are not automatically portable just because chat requests are. Confirm that the provider exposes an embeddings endpoint, identify the correct embedding model, and record the vector dimension in your index configuration. Documents and queries must use compatible embedding settings; changing the model generally requires planning for re-indexing or maintaining separate indexes.

Before switching a RAG pipeline, compare retrieval results on a fixed evaluation set. The RAG prompt engineering guide can help with grounded-answer testing, while an LLM evaluation workflow provides a useful structure for regression checks.

What to double-check

Base URL and routing: Determine whether the SDK appends endpoint paths. A duplicated path can produce confusing 404 errors, while a missing version segment can route to the wrong service.

Authentication: Check whether the provider expects a bearer token, a custom header, a gateway key, or no key for a local development server. Keep authentication handling in one adapter rather than scattering it across requests.

Model and deployment names: Some services distinguish between a public model identifier and a private deployment name. Make the configured value explicit and log a redacted provider and model label for troubleshooting.

Limits and timeouts: Compare context limits, maximum output tokens, request size, concurrency, and timeout behavior. A prompt that succeeds with one model may be rejected or truncated by another.

Errors and retries: Categorize authentication failures, invalid requests, rate limits, timeouts, overload responses, and server errors separately. Retry only transient failures, use bounded exponential backoff, and add a request ID to logs when available.

Observability and privacy: Track latency, status codes, token usage when available, model identifiers, and failure categories. Avoid logging secrets or sensitive prompt content by default. For security considerations around hostile instructions in retrieved or user-provided text, review the prompt injection prevention guide.

Common mistakes

• Changing only the API key: A different provider normally requires a different base URL and model identifier too.
• Assuming SDK compatibility means model compatibility: The client may accept a request that the selected model cannot fulfill.
• Skipping a capability matrix: Record support for streaming, tools, structured output, vision, embeddings, and batch operations before committing to a migration.
• Retrying every error: Repeating invalid requests increases cost and latency without fixing the cause.
• Mixing provider logic into prompts: Keep provider-specific parameters in configuration or an adapter so prompt tests remain comparable.
• Comparing outputs without a test set: Use representative inputs and expected properties. A single successful response does not establish production equivalence.
• Ignoring downstream effects: Changes in tokenization, formatting, tool arguments, or embedding dimensions can affect storage, parsing, retrieval, and user interfaces.

When to revisit

Revisit this checklist before switching providers, moving from a local model to a hosted endpoint, changing a model deployment, upgrading an SDK, or adding streaming, tools, structured output, or embeddings. It is also worth reviewing before a planned seasonal traffic cycle or any major change to your cloud architecture.

Keep a small provider migration record with the following items:

1. The tested base URL format and authentication method.
2. Model and embedding identifiers used in each environment.
3. Supported capabilities and known differences.
4. Representative prompts, tool schemas, and expected output checks.
5. Observed timeout, retry, rate-limit, and error behavior.
6. Rollback configuration for the previous endpoint.

After every endpoint change, run a smoke test, then a regression suite covering quality, parsing, latency, cost controls, security, and failure handling. If the application uses RAG, evaluate retrieval and grounded answers separately from generation. If it uses automated actions, require schema validation and authorization checks before executing any tool call. This approach lets you switch LLM providers deliberately while preserving the parts of your application that should remain stable.