Your OpenAI bill arrived and it looked like a mortgage payment. You stared at it, did the math backward, and realized your clever little AI feature is hemorrhaging money at a rate that makes the business case evaporate. Sound familiar?
The good news: most teams are leaving 40–70% cost savings on the table through fixable inefficiencies. Not by switching to a worse model or lobotomizing your prompts — but by being smarter about when, how, and what you send to the API. This guide walks through every lever worth pulling, with real numbers and code you can drop in today.
Understand the Cost Levers First
Before you optimize, you need to know what you're paying for. OpenAI charges on tokens: input tokens (your prompt + context) and output tokens (the model's response). The ratio matters enormously.
- Input tokens are cheaper than output tokens — usually by 3–5x depending on the model.
- Model choice is the single biggest multiplier. GPT-4o can cost 30x more than GPT-4o-mini per token.
- Context window bloat silently inflates every request when you're stuffing in unnecessary history or documents.
Run this snippet to get a real-time token count before you commit to a request:
import tiktoken
def count_tokens(messages: list[dict], model: str = "gpt-4o") -> int:
enc = tiktoken.encoding_for_model(model)
total = 0
for msg in messages:
# 4 tokens overhead per message
total += 4
for key, value in msg.items():
total += len(enc.encode(str(value)))
total += 2 # reply priming
return total
messages = [
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "Summarize this document for me..."}
]
print(f"Estimated tokens: {count_tokens(messages)}")
Make this a pre-flight check in your pipeline. If a request is going to exceed a threshold you define (say, 8,000 tokens for a task that should need 2,000), something upstream is broken.
Model Routing: The Highest-ROI Change You Can Make
Not every task deserves GPT-4o. This is the most consistently overlooked optimization, and fixing it alone can cut bills in half.
Build a Simple Task Classifier
Route requests to the cheapest model that can handle them adequately. A classification rubric might look like this:
- GPT-4o-mini: Classification, extraction, simple Q&A, formatting, single-step reasoning
- GPT-4o: Multi-step reasoning, code generation, complex analysis, synthesis across sources
- o1 / o3-mini: Mathematical proofs, deep logical chains, tasks where accuracy is worth the latency cost
import openai
ROUTING_RULES = {
"classify": "gpt-4o-mini",
"extract": "gpt-4o-mini",
"summarize": "gpt-4o-mini",
"reason": "gpt-4o",
"code": "gpt-4o",
"analyze": "gpt-4o",
}
def route_model(task_type: str) -> str:
return ROUTING_RULES.get(task_type, "gpt-4o-mini")
def call_api(prompt: str, task_type: str) -> str:
model = route_model(task_type)
response = openai.chat.completions.create(
model=model,
messages=[{"role": "user", "content": prompt}],
max_tokens=512,
)
return response.choices[0].message.content
You can make the router itself intelligent — use a fast, cheap model to classify incoming tasks before sending them to the appropriate model. The classification call costs almost nothing and pays for itself immediately.
Prompt Compression Without Losing Signal
Fat prompts are a tax on every request. Here's where teams consistently over-inflate their inputs:
1. Trim Your System Prompt Religiously
Audit your system prompt. Strip redundancy, verbose instructions that repeat themselves, and anything that doesn't change model behavior measurably. A 2,000-token system prompt that could be 400 tokens costs you 1,600 tokens on every single call. At scale, this is thousands of dollars per month.
2. Compress Retrieved Context
If you're doing RAG, don't dump entire document chunks into the prompt. Pre-summarize or extract only the sentences most relevant to the query. The semantic search approach we covered in the embeddings article lets you retrieve small, targeted chunks rather than whole pages.
3. Use Structured Output to Control Response Length
Unconstrained generation is expensive. When you know the shape of the output you want, enforce it:
from openai import OpenAI
from pydantic import BaseModel
client = OpenAI()
class SentimentResult(BaseModel):
label: str # "positive", "negative", "neutral"
confidence: float
reasoning: str # max one sentence
response = client.beta.chat.completions.parse(
model="gpt-4o-mini",
messages=[
{"role": "user", "content": "Classify sentiment: 'The product broke after two days.'"}
],
response_format=SentimentResult,
)
result = response.choices[0].message.parsed
print(result.label, result.confidence)
Structured outputs prevent the model from padding responses with unnecessary explanation. You get exactly what you asked for, nothing more.
Caching: Pay Once, Use Many Times
OpenAI's Prompt Caching (available on GPT-4o and newer models) automatically discounts repeated prefixes at 50% off. If your system prompt or context is consistent across requests, you're already getting this — but only if your messages are structured to take advantage of it.
Rules for effective prompt caching:
- Keep the static part of your prompt (system message, shared context) at the beginning of your messages array.
- Put the variable, user-specific content at the end.
- Maintain consistent phrasing in your static sections — even minor wording changes break the cache.
Application-Level Semantic Caching
For repeated or near-identical queries, add a semantic cache layer before hitting the API at all:
import numpy as np
from openai import OpenAI
client = OpenAI()
cache: list[dict] = [] # In production: use Redis or a vector DB
def cosine_similarity(a, b):
return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))
def get_embedding(text: str) -> list[float]:
response = client.embeddings.create(
model="text-embedding-3-small",
input=text
)
return response.data[0].embedding
def cached_completion(prompt: str, threshold: float = 0.95) -> str:
prompt_embedding = get_embedding(prompt)
for entry in cache:
similarity = cosine_similarity(prompt_embedding, entry["embedding"])
if similarity >= threshold:
print(f"Cache hit! Similarity: {similarity:.3f}")
return entry["response"]
# Cache miss — call the API
response = client.chat.completions.create(
model="gpt-4o-mini",
messages=[{"role": "user", "content": prompt}]
)
result = response.choices[0].message.content
cache.append({
"prompt": prompt,
"embedding": prompt_embedding,
"response": result
})
return result
This pattern is especially effective for chatbots, FAQ systems, or any product where users ask similar questions repeatedly. The embedding call is ~100x cheaper than a full completion.
Batching and Async: Stop Paying for Urgency You Don't Need
OpenAI's Batch API offers 50% off standard pricing for requests that don't need real-time responses. If you're processing documents, running evals, or doing background enrichment — you should be using it.
import json
from openai import OpenAI
client = OpenAI()
# Prepare batch requests
requests = [
{
"custom_id": f"task-{i}",
"method": "POST",
"url": "/v1/chat/completions",
"body": {
"model": "gpt-4o-mini",
"messages": [{"role": "user", "content": f"Summarize: {doc}"}],
"max_tokens": 200
}
}
for i, doc in enumerate(["Doc 1 content...", "Doc 2 content..."])
]
# Write to JSONL file
with open("batch_input.jsonl", "w") as f:
for req in requests:
f.write(json.dumps(req) + "\
")
# Upload and create batch
batch_file = client.files.create(
file=open("batch_input.jsonl", "rb"),
purpose="batch"
)
batch = client.batches.create(
input_file_id=batch_file.id,
endpoint="/v1/chat/completions",
completion_window="24h"
)
print(f"Batch created: {batch.id}")
Pair this with the production deployment patterns we've documented — batching works best when you've already separated your synchronous (user-facing) and asynchronous (background) processing paths.
max_tokens, Temperature, and the Small Settings That Add Up
A few configuration habits that compound over millions of requests:
- Always set
max_tokens: Never leave this unset on production calls. Unbounded generation can 10x your expected output length on edge cases. - Lower temperature for deterministic tasks: Temperature doesn't affect cost directly, but high-temperature outputs tend to be wordier and less precise, leading to retry loops that absolutely do cost money.
- Use
stopsequences: For structured outputs, defining stop tokens prevents the model from generating beyond what you need. - Skip
logprobsunless you need them: Requesting log probabilities adds overhead you rarely need in production.
Monitor Before You Optimize Further
You can't sustain cost discipline without visibility. At minimum, track:
- Tokens per request (input and output separately)
- Cost per feature or endpoint
- Cache hit rate
- Model distribution across your fleet
This feeds directly into the agentic workflow design decisions you'll face when your system grows — knowing which sub-agents are expensive is prerequisite to optimizing them. If you're building persistent agent systems, the infrastructure patterns in our AI agent memory and sandbox piece show how to architect context storage that doesn't just dump everything into every prompt.
Practical Takeaways
Pull these levers in order of impact:
- Route to cheaper models first — GPT-4o-mini handles more than you think. Validate with evals before assuming you need the big model.
- Audit and compress system prompts — Run a token count on your current system prompt right now. You'll find waste.
- Add semantic caching for any product with repetitive query patterns.
- Use the Batch API for all non-real-time processing.
- Enforce output structure with Pydantic models and
max_tokenson every call. - Instrument everything — cost visibility prevents regressions as your product evolves.
None of these require sacrificing quality. The teams paying the most per useful output are usually the ones who never measured what "useful" costs in the first place. Now you have the tools to fix that.
