LLM Cost Comparison: A Practical TCO Framework for Open-Weight and Proprietary Models

For many AI teams, LLM spend is no longer an experimental line item. But the key question is not whether open-weight or proprietary models have the lower headline token price. It is which architecture delivers the required quality, latency, reliability, compliance, and cost per successful task for a specific workload.

Proprietary APIs often provide the simplest starting point for low, uncertain, or rapidly changing demand. Self-hosted open-weight models may become economically attractive when demand is sustained, utilization is high, the model meets the quality threshold, and the organization can support the operational requirements. Hosted open-model APIs offer another option: they can provide access to open-weight models without requiring the team to operate GPU infrastructure.

Why token price is not TCO

The most visible number in an LLM comparison is usually the provider’s price per million input and output tokens. That number is useful, but it does not answer the full budget question.

A production comparison should include at least five categories:

• API or inference costs: input tokens, cached input, cache writes, output tokens, batch processing, service tiers, and provider-specific tools.
• Platform and infrastructure: GPUs, CPU and memory, storage, networking, load balancing, vector search, embeddings, reranking, and observability.
• Customization: prompt development, retrieval-augmented generation, fine-tuning, evaluation datasets, and quality assurance.
• Operations: deployment, monitoring, incident response, autoscaling, security, compliance, patching, capacity planning, and on-call coverage.
• Transition and switching: migration work, prompt retuning, integration changes, evaluation reruns, license review, and the risk of degraded quality during a change.

A useful baseline:

For quality-sensitive workloads, also track cost per successful task: total cost divided by tasks meeting the agreed quality and latency threshold. A lower-cost model may require more retries, longer prompts, additional verification, or human review — a lower token rate does not automatically produce a lower cost per business outcome.

Proprietary API pricing

Commercial LLM APIs generally charge separately for input and output tokens. Output is often more expensive than input, but the ratio varies by provider and model — check the current rate card before estimating costs.

API pricing may also include several modifiers:

• Cached input: reusable prompt prefixes may be billed at a lower rate, subject to provider-specific eligibility, cache-write charges, retention, and cache-hit rates.
• Batch processing: asynchronous jobs may receive lower rates when they don’t require interactive responses.
• Long-context pricing: some models apply different rates above a context-length threshold.
• Service tiers: faster or higher-priority processing may carry a premium.
• Tool and agent costs: tool schemas, arguments, results, repeated context, retries, and additional model turns can contribute to token usage.

For current rates, consult the OpenAI API pricing page, Anthropic pricing documentation, and Google Gemini API pricing documentation.

Indicative pricing structure

The table below illustrates how commercial rate cards may be structured. It is not intended to rank providers or serve as a permanent market comparison — verify prices and model availability directly on each provider’s page.

A transparent API example

Consider a support workflow processing 50,000 conversations each month, where each conversation generates 2,000 input tokens and 500 output tokens, with no tool call, at $2 per million input tokens and $10 per million output tokens.

If a second tool call adds 1,000 input tokens and 500 output tokens per conversation, the incremental volume — 50 million input tokens and 25 million output tokens — adds $100 and $250 respectively, bringing the model-only subtotal to $800 per month. The actual impact of a tool call depends on schema size, arguments, results, repeated context, model turns, retries, and caching — measure it from representative traces rather than applying a universal multiplier.

Basic API cost = input tokens × input rate + output tokens × output rate.
A more complete production formula: Total API cost = standard input + cached input + cache writes + output + batch/service-tier adjustments + tool and retrieval charges + platform charges.

Open-weight model costs

“Open-source” and “open-weight” are not interchangeable terms. Open-weight refers to models whose trained parameters are available for download but whose training data, code, or license may impose additional conditions.

Self-hosting does not make inference free — it changes the billing structure from a metered API bill to a combination of accelerator capacity, platform services, engineering labor, and operational risk. Before using any model commercially, check the model card and license directly; the Open Source AI Definition explains why access to weights alone doesn’t establish that a model is open source.

GPU and serving economics

Cloud GPU costs vary by provider, region, instance type, purchasing model, and availability. Neocloud providers may offer lower hourly rates than hyperscalers, while hyperscalers may offer stronger enterprise integration, compliance controls, and support. For current infrastructure pricing, consult the AWS EC2 pricing page, AWS Capacity Blocks pricing, Lambda pricing, and RunPod pricing.

Compute-only planning example

Monthly compute cost = hourly GPU rate × 730. This is a compute-only planning estimate; it excludes or may not fully represent storage, networking, orchestration, monitoring, redundancy, support, and engineering labor.

A GPU’s hourly price does not determine cost per million tokens — that requires measured useful throughput: cost per million tokens = total hourly serving cost ÷ useful tokens served per hour × 1,000,000. Useful throughput depends on the model, precision, context length, batching, concurrency, input/output mix, and latency target. See the vLLM documentation for current serving and benchmarking guidance.

Utilization matters

A dedicated GPU incurs cost during idle periods. For example, if a GPU costs $3,000 per month in compute:

These are capacity-allocation examples, not provider invoices. They show why self-hosting can look attractive in a high-volume benchmark but unattractive when traffic is uncertain or spiky. High availability adds further fixed cost: two replicas for failover may roughly double accelerator capacity before autoscaling headroom or multi-zone deployment. A single GPU may suit development or low-availability workloads, but it shouldn’t be treated as a universal production architecture.

Model memory and hardware assumptions

Model weights are only one part of serving memory — KV cache, runtime overhead, batching, context length, and concurrency also require headroom. Consult the current model card and deployment documentation rather than inferring hardware requirements from parameter count alone. Quantization can substantially reduce memory requirements, but the result depends on the checkpoint, quantization format, runtime, context length, and serving configuration; benchmark quality, latency, throughput, and failure rates before production deployment.

Other cost categories

Retrieval and data services

A production RAG system may incur costs for document extraction, chunking, embeddings, vector search, reranking, object storage, metadata stores, network traffic, and application hosting — costs that can remain even when the generation model changes.

Operations and staffing

Self-hosting requires more than downloading weights and exposing an endpoint. The operating model may include deployment automation, GPU scheduling, model updates, health checks, observability, autoscaling, incident response, capacity planning, security controls, and evaluation after model or prompt changes. Monthly operations cost = allocated engineering cost + on-call cost + platform overhead. If a team doesn’t already operate GPU inference, the first relevant cost may be engineering time or specialist support rather than the accelerator itself.

Customization and evaluation

Fine-tuning costs depend on model size, training method, dataset volume, sequence length, GPU type, number of experiments, and evaluation requirements. LoRA and QLoRA may reduce compute requirements, but full cost also includes data preparation, experiment management, validation, and deployment. RAG and prompt engineering create recurring costs too — a lower-cost model needing more retrieval context or verification may not be cheaper at the workflow level.

Switching and licensing

Migration costs can include prompt retuning, API integration, structured-output validation, tool-call changes, regression testing, safety review, and quality revalidation. Downloadable weights do not necessarily imply unrestricted commercial use.

Decision framework

There is no universal token-volume threshold at which self-hosting becomes cheaper. The break-even point depends on the model, GPU provider, workload shape, quality target, throughput, utilization, redundancy, engineering effort, and the API being displaced.

Hybrid architecture

A hybrid design can combine the benefits of both approaches:

• Self-host a model for predictable baseline traffic.
• Use a proprietary API for overflow and demand spikes.
• Route simple tasks to lower-cost models.
• Reserve frontier models for complex reasoning, coding, or quality-critical cases.
• Use batch processing for offline classification, evaluation, enrichment, and backfills.
• Apply prompt caching where repeated prefixes are large and cache hits are measurable.

This approach can reduce the risk of paying for a large idle GPU fleet while preserving a fallback path during traffic spikes or model upgrades, and lets a team validate demand before making a fixed infrastructure commitment.

Cost-management practices

Before changing models or deployment architecture:

• Measure input and output tokens separately by workflow, tenant, model, and feature.
• Record tool-call count, tool-result size, retries, and repeated context.
• Route simple classification, extraction, and validation tasks to lower-cost models.
• Bound agent loops with explicit step, token, time, and retry limits.
• Use structured outputs to reduce unnecessary generation.
• Cache stable prompt prefixes and measure actual cache-hit rates.
• Use batch processing for jobs that don’t require interactive responses.
• Quantize only after evaluating quality, latency, and failure rates.
• Track cost per successful task, not only cost per token.
• Re-run the comparison after material changes in traffic, pricing, model quality, or architecture.

Conclusion

There is no universal cost winner between proprietary APIs and self-hosted open-weight models. APIs often offer the simplest and lowest-risk option for early-stage, low-volume, unpredictable, or rapidly changing workloads. Self-hosted inference may become economically attractive when demand is sustained, utilization is high, the model meets the quality bar, and the organization can support the operational requirements.

Self-hosting may reduce inference cost compared with premium proprietary APIs under favorable utilization and throughput assumptions, but the advantage can disappear when GPUs are idle, redundancy is required, operations are included, or a lower-cost hosted API is available. A credible TCO analysis should begin with an observed workload trace rather than a pricing-page comparison — model the request mix, token distribution, tool behavior, throughput, latency target, availability requirement, GPU price, utilization, engineering allocation, and quality threshold, then compare proprietary API, hosted open-model, managed dedicated, and self-hosted options on the same basis.