Running open-weight and frontier models together

A default split that holds up

Most regulated estates settle somewhere close to this shape. It is not a rule, but it is a reasonable default to challenge from.

• Self-hosted open-weight models handle internal business processing, document and record work, extraction, classification and anything touching personal or commercially sensitive data.
• Commercial frontier models handle open-domain reasoning, complex drafting and tasks where measured quality gaps justify the exposure — on content cleared to leave.
• Routing is declared per agent and per tool, so placement is a reviewable decision rather than a runtime accident.
• Egress allow-lists make an unintended external call a failure rather than a surprise on an invoice.

Operating self-hosted models honestly

Hosting models has real costs: capacity planning, upgrade cycles, evaluation work and the discipline of pinning versions so behaviour does not drift under a process that depends on it. Those costs are usually worth it for the highest-volume internal work, where per-token pricing on someone else's platform compounds fastest.

The advantage compounds too. Once internal models carry the bulk of usage, external spend becomes a small, predictable line rather than the dominant one.

Version pinning is especially important. Open-weight models are updated frequently, and a process that depends on a specific behaviour can break when a new release changes the output distribution. Pin versions explicitly, test before upgrading, and maintain the ability to roll back.

Model cards and evaluation discipline

Every model in production should have a model card that records what it is for, what it is not for, which versions are approved, and what evaluation results support that approval. This is not paperwork for its own sake. It is the evidence an auditor or safety reviewer will ask for.

Evaluations should include task-specific tests, not just general benchmarks. A model that scores well on reasoning may hallucinate names, dates or amounts in your documents. The only way to know is to test it on your data, with your prompts, in your workflow.

Keep evaluation datasets inside the boundary. Sending production-like data to an external evaluation service undermines the same sovereignty you are trying to protect.

Cost accounting per model

Attribute spend by agent, project and model. Without that, the first real budget conversation happens after the fact and no one can say which workload caused it.

Cost attribution also reveals whether the model split is working. If the intention was for internal models to handle the bulk of volume, the spend should reflect that. A disproportionate frontier spend is a signal that the routing policy is not being enforced or that internal model capacity is insufficient.

Include all-in costs for self-hosted models: compute, storage, network, operations time and the evaluation overhead. Comparing only per-token API prices will make frontier models look cheaper than they are.

The fallback question

A well-designed dual-model system should answer a simple fallback question: if the frontier provider becomes unavailable, what breaks? The ideal answer is: nothing critical.

That requires more than having an internal model installed. It requires that internal models are already carrying production traffic for the processes that matter, with proven latency, quality and capacity. A standby model that has never been used is not a fallback.

Test fallback paths regularly. Switch a fraction of traffic to the internal model, compare outputs, and refine prompts until the gap is acceptable. When the external model is eventually withdrawn or repriced, the transition becomes routine.

Hugging Face and the open model ecosystem

The open model ecosystem is larger than a handful of well-known names. Through Hugging Face and compatible serving stacks, organisations can publish and run any model that fits their hardware and task requirements inside their own boundary.

This matters for specialised work: a fine-tuned model for a specific document type, a smaller model for low-latency classification, or an experimental architecture for a research team. Instead of waiting for a provider to offer the right model, the organisation can serve it itself.

The governance principle is the same: any model served inside the boundary is under the organisation's control. The control plane treats it like any other model, with the same policy, approval and evidence requirements.