Self-hosted AI inference for a client organization

The situation.

An organization wanted to use large language models on its own documents and data, and could not send that data to a hosted API.

The constraint.

The models had to run on hardware the client owns, inside their own network, and still be reachable by the people who need them wherever they are, without exposing anything to the internet.

What we found.

The workload did not need a rack of GPUs. A single Apple silicon machine with enough unified memory could run the models the team needed while keeping inference on premises.

What we built.

An inference server running Ollama on Apple silicon, on the client's premises. It is published through a Cloudflare Tunnel, so nothing is port-forwarded, and access is gated by Cloudflare Access so only the client's people, on their identities, can reach it. Models, prompts, and data never leave the machine.

What changed.

The client uses AI on its own material with no third party in the loop, at a fixed cost they already paid.

• Apple silicon
• Ollama
• Cloudflare Tunnel and Access

Related work

• A nine-company holding group, one engineering team
• Image compliance at scale, on the client's own hardware
• A content ingestion kiosk that cannot lose a byte