On-Premise AI Deployment: When Cloud APIs Are Wrong

Opening answer

Cloud chat APIs are the default because they are easy to try. They are the wrong architecture when client files, health records, or nonpublic financial data cannot leave your network, when a model vendor will not (or should not) sit in the prompt path, or when an outage at that vendor would halt work you still have to finish. On-premise AI deployment means the model runs on hardware you control, so prompts and documents stay on your LAN instead of being sent to a third-party inference API. That is a custody decision first, and a cost decision second. U.S. regulators already treat model-as-a-service vendors as companies with a continuous appetite for customer data, and they treat covered healthcare and banking work as activities you still own even after you hire a vendor.[1][3][6]

Cloud APIs put someone else in the prompt path

A consumer or business LLM API looks like a text box. Architecturally it is a pipeline. Your application ships a prompt (often with retrieved documents, form fields, or chat history) to a provider that tokenizes it, runs it on GPUs you do not operate, and returns text. The provider may log inputs for abuse review, retain them for a stated period, or, depending on the product tier and contract, use them to improve models. You do not see the racks. You do not control the subprocessors. You inherit the vendor's outage calendar.

That pipeline is fine for public marketing copy, generic research, and anything you would be willing to paste into a web form. It is a poor fit the moment the prompt contains another person's data that you hold in confidence. The Federal Trade Commission has been explicit that model-as-a-service companies (hosts that sell LLM access over an API or chat interface) have a business incentive to ingest more data, and that customers often reveal sensitive or confidential information, including internal documents and their own users' data, when they use those models.[1] The agency also warned that firms which promise not to train on customer data, then do so through workarounds or quiet terms changes, can face enforcement under existing consumer-protection law. There is no AI exemption from the laws already on the books.[1]

If your legal, clinical, or finance workflow cannot survive that description, you are not looking at a "better prompt." You are looking at a different place to run the model.

What data custody actually requires

Data custody is not a slogan. It is a set of answers you can give a regulator, a client, or your own counsel:

• Who can read the prompt and the retrieved chunks?
• Who can copy them, train on them, or be compelled to produce them?
• Where do logs live, and for how long?
• What happens to the weights, caches, and vector store if the vendor is acquired, subpoenaed, or taken offline?

On a cloud API, the honest answers are "the vendor, its subprocessors, and anyone they must obey." On-premise AI deployment changes the answers: inference happens on a box (or a small cluster) on your network, behind your identity provider, your firewall, and your backup policy. The model files may be open-weight (Llama, Mistral, Qwen, and similar families we specify on our on-premise deployment page). The documents never have to leave the building.

NIST's AI Risk Management Framework, released in January 2023 as a voluntary, use-case-agnostic resource, lists "privacy-enhanced" as a characteristic of trustworthy AI. Privacy here is not marketing language. NIST describes it as the norms that protect autonomy, identity, and dignity, including limits on observation and control over disclosure, and it says confidentiality and control should guide design, development, and deployment choices.[2][4] Generative systems add a sharper version of the same problem. In its July 2024 Generative AI Profile, NIST flags data privacy as a risk unique to or exacerbated by generative models: leakage, unauthorized use, disclosure, or de-anonymization of health, location, biometric, or other sensitive data, plus the related issue that training data is often opaque to the people whose records ended up inside it.[2]

CISA's June 2022 Cloud Security Technical Reference Architecture (now listed as archived on CISA.gov) put the same split in infrastructure terms: cloud adoption has advantages, and it has inherent risks, and agencies that move workloads off their own premises inherit a shared-risk model rather than a fully owned one.[5] You can accept that trade for email or commodity compute. You should not accept it unexamined for a model that reads client files.

Custody also includes availability. A cloud LLM is a dependency. If the provider throttles you, changes a model, raises prices, or has a regional incident, your "AI feature" is a waiting room. Our on-premise deployment work starts from the opposite assumption: the system should keep answering when an external LLM vendor does not.

Healthcare: a BAA is not a magic wand

HIPAA does not ban cloud. HHS OCR's cloud-computing FAQ (last reviewed January 9, 2023) says a covered entity or business associate may use a cloud service to store or process electronic protected health information, provided it signs a HIPAA-compliant business associate agreement with the cloud provider and otherwise follows the HIPAA Rules.[3] The BAA sets permitted uses and requires the vendor to safeguard ePHI under the Security Rule. OCR also reminds covered entities to understand the actual cloud configuration they are buying, because that configuration changes the risk analysis.[3]

Two practical facts follow.

First, many consumer chat APIs will not sign a BAA. If the product page is silent, assume the answer is no until counsel has a signed contract. Sending a chart note, a discharge summary, or a patient email into that API is not "trying ChatGPT." It is creating, transmitting, or maintaining ePHI with a party that has no business-associate relationship.

Second, a BAA does not finish the job. The HIPAA Security Rule still requires covered entities and business associates to run an accurate risk analysis of threats to the confidentiality, integrity, and availability of ePHI, and to implement measures that reduce those risks to a reasonable level.[7] 45 CFR 164.308 also requires written satisfactory assurances before a business associate creates, receives, maintains, or transmits ePHI on your behalf.[7] A clinic that pastes PHI into an unsanctioned chatbot has skipped both the contract and the analysis.

On-premise inference does not make you "HIPAA certified." Nothing we sell does. Our service page is specific on this point: on-prem deployment helps an organization meet regulatory obligations without guaranteeing certification. What it does change is the data-flow diagram. If the model, the retrieval index, and the documents stay on the clinic's network, you are no longer arguing about a consumer LLM's subprocessors. You are arguing about hardware you can inventory, logs you can retain, and access controls you already run for the EHR.

Legal and finance: you still own the risk

Law firms and financial SMBs hit a different version of the same wall. Client files, deal rooms, loan files, and tax records are not "prompts." They are records you hold under professional or statutory duties.

Banking supervisors said this plainly in June 2023. The FDIC, Federal Reserve, and OCC issued joint third-party risk guidance. The FDIC's summary is the sentence that should sit on a finance team's whiteboard: using third parties can increase risk, but it does not diminish or remove the banking organization's responsibility to perform activities in a safe and sound manner, including consumer protection and the security of customer information.[6] The OCC's companion bulletin applies that life-cycle view (planning, due diligence, contracting, monitoring, termination) to every bank with third-party relationships, scaled to the criticality of the activity.[8]

Nonbank finance shops are not off the hook. The FTC's Safeguards Rule, at 16 CFR 314.3, requires covered financial institutions to maintain a written information security program with administrative, technical, and physical safeguards appropriate to their size, complexity, and the sensitivity of customer information, reasonably designed to keep that information confidential and to protect it against unauthorized use that could harm a customer.[9] Dropping account numbers or tax data into a general-purpose LLM API is a vendor-risk and a safeguards-program question, not a productivity hack.

For a small law practice the analysis is similar even without a bank charter. If a prompt includes information relating to a representation, you have moved that information to a third party. Some firms will get a negotiated enterprise contract, a no-training clause, and a data-processing addendum that counsel can live with. Many will not. On-premise AI deployment is the option that keeps the model next to the document management system instead of next to a public API.

Cost: run the numbers, do not buy a myth

The cost story for on-prem used to write itself: APIs billed per token, hardware was a one-time check, high volume won. The second half is still true. The first half moved.

Stanford's 2025 AI Index (Chapter 1, April 2025) reported that the inference cost of a model scoring at GPT-3.5 level on MMLU fell from $20.00 per million tokens in November 2022 to $0.07 per million tokens by October 2024, more than a 280-fold drop in about 18 months. Depending on the task, LLM inference prices fell anywhere from 9 to 900 times per year. At the hardware layer, the same chapter reported machine-learning hardware price-performance improving about 30 percent per year and energy efficiency improving about 40 percent per year. Open-weight models also closed much of the quality gap with closed models on some benchmarks, from about 8 percent in early 2024 to about 1.7 percent by early 2025.[10]

Those figures cut both ways. Cheaper APIs weaken a naive "on-prem always wins on cost" pitch. Cheaper, more efficient GPUs and stronger open-weight models strengthen the case for owning a box if your volume is high or your data cannot leave. The honest evaluation is a spreadsheet, not a slogan:

1. Count monthly tokens (prompt plus completion) for the workloads that actually need a model.
2. Price those tokens at current API rates for a model of comparable quality, including retries, RAG context, and tool-calling overhead.
3. Price hardware (GPU, CPU, RAM, storage, power, cooling) and the staff time to patch, monitor, and restore it.
4. Separate sensitive workloads (must stay on-LAN) from public ones (API is fine).
5. Add the cost of downtime if the API is the only path, and the cost of a breach or a failed audit if it is the wrong path.

Our on-premise deployment page lists typical hardware in the $5,000 to $50,000 range depending on scale, with consulting for specifications starting at $2,500. It also notes that at high volumes, on-premise often pays for itself within 6 to 12 months versus cloud API fees. That is our published planning range, not an independent study, and it has to be recomputed against 2024 to 2026 token prices, not 2023 ones.[10] If your volume is a few thousand queries a month of non-sensitive text, a metered API will usually win on cash. If you are embedding every contract, claim file, or knowledge base document, or if you cannot send the text off-network at all, the API price is not the binding constraint. Custody is.

How to tell if on-prem is the right architecture

Use this as a go/no-go, not as a personality test.

Cloud APIs are usually the right call when the data is already public or would be fine in email; you need the newest frontier model this week; volume is low and spiky; and you have a contract that matches the sensitivity of the work.

On-premise AI deployment is usually the right call when one or more of these are true:

• Counsel or a regulator would treat the prompt as a record you are disclosing to a vendor.
• You cannot obtain a BAA, GLBA-grade vendor controls, or a no-training clause you actually believe.
• The workload has to survive a model-provider incident.
• Retrieval is over a private corpus (policies, charts, matter files) that should never be copied into a third-party context window.
• Token volume is high enough that per-request fees dominate, after you have updated the rates.

Hybrid is common. Many SMBs we work with keep a small local model for private retrieval and drafting, and they keep a cloud API for tasks that never touch client data. The mistake is using one architecture for both.

If you need a knowledge assistant that searches your own documents, our product Thea is built for internal search with source citations, and clients on the Enterprise plan can request on-premises deployment. Thea does not train on customer data. For broader process work (support routing, document intake, ops automation), start with AI solutions and decide the runtime (API vs. on-LAN) after you map the data, not before.

Practical takeaways

• Treat every prompt that contains client, patient, or customer data as a disclosure to whoever runs the model. If that party is a public API, you need a contract that matches the disclosure, or you need a different runtime.
• A BAA, a no-training clause, or a SOC 2 report is evidence in a risk analysis. It is not a substitute for knowing where the bits go.
• HIPAA, GLBA, and bank third-party guidance all leave you on the hook. Hiring a model vendor does not move the duty off your books.[6][7][9]
• On-premise AI deployment is a custody architecture: inference and retrieval on your network, with no external LLM required for the private path.
• Cost math has to use current token prices. Stanford's 2025 Index showed API inference getting dramatically cheaper through October 2024; hardware got cheaper and more efficient too. Run both sides of the ledger.[10]
• Split workloads. Public tasks can stay on an API. Private corpora should not.
• We do not certify you for HIPAA or SOC 2. On-prem helps you meet obligations you already have. Certification, if you need it, is a separate audit.

How we can help

We design and deploy on-premise AI for small and mid-sized firms that actually have a custody problem: legal, healthcare, and financial teams, not hobbyist GPU racks and not Fortune-50 platform groups. That work covers requirements, hardware specs, throughput estimates, network layout, model choice, install, and handoff so your staff can run the box. If a cloud API is the better fit, we will say so. Have more questions or want to get in touch? Start with our on-premise deployment overview, or contact our team and tell us what data you cannot send off-network.

Citations

1. Federal Trade Commission, Office of Technology, "AI Companies: Uphold Your Privacy and Confidentiality Commitments" (2024-01-09)
2. National Institute of Standards and Technology, "AI Risk Management Framework" (AI RMF 1.0 released 2023-01-26; NIST AI 600-1 Generative AI Profile released 2024-07-26)
3. U.S. Department of Health and Human Services, Office for Civil Rights, "May a HIPAA covered entity or business associate use a cloud service to store or process ePHI?" (content last reviewed 2023-01-09)
4. NIST Trustworthy and Responsible AI Resource Center, "3 AI Risks and Trustworthiness" (excerpt from AI RMF 1.0, 2023)
5. Cybersecurity and Infrastructure Security Agency, "Cloud Security Technical Reference Architecture" Version 2 (2022-06-21; page listed as archived content)
6. Federal Deposit Insurance Corporation, "Interagency Guidance on Third-Party Relationships: Risk Management" (FIL-29-2023, 2023-06-06)
7. Legal Information Institute, Cornell Law School, "45 CFR § 164.308 - Administrative safeguards" (HIPAA Security Rule)
8. Office of the Comptroller of the Currency, "Third-Party Relationships: Interagency Guidance on Risk Management" (OCC Bulletin 2023-17, 2023-06-06)
9. Legal Information Institute, Cornell Law School, "16 CFR § 314.3 - Standards for safeguarding customer information" (FTC Safeguards Rule, as amended 2021-12-09)
10. Stanford Institute for Human-Centered Artificial Intelligence, "Artificial Intelligence Index Report 2025, Chapter 1: Research and Development" (2025)