Softude
Self-hosting AI agents is useful when workloads are large and predictable enough to maintain high GPU utilization, data governance requires direct infrastructure control, and your engineering team can support production-grade model serving. For workloads below several billion tokens per month, hosted AI agents or hybrid deployment is more cost-effective and operationally efficient.
Key highlights
- Before buying any GPUs, run your agents’ real token usage against the break-even point. Below several billion tokens a month, hosted APIs usually cost less.
- You can run the agent and its tools in-house while still calling an external model, which gives you control without taking on GPU operations.
- Self-hosting only saves money when your GPUs stay busy.
- Keeping data in-house doesn’t make you compliant. Your controls do.
- For most fintech teams, the right answer is a mix of hosted and self-hosted.
What are self-hosted AI agents and how do they work?
Self-hosted or local AI agents are agents whose orchestration, tools, and integrations run on infrastructure your organization controls: on-premises servers, a private cloud, or a dedicated environment in a public cloud. The model the agent calls can run inside that environment too, or it can sit behind an external API.
A production agent has five main parts:
- Orchestration and runtime: plans steps, calls tools, manages memory and retries.
- Model serving: an inference server such as vLLM running an open-weight model on GPUs, if you host the model.
- Data and retrieval: document stores, vector databases, and links to core banking or payments systems.
- Tool integrations: actions such as pulling a transaction history or opening a fraud case.
- Monitoring and security: traces, audit logs, identity and access management.
The orchestration layer shapes cost and risk more than most teams expect. To see how it coordinates tools, models, and data, explore how AI agent orchestration works in enterprise systems.
Self-hosting an AI agent vs. self-hosting its LLM: what does it mean
Self-hosting the agent and self-hosting the LLM are separate decisions. You can run orchestration, data access, and tool integrations privately while sending model calls to an external API. Where you draw that line changes your cost structure, your data flow, and how much infrastructure you own.
| Deployment model | Agent environment | Model environment | What leaves your environment |
| Fully hosted | Provider infrastructure | Provider infrastructure | Prompts, retrieved records, tool results and outputs |
| Self-hosted agent with external LLM | Your infrastructure | External model API | Only what you include in model calls |
| Fully self-hosted | Your infrastructure | Your infrastructure | Nothing, if network egress is locked down |
Many fintech teams start in the middle row, keeping credentials, tool permissions, and audit logs in-house without GPU operations.
Hosted vs. self-hosted AI agents: what are the key differences?
Hosted AI agents prioritize speed and ease of deployment, with the provider managing infrastructure, scaling, and model updates. Self-hosted agents offer greater control over data boundaries, model selection, and configuration, but require your team to handle provisioning, maintenance, and incident response. The cost advantage depends primarily on workload volume.
| Decision factor | Hosted AI agents | Self-hosted AI agents |
| Data control | Depends on provider infrastructure and data policies | More control over infrastructure, data boundaries, and access |
| Initial investment | Low setup cost | Infrastructure provisioning and engineering investment |
| Operating costs | Usage or token-based pricing | Compute, storage, networking, model serving, and engineering |
| Maintenance | Provider manages most platform infrastructure | You manage the deployment and its components |
| Model choice | Limited to supported models | Any compatible model, including open-weight models |
| Scalability | Provider-managed, within service limits | You plan capacity and add hardware |
| Vendor dependency | Tied to provider APIs and features | Less platform dependency; hardware and model dependencies remain |
Are self-hosted AI agents more secure than hosted AI agents?
Not automatically. Self-hosting AI agents keeps data inside your network, but security depends on access controls, encryption, logging, and patching. A well-governed hosted agent with strong data terms can be safer than a poorly configured self-hosted one.
What is the cost of running self-hosted AI agents for fintech?
Self-hosted AI agent costs are largely fixed, covering GPUs, supporting infrastructure, and engineering resources. Hosted costs are variable, billed per token processed. The more economical option depends on overall workload volume and GPU utilization, rather than the unit price per token.
Token prices also understate agent costs. A Forbes Finance Council article argues that banks should measure cost per governed outcome, such as a completed KYC review, because instructions, documents, tool results, and retries all consume tokens.
The strongest public evidence on break-even comes from the OECD’s May 2026 paper, Benefits of AI Openness. It compares privately hosting open-weight models with pay-as-you-go cloud APIs, counting GPU installation, electricity, connectivity, colocation, engineering support, insurance, and depreciation:
| Monthly token workload | Estimated private-hosting break-even |
| Less than 100 million tokens | No break-even under the modeled assumptions |
| 1 billion tokens | About 30 months |
| 10 billion tokens | About 2 months |
| 50 billion tokens | About 1 month |
These are model estimates under specific assumptions, not fintech benchmarks. The pattern matters: between 1 and 10 billion tokens a month, the economics shift sharply.
What infrastructure is required to run local AI agents?
Local AI agents need six infrastructure layers. Which ones you run depends on one decision: do you host only the agent, or the agent and its LLM too? Does the self-managed AI agent run on standard cloud or on-premises servers?
Here are all the layers of self-hosted AI agent infrastructure explained:
1. Agent runtime and orchestration
This is where the agent plans steps, calls tools, keeps memory, and handles retries. It runs on ordinary CPU servers or containers, usually managed with Kubernetes. It also needs a queue or workflow engine so long-running tasks, such as a KYC review, survive restarts.
2. Data and retrieval
On-premises AI agents need the right records at the right time. That means document storage, a vector database for search, and a cache for frequently used context. For fintech, it also means read access to core banking, payments, lending, or case management systems, limited to the data each agent actually needs.
3. Tool integrations and APIs
Every action an agent takes, such as pulling a transaction history or opening a fraud case, runs through an API. You need an API gateway, service accounts with scoped permissions, and rate limits so a looping agent can’t overload a core system.
4. Model serving (only if you host the LLM)
This layer needs GPUs, an inference server such as vLLM, and storage for model weights and versions. GPU needs grow with volume. The OECD’s 2026 break-even analysis gives a rough guide:
- Under 100 million tokens a month: 1 L4 GPU
- About 1 billion tokens a month: 1 H100
- About 10 billion tokens a month: 2 to 3 H100S
- About 50 billion tokens a month: 8 H100S
Add spare capacity for peaks and failover.
5. Security and network controls
These include:
- private networking and controls on what data can leave
- encryption in transit and at rest
- identity and access management for both people and agents
- secrets management for credentials
- data residency controls
6. Monitoring and audit
You need step-level traces of what the local AI agent did and why, logs of every tool call and model response, cost and token tracking per task, and alerts for failures. In financial services, these logs are also your evidence for auditors and model risk reviews.
The most important infrastructure for self-hosted AI agents people forget: the team. Running these layers needs platform engineering, security, and, if you host the model, MLOps skills on call around the clock. Without that team, the hardware alone won’t keep agents running reliably.
Softude brings together AI agent development, DevOps, and MLOps expertise to build and deploy on-premises AI agents within your existing technology ecosystem. We understand the security, governance, and compliance considerations involved in deploying AI solutions in financial services. Backed by ISO 27001 and CMMI Level 5 practices, our team helps you evaluate the right deployment options for private AI agents based on your workloads, infrastructure, and risk requirements.
Does self-hosting give us more control over our financial data?
Yes, over where the data travels, but control alone doesn’t make you compliant. In an agent workflow, data leaves your systems far more often than in a chatbot. Each model call can carry KYC documents, transaction histories, and tool outputs. Self-hosting keeps that traffic inside your own network.
The trade-off is that the controls become your job. That means private networking, encryption, scoped permissions for every agent tool, audit logs, and data residency rules. A self-hosted agent or privately managed AI agent with broad read access to your customer database carries more risk than a well-governed hosted one.
If you stay with a hosted LLM, get three things confirmed in writing:
- where your requests are processed
- how long the provider keeps your data
- whether your data can be used for training
Who maintains self-hosted AI agents, and what does it take?
Your team does, around the clock. With a hosted deployment, the provider keeps the model running, and you manage the agent. With AI agents running on your own infrastructure, your team also owns:
- GPU provisioning
- model updates
- security patches
- capacity planning
- monitoring
- incident response
That takes DevOps, MLOps, and security skills, plus someone on call.
Picture a reconciliation agent that processes overnight settlement files. If your inference server fails at 2 a.m., your engineer has to restore it before the morning cutoff, or finance starts the day with unmatched transactions. Hosted APIs have outages too, so plan a fallback either way. The real difference is who does the fixing.
Can we choose our own model for self-hosting AI agents?
Yes, but only among open-weight models. You can run any open-weight model your hardware supports, such as Llama or Qwen. You can pin a version, fine-tune it on your own data, and tune inference for your setup. Proprietary frontier models from providers such as OpenAI or Anthropic don’t release their weights, so you can only use them through an API. Self-hosting widens your control but narrows your model catalog.
That control matters in financial services. If your model risk team validates an agent, a hosted model that changes underneath it means validating again. A pinned self-hosted model stays exactly as tested until you decide to change it.
Open-weight models can also cost less. Research in the American Economic Association’s Journal of Economic Perspectives found that open-source models cost about 90% less than comparable closed-source models in the OpenRouter data studied. That compares API prices, not self-hosting costs.
You can buy open-weight inference from hosted providers without running GPUs, which is often the cheaper way to test a model before committing to host it.
When should fintech self-host AI agents
Self-host AI agents when your agents process billions of tokens a month, steadily enough to keep GPUs busy. Your data or policy rules need more control than hosted providers can offer. And you have engineers who can run model infrastructure around the clock. If any one of these is missing, hosted or hybrid deployment will usually cost less and carry less risk.
- Your workloads are large and predictable enough to justify infrastructure investment
Self-hosting becomes cost-effective when GPUs are consistently utilized. For example, a fraud-alert triage agent operating continuously will maximize hardware usage, while a month-end reporting agent will not. Both analyses above indicate that sustained workloads of several billion tokens per month are required.
- You need greater control over data processing and infrastructure
Some workloads can’t send customer records to a third-party API under your policies or contracts. Private deployment keeps that data inside boundaries you define, though regulators still assess the whole system, from access controls to human oversight.
- You need greater control over model deployment
Self-hosting fits when you need to fine-tune on proprietary transaction patterns, freeze a validated model version, or avoid one provider’s roadmap, especially where a model change triggers costly revalidation.
- You have the engineering capacity to operate AI infrastructure
Production model serving needs people skilled in GPU scheduling, inference optimization, observability, and security hardening who can respond to incidents at night. If that means new hires, include the cost and timeline.
- Your performance requirements justify deployment control
Deploying inference close to core systems can reduce network overhead and help maintain predictable throughput during peak periods. However, latency is determined by factors such as model size, GPU type, batching strategy, and network architecture, so benchmarking is essential before making deployment decisions.
When is hosted AI agent deployment the better choice?
Hosted deployment is preferable for small or unpredictable workloads, during early-stage experimentation, when rapid deployment is a priority, or when your team lacks the capacity to operate model infrastructure. It also provides access to proprietary models that are not available for self-hosting.
Consider a fintech startup testing an internal customer-support agent with uncertain usage. Per-token billing costs a few thousand dollars a month at most, while dedicated GPUs plus MLOps support would cost several times more and sit mostly idle. If the provider’s security terms meet your requirements, hosted is a sound long-term choice.
Should you consider a hybrid AI agent deployment?
Hybrid AI agent deployment runs the agent and sensitive data privately, routes approved model calls to external APIs, and keeps specific workloads on internally hosted open-weight models. For many fintech teams, it balances model capability, control, and cost better than either extreme.
A typical hybrid architecture:
- Agent orchestration runs in your private infrastructure.
- Business records and internal integrations stay in controlled environments.
- Approved, lower-sensitivity tasks call external LLM APIs.
- Sensitive or high-volume tasks use internally hosted open-weight models.
- Access policies and data controls govern every call between components.
Keep tasks with raw customer identifiers on the private model, and redact sensitive fields before any external call. Route by sensitivity first and complexity second, and log which model handled each step, because auditors will ask. The router becomes a security component, so test its rules.
How should a fintech CTO choose between hosted and self-hosted AI agents?
Choose based on total cost of ownership, data requirements, and engineering capacity. Measure token consumption per completed task, compare it with the break-even ranges above, then check whether your team can operate what the numbers favor.
| Evaluation question | If yes | If no |
| Do your agents generate sustained demand in the billions of tokens a month? | Evaluate self-hosting economics | Hosted is likely cheaper |
| Do your policies require specific infrastructure or data-processing controls? | Evaluate self-hosted or hybrid | Hosted may be sufficient |
| Can your team run GPU infrastructure and respond to incidents 24/7? | Self-hosting is feasible | Stay hosted or hybrid |
| Do you need control over model versions and serving? | Evaluate self-hosted models | Hosted options may be enough |
| Do you need to launch within weeks? | Start hosted or hybrid | Weigh the other factors |
- Choose hosted when speed, managed infrastructure, and lower operational load matter most.
- Choose self-hosted when sustained volume, infrastructure control, and model deployment needs justify owning the stack.
- Choose hybrid when some workloads need tighter control and external models still add value.
If you are still scoping the agent itself, compare AI agent use cases in financial services.
Choose AI agent infrastructure based on your workload and business requirements
Self-hosting is the right move when volume, control, and model requirements justify the extra investment and responsibility. Measuring cost per completed task on real workloads turns the debate into a spreadsheet decision.
Softude’s AI agent development team can help you model those numbers and design an agent architecture that fits your infrastructure, security requirements, and workflows.
FAQs
Can you self-host an AI agent without hosting its LLM?
Yes. You can run orchestration, tools, and data integrations in your environment and send model calls to an external API. This keeps system access in-house without GPU operations.
Are self-hosted AI agents better for fintech companies?
Only for some workloads. Fintechs with high-volume agents, strict data policies, and strong platform teams often benefit. Low-volume teams usually do better with hosted or hybrid deployment.
Can self-hosted AI agents use external LLM APIs?
Yes. A self-hosted agent can call any external LLM API you approve, usually with redaction and routing rules that keep sensitive data on an internally hosted model.
When should a company switch from hosted to self-hosted AI agents?
Switch when measured volume stays in the billions of tokens a month, projected self-hosted cost per completed task drops below hosted, and your team can operate the infrastructure. Data requirements hosted providers can’t meet are another valid trigger.
