Contents
Introduction
Once an organisation has chosen its architecture and language model, it must decide where the platform will run. Hardware procurement should not be the first design decision. The platform must be sized from business use cases, security constraints, and an observed or forecast workload.
Why it matters
An undersized system creates slow responses and weak user trust. An oversized one raises total cost of ownership without producing business value. The architect's task is to meet today's demand while retaining a path to grow without rebuilding the platform.
Core explanation
Start with the workload profile: concurrent users, model size, knowledge-base volume, response-time targets, expected growth, and availability requirements. GPU capacity is only one output of that assessment.
flowchart TD
business[Business workload] --> requirements[Architecture requirements]
requirements --> model[Model selection]
requirements --> sizing[Infrastructure sizing]
sizing --> compute[CPU and GPU]
sizing --> memory[Memory]
sizing --> storage[Storage]
sizing --> network[Network]
In most enterprises, model serving, retrieval, databases, and observability should be separated. Each can then scale independently, and failure in one component is less likely to stop the entire service. A pilot may fit on one server, but the target architecture should support additional nodes.
Security
| Requirement | Practical effect |
|---|---|
| Protected environment | Enterprise data stays in an approved boundary |
| Network segmentation | Services receive only the access they need |
| Backups | Critical data remains recoverable |
| Hardware monitoring | Failures are detected before service disruption |
| Controlled updates | Infrastructure changes remain auditable |
Security belongs in the infrastructure design from the beginning, not after equipment has been acquired.
Enterprise example
An industrial group begins with a pilot for several dozen users. Instead of buying the largest possible machine, it separates model serving, knowledge retrieval and storage, and central monitoring. When the pilot demonstrates value, the team adds compute nodes without redesigning the whole platform.
Industrial safety example
Safety specialists work with project documents, inspection archives, and regulations. They need reliable access during busy periods as much as they need good answers. The design therefore includes redundancy for critical components, regular backups, and continuous equipment monitoring.
AI helps the expert retrieve and compare evidence; it does not replace the professional accountable for a safety-critical decision.
Common mistakes
- Buying hardware before estimating workload.
- Designing only for the first pilot.
- Treating redundancy and storage as secondary concerns.
- Assuming GPU is the only component that matters.
- Ignoring networking, monitoring, and operations.
Practical conclusions
Before procurement, prepare a workload forecast, response-time and availability targets, a scaling plan, a redundancy design, and a total-cost-of-ownership estimate. These artefacts make infrastructure selection an engineering decision rather than a response to vendor specifications.
Key takeaways
- Infrastructure follows business workload and architecture.
- GPUs are important, but not sufficient by themselves.
- Separated services improve resilience and make growth easier.
- Security and redundancy must be designed upfront.
- Incremental scaling is usually better than maximum capacity on day one.