What you need to run your own AI in-house

Running AI on your own hardware gives you control over your data and your costs. It also means the parts a cloud provider normally hides become yours to think about. This checklist covers what you need, in the order most teams work through it.

1. A clear first job

Decide what the system has to do first: answer questions over your documents, fine-tune a model on your data, serve a model to an app, or run inside a device. The job decides almost everything below.

2. Compute

• GPU memory sets which models you can run. Check it before anything else.
• The form it comes in. A complete desktop system, a GPU for a workstation, a card for a server, or a small computer for a device.
• Room to grow. Whether you can add a second card or a second unit later.

See Will it run? and Which GPU do I need for fine-tuning?

3. Something to put it in

• A workstation or server with a power supply that can carry the card.
• The right slot, enough physical space and enough airflow.
• Server cards need a server. They rely on its fans and won't cool themselves in a desktop.

4. Storage

• Fast local storage for the models and data in use.
• Shared storage for datasets and checkpoints if more than one person or machine needs them.
• A backup that doesn't depend on someone remembering.

5. Power and cooling

• A circuit that can supply the total draw. High-power GPUs add up quickly.
• A UPS, so a power cut doesn't end a long run or damage hardware.
• Somewhere the heat and fan noise can go.

6. Software

• An operating system and GPU drivers.
• A way to run models, and the models themselves. Open models each have their own licence, so check it suits your use.
• For production, something to serve the model and keep it running.

Our Resources page links to the official guides and downloads for each of these.

7. Someone to look after it

Hardware you own needs an owner. Someone has to install updates, watch disk space, replace a failed drive and restart things when they stop. For a single desktop system that is an hour here and there. For a rack it is part of someone's job.

8. A realistic budget

Add the workstation or server, storage, drives, a UPS and electricity to the price of the GPU. Then compare the total with what you would spend renting the same capacity. See Buying hardware vs renting cloud GPUs.

Where to start

If you are early, start small: one desktop system or one workstation GPU, and learn what you need from using it. Our solutions group the hardware by job, and a free consultation will help you work out which fits.