AI Chips and GPUs Explained: The Hardware Behind the Hype

Modern AI runs on specialised processors. Understanding them explains costs, shortages and why some models run on your laptop while others need a data centre.

• CPU: a flexible, general-purpose processor. Good at varied tasks, fewer parallel operations.
• GPU: originally for graphics, it performs thousands of calculations in parallel, ideal for the matrix maths in AI.
• AI accelerators: chips designed specifically for AI workloads (often called NPUs, TPUs or similar names).

Neural networks multiply huge grids of numbers. Doing many of these multiplications at once is exactly what GPUs and accelerators excel at.

• Training vs inference: training builds the model (very demanding). Inference runs it to produce answers (lighter, but at massive scale).
• VRAM / memory: the model must fit in memory. More memory means larger models.
• Memory bandwidth: how fast data moves, often a bottleneck.
• Quantisation: compressing a model to use less memory with some loss of precision.
• NPU: a neural processing unit in many new laptops and phones for on-device AI.

• Local AI: a graphics card with plenty of memory lets you run bigger models at home.
• "AI PC" and phone features: small on-device models for tasks like transcription and photo editing, which can improve privacy and speed.
• Cloud costs: access to powerful chips drives pricing.

• Supply and cost: demand for top chips has been intense.
• Energy and cooling for data centres.
• Geopolitics and export controls on advanced chips.
• Concentration: a small number of companies build the most advanced chips.

1. For local AI, prioritise GPU memory over raw speed.
2. Check software support for your hardware.
3. Don't overbuy. Small models run well on modest machines.

AI is as much about hardware as software. Memory and parallel compute determine what's practical, on your desk or in the cloud.