What is TOPS? The AI Performance Metric Explained
The first Copilot Plus PCs will be powered by the 45 TOPS in their Qualcomm Snapdragon X chips. New AMD AI 300 chips promise 50 TOPS for future laptops, and Intel says its upcoming Lunar Lake chips will offer 48 TOPS to new PCs in the near future. In fact, to be considered part of the next-gen era of AI PCs, there's a 40 TOPS requirement for your Neural Processing Unit (NPU).
But it begs the question: what exactly are TOPS, and do they really matter?
What are TOPS?
TOPS is an acronym for Tera Operations Per Second, but is sometimes alternatively described as Trillions of Operations Per Second. Simply put, a TOPS figure describes how many trillions of operations per second an NPU can handle. 40 TOPS means 40 trillion operations per second.
Ok—but what’s an operation? Matrix multiplication is a common example found across a variety of neural networks. To put it simply, many neural networks generate results by multiplying inputs and weights, which can involve tens or hundreds of billions of multiplication calculations. It’s a bit like multiplying the numbers in two Excel spreadsheets, except each spreadsheet is massive beyond imagining.
With this in mind, the meaning of TOPS becomes a bit easier to understand. TOPS is a measure of how quickly a processor can breeze through these specific calculations, and a higher figure is better.
How NPUs add TOPS
However, there’s a gap in this explanation large enough to drive a truck through. If TOPS measures “operations,” and the operations are math, what’s so special about TOPS? Why do we need a new metric? Or an NPU, for that matter?
The answer again comes down to matrix multiplications in neural networks. These calculations aren’t too complex individually, but happen at a vast scale with as much parallelization as possible. NPUs meet this task with lots of specialized hardware.
Take a peak at Intel’s NPU Acceleration Library documentation, for example, and you’ll see a diagram that shows each Neural Compute Engine has an Inference Pipeline that includes a MAC Array. That array is composed of hundreds of individual Multiple-Accumulate Units (MACs).
That’s different from a CPU’s architecture. The x86 instruction set has thousands of defined instructions, each of which enable different calculations. Modern CPUs can handle sorts of mathematical tasks, but that means each CPU core is more complex and takes up more physical space, so fewer cores can fit. A diagram of a CPU looks like a collection of big blocks, while a diagram of an NPU looks like clusters of tiny blocks strung together.
That’s why TOPS is relevant to an NPU, but not a CPU. A CPU is designed to handle many different types of calculations, not a few types of calculations en-mass.
How much does TOPS matter?
TOPS is a measure of AI performance that summarizes a complex topic in a single, easy-to-understand number. That makes it a useful bit of shorthand, so Intel, AMD, and others are talking TOPS while promoting new processors that pair an NPU with a CPU and integrated GPU.
But TOPS isn't the end-all of specs for AI. The technology industry hasn’t settled on a firm definition for how TOPS should be measured and, frankly, it probably never will. Different companies can define operations differently, or calculate with different levels of precision, which sways the results.
There’s also a lot of messy details in quoting AI performance, like batch size, which can further influence the results. Optimizations in an AI model for a particular piece of hardware will affect results even more, adding another variable.
Hardcore PC enthusiasts are already familiar with this problem. CPUs quote a clock speed in gigahertz (GHz), but most enthusiasts know you can’t just look at a CPU’s clock speed to decide which is best. Gamers, meanwhile, are familiar with looking at Tera Floating-Point Operations Per Second (TFLOPS), which most video cards and game consoles quote. TFLOPs can be useful when comparing two video cards in the same line-up, like an RTX 4060 vs an RTX 4080, but it’s otherwise it can be apples to oranges.
This isn’t just a matter of performance—though that is relevant—but also a matter of memory. X.ai’s Grok-1, which has 314 billion parameters, is the largest AI “open-source” large language model. In theory, anyone can download it and run it (after fine-tuning the model, which is its own challenge). But good luck running Grok-1 without an Nvidia H100 cluster in your closet. It consumes hundreds and hundreds of gigabytes of memory.
That leaves local NPUs and GPUs to pick up specialized AI tasks. They can still be useful, but they won’t wholly define a PC’s AI performance. A PC with an NPU might perform better when upscaling video in Topaz Video Enhance AI, for example, but that doesn’t make cloud-based tools like ChatGPT any quicker.
At the end of the day, your best is to stick to a tried-and-true rule of buying PC hardware. First decide the sort of software and tasks you need or want to run, then look for components with real-world benchmarks specific to them.
Read more: AI Tools and Tips
- Microsoft Launches a New Era of Copilot Plus PCs
- Hands-on with the Faster, Smarter ChatGPT-4o AI
- Qualcomm Intros AI-Ready Snapdragon X Plus Chip
- Why Coders are Learning to Love Copilot
- Roll Your Own GPT: Setting Up Your Computer for Local AI
- How to Get NVIDIA Chat with RTX: Local AI for Everyone
- How to Make Sure Your Next Computer Is AI Ready
Matthew S. Smith is a prolific tech journalist, critic, product reviewer, and influencer from Portland, Oregon. Over 16 years covering tech he has reviewed thousands of PC laptops, desktops, monitors, and other consumer gadgets. Matthew also hosts Computer Gaming Yesterday, a YouTube channel dedicated to retro PC gaming, and covers the latest artificial intelligence research for IEEE Spectrum.
Read more: PC laptop and desktop resources
- See top new laptops at Micro Center
- See top new desktops at Micro Center
- See top new GPUs at Micro Center
- See top new CPUs at Micro Center
- Start building a new PC with Micro Center's PC builder
