Intel Performance Testing Lab Reveals Methodology

This video features an interview with Hannah Colett, Development Tools & Software Engineering Manager at Intel, who discusses the company's internal CPU performance testing methodology.

Key aspects of Intel's CPU testing process:

Goal of the Performance Lab (1:00): Hannah explains that the primary goal is to ensure Intel chips are useful, function as expected, and provide a good user experience. She emphasizes generating accurate, consistent, and reliable data that represents how products are actually used, rather than just making them look good on bar charts (2:25).
Picking Benchmarks (4:50): The selection of benchmarks focuses on relevancy, ensuring that a benchmark measures what it's intended to measure. The team aims for "breadth, not just depth," looking for measurements that flex different parts of the computer (e.g., memory, speed, RAM, power, GPU, NPU) to tell a comprehensive story about each component (6:08). Unnecessary tests are swept aside to focus on core fundamentals.
Early Testing (6:50): Testing begins as early as possible, typically six to eight months before a product's launch. This allows for iterative testing with raw products, without the newest BIOS or drivers, to track how the product evolves and performs over time and to identify and fix issues early on.
Testing Methods (10:13): While other labs at Intel might conduct highly granular tests (e.g., different RAM timings, PCIe slot configurations), Hannah's team focuses on end-product testing. This includes testing with four different types of memory, various memory manufacturers, different GPUs, and tweaking BIOS settings, especially for gaming to ensure the right user experience.
Competitive Analysis (11:32): When testing against competitors, Intel aims to create a realistic and fair scenario by giving competitors the best possible setup, mirroring how competitors would recommend their products be used. This ensures an "apples to apples" comparison, not necessarily meaning exact duplication, but rather a fair and realistic representation.
Product Readiness (12:24): A product is generally considered ready to ship ("Product Validation" or PV) about a month before release. At this stage, extensive and thorough testing is conducted, including competitive analysis, to gather the true story of the product's performance.
Addressing Problems (14:34): If a processor isn't performing up to expectations, Intel prioritizes honesty and continual improvement (Kaizen). In cases like the Raptor Lake issue, where the data didn't initially show the problem, the team learned from it and adjusted their methodology for future launches to prevent similar pitfalls (15:36). Hannah also personally tests the user experience from a fresh build to identify roadblocks.
Lawyer Vetting of Benchmarks (17:10): All public numbers and marketing claims are vetted by lawyers to ensure accuracy and fairness to customers. Every data point can be traced back to a specific test plan, devices, and instruction sets, with thorough logging of game versions, test dates, and personnel. The focus is on robust data, including medians, standard deviations, power traces, and thermals, to ensure the data is correct regardless of whether Intel "wins or loses."
The Art of Benchmarking (18:56): Hannah emphasizes that benchmarking is not just a science but also an art. There are many ways to achieve a goal, and the "how you test matters." Ultimately, the goal is to understand how users will experience the device off the shelf, beyond just synthetic scores.