The Art of (Java) Benchmarking

People write toy Java benchmarks all the time. Nearly always they "get it wrong" -- wrong in the sense that the code they write doesn't measure what they think it does. Oh, it measures something all right -- just not what they want. This session presents some common benchmarking pitfalls, demonstrating pieces of real, bad (and usually really bad) benchmarks. The session is for any programmer who has tried to benchmark anything. It provides specific advice on how to benchmark, stumbling blocks to look out for, and real-world examples of how well-known benchmarks fail to actually measure what they intended to measure.

People write toy Java benchmarks all the time. Nearly always they "get it wrong" -- wrong in the sense that the code they write doesn't measure what they think it does. Oh, it measures something all right -- just not what they want. This session presents some common benchmarking pitfalls, demonstrating pieces of real, bad (and usually really bad) benchmarks such as the following: SpecJVM98 209_db isn't a DB test; it's a bad string-sort test and indirectly a measure of the size of your TLBs and caches. SpecJAppServer2004 is a test of your DB and network speed, not your JVM. SpecJBB2000 isn't a middleware test; it's a perfect young-gen-only garbage collection test. The session goes through some of the steps any programmer would go through to make a canned program run fast -- that is, it shows you how benchmarks get "spammed."

The session is for any programmer who has tried to benchmark anything. It provides specific advice on how to benchmark, stumbling blocks to look out for, and real-world examples of how well-known benchmarks fail to actually measure what they intended to measure.

About Cliff Click

With more than twenty-five years experience developing compilers, Cliff serves as Azul Systems' Chief JVM Architect. Cliff joined Azul in 2002 from Sun Microsystems where he was the architect and lead developer of the HotSpot Server Compiler, a technology that has delivered dramatic improvements in Java performance since its inception.

Previously he was with Motorola where he helped deliver industry leading SpecInt2000 scores on PowerPC chips, and before that he researched compiler technology at HP Labs. Cliff has been writing optimizing compilers and JITs for over 20 years. He is invited to speak regularly at industry and academic conferences including JavaOne, ECOOP, JVM and VEE; serves on the Program Committee of many conferences (including PLDI and OOPSLA); and has published many papers about HotSpot technology and more than a dozen related patents. Cliff holds a PhD in Computer Science from Rice University.

More About Cliff »