BenchmarkRunner.java - Benchmark Runner Program

This section provides a tutorial example on how to write a benchmark runner program that takes care of JVM warming up, JIT compilation, measuring time, and other non-test related tasks.

The best way to write a micro benchmark program is probably to separate it in 2 parts:

Here my test runner program, BenchmarkRunner.java:

/**
 * BenchmarkRunner.java
 * Copyright (c) 2010, HerongYang.com, All Rights Reserved.
 */
class BenchmarkRunner {
   static java.io.PrintStream out = System.out;
   static java.io.InputStream in = System.in;
   long startTime = 0;
   long endTime = 0;
   long[] timeRecords = null;

   public static void main(String[] a) {
      if (a.length<5) {
        out.println("Missing arguments. Usage: ");
        out.println(
           "BenchmarkRunner class method warmups runs steps");
        return;
      }
      try {
      	 String className = a[0];
      	 String methodName = a[1];
      	 int numberOfWarmups = Integer.parseInt(a[2]);
      	 int numberOfRuns = Integer.parseInt(a[3]);
      	 int numberOfSteps = Integer.parseInt(a[4]);
     
         // Warming up the JVM
         out.println("Are you ready?");
         in.read(new byte[1]);

         // Loading the benchmark class and method
     	 Class testClass = Class.forName(className);
     	 java.lang.reflect.Method testMethod 
     	    = testClass.getMethod(methodName, int.class,
     	    BenchmarkRunner.class);
         BenchmarkRunner testRunner
            = new BenchmarkRunner(numberOfRuns);
         Object testObject = testClass.newInstance();

         // JIT warmup
         out.println();
         out.println("Waking up the JIT compiler...");
         for (int i=0; i<numberOfWarmups; i++) {
            Object testResult = 
            testMethod.invoke(testObject, numberOfSteps, testRunner);
            out.println("Run: "+i+", Time: "+testRunner.returnTime()
               +", Test returns: "+testResult);
         }

         // Benchmark runs
         out.println();
         out.println("Starting benchmark test runs...");
         for (int i=0; i<numberOfRuns; i++) {
            Object testResult = 
            testMethod.invoke(testObject, numberOfSteps, testRunner);
            testRunner.recordTime(i);
            out.println("Run: "+i+", Time: "+testRunner.returnTime()
               +", Test returns: "+testResult);
         }

         // Benchmark report
         out.println();
         out.println("Benchmark test time report...");
         testRunner.report(numberOfRuns, numberOfSteps);

      } catch (Exception e) {
         e.printStackTrace();
      }

   }

   // Constructor
   public BenchmarkRunner(int runs) {
      timeRecords = new long[runs];
   }

   // Starting the timer - to be called by test method
   public void startTimer() {
      startTime = System.nanoTime();
   }

   // Stopping the timer - to be called by test method
   public void stopTimer() {
      endTime = System.nanoTime();
   }

   // Returning time from the timer
   public long returnTime() {
      return endTime - startTime;
   }

   // Recording time from the timer
   public void recordTime(int i) {
      timeRecords[i] = endTime - startTime;
   }

   // Reportting benchmark result
   public void report(int runs, int steps) {
      long total = 0;
      long minimum = Long.MAX_VALUE;
      long maximum = 0;
      for (int i=0; i<runs; i++) {
      	 long t = timeRecords[i];
      	 total += t;
      	 if (t>maximum) maximum = t;
      	 if (t<minimum) minimum = t;
      }
      long average = total/runs;
      out.println("Runs: "+runs+", Ave: "+average/steps
         +", Min: "+minimum/steps
         +", Max: "+maximum/steps
         +" - Per step in nanoseconds");
      out.println("Runs: "+runs+", Ave: "+average
         +", Min: "+minimum+", Max: "+maximum
         +" - All steps in nanoseconds");
      out.println("Runs: "+runs+", Ave: "+average/1000000
         +", Min: "+minimum/1000000
         +", Max: "+maximum/1000000
         +" - All steps in milliseconds");
      out.println("Runs: "+runs+", Ave: "+average/1000000000
         +", Min: "+minimum/1000000000
         +", Max: "+maximum/1000000000
         +" - All steps in seconds");
   }
   
   // Constructor needed as a sample benchmark test class
   public BenchmarkRunner() {
   }

   // A sample benchmark test method
   public static long sampleTest(int steps, BenchmarkRunner runner) {
      long total = 0;
      runner.startTimer();
      for (int i=0; i<steps; i++) {
         total += i;
      }
      runner.stopTimer();
      return total;
   }
}

Note that:

Last update: 2010.

Table of Contents

 About This Book

 Downloading and Installing JDK 1.8.0 on Windows

 Downloading and Installing JDK 1.7.0 on Windows

 java.lang.Runtime Class - The JVM Instance

 java.lang.System Class - The Operating System

 ClassLoader Class - Class Loaders

 Class Class - Class Reflections

 Sun's JVM - Java HotSpot VM

 JRockit JVM 28.2.7 by Oracle Corporation

 JVM Runtime Data Areas

 Memory Management and Garbage Collectors

 Garbage Collection Tests

 JVM Stack, Frame and Stack Overflow

 Thread Testing Program and Result

 CPU Impact of Multi-Thread Applications

 I/O Impact of Multi-Thread Applications

 CDS (Class Data Sharing)

Micro Benchmark Runner and JVM Options

 What Is Micro Benchmark?

BenchmarkRunner.java - Benchmark Runner Program

 emptyLoop() - The Empty Loop Test Method

 "-XX:+PrintCompilation" - Watching JIT Compilation Logs

 "-XX:+PrintGC" - Watching GC (Garbage Collection) Logs

 "-Xms" and "-Xmx" - Avoiding GC with Large Memory Size

 Benchmark Affected by Other Running Applications

 "-Xint" - Running in Interpreted-Only Mode

 Micro Benchmark Tests on "int" Operations

 Micro Benchmark Tests on "long" Operations

 Micro Benchmark Tests in JIT Compilation Mode

 Micro Benchmark Tests on "float" and "double" Operations

 Outdated Tutorials

 References

 PDF Printing Version