Whetstone (benchmark)

The Whetstone benchmark is a synthetic benchmark for evaluating the performance of computers. It was first written in ALGOL 60 in 1972 at the Technical Support Unit of the Department of Trade and Industry (later part of the Central Computer and Telecommunications Agency) in the United Kingdom. It was derived from statistics on program behaviour gathered on the KDF9 computer at NPL National Physical Laboratory, using a modified version of its Whetstone ALGOL 60 compiler. The workload on the machine was represented as a set of frequencies of execution of the 124 instructions of the Whetstone Code. The Whetstone Compiler was built at the Atomic Power Division of the English Electric Company in Whetstone, Leicestershire, England, hence its name. Dr. B.A. Wichman at NPL produced a set of 42 simple ALGOL 60 statements, which in a suitable combination matched the execution statistics.

To make a more practical benchmark Harold Curnow of TSU wrote a program incorporating the 42 statements. This program worked in its ALGOL 60 version, but when translated into FORTRAN it was not executed correctly by the IBM optimizing compiler. Calculations whose results were not output were omitted. He then produced a set of program fragments which were more like real code and which collectively matched the original 124 Whetstone instructions. Timing this program gave a measure of the machine’s speed in thousands of Whetstone instructions per second (kWIPS). The Fortran version became the first general purpose benchmark that set industry standards of computer system performance. Further development was carried out by Roy Longbottom, also of TSU/CCTA, who became the official design authority.

The Algol 60 program ran under the Whetstone compiler in July 2010, for the first time since the last KDF9 was shut down in 1980, but now executed by a KDF9 emulator..

Benchmark Content and Enhancements

The benchmark employs 8 test procedures, with three executing standard floating point calculations, two with such as COS or EXP functions, one each for integer arithmetic, branching or memory assignments. Output from the original comprised parameters used for each test, numeric results produced and the overall KWIPS performance rating. In 1978, the program was updated to log running time of each of the tests, allowing MFLOPS (Millions of Floating Point Operations Per Second) to be included in reports, along with an estimation of Integer MIPS (Millions of Instructions Per Second). In 1987, MFLOPS calculations were included in the log for the three appropriate tests and MOPS (Millions of Operations Per Second) for the others. Code changes were also carried out, including by by Bangor University, necessary to identify unexpected behaviour, without changing the implementation of the original 124 Whetstone instructions. One necessary change was to maintain measurement accuracy at increasing CPU speeds, with self calibration to run for a noticeable finite time, typically set for 10 seconds or 100 for early PCs with low timer resolution.

Note that there are other versions of the Whetstone Benchmark available online, some claiming copyright, without reference to CCTA or the design authority.

Initial CCTA Results

In conjunction with the undertaking controlled by the Contracts Division, CCTA engineers had responsibility to design and supervise acceptance trials of all UK Government computers and those for centrally funded for Universities and Research Councils, with systems varying from minicomputers to supercomputers. This provided the opportunity to gather verified Whetstone Benchmark results. Other results were obtained via new computer system appraisal activities.

CCTA records are now available in The UK National Archives , including technical reports. Original Whetstone Benchmark results are in the 1985 CCTA Technical Memorandum 1182, where overall speed is only shown as MWIPS. This contains more than 1000 results for 244 computers from 32 manufacturers, including the first for PCs and Supercomputers. The report might well be accessible from the Archive. The details were later included in a publicly available report (see Available Reports below).

Vector Processing Version

Roy Longbottom converted the original Whetstone Benchmark to fully exploit capabilities of the new vector processors. Results were included in the paper “Performance of Multi-User Supercomputing Facilities” presented in the 1989 Fourth International Conference on Supercomputing, Santa Clara .

This was also repeated in the Harold Curnow paper “Whither Whetstone? The synthetic benchmark after 15 years” presented at the “Evaluating supercomputers: strategies for exploiting, evaluating and benchmarking computers with advanced architecture” conference in 1990, in book .

Whetstone Benchmark Influences

Harold also reported comments from the 1989 conference “Software for Parallel Computers” in a presentation by Gordon Bell, designer of the Digital Equipment Corporation VAX range of minicomputers, indicating that the range was designed to perform well on the Whetstone Benchmark.

The Whetstone Benchmark also had high visibility concerning floating point performance of Intel CPUs and PCs, starting with the 1980 Intel 8087 coprocessor. This was reported in the 1986 Intel Application Report “High Speed Numerics with the 80186/80188 and 8087” . The latter includes hardware functions for exponential, logarithmic or trigonometric calculations, as used in two of the eight Whetstone Benchmark tests, where these can dominate running time. Only two other benchmarks were included in the Intel procedures, showing huge gains over the earlier software based routines on all three programs.

Later tests, by a SSEMC Laboratory, evaluated Intel 80486 compatible CPU chips using their Universal Chip Analyzer . Considering two floating point benchmarks, as used by Intel in the above report, they preferred Whetstone, stating “Whetstone utilizes the complete set of instructions available on early x87 FPUs”. This might suggest that the Whetstone Benchmark influenced the hardware instruction set.

By the 1990s the Whetstone Benchmark and results had become relatively popular. A notable quotation in 1985 was in “A portable seismic computing benchmark” quoting "The only commonly used benchmark to my knowledge is the venerable Whetstone benchmark, designed many years ago to test floating point operations" in the European Association of Geoscientists and Engineers Journal .

Details of the Vector Whetstone Benchmark performance were also repeated, by Roy Longbottom, at the June 1990 Advanced Computing Seminar at Natural Environment Research Council Wallingford. This lead to Council for the Central Laboratory of the Research Councils Distributed Computing Support collecting results from running “on a variety of machines, including vector supercomputers, minisupers, super-workstations and workstations, together with that obtained on a number of vector CPUs and on single nodes of various MPP machines “. More than 200 results are included, up to 2006, in the report available on the Wayback Machine Archive in entries to at least the year 2007 section . The report also indicated “The wide variety of standard functions exercised (sqrt, exp, cos etc.) consume a far larger fraction of the reported times.”.

The First 1 MIPS Minicomputer and Dhrystone Benchmark

On achieving 1 MWIPS, the Digital Equipment Corporation VAX-11/780 minicomputer became accepted as the first commercially available 32-bit computer to demonstrate 1 MIPS (Millions of Instructions Per Second), CERN , not really appropriate for a benchmark dependent on floating point speed. This had an impact on the Dhrystone Benchmark, the second accepted general purpose computer performance measurement program, with no floating point calculations. This produced a result of 1757 Dhrystones Per Second on the VAX 11/780, leading to a revised measurement of 1 DMIPS, (AKA Vax MIPS), by dividing the original result by 1757.

Available Reports

Source code and pre-compiled versions for PCs in C/C++, Basic, Visual Basic, Fortran and Java are available.

The Whetstone benchmark primarily measures the floating-point arithmetic performance. A similar benchmark for integer and string operations is the Dhrystone.

See also

• Dhrystone
• FLOPS
• Gibson Mix
• LINPACK benchmarks
• Million instructions per second (MIPS)

External links

• Benchmark Programs and Reports (see also Netlib)
• Whetstone Algol Revisited, or Confessions of a compiler writer PDF file (B. Randell, 1964)