Benchmarking Methodology

The complexity of today’s designs, combined with the wealth of available FPGA and computer-aided design (CAD) tool features, make benchmarking a difficult, time-consuming, and expensive task. A detailed understanding of the designs used in comparisons, and an intimate knowledge of FPGA features and CAD tools is necessary to extract meaningful benchmarking results. A poor benchmarking process can easily result in inconclusive—and even worse, incorrect—results.

Altera's benchmarking methodology has been endorsed by third-party industry experts. Two such experts are quoted below:

To evaluate its FPGAs, Altera has created a benchmarking methodology which fairly considers the intricacies and optimizations of its competitors' tools and devices, as well as its own. Experiments which consider a variety of end-user operating conditions have been run on a suite of industrial benchmark circuits. The results of these experiments have been analyzed to accentuate experimental variations and to clearly identify result trends. I am convinced that the experimental methodology that has been used fairly characterizes appropriate user expectations for Altera's devices in terms of area utilization and speed versus their competitors.

- Russell Tessier, Associate Professor at the University of Massachusetts, Amherst

Altera has created an impressive methodology for measuring and monitoring key performance characteristics of their and competitors’ devices and tools. Designing a fair and accurate test methodology that yields reasonable metrics is a daunting task, and Altera has clearly made a considerable investment that has paid off in this area. Their system includes a comprehensive and representative test suite, automatic compensation for technology and IP differences between various FPGA families, and automatic generation of design constraints to get optimal results from synthesis and layout tools.

- Kevin Morris, Editor of the FPGA and Programmable Logic Journal

The following factors affect benchmarking results:

  • Benchmark design selection
  • HDL optimization
  • Software tool settings and user constraints
  • Timing analysis techniques
  • Benchmark result reporting

The FPGA Performance Benchmark Methodology white paper (PDF) contains detailed discussion about the above factors.

Benchmark Design Selection

Altera uses real customer designs for benchmarking and does not generate artificial circuits or use intellectual property (IP) cores in isolation. By using a broad suite of customer designs from various market segments (such as telecommunications, networking, storage, wireless, and medical applications) Altera ensures that the logic and connectivity present in designs is real and can therefore be used to represent the complex interaction of large circuits and FPGA CAD tools.

HDL Optimization

The customer designs Altera uses are originally targeted for technologies such as Altera® and Xilinx FPGAs, ASICs, and gate arrays. Taking a design’s HDL code that is optimized for a particular technology and blindly benchmarking it for another technology produces very misleading results. Therefore, a team of Altera engineers is dedicated to converting each individual design to optimize its performance to its targeted architecture used in these benchmarks. For each design, the conversion and performance optimization process takes weeks to be completed so that each design can take full advantage of the dedicated features present in each architecture.

Software Settings and Constraints

All CAD tools make trade-offs between design performance in the areas of amount of logic used, compile time, and memory usage. The benchmarking results depend on the many settings for the tools used to compile an FPGA design. The outcome of benchmarking varies significantly with software settings and constraints applied.

The following must be considered when benchmarking:

  • Least-Effort Results. This involves very little user intervention for optimizing performance with the FPGA CAD tools such as Altera’s Quartus® II software or Xilinx’s ISE software. Least-effort results give an initial estimate on performance and resource usage.
  • Best-Effort Results. Best-effort results are based on significant user intervention to achieve the best possible results. Best-effort results typically require appropriate software settings and timing constraints for each design; they can also involve multiple compilations and iterations.

Figures 1 and 2 show how the average results swing, depending on whether the least- or best-effort settings were used in benchmarking activities.

Figure 1. Best-Effort Performance Comparison (Note 1)

Figure 2. Least-Effort Performance Comparison (Note 1)

Note to Figures 1 and 2:

1. Benchmarking data is based on comparing an Altera device to an equivalent Xilinx device using the latest available Quartus II and ISE software at the time of analysis.

Quartus II design software provides the design space explorer (DSE) feature that automatically determines the best settings for a particular design. Quartus II software also provides advanced algorithms for physical synthesis. The DSE and physical synthesis features offer easy push-button flows for obtaining large performance gain. Xilinx ISE software now also provides a similar tool called Xplorer

Benchmark Result Reporting

The choice of circuits used for benchmarking affects the results significantly. Typically, a distribution of results, due to differences in FPGA architectures and heuristic-based CAD tools, can occur. There are designs that can be implemented particularly well in a specific architecture and CAD tool, whereas other designs cannot be implemented at all. Many different conclusions can be reached by using a subset of designs.

Figure 3 shows how only selecting the top circuits can produce a different average result than when selecting the full set. A large number of designs must be used to understand the overall performance of the device and CAD tool.

Figure 3. Choice of Circuits Impacts Benchmark Results (Note 1)

Note to Figure 3:

1. Benchmarking data is based on comparing an Altera device to an equivalent Xilinx device using the latest available Quartus II and ISE software at the time of analysis.

Conclusion

The benchmarking results obtained using poor benchmarking processes can produce severely misleading conclusions. Therefore, following a careful benchmarking methodology is critical, and communicating this methodology is equally important.

For more information about Altera's rigorous benchmarking methodology, refer to the FPGA Performance Benchmark Methodology white paper (PDF).