Caliper Benchmarking


1 What is Caliper?

Caliper is a benchmarking framework for hardware platforms, integrated with industry standard tools and test cases. It provides detailed performance assessments for server platforms based on ARM64 and X86_64.

Caliper executes on a host system which is connected to the target platforms through network. The performance parameters are captured from the targets and converted to score through series of formulas. Thereby caliper intuitively presents various performance values and shows the performance gaps of the hardware platforms.

The uniqueness of caliper is a consolidated score based evaluation. The best performance in each test will get a score of 100. All other test scores are calculated relative to this. This provides a clear relative positioning of each target platform. It enables easy analysis of the performance levels and bottlenecks.

As the performance testing can depends on system hardware resources at the time of execution, it is bound to vary over repeated testing. In general, this makes it difficult to analyze the performance results. Caliper tries to solve by executing multiple iterations of each test on every platform and ensure low variance. This provides a more believable and stable scores throughout the report.

2 Caliper Setup

http://open-estuary.org/wp-content/uploads/2015/08/caliper_setup1.png

Caliper requires:

  • Two network switches – one for 1G network connection and another for 10G network connection.
  • Target platform on which all test tools will be execute.
  • Two machines which acts as clients.
  • One machine for caliper host.
  • Extra client (connected through eth3) for specific test cases where target to be stressed with multiple network interfaces.

3 Caliper Configuration

This section will discuss about the high level steps involved for caliper configurations.

Please see the detailed steps from the manual in the following link:

https://github.com/open-estuary/caliper/blob/master/docs/Caliper_User_Manual.pdf

Get the caliper tool from https://github.com/open-estuary/caliper.git

Install the caliper in your system using below steps:

  • Install the dependency packages for all the machines to execute tools.
  • Configure system login information in ~/caliper_output/configuration/config/client_config.cfg file.
  • Choose the tools which user wants to execute from ~/caliper_output/configuration/test_cases_cfg/ directory.

4 Caliper Execution

Executes the caliper with “caliper -brps” command. By default caliper command is configured with “brps” option.

To execute caliper with the supported options, refer caliper user manual section 5.1.

5 Generate caliper report in HTML format

Refer section 5.2 in the user manual. High level steps are as follows:

  • After the successful caliper execution, it generated three files form each platform (platform.yaml, platform_score.yaml, platform_hw_info.yaml) inside the ~/caliper_output/<workspace directory>/output/results/yaml directory.
  • Copy these files into ~/caliper_output/frontend/frontend/data_files/Input_Logs/ directory as specified in caliper user manual.
  • Execute “caliper -BRPSw” command.
  • Report will be generated as ~/caliper_output/<_WS_current_data_time>/output/results/test_results.tar.gz.
  • Untar the test_results.tar.gz file and open index.html file with google chrome explorer.

6 General Tips

6.1 Checklist for caliper execution

  • Confirm the hostname configuration in below files for hadoop tool execution:

In /etc/hostname file:
localhost.localdomain

In /etc/hosts add this line:
127.0.0.1 localhost

  • After updating above files, user has to reboot the target platform.
  • Mount separate disk (Example: /dev/sdb1) at /mnt/sdb (create the mount if not exists) for storage tools testing.
  • Start mysql server for centos target platform: cd /usr/local/mysql; ./bin/mysqld_safe &
  • Check all Ethernet interfaces which will be use for caliper execution.

Use this command to check the Ethernet interfaces: ethtool <interface name>

  • Check the ping response of all the ip addresses which will be use for caliper execution. (Refer ~/caliper_output/configuration/config/client_config.cfg file for ip addresses which will be used for caliper execution)

6.2 Caliper output log verification

After caliper execution has been completed, verify below files are present in the workspace.

  • output/final_parsing_logs.yaml – This file will be used for caliper framework. It is an intermediate file, which is not relevant for end user.
  • output/results_summary.log – summary information of tool execution status.
  • output/caliper_exe.log – execution time of tool
  • caliper_build/<each tool build log> – If the tool is successfully build, then the file name will be <tool_name_arch_name>.suc. If the tool in not able to build successfully, then the file name will be <tool_name_arch_name>.fail. User has to review the reason for failure and fix the issue.
  • caliper_exec/<each tool output log> – The output log of each tool will be present in caliper_exec directory. If tool execution has been failed, then user can review the log file and fix the issue accordingly.
  • result/yaml/<platform name given in client_config.cfg file>.yaml – This file contain the test case results of each tool.
  • results/yaml/<platform name given in client_config.cfg file>_score.yaml – This file contain the test case results of each tool in compressed way.
  • results/yaml/<platform name given in client_config.cfg file>_hw_info.yaml – This file contain the hardware information of the platform.

6.3 Caliper re-execution for failed tools

Refer below steps if user wants to execute caliper for those tools which has been failed.

  • Execute this command: cd ~/caliper_output/<workspace created by caliper>/test_cases_cfg/
  • Choose only failed tools in following files:

common_cases_def.cfg

server_cases_def.cfg

application_cases_def.cfg

  • Execute this command: caliper -brpsf <workspace created by caliper>
  • After successful execution of all the tools, uncomment all the tools in common_cases_def.cfg, server_cases_def.cfg and application_cases_def.cfg files.
  • Perform parsing and scoring using the command: caliper -BRpsf <workspace created by caliper>
  • Please verify that all the yaml files has been created in “output/results/yaml” directory.

6.4 Change platform name in HTML report

If user wants to change the name of the platform in caliper HTML report, then make following changes:

  • Execute this command: cd ~/caliper_output/frontend/frontend/data_files/Input_Logs/
  • Update the “platform_name>.yaml” files in Input_Consolidated directory and change the name inside this file at 2 places: (see highlighted text)

Configuration:

Byte_order: Little Endian

CPU: ’64’

CPU_type: ‘*TBA’

Hostname: platform_name

L1d_cache: 1024 kB

L1i_cache: 1536 kB

L2_cache: 8192 kB

L3_cache: 32768 kB

Machine_arch: aarch64

Memory: 523535MB

OS_Version: Linux 4.9.20+

name: platform_name

  • Follow same procedure for <platform_name>_score.yaml files in Input_Report directory.
  • Update the name of <platform_name>_hw_info.yaml files in Input_Hardware directory.
  • Execute command: caliper -BRPSw
  • New workspace will be created in ~/caliper_output directory.
  • Switch to the new workspace and extract test_results.tar.gz file in output/results directory.
  • Open index.html web page.

7 Appendix

Caliper user manual: https://github.com/open-estuary/caliper/blob/master/docs/Caliper_User_Manual.pdf

Caliper developer manual: https://github.com/open-estuary/caliper/blob/master/docs/Caliper_Developer_Manual.pdf

Nginx tool user guide: https://github.com/open-estuary/caliper/blob/master/docs/Nginx_Tool_User_Guide.pdf

Redis tool user guide: https://github.com/open-estuary/caliper/blob/master/docs/Redis_Tool_User_Guide.pdf

8 List of Benchmarks Supported in Caliper

1 openssl https://www.openssl.org/
2 coremark EEMBC, https://www.eembc.org/
3 linpack http://www.netlib.org/linpack/
4 nbench http://www.tux.org/~mayer/linux/bmark.html
5 lmbench http://www.bitmover.com/lmbench/
6 cachebench http://icl.cs.utk.edu/projects/llcbench/cachebench.html
7 iozone http://www.iozone.org/
8 openblas https://github.com/xianyi/OpenBLAS.git
9 netperf http://www.netperf.org/netperf/
10 dhrystone http://fossies.org/linux/privat/old/dhrystone-2.1.tar.gz/
11  qperf install qperf – “apt-get install qperf” or “yum install qperf”
12  nginx http://nginx.org/download/nginx-1.11.13.tar.gz
13  redis https://github.com/sjtuhjh/applications
14 fio GPL v2, https://github.com/axboe/fio/blob/master/COPYING
15 hibench https://github.com/intel-hadoop/HiBench
16 iperf3 BSD license, http://iperf.sourceforge.net/ ,https://github.com/esnet/iperf
17 netperf HP license, http://www.calculate-linux.org/packages/licenses/netperf
18 scimark2 public domain, http://math.nist.gov/scimark2/credits.html
19 sysbench GPL v2, https://github.com/akopytov/sysbench , http://imysql.com/wp-content/uploads/2014/10/sysbench-manual.pdf
20  unixbench https://github.com/kdlucas/byte-unixbench
21 stress-ng http://kernel.ubuntu.com/~cking/stress-ng/
22 Unzip Compression benchmark
23 Compile Compilation performance benchmark

Suggestions/ feedbacks @ shyju.pv@gmail.com