Table of Contents
- 1 What is Caliper?
- 2 Caliper Setup
- 3 Caliper Configuration
- 4 Caliper Execution
- 5 Generate caliper report in HTML format
- 6 General Tips
- 7 Appendix
- 8 List of Benchmarks Supported in Caliper
1 What is Caliper?
Caliper is a benchmarking framework for hardware platforms, integrated with industry standard tools and test cases. It provides detailed performance assessments for server platforms based on ARM64 and X86_64.
Caliper executes on a host system which is connected to the target platforms through network. The performance parameters are captured from the targets and converted to score through series of formulas. Thereby caliper intuitively presents various performance values and shows the performance gaps of the hardware platforms.
The uniqueness of caliper is a consolidated score based evaluation. The best performance in each test will get a score of 100. All other test scores are calculated relative to this. This provides a clear relative positioning of each target platform. It enables easy analysis of the performance levels and bottlenecks.
As the performance testing can depends on system hardware resources at the time of execution, it is bound to vary over repeated testing. In general, this makes it difficult to analyze the performance results. Caliper tries to solve by executing multiple iterations of each test on every platform and ensure low variance. This provides a more believable and stable scores throughout the report.
2 Caliper Setup
- Two network switches – one for 1G network connection and another for 10G network connection.
- Target platform on which all test tools will be execute.
- Two machines which acts as clients.
- One machine for caliper host.
- Extra client (connected through eth3) for specific test cases where target to be stressed with multiple network interfaces.
3 Caliper Configuration
This section will discuss about the high level steps involved for caliper configurations.
Please see the detailed steps from the manual in the following link:
Get the caliper tool from https://github.com/open-estuary/caliper.git
Install the caliper in your system using below steps:
- Install the dependency packages for all the machines to execute tools.
- Configure system login information in ~/caliper_output/configuration/config/client_config.cfg file.
- Choose the tools which user wants to execute from ~/caliper_output/configuration/test_cases_cfg/ directory.
4 Caliper Execution
Executes the caliper with “caliper -brps” command. By default caliper command is configured with “brps” option.
To execute caliper with the supported options, refer caliper user manual section 5.1.
5 Generate caliper report in HTML format
Refer section 5.2 in the user manual. High level steps are as follows:
- After the successful caliper execution, it generated three files form each platform (platform.yaml, platform_score.yaml, platform_hw_info.yaml) inside the ~/caliper_output/<workspace directory>/output/results/yaml directory.
- Copy these files into ~/caliper_output/frontend/frontend/data_files/Input_Logs/ directory as specified in caliper user manual.
- Execute “caliper -BRPSw” command.
- Report will be generated as ~/caliper_output/<_WS_current_data_time>/output/results/test_results.tar.gz.
- Untar the test_results.tar.gz file and open index.html file with google chrome explorer.
6 General Tips
6.1 Checklist for caliper execution
- Confirm the hostname configuration in below files for hadoop tool execution:
In /etc/hostname file:
In /etc/hosts add this line:
- After updating above files, user has to reboot the target platform.
- Mount separate disk (Example: /dev/sdb1) at /mnt/sdb (create the mount if not exists) for storage tools testing.
- Start mysql server for centos target platform: cd /usr/local/mysql; ./bin/mysqld_safe &
- Check all Ethernet interfaces which will be use for caliper execution.
Use this command to check the Ethernet interfaces: ethtool <interface name>
- Check the ping response of all the ip addresses which will be use for caliper execution. (Refer ~/caliper_output/configuration/config/client_config.cfg file for ip addresses which will be used for caliper execution)
6.2 Caliper output log verification
After caliper execution has been completed, verify below files are present in the workspace.
- output/final_parsing_logs.yaml – This file will be used for caliper framework. It is an intermediate file, which is not relevant for end user.
- output/results_summary.log – summary information of tool execution status.
- output/caliper_exe.log – execution time of tool
- caliper_build/<each tool build log> – If the tool is successfully build, then the file name will be <tool_name_arch_name>.suc. If the tool in not able to build successfully, then the file name will be <tool_name_arch_name>.fail. User has to review the reason for failure and fix the issue.
- caliper_exec/<each tool output log> – The output log of each tool will be present in caliper_exec directory. If tool execution has been failed, then user can review the log file and fix the issue accordingly.
- result/yaml/<platform name given in client_config.cfg file>.yaml – This file contain the test case results of each tool.
- results/yaml/<platform name given in client_config.cfg file>_score.yaml – This file contain the test case results of each tool in compressed way.
- results/yaml/<platform name given in client_config.cfg file>_hw_info.yaml – This file contain the hardware information of the platform.
6.3 Caliper re-execution for failed tools
Refer below steps if user wants to execute caliper for those tools which has been failed.
- Execute this command: cd ~/caliper_output/<workspace created by caliper>/test_cases_cfg/
- Choose only failed tools in following files:
- Execute this command: caliper -brpsf <workspace created by caliper>
- After successful execution of all the tools, uncomment all the tools in common_cases_def.cfg, server_cases_def.cfg and application_cases_def.cfg files.
- Perform parsing and scoring using the command: caliper -BRpsf <workspace created by caliper>
- Please verify that all the yaml files has been created in “output/results/yaml” directory.
6.4 Change platform name in HTML report
If user wants to change the name of the platform in caliper HTML report, then make following changes:
- Execute this command: cd ~/caliper_output/frontend/frontend/data_files/Input_Logs/
- Update the “platform_name>.yaml” files in Input_Consolidated directory and change the name inside this file at 2 places: (see highlighted text)
Byte_order: Little Endian
L1d_cache: 1024 kB
L1i_cache: 1536 kB
L2_cache: 8192 kB
L3_cache: 32768 kB
OS_Version: Linux 4.9.20+
- Follow same procedure for <platform_name>_score.yaml files in Input_Report directory.
- Update the name of <platform_name>_hw_info.yaml files in Input_Hardware directory.
- Execute command: caliper -BRPSw
- New workspace will be created in ~/caliper_output directory.
- Switch to the new workspace and extract test_results.tar.gz file in output/results directory.
- Open index.html web page.
Caliper user manual: https://github.com/open-estuary/caliper/blob/master/docs/Caliper_User_Manual.pdf
Caliper developer manual: https://github.com/open-estuary/caliper/blob/master/docs/Caliper_Developer_Manual.pdf
Nginx tool user guide: https://github.com/open-estuary/caliper/blob/master/docs/Nginx_Tool_User_Guide.pdf
Redis tool user guide: https://github.com/open-estuary/caliper/blob/master/docs/Redis_Tool_User_Guide.pdf
8 List of Benchmarks Supported in Caliper
|11||qperf||install qperf – “apt-get install qperf” or “yum install qperf”|
|14||fio||GPL v2, https://github.com/axboe/fio/blob/master/COPYING|
|16||iperf3||BSD license, http://iperf.sourceforge.net/ ,https://github.com/esnet/iperf|
|17||netperf||HP license, http://www.calculate-linux.org/packages/licenses/netperf|
|18||scimark2||public domain, http://math.nist.gov/scimark2/credits.html|
|19||sysbench||GPL v2, https://github.com/akopytov/sysbench , http://imysql.com/wp-content/uploads/2014/10/sysbench-manual.pdf|
|23||Compile||Compilation performance benchmark|
Suggestions/ feedbacks @ firstname.lastname@example.org