librewolf/lkrg
1
0
Fork 0
mirror of https://github.com/openwall/lkrg.git synced 2023-12-13 21:30:29 +01:00
Code Issues Projects Releases Wiki Activity
lkrg/PERFORMANCE
Vitaly Chikunov 2648a1d607 Various spelling and grammar corrections (#7) 
* Various spelling corrections by codespell 1.17.1
* Various grammar corrections

Signed-off-by: Vitaly Chikunov <vt@altlinux.org>
Co-authored-by: Solar Designer <solar@openwall.com>
2020-08-19 00:59:35 +02:00

605 lines
28 KiB
Text
Raw Permalink Blame History

LKRG benchmarks with Phoronix Test Suite
========================================
Michael Larabel of Phoronix published an article on his benchmarks of LKRG as
packaged for Whonix as of February 25, 2020. This means a development version
between LKRG 0.7 and 0.8. The benchmark system was a "Core i9 9900KS desktop
running an Ubuntu 20.04 snapshot and using the Linux 5.4 Ubuntu kernel":
https://www.phoronix.com/scan.php?page=article&item=linux-lkrg-performance&num=1
The raw results are available at:
https://openbenchmarking.org/result/2002252-PTS-COREI99901
As LKRG developers, we found most of the results reasonable, but were surprised
by some. Anyhow, the overall performance impact of LKRG as seen from the
geometric mean of all test results was around 4.4%:
Geometric Mean Of All Test Results: Result Composite 112.550 107.581
which gives 1-107.581/112.550 = 0.0441.
Working towards the LKRG 0.8 release, we've made numerous changes to improve
its performance and scalability, which were not yet included in the development
version benchmarked above. We contacted Michael, and he kindly instructed us
how to rerun the exact same set of 58 tests:
sudo apt install php-cli php-xml git-core apt-file
git clone https://github.com/phoronix-test-suite/phoronix-test-suite
cd phoronix-test-suite
./phoronix-test-suite benchmark 2002252-PTS-COREI99901
which we did (albeit on a somewhat different system configuration), using
phoronix-test-suite as of commit e7f3a2b7fe083ecce28c7e42b315bd314dfe7070.
We ran these tests in June 2020 using LKRG 0.8 with two sets of settings:
"LKRG 0.8" means its default profile (profile_validate=3) with msr_validate=1
enabled on top of it (which might have added very slight additional overhead).
"LKRG 0.8 light" means the light profile (profile_validate=1).
We also happened to run the tests without LKRG twice (along with each one of
the two LKRG configurations above), which shows the jitter unrelated to LKRG.
The overall performance impact of LKRG 0.8 as seen from the geometric mean of
all test results is around 2.5% for LKRG defaults and around 2.0% for "light":
Geometric Mean Of All Test Results
Result Composite
Geometric Mean > Higher Is Better
LKRG 0.8 ........................................ 79.64 |==============================================================================================
LKRG 0.8 light .................................. 80.11 |==============================================================================================
Without LKRG 1 .................................. 81.69 |================================================================================================
Without LKRG 2 .................................. 81.74 |================================================================================================
which we process as 1-79.64/81.69 = 0.02509 and 1-80.11/81.74 = 0.01994.
System configuration:
Processor: Intel Xeon E-2176G @ 4.70GHz (6 Cores / 12 Threads), Motherboard: HP 8458 (Q50 Ver. 01.04.01 BIOS), Chipset: Intel Cannon Lake PCH, Memory: 32GB, Disk: 1024GB SAMSUNG MZVLB1T0HALR-000H2, Graphics: NVIDIA Quadro P1000 Mobile 4GB, Audio: Conexant CX20632, Network: Intel I219-LM + Intel-AC 9560
OS: Ubuntu 18.04, Kernel: 4.15.0-101-generic (x86_64), Desktop: Xfce, Display Server: X Server 1.19.6, OpenCL: OpenCL 1.2 CUDA 10.2.159, Compiler: GCC 7.5.0, File-System: ext4, Screen Resolution: 3840x2160
Below are our raw per-test results, including all the curiosities. (LKRG
speeds up "Socket Activity" by 30% or 35%?! We don't think it normally does.
We also don't think it normally slows down context switches. This is where the
science is in CS, with experiments sometimes showing weird results. Luckily,
with as many as 58 tests these weird effects mostly cancel out.)
SQLite 3.30.1
Threads / Copies: 1
Seconds < Lower Is Better
Without LKRG 1 .................................. 47.50 |==================
LKRG 0.8 ........................................ 51.25 |===================
LKRG 0.8 light .................................. 50.28 |===================
Without LKRG 2 .................................. 49.05 |==================
SQLite 3.30.1
Threads / Copies: 8
Seconds < Lower Is Better
Without LKRG 1 .................................. 207.52 |==================
LKRG 0.8 ........................................ 198.15 |=================
LKRG 0.8 light .................................. 192.24 |=================
Without LKRG 2 .................................. 207.05 |==================
PostMark 1.51
Disk Transaction Performance
TPS > Higher Is Better
Without LKRG 1 .................................. 6466 |====================
LKRG 0.8 ........................................ 5173 |================
LKRG 0.8 light .................................. 5725 |==================
Without LKRG 2 .................................. 6522 |====================
t-test1 2017-01-13
Threads: 1
Seconds < Lower Is Better
Without LKRG 1 .................................. 18.33 |===================
LKRG 0.8 ........................................ 18.51 |===================
LKRG 0.8 light .................................. 18.82 |===================
Without LKRG 2 .................................. 18.59 |===================
t-test1 2017-01-13
Threads: 2
Seconds < Lower Is Better
Without LKRG 1 .................................. 6.094 |===================
LKRG 0.8 ........................................ 6.118 |===================
LKRG 0.8 light .................................. 6.126 |===================
Without LKRG 2 .................................. 6.084 |===================
pmbench
Concurrent Worker Threads: 1 - Read-Write Ratio: 100% Reads
us - Average Page Latency < Lower Is Better
Without LKRG 1 .................................. 0.0259 |==================
LKRG 0.8 ........................................ 0.0257 |==================
LKRG 0.8 light .................................. 0.0258 |==================
Without LKRG 2 .................................. 0.0258 |==================
pmbench
Concurrent Worker Threads: 1 - Read-Write Ratio: 100% Writes
us - Average Page Latency < Lower Is Better
Without LKRG 1 .................................. 0.0289 |==================
LKRG 0.8 ........................................ 0.0285 |==================
LKRG 0.8 light .................................. 0.0285 |==================
Without LKRG 2 .................................. 0.0286 |==================
pmbench
Concurrent Worker Threads: 16 - Read-Write Ratio: 100% Reads
us - Average Page Latency < Lower Is Better
Without LKRG 1 .................................. 0.0690 |==================
LKRG 0.8 ........................................ 0.0638 |=================
LKRG 0.8 light .................................. 0.0642 |=================
Without LKRG 2 .................................. 0.0616 |================
pmbench
Concurrent Worker Threads: 16 - Read-Write Ratio: 100% Writes
us - Average Page Latency < Lower Is Better
Without LKRG 1 .................................. 0.0702 |=================
LKRG 0.8 ........................................ 0.0753 |==================
LKRG 0.8 light .................................. 0.0744 |==================
Without LKRG 2 .................................. 0.0685 |================
pmbench
Concurrent Worker Threads: 1 - Read-Write Ratio: 80% Reads 20% Writes
us - Average Page Latency < Lower Is Better
Without LKRG 1 .................................. 0.0791 |==================
LKRG 0.8 ........................................ 0.0783 |==================
LKRG 0.8 light .................................. 0.0787 |==================
Without LKRG 2 .................................. 0.0785 |==================
pmbench
Concurrent Worker Threads: 16 - Read-Write Ratio: 80% Reads 20% Writes
us - Average Page Latency < Lower Is Better
Without LKRG 1 .................................. 0.1183 |==================
LKRG 0.8 ........................................ 0.1197 |==================
LKRG 0.8 light .................................. 0.1191 |==================
Without LKRG 2 .................................. 0.1104 |=================
Sockperf 3.4
Test: Throughput
Messages Per Second > Higher Is Better
Without LKRG 1 .................................. 528116 |==================
LKRG 0.8 ........................................ 480136 |================
LKRG 0.8 light .................................. 480136 |================
Without LKRG 2 .................................. 523773 |==================
Sockperf 3.4
Test: Latency Ping Pong
usec < Lower Is Better
Without LKRG 1 .................................. 3.545 |=================
LKRG 0.8 ........................................ 4.028 |===================
LKRG 0.8 light .................................. 3.910 |==================
Without LKRG 2 .................................. 3.606 |=================
Sockperf 3.4
Test: Latency Under Load
usec < Lower Is Better
Without LKRG 1 .................................. 12.62 |===============
LKRG 0.8 ........................................ 14.13 |=================
LKRG 0.8 light .................................. 15.32 |===================
Without LKRG 2 .................................. 15.55 |===================
Ethr 2019-01-02
Server Address: localhost - Protocol: TCP - Test: Latency - Threads: 1
Microseconds < Lower Is Better
Without LKRG 1 .................................. 13.40 |==================
LKRG 0.8 ........................................ 14.38 |===================
LKRG 0.8 light .................................. 13.99 |==================
Without LKRG 2 .................................. 13.31 |==================
Ethr 2019-01-02
Server Address: localhost - Protocol: TCP - Test: Connections/s - Threads: 1
Connections/sec > Higher Is Better
Without LKRG 1 .................................. 12483 |===================
LKRG 0.8 ........................................ 11357 |=================
LKRG 0.8 light .................................. 11627 |==================
Without LKRG 2 .................................. 12497 |===================
iPerf 3.7
Server Address: localhost - Server Port: 5201 - Duration: 10 Seconds - Test: TCP - Parallel: 1
Mbits/sec > Higher Is Better
Without LKRG 1 .................................. 87761 |===================
LKRG 0.8 ........................................ 83260 |==================
LKRG 0.8 light .................................. 82774 |==================
Without LKRG 2 .................................. 87674 |===================
iPerf 3.7
Server Address: localhost - Server Port: 5201 - Duration: 10 Seconds - Test: UDP - Parallel: 1
Mbits/sec > Higher Is Better
Without LKRG 1 .................................. 1.05 |====================
LKRG 0.8 ........................................ 1.05 |====================
LKRG 0.8 light .................................. 1.05 |====================
Without LKRG 2 .................................. 1.05 |====================
iPerf 3.7
Server Address: localhost - Server Port: 5201 - Duration: 10 Seconds - Test: TCP - Parallel: 10
Mbits/sec > Higher Is Better
Without LKRG 1 .................................. 102445 |==================
LKRG 0.8 ........................................ 100749 |==================
LKRG 0.8 light .................................. 100253 |==================
Without LKRG 2 .................................. 101773 |==================
iPerf 3.7
Server Address: localhost - Server Port: 5201 - Duration: 10 Seconds - Test: UDP - Parallel: 10
Mbits/sec > Higher Is Better
Without LKRG 1 .................................. 10.5 |====================
LKRG 0.8 ........................................ 10.5 |====================
LKRG 0.8 light .................................. 10.5 |====================
Without LKRG 2 .................................. 10.5 |====================
Java SciMark 2.0
Computational Test: Composite
Mflops > Higher Is Better
Without LKRG 1 .................................. 2560.11 |=================
LKRG 0.8 ........................................ 2576.03 |=================
LKRG 0.8 light .................................. 2572.08 |=================
Without LKRG 2 .................................. 2571.91 |=================
SVT-AV1 0.8
Encoder Mode: Enc Mode 8 - Input: 1080p
Frames Per Second > Higher Is Better
Without LKRG 1 .................................. 19.63 |===================
LKRG 0.8 ........................................ 19.60 |===================
LKRG 0.8 light .................................. 19.59 |===================
Without LKRG 2 .................................. 19.64 |===================
VP9 libvpx Encoding 1.8.2
Speed: Speed 5
Frames Per Second > Higher Is Better
Without LKRG 1 .................................. 25.90 |===================
LKRG 0.8 ........................................ 25.96 |===================
LKRG 0.8 light .................................. 25.96 |===================
Without LKRG 2 .................................. 26.05 |===================
x264 2019-12-17
H.264 Video Encoding
Frames Per Second > Higher Is Better
Without LKRG 1 .................................. 63.15 |===================
LKRG 0.8 ........................................ 62.96 |===================
LKRG 0.8 light .................................. 63.61 |===================
Without LKRG 2 .................................. 63.11 |===================
Coremark 1.0
CoreMark Size 666 - Iterations Per Second
Iterations/Sec > Higher Is Better
Without LKRG 1 .................................. 231557.92 |===============
LKRG 0.8 ........................................ 232499.71 |===============
LKRG 0.8 light .................................. 232255.06 |===============
Without LKRG 2 .................................. 232890.77 |===============
Timed Apache Compilation 2.4.41
Time To Compile
Seconds < Lower Is Better
Without LKRG 1 .................................. 20.63 |================
LKRG 0.8 ........................................ 23.77 |===================
LKRG 0.8 light .................................. 23.28 |===================
Without LKRG 2 .................................. 20.56 |================
Timed FFmpeg Compilation 4.2.2
Time To Compile
Seconds < Lower Is Better
Without LKRG 1 .................................. 75.06 |===================
LKRG 0.8 ........................................ 76.40 |===================
LKRG 0.8 light .................................. 76.10 |===================
Without LKRG 2 .................................. 75.03 |===================
Timed GCC Compilation 8.2
Time To Compile
Seconds < Lower Is Better
Without LKRG 1 .................................. 1058.20 |==============
LKRG 0.8 ........................................ 1306.89 |=================
LKRG 0.8 light .................................. 1262.02 |================
Without LKRG 2 .................................. 1057.01 |==============
Timed GDB GNU Debugger Compilation 9.1
Time To Compile
Seconds < Lower Is Better
Without LKRG 1 .................................. 117.11 |=================
LKRG 0.8 ........................................ 121.68 |==================
LKRG 0.8 light .................................. 120.11 |==================
Without LKRG 2 .................................. 116.06 |=================
Timed ImageMagick Compilation 6.9.0
Time To Compile
Seconds < Lower Is Better
Without LKRG 1 .................................. 40.43 |===================
LKRG 0.8 ........................................ 41.39 |===================
LKRG 0.8 light .................................. 41.13 |===================
Without LKRG 2 .................................. 40.27 |==================
Timed Linux Kernel Compilation 5.4
Time To Compile
Seconds < Lower Is Better
Without LKRG 1 .................................. 107.59 |==================
LKRG 0.8 ........................................ 110.36 |==================
LKRG 0.8 light .................................. 109.44 |==================
Without LKRG 2 .................................. 107.60 |==================
Timed LLVM Compilation 6.0.1
Time To Compile
Seconds < Lower Is Better
Without LKRG 1 .................................. 571.60 |==================
LKRG 0.8 ........................................ 577.02 |==================
LKRG 0.8 light .................................. 567.23 |==================
Without LKRG 2 .................................. 564.93 |==================
Timed MPlayer Compilation 1.4
Time To Compile
Seconds < Lower Is Better
Without LKRG 1 .................................. 51.27 |==================
LKRG 0.8 ........................................ 52.81 |===================
LKRG 0.8 light .................................. 52.34 |===================
Without LKRG 2 .................................. 51.18 |==================
Timed PHP Compilation 7.4.2
Time To Compile
Seconds < Lower Is Better
Without LKRG 1 .................................. 61.86 |==================
LKRG 0.8 ........................................ 63.78 |===================
LKRG 0.8 light .................................. 63.24 |===================
Without LKRG 2 .................................. 61.76 |==================
Build2 0.12
Time To Compile
Seconds < Lower Is Better
Without LKRG 1 .................................. 135.47 |=================
LKRG 0.8 ........................................ 139.57 |==================
LKRG 0.8 light .................................. 139.06 |==================
Without LKRG 2 .................................. 135.52 |=================
rays1bench 2020-01-09
Large Scene
mrays/s > Higher Is Better
Without LKRG 1 .................................. 33.93 |===================
LKRG 0.8 ........................................ 33.18 |===================
LKRG 0.8 light .................................. 33.45 |===================
Without LKRG 2 .................................. 33.55 |===================
Numpy Benchmark
Score > Higher Is Better
Without LKRG 1 .................................. 373.58 |==================
LKRG 0.8 ........................................ 373.55 |==================
LKRG 0.8 light .................................. 373.60 |==================
Without LKRG 2 .................................. 373.80 |==================
DeepSpeech 0.6
Acceleration: CPU
Seconds < Lower Is Better
Without LKRG 1 .................................. 72.88 |===================
LKRG 0.8 ........................................ 73.01 |===================
LKRG 0.8 light .................................. 73.15 |===================
Without LKRG 2 .................................. 72.81 |===================
Tachyon 0.99b6
Total Time
Seconds < Lower Is Better
Without LKRG 1 .................................. 145.07 |==================
LKRG 0.8 ........................................ 145.22 |==================
LKRG 0.8 light .................................. 144.81 |==================
Without LKRG 2 .................................. 145.07 |==================
PostgreSQL pgbench 12.0
Scaling: Buffer Test - Test: Normal Load - Mode: Read Only
TPS > Higher Is Better
Without LKRG 1 .................................. 148168.70 |===============
LKRG 0.8 ........................................ 144992.92 |===============
LKRG 0.8 light .................................. 145926.02 |===============
Without LKRG 2 .................................. 148208.80 |===============
PostgreSQL pgbench 12.0
Scaling: Buffer Test - Test: Normal Load - Mode: Read Write
TPS > Higher Is Better
Without LKRG 1 .................................. 3664.46 |=================
LKRG 0.8 ........................................ 3593.93 |================
LKRG 0.8 light .................................. 3710.23 |=================
Without LKRG 2 .................................. 3701.83 |=================
SQLite Speedtest 3.30
Timed Time - Size 1,000
Seconds < Lower Is Better
Without LKRG 1 .................................. 63.68 |===================
LKRG 0.8 ........................................ 64.52 |===================
LKRG 0.8 light .................................. 64.29 |===================
Without LKRG 2 .................................. 63.87 |===================
Inkscape
Operation: SVG Files To PNG
Seconds < Lower Is Better
Without LKRG 1 .................................. 21.58 |=================
LKRG 0.8 ........................................ 24.37 |===================
LKRG 0.8 light .................................. 23.65 |==================
Without LKRG 2 .................................. 21.51 |=================
RawTherapee
Total Benchmark Time
Seconds < Lower Is Better
Without LKRG 1 .................................. 64.32 |===================
LKRG 0.8 ........................................ 64.68 |===================
LKRG 0.8 light .................................. 64.66 |===================
Without LKRG 2 .................................. 64.33 |===================
BenchmarkMutex
Benchmark: Mutex Lock Unlock spinlock
ns < Lower Is Better
Without LKRG 1 .................................. 38 |=====================
LKRG 0.8 ........................................ 39 |======================
LKRG 0.8 light .................................. 38 |=====================
Without LKRG 2 .................................. 39 |======================
BenchmarkMutex
Benchmark: Mutex Lock Unlock std::mutex
ns < Lower Is Better
Without LKRG 1 .................................. 19 |======================
LKRG 0.8 ........................................ 18 |=====================
LKRG 0.8 light .................................. 18 |=====================
Without LKRG 2 .................................. 19 |======================
Stress-NG 0.07.26
Test: Forking
Bogo Ops/s > Higher Is Better
Without LKRG 1 .................................. 55337.95 |================
LKRG 0.8 ........................................ 52709.65 |===============
LKRG 0.8 light .................................. 54567.02 |================
Without LKRG 2 .................................. 55723.36 |================
Stress-NG 0.07.26
Test: Semaphores
Bogo Ops/s > Higher Is Better
Without LKRG 1 .................................. 4339272.09 |==============
LKRG 0.8 ........................................ 4387407.74 |==============
LKRG 0.8 light .................................. 4323867.93 |==============
Without LKRG 2 .................................. 4388385.72 |==============
Stress-NG 0.07.26
Test: Socket Activity
Bogo Ops/s > Higher Is Better
Without LKRG 1 .................................. 3912.43 |=============
LKRG 0.8 ........................................ 5304.83 |=================
LKRG 0.8 light .................................. 5144.24 |================
Without LKRG 2 .................................. 3935.55 |=============
Stress-NG 0.07.26
Test: Context Switching
Bogo Ops/s > Higher Is Better
Without LKRG 1 .................................. 1917531.34 |==============
LKRG 0.8 ........................................ 1523650.72 |===========
LKRG 0.8 light .................................. 1541390.37 |===========
Without LKRG 2 .................................. 1909769.07 |==============
Stress-NG 0.07.26
Test: System V Message Passing
Bogo Ops/s > Higher Is Better
Without LKRG 1 .................................. 5857712.96 |==============
LKRG 0.8 ........................................ 5990669.18 |==============
LKRG 0.8 light .................................. 5974934.61 |==============
Without LKRG 2 .................................. 5952706.40 |==============
ctx_clock
Context Switch Time
Clocks < Lower Is Better
Without LKRG 1 .................................. 1130 |====================
LKRG 0.8 ........................................ 1138 |====================
LKRG 0.8 light .................................. 1141 |====================
Without LKRG 2 .................................. 1130 |====================
Blender 2.82
Blend File: BMW27 - Compute: CPU-Only
Seconds < Lower Is Better
Without LKRG 1 .................................. 261.62 |==================
LKRG 0.8 ........................................ 262.28 |==================
LKRG 0.8 light .................................. 262.44 |==================
Without LKRG 2 .................................. 260.84 |==================
PyBench 2018-02-16
Total For Average Test Times
Milliseconds < Lower Is Better
Without LKRG 1 .................................. 973 |=====================
LKRG 0.8 ........................................ 983 |=====================
LKRG 0.8 light .................................. 975 |=====================
Without LKRG 2 .................................. 975 |=====================
PHPBench 0.8.1
PHP Benchmark Suite
Score > Higher Is Better
Without LKRG 1 .................................. 730291 |==================
LKRG 0.8 ........................................ 728913 |==================
LKRG 0.8 light .................................. 726232 |==================
Without LKRG 2 .................................. 729524 |==================
Mlpack Benchmark
Benchmark: scikit_svm
Seconds < Lower Is Better
Without LKRG 1 .................................. 15.30 |===================
LKRG 0.8 ........................................ 15.22 |===================
LKRG 0.8 light .................................. 15.22 |===================
Without LKRG 2 .................................. 15.23 |===================
Mlpack Benchmark
Benchmark: scikit_linearridgeregression
Seconds < Lower Is Better
Without LKRG 1 .................................. 236.57 |==================
LKRG 0.8 ........................................ 236.30 |==================
LKRG 0.8 light .................................. 236.68 |==================
Without LKRG 2 .................................. 237.05 |==================
Sunflow Rendering System 0.07.2
Global Illumination + Image Synthesis
Seconds < Lower Is Better
Without LKRG 1 .................................. 1.638 |===================
LKRG 0.8 ........................................ 1.617 |===================
LKRG 0.8 light .................................. 1.653 |===================
Without LKRG 2 .................................. 1.626 |===================
Shortly after the LKRG 0.8 release, Michael himself ran different benchmarks of
it (as many as 119 tests) and published an article and the raw results here:
https://www.phoronix.com/scan.php?page=article&item=lkrg-08-linux&num=1
https://openbenchmarking.org/result/2006277-NE-LKRG08BEN46
Once again, we found most of the results reasonable, but were surprised by
some, which we'll be looking into. Unfortunately, automated analytics of the raw
results above show inconsistent geometric means in two places (a bug, which
Michael acknowledged), so we cannot easily and confidently state LKRG's overall
performance impact as seen there, but the individual test results are usable.
Reference in a new issue View git blame Copy permalink