Tool for benchmarking and comparing various engine and mod variants
Introduction
Benchmarker is a simple tool that lets you test various engine and mod variants in an easy way and compare results of benchmarks.
You can start it from Zero-K lobby - settings-> benchmarker or by clicking spring://@benchmark link
Installing benchmarks
- Git checkout https://github.com/ZeroK-RTS/Benchmarks to "Benchmarks" folder - best in Spring data dir (my games/spring on windows), also works in current directory of benchmarker
On Linux
- Check out to ~/.spring/Benchmarks
Key features
- automatically downloads all needed dependencies (engine, mod, deps, maps)
- lets you specify configuration for your test run, (engine, script and mod settings)
- automatically modifies modinfo.lua (change dependency - override mod)
- automatically modifies startscript as needed (map override)
- compares results and sums them up in a neat table
- auto calculates avg,min,max of repeating values
- allows bisecting to find faulty engine/mod
How to modify configuration presets
- configurations must be stored in "Configs" or "Benchmarks/Configs" folder, you can create a new subfolder there and place springsettings.cfg and various luaui files there as needed. This folder will be used as primary so it should override everything else.
How to create benchmark mutator
- create any mod or mutator in a standard way
- put mutator into "Mods" or "Benchmarks/Mods" folder into "mytest.sdd" subfolder
How to create a start script
- create my_Script.txt and put it to "Scripts" or "Benchmarks/Scripts" folder.
How to output measured stats
- echo any text in the format "!transmitlobby some name: 123"
Example values:
- "!transmitlobby pathing validity:0.3"
- "!transmitlobby avg. fps during nuke: 15"
- "!transmitlobby max gameframe ms: 50"
If same key is repeated many times - like "max gameframe ms" then the Benchmarker auto creates avg, min and max values.