New benchmarking tool for Spring - now part of ZKL

Recent problems show, that we need an objective way to measure performance changes over time and across various versions of game, engine and configurations. Benchmarker is a tool to make this easier.

Wiki page: http://zero-k.info/Wiki/Benchmarker

Benchmarker is so far intended for developers and needs people to write more tests (mutators).

Please help if you can :)

Interesting, though I would like to see what the error is for each test (I'm assuming jK did more than one test of each scenario).

The most notable difference seems to be with the unit creation scenario, with 94 having 1/5th the FPS of the other versions. There is a trend towards newer engines being slower, which isn't terribly surprising (feature creep and the overall common "next year's computers will be faster" attitude), but it's hard to tell whether those changes are beyond measurement error.

Is this available in linux or only ZKL?

TheSponge it is currently included in ZKL, but its a separate program that would run on mono under linux fine.

Except for automated engine downloader, that does not work on linux pre 91.0 and is untested - so someone would have to modify code to make multi engine work on linux.

I will post it once its tested for linux

http://zero-k.info/temp/bench.zip

Yes http://zero-k.googlecode.com/svn/trunk/other/Benchmarker/Benchmarks
svn checkout this to Benchmarks in my games/spring (or your spring data folder or folder with benchmarker)

quote: I did that and when I clicked start it did this.

Well duh, it says you haven't checked any of the checkboxes, right there in the error message in the middle of your 1920x1080 pixels.

yeah some shell script could be nice..

Benchmarker is now updated with a new bisect tool to locate faulty commit.
Engine downloader now works properly on linux and wiki was updated with proper linux install instructions :-)

I think the wiki page should add the link to the spring startscript page https://github.com/spring/spring/blob/master/doc/StartScriptFormat.txt . Not everybody knows the format in times of ZKL and weblobby.

Edit: moved end of sentence fullstop away from link