Benchmarking Required

It is nearing time to update the engine that runs Zero-K (the Spring Engine), but first we need to pin down the performance issues that affected many users when such an update was last attempted. To facilitate this we have created an easy to use automatic benchmarker. It currently contains benchmarks for our most pressing issue, but more are likely to be added in the future.

If you are the kind of person who does not use their computer while sleeping then consider running a benchmark and submitting the results to help us out.

The build system seems to be down, so the stable with the benchmark is not live.

I'll do this sometime in the next few days

I just ran the 1x benchmark on my Linux machine (Steam Linux version of ZK). After it completed, it closed ZK, but did not show any sending a report dialog. This might be the reason:
Zero-K.exe Warning: 0 : Problem reporting a bug: System.AggregateException: One or more errors occurred. ---> System.Net.WebException: Error: SecureChannelFailure (The authentication or decryption has failed.) ---> System.IO.IOException: The authentication or decryption has failed. ---> System.IO.IOException: The authentication or decryption has failed. ---> Mono.Security.Protocol.Tls.TlsException: The authentication or decryption has failed.
Full infolog:
https://pastebin.com/k9rjR0Fd

I do see this stuff in infolog_full, maybe it's helpful:

[f=-000001] ,average dt,min dt,max dt,dt_0,dt_3,dt_6,dt_9,dt_12,dt_15,dt_18,dt_21,dt_24,dt_27,dt_30,dt_33,dt_36,dt_39,dt_42,dt_45,average time,min time,max time,time_0,time_3,time_6,time_9,time_12,time_15,time_18,time_21,time_24,time_27,time_30,time_33,time_36,time_39,time_42,time_45,end unit count
[f=-000001] Engine 287 r3,0.07801547,0.02515293,0.30483779,0,23,565,899,602,737,373,391,252,64,4,1,3,1,0,0,0.07799787,0.023,0.30499998,0,26,555,898,621,723,378,388,226,88,6,1,1,2,1,0,911
[f=-000001] Engine 308 r1,0.05177741,0.01864615,0.18432684,0,1,9,207,765,1207,811,933,1174,606,87,50,20,7,2,15,0.05176713,0.018,0.134,0,0,9,204,771,1217,807,961,1003,726,104,51,8,12,3,17,934
[f=-000001] Engine 309 r2,0.08401322,0.02107306,0.41659334,1,72,756,718,715,681,158,313,234,71,11,1,0,0,0,2,0.08399519,0.02,0.41799998,1,73,747,724,721,676,153,327,204,89,14,1,0,0,0,2,934
Zero-K.exe Information: 0 : Chobbyla << SendBugReport {"Title":"GCBenchmarkv1","Description":"Benchmark by Vermind\n```,average dt,min dt,max dt,dt_0,dt_3,dt_6,dt_9,dt_12,dt_15,dt_18,dt_21,dt_24,dt_27,dt_30,dt_33,dt_36,dt_39,dt_42,dt_45,average time,min time,max time,time_0,time_3,time_6,time_9,time_12,time_15,time_18,time_21,time_24,time_27,time_30,time_33,time_36,time_39,time_42,time_45,end unit count```\n```Engine 287 r3,0.07801547,0.02515293,0.30483779,0,23,565,899,602,737,373,391,252,64,4,1,3,1,0,0,0.07799787,0.023,0.30499998,0,26,555,898,621,723,378,388,226,88,6,1,1,2,1,0,911```\n```Engine 308 r1,0.05177741,0.01864615,0.18432684,0,1,9,207,765,1207,811,933,1174,606,87,50,20,7,2,15,0.05176713,0.018,0.134,0,0,9,204,771,1217,807,961,1003,726,104,51,8,12,3,17,934```\n```Engine 309 r2,0.08401322,0.02107306,0.41659334,1,72,756,718,715,681,158,313,234,71,11,1,0,0,0,2,0.08399519,0.02,0.41799998,1,73,747,724,721,676,153,327,204,89,14,1,0,0,0,2,934```\n```"}
Zero-K.exe Information: 0 : Chobby closed connection

Hello,
I have not sent a report on the first run because I moved the mouse around a bit.
The display was not fine though. First run the screen was a bit shifted to the right (minimap not completely visible) but got worst on the second run (minimap nearly completely invisible, apparently because the image was more horizontally stretched). For the third run, it was probably like the second one. I will still run it without moving the mouse to send the report.

GoogleFrog, you probably missed this bit "but did not show any sending a report dialog." So no, I saw no prompt. There's something wrong with Mono's handling of certificates probably, which prevents the sending or the dialog appearing.

Send me the csv found in Zero-K/benchmarks. It should take 20 times longer than the short benchmark, which may go over 7 hours if it runs slowly.

Maybe is a good idea to post one/some summaries at some point (like, how many people run, how many issues you still saw, how many runs would you consider are needed, etc.). Just so that the testers get a better feeling on how they contributed (will also run it when I get back home from travelling)

The issue has been resolved. Apparently kloot already made a toggle for the performance-impacting GC change, but did not mention this on the ticket until I reopened it. I'll see about making another benchmark to test the current engines performance against latest dev. This is much more difficult because the engines which are 550 commits apart do not sync, so running the same replay on them does not result in the same game. This means that the short benchmark will have to be longer.

Limited multiplayer testing is blocked by a few bugs. In my testing I saw units teleport across the map upon completion and there are some issues relating to formation moves. General testing/adoption of the dev engine is blocked by a regression in the skirmish AI. Hopefully lamer is around to fix it.

Yeah, he probably does needs another round because of the new engine release to check for regressions.

Your setup is similar to mine. I am also on Linux using AMD Graphics with AMDGPU but I am using an Radeon RX 470.