Loading...
  OR  Zero-K Name:    Password:   

Forum index  > News   >

Benchmarking Required

17 posts, 1187 views
Post comment
Filter:    Player:  
sort



AUrankAdminGoogleFrog
13 days ago

It is nearing time to update the engine that runs Zero-K (the Spring Engine), but first we need to pin down the performance issues that affected many users when such an update was last attempted. To facilitate this we have created an easy to use automatic benchmarker. It currently contains benchmarks for our most pressing issue, but more are likely to be added in the future.

If you are the kind of person who does not use their computer while sleeping then consider running a benchmark and submitting the results to help us out.
+7 / -0

MXrankmoleculeman88
12 days ago
okay sounds like fun

+0 / -0



AUrankAdminGoogleFrog
12 days ago
The build system seems to be down, so the stable with the benchmark is not live.
+0 / -0



AUrankAdminGoogleFrog
12 days ago
Well MXrankmoleculeman88 submitted a benchmark, even though I am unable to get my non-dev ZK install to update.
+0 / -0

AUrankisaach
12 days ago
I'll do this sometime in the next few days
+0 / -0



AUrankAdminGoogleFrog
12 days ago
The release is now live.
+0 / -0


FIrankVermind
12 days ago
(edited 12 days ago)

I just ran the 1x benchmark on my Linux machine (Steam Linux version of ZK). After it completed, it closed ZK, but did not show any sending a report dialog. This might be the reason:
Zero-K.exe Warning: 0 : Problem reporting a bug: System.AggregateException: One or more errors occurred. ---> System.Net.WebException: Error: SecureChannelFailure (The authentication or decryption has failed.) ---> System.IO.IOException: The authentication or decryption has failed. ---> System.IO.IOException: The authentication or decryption has failed. ---> Mono.Security.Protocol.Tls.TlsException: The authentication or decryption has failed.
Full infolog:
https://pastebin.com/k9rjR0Fd

I do see this stuff in infolog_full, maybe it's helpful:

[f=-000001] ,average dt,min dt,max dt,dt_0,dt_3,dt_6,dt_9,dt_12,dt_15,dt_18,dt_21,dt_24,dt_27,dt_30,dt_33,dt_36,dt_39,dt_42,dt_45,average time,min time,max time,time_0,time_3,time_6,time_9,time_12,time_15,time_18,time_21,time_24,time_27,time_30,time_33,time_36,time_39,time_42,time_45,end unit count
[f=-000001] Engine 287 r3,0.07801547,0.02515293,0.30483779,0,23,565,899,602,737,373,391,252,64,4,1,3,1,0,0,0.07799787,0.023,0.30499998,0,26,555,898,621,723,378,388,226,88,6,1,1,2,1,0,911
[f=-000001] Engine 308 r1,0.05177741,0.01864615,0.18432684,0,1,9,207,765,1207,811,933,1174,606,87,50,20,7,2,15,0.05176713,0.018,0.134,0,0,9,204,771,1217,807,961,1003,726,104,51,8,12,3,17,934
[f=-000001] Engine 309 r2,0.08401322,0.02107306,0.41659334,1,72,756,718,715,681,158,313,234,71,11,1,0,0,0,2,0.08399519,0.02,0.41799998,1,73,747,724,721,676,153,327,204,89,14,1,0,0,0,2,934
Zero-K.exe Information: 0 : Chobbyla << SendBugReport {"Title":"GCBenchmarkv1","Description":"Benchmark by Vermind\n```,average dt,min dt,max dt,dt_0,dt_3,dt_6,dt_9,dt_12,dt_15,dt_18,dt_21,dt_24,dt_27,dt_30,dt_33,dt_36,dt_39,dt_42,dt_45,average time,min time,max time,time_0,time_3,time_6,time_9,time_12,time_15,time_18,time_21,time_24,time_27,time_30,time_33,time_36,time_39,time_42,time_45,end unit count```\n```Engine 287 r3,0.07801547,0.02515293,0.30483779,0,23,565,899,602,737,373,391,252,64,4,1,3,1,0,0,0.07799787,0.023,0.30499998,0,26,555,898,621,723,378,388,226,88,6,1,1,2,1,0,911```\n```Engine 308 r1,0.05177741,0.01864615,0.18432684,0,1,9,207,765,1207,811,933,1174,606,87,50,20,7,2,15,0.05176713,0.018,0.134,0,0,9,204,771,1217,807,961,1003,726,104,51,8,12,3,17,934```\n```Engine 309 r2,0.08401322,0.02107306,0.41659334,1,72,756,718,715,681,158,313,234,71,11,1,0,0,0,2,0.08399519,0.02,0.41799998,1,73,747,724,721,676,153,327,204,89,14,1,0,0,0,2,934```\n```"}
Zero-K.exe Information: 0 : Chobby closed connection
+0 / -0



AUrankAdminGoogleFrog
12 days ago
Did it prompt you to submit a report?
+0 / -0

FRrankduser1
12 days ago
Hello,
I have not sent a report on the first run because I moved the mouse around a bit.
The display was not fine though. First run the screen was a bit shifted to the right (minimap not completely visible) but got worst on the second run (minimap nearly completely invisible, apparently because the image was more horizontally stretched). For the third run, it was probably like the second one. I will still run it without moving the mouse to send the report.
+0 / -0

FRrankduser1
12 days ago
For the second execution of the benchmark the first looked like the other two. It was consistent. The image is a bit shifted down too. It is probably related to the special configuration used to run the benchmark.
+0 / -0


FIrankVermind
12 days ago
(edited 12 days ago)

GoogleFrog, you probably missed this bit "but did not show any sending a report dialog." So no, I saw no prompt. There's something wrong with Mono's handling of certificates probably, which prevents the sending or the dialog appearing.
+0 / -0

MXrankmoleculeman88
12 days ago
hey google look i did 7 hour today in the morning since 12:45 and le tthe machin at it like for 9:00 and it went over the 7 hours needless to say i wait 20 mins. untill i wanted to send it. try to crash the game but got aborted. i'll do it again later. but perhaps there something u can do that if the 7 hours is inacurrate. and it was more or there was a bug. becuase it contiued and i coulnd't send the crash report. i guess i lsot that. but will try again. but maybe u can help me. becuase i did the 7 hour thing, but it didn't stop at 7 hours but like 8 hours and 30 mins and probably would of contiued and went beyond the 20 games. i'm just saying that perhaps it has a bug it contiued and would of proabably never give me the crash report. i'm not sure just giving ya feedback ;D
+0 / -0



AUrankAdminGoogleFrog
12 days ago
Send me the csv found in Zero-K/benchmarks. It should take 20 times longer than the short benchmark, which may go over 7 hours if it runs slowly.
+0 / -0


USrankOflameo
10 days ago
https://github.com/ZeroK-RTS/Zero-K/issues/3325

I submitted my benchmarks. I started the Benchmark x20 before I went to works.
+0 / -0

NLrankmalric
10 days ago
Maybe is a good idea to post one/some summaries at some point (like, how many people run, how many issues you still saw, how many runs would you consider are needed, etc.). Just so that the testers get a better feeling on how they contributed (will also run it when I get back home from travelling)
+0 / -0



AUrankAdminGoogleFrog
9 days ago
We've received 8 long benchmarks and 70 short ones. The long benchmarks have been more useful because the variance between individual runs appears to be high. I've submitted an engine report as the data appears to be good enough to show that there is a regression between the engine versions that are being tested.
+1 / -0



AUrankAdminGoogleFrog
8 days ago
The issue has been resolved. Apparently kloot already made a toggle for the performance-impacting GC change, but did not mention this on the ticket until I reopened it. I'll see about making another benchmark to test the current engines performance against latest dev. This is much more difficult because the engines which are 550 commits apart do not sync, so running the same replay on them does not result in the same game. This means that the short benchmark will have to be longer.

Limited multiplayer testing is blocked by a few bugs. In my testing I saw units teleport across the map upon completion and there are some issues relating to formation moves. General testing/adoption of the dev engine is blocked by a regression in the skirmish AI. Hopefully UAranklamer is around to fix it.
+5 / -0