Loading...
  OR  Zero-K Name:    Password:   

Forum index  > News   >

Benchmarking Required

27 posts, 3268 views
Post comment
Filter:    Player:  
Page of 2 (27 records)
sort


5 years ago

It is nearing time to update the engine that runs Zero-K (the Spring Engine), but first we need to pin down the performance issues that affected many users when such an update was last attempted. To facilitate this we have created an easy to use automatic benchmarker. It currently contains benchmarks for our most pressing issue, but more are likely to be added in the future.

If you are the kind of person who does not use their computer while sleeping then consider running a benchmark and submitting the results to help us out.
+7 / -0
5 years ago
okay sounds like fun
+0 / -0


5 years ago
The build system seems to be down, so the stable with the benchmark is not live.
+0 / -0


5 years ago
Well MXrankmoleculeman88 submitted a benchmark, even though I am unable to get my non-dev ZK install to update.
+0 / -0
5 years ago
I'll do this sometime in the next few days
+0 / -0


5 years ago
The release is now live.
+0 / -0
I just ran the 1x benchmark on my Linux machine (Steam Linux version of ZK). After it completed, it closed ZK, but did not show any sending a report dialog. This might be the reason:
Zero-K.exe Warning: 0 : Problem reporting a bug: System.AggregateException: One or more errors occurred. ---> System.Net.WebException: Error: SecureChannelFailure (The authentication or decryption has failed.) ---> System.IO.IOException: The authentication or decryption has failed. ---> System.IO.IOException: The authentication or decryption has failed. ---> Mono.Security.Protocol.Tls.TlsException: The authentication or decryption has failed.
Full infolog:
https://pastebin.com/k9rjR0Fd

I do see this stuff in infolog_full, maybe it's helpful:

[f=-000001] ,average dt,min dt,max dt,dt_0,dt_3,dt_6,dt_9,dt_12,dt_15,dt_18,dt_21,dt_24,dt_27,dt_30,dt_33,dt_36,dt_39,dt_42,dt_45,average time,min time,max time,time_0,time_3,time_6,time_9,time_12,time_15,time_18,time_21,time_24,time_27,time_30,time_33,time_36,time_39,time_42,time_45,end unit count
[f=-000001] Engine 287 r3,0.07801547,0.02515293,0.30483779,0,23,565,899,602,737,373,391,252,64,4,1,3,1,0,0,0.07799787,0.023,0.30499998,0,26,555,898,621,723,378,388,226,88,6,1,1,2,1,0,911
[f=-000001] Engine 308 r1,0.05177741,0.01864615,0.18432684,0,1,9,207,765,1207,811,933,1174,606,87,50,20,7,2,15,0.05176713,0.018,0.134,0,0,9,204,771,1217,807,961,1003,726,104,51,8,12,3,17,934
[f=-000001] Engine 309 r2,0.08401322,0.02107306,0.41659334,1,72,756,718,715,681,158,313,234,71,11,1,0,0,0,2,0.08399519,0.02,0.41799998,1,73,747,724,721,676,153,327,204,89,14,1,0,0,0,2,934
Zero-K.exe Information: 0 : Chobbyla << SendBugReport {"Title":"GCBenchmarkv1","Description":"Benchmark by Vermind\n```,average dt,min dt,max dt,dt_0,dt_3,dt_6,dt_9,dt_12,dt_15,dt_18,dt_21,dt_24,dt_27,dt_30,dt_33,dt_36,dt_39,dt_42,dt_45,average time,min time,max time,time_0,time_3,time_6,time_9,time_12,time_15,time_18,time_21,time_24,time_27,time_30,time_33,time_36,time_39,time_42,time_45,end unit count```\n```Engine 287 r3,0.07801547,0.02515293,0.30483779,0,23,565,899,602,737,373,391,252,64,4,1,3,1,0,0,0.07799787,0.023,0.30499998,0,26,555,898,621,723,378,388,226,88,6,1,1,2,1,0,911```\n```Engine 308 r1,0.05177741,0.01864615,0.18432684,0,1,9,207,765,1207,811,933,1174,606,87,50,20,7,2,15,0.05176713,0.018,0.134,0,0,9,204,771,1217,807,961,1003,726,104,51,8,12,3,17,934```\n```Engine 309 r2,0.08401322,0.02107306,0.41659334,1,72,756,718,715,681,158,313,234,71,11,1,0,0,0,2,0.08399519,0.02,0.41799998,1,73,747,724,721,676,153,327,204,89,14,1,0,0,0,2,934```\n```"}
Zero-K.exe Information: 0 : Chobby closed connection
+0 / -0


5 years ago
Did it prompt you to submit a report?
+0 / -0

5 years ago
Hello,
I have not sent a report on the first run because I moved the mouse around a bit.
The display was not fine though. First run the screen was a bit shifted to the right (minimap not completely visible) but got worst on the second run (minimap nearly completely invisible, apparently because the image was more horizontally stretched). For the third run, it was probably like the second one. I will still run it without moving the mouse to send the report.
+0 / -0

5 years ago
For the second execution of the benchmark the first looked like the other two. It was consistent. The image is a bit shifted down too. It is probably related to the special configuration used to run the benchmark.
+0 / -0
GoogleFrog, you probably missed this bit "but did not show any sending a report dialog." So no, I saw no prompt. There's something wrong with Mono's handling of certificates probably, which prevents the sending or the dialog appearing.
+1 / -0
5 years ago
hey google look i did 7 hour today in the morning since 12:45 and le tthe machin at it like for 9:00 and it went over the 7 hours needless to say i wait 20 mins. untill i wanted to send it. try to crash the game but got aborted. i'll do it again later. but perhaps there something u can do that if the 7 hours is inacurrate. and it was more or there was a bug. becuase it contiued and i coulnd't send the crash report. i guess i lsot that. but will try again. but maybe u can help me. becuase i did the 7 hour thing, but it didn't stop at 7 hours but like 8 hours and 30 mins and probably would of contiued and went beyond the 20 games. i'm just saying that perhaps it has a bug it contiued and would of proabably never give me the crash report. i'm not sure just giving ya feedback ;D
+0 / -0


5 years ago
Send me the csv found in Zero-K/benchmarks. It should take 20 times longer than the short benchmark, which may go over 7 hours if it runs slowly.
+0 / -0

5 years ago
https://github.com/ZeroK-RTS/Zero-K/issues/3325

I submitted my benchmarks. I started the Benchmark x20 before I went to works.
+0 / -0
5 years ago
Maybe is a good idea to post one/some summaries at some point (like, how many people run, how many issues you still saw, how many runs would you consider are needed, etc.). Just so that the testers get a better feeling on how they contributed (will also run it when I get back home from travelling)
+0 / -0


5 years ago
We've received 8 long benchmarks and 70 short ones. The long benchmarks have been more useful because the variance between individual runs appears to be high. I've submitted an engine report as the data appears to be good enough to show that there is a regression between the engine versions that are being tested.
+1 / -0


5 years ago
The issue has been resolved. Apparently kloot already made a toggle for the performance-impacting GC change, but did not mention this on the ticket until I reopened it. I'll see about making another benchmark to test the current engines performance against latest dev. This is much more difficult because the engines which are 550 commits apart do not sync, so running the same replay on them does not result in the same game. This means that the short benchmark will have to be longer.

Limited multiplayer testing is blocked by a few bugs. In my testing I saw units teleport across the map upon completion and there are some issues relating to formation moves. General testing/adoption of the dev engine is blocked by a regression in the skirmish AI. Hopefully UAranklamer is around to fix it.
+5 / -0
So we will have to manually submit the benchmark? It looks like it's crashing still? Or maybe it sent, I am not sure, it didn't look like in the logs.
-rw-rw-r-- 1 grats grats 4.3K Nov 20 11:05 zerokBenchmark02.sdz
-rw-rw-r-- 1 grats grats 359M Nov 20 11:05 39d82368167c5ae54652065fe37d5445.sdz
-rw-rw-r-- 1 grats grats 149M Nov 20 11:04 16633d26101f49d2b24c5f2d7ab51232.sdz


I have these 3 that seem to have been generated. Not sure if they're part of the bench or just the named one is.
NVM I found the benchmarks CSV too.

I'm running the long test now though, I have ryzen 1700X and AMDGPU on Linux, I figured it was kind of a unique setup.
Do you still need these benchmarks?
+0 / -0

5 years ago
Yeah, he probably does needs another round because of the new engine release to check for regressions.

Your setup is similar to mine. I am also on Linux using AMD Graphics with AMDGPU but I am using an Radeon RX 470.
+0 / -0
Ryzen 1700X is my CPU

I'm using 7850 GPU, zero-K crashes on any nvidia GPU I try (under linux and mono), not sure what that is about.
I just bought a RX 580 though.

https://drive.google.com/file/d/120B6c0YtztWjTMkTxYWLryrdwN8ntwkj/view?usp=sharing

Do you want windows benches? I have 17 computers here.. It's very easy to do if this helps you out.
From reading the data you're just trying to get outliers of data or something in the numbers? Hardware is irrelevant?
+1 / -0
Page of 2 (27 records)