Computation time reduction?

Feedback
#1

Hi everyone!

Yesterday during the final I said few things about this, this post is just to get into the details.

Aside from messing around with ML algos, I was pretty much away for season 4. About a week before the deadline, I decide to put back my Genoracle algo online. I was a bit surprised because the decisions it made had an unfinished color, some moves were clearly sub optimal (like sending lone pings, which has also been seen during the final). So I dig a little to inspect this.

I remembered that C1 said that for season 4 computation time will be better regulated. As my strategy is very hungry in computation power, I assumed it would affect me (it was very likely that my algos were using the opponent’s time as they said). At the end of season 3, my algo could do between 40k and 120k simulations per turn and was expecting for it be divided by at most 2 with the season 4 system. So I was quite surprised when I noticed that it was now between 10k and 30k.

So I looked back at my old posts and simulators to find some numbers.

On January 2019, my algo was computing 9k simulations per turn on my computer and 40k on the server.
Today (January 2020), my algo can compute 80k simultations per turn on my computer. But it can only achieve 20k on the server.

This confirm what I remembered: The server, in past seasons, had always outperformed my computer in computation powers. So I was a bit surprised that it reversed completely.

Has anyone also noticed this? Or I am just finding excuses for the losses of my creations ? :laughing:

On a side note, as it is now, the strategy behind Genoracle is unsustainable. I supposed it is my fault for making such a computational heavy strategy.
For the final I named it “Genoracle_is_not_dead” as a wish for it to have some kind of last show and I’m glad it made it in!

Anyways, it was so interesting to see the beautiful algos you all made this season! Thanks everyone for the effort and talent you put in this :grinning:. It is always a learning experience to see your creations playing!

2 Likes
#2

I have an i7-6820HQ, and I consistently see that the server does only about half the number of simulations as my laptop does, which is fairly inline with what you seem to be getting now after the rebalancing.

1 Like
#3

So the promise of better computation time consistency for Season 4 kind of failed because the approach we used with Docker caused all sorts of problems so we had to revert it.

So computation should be the same throughout the seasons on paper.

What I suspect could be happening is that the algos you are versing hog more/less CPU. If this is the case we could tell by seeing how many simulations happen between a ranked game with a “simple” algo, and a ranked game with a “high performance” algo. If the performance even against a “simple” dumb algo is bad then maybe there really is something wrong and we can spend some time looking at the settings. If you already have this data thats great too!

How do you know how many simulations your algo is doing in ranked matches though since you can’t read error logs unless it crashes? If these are the numbers from Playground matches, then that doesn’t really matter unlike our ranked matches Playground servers are essentially big servers that handle dozens of matches each so is super variable the compute you get as its based on the traffic and how busy playground is.

1 Like
#4

Would it be possible to add some sort of way for algos to log some statistics like the number of simulations run? Logging statistics could be useful not just for debugging issues like this one, but also for improving our algos

2 Likes
#5

It would be a fairly complicated feature, but something I will think about, maybe there is an easier way.

For now, you could always just have your algo crash if it wants to report something, and then you can read your error log that way :upside_down_face: Just be careful since algos stop playing matches after a certain number of crashes.

#6

That’s a fun idea, I can definitely find a use for that for at least some parameter tuning. If I only crash with like 10% probability then hopefully my algo could reach high-enough elo anyways that it plays some matches against good opponents.

#7

I had this idea for reporting, that I never actually had a use case for:
Add a “reporting / debug mode” flag and if enabled, make a specific move on the board on location not otherwise used (lets say on turn 10)
So with 1-4 “bit” of information you can log things like:

  • successful first turn rush
  • asymmetric, left attacking enemy
  • more then 500 sims per turn

You can then upload 2 algos, one in debug mode, get all the replays from it and run an automated parsing for the specific data.

3 Likes
#8

My apologies to have left this thread to die just after starting it. But now that my studies have ended and with the current situation, I have finally some time!

It’s now a bit late to make any comparison with previous seasons as things have changed again but for the future, it could be interesting.

using Genoracle_is_not_dead_10 as it was in season 4 (with minor patches to make it work under the current season) here is some data:

Season 5, 2020/04/23
raw number of simulations per turn

On 95 matches (1506 turns)

by matches:
simulations

all matches together:
all_simulations

For all matches the mean number of simulation is 39 448 and the standard deviation is 11 080.

number of simulations per turn vs opponent computation time

regression

As shown above, with a simple regression we get a r2 of 0.11. This means that there is some correlation between the number of simulations made by Genoracle and the time used by its opponent. For those who aren’t familiar with the Oracle series, one of their features is to be as greedy as possible on computation time. So this could mean that the computation power decrease if both opponents use a significant portion of the allocated time.
However as we know it, correlations needs to be handle carefully. For example, the opponents using the longest time to compute could also be the ones generating the hardest game boards for Oracle’s simulator.

To collect the data, I made the algo crash randomly as @C1Junaid suggested.

[edit 2020/04/23: addition of the number of simulations versus opponent time graph]

2 Likes