So with these classifiers and unambiguous accuracy numbers when compared to ground truth, it’s dead simple to evaluate success. But what about this checkers game? Unlike the pong from pixels example, my game won’t be playing against a functioning, trained AI. It will be playing against itself. Assuming it’s even capable of learning with meaningful success, its win/loss ratio should still hover at around 1. So what to do?

I’ve given this a good deal of thought and here’s the plan. I need bootstrap metrics, so I will start by having the system learn against itself for some reasonably large number of epochs. I’m assuming W/L will hover near 1. Then, after a certain number of epochs, I will have the system play against a freshly initialized version of itself (i.e. one that is playing a near random game, and which I will call Checkpoint 0) for some reasonably large number of games. If there is a W/L that is meaningfully above 1, I will consider the first battle won: I have created a NN that is learning to play checkers. I will also save that improved NN as Checkpoint 1. I will then continue to train the NN against itself for a similarly large number of epochs and then test it against Checkpoint 1 and Checkpoint 0. Success will be a meaningfully >1 W/L over Checkpoint 1, and an even larger one over Checkpoint 0. Failure will be no meaningfully >1 W/L over Checkpoints 1 and 0.

Assuming success, I’m figuring at some point things will level out, and at that point, it will be time to start switching in improved models.