TCEC Opening Book creation guide

[Martin Thoresen]

Now found here:
http://www.tcec-chess.net/opening_book.php

If works for me so I thought I'd share it.

PS: TCEC News will now be found here:
http://www.facebook.com/tcec.chess (you can read without being registered to facebook)

Best,
Martin

[crystalclear]

You can change the mnv= and mxv= parameters to widen or tighten the evaluation window. For TCEC, I am allowing an advantage within 0.15 pawns of zero.

Am I right in thinking that the idea with your opening book is to create balanced openings for a chess engine competition, so that neither engine starts the real game with an advantage?

Or are you just saying that an engine using the opening book should not come out of the opening with more than a 0.15 pawn disadvantage?


I have been working on creation of my own opening book, but had planned on mentioning it later rather than now. But you having started an opening book thread, I'll mention it now for contrast.

The aim with my book is not to have a balanced opening, but to come out of the opening with a large advantage if possible. So for certain lines, the intention is to store white moves only, for other lines black moves only. The opening book goes for depth rather than breadth. I figure the larger the depth, the larger the potential advantage, and the more thinking time spared and available for the middle game and endgame , with the triple benefit of
1. better score
2. less future moves to think about
3. more time to think about them

I wanted some sort of measure for the quality of my opening book.
I have devised a measure of my own. An Out of Book to Checkmate ratio. I had planned to mention it when I got this to 1:1, but I'd rather have a discussion about opening books than wait until my book is better.

Today, to write this posting, I did the first real measurement, and I have a ratio of about 12:1, though I'd need to test for longer to get some accuracy for the figure.

The idea is as follows.
Look at a chess game and imagine that your program had played for one of the sides using the opening book, eg White.
There are three possibilities.

1. The program remains in book for the duration of the game being analyzed. (Stay In Book)
2. The program gets taken out of book by the opponent playing an unforeseen strange move. (Taken Out Of Book)
3. The program's book move for a game position for that player are different to what was actually played. (Play Out Of Book)

Since my opening book is not very wide, a random reference game usually contains a move that differs from the one stored in my opening book. (Play Out Of Book)
The program would have simply played differently. These games are of low interest to me at present.

Of the remaining games, I am interested in how long my program can stay inside its opening book.

Here is a little test script I ran today, and the results.

Code: Select all

set opening_book {F:\book.bin}
set pgnfile {FICS_Jan_2012.pgn}

uci "setoption name Opening Book File value $opening_book"

set games 0
foreachgame {tag moves result} $pgnfile \
{
   incr games
   set oob [oob plycount White $moves]
   if {($games < 10) || ($oob == "Taken Out Of Book")} {puts "Game number $games: $oob (plycount = $plycount)"}
   if {$oob eq "Stay In Book"} \
      {puts "Game number $games: $oob (plycount = $plycount) Checkmate? [checkmate [make $moves]], Result $result"}
   if {$games == 200} break
}

puts "====================="

set games 0
set taken_out_of_book 0
set play_out_of_book 0
set stay_in_book 0

foreachgame {tag moves} $pgnfile \
{
   incr games
   set oob [oob plycount White $moves]
   if {$oob eq {Taken Out Of Book}} {incr taken_out_of_book}
   if {$oob eq {Play Out Of Book}}  {incr play_out_of_book}
   if {$oob eq {Stay In Book}}      {incr stay_in_book}
   if {$games == 100} break
}

puts "Games             $games"
puts "Taken Out Of Book $taken_out_of_book"
puts "Play Out Of Book  $play_out_of_book"
puts "Stay In Book      $stay_in_book"

puts "====================="

set games 0
set taken_out_of_book 0
set play_out_of_book 0
set stay_in_book 0
set checkmates(total) 0
set checkmates(1-0) 0
set checkmates(0-1) 0
set checkmates(1/2-1/2) 0

foreachgame {tag moves result} $pgnfile \
{
   incr games
   set oob [oob plycount White $moves]
   if {$oob eq {Taken Out Of Book}} {incr taken_out_of_book}
   if {$oob eq {Play Out Of Book}}  {incr play_out_of_book}
   if {$oob eq {Stay In Book}}      \
   {
      incr stay_in_book
      if {[checkmate [make $moves]]} \
      {
         incr checkmates(total)
         incr checkmates($result)
      }
   }
   if {($taken_out_of_book + $checkmates(total) == 25)} break
}

puts "Games             $games"
puts "Taken Out Of Book $taken_out_of_book"
puts "Play Out Of Book  $play_out_of_book"
puts "Stay In Book      $stay_in_book"
puts "Checkmates        $checkmates(total)"
puts " 1-0              $checkmates(1-0)"
puts " ½-½              $checkmates(1/2-1/2)"
puts " 0-1              $checkmates(0-1)"


Code: Select all

Game number 1: Play Out Of Book (plycount = 2)
Game number 2: Play Out Of Book (plycount = 2)
Game number 3: Play Out Of Book (plycount = 4)
Game number 4: Play Out Of Book (plycount = 2)
Game number 5: Play Out Of Book (plycount = 8)
Game number 6: Play Out Of Book (plycount = 0)
Game number 7: Play Out Of Book (plycount = 0)
Game number 8: Play Out Of Book (plycount = 0)
Game number 9: Play Out Of Book (plycount = 0)
=====================
Games             100
Taken Out Of Book 0
Play Out Of Book  100
Stay In Book      0
=====================
Games             14377
Taken Out Of Book 23
Play Out Of Book  14247
Stay In Book      107
Checkmates        2
 1-0              2
 ½-½              0
 0-1              0


The script looks at game from a Jan 2012 FICS PGN file.
The first part shows that most of the games on FICS don't follow my opening book.
eg the first game, differs on ply=2. White played an opening book move on ply 0 but something else on ply 2.
In the third game, white followed the opening book on ply 0 and ply 2, but played something else on ply 4.
In game six, white played moves in my opening book on ply 0, ply 2, ply 4 and ply 6, but not on ply 8.
In games seven to ten, white didn't even play an opening in my book. (He probably opened d4 and my book has 1. e4 for white, like Bobby Fischer.)

The next part of the script was an attempt to see what percentage of FICS game fall into each category, so I looked at the first 100 games in the Jan 2012 file.
All were games where White played moves outside of my opening book's white moves.

The third part of the script looks at games until it has found 25 games where either
1. The opponent takes us out of book (Out of Book) or
2. we stay in book until a checkmate. (Stay In Book)
(I added the checkmate test, as there are games where we stay in book just because someone resigns very early for whatever reason.)
Also, it'd be nice to know that if you stay in the opening book until a checkmate, that it's us that have caught an opponent off guard and not vice versa.
In hindsight it should have been an end-of-game test really, to spot stalemates, repetition, & 50 move rule games as drawn.

Anyway, here is ratio I was after 23:2.

Games 14377
Taken Out Of Book 23
Play Out Of Book 14247
Checkmates 2

My opening book will get taken out of book about twelve times as often as it would trap a typical FICS player with one of its known checkmate games, played from start to finish without even invoking the search engine. I am aiming to get the ratio to 1:1. I think the ratio is actually much closer, but the small sample of 25 relevent games from the 14377 looked at isn't enough for proper statistic; it is only a demonstration of what I shall measure as an opening book quality metric.