Is it possible to delete a move from a Polyglot book?

[H.G.Muller]

No, there is no restriction as to what these numbers sum up to. I saw that in the performance.bin book by Marc Lacrosse they even add up to 3 for the opening position: e2e4, d2d4 and c2c4 are the only moves, each with weight 1. This means they will be played with 33% probability each.

When Polyglot builds a book from a set of PGN games, the weights are simply the number of half-points scored with this move. Which could be anything. They are stored in the book as 16-bit unsigned integers, though, which means they cannot be larger than 65,535. But anything from 0 to 65,535 would work in WinBoard.

The percentage is simply the weight divided by the sum of the weights, calculated when the window is initialized (i.e. they are not automatically adapted when you change the weights, but they would when you reopen the window after OK'ing a change, and the data is re-read from the book). They are provided precisely because the weights can be anything, which makes it difficult for the user to asses at a glance what the playing probability is.

[K Inuen]

To edit the book, you would simply replace the weights (7800 or 65520 in this case) by what you want it to be, by typing into the Edit Book window, and press OK. The info in the window is then written back into the book. (You'd better not write something invalid there, like illegal moves or nonsense messages, as in this primitive implementaton there is no validity checking before writing the info back to the book, and it will only write back the stuff upto the point where it encounters something invalid. So you could lose moves from the book by this!) The learning info tells you that you played two games with 1.d4, and won them both (4 half-points), and 1 game with 1.e4, which was a draw (1 half-point). Results when the opponent played white are not incorporated. You could also edit that before you OK the dialog (but there seems little point in doing that).

Thanks for the response. I am inclined to think these numerical values assigned on each opening is for the probability of being played (as assigned in %) but I'm not exactly sure. But to be sure, what are these numerical values with 65520 for d4 & e4 and 7800 for g3, Nf3 & c4?

Also to be sure, can the Edit Book option be used to edit book-openings (or change preferred openings), during ICS-mode, by assigning high / low probabilities (%) to each openings without having to log-off to use SCID for this?

Thanks.

[H.G.Muller]

Idon't think you would ever have to log off, to edit the opening book. Whether you do it with SCID or with WinBoard. The weights are the nominal realtive playing probabilities. WinBoard used them if 'book variation' is set to 50. At other values of book variation it raises them to a power (ranging from 0 to infinity) first.

[K Inuen]

Idon't think you would ever have to log off, to edit the opening book. Whether you do it with SCID or with WinBoard. The weights are the nominal realtive playing probabilities. WinBoard used them if 'book variation' is set to 50. At other values of book variation it raises them to a power (ranging from 0 to infinity) first.

Okay thanks for confirming the Book Edition in relation to ICS mode. I'm glad you mentioned the Book Variation option which puzzled me in this aspect; considering my openings (for White: d4, e4, g3, Nf3, & c4), what happens if Book Variation is set to, say, 10? Set to 40?, Set to 100?

Can Book Variation be set to a 'tenth' of number +5, such as 15, 45, 75....?

[H.G.Muller]

The exponent is derived from the book variation x as (100-x)/x. So for x=10 this would be 9, and the weights would be taken to the 9th power. That means weights 40-30-30 would become 87-6.5-6.5, i.e. a very large emphasis on the best move. Any integer setting is possible, even 0 or 100.

[K Inuen]

Okay thanks H.G.Muller. I was also hoping Michel would reconsider things on-behalf of PolyGlot but haven't heard much from him (I know he's busy). I guess this auto book tuning/probing on-the-fly by PolyGlot is unlikely. He usually responds to bugs, suggestions, requests and willing to try out new features.

[H.G.Muller]

Well, if you are impatient, and Michel is busy, you could try the version at

http://hgm.nubati.net/polyglot66hgm.zip .

You still have to put the BookLearn=true before the book filename, but I did make the learning there dependent on if the engine played all moves for one side (irrespective of if it is in force mode or what color it playes at game end), and have it use the learn info as I suggested when BookLearn=true.

Beware, it is completely untested.

[K Inuen]

Thank you H.G.Muller. I'll test it and give feed backs.

By the way, this is how I have my book features arranged:

Code: Select all

Book=true
BookLearn=true
BookRandom=true
BookDepth=99
BookTreshold=5
BookFile=C:\PolyGlot\Bin\PCG-Beta3-X.Bin

Notice the BookLearn=true is 4 steps ahead of BookFile. Is this still fine with PolyGlot and the BookLearn=true for the learn feature to work?

[H.G.Muller]

Yes, it should be fine as long as it is somewhere before the specification of the book file.

The problem is that on reading the BookFile option, Polyglot opens the book immediately (to check if the given name is valid; otherwise it switches off book). And if at that point the option BookLearn was not set yet, it will open it read-only, and later learning will cause a crash. But if it is already set at that point, it will open the file read/write, as it should for book learning.

[K Inuen]

H.G.Muller,

An observation: When using your Polyglot 1.4.66b, it takes a while for either side to make its first, second, third....moves. In some cases, it can take black about 38 seconds (or more) just to make its first move. When switched to Michel's Polyglot (1.4.66b), the opening-delay is gone.

Is this opening-delay an expected behaviour with your version of Polyglot 1.4.66b? If so, how or what parameter should be tweaked to reduce the opening-delay?

[H.G.Muller]

Oh, you are right. My modification of the move-selection code (to incorporate the learn info) was fatally flawed: all weights ended up as zero, (due to rounding), and apparently Polyglot has trouble selecting a move in that case.

The version I uploaded now should fix it.

[K Inuen]

The version I uploaded now should fix it.

Thanks for the quick fix on notice. Is the fix from the same previous (above) link?

[Michel]

I am busy right now but I think the whole BookLearn thing is currently fatally flawed.

It would be much better (for a number of obvious reasons) if Polyglot saved the learning information as a separate book (specified in the ini file). I.e. something like

BookLearn=true
LearnFile=learn.bin
UseLearn=true

One reason is that I fear the (possibly accidental) proliferation of polluted books (i.e. performance.bin "enhanced" with learning information). Up to now this was not a problem since Polyglot did not use the information.

[H.G.Muller]

Well, a separate UseLearn option could be useful. The patch I made makes the use of learn info subject to the BookLearn option, so people that would swicth BookLearn off would not be bothered by it. Only people that want to use BookLearn would have to be worried about not starting with a polluted book.

But I don't think the danger is very bad, and the solution you propose is quite inefficient (as space is already reserved for the learn info, and would presumably stay reserved for backward comptibility). And the two-books approach is very error prone, because the books can get out of phase on editing (adding or deleting a move). That seems much worse, and more difficult to check by the user than the problem of books poluted with prior learn info. (To check that, the user can simply open the entry for the opening position for editing, and see if there is learn info in there.)

I think it would be more useful to simply have an option in Polyglot to purge a book of all learn info. E.g. just a button in the Engine Settings dialog, 'Purge Book-Learn Info'. Possibly subject to the popup of a warning through a 'telluser' command, and a second 'Yes, I am sure' button (or pressing the original Purge button two more times).

What I actually would like is to have a book with ECO codes in the learn field, for use as a GUI book in WinBoard.

[Michel]

I generally cringe at the idea of writing directly on books created by other people:-( There might be even copyright
issues if the book is free for distribution but not for modification. If I were to distribute a book then I certainly would not like that.

Well, a separate UseLearn option could be useful. The patch I made makes the use of learn info subject to the BookLearn option, so people that would swicth BookLearn off would not be bothered by it. Only people that want to use BookLearn would have to be worried about not starting with a polluted book.

UseLearn is necessary as using the learning information is quite different from generating it. The latter you would only do under very controlled circumstances regarding opponent rating and time control.

But I don't think the danger is very bad, and the solution you propose is quite inefficient (as space is already reserved for the learn info, and would presumably stay reserved for backward comptibility). And the two-books approach is very error prone, because the books can get out of phase on editing (adding or deleting a move). That seems much worse, and more difficult to check by the user than the problem of books poluted with prior learn info. (To check that, the user can simply open the entry for the opening position for editing, and see if there is learn info in there.)

Well PG books are very small so the 32 extra bit/entry are not a problem. Besides they could be recycled for other uses.


I don't think there is a danger of the root book and the learn book going out of sync as they are independent (they don't have to contain the same moves or positions). The idea that the user would check every entry of the book to see what the learning information is does not seem feasible.

[H.G.Muller]

I am not sure we have the same perspective on this book learning. IMO it is not a way to automate generation of good-quality books for general use. The type of learning applied in Polyglot is way too primitive for that. The main application I see is to adapt an aready good book to peculiarities of a specific engine (weeding out lines leading to positions it does not understand well, as evidened by their results). I don't see the need for very controlled conditions for that. (And you might want to have a diffeent book for blitz and long TC anyway.)

In that perspective, I also don't see any serious legal problems. Learn info is not part of the book. It happens to be stored interleaved with it, on the same medium. Well, so are the sectors on my hard disk of copyrighted files and the trash can. It makes no sense to release a book with learn info in it; you might as well change the weights. The list of zeros constituting the learn info in that case is not copyrightable.

[Michel]

The main application I see is to adapt an aready good book to peculiarities of a specific engine (weeding out lines leading to positions it does not understand well, as evidened by their results). I don't see the need for very controlled conditions for that. (And you might want to have a diffeent book for blitz and long TC anyway.)

Well you definitely need controlled conditions. It is like engine parameter tuning. You tune against a carefully selected
set of engines.

Furthermore, apart from my somewhat general objection against writing on the book itself, learning information seems more tied to an engine than to a book (it is in fact independent of the book). Therefore, it still seems more useful to me to store it in a separate file.

(In fact a better way to do learning might be as follows: save the games in a pgn-file, use pgn-extract to extract the useful ones
and then use the book merge command of polyglot to make a new book. Possibly the merge command should be enhanced to make this really work)

[Michel]

I looked a bit on the web for ChessBase's handling of book learning. As far as I understand they also modify the original book.
Does anyone know how they deal with copyright? Or is it possible for the original author to write protect a book so that
it cannot be modified?

[H.G.Muller]

(In fact a better way to do learning might be as follows: save the games in a pgn-file, use pgn-extract to extract the useful ones
and then use the book merge command of polyglot to make a new book. Possibly the merge command should be enhanced to make this really work)

I don't expect this to be a better way, as you cannot use the information while it is laying the games. The best way is probably to use a Monte-Carlo UTC-like algorithm, that does some minimal amount of exploration on bad lines to verify that they are bad, and then focuses the main effort on the lines that are promising, to explore them more deeply.

[Michel]

I don't expect this to be a better way, as you cannot use the information while it is laying the games.

It is better in the sense that you use pgn-extract to select the interesting games after the fact (e.g. using opponent rating and time control). With the current book learning setup you have no way to select the games you want/don't want. So you have to set up
things in such a way that you only get the correct type of games, which is clearly inconvenient.

I assume UCT would be very useful for the creation of books. But that looks like an entirely different issue.