Crazyhouse, tournaments etc

[yolin]

Last time I tried winboard was in 2003, and only recently I have installed it again. Oh boy what an improvement!!! I especially like the fact that the different windows are now glued together and the much better support for crazyhouse (like the display of the captured pieces). It seems to have much more options overall too. So a big kudos to the programmers.

I am currently running a crazyhouse tournament using the in-build tournament manager. I have some questions/suggestions:
- Is there support for a mutual crazyhouse book? If so in which format? Some engines come with a book, others don't so I would like to have that uniform.
- are the options 'file with start positions' and 'game file with opening lines' supported for crazyhouse?
- at the end of the tournament, I got a couple (5) windows open saying "illegal first move by first program". Since I didnt watch all the games, I am not sure which program is in error. Perhaps the message could substitute "first program' with the name of the engine?
- The same actually goes for "white" and "black" on top of the board, I never know who is what. Is there an option to have the players name on either side of the board (like in e.g. http://www.thief-interface.com/target4.html). From the position of the players name it is immediately clear who is playing black and who is playing white.
- Also regarding this black/white. I find this currently very confusing. So if it is white's turn, the word white is written in white on a black background, black is then simply black on a white background. But if it is black's turn, it is also written in white on a black background and white black on a white background. Although it is consistent, it is very confusing. It is like on those IQ tests, where you have to say which color a series of words are, and then the word green is coloured blue, yellow in red , etc.. Now, as I am running lightning games, you can imagine what this does to me, when I try to figure out who is playing white and who is playing black, and sides change every second. I also think thief-style is better, where the players name doesn't change colour, and just says the name of the player (only the clock changes from blink to no blink).
- Is there an option to build a crosstable after/during the tournmaent. If not, does anyone know software which supports crazyhouse pgn to generate crosstables? I would like to post the results of my crazyhouse tournament

Thanks a lot and best regards,

yolin

[H.G.Muller]

Thanks for your appreciation! As to your questions/remarks:

- Is there support for a mutual crazyhouse book? If so in which format? Some engines come with a book, others don't so I would like to have that uniform.

WinBoard supports books in Polyglot format, and it has extended the calculation of hash keys and encoding of moves to other variants. Unfortunately, Polyglot can only make books for orthodox Chess. So it has probing code built in for the GUI book, but AFAIK no such books exist for variants (apart from a Xiangqi book that I made myself).

For now the only way to obtain such a book is use WinBoard Edit Book function to create the book by hand (typing moves and weights in each position). I am considering to equip WinBoard with a book-building function, where it would extract statistics from a loaded PGN file (only using the games currently displayed in the Game List, so you could filter by Elo, date, result etc.), and write it to a book in the required format.

- are the options 'file with start positions' and 'game file with opening lines' supported for crazyhouse?

Yes, this is a completely variant-independent feature. Of course they would have to be crazyhouse games. Note that in FEN WinBoard writes holdings between [] brackets behind the board, although I think that on reading it would understand bFEN (which writes the holdings as an extra rank) as well.

- at the end of the tournament, I got a couple (5) windows open saying "illegal first move by first program". Since I didnt watch all the games, I am not sure which program is in error. Perhaps the message could substitute "first program' with the name of the engine?

I guess this is in principle a good idea, but unfortunately not a good remedy against the problem at hand. Problems like this are quite common with engines that do not support the WB ping feature, and ignore all input when they are thinking. When such engines play against an opponent that plays a move before it resigns (another silly habit...), they have already started thinking on a reply before it becomes clear the game has ended. WinBoard informs them of the game end, and starts a new game. But the engine ignores that as it is thinking, and eventually produces a move. WinBoard has no way of knowing that this move actually was a 'leftover' from the previous game, and treats it like the first move of the newly started game. Where it most likely is illegal, and leads to a forfeit of that game even before the engine has started it!

In the PGN you might be able to see which engines suffer most from this problem, because the forfeits because of illegal move will be mentioned in the 'result details' comment. If you make sure the result comment is printed in the Game List header lines, you can even filter on the word 'forfeit' or 'illegal move' to select all games where it happened. (Games without moves would not be saved, though.)

One way to avoid it is to specify a longer 'match pause'. Another method could be to install troublesome engines with the /xreuse option, so WinBoard would kill the process of the previous game before it could produce the left-over move. Or select a tournament form where an engine would never have to play to consecutive games in the first place (do many cycles, and keep games-er-pairing (/defaultMatchGames) at 1).

- The same actually goes for "white" and "black" on top of the board, I never know who is what. Is there an option to have the players name on either side of the board (like in e.g. http://www.thief-interface.com/target4.html). From the position of the players name it is immediately clear who is playing black and who is playing white.

WinBoard has the option to display engine logos next to the clock. This doesn't help much in case where the players are not engines, e.g. when observing a FICS game. (Both sides would then show the FICS logo.) But the players names are already written in the title bar, the white player mentioned first, so it would normally be above the white clock.

- Also regarding this black/white. I find this currently very confusing. So if it is white's turn, the word white is written in white on a black background, black is then simply black on a white background. But if it is black's turn, it is also written in white on a black background and white black on a white background. Although it is consistent, it is very confusing. It is like on those IQ tests, where you have to say which color a series of words are, and then the word green is coloured blue, yellow in red , etc.. Now, as I am running lightning games, you can imagine what this does to me, when I try to figure out who is playing white and who is playing black, and sides change every second. I also think thief-style is better, where the players name doesn't change colour, and just says the name of the player (only the clock changes from blink to no blink).

I admit this confused me very much too, when I first started using WinBoard. Once you learn that reverse-video highlighting is used to indicate the active clock, so that the black background indicates the side to move, your troubles are over, however, and now it never confuses me at all. The problem is really that the highlighted area really fills all available space, so that there is no unhighlighted area around it to contrast it. In XBoard, where there is white padding between the clock widgets, and only the widgets themselves are then highlighted in black (background), it is much more obvious that the highlighting is black.

- Is there an option to build a crosstable after/during the tournmaent. If not, does anyone know software which supports crazyhouse pgn to generate crosstables? I would like to post the results of my crazyhouse tournament

There are lots of PGN-to-cross-table converters, and I don't think they care about the variant at all. (They just look for player and result tags). I think I even included a simple one (making cross tables in ascii text) in the WinBoard install (hgtbl.exe). WinBoard has an /afterGame option, where you can give a text string to be executed as a command after every game. If you specify that as "hgtbl FILENAME" it would update cross-table and standings files from the mentioned PGN file automatically after every game.

[yolin]

Thank you so much for your detailed and informative reply!

The illegal moves: they mostly come from pulsar, I will try the /xreuse option and see how that goes.

For the opening book; I think I'll go ahead with just using a set number of positions, which I consider about equal for black/white.

As for the crosstable: I was actually surprised how hard it is to find a generator. The ones I found also parsed the games to retrieved ECO code statistics and so were not usuable. I had a look for
hgtbl.exe but don't think it is included in the current distribution (at least not in the c:\winboard\ directory).

Thanks again, yolin

[H.G.Muller]

Ah, you are right! I did not include it in the WinBoard package, but in the ChessLive! HTML-viewer package. It is called hgtblB.exe there.

[yolin]

Thanks again for the help. Now I have some experience running crazyhouse tournaments with winboard. It rocks!

In case anyone needs the following, sent me a message:
- a database with 100,000 crazyhouse games with an average rating over 2200 (FICS, mostly from ficsgames.org, but also from before 2007) - good for book building
- A pgn with the 300 most played crazyhouse positions (distilled from the database) - good for engine testing

[H.G.Muller]

Do there exist tools for building crazyhouse books from PGN?

One of the things that is lingering on my to-do list is to add book-building functionality to WinBoard. But I haven't worked out any detailed design yet. I was thinking about adding a button somewhere, "Ad Game Selection to Book". (Probably in the Common Engine Options dialog, where most other book options currently are.) This option would then take all (position, move) pairs of all games selected in the current Game List, and add those to the book that is currently installed as GUI book. This subject to the 'Book Depth' parameter we already have, and perhaps to a new 'Threshold Number of Games' field for how often the position minimally has to occur to allow its incorporation. (The latter is tricky, though, when you want to slowly build up the book from several PGN files. So perhaps it is better to have a button 'Remove rare Positions', which could be used afterwards to remove positions that occurred fewer times than the specified threshold from the book file. (Or set their weight to zero, while preserving statistics for them in the 'learn' field.

Would this be a useful feature?

[Michel]

Do there exist tools for building crazyhouse books from PGN?

One of the things that is lingering on my to-do list is to add book-building functionality to WinBoard. But I haven't worked out any detailed design yet. I was thinking about adding a button somewhere, "Ad Game Selection to Book". (Probably in the Common Engine Options dialog, where most other book options currently are.) This option would then take all (position, move) pairs of all games selected in the current Game List, and add those to the book that is currently installed as GUI book. This subject to the 'Book Depth' parameter we already have, and perhaps to a new 'Threshold Number of Games' field for how often the position minimally has to occur to allow its incorporation. (The latter is tricky, though, when you want to slowly build up the book from several PGN files. So perhaps it is better to have a button 'Remove rare Positions', which could be used afterwards to remove positions that occurred fewer times than the specified threshold from the book file. (Or set their weight to zero, while preserving statistics for them in the 'learn' field.

Would this be a useful feature?

The problem is that polyglot books are not good for incremental merging since the score is relative. I proposed long ago
that the learn field rather than using it for learning information, would be used to store W/D/L statistics.

One could store the total number of games in a logaritmic way. And the W/D scores one could store relative to the
total number of games. In this way one could put quite accurate W/D/L information in 32 bit.

EDIT:

And one could indicate in the header that the learning field contains W/D/L information. I would like to add a key/value
dictionary to the header in v1.1 of the format (since the header is a tree this is trivial to do). One could then put WDL:1 in the header

[H.G.Muller]

I am not sure I get that. With 'score' you mean the move weight? Polyglot sets that to the number of half-points scored with the move, right? That should be incremental. Unless you overflow, and it rescales. But that happens only at 65,000 games. But by that time you hardly care.

[Michel]

Well the overflow has to occur only for one move for the global rescaling to happen...

Much nicer seems to be to record W/D/L information since there are plenty of bits to do it....
(48 bits if you count the score field). Then the user of the book can use this information for his
own scoring algorithm.

One idea is to let the score field overflow into the learning field. This would be backward
compatible and one would have the quantity 2*W+D with only a few bits of the learning field. With some information
in the remaining fields one could then extract approximations for W/D/L from it.

EDIT: I have an even better idea. Since the overflow occurs globally it can simply be recorded in the header....
This can be implemented independently of the use of the learning field for W/D/L information.

[H.G.Muller]

Why would you want the rescaling to be global? I certainly would not do that if I had WinBoard build book. I ould just scale the weights for the position with the overflowing move. Anything else seems needless destruction of information.

[Michel]

Why would you want the rescaling to be global? I certainly would not do that if I had WinBoard build book. I ould just scale the weights for the position with the overflowing move. Anything else seems needless destruction of information.

You are right polyglot only rescales the scores for a given position. That makes it still non-linear though. I still vote for W/D/L as is is so much more flexible.

[H.G.Muller]

Well, you can redefine the meaning of weight & learn bytes in any way you want, depending on a header flag. I propably would prefer to let WinBoard build books with traditional meaning of those fields. The occasional move that would overflow and rescaled I don't see as a real problem. If you build the book in one run, rescaling is perfetly safe. If you merge two differently rescaled books the root moves from one set of PGN might be weighted a bit more than those of the other. So what? It is not that you could not trust the weights even from just one of the merged books, if you reach weights were rescaling becomes necessary. And how many such moves can there be in the entire book?

Btw, how about a format extension where you just add 'overflow entries', in stead of recruting other fields in the same entry? If e2-e4 is played more than 65,000 times, just create a second e2-e4 entry for that position, immediately behind the other one, which contains the scaling shift to be applied to the primary entry and all other moves from that position, so that it effectively becomes a floating-point representation. This is compatible with existing probing routines, which would see the extra entry as yet another move, with weights so small that it will never be played (as the scaling shift will never be very large, and the mantisse will be normalized to be >= 32K). It might look a bit weird in the Edit Book window to see the same move twice, but it would not be fatal. Just adopt the rule that if any of the moves from a position ocurs twice in consecutive entries, the second one contains the shift to apply to all moves from the position. Then you don't have to fool around if on merging books another move gets the largest weight, or another move becomes the first one.

[yolin]

Sorry for being slow in replying: as far as I know, they are no public programs which can build a crazyhouse book. Some engines support a book, but these all use different formats. Bpgnview can build an opening three from crazyhouse/bughouse games, but that is not meant to be used for engines, but rather to analyse opening lines.

A book building feature for variant games would be an excellent addition to winboard. Ideally, one would then like an option to filter moves for occurrence (a move should have been played a few dozens of times) and win percentage (excluding the low ones) and perhaps average rating. Like you discussed, incremental would be nice, but seems much more complex to implement.

[H.G.Muller]

I have some more about this, and the following seems a good way to implement it:

WinBoard could get four new buttons in some dialog (probably Common Engine Options, because that is where the other book stuff is as well). These would be

Import Book
Pressing it would load the currently specified Polyglot book, and merge it with a memory buffer. (I.e. it adds the weights to the weights of (position, move) pairs already in the buffer, and would just copy new pairs.

Clear Buffer
Pressing it clears the memory buffer.

Add Games
All games currently selected in the Game List are merged into the memory buffer

Save Buffer
Calls up a file-browse window to save the memory buffer as a Polyglot book. The Book Depth setting in the dialog would be respected. Filtering on number of games would take place here.

This would allow a user to merge as many prevvious books and PGN files as he wants. Slight problem is that in data imported from books don't have any information on the playing frequence, but only on the number of half-points scored with it (with conventional meaning of the weights). I can't think of anything better than to use the weight also for the number of games for now (i.e. assume a 50% score). If the book has a header hat indicates the learning info fields contain this info, it should use it, of course.

If we assume this is a possible format, we would probably want to be able to create it as well. So there should be a checkbox "Save WDL in learning field". And probably also a checkbox to force 'scalingless saving'. We should have an entry field for the minimum number of games as well.

Perhaps it should be possible to select a 'lossless' save format, where even sub-threshold moves would be saved (because later merging could bring them above the threshold). We could save them with weight 0, so the book would still be usable, and save the actual stats in the learning field (only available when you save in the WDL format). This suggests we should define the WDL format such that when the weight vanishes, the full WDL can be stored in the learning fields (e.g. 2 bytes for total games, 1 byte for W and D each). For true weights you would probably want to use the full 16 bits for wins (or (W+D)*3/2), as the weight would be 2*W+D, so you could derive both W and D from them.

[yolin]

Don't know enough to comment on the specifics of the polyglot format. I hope others join the discussion; it seems important enough if this going to be in winboard and be useful for all variants! If you are going to implement the book building stuff - which would be fantastic, i would be very happy to test it.

[H.G.Muller]

I made some progress on this:

The latest WinBoard beta (at http://hgm.nubati.net/WinBoard-4.7.beta.zip ) includes a menu item File -> Create Opening Book. This item will cause all opening positions (upto the specified Book Depth) in a loaded PGN file to be collected in a memory buffer, and then write out the buffer as Polyglot book (with the name specified for the GUI book, so be careful not to overwrite an existing book!). The buffer is not cleared, so you could repeat the procedure (in the same session) with another PGN file, to add that to the buffer too, and write a book of the combined games.

See also: http://www.talkchess.com/forum/viewtopi ... 690#494690 .