Crafty loses too much!

[Gabor Szots]

Geschrieben von:/Posted by: Gabor Szots at 07 March 2000 13:15:50:
Hi, everyone,
I have to tell you, I am astonished seeing my WB-tournament results (GS-2 and GS-3 at Frank's site): I can't believe Crafty 17.7-17.9 lost 7 of its 18 games and concluded both tournaments with a meagre 50%.
Time control was game in 30 minutes, and the games were played on my Celeron 433 with 32 MB hash. I think this is far from blitz, and I suspect statistical considerations will not provide a satisfying explanation either. It is remarkable (look at GS-3) that Crafty could only win against programs of the second half of the field.
For me that means that either the rest have reached Crafty's strength or Crafty has fallen back. From version 17.6 there's a significant (25% !) decrease in NPS measured by Crafty benchmarks. Perhaps that is the reason.
I am looking forward to Dann's internet tournament played under even longer time controls. After that we'll see more clearly.
Cheers,
Gabor

[WYx]

Geschrieben von:/Posted by: WYx at 07 March 2000 13:42:22:
Als Antwort auf:/As an answer to: Crafty loses too much! geschrieben von:/posted by: Gabor Szots at 07 March 2000 13:15:50:

Hi, everyone,
I have to tell you, I am astonished seeing my WB-tournament results (GS-2 and GS-3 at Frank's site): I can't believe Crafty 17.7-17.9 lost 7 of its 18 games and concluded both tournaments with a meagre 50%.
Time control was game in 30 minutes, and the games were played on my Celeron 433 with 32 MB hash. I think this is far from blitz, and I suspect statistical considerations will not provide a satisfying explanation either. It is remarkable (look at GS-3) that Crafty could only win against programs of the second half of the field.
For me that means that either the rest have reached Crafty's strength or Crafty has fallen back. From version 17.6 there's a significant (25% !) decrease in NPS measured by Crafty benchmarks. Perhaps that is the reason.
I am looking forward to Dann's internet tournament played under even longer time controls. After that we'll see more clearly.
Cheers,
Gabor

Hi
Crafty on my Tours (BLITZ) plays on the average.
Perhaps 17.09 is a bit weaker, but not striking.
Regards
WYx

[Mogens Larsen]

Geschrieben von:/Posted by: Mogens Larsen at 07 March 2000 13:53:35:
Als Antwort auf:/As an answer to: Crafty loses too much! geschrieben von:/posted by: Gabor Szots at 07 March 2000 13:15:50:

Hi, everyone,
I have to tell you, I am astonished seeing my WB-tournament results (GS-2 and GS-3 at Frank's site): I can't believe Crafty 17.7-17.9 lost 7 of its 18 games and concluded both tournaments with a meagre 50%.
Time control was game in 30 minutes, and the games were played on my Celeron 433 with 32 MB hash. I think this is far from blitz, and I suspect statistical considerations will not provide a satisfying explanation either. It is remarkable (look at GS-3) that Crafty could only win against programs of the second half of the field.
For me that means that either the rest have reached Crafty's strength or Crafty has fallen back. From version 17.6 there's a significant (25% !) decrease in NPS measured by Crafty benchmarks. Perhaps that is the reason.
I am looking forward to Dann's internet tournament played under even longer time controls. After that we'll see more clearly.
Cheers,
Gabor

The new engines are closer to the strength of Crafty than ever before in my opinion. The added chess knowledge is an advantage when playing longer games. I've noticed the difference between my tournaments ML-4 (2*40/40) and ML-6 (4*90/G). I'm sure the amount of games will benefit Crafty greatly as the chances of defeat in one of two games is increasing with likes of AnMon, LG2000 and SOS improving. The development of Crafty is aimed at 40/2h as far as I know, which is the reason for the adding of chess knowledge and the reduction in NPS.
Best wishes...
Mogens

[Jouni Uski]

Geschrieben von:/Posted by: Jouni Uski at 07 March 2000 14:03:02:
Als Antwort auf:/As an answer to: Re: Crafty loses too much! geschrieben von:/posted by: Mogens Larsen at 07 March 2000 13:53:35:

The development of Crafty is aimed at 40/2h as far as I know, which is the >reason for the adding of chess knowledge and the reduction in NPS.
Best wishes...
Mogens

Let's wait for next SSDF list. Then we finally! get idea of Crafty's strength.
My quess about 2600 SSDF elo (it has + score against Fritz5.32 200MMX.
Jouni

[Dann Corbit]

Geschrieben von:/Posted by: Dann Corbit at 07 March 2000 19:13:20:
Als Antwort auf:/As an answer to: Crafty loses too much! geschrieben von:/posted by: Gabor Szots at 07 March 2000 13:15:50:

Hi, everyone,
I have to tell you, I am astonished seeing my WB-tournament results (GS-2 and GS-3 at Frank's site): I can't believe Crafty 17.7-17.9 lost 7 of its 18 games and concluded both tournaments with a meagre 50%.
Time control was game in 30 minutes, and the games were played on my Celeron 433 with 32 MB hash. I think this is far from blitz, and I suspect statistical considerations will not provide a satisfying explanation either. It is remarkable (look at GS-3) that Crafty could only win against programs of the second half of the field.
For me that means that either the rest have reached Crafty's strength or Crafty has fallen back. From version 17.6 there's a significant (25% !) decrease in NPS measured by Crafty benchmarks. Perhaps that is the reason.
I am looking forward to Dann's internet tournament played under even longer time controls. After that we'll see more clearly.

Crafty works best under the following conditions:
1. Crafty gets its own machine (not two programs on the same machine).
2. Pondering is allowed.
3. Against computers, use the computer opening book.
4. Against humans, use the human opening book.
Did you set it up like that?

[Gabor Szots]

Geschrieben von:/Posted by: Gabor Szots at 08 March 2000 07:43:20:
Als Antwort auf:/As an answer to: Re: Crafty loses too much! geschrieben von:/posted by: Dann Corbit at 07 March 2000 19:13:20:

Hi, everyone,
I have to tell you, I am astonished seeing my WB-tournament results (GS-2 and GS-3 at Frank's site): I can't believe Crafty 17.7-17.9 lost 7 of its 18 games and concluded both tournaments with a meagre 50%.
Time control was game in 30 minutes, and the games were played on my Celeron 433 with 32 MB hash. I think this is far from blitz, and I suspect statistical considerations will not provide a satisfying explanation either. It is remarkable (look at GS-3) that Crafty could only win against programs of the second half of the field.
For me that means that either the rest have reached Crafty's strength or Crafty has fallen back. From version 17.6 there's a significant (25% !) decrease in NPS measured by Crafty benchmarks. Perhaps that is the reason.
I am looking forward to Dann's internet tournament played under even longer time controls. After that we'll see more clearly.

Crafty works best under the following conditions:
1. Crafty gets its own machine (not two programs on the same machine).
2. Pondering is allowed.
3. Against computers, use the computer opening book.
4. Against humans, use the human opening book.
Did you set it up like that?

[Mogens Larsen]

Geschrieben von:/Posted by: Mogens Larsen at 08 March 2000 12:04:56:
Als Antwort auf:/As an answer to: Re: Crafty loses too much! geschrieben von:/posted by: Gabor Szots at 08 March 2000 07:43:20:

No.
1. I have only one machine.
2. Pondering was off.
3. Maybe I don't know how to use different books against computers and humans. My Crafty.rc looks like this:
hash 24M
hashp 4M
cache 2M
computer
drawscore=0
log off
book on
resign 9
tbpath=C:\SAKK\TB.emd
egtb
My Crafty directory includes book.bin (13 MB), bookc.bin (132k) and books.bin (132k). All have been downloaded from your site. Crafty seems to use the big book (lines are often more than 40 ply long).

This a fine setup for playing Crafty on one computer. The hash is fine (maybe 8Mb for hashp would be better) and it uses computer lines from book.
Best wishes...
Mogens

[Gabor Szots]

Geschrieben von:/Posted by: Gabor Szots at 08 March 2000 14:50:19:
Als Antwort auf:/As an answer to: Re: Crafty loses too much! geschrieben von:/posted by: Mogens Larsen at 08 March 2000 12:04:56:
My Crafty.rc looks like this:

hash 24M
hashp 4M
cache 2M
computer
drawscore=0
log off
book on
resign 9
tbpath=C:\SAKK\TB.emd
egtb
My Crafty directory includes book.bin (13 MB), bookc.bin (132k) and books.bin (132k). Crafty seems to use the big book (lines are often more than 40 ply long).

This a fine setup for playing Crafty on one computer. The hash is fine (maybe 8Mb for hashp would be better) and it uses computer lines from book.
Best wishes...
Mogens

Thanks, Mogens. In fact, I have taken this setup from the config file on Frank's site. I apply it mechanically. I don't even know what hashp is for. Some programs appear to use only one hash, others more. I once read (if memory serves me well, in Gromit's setup file) that to use more than 2 MB for hashpawn doesn't make much sense. Perhaps programmers use hashp in different ways.
A clarification on this matter would be welcome - at least by me. (Maybe on another forum - to how much extent may we consider a topic Winboard related?)
Best regards,
Gabor

[Mogens Larsen]

Geschrieben von:/Posted by: Mogens Larsen at 08 March 2000 15:48:14:
Als Antwort auf:/As an answer to: Re: Crafty loses too much! geschrieben von:/posted by: Gabor Szots at 08 March 2000 14:50:19:

Thanks, Mogens. In fact, I have taken this setup from the config file on Frank's site. I apply it mechanically. I don't even know what hashp is for. Some programs appear to use only one hash, others more. I once read (if memory serves me well, in Gromit's setup file) that to use more than 2 MB for hashpawn doesn't make much sense. Perhaps programmers use hashp in different ways.
A clarification on this matter would be welcome - at least by me. (Maybe on another forum - to how much extent may we consider a topic Winboard related?)
Best regards,
Gabor

If you use Franks setup there should be about 32Mb for each engine, which is why 24Mb hash+8Mb hashp is reasonable. I prefer the program to divide the hash itself, otherwise there's too much difference in setup from tournament to tournament. However, I don't think you'll see a large increase in strength due to 4Mb extra.
Best wishes...
Mogens

[Dann Corbit]

Geschrieben von:/Posted by: Dann Corbit at 09 March 2000 02:21:01:
Als Antwort auf:/As an answer to: Re: Crafty loses too much! geschrieben von:/posted by: Gabor Szots at 08 March 2000 07:43:20:
[snip]

No.
1. I have only one machine.
2. Pondering was off.

All the other settings look fine. If you want crafty to be a world-beater, you need two machines (or play against an internet opponent on FICS or ICC). Crafty's clock management is built for pondering and it does much better when allowed to ponder.
Dr. Hyatt is almost militant against contests with crafty not pondering on a single machine. But (despite his protests) crafty plays well under those conditions anyway. But just not nearly as good as when the program gets a whole machine (preferably with multiple CPU's!) and pondering is turned on.
[snip]


My FTP site

[Gabor Szots]

Geschrieben von:/Posted by: Gabor Szots at 09 March 2000 15:31:38:
Als Antwort auf:/As an answer to: Re: Crafty loses too much! geschrieben von:/posted by: Dann Corbit at 09 March 2000 02:21:01:

[snip]
If you want crafty to be a world-beater, you need two machines (or play against an internet opponent on FICS or ICC). Crafty's clock management is built for pondering and it does much better when allowed to ponder.
Dr. Hyatt is almost militant against contests with crafty not pondering on a single machine. But (despite his protests) crafty plays well under those conditions anyway. But just not nearly as good as when the program gets a whole machine (preferably with multiple CPU's!) and pondering is turned on.
[snip]

Does not the same apply to all programs? For comparison purposes, don't we have to use similar hardware for all programs?

[Aaron]

Geschrieben von:/Posted by: Aaron at 09 March 2000 18:01:28:
Als Antwort auf:/As an answer to: Re: Crafty loses too much! geschrieben von:/posted by: Gabor Szots at 09 March 2000 15:31:38:
One thing I've noticed from the programmers of chess engines , they like to claim 2 things...
1)Their program is stronger in longer time controls and weak at Blitz
Well, this can be tested easily in matches etc..Personally i always found (vague impression actually) that crafty is super strong/far ahead of the other winboard engines at Blitz and a lot more vulnerable at longer time limits where gormit, SOS,anmon begin to have a almost 50% chance..

2) Their program works best/better with pondering on..better hardware etc..
Leaving aside the first claim, the second seems obvious since pondering on/better hardware will make a stronger program. But are there programs that are more handicapped than usual/other programs with ponder off/poor hardware..
Could we somehow test this??Test suites(that can't work ) or games or whatever?? Maybe self play with itself on 2 computers, 1 with ponder-off another ponder on...Not sure if that will prove anything if it leads to a 100% onesided match..

[Dann Corbit]

Geschrieben von:/Posted by: Dann Corbit at 09 March 2000 20:01:21:
Als Antwort auf:/As an answer to: Re: Crafty loses too much! geschrieben von:/posted by: Gabor Szots at 09 March 2000 15:31:38:

[snip]
If you want crafty to be a world-beater, you need two machines (or play against an internet opponent on FICS or ICC). Crafty's clock management is built for pondering and it does much better when allowed to ponder.
Dr. Hyatt is almost militant against contests with crafty not pondering on a single machine. But (despite his protests) crafty plays well under those conditions anyway. But just not nearly as good as when the program gets a whole machine (preferably with multiple CPU's!) and pondering is turned on.
[snip]

Does not the same apply to all programs? For comparison purposes, don't we have to use similar hardware for all programs?

When you make assumptions about how your program is going to be used, it changes the way you make design decisions.
It is certainly a testable theory. A couple hundred games should provide a clear answer if anyone is interested to do that.
Dr. Hyatt feels very strongly that crafty is assuming to be able to ponder and if you disable that, it does not play nearly as well (not just a 50% drop off in think time -- somehow more damaging than that).
I have not tested it. I am only going by the impressions of the tool's author.


My FTP site

[Bryan K]

Geschrieben von: / Posted by: Bryan K at 15 March 2000 03:29:09:
Als Antwort auf: / As an answer to: Re: Crafty loses too much! geschrieben von: / posted by: WYx at 07 March 2000 13:42:22:

Hi, everyone,
I have to tell you, I am astonished seeing my WB-tournament results (GS-2 and GS-3 at Frank's site): I can't believe Crafty 17.7-17.9 lost 7 of its 18 games and concluded both tournaments with a meagre 50%.
Time control was game in 30 minutes, and the games were played on my Celeron 433 with 32 MB hash. I think this is far from blitz, and I suspect statistical considerations will not provide a satisfying explanation either. It is remarkable (look at GS-3) that Crafty could only win against programs of the second half of the field.
For me that means that either the rest have reached Crafty's strength or Crafty has fallen back. From version 17.6 there's a significant (25% !) decrease in NPS measured by Crafty benchmarks. Perhaps that is the reason.
I am looking forward to Dann's internet tournament played under even longer time controls. After that we'll see more clearly.
Cheers,
Gabor

Hi
Crafty on my Tours (BLITZ) plays on the average.
Perhaps 17.09 is a bit weaker, but not striking.
Regards
WYx

You are very right