Programming Computers to Play Games

?

Backgammon is the oldest board game in the world. It was first played in ancient Mesopotamia, starting around 3000 B.c. The rules of backgammon were codified in the seventeenth century, and the game has changed little since. The same can’t be said about the players of the game. One of the best backgammon players in the world is now a software program. In the early 1990s. Gerald Tesauro, a computer programmer at IBM, began developing a new kind of artificial intelligence (Al). At the time, most Al programs relied on the brute computational power of microchips. This was the approach used by Deep Blue, the powerful set of IBM mainframes that managed to defeat chess grand master Garry Kasparoy in 1997. Deep Blue was capable of analyzing more than two hundred million possible chess moves per second, allowing it to consistently select the optimal chess strategy. (Kasparov’s brain, on the other I hand. evaluated only about five moves per second.) But all this strategic firepower consumed a lot of energy: while playing chess. Deep Blue was a ire hazard and required specialized heat-dissipating equipment so that it didn’t burst into flames. Kasparov, meanwhile, barely broke a sweat. That’s because the human brain is a model of efficiency: even when it’s deep in thought, the cortex consumes less energy than a lightbulb.

While the popular press was celebrating Deep Blue’s stunning achievement-a machine had outwitted the greatest chess player in the world!-Tesauro was puzzled by its limitations. Here was a machine capable of thinking millions of times faster than its human opponent, and yet it had barely won the match. Tesauro realized that the problem with all conventional Al programs, even brilliant ones like Deep Blue’s, was their rigidity. Most of Deep Blue’s intelligence was derived from other chess grand masters, whose wisdom was painstakingly programmed into the machine. (IBM programmers also studied Kasparov’s previous chess matches and engineered the software to exploit his recurring strategic mistakes.) But the machine itself was incapable of learning. Instead, it made decisions by predicting the probable outcomes of several million different chess moves. The move with the highest predicted value was what the computer ended up executing. For Deep Blue, the game of chess was just an endless series of math problems.

Of course, this sort of artificial intelligence isn’t an accurate model of human cognition. Kasparov managed to compete on the same level as Deep Blue even though his mind had far less computational power. Tesauro’s surprising insight was that Kasparov’s neurons were effective because they had trained themselves. They had been refined by decades of experience to detect subtle spatial patterns on the chessboard. Unlike Deep Blue, which analyzed every possible move. Kasparov was. able to instantly isolate his best options and focus his mental energies on evaluating only the most useful strategic alternatives.

Tesauro set out to create an Al program that acted like Garry Kasparov. He chose backgammon as his model and named the program TD-Gammon. (The TD stands for temporal difference.) Deep Blue had been preprogrammed with chess acumen, but Tesauro’s software began with absolutely zero knowledge. At first, its backgammon moves were entirely random. It lost every match and made stupid mistakes. But the computer didn’t remain a novice for long; TD-Gammon was designed to learn from its own experience. Day and night, the software played backgammon against itself, patiently learning which moves were most effective After a few hundred thousand games of backgammon, TD-Gammon was able to defeat the best human players in the world.

How did the machine turn itself into an expert? Although the mathematical details of Tesauro’s software are numbingly complex, the basic approach is simple. TD-Gammon generates a set of predictions about how the backgammon game will unfold. Unlike Deep Blue, the computer program doesn’t investigate every possible permutation. Instead, it acts like Garry Kasparov and generates its predictions from its previous experiences. The software compares these predictions to what actually happens during the backgammon game. The ensuing discrepancies provide the substance of its education, and the software strives to continually decrease this error signal. As a result, its predictions constantly increase in accuracy, which means that its strategic decisions get more and more effective and intelligent.

1.Backgammon is the oldest board game in the world. It was first played in ancient Mesopotamia, starting around 3000 B.c. The rules of backgammon were codified in the seventeenth century, and the game has changed little since. The same can’t be said about the players of the game. One of the best backgammon players in the world is now a software program. In the early 1990s. Gerald Tesauro, a computer programmer at IBM, began developing a new kind of artificial intelligence (Al). At the time, most Al programs relied on the brute computational power of microchips. This was the approach used by Deep Blue, the powerful set of IBM mainframes that managed to defeat chess grand master Garry Kasparoy in 1997. Deep Blue was capable of analyzing more than two hundred million possible chess moves per second, allowing it to consistently select the optimal chess strategy. (Kasparov’s brain, on the other I hand. evaluated only about five moves per second.) But all this strategic firepower consumed a lot of energy: while playing chess. Deep Blue was a ire hazard and required specialized heat-dissipating equipment so that it didn’t burst into flames. Kasparov, meanwhile, barely broke a sweat. That’s because the human brain is a model of efficiency: even when it’s deep in thought, the cortex consumes less energy than a lightbulb.