The Rise of AI: A Weekend Read. The story moves from early machine learning to the moment computers learned to see. Here is the third chapter of the series.
How an unlikely alliance of neural networks, ImageNet and NVIDIA’s gaming hardware opened an eye
On a winter morning in Toronto, daylight arrives as a rumour. Snow gathers along the kerbs, salt whitens the pavements, and the towers of the university emerge slowly from a sky the colour of old paper.
Inside one of those buildings, Geoffrey Hinton was working on an idea that had spent most of his career in a climate colder than the weather.
Hinton was a University Professor in the Department of Computer Science—a title of considerable distinction. Yet the branch of artificial intelligence to which he had devoted his life remained near the frigid frontier of the discipline.
Unlike most AI researchers of his generation, Hinton did not believe that intelligence could be constructed simply by writing enough rules into a machine. He believed that useful knowledge should arise through learning, from the bottom up.
The algorithms he championed were called artificial neural networks. The name invited comparison with the brain, although the resemblance was always loose. Instead of receiving an exhaustive catalogue of instructions, a neural network learned from examples, gradually adjusting the strengths of the connections within itself.
A child does not arrive in the world carrying a manual that defines a face, a voice or a cat. It sees, hears, touches, misjudges and is corrected. Through experience and guidance, certain patterns slowly become familiar.
A neural network attempted a crude mathematical version of that process.
This did not mean that the machine sprang to life and taught itself unaided. Human beings still designed the network, chose its task and supplied the material from which it learned. But they did not have to write a complete definition of cat-ness. Show the network enough labelled pictures of cats, together with enough pictures of things that were not cats, and it might gradually discover useful distinctions for itself.
At first, it discovered very little.
The early neural networks were slow, temperamental and painfully limited. They promised minds and delivered toys.
The attention of the field moved elsewhere.
Hinton did not.
He and a scattered fellowship of researchers continued experimenting with networks containing multiple layers. These would eventually become known as deep neural networks. The hope was that each layer might learn a progressively more complicated representation of the same material.
An image entered the network not as a dog, a car or a teapot, but as columns of numbers describing the colour and brightness of its pixels. The first layers might become sensitive to edges and contrasts. Later layers might detect corners, curves and textures. Deeper still, combinations resembling eyes, wheels, fur or leaves might begin to appear.
At the beginning, the network guessed blindly. It produced wrong answers more often than right ones.
Then came guidance.
The network was shown an image whose correct label was already known. Its prediction was compared with that label, and the difference between them became an error. The error was carried backwards through the network so that each connection could be adjusted according to how much it had contributed to the failure.
This was backpropagation: a mathematical system for distributing blame.
In 1986, David Rumelhart, Hinton and Ronald Williams published an influential paper demonstrating how backpropagation could train multilayer neural networks to develop useful internal representations. The network made a prediction, measured how wrong it had been, adjusted itself and tried again.
Failure was not an interruption of the learning process. Failure was the information from which learning arose. A child who is never allowed to wobble will never discover balance. In much the same way, the neural network required the freedom to be wrong before it could learn to be right.
But an elegant method was not the same thing as a working revolution. Neural networks still lacked three things they required in abundance: deeper architectures, vast quantities of data and enormous computational power. For years, Hinton’s machines remained too small to vindicate his ambitions and too peculiar to command the centre of artificial-intelligence research.
Elsewhere, the field presented a warmer and more confident face.
At the Massachusetts Institute of Technology, Marvin Minsky had helped establish the Artificial Intelligence Laboratory and had become one of the commanding figures of the young science. MIT possessed famous researchers, ambitious machines, institutional money and the attention of the world. It was one of the places to which brilliant young people travelled when they wanted to glimpse the future.
Much of the work celebrated in such laboratories approached intelligence through symbols, representations, rules and logical operations. Tell a machine what the world contains. Give it procedures for manipulating that knowledge. Construct a sufficiently elaborate architecture of reason, and intelligence might emerge.
Minsky himself was more complicated than this neat division suggests. He had built an early neural-network machine and believed intelligence would require many different methods. But MIT stood close to the illuminated centre of artificial intelligence, while Hinton’s neural networks survived at its colder margins.
MIT attracted disciples, visitors and prodigies. Among the young people drawn into Minsky’s orbit was a teenage inventor named Ray Kurzweil.
His role in this story would not become clear for decades.
Back in Toronto, the winter eventually began attracting company.
In 2003, a teenager named Ilya Sutskever knocked on Hinton’s door. He had been earning money during the summer by cooking fries and wanted to know whether he could work on artificial intelligence instead.
Hinton gave him several research papers to read. Sutskever returned having seen implications that had escaped more experienced researchers. His answers were so clear—and his conviction so intense—that Hinton invited him into the laboratory.
Alex Krizhevsky arrived by a less dramatic route. He was a gifted programmer who was reluctant to leave university for an ordinary coding job, so he contacted Hinton about doctoral research. Quiet and extraordinarily persistent, he possessed the engineering temperament needed to turn difficult mathematical ideas into efficient working software.
Sutskever supplied the conviction that larger neural networks, trained on more data, would eventually overwhelm the established methods.
Krizhevsky could make that conviction run on a machine.
Together with Hinton, they now possessed the theory, the determination and the engineering skill. What they still lacked were sufficient data and sufficient computational power.
Then, in 2009, the data arrived.
Fei-Fei Li and her collaborators introduced ImageNet, an immense, carefully organized library of annotated images designed to teach machines about the visual world.
ImageNet was not glamorous in the manner artificial intelligence preferred to imagine itself.
It was not a robot. It was not a talking machine. It did not play chess beneath television lights or roll through laboratories on mechanical legs. It was, in essence, a colossal labelled warehouse of pictures.
Fei-Fei Li and her collaborators had begun with a simple but radical conviction. If a machine were ever to understand the visual world, it could not be trained on a few thousand carefully chosen photographs.
It would need millions.
When ImageNet was introduced in 2009, it already contained more than three million images. It would eventually grow to over fourteen million, organized into more than twenty thousand categories derived from WordNet.
There were cats divided into breeds, dogs divided into still more breeds, musical instruments, household objects, fruits, insects, flowers, vehicles and tools. The disorder of the visible world had been arranged into an enormous educational catalogue for machines.
But a database alone could not establish whether machines were getting better at seeing. Researchers needed an examination—a common set of questions against which every competing system could be judged.
That examination became the ImageNet Large Scale Visual Recognition Challenge.
There was no stadium, no starting pistol and no audience watching the contestants work. The competition unfolded silently inside laboratories around the world, among humming computers, spinning fans, unfinished code and progress bars that could remain motionless for hours.
In June 2012, registered teams received approximately 1.28 million training images, already divided among one thousand categories. A separate collection of fifty thousand images allowed them to test and refine their systems.
In July, the organizers released one hundred thousand test images.
The answers to those images were withheld.
The machines would have to confront pictures they had never encountered before: animals photographed from strange angles, objects partially hidden behind other objects, badly lit rooms, crowded scenes and creatures whose differences might challenge even a knowledgeable human observer.
The competition contained several tasks. A system might be asked to classify the main object in an image, locate it within the frame or distinguish among 120 closely related breeds of dog.
For the principal classification task, each machine was permitted five guesses.
This was partly an acknowledgment of the untidiness of photographs. A picture labelled “dog” might also contain a ball, a sofa and a person’s leg. If the correct answer appeared anywhere among the machine’s five most confident predictions, the answer was accepted.
Every incorrect set of guesses increased the system’s top-five error.
The final submission deadline was September 30 at eleven o’clock at night, Greenwich Mean Time. Until then, teams could train their systems, test them against the validation images, rewrite their code and begin again.
For the Toronto group, ImageNet presented an almost insolent challenge.
The dataset contained precisely what Hinton’s neural networks had lacked for decades: examples in abundance. But abundance had created a new scarcity: computation.
A deep network would have to pass through more than a million images repeatedly. Every pass required billions of mathematical operations. Each of the network’s connections had to contribute to a prediction, receive its portion of the blame and adjust itself before the process began again.
A conventional CPU could perform this work. Given enough CPUs and enough time, an entire cluster could perform it well.
But Hinton’s university group could not afford a vast installation of high-end processors. They did not possess the computing resources of Google, Microsoft or the largest American laboratories. Even if they assembled a smaller CPU cluster, every experiment could consume so much time that a mistaken design choice might cost weeks.
They had the theory.
They had the data.
They could not afford the machine.
Krizhevsky began searching for another way.
He had already been experimenting with the use of graphics processors for neural-network calculations. A GPU lacked the general versatility of a high-end CPU, but it contained hundreds of smaller workers capable of performing similar mathematical operations simultaneously.
ImageNet did not require one brilliant processor solving a single intricate problem.
It required an army performing the same kinds of calculation millions of times.
The instrument they needed might not be sitting inside an expensive university computing centre. It might be advertised to gamers.
After several days of calculation and hesitation, Krizhevsky opened Newegg.
The website was the fluorescent electronics aisle of the digital age: processors, motherboards, cooling fans and graphics cards arranged beneath photographs, technical specifications and the judgments of strangers.
Among them was NVIDIA’s GeForce GTX 580.
It was a consumer gaming card, built to produce smoke, reflections, shadows and explosions without allowing the imaginary world on the screen to stutter. Each card possessed three gigabytes of memory. More importantly, each could be programmed through CUDA.
Krizhevsky ordered two.
A few days later, the boxes arrived in Toronto.
The two cards together cost a fraction of the high-end CPU installation the group could not afford. They were not a supercomputer. They were pieces of gaming hardware that could be placed inside an ordinary workstation.
But they offered something more valuable than prestige.
They offered parallel labour.
Krizhevsky began writing a highly optimized CUDA implementation of the operations required by a convolutional neural network. The complete model was too large to fit conveniently inside either GTX 580, so he divided it between the two cards.
Each GPU carried roughly half the computational burden. The two halves communicated fully at some layers and remained largely separate at others. Information crossed between the cards only where the benefit justified the delay.
The resulting network contained approximately sixty million adjustable parameters and 650,000 artificial neurons.
Each complete training run required five or six days.
During those days, the cards worked continuously. Their fans spun. Heat accumulated inside the machine. More than a million photographs passed through the network: dogs, birds, cars, tools, fruits, furniture and faces.
At the end of a run, Krizhevsky examined the errors, changed the architecture or adjusted the training procedure, and began again.
Five or six more days.
Then five or six more.
For months, the network failed to distinguish itself. It took roughly half a year merely to equal the strongest existing ImageNet systems. Krizhevsky continued refining the code. Sutskever continued pressing the conviction that the network could go further. Hinton advised, questioned and occasionally resisted.
Slowly, the error rate began to fall.
The system would later become known as AlexNet. For the competition, however, the Toronto researchers submitted it under another name: SuperVision.
On September 30, their predictions were uploaded to the ImageNet evaluation server. For each of the one hundred thousand test images, the file contained five guesses arranged in descending order of confidence.
Then the team waited.
Preliminary results were released to the participants on October 8. Four days later, researchers gathered at the European Conference on Computer Vision in Florence, Italy, for the ImageNet workshop.
The SuperVision entry had recorded a top-five error rate of 15.3 per cent when supplemented with additional ImageNet training data. A version trained only on the material supplied for the competition recorded 16.4 per cent.
The second-best team scored 26.2 per cent.
In a discipline accustomed to victories measured in fractions, the Toronto entry had opened a gap of more than ten percentage points.
Researchers studied the numbers with disbelief. No recent computer-vision competition had prepared them for a result of that magnitude. A university group unable to purchase a grand CPU installation had defeated the established systems using two graphics cards designed for gamers.
This was not an improvement. It was a rupture.
ImageNet had not been solved—not completely and not finally. But it had been broken open decisively enough that everyone understood the direction of the field had changed.
AlexNet did not depend upon human specialists designing every visual feature in advance. It learned useful representations directly from pixels. Its shallow layers detected elementary patterns. Its intermediate layers combined them into textures and parts. Its deeper layers assembled those parts into increasingly recognizable objects.
Deep neural networks had already begun producing striking improvements in speech recognition. After ImageNet, however, the larger scientific world could no longer dismiss the formula as a curiosity confined to Hinton’s winter outpost.
Researchers purchased GPUs. Laboratories rewrote their software in CUDA. Companies began recruiting the small community of specialists who understood deep learning. Image recognition changed. Speech recognition changed. Translation and recommendation systems followed.
In time, language itself would be submitted to the same formula.
For NVIDIA, AlexNet was a revelation.
CUDA had spent years appearing to be a strange and commercially barren wager. Now it had found the problem for which it seemed almost providentially designed. The new form of artificial intelligence did not merely benefit from graphics processors.
It was hungry for them.
NVIDIA had not set out to build the engine room of artificial intelligence. It had manufactured chips for drawing game worlds, then created software that allowed those chips to perform general mathematics. After that, it had waited—without knowing exactly what it was waiting for—until the right problem arrived.
The right problem was learning.
I had not yet learned that lesson myself.
To me, a graphics card still belonged principally to the world of games. In 2013, shortly after completing my postgraduate studies and after considerable thought about the expense, I finally bought one of my own.
It was a Zotac NVIDIA GeForce GTX 650 Ti, an affordable gaming card released in October 2012.
It would be both my first graphics card and my last.
History had already transformed the object of our youthful desire into something else. The graphics card was no longer merely an indulgence that made explosions brighter and water more convincing. It had become a scientific instrument: a machine for training other machines.
During the years that followed, demand for GPUs arrived from several directions at once. Gamers wanted richer virtual worlds. Researchers wanted larger neural networks. Cryptocurrency miners, particularly during later speculative booms, filled rooms with graphics cards and emptied them from retail shelves.
Cards that did appear were frequently sold at two or three times their expected prices. The graphics hardware I had once coveted gradually moved beyond my budget.
The furnace had found too many customers.
Hinton, Sutskever and Krizhevsky soon discovered that they themselves had become objects of extraordinary demand.
The three formed a company called DNNresearch. It had three people, no conventional product and almost no history. What it possessed was more valuable: the knowledge required to reproduce one of the most consequential breakthroughs in modern computer science.
Google wanted it.
So did Microsoft and the Chinese technology company Baidu.
The fourth bidder was a little-known London startup called DeepMind.
Hinton conducted an auction by email from a hotel room near Lake Tahoe, where the researchers had gathered for an academic conference. The bidding rose through tens of millions of dollars. Microsoft withdrew and then returned. Baidu and Google continued pursuing the group. DeepMind, still almost unknown outside a narrow circle, attempted to compete with companies many times its size.
When the bidding reached forty-four million dollars, Hinton stopped the auction.
The companies were prepared to continue. But Hinton had decided that finding the right home for the research mattered more than extracting the final possible dollar.
He chose Google.
The eventual purchase price was never publicly disclosed. In March 2013, the University of Toronto announced that Google had acquired DNNresearch. Hinton would divide his time between Google and the university, while Krizhevsky and Sutskever joined Google’s researchers as research scientists.
The acquisition brought several currents of the story together.
Ray Kurzweil—the teenage prodigy who had once entered Marvin Minsky’s orbit at MIT—had already joined Google in December 2012. He arrived as a director of engineering to work on machine learning and language processing.
Kurzweil had spent years predicting the eventual convergence of human and machine intelligence. He supplied prophecy, a timetable and almost limitless ambition.
The Toronto group brought a method that had now prevailed in open competition.
If Google was collecting rings of power, it now possessed two.
Ray Kurzweil, the prophet of the coming posthuman age, was inside the company.
So was Geoffrey Hinton, the priest of bottom-up intelligence, whose machines had finally learned to see.
Meanwhile, the third ring was being forged in London.
Comments
Post a Comment