at what they’ve been able to produce already: cars that drive
themselves, computers that detect cancer, machines that
instantly translate spoken language. And look at this charming British scientist talking about gradient descent in high-dimensional spaces!
It’s only when you leave the room that you remember:
these “deep learning” systems are still pretty dumb, in spite
of how smart they sometimes seem. A computer that sees a
picture of a pile of doughnuts piled up on a table and captions it, automatically, as “a pile of doughnuts piled on a table”
seems to understand the world; but when that same program
sees a picture of a girl brushing her teeth and says “The boy is
holding a baseball bat,” you realize how thin that understanding really is, if ever it was there at all. Neural nets are just
thoughtless fuzzy pattern recognizers, and as useful as fuzzy
pattern recognizers can be—hence the rush to integrate them
into just about every kind of software—they represent, at
best, a limited brand of intelligence, one that is easily fooled.
A deep neural net that recognizes images can be totally stymied when you change a single pixel, or add visual noise that’s
imperceptible to a human. Indeed, almost as often as we’re
finding new ways to apply deep learning, we’re finding more
of its limits. Self-driving cars can fail to navigate conditions
they’ve never seen before. Machines have trouble parsing sentences that demand common-sense understanding of how the
Deep learning in some ways mimics what goes on in the
human brain, but only in a shallow way—which perhaps
explains why its intelligence can sometimes seem so shallow.
Indeed, backprop wasn’t discovered by probing deep into the
brain, decoding thought itself; it grew out of models of how
animals learn by trial and error in old classical-conditioning
experiments. And most of the big leaps that came about as it
developed didn’t involve some new insight about neuroscience;
they were technical improvements, reached by years of mathematics and engineering. What we know about intelligence is
nothing against the vastness of what we still don’t know.
David Duvenaud, an assistant professor in the same
department as Hinton at the University of Toronto, says deep
learning has been somewhat like engineering before physics.
“Someone writes a paper and says, ‘I made this bridge and it
stood up!’ Another guy has a paper: ‘I made this bridge and
it fell down—but then I added pillars, and then it stayed up.’
Then pillars are a hot new thing. Someone comes up with
arches, and it’s like, ‘Arches are great!’” With physics, he says,
“you can actually understand what’s going to work and why.”
Only recently, he says, have we begun to move into that phase
of actual understanding with artificial intelligence.
Hinton himself says, “Most conferences consist of making
minor variations … as opposed to thinking hard and saying,
‘What is it about what we’re doing now that’s really deficient?
What does it have difficulty with? Let’s focus on that.’”
It can be hard to appreciate this from the outside, when
all you see is one great advance touted after another. But the
latest sweep of progress in AI has been less science than engi-
neering, even tinkering. And though we’ve started to get a
better handle on what kinds of changes will improve deep-
learning systems, we’re still largely in the dark about how those
systems work, or whether they could ever add up to something
as powerful as the human mind.
It’s worth asking whether we’ve wrung nearly all we can
out of backprop. If so, that might mean a plateau for progress
in artificial intelligence.
If you want to see the next big thing, something that could
form the basis of machines with a much more flexible intelligence, you should probably check out research that resembles what you would’ve found had you encountered backprop
in the ’80s: smart people plugging away on ideas that don’t
really work yet.
A few months ago I went to the Center for Minds, Brains,
and Machines, a multi-institutional effort headquartered at
MIT, to watch a friend of mine, Eyal Dechter, defend his dissertation in cognitive science. Just before the talk started, his
wife Amy, their dog Ruby, and their daughter Susannah were
milling around, wishing him well. On the screen was a picture of Ruby, and next to it one of Susannah as a baby. When
Dad asked Susannah to point herself out, she happily slapped
a long retractable pointer against her own baby picture. On
the way out of the room, she wheeled a toy stroller behind
her mom and yelled “Good luck, Daddy!” over her shoulder.
“Vámanos!” she said finally. She’s two.
Eyal started his talk with a beguiling question: How is it
that Susannah, after two years of experience, can learn to talk,
to play, to follow stories? What is it about the human brain
that makes it learn so well? Will a computer ever be able to
learn so quickly and so fluidly?
We make sense of new phenomena in terms of things we
already understand. We break a domain down into pieces and
learn the pieces. Eyal is a mathematician and computer programmer, and he thinks about tasks—like making a soufflé—as
really complex computer programs. But it’s not as if you learn
to make a soufflé by learning every one of the program’s zillion
micro-instructions, like “Rotate your elbow 30 degrees, then
look down at the countertop, then extend your pointer finger,