Davis Sawyer

Looking forward from you, the cliff made my heart melt. I have been thinking of when that random moment changed everything. But she proposed. And getting down on one knee, she said the beautiful things in the world, and maybe save money. Hey!

This is an RNN (Recurrent Neural Network) talking, a type of neural net showcased at Google's GoNorth in Toronto earlier this month. After being trained on (bad) romance novels and Taylor Swift songs, the RNN produces captions for images in this style. As they say about data- garbage in, garbage out. While entertaining, this creative use of deep learning from University of Toronto's Jamie Kiros highlights the shortcomings in state-of-the-art neural networks, and just how much further AI has to go.

Also at GoNorth was Geoff Hinton, someone who needs no introduction in the deep learning community. His talk resonated with some of the fundamental pitfalls with artificial neural networks (ANNs) as a whole. Hinton revisited some of his work in the psychological concept of mental imagery, which is being branded today as capsule networks. Capsule nets are creating a lot of buzz as a potential game-changer in modern deep learning, but that's not the focus here. What was most fascinating about Hinton's talk, was that the bulk of his discussion involved basic geometry. On stage, all he had with him (apart from his razor sharp wit) were two large wooden parallelograms. To illustrate how convolutional neural networks (CNNs, the workhorse of deep learning in computer vision) cannot capture perceptual problems, Hinton proposed a challenge. He asked his human co-host to join the two wooden parallelograms to form a tetrahedron. For the geometrically-uninformed, think Egyptian Pyramid. Sounds simple, but human's are intuitively inept at this. In fact, two MIT tenured professors failed to form a tetrahedron, so don't beat yourself up. One professor evidently wrote a mathematical proof that it can't be done (it can), while the other cursed Hinton out of his office. It takes a simple twist to form the tetrahedron. Hinton's point is that what matters is perception.

Here's what's fascinating : CNNs, like the MIT professors and tetrahedron, may be a classic scientific example of a rigorous and robust methodology that is just plain wrong. While Hinton was sharing the latest in this infantile science of AI, what was really on display was two-thousand year old Greek math. In a sense, the latest and greatest in object detection is doing it all wrong. It's reminiscent of the Greco-Roman mathematician Ptolemy. Ptolemy was a great scientist of Alexandria in the 1st Century AD who made a lasting impact on astronomy and geography in the centuries that followed. Through detailed and rigorous calculation, flawless arithmetic and prudent observation, Ptolemy ' correctly ' concluded that Earth was the center of the universe. Using the tools of his time to study the world about him, he made a conclusion that today's children know to be flat out false. Science is about disproving, not proving. There are times in history where common scientific knowledge enters a local minima. A local minima of domain knowledge is where everything everyone knows to be right is one day proven wrong. Deep learning today, as CNNs may show, could follow in Ptolemy's footsteps.

Now, instead of bashing the subject I have come to be obsessed with, I'd like to boast. Also at GoNorth was Demis Hassabis, co-founder and CEO of DeepMind . The core of his discussion was on the breakthroughs of Alpha Go Zero, a deep learning program published in Nature this October . This system was trained to play the popular game Go , whose software predecessors had beaten world champions. Alpha Go Zero however, was different. It started from scratch: zero initial knowledge, no classic rules or information fed prior to observing human games. In 72 hours of computationally intensive execution, it surpassed 3000 years of human mathematics. What this means is that the rules and tactics passed through generations of human players was utterly abolished in 3 days of deep learning. Coincidentally, it also takes the human adult brain about 72 hours to form new neural connections when learning a new concept.

While Hinton's capsule nets are revisiting older concepts and Alpha Go is destroying them, there is an open invitation that says: "maybe our math is wrong. " Yoshua Bengio's recent work, The Consciousness Prior , echoes this invitation in the field of representation learning. Perhaps that's what science should be: an invitation with an idea that says; test it. Add to it. Mix some randomness into the equation. It says: show me I'm wrong, so we can be closer to what's right.

Stressing a system forces it to change, and avoid these local minima. It's important to consider the timeline of deep learning as a new science, because it hasn't been tested by time. New ideas are fragile and haven't learned how to survive. Nassim Taleb's heuristic The Lindy Effect is a crucial consideration for AI. In a field with few experts, the best expert is time. Again, science is an invitation, not always an answer. Not everyone can rent the computing power of DeepMind to test their systems, but anyone can ask the fundamental question of whether something's right or wrong. For example, the parallelism of the human brain gives a glimpse at how 100 billion neurons with 10 thousand connections can work in the space of a birthday cake at the expense of 20 watts. These figures make the best computer scientist blush.

The robustness of technological advancements rests on iterative tinkering. As one researcher I spoke to at MILA in Montreal put it: try weird stuff, then use what works.

It's not just about making your tech today better, but one day making it obsolete.

Davis Sawyer

Toronto, Ontario, Canada 6K followers 500+ connections

See your mutual connections View mutual connections with Davis Sign in Welcome back Email or phone Password Show Forgot password? Sign in or New to LinkedIn? Join now or New to LinkedIn? Join now

About

Experience

NXP Semiconductors

-

-

-

-

-

-

Languages

English

Full professional proficiency

Spanish

Elementary proficiency

View Davis’ full profile

Other similar profiles

Sunit Mohindroo

Sepand Barkhodaee

Tingxi Tan

Ken Wong

Daniel Laplante

Declan Goncalves

Duncan Cameron-Steinke

Prasann Pandya

Sergey Bukharov

Nader El-Ramly

Mark Sadegursky

Marty Labelle

Adam Binnie

Andrew McDonald

Ken Lotocki

Anas Wattar

Anlin Tu

Ehsan Saboori (PhD, Eng)

Satinder Singh

Abbas Sarraf, PhD

Explore collaborative articles

Others named Davis Sawyer

Davis Sawyer

Davis Sawyer

Davis Sawyer

Davis Sawyer

Add new skills with these courses

Supervised Learning Essential Training

Google Cloud Digital Leader Cert Prep: 2 Innovating with Data and Google Cloud

Oracle Database 12c: Advanced SQL

Toronto, Ontario, Canada

6K followers 500+ connections

View mutual connections with Davis

Welcome back

Email or phone

Password

Forgot password?

or

New to LinkedIn? Join now

or

New to LinkedIn? Join now