Download as pdf or txt
Download as pdf or txt
You are on page 1of 34

Linear Regression and Neural

Networks
COMP3314 — Week 5

Lingpeng Kong
Department of Computer Science, The University of Hong Kong

Based on: Probabilistic Machine Learning by Kevin Murphy


Slides from: Saw Shier Nee with special thanks!
Linear Regression
Linear Regression

Logistic Regression

If input is 1-D, simple linear regression


<latexit sha1_base64="mdFsemaZXAuPyhwistu2gKtAsD4=">AAAF23icjVRba9RAFE6166VeWvXRl9AiVLRLUqgKIlSlYkGliq3CZimTyclm2LnEmUm72yFPvomP+uSr/gP/jP/Gk+y6drMtOhA4l++bM+ecnBPnnBkbBL/mzpydb507f+HiwqXLV64uLl27vmdUoSnsUsWVfh8TA5xJ2LXMcnifayAi5vAu7j+t/O8OQBum5Fs7zKErSE+ylFFi0bS/tJiuDh4e3vYf+WTg3/Hj/aWVoB3Ux58VwrGwsrkc3fnya3O4s39t/meUKFoIkJZyYkwnDHLbdURbRjmUC1FhICe0T3rQQVESAabr6peX/i20JH6qNH7S+rV1ipEcsNyMOYP/JDkijBmKGJGC2Mw0fZXxJF+nsOmDrmMyLyxIOgqUFty3yq9q5ydMA7V8iAKhmmGCPs2IJtRihaeiWNY/GqVeSZzFmuihS5m9myvDqtIz2bubAFW6boRp5/gaoXSeVQ5KOEW6hEOqhCAyiWQhYtA2Y8ZFJEmsoqqQGNbBh6K+oXRhGVnSc5HNYGKcusRFB0SXnbDroip3Ldwe6hGH1HZWwkizXma7DcbW4C8hjt3WP+A8LkdYzMC9aIbPtfrjx7t2mu6Xx7gvm85YSZhwOSbb8AvB5PipTEYOHziLIJNkBscRIA+YVrL6gx2zINgRmA8FMxnGi6HHpKsGERVOYhy0CoEKtpaD7NnMRZUJe1+6taC9DqI87qs5hyzBX84F7Y2mF4spiMYY5RR01OPTUVW0EhXAxOrHNfuQ90epWhjYekBczIuqgHHqRzgBnPvYRH9tDZHdGfZRk6whOYl7NENluF9AQlWtim2t264sr8CG/TIaGBwQaFAoS/E3nOCfbj97/GYtDIJT4EIlwMdwQ51meY5r5mSsTskxJCoTGO65sLnVZoW99XZ4r73xGhfeE290Lng3vWVv1Qu9+96m99zb8XY96hXeN++796PVbX1sfWp9HkHPzI05N7yp0/r6G0+VITg=</latexit>

f (x; w) = ax + b
If input is N-D/output is N-D, multiple/multivariate linear regression
Linear Regression

<latexit sha1_base64="jIMc4foUgqs5vTSq8h0Td8ODCJk=">AAAFoHichVRZb9NAEDaFQLkpPPJitUIC0UQ24nosIBBIHAXRQ2Sjar0ex6vs4e6u2yYriz8Br/C7+DeMkxASp4iVLM3xfTvHeiYpBLcuin6dWTl7rnX+wurFS5evXL12/cbazV2rS8Ngh2mhzX5CLQiuYMdxJ2C/MEBlImAvGbyo/XtHYCzX6rMbFtCTtK94xhl1aOqSIud3CUu1u3dwYyPqROMTLgvxVNjYWif3v/3aGm4frJ37SlLNSgnKMUGt7cZR4XqeGseZgOoSKS0UlA1oH7ooKirB9vw45yq8g5Y0zLTBT7lwbJ1neCqtHcoEkZK63DZ9tfE0X7d02dOe56ooHSg2CZSVInQ6rBsQptwAc2KIAmWGY64hy6mhzGGbFqI4PhhNqqglwRNDzdBn3G0W2vK6f1z1N1Ng2oy7aTsFZiO1waaig1HBkK7gmGkpqUqJKmUCxuXcekLT1GmmS4VhPRyW4xsqH1fE0b4nLoeZceEST46oqbpxz5O6diP9LupEQOa6GzExvJ+7XoPx8uQvIUn8y//ARVJNsFiBf9sMXxj9x493bTfd7+a475rORCuYcQUW2/BLydU0Va6IxwSXEXRWzMk8AtQRN1rVP6PnDiQfgT0suc0xXgJ9rnw9TagImuC01AhU8GkFqL7LPalN+PaVb0edByCred+Yc8xT/OV81HnU9GIzJTUYo1qATt7436g6WoUKYGHj5JrvUAwmpTo4ceMB8Yko6wYmWUhwAoQI8RHDdhuRvSX2qEk2kJ7GHS1ROS4JUFB3q2Y759/Ulvfg4kFFTiwOCDQojGf4G87wL968evapHUfRP+BSpyCmcMu84UWBG+N0rMnoHBKVGQxXVtxcUMvC7oNO/Ljz8CPurufB5KwGt4P14G4QB0+CreB1sB3sBCzQwffgR/Cztd563frQ+jiBrpyZcm4FC6f15Tenvgsd</latexit>

Polynomial Regression (·) Feature Extractor


Linear Regression

1
<latexit sha1_base64="OTUVl4wkp7RN4mvp5SbxC5jQSe8=">AAAGDnicjVRLbxRHEB6IF4ghYCcHFHEZYSERCa9mLDnhEomHjEDCyGBskLZXVk9PzW5r+zF099hrt+Y/5GfkByBuKNecEDceF675GaneXRbvrBG0NJp6fF9XV1dXZaXg1iXJh1Onf1honTl77sfF8xd+unhpafnnXasrw2CHaaHN84xaEFzBjuNOwPPSAJWZgGfZ4G7wP9sHY7lWT91hCV1Je4oXnFGHpr2lO8TB0PnN7Y36Osm0yO2hxJ8/qH+L/4xJYSjzae0f1WPck+3tOdze0krSTkYrnhfSibBy6+ant5c//vfr1t7ywkuSa1ZJUI4Jam0nTUrX9dQ4zgTUi6SyUFI2oD3ooKioBNv1o2Tr+Bpa8rjQBj/l4pF1hpHv89JOOMPvJHkqbcgIkZK6vm36gvEkX6dyxc2u56qsHCg2DlRUInY6Dtcd59wAc+IQBcoMxwRj1qd4qw6LMhPF8cHROPUgCZ4Zag59wd2NUlseqsVV70YOTJtR7Wy7xNNIbcp+cDAqGNIVHDAtJVU5UZXMwLg+t57QPHea6UphWA8vqtEONVaWONrzxPVhapzZxJN9aupO2vUk5G6k30WdCChcZyUlhvf6rttgbAy/ELLMb3wDLrJ6jMUM/MNm+NLoz37ca6vp3jzG3Ww6M61gyhWYbMMvJVeTo3JFPB5wHkGnyQyPI0Dtc6NVeMGeO5D8COyLits+xsugx5UPvYuKoBn2ZkCggqUVoHqu70kwYe1rv5q010DWx30jzgHP8cn5pL3e9OJlSmowRj0DHdf466gQrUYFMLHR4Zp1KAfjVEOjjxrEZ6IKF5gVMcEOECLGIsarq4jszrGPmmQD+Uncozkqx5EECsJtBbZz/kGwPAKXDmoytNgg0KAwXuAznOLvPrh3+8lqmiRfgUudg5jALfOGlyWOmZOxpqDHkKhMYTjn0uZUmxd219rp7+31xzjw7kTjdS66El2Nrkdp9Ed0K7ofbUU7EYv+jt5E76L3rb9ar1qvW/+MoadPTTi/RDOr9e//sOU51Q==</latexit>

MSE(w) = RSS(w)
N
residual sum of squares mean squared error
Linear Regression

1
<latexit sha1_base64="ezzYS+Y7vsNZ/oVYHJ/Tb99EAn4=">AAAGMXicjVRNjxQ3EG0Iw1cCLOTIpcUKaZGYUfdKC1yQNkEgkCBaouyCNB5Wbnf1jDX+6LXdy+xa/QP4PeRf5Ia4cItyzYkrXCj3DJOZnkXBUqvL9d5zuVwuZ6Xg1iXJh1OnfzjTOXvu/IWLP/506fKVtavX9qyuDINdpoU2LzNqQXAFu447AS9LA1RmAl5k4wcBf3EIxnKt/nBHJQwkHSpecEYduvbXoNiY3LrvSWEoi31ae2L5UNIY/wfGxX6TlDyuccAr312gbdY1EVC4ja++SZfIKp7rETZ8OHK3XgXq/tp60kuaEa8a6cxY395+X5I38t3O/tUzf5Jcs0qCckxQa/tpUrqBp8ZxJqC+SCoLJWVjOoQ+mopKsAPfnEcd30RPHhfa4Kdc3HiXFPkhL+1MM/lOkafS2iOZIVNSN7JtLDhPwvqVK+4NPFdl5UCxaaCiErHTcahInHMDzIkjNCgzHBOM2YjioTqs21IUx8fH09SDJXhmqDnyBXe3S215KChXw9s5MG2a8tpeibuR2pSjADAqGMoVvGZaSqpyoiqZgXEjbj2hee4005XCsB4OqmaFGmtNHB164kYwdy4t4skhNXU/HXgScjfS7+G8uRz99XR6CwYtxcPJf4Is8w//hy6yesrFDPzTdvjS6K84rrXThp8taJ+1wUwrmGsFJtvCpeRqtlWuiMcNrjLoPJnJIgPUITdahRvsuQPJj8EeVNyOMF4GQ658aG+cCJph+wZG6B7AFlZDN/IkuLD2te8mvU2Q9SLWaF7zHK+cT3pbbRQPU1KDMeol6rTG32aFaDVOABNrNteuQzmepupg4poG8ZmowgFmRUywA4SIsYhxt4vMwYr6uC02kJ+kPV6Rcny1QEE4raB2zj8Jnt/ApeOaTCw2CLQkjBd4Def8B08e/fJ7N02Sb9ClzkHM6JZ5w8sSn5mTuaagC0yczGn4zqXtV23V2NvspXd6W8/xwfs1mo7z0fXoRrQRpdHdaDt6HO1EuxGL/oo+Rp+iz523nQ+dvzv/TKmnT800P0dLo/PvFxiqR4U=</latexit>

x µ 2
f (x) = p e
1
2 ( )
2⇡

<latexit sha1_base64="hqWOK7bmGJ8j4W4tMG7RPtwQtL4=">AAAGJXicjVTNjhQ3EG5+hvATYIFjLi1WkUBiR90rEeCABEGLALHRAtkFNB6N3O7qaWv809ju3dm1+lW45gVy5wm4IaSccuIKUp4g5ZnJZKdnEVhqdbm+73O5XC5nleDWJclfx46fONk59cPpM2fP/Xj+wsWVS5d3rK4Ng22mhTavMmpBcAXbjjsBryoDVGYCXmajBwF/uQvGcq1+d/sV9CUdKl5wRh26BiuvSUmdJ5kWud2X+PN7TTPwxMHY+c2nG00T340JNcOYSK4GbWY8JT5/8aK5tghdH6ysJt1kMuJlI50Zq/fu0Cf9f1bfbg0unfyT5JrVEpRjglrbS5PK9T01jjMBzVlSW6goG9Eh9NBUVILt+8kZNPHP6MnjQhv8lIsn3gVFvssrO9OMv1PkqbQhI2RK6krbxoLzKKxXu+J233NV1Q4UmwYqahE7HYcqxDk3wJzYR4MywzHBmJXUUOawVgtRHB8dTFMPluCZoWbfF9zdqLTloYhcDW/kwLSZlNR2K9yN1KYqA8CoYChXsMe0lFTlRNUyA+NKbj2hee4007XCsB7e1JMVGp82xNEhXoIS5s6FRTzZpabppX1PQu5G+h2cEwGF662mxPBh6fotxcb4f0GW+Y1v0EXWTLmYgX/aDl8Z/R+Oa2214c1D2s02mGkFc63AZFu4xIs+2ypXxOMGlxl0nsz4MAPULjdahRvsuQPJD8C+qbktMV4GQ658aGmcCJphywYGTrC0AtTQlZ4EF9a+8WtJdx1kcxibaPZ4jlfOJ92bbRQPU2Kf4t4XqNMaf50VojU4AUxssrl2HarRNNXQ6JMG8ZmowwFmRUywA4SIsYjx2hoy+0vqg7bYQH6U9mBJyvGlAgXhtILaOf84eH4Dl44aMrbYINCSMF7gNZzzHzx+eP/5WpokX6FLnYOY0S3zhlcVPjNHc01BDzFxMqfhO5e2X7VlY2e9m/7SvfkMH7xfo+k4Hf0UXY2uRWl0K7oXPYq2ou2IRe+iT9Hn6Evnj877zofOxyn1+LGZ5kq0MDp//wuGUEPO</latexit>

ŵMLE = arg min RSS(w)


w
Optimization

Gradient Descent
Ridge / Lasso Regression
Ridge Regression (MAP derivation)

Lasso Regression (Loss function derivation)


Lasso Regression

Ridge allows parameters to be small but cannot reach zero.

Lasso allow parameters to be exactly zero.

This is useful because it can be used to perform feature selection,


where the weight of certain features can be zero.
Q-norm
General Equation

D
<latexit sha1_base64="7B0/4p15HQBEDW7sIvenlOAN9hU=">AAAGAnicjVTNjxM3FB8oKRRoWdpjLyNWSCCx6QwS0AsS7ZYCEqAFsQtSnI08njeJFX9MbM9udp259dQbd/4CbgiOnLi2F479ZyqekzTdTBa1lkbzPn4/Pz8/v5eVgluXJH+dOPnFqdaXp898dfbc+a+/ubB28dsdqyvDYJtpoc2LjFoQXMG2407Ai9IAlZmA59lwM/if74GxXKtn7qCErqR9xQvOqENTb+3mZEIyLXJ7IPHn9+vJpOdHdXw7vkJsJXs+v53Wu/6XerLfyye7o6u7Pv1hVPfW1pN2Ml3xqpDOhfU7937f/Ju8/LjVu3jqHck1qyQoxwS1tpMmpet6ahxnAuqzpLJQUjakfeigqKgE2/XTBOv4MlryuNAGP+XiqXWJke/x0s454/9J8lTakDYiJXUD2/QF43G+TuWKH7ueq7JyoNgsUFGJ2Ok4XHGccwPMiQMUKDMcE4zZgBrKHBZiKYrjw8NZ6kESPDPUHPiCu2ultjxUiKv+tRyYNtN62XaJp5HalIPgYFQwpCvYZ1pKqnKiKpmBcQNuPaF57jTTlcKwHkbVdIfapzVxtO+JG8DCuLSJJ3vU1J2060nI3Ui/gzoRULjOekoM7w9ct8G4O/6XkGX+7n/ARVbPsJiBf9gMXxr9jx/32mq6Hx3hPmo6M61gwRWYbMMvJVfzo3JFPB5wFUEXyYyPIkDtcaNVeMGeO5D8EOyo4naA8TLoc+VDv6IiaIb9GBCoYGkFqL4beBJMWPvabyTt6yDro74pZ5/n+OR80r7R9OJlSmowRr0EndX486gQrUYFMLHp4Zp1KIezVB2M3bRBfCaqcIFZERPsACFiLGK8sYHI7gr7sEk2kB/HPVyhchxDoCDcVmA75x8Ey2Nw6bAmY4sNAg0K4wU+wwV+88GvPz3dSJPkM3CpcxBzuGXe8LLEMXM81hT0CBKVBQznXNqcaqvCzvV2erN94wkOvJ+j2ToTfR9diq5EaXQruhPdj7ai7YhFr6IP0R/Rn63fWq9bb1pvZ9CTJ+ac76Kl1Xr/CYQSNN8=</latexit>

X
q 1/q
||w||q = ( |wd | )
d=1

L0 Regularization L1 Regularization L2 Regularization

Elastic Net — Lasso + Ridge

“elastic net” regularization


(Zou and Hastie, 2005; Friedman et al., 2008)
Example — Cancer Data

Least Square – Worst


Ridge – weight is smaller but won’t reach zero, better than LS
Lasso – some features’ weight are zero; features eliminated
Elastic Net - Best
Learning Linear Models
General training setup:

We observe a collection of examples.

Perform statistical analysis to discover from the data.

Ranges from “count and normalize” to complex optimization routines.

Optimization view:
Neural Networks
Basis Function Expansion
Logistic Regression
Basis Function Expansion

Deep Neural Networks (DNNs)


Multilayer Perceptrons (MLPs)
perceptron: a deterministic version of logistic regression

heaviside step function / linear threshold function

x1 x2 XOR(x1, x2)

0 0 0

0 1 1

1 0 1

1 1 0
Multilayer Perceptrons (MLPs)
Differentiable MLPs

Activation function What’s bad here?


(ReLU: rectified linear unit)
Computational Graphs

x1 <latexit sha1_base64="Y1MfsLqoTt9iL03xQrfEyz0C1Wk=">AAACPHicbVA9SwNBEN2L3/EraqnFYRAsQrgTUcuAhZaKJhGSGPY2e8mSvd1ld06MR0pLW/0h9vkf9nZia+1eLoVGBwYe780wb16gODPgeW9ObmZ2bn5hcSm/vLK6tl7Y2KwZGWtCq0RyqW8CbChnglaBAac3SlMcBZzWg/5pqtfvqDZMimsYKNqKcFewkBEMlrq6b/vtQtEre+Ny/wJ/AoqVnceV29et0UV7wyk2O5LEERVAODam4XsKWgnWwAinw3wzNlRh0sdd2rBQ4IiaVjL2OnT3LNNxQ6ltC3DH7M+NBEfGDKLATkYYemZaS8n/tEYM4UkrYULFQAXJDoUxd0G66eNuh2lKgA8swEQz69UlPawxARvPryvA+g/ZFyniLNBYD5KQQUlJw9LcmOiWOpRIPU7RlJV1E0mteqlAMCfDvI3Vnw7xL6gdlP2j8uGlzfcMZbWIttEu2kc+OkYVdI4uUBUR1EVP6Bm9OCPn3flwPrPRnDPZ2UK/yvn6BrYIseg=</latexit>

p1 := p(y = 1 | x1 , x2 )
<latexit sha1_base64="/Gbl26h/eQpNHKJx0aq97lY8VFk=">AAACVnicbVBdSxtBFJ3d1pqmH8b46MvQUGohLLtSqgiC0Af7aKFRIQnL7OxsMmS+mLkrrkse+mt8tT/C/6D+GHE28aFqD8xwOOde7r0nM4I7iOPbIHz1euXNautt+937Dx/XOuvdY6dLS9mAaqHtaUYcE1yxAXAQ7NRYRmQm2Ek2+9H4J2fMOq7Vb6gMG0syUbzglICX0s6mSRO8t4/NVrWf4JHkOT5Pk77/tr+mnV4cxQvglyR5JL2DL+V1ZO+6R+l60BvlmpaSKaCCODdMYgPjmljgVLB5e1Q6ZgidkQkbeqqIZG5cL66Y489eyXGhrX8K8EL9t6Mm0rlKZr5SEpi6514j/s8bllDsjmuuTAlM0eWgohQYNG4iwTm3jIKoPCHUcr8rplNiCQUf3JMpwGcXyysaJnhmia3qgkPfaMebRLma9HNGtV3k6yLjt5HammljUCLovO1jTZ6H+JIcb0fJ9+jbL5/vIVqihTbRJ7SFErSDDtBPdIQGiKI/6BJdob/BTXAfroSry9IweOzZQE8Qdh4AdaK29A==</latexit>

h1 p1
<latexit sha1_base64="0aMS+62nMB/pVmVWoZAvPgOQQf8=">AAACUHicbZDPThsxEMZnU0ohpZAU9QSHFVGlHlC0ixDtEakHOFKpAaQkjbzObGLFa1v2bNV0tcc+SK/lyJ2H4NY36Y16NyDxbyTLn77P9ox/iZHCURT9DRovll4uv1pZbb5ee7O+0Wq/PXU6txx7XEttzxPmUAqFPRIk8dxYZFki8SyZfa7ys+9ondDqK80NDjM2USIVnJG3Rq13A8IfVL9TJDLHspiO4nLU6kTdqK7wqYhvRedw+9fat8vNq5NRO+gMxprnGSrikjnXjyNDw4JZElxi2RzkDg3jMzbBvpeKZeiGRd23DN97Zxym2vqlKKzd+zcKljk3zxJ/MmM0dY+zynwu6+eUfhoWQpmcUPFFozSXIemwghGOhUVOcu4F41b4WUM+ZZZx8sgedCEx+7n4RaWkSCyz8yIVtGu0ExVLoSa7Y+Ta1mRd1/hpMm3NtAo4k7xseqzxY4hPxeleNz7o7n/xfI9gUSuwBTvwAWL4CIdwDCfQAw4F/IY/cBFcB/+Cm0awOHq3wyY8qEbzP1gGuJc=</latexit>

<latexit sha1_base64="4UhpAX1AVR8YjJNAOx3goAoXdmg=">AAACUHicbZDBThsxEIZnQ0tpaCEB9dTLigipBxTtIkQ59IDEoZW4pFIDVEkUeZ3ZxIrXtuxZ1LDad+AVuLZvw4034Ua9m1Yq0JEs//p/2zP+EiOFoyi6CxorL16uvlp73Vx/83Zjs9XeOnM6txz7XEttLxLmUAqFfRIk8cJYZFki8TyZn1T5+SVaJ7T6RguDo4xNlUgFZ+StcevdkPAH1e8UicyxLMw4LsetTtSN6gqfi/iP6BzvfO+1r08/9cbtoDOcaJ5nqIhL5twgjgyNCmZJcIllc5g7NIzP2RQHXiqWoRsVdd8y3PXOJEy19UtRWLv/3ihY5twiS/zJjNHMPc0q83/ZIKf0aFQIZXJCxZeN0lyGpMMKRjgRFjnJhReMW+FnDfmMWcbJI3vUhcT8avmLSkmRWGYXRSpoz2gnKpZCTfcmyLWtybqu8dNk2ppZFXAmedn0WOOnEJ+Ls/1ufNg9+Or5foZlrcF72IEPEMNHOIYv0IM+cCjgBn7Cr+A2uA8eGsHy6N8dtuFRNZq/AWReuAM=</latexit>

x2
<latexit sha1_base64="+QD6lLYrlFcju7OsE+tVFe8fWe4=">AAACPHicbVA9SwNBEN3z2/gZLbU4DIJFCHciahmw0FLRfEASw95mL1myt7vszonxSGlpqz/E3v9hbye21u7lUhjjwMDjvRnmzQsUZwY8792ZmZ2bX1hcWs6trK6tb2zmt6pGxprQCpFc6nqADeVM0Aow4LSuNMVRwGkt6J+leu2OasOkuIGBoq0IdwULGcFgqev79mF7s+CVvFG508Afg0J593H19nX77bKddwrNjiRxRAUQjo1p+J6CVoI1MMLpMNeMDVWY9HGXNiwUOKKmlYy8Dt19y3TcUGrbAtwR+3sjwZExgyiwkxGGnvmrpeR/WiOG8LSVMKFioIJkh8KYuyDd9HG3wzQlwAcWYKKZ9eqSHtaYgI1n4gqw/kP2RYo4CzTWgyRkUFTSsDQ3JrrFDiVSj1I0JWXdRFKrXioQzMkwZ2P1/4Y4DaqHJf+4dHRl8z1HWS2hHbSHDpCPTlAZXaBLVEEEddETekYvzpvz4Xw6X9nojDPe2UYT5Xz/ALfesek=</latexit>

h1 = w 1 x 1 + w 2 x 2 + b
<latexit sha1_base64="nsjx/2YQDDjEDEzW/d54j+h+MQQ=">AAACmnicbVHLbhMxFHWGV0kLTemSLqxGSEhU0UxUFTZIFV0UxKZITVspCSOP505ixWNb9h1IGM2yX9Q9Oz6Cb2ALezyTLvrIlSwdnXNfPjcxUjgMw9+t4MHDR4+frD1tr288e77Z2Xpx5nRhOQy4ltpeJMyBFAoGKFDChbHA8kTCeTI7qvXzb2Cd0OoUFwbGOZsokQnO0FNxJx4hzLHpUyaygKqcxlFF39MbvIW0Kr/X9DyO6JsVUr+ax/0VSlLFnW7YC5ug90F0DbqHO5cbX6+2f57EW63uKNW8yEEhl8y5YRQaHJfMouASqvaocGAYn7EJDD1ULAc3LpuhFX3lmZRm2vqnkDbszYqS5c4t8sRn5gyn7q5Wk6u0YYHZu3EplCkQFF8OygpJUdPaVpoKCxzlwgPGrfC7Uj5llnH05t+agmL2Y/mLGkmRWGYXZSZwz2gn6qsINdlLgWvb3Mj1jN8m19ZMa4Ezyau2tzW6a+J9cNbvRQe9/S/e32OyjDXykuyS1yQib8kh+UhOyIBw8ov8IX/Jv2An+BB8Cj4vU4PWdc02uRXB6X8ZtdS0</latexit>

1
<latexit sha1_base64="BrTycRZUCAsdXHJMQtfJPv0vXjM=">AAACfXicbVFNbxMxEHW2fJTw0bQcuVhElYoIq12EaC+VKhCCYxGkrZSNIq8zm1jx2pY9WzW1ll/Dgb/ChSvwa8C76YE2jGTp6b03nvFzbqRwmCS/O9HGrdt37m7e695/8PDRVm9758TpynIYci21PcuZAykUDFGghDNjgZW5hNN88bbRT8/BOqHVZ1waGJdspkQhOMNATXpvMoQLbO/xuayg9maS1vSQZoVl3Ke1T59ncGH83os15zw4n9X1pNdP4qQtug7SK9A/evf105fB8PvxZLvTz6aaVyUo5JI5N0oTg2PPLAouoe5mlQPD+ILNYBSgYiW4sW9H13Q3MFNaaBuOQtqy/3Z4Vjq3LPPgLBnO3U2tIf+njSosDsZeKFMhKL4aVFSSoqZNcnQqLHCUywAYtyLsSvmchZQw5HttCorF5eoVDZIit8wufSFwYLQTTfBCzQZT4Nq23+BiE7YptTXzRuBM8robYk1vhrgOTl7G6ev41ceQ73uyqk3yhDwleyQl++SIfCDHZEg4+UZ+kJ/kV+dPtBsNonhljTpXPY/JtYr2/wJsq8m5</latexit>

p1 =
1 + exp ( h1 )
Logistic Regression

x1 <latexit sha1_base64="Y1MfsLqoTt9iL03xQrfEyz0C1Wk=">AAACPHicbVA9SwNBEN2L3/EraqnFYRAsQrgTUcuAhZaKJhGSGPY2e8mSvd1ld06MR0pLW/0h9vkf9nZia+1eLoVGBwYe780wb16gODPgeW9ObmZ2bn5hcSm/vLK6tl7Y2KwZGWtCq0RyqW8CbChnglaBAac3SlMcBZzWg/5pqtfvqDZMimsYKNqKcFewkBEMlrq6b/vtQtEre+Ny/wJ/AoqVnceV29et0UV7wyk2O5LEERVAODam4XsKWgnWwAinw3wzNlRh0sdd2rBQ4IiaVjL2OnT3LNNxQ6ltC3DH7M+NBEfGDKLATkYYemZaS8n/tEYM4UkrYULFQAXJDoUxd0G66eNuh2lKgA8swEQz69UlPawxARvPryvA+g/ZFyniLNBYD5KQQUlJw9LcmOiWOpRIPU7RlJV1E0mteqlAMCfDvI3Vnw7xL6gdlP2j8uGlzfcMZbWIttEu2kc+OkYVdI4uUBUR1EVP6Bm9OCPn3flwPrPRnDPZ2UK/yvn6BrYIseg=</latexit>

p2 := p(y = 0 | x1 , x2 )
<latexit sha1_base64="0fTBFe32HCggbA9vTFAQiYqBBn0=">AAACVnicbVBdSxtBFJ3d1o/GVqM++jIYpBbCshtERRCEPrSPFhoVkrDMzs4mQ+aLmbviuuShv8ZX+yP8D21/TOls4oMfPTDD4Zx7ufeezAjuII5/B+Gbt0vLK6vvWmvvP6xvtDe3LpwuLWV9qoW2VxlxTHDF+sBBsCtjGZGZYJfZ9HPjX14z67hW36EybCTJWPGCUwJeSts7Ju3hk1Ns9qvTGA8lz/FNmnT91/uUtjtxFM+BX5PkkXTOPpYPkf2zdZ5uBp1hrmkpmQIqiHODJDYwqokFTgWbtYalY4bQKRmzgaeKSOZG9fyKGd7zSo4Lbf1TgOfq046aSOcqmflKSWDiXnqN+D9vUEJxPKq5MiUwRReDilJg0LiJBOfcMgqi8oRQy/2umE6IJRR8cM+mAJ/eLq5omOCZJbaqCw5dox1vEuVq3M0Z1Xaer4uM30ZqayaNQYmgs5aPNXkZ4mty0YuSw+jgm8/3C1pgFe2gXbSPEnSEztBXdI76iKIf6A7do5/Br+BvuBSuLErD4LFnGz1D2P4Hdau29A==</latexit>

h1 p2
<latexit sha1_base64="0aMS+62nMB/pVmVWoZAvPgOQQf8=">AAACUHicbZDPThsxEMZnU0ohpZAU9QSHFVGlHlC0ixDtEakHOFKpAaQkjbzObGLFa1v2bNV0tcc+SK/lyJ2H4NY36Y16NyDxbyTLn77P9ox/iZHCURT9DRovll4uv1pZbb5ee7O+0Wq/PXU6txx7XEttzxPmUAqFPRIk8dxYZFki8SyZfa7ys+9ondDqK80NDjM2USIVnJG3Rq13A8IfVL9TJDLHspiO4nLU6kTdqK7wqYhvRedw+9fat8vNq5NRO+gMxprnGSrikjnXjyNDw4JZElxi2RzkDg3jMzbBvpeKZeiGRd23DN97Zxym2vqlKKzd+zcKljk3zxJ/MmM0dY+zynwu6+eUfhoWQpmcUPFFozSXIemwghGOhUVOcu4F41b4WUM+ZZZx8sgedCEx+7n4RaWkSCyz8yIVtGu0ExVLoSa7Y+Ta1mRd1/hpMm3NtAo4k7xseqzxY4hPxeleNz7o7n/xfI9gUSuwBTvwAWL4CIdwDCfQAw4F/IY/cBFcB/+Cm0awOHq3wyY8qEbzP1gGuJc=</latexit>

<latexit sha1_base64="17pf7sl6FYcbed08j6a9qE8zAV4=">AAACUHicbZDLahsxFIbPuLfESRu7JatuhphAFsHMhJB20UUgiwS6caHOBdsYjXzGFtZIQjpT6gzzDnmFbJO36a5vkl2qGbfQXA4I/fy/pHP0JUYKR1H0O2i8ePnq9ZuV1eba+tt3G632+1Onc8uxz7XU9jxhDqVQ2CdBEs+NRZYlEs+S+VGVn/1A64RW32lhcJSxqRKp4Iy8NW5tDgl/Uv1Okcgcy8KM98pxqxN1o7rCpyL+KzqHWxe99tXXL71xO+gMJ5rnGSrikjk3iCNDo4JZElxi2RzmDg3jczbFgZeKZehGRd23DLe9MwlTbf1SFNbu/zcKljm3yBJ/MmM0c4+zynwuG+SUfh4VQpmcUPFlozSXIemwghFOhEVOcuEF41b4WUM+Y5Zx8sgedCExv1z+olJSJJbZRZEK2jXaiYqlUNPdCXJta7Kua/w0mbZmVgWcSV42Pdb4McSn4nSvGx909795vsewrBX4CFuwAzF8gkM4gR70gUMB13ADt8Gv4C64bwTLo/92+AAPqtH8A2Y1uAQ=</latexit>

x2
<latexit sha1_base64="+QD6lLYrlFcju7OsE+tVFe8fWe4=">AAACPHicbVA9SwNBEN3z2/gZLbU4DIJFCHciahmw0FLRfEASw95mL1myt7vszonxSGlpqz/E3v9hbye21u7lUhjjwMDjvRnmzQsUZwY8792ZmZ2bX1hcWs6trK6tb2zmt6pGxprQCpFc6nqADeVM0Aow4LSuNMVRwGkt6J+leu2OasOkuIGBoq0IdwULGcFgqev79mF7s+CVvFG508Afg0J593H19nX77bKddwrNjiRxRAUQjo1p+J6CVoI1MMLpMNeMDVWY9HGXNiwUOKKmlYy8Dt19y3TcUGrbAtwR+3sjwZExgyiwkxGGnvmrpeR/WiOG8LSVMKFioIJkh8KYuyDd9HG3wzQlwAcWYKKZ9eqSHtaYgI1n4gqw/kP2RYo4CzTWgyRkUFTSsDQ3JrrFDiVSj1I0JWXdRFKrXioQzMkwZ2P1/4Y4DaqHJf+4dHRl8z1HWS2hHbSHDpCPTlAZXaBLVEEEddETekYvzpvz4Xw6X9nojDPe2UYT5Xz/ALfesek=</latexit>

h1 = w 1 x 1 + w 2 x 2 + b
<latexit sha1_base64="nsjx/2YQDDjEDEzW/d54j+h+MQQ=">AAACmnicbVHLbhMxFHWGV0kLTemSLqxGSEhU0UxUFTZIFV0UxKZITVspCSOP505ixWNb9h1IGM2yX9Q9Oz6Cb2ALezyTLvrIlSwdnXNfPjcxUjgMw9+t4MHDR4+frD1tr288e77Z2Xpx5nRhOQy4ltpeJMyBFAoGKFDChbHA8kTCeTI7qvXzb2Cd0OoUFwbGOZsokQnO0FNxJx4hzLHpUyaygKqcxlFF39MbvIW0Kr/X9DyO6JsVUr+ax/0VSlLFnW7YC5ug90F0DbqHO5cbX6+2f57EW63uKNW8yEEhl8y5YRQaHJfMouASqvaocGAYn7EJDD1ULAc3LpuhFX3lmZRm2vqnkDbszYqS5c4t8sRn5gyn7q5Wk6u0YYHZu3EplCkQFF8OygpJUdPaVpoKCxzlwgPGrfC7Uj5llnH05t+agmL2Y/mLGkmRWGYXZSZwz2gn6qsINdlLgWvb3Mj1jN8m19ZMa4Ezyau2tzW6a+J9cNbvRQe9/S/e32OyjDXykuyS1yQib8kh+UhOyIBw8ov8IX/Jv2An+BB8Cj4vU4PWdc02uRXB6X8ZtdS0</latexit>

<latexit sha1_base64="PdPkAzDXiY/cu/rXNDLRetcRj9A=">AAACy3ichVHLahRBFK1uX3F8TXSpi8IhEnEydAcxboSAiG6EiE4SmB6a6prbM8VUdxVVt0MmZbtz70f4M36CP+La6u4sTEbIhYLDuec+6txMS2Exin4H4bXrN27e2rjdu3P33v0H/c2Hh1ZVhsOYK6nMccYsSFHCGAVKONYGWJFJOMqWb5v80QkYK1T5BVcapgWblyIXnKGn0v6PBOEU2z4ukxXUTqe7NX1DY7pDk9ww7uLaxS8SONVue2dNvUjj+nndFHTiK3RXd0r7g2gUtUHXQXwOBvvvfn7+Nhz/Okg3g0EyU7wqoEQumbWTONI4dcyg4BLqXlJZ0Iwv2RwmHpasADt17eiabnlmRnNl/CuRtuy/FY4V1q6KzCsLhgt7OdeQ/8tNKsxfT50odYVQ8m5QXkmKijZ3oDNhgKNcecC4EX5XyhfMW4j+WhemoFiedb9okBSZYWblcoFDraxozijK+XAGXJn2qHak/TaFMnrRJDiTvO55W+PLJq6Dw91R/Gr08pP39z3pYoM8Jk/JNonJHtknH8gBGRNO/gRPgq3gWfgxtOFZ+LWThsF5zSNyIcLvfwHw1+YH</latexit>

1 exp ( h1 )
p2 = 1 =
1 + exp ( h1 ) 1 + exp ( h1 )
Loss Function
case y = 1 :
<latexit sha1_base64="bndwrjvF/SSKDckENS5bTLw39R8=">AAACU3icbVBdaxNBFJ1dq9ZoNdEXwZehQfChhF0RLUWh4IOCLxFMW0lCmL25mwyZnRlm7orrsv4Gf4Sv+m988Lf44my2D7b1wsCZc+7nyaySnpLkdxRf27l+4+burd7tO3t37/UH90+8KR3gBIwy7iwTHpXUOCFJCs+sQ1FkCk+zzetWP/2EzkujP1BlcV6IlZa5BEGBWvQfzgg/Uw2hBW+qV2n35UfNoj9MRsk2+FWQnoPh8f7H8eDbu5fjxSAazpYGygI1gRLeT9PE0rwWjiQobHqz0qMVsBErnAaoRYF+Xm9PaPjjwCx5blx4mviW/beiFoX3VZGFzELQ2l/WWvJ/2rSk/HBeS21LQg3doLxUnAxv/eBL6RBIVQEIcDLsymEtnAAKrl2YQnLzpbuiRUpmTriqziUdWONla6fUq4MlgnFbc/3Ihm0K4+y6FUAoaHrB1vSyiVfBydNR+nz07H3w9w3rYpc9YvvsCUvZC3bM3rIxmzBgX9l39oP9jH5Ff+I43ulS4+i85gG7EPHeX2W8t3k=</latexit>

x1
<latexit sha1_base64="Y1MfsLqoTt9iL03xQrfEyz0C1Wk=">AAACPHicbVA9SwNBEN2L3/EraqnFYRAsQrgTUcuAhZaKJhGSGPY2e8mSvd1ld06MR0pLW/0h9vkf9nZia+1eLoVGBwYe780wb16gODPgeW9ObmZ2bn5hcSm/vLK6tl7Y2KwZGWtCq0RyqW8CbChnglaBAac3SlMcBZzWg/5pqtfvqDZMimsYKNqKcFewkBEMlrq6b/vtQtEre+Ny/wJ/AoqVnceV29et0UV7wyk2O5LEERVAODam4XsKWgnWwAinw3wzNlRh0sdd2rBQ4IiaVjL2OnT3LNNxQ6ltC3DH7M+NBEfGDKLATkYYemZaS8n/tEYM4UkrYULFQAXJDoUxd0G66eNuh2lKgA8swEQz69UlPawxARvPryvA+g/ZFyniLNBYD5KQQUlJw9LcmOiWOpRIPU7RlJV1E0mteqlAMCfDvI3Vnw7xL6gdlP2j8uGlzfcMZbWIttEu2kc+OkYVdI4uUBUR1EVP6Bm9OCPn3flwPrPRnDPZ2UK/yvn6BrYIseg=</latexit>

+
<latexit sha1_base64="hSo+8nFIIzdpadb5Q6VOnv+Q6mI=">AAACgHicbVHLjtMwFHXDayivFpZsrKmQRqIKyRQxA2IxEkIgsSkSnRnUVpXj3KRWbceybxAhyj/wH3wNS+BncNJZMB2uZOvonPvycWKkcBhFv3rBtes3bt7au92/c/fe/QeD4cNTV5SWw4wXsrDnCXMghYYZCpRwbiwwlUg4SzZvWv3sC1gnCv0JKwNLxXItMsEZemo1eLtIIfO1XaeaWVXlFkA3tc2Tpo7Cw5djGoWTSXvHk6a/QPiKu8lPV4NRFEZd0KsgvgCjk/3P0+H3D6+nq2FvtEgLXirQyCVzbh5HBpe+Jwouwc8pHRjGNyyHuYeaKXDLuhvc0CeeSWlWWH800o79t6JmyrlKJT5TMVy7Xa0l/6fNS8yOl7XQpkTQfDsoKyXFgrbm0VRY4CgrDxi3wu9K+ZpZxtFbfGkKis237StaJEVima3qTODYFE603gudj1Pvu+1+woXGb6MKa9atwJnkTd/bGu+aeBWcHobxi/D5R+/vO7KNPfKY7JMDEpMjckLekymZEU5+kJ/kN/kTBMFB8CyIt6lB76LmEbkUwau/nLHG/w==</latexit>

h1 p1 loss
<latexit sha1_base64="0aMS+62nMB/pVmVWoZAvPgOQQf8=">AAACUHicbZDPThsxEMZnU0ohpZAU9QSHFVGlHlC0ixDtEakHOFKpAaQkjbzObGLFa1v2bNV0tcc+SK/lyJ2H4NY36Y16NyDxbyTLn77P9ox/iZHCURT9DRovll4uv1pZbb5ee7O+0Wq/PXU6txx7XEttzxPmUAqFPRIk8dxYZFki8SyZfa7ys+9ondDqK80NDjM2USIVnJG3Rq13A8IfVL9TJDLHspiO4nLU6kTdqK7wqYhvRedw+9fat8vNq5NRO+gMxprnGSrikjnXjyNDw4JZElxi2RzkDg3jMzbBvpeKZeiGRd23DN97Zxym2vqlKKzd+zcKljk3zxJ/MmM0dY+zynwu6+eUfhoWQpmcUPFFozSXIemwghGOhUVOcu4F41b4WUM+ZZZx8sgedCEx+7n4RaWkSCyz8yIVtGu0ExVLoSa7Y+Ta1mRd1/hpMm3NtAo4k7xseqzxY4hPxeleNz7o7n/xfI9gUSuwBTvwAWL4CIdwDCfQAw4F/IY/cBFcB/+Cm0awOHq3wyY8qEbzP1gGuJc=</latexit>

<latexit sha1_base64="/WnQCs7ketQzMhW2Yz5e6u/ygwU=">AAACWHicbVBNa9tAEB0raRI7beqkxxIQMYEegpFCaXIpMeTQHlOIk4BtzGo9shevdre7o1JX6NSf0lOu7e/oOaS/pis5h3wNLDzem5k3+xIjhaMoum0EK6sv1tY3mq3Nl6+2Xre3dy6czi3HPtdS26uEOZRCYZ8ESbwyFlmWSLxM5qeVfvkNrRNandPC4ChjUyVSwRl5atzeHRJ+p3pPkcgcy6ImCqmdK8txuxN1o7rCpyC+A52Tv+bXTevj17PxdqMznGieZ6iIS+bcII4MjQpmSXCJZWuYOzSMz9kUBx4qlqEbFbV/Ge57ZhKm2vqnKKzZ+xMFy5xbZInvzBjN3GOtIp/TBjmlx6NCKJMTKr40SnMZkg6rUMKJsMhJLjxg3Ap/a8hnzDJOProHLiTmP5a/qJAUiWV2UaSCDox2ospUqOnBBLm2dcKua/w1mbZmVgmcSV62fKzx4xCfgovDbvyh+/5L1Ol9gmVtwFvYg3cQwxH04DOcQR84/IRr+A1/Gv8CCNaD5rI1aNzNvIEHFez8Byncu6o=</latexit>

<latexit sha1_base64="4UhpAX1AVR8YjJNAOx3goAoXdmg=">AAACUHicbZDBThsxEIZnQ0tpaCEB9dTLigipBxTtIkQ59IDEoZW4pFIDVEkUeZ3ZxIrXtuxZ1LDad+AVuLZvw4034Ua9m1Yq0JEs//p/2zP+EiOFoyi6CxorL16uvlp73Vx/83Zjs9XeOnM6txz7XEttLxLmUAqFfRIk8cJYZFki8TyZn1T5+SVaJ7T6RguDo4xNlUgFZ+StcevdkPAH1e8UicyxLMw4LsetTtSN6gqfi/iP6BzvfO+1r08/9cbtoDOcaJ5nqIhL5twgjgyNCmZJcIllc5g7NIzP2RQHXiqWoRsVdd8y3PXOJEy19UtRWLv/3ihY5twiS/zJjNHMPc0q83/ZIKf0aFQIZXJCxZeN0lyGpMMKRjgRFjnJhReMW+FnDfmMWcbJI3vUhcT8avmLSkmRWGYXRSpoz2gnKpZCTfcmyLWtybqu8dNk2ppZFXAmedn0WOOnEJ+Ls/1ufNg9+Or5foZlrcF72IEPEMNHOIYv0IM+cCjgBn7Cr+A2uA8eGsHy6N8dtuFRNZq/AWReuAM=</latexit>


<latexit sha1_base64="ZEhmNtmWna9lR+bGQjd+LwCuntg=">AAAChXicbVFNj9MwEHXDxy7lY1s4crG2QuJQomS7WkBCYiUOILgUie4uaqvKcSapVdux7AkiRPkP/BR+DRIn+Ck46R7YLiPZepo3M2/8nBgpHEbRr15w4+at23v7d/p3791/cDAYPjxzRWk5zHghC3uRMAdSaJihQAkXxgJTiYTzZPOm5c+/gHWi0J+wMrBULNciE5yhT60G7xcpZL63m1Qzq6rcAuimtnnS1FF49HJMo3Ayae940vQXCF9xt3iBQoFbDUZRGHVBr4P4EoxODz9Ph98/vJquhr3RIi14qUAjl8y5eRwZXPrBKLgEL1Y6MIxvWA5zDzXzKsu6U2/oE59JaVZYfzTSLvtvR82Uc5VKfKViuHa7XJv8HzcvMXuxrIU2JYLmW6GslBQL2jpIU2GBo6w8YNwKvyvla2YZR+/zFRUUm2/bV7RIisQyW9WZwLEpnGg/QOh8nHrzbfcdLjR+G1VYs24JziRv+t7WeNfE6+DsKIxPwuOP3t+3ZBv75DE5JE9JTJ6TU/KOTMmMcPKD/CS/yZ9gL3gWHAcn29Kgd9nziFyJ4PVfIMbJhA==</latexit>

sigmoid log
<latexit sha1_base64="lUzUAnzIFb5iwZmg1DHP9+oSaps=">AAACo3icbVFNa9tAEF2rX6n7ZbfHXkSMoQcjpLgkLeQQ6KGFXNwSJym2MKvVSF68H2J3VKIK/Yf+st77P3pt6UrOIYk7sMvjvZmd2XlJIbjFMPzV8+7df/Dw0d7j/pOnz56/GAxfnltdGgZzpoU2lwm1ILiCOXIUcFkYoDIRcJFsPrT6xTcwlmt1hlUBsaS54hlnFB21GqyXKWSutnuppkZWuQFQTW3ypKnD4OD9xA+D6bS9o2nTXyJc4U5yx9aW51LztGn64xtpBtKmTprVYBQGYRf+Loiuwehk/+ts+OP0eLYa9kbLVLNSgkImqLWLKCwwdm2RMwFulNJCQdmG5rBwUFEJNq67po0/dkzqZ9q4o9Dv2JsVNZXWVjJxmZLi2t7VWvJ/2qLE7F1cc1WUCIptG2Wl8FH77X79lBtgKCoHKDPczeqzNTWUoXPhVhfkm+/bX7RI8MRQU9UZx0mhLW/t4SqfpM4a05llg8JNI7Up1q3AqGBN3601urvEXXB+EESHwdvPbr8fyTb2yGuyT96QiByRE/KJzMicMPKT/CZ/yF9v7J16X7yzbarXu655RW6FF/8DZm3XKw==</latexit> <latexit sha1_base64="Xyx7hozIAJqxe7UkbLyu9FAGh4k=">AAACmnicbVFNb9NAEN24fJTw0QSOcFg1qsQhWHaDWpB6qOAAqJcgNW1REkXr9dhZZb1r7Y5RXcv/gT/Fj+A3cIU7a6eHtmGkXT3Nm9k3+ybKpbAYBL863ta9+w8ebj/qPn7y9NlOr//8zOrCcJhwLbW5iJgFKRRMUKCEi9wAyyIJ59HqY8OffwdjhVanWOYwz1iqRCI4Q5da9BazGBLX275UMZOVqQFQdWXSqK4Cf//9kAb+aNTc4ajuzhAucaP4zUzqtO7u3WANxHUV1YveIPCDNugmCK/B4Hj327j/4+RovOh3BrNY8yIDhVwya6dhkOPcqaHgEtwEhYWc8RVLYeqgYhnYedWK1nTPZWKaaOOOQtpmb3ZULLO2zCJXmTFc2rtck/wfNy0weTevhMoLBMXXQkkhKWra2EpjYYCjLB1g3Ag3K+VLZhhHZ/4tFRSrq/UvGiRFZJgpq0TgMNdWNFsRKh3GbiOm3ZH1czdNpk2+bAjOJK+7ztbwromb4GzfDw/8t1+dv5/IOrbJS7JLXpOQHJJj8pmMyYRw8pP8Jn/IX++V98H74p2sS73Odc8Lciu803/si9KX</latexit>

w1
<latexit sha1_base64="COapl/sdFPPJrahwwPVDqoV/7GU=">AAACT3icbZDNTuMwFIWdDr8dBsrADhYRFdIsUJUgBLNEmgUsQaKA1HYqx7lprTq2Zd8AJcpy3mO2sJ39vANLnoQdwkm74O9Klj6dc+17fSItuMUgePRqX2Zm5+YXFutfl74trzRWv59blRkGbaaEMpcRtSC4hDZyFHCpDdA0EnARjX6V/sUVGMuVPMOxhl5KB5InnFF0Ur+x3kW4weqd3EBc5Nf9sOg3mkErqMr/COEUmoebf5Z+/1v7f9Jf9ZrdWLEsBYlMUGs7YaCxl1ODnAko6t3MgqZsRAfQcShpCraXV2MLf9spsZ8o445Ev1Jf38hpau04jVxnSnFo33ul+JnXyTD52cu51BmCZJNBSSZ8VH6ZhR9zAwzF2AFlhrtdfTakhjJ0ib2Zgnx0O/lFSYJHhppxnnDc0cryMkouBzsxMGWqYG1Lu21SZfSwNBgVrKi7WMP3IX6E891WuN/aO3X5HpFJLZANskV+kJAckENyTE5ImzByS/6SO3LvPXhP3nNt2lrzprBG3lRt8QV9orgv</latexit>

⇥ b
<latexit sha1_base64="D4FdYbbBfh/yXoNhfuguCRKIUvI=">AAAClHicbVHbitswEFXc29a9JS30pS9mw0Khwdib0t3CPiwspYVCSaHZC0kIsjx2RCRZSONS1/gf+kf9jn5AX9tfqOzsw262AxKHc2Y0ozOJFtxiFP3qebdu37l7b+e+/+Dho8dP+oOnp7YoDYMpK0RhzhNqQXAFU+Qo4FwboDIRcJasT1r97CsYywv1BSsNC0lzxTPOKDpq2b+Yp5C52u6lmhpZ5QZANbXJk6aOwv23oyAKx+P2jseNvzdH+Ibb2a/8K7SBtKmTZtkfRmHURXATxJdgeLx7MRn8+Hg0WQ56w3lasFKCQiaotbM40rhwbZAzAY0/Ly1oytY0h5mDikqwi7pr2gR7jkmDrDDuKAw69mpFTaW1lUxcpqS4sttaS/5Pm5WYHS5qrnSJoNimUVaKAIugNTRIuQGGonKAMsPdrAFbUUMZOtuvdUG+/r75RYsETww1VZ1xHOnC8nYfXOWj1O3CdNuxoXbTyMLoVSswKljjO1vjbRNvgtP9MH4Tvv7s/H1PNrFDXpBd8pLE5IAckw9kQqaEkZ/kN/lD/nrPvSPvxHu3SfV6lzXPyLXwPv0DbsHPww==</latexit>

<latexit sha1_base64="ZEhmNtmWna9lR+bGQjd+LwCuntg=">AAAChXicbVFNj9MwEHXDxy7lY1s4crG2QuJQomS7WkBCYiUOILgUie4uaqvKcSapVdux7AkiRPkP/BR+DRIn+Ck46R7YLiPZepo3M2/8nBgpHEbRr15w4+at23v7d/p3791/cDAYPjxzRWk5zHghC3uRMAdSaJihQAkXxgJTiYTzZPOm5c+/gHWi0J+wMrBULNciE5yhT60G7xcpZL63m1Qzq6rcAuimtnnS1FF49HJMo3Ayae940vQXCF9xt3iBQoFbDUZRGHVBr4P4EoxODz9Ph98/vJquhr3RIi14qUAjl8y5eRwZXPrBKLgEL1Y6MIxvWA5zDzXzKsu6U2/oE59JaVZYfzTSLvtvR82Uc5VKfKViuHa7XJv8HzcvMXuxrIU2JYLmW6GslBQL2jpIU2GBo6w8YNwKvyvla2YZR+/zFRUUm2/bV7RIisQyW9WZwLEpnGg/QOh8nHrzbfcdLjR+G1VYs24JziRv+t7WeNfE6+DsKIxPwuOP3t+3ZBv75DE5JE9JTJ6TU/KOTMmMcPKD/CS/yZ9gL3gWHAcn29Kgd9nziFyJ4PVfIMbJhA==</latexit>

x2
<latexit sha1_base64="+QD6lLYrlFcju7OsE+tVFe8fWe4=">AAACPHicbVA9SwNBEN3z2/gZLbU4DIJFCHciahmw0FLRfEASw95mL1myt7vszonxSGlpqz/E3v9hbye21u7lUhjjwMDjvRnmzQsUZwY8792ZmZ2bX1hcWs6trK6tb2zmt6pGxprQCpFc6nqADeVM0Aow4LSuNMVRwGkt6J+leu2OasOkuIGBoq0IdwULGcFgqev79mF7s+CVvFG508Afg0J593H19nX77bKddwrNjiRxRAUQjo1p+J6CVoI1MMLpMNeMDVWY9HGXNiwUOKKmlYy8Dt19y3TcUGrbAtwR+3sjwZExgyiwkxGGnvmrpeR/WiOG8LSVMKFioIJkh8KYuyDd9HG3wzQlwAcWYKKZ9eqSHtaYgI1n4gqw/kP2RYo4CzTWgyRkUFTSsDQ3JrrFDiVSj1I0JWXdRFKrXioQzMkwZ2P1/4Y4DaqHJf+4dHRl8z1HWS2hHbSHDpCPTlAZXaBLVEEEddETekYvzpvz4Xw6X9nojDPe2UYT5Xz/ALfesek=</latexit>

w2
<latexit sha1_base64="bS4yJRC3gCcytIHoJK4H5zT95/g=">AAACT3icbZDPThsxEMa9KZQQoA1Nb3BYNULigKJdVAFHJA5wpBIBpCREXu9sYsVrW/YsbVjtkffgClfuvEOPfZLeKrybHPjTkSz99H1jz/iLtOAWg+CPV/uwsPhxqb7cWFld+/S5uf7l3KrMMOgyJZS5jKgFwSV0kaOAS22AppGAi2hyVPoX12AsV/IMpxoGKR1JnnBG0UnD5tc+wi+s3skNxEX+c7hbDJvtoBNU5b+HcA7tw83b1avH1tPpcN1r92PFshQkMkGt7YWBxkFODXImoGj0MwuasgkdQc+hpCnYQV6NLfwtp8R+oow7Ev1KfXkjp6m10zRynSnFsX3rleL/vF6GycEg51JnCJLNBiWZ8FH5ZRZ+zA0wFFMHlBnudvXZmBrK0CX2agryyc3sFyUJHhlqpnnCcUcry8souRztxMCUqYK1He22SZXR49JgVLCi4WIN34b4Hs53O+Fe5/sPl+8xmVWdbJBvZJuEZJ8ckhNySrqEkRtyR+7Jg/fb++v9q81ba94cWuRV1ZafAX95uDA=</latexit>
Computational Graphs
Input x1
<latexit sha1_base64="Y1MfsLqoTt9iL03xQrfEyz0C1Wk=">AAACPHicbVA9SwNBEN2L3/EraqnFYRAsQrgTUcuAhZaKJhGSGPY2e8mSvd1ld06MR0pLW/0h9vkf9nZia+1eLoVGBwYe780wb16gODPgeW9ObmZ2bn5hcSm/vLK6tl7Y2KwZGWtCq0RyqW8CbChnglaBAac3SlMcBZzWg/5pqtfvqDZMimsYKNqKcFewkBEMlrq6b/vtQtEre+Ny/wJ/AoqVnceV29et0UV7wyk2O5LEERVAODam4XsKWgnWwAinw3wzNlRh0sdd2rBQ4IiaVjL2OnT3LNNxQ6ltC3DH7M+NBEfGDKLATkYYemZaS8n/tEYM4UkrYULFQAXJDoUxd0G66eNuh2lKgA8swEQz69UlPawxARvPryvA+g/ZFyniLNBYD5KQQUlJw9LcmOiWOpRIPU7RlJV1E0mteqlAMCfDvI3Vnw7xL6gdlP2j8uGlzfcMZbWIttEu2kc+OkYVdI4uUBUR1EVP6Bm9OCPn3flwPrPRnDPZ2UK/yvn6BrYIseg=</latexit>

x2
<latexit sha1_base64="+QD6lLYrlFcju7OsE+tVFe8fWe4=">AAACPHicbVA9SwNBEN3z2/gZLbU4DIJFCHciahmw0FLRfEASw95mL1myt7vszonxSGlpqz/E3v9hbye21u7lUhjjwMDjvRnmzQsUZwY8792ZmZ2bX1hcWs6trK6tb2zmt6pGxprQCpFc6nqADeVM0Aow4LSuNMVRwGkt6J+leu2OasOkuIGBoq0IdwULGcFgqev79mF7s+CVvFG508Afg0J593H19nX77bKddwrNjiRxRAUQjo1p+J6CVoI1MMLpMNeMDVWY9HGXNiwUOKKmlYy8Dt19y3TcUGrbAtwR+3sjwZExgyiwkxGGnvmrpeR/WiOG8LSVMKFioIJkh8KYuyDd9HG3wzQlwAcWYKKZ9eqSHtaYgI1n4gqw/kP2RYo4CzTWgyRkUFTSsDQ3JrrFDiVSj1I0JWXdRFKrXioQzMkwZ2P1/4Y4DaqHJf+4dHRl8z1HWS2hHbSHDpCPTlAZXaBLVEEEddETekYvzpvz4Xw6X9nojDPe2UYT5Xz/ALfesek=</latexit>

Special
Parameter w1 w2
<latexit sha1_base64="bS4yJRC3gCcytIHoJK4H5zT95/g=">AAACT3icbZDPThsxEMa9KZQQoA1Nb3BYNULigKJdVAFHJA5wpBIBpCREXu9sYsVrW/YsbVjtkffgClfuvEOPfZLeKrybHPjTkSz99H1jz/iLtOAWg+CPV/uwsPhxqb7cWFld+/S5uf7l3KrMMOgyJZS5jKgFwSV0kaOAS22AppGAi2hyVPoX12AsV/IMpxoGKR1JnnBG0UnD5tc+wi+s3skNxEX+c7hbDJvtoBNU5b+HcA7tw83b1avH1tPpcN1r92PFshQkMkGt7YWBxkFODXImoGj0MwuasgkdQc+hpCnYQV6NLfwtp8R+oow7Ev1KfXkjp6m10zRynSnFsX3rleL/vF6GycEg51JnCJLNBiWZ8FH5ZRZ+zA0wFFMHlBnudvXZmBrK0CX2agryyc3sFyUJHhlqpnnCcUcry8souRztxMCUqYK1He22SZXR49JgVLCi4WIN34b4Hs53O+Fe5/sPl+8xmVWdbJBvZJuEZJ8ckhNySrqEkRtyR+7Jg/fb++v9q81ba94cWuRV1ZafAX95uDA=</latexit>

<latexit sha1_base64="COapl/sdFPPJrahwwPVDqoV/7GU=">AAACT3icbZDNTuMwFIWdDr8dBsrADhYRFdIsUJUgBLNEmgUsQaKA1HYqx7lprTq2Zd8AJcpy3mO2sJ39vANLnoQdwkm74O9Klj6dc+17fSItuMUgePRqX2Zm5+YXFutfl74trzRWv59blRkGbaaEMpcRtSC4hDZyFHCpDdA0EnARjX6V/sUVGMuVPMOxhl5KB5InnFF0Ur+x3kW4weqd3EBc5Nf9sOg3mkErqMr/COEUmoebf5Z+/1v7f9Jf9ZrdWLEsBYlMUGs7YaCxl1ODnAko6t3MgqZsRAfQcShpCraXV2MLf9spsZ8o445Ev1Jf38hpau04jVxnSnFo33ul+JnXyTD52cu51BmCZJNBSSZ8VH6ZhR9zAwzF2AFlhrtdfTakhjJ0ib2Zgnx0O/lFSYJHhppxnnDc0cryMkouBzsxMGWqYG1Lu21SZfSwNBgVrKi7WMP3IX6E891WuN/aO3X5HpFJLZANskV+kJAckENyTE5ImzByS/6SO3LvPXhP3nNt2lrzprBG3lRt8QV9orgv</latexit>

Expression h1 p1 loss
<latexit sha1_base64="0aMS+62nMB/pVmVWoZAvPgOQQf8=">AAACUHicbZDPThsxEMZnU0ohpZAU9QSHFVGlHlC0ixDtEakHOFKpAaQkjbzObGLFa1v2bNV0tcc+SK/lyJ2H4NY36Y16NyDxbyTLn77P9ox/iZHCURT9DRovll4uv1pZbb5ee7O+0Wq/PXU6txx7XEttzxPmUAqFPRIk8dxYZFki8SyZfa7ys+9ondDqK80NDjM2USIVnJG3Rq13A8IfVL9TJDLHspiO4nLU6kTdqK7wqYhvRedw+9fat8vNq5NRO+gMxprnGSrikjnXjyNDw4JZElxi2RzkDg3jMzbBvpeKZeiGRd23DN97Zxym2vqlKKzd+zcKljk3zxJ/MmM0dY+zynwu6+eUfhoWQpmcUPFFozSXIemwghGOhUVOcu4F41b4WUM+ZZZx8sgedCEx+7n4RaWkSCyz8yIVtGu0ExVLoSa7Y+Ta1mRd1/hpMm3NtAo4k7xseqzxY4hPxeleNz7o7n/xfI9gUSuwBTvwAWL4CIdwDCfQAw4F/IY/cBFcB/+Cm0awOHq3wyY8qEbzP1gGuJc=</latexit>

<latexit sha1_base64="/WnQCs7ketQzMhW2Yz5e6u/ygwU=">AAACWHicbVBNa9tAEB0raRI7beqkxxIQMYEegpFCaXIpMeTQHlOIk4BtzGo9shevdre7o1JX6NSf0lOu7e/oOaS/pis5h3wNLDzem5k3+xIjhaMoum0EK6sv1tY3mq3Nl6+2Xre3dy6czi3HPtdS26uEOZRCYZ8ESbwyFlmWSLxM5qeVfvkNrRNandPC4ChjUyVSwRl5atzeHRJ+p3pPkcgcy6ImCqmdK8txuxN1o7rCpyC+A52Tv+bXTevj17PxdqMznGieZ6iIS+bcII4MjQpmSXCJZWuYOzSMz9kUBx4qlqEbFbV/Ge57ZhKm2vqnKKzZ+xMFy5xbZInvzBjN3GOtIp/TBjmlx6NCKJMTKr40SnMZkg6rUMKJsMhJLjxg3Ap/a8hnzDJOProHLiTmP5a/qJAUiWV2UaSCDox2ospUqOnBBLm2dcKua/w1mbZmVgmcSV62fKzx4xCfgovDbvyh+/5L1Ol9gmVtwFvYg3cQwxH04DOcQR84/IRr+A1/Gv8CCNaD5rI1aNzNvIEHFez8Byncu6o=</latexit>

<latexit sha1_base64="4UhpAX1AVR8YjJNAOx3goAoXdmg=">AAACUHicbZDBThsxEIZnQ0tpaCEB9dTLigipBxTtIkQ59IDEoZW4pFIDVEkUeZ3ZxIrXtuxZ1LDad+AVuLZvw4034Ua9m1Yq0JEs//p/2zP+EiOFoyi6CxorL16uvlp73Vx/83Zjs9XeOnM6txz7XEttLxLmUAqFfRIk8cJYZFki8TyZn1T5+SVaJ7T6RguDo4xNlUgFZ+StcevdkPAH1e8UicyxLMw4LsetTtSN6gqfi/iP6BzvfO+1r08/9cbtoDOcaJ5nqIhL5twgjgyNCmZJcIllc5g7NIzP2RQHXiqWoRsVdd8y3PXOJEy19UtRWLv/3ihY5twiS/zJjNHMPc0q83/ZIKf0aFQIZXJCxZeN0lyGpMMKRjgRFjnJhReMW+FnDfmMWcbJI3vUhcT8avmLSkmRWGYXRSpoz2gnKpZCTfcmyLWtybqu8dNk2ppZFXAmedn0WOOnEJ+Ls/1ufNg9+Or5foZlrcF72IEPEMNHOIYv0IM+cCjgBn7Cr+A2uA8eGsHy6N8dtuFRNZq/AWReuAM=</latexit>

Operation ⇥ + sigmoid log


<latexit sha1_base64="lUzUAnzIFb5iwZmg1DHP9+oSaps=">AAACo3icbVFNa9tAEF2rX6n7ZbfHXkSMoQcjpLgkLeQQ6KGFXNwSJym2MKvVSF68H2J3VKIK/Yf+st77P3pt6UrOIYk7sMvjvZmd2XlJIbjFMPzV8+7df/Dw0d7j/pOnz56/GAxfnltdGgZzpoU2lwm1ILiCOXIUcFkYoDIRcJFsPrT6xTcwlmt1hlUBsaS54hlnFB21GqyXKWSutnuppkZWuQFQTW3ypKnD4OD9xA+D6bS9o2nTXyJc4U5yx9aW51LztGn64xtpBtKmTprVYBQGYRf+Loiuwehk/+ts+OP0eLYa9kbLVLNSgkImqLWLKCwwdm2RMwFulNJCQdmG5rBwUFEJNq67po0/dkzqZ9q4o9Dv2JsVNZXWVjJxmZLi2t7VWvJ/2qLE7F1cc1WUCIptG2Wl8FH77X79lBtgKCoHKDPczeqzNTWUoXPhVhfkm+/bX7RI8MRQU9UZx0mhLW/t4SqfpM4a05llg8JNI7Up1q3AqGBN3601urvEXXB+EESHwdvPbr8fyTb2yGuyT96QiByRE/KJzMicMPKT/CZ/yF9v7J16X7yzbarXu655RW6FF/8DZm3XKw==</latexit> <latexit sha1_base64="Xyx7hozIAJqxe7UkbLyu9FAGh4k=">AAACmnicbVFNb9NAEN24fJTw0QSOcFg1qsQhWHaDWpB6qOAAqJcgNW1REkXr9dhZZb1r7Y5RXcv/gT/Fj+A3cIU7a6eHtmGkXT3Nm9k3+ybKpbAYBL863ta9+w8ebj/qPn7y9NlOr//8zOrCcJhwLbW5iJgFKRRMUKCEi9wAyyIJ59HqY8OffwdjhVanWOYwz1iqRCI4Q5da9BazGBLX275UMZOVqQFQdWXSqK4Cf//9kAb+aNTc4ajuzhAucaP4zUzqtO7u3WANxHUV1YveIPCDNugmCK/B4Hj327j/4+RovOh3BrNY8yIDhVwya6dhkOPcqaHgEtwEhYWc8RVLYeqgYhnYedWK1nTPZWKaaOOOQtpmb3ZULLO2zCJXmTFc2rtck/wfNy0weTevhMoLBMXXQkkhKWra2EpjYYCjLB1g3Ag3K+VLZhhHZ/4tFRSrq/UvGiRFZJgpq0TgMNdWNFsRKh3GbiOm3ZH1czdNpk2+bAjOJK+7ztbwromb4GzfDw/8t1+dv5/IOrbJS7JLXpOQHJJj8pmMyYRw8pP8Jn/IX++V98H74p2sS73Odc8Lciu803/si9KX</latexit>

<latexit sha1_base64="hSo+8nFIIzdpadb5Q6VOnv+Q6mI=">AAACgHicbVHLjtMwFHXDayivFpZsrKmQRqIKyRQxA2IxEkIgsSkSnRnUVpXj3KRWbceybxAhyj/wH3wNS+BncNJZMB2uZOvonPvycWKkcBhFv3rBtes3bt7au92/c/fe/QeD4cNTV5SWw4wXsrDnCXMghYYZCpRwbiwwlUg4SzZvWv3sC1gnCv0JKwNLxXItMsEZemo1eLtIIfO1XaeaWVXlFkA3tc2Tpo7Cw5djGoWTSXvHk6a/QPiKu8lPV4NRFEZd0KsgvgCjk/3P0+H3D6+nq2FvtEgLXirQyCVzbh5HBpe+Jwouwc8pHRjGNyyHuYeaKXDLuhvc0CeeSWlWWH800o79t6JmyrlKJT5TMVy7Xa0l/6fNS8yOl7XQpkTQfDsoKyXFgrbm0VRY4CgrDxi3wu9K+ZpZxtFbfGkKis237StaJEVima3qTODYFE603gudj1Pvu+1+woXGb6MKa9atwJnkTd/bGu+aeBWcHobxi/D5R+/vO7KNPfKY7JMDEpMjckLekymZEU5+kJ/kN/kTBMFB8CyIt6lB76LmEbkUwau/nLHG/w==</latexit>

<latexit sha1_base64="ZEhmNtmWna9lR+bGQjd+LwCuntg=">AAAChXicbVFNj9MwEHXDxy7lY1s4crG2QuJQomS7WkBCYiUOILgUie4uaqvKcSapVdux7AkiRPkP/BR+DRIn+Ck46R7YLiPZepo3M2/8nBgpHEbRr15w4+at23v7d/p3791/cDAYPjxzRWk5zHghC3uRMAdSaJihQAkXxgJTiYTzZPOm5c+/gHWi0J+wMrBULNciE5yhT60G7xcpZL63m1Qzq6rcAuimtnnS1FF49HJMo3Ayae940vQXCF9xt3iBQoFbDUZRGHVBr4P4EoxODz9Ph98/vJquhr3RIi14qUAjl8y5eRwZXPrBKLgEL1Y6MIxvWA5zDzXzKsu6U2/oE59JaVZYfzTSLvtvR82Uc5VKfKViuHa7XJv8HzcvMXuxrIU2JYLmW6GslBQL2jpIU2GBo6w8YNwKvyvla2YZR+/zFRUUm2/bV7RIisQyW9WZwLEpnGg/QOh8nHrzbfcdLjR+G1VYs24JziRv+t7WeNfE6+DsKIxPwuOP3t+3ZBv75DE5JE9JTJ6TU/KOTMmMcPKD/CS/yZ9gL3gWHAcn29Kgd9nziFyJ4PVfIMbJhA==</latexit>

x1
<latexit sha1_base64="Y1MfsLqoTt9iL03xQrfEyz0C1Wk=">AAACPHicbVA9SwNBEN2L3/EraqnFYRAsQrgTUcuAhZaKJhGSGPY2e8mSvd1ld06MR0pLW/0h9vkf9nZia+1eLoVGBwYe780wb16gODPgeW9ObmZ2bn5hcSm/vLK6tl7Y2KwZGWtCq0RyqW8CbChnglaBAac3SlMcBZzWg/5pqtfvqDZMimsYKNqKcFewkBEMlrq6b/vtQtEre+Ny/wJ/AoqVnceV29et0UV7wyk2O5LEERVAODam4XsKWgnWwAinw3wzNlRh0sdd2rBQ4IiaVjL2OnT3LNNxQ6ltC3DH7M+NBEfGDKLATkYYemZaS8n/tEYM4UkrYULFQAXJDoUxd0G66eNuh2lKgA8swEQz69UlPawxARvPryvA+g/ZFyniLNBYD5KQQUlJw9LcmOiWOpRIPU7RlJV1E0mteqlAMCfDvI3Vnw7xL6gdlP2j8uGlzfcMZbWIttEu2kc+OkYVdI4uUBUR1EVP6Bm9OCPn3flwPrPRnDPZ2UK/yvn6BrYIseg=</latexit>

+
<latexit sha1_base64="hSo+8nFIIzdpadb5Q6VOnv+Q6mI=">AAACgHicbVHLjtMwFHXDayivFpZsrKmQRqIKyRQxA2IxEkIgsSkSnRnUVpXj3KRWbceybxAhyj/wH3wNS+BncNJZMB2uZOvonPvycWKkcBhFv3rBtes3bt7au92/c/fe/QeD4cNTV5SWw4wXsrDnCXMghYYZCpRwbiwwlUg4SzZvWv3sC1gnCv0JKwNLxXItMsEZemo1eLtIIfO1XaeaWVXlFkA3tc2Tpo7Cw5djGoWTSXvHk6a/QPiKu8lPV4NRFEZd0KsgvgCjk/3P0+H3D6+nq2FvtEgLXirQyCVzbh5HBpe+Jwouwc8pHRjGNyyHuYeaKXDLuhvc0CeeSWlWWH800o79t6JmyrlKJT5TMVy7Xa0l/6fNS8yOl7XQpkTQfDsoKyXFgrbm0VRY4CgrDxi3wu9K+ZpZxtFbfGkKis237StaJEVima3qTODYFE603gudj1Pvu+1+woXGb6MKa9atwJnkTd/bGu+aeBWcHobxi/D5R+/vO7KNPfKY7JMDEpMjckLekymZEU5+kJ/kN/kTBMFB8CyIt6lB76LmEbkUwau/nLHG/w==</latexit>

h1 p1 loss
<latexit sha1_base64="0aMS+62nMB/pVmVWoZAvPgOQQf8=">AAACUHicbZDPThsxEMZnU0ohpZAU9QSHFVGlHlC0ixDtEakHOFKpAaQkjbzObGLFa1v2bNV0tcc+SK/lyJ2H4NY36Y16NyDxbyTLn77P9ox/iZHCURT9DRovll4uv1pZbb5ee7O+0Wq/PXU6txx7XEttzxPmUAqFPRIk8dxYZFki8SyZfa7ys+9ondDqK80NDjM2USIVnJG3Rq13A8IfVL9TJDLHspiO4nLU6kTdqK7wqYhvRedw+9fat8vNq5NRO+gMxprnGSrikjnXjyNDw4JZElxi2RzkDg3jMzbBvpeKZeiGRd23DN97Zxym2vqlKKzd+zcKljk3zxJ/MmM0dY+zynwu6+eUfhoWQpmcUPFFozSXIemwghGOhUVOcu4F41b4WUM+ZZZx8sgedCEx+7n4RaWkSCyz8yIVtGu0ExVLoSa7Y+Ta1mRd1/hpMm3NtAo4k7xseqzxY4hPxeleNz7o7n/xfI9gUSuwBTvwAWL4CIdwDCfQAw4F/IY/cBFcB/+Cm0awOHq3wyY8qEbzP1gGuJc=</latexit>

<latexit sha1_base64="/WnQCs7ketQzMhW2Yz5e6u/ygwU=">AAACWHicbVBNa9tAEB0raRI7beqkxxIQMYEegpFCaXIpMeTQHlOIk4BtzGo9shevdre7o1JX6NSf0lOu7e/oOaS/pis5h3wNLDzem5k3+xIjhaMoum0EK6sv1tY3mq3Nl6+2Xre3dy6czi3HPtdS26uEOZRCYZ8ESbwyFlmWSLxM5qeVfvkNrRNandPC4ChjUyVSwRl5atzeHRJ+p3pPkcgcy6ImCqmdK8txuxN1o7rCpyC+A52Tv+bXTevj17PxdqMznGieZ6iIS+bcII4MjQpmSXCJZWuYOzSMz9kUBx4qlqEbFbV/Ge57ZhKm2vqnKKzZ+xMFy5xbZInvzBjN3GOtIp/TBjmlx6NCKJMTKr40SnMZkg6rUMKJsMhJLjxg3Ap/a8hnzDJOProHLiTmP5a/qJAUiWV2UaSCDox2ospUqOnBBLm2dcKua/w1mbZmVgmcSV62fKzx4xCfgovDbvyh+/5L1Ol9gmVtwFvYg3cQwxH04DOcQR84/IRr+A1/Gv8CCNaD5rI1aNzNvIEHFez8Byncu6o=</latexit>

<latexit sha1_base64="4UhpAX1AVR8YjJNAOx3goAoXdmg=">AAACUHicbZDBThsxEIZnQ0tpaCEB9dTLigipBxTtIkQ59IDEoZW4pFIDVEkUeZ3ZxIrXtuxZ1LDad+AVuLZvw4034Ua9m1Yq0JEs//p/2zP+EiOFoyi6CxorL16uvlp73Vx/83Zjs9XeOnM6txz7XEttLxLmUAqFfRIk8cJYZFki8TyZn1T5+SVaJ7T6RguDo4xNlUgFZ+StcevdkPAH1e8UicyxLMw4LsetTtSN6gqfi/iP6BzvfO+1r08/9cbtoDOcaJ5nqIhL5twgjgyNCmZJcIllc5g7NIzP2RQHXiqWoRsVdd8y3PXOJEy19UtRWLv/3ihY5twiS/zJjNHMPc0q83/ZIKf0aFQIZXJCxZeN0lyGpMMKRjgRFjnJhReMW+FnDfmMWcbJI3vUhcT8avmLSkmRWGYXRSpoz2gnKpZCTfcmyLWtybqu8dNk2ppZFXAmedn0WOOnEJ+Ls/1ufNg9+Or5foZlrcF72IEPEMNHOIYv0IM+cCjgBn7Cr+A2uA8eGsHy6N8dtuFRNZq/AWReuAM=</latexit>


<latexit sha1_base64="ZEhmNtmWna9lR+bGQjd+LwCuntg=">AAAChXicbVFNj9MwEHXDxy7lY1s4crG2QuJQomS7WkBCYiUOILgUie4uaqvKcSapVdux7AkiRPkP/BR+DRIn+Ck46R7YLiPZepo3M2/8nBgpHEbRr15w4+at23v7d/p3791/cDAYPjxzRWk5zHghC3uRMAdSaJihQAkXxgJTiYTzZPOm5c+/gHWi0J+wMrBULNciE5yhT60G7xcpZL63m1Qzq6rcAuimtnnS1FF49HJMo3Ayae940vQXCF9xt3iBQoFbDUZRGHVBr4P4EoxODz9Ph98/vJquhr3RIi14qUAjl8y5eRwZXPrBKLgEL1Y6MIxvWA5zDzXzKsu6U2/oE59JaVZYfzTSLvtvR82Uc5VKfKViuHa7XJv8HzcvMXuxrIU2JYLmW6GslBQL2jpIU2GBo6w8YNwKvyvla2YZR+/zFRUUm2/bV7RIisQyW9WZwLEpnGg/QOh8nHrzbfcdLjR+G1VYs24JziRv+t7WeNfE6+DsKIxPwuOP3t+3ZBv75DE5JE9JTJ6TU/KOTMmMcPKD/CS/yZ9gL3gWHAcn29Kgd9nziFyJ4PVfIMbJhA==</latexit>

sigmoid log
<latexit sha1_base64="lUzUAnzIFb5iwZmg1DHP9+oSaps=">AAACo3icbVFNa9tAEF2rX6n7ZbfHXkSMoQcjpLgkLeQQ6KGFXNwSJym2MKvVSF68H2J3VKIK/Yf+st77P3pt6UrOIYk7sMvjvZmd2XlJIbjFMPzV8+7df/Dw0d7j/pOnz56/GAxfnltdGgZzpoU2lwm1ILiCOXIUcFkYoDIRcJFsPrT6xTcwlmt1hlUBsaS54hlnFB21GqyXKWSutnuppkZWuQFQTW3ypKnD4OD9xA+D6bS9o2nTXyJc4U5yx9aW51LztGn64xtpBtKmTprVYBQGYRf+Loiuwehk/+ts+OP0eLYa9kbLVLNSgkImqLWLKCwwdm2RMwFulNJCQdmG5rBwUFEJNq67po0/dkzqZ9q4o9Dv2JsVNZXWVjJxmZLi2t7VWvJ/2qLE7F1cc1WUCIptG2Wl8FH77X79lBtgKCoHKDPczeqzNTWUoXPhVhfkm+/bX7RI8MRQU9UZx0mhLW/t4SqfpM4a05llg8JNI7Up1q3AqGBN3601urvEXXB+EESHwdvPbr8fyTb2yGuyT96QiByRE/KJzMicMPKT/CZ/yF9v7J16X7yzbarXu655RW6FF/8DZm3XKw==</latexit> <latexit sha1_base64="Xyx7hozIAJqxe7UkbLyu9FAGh4k=">AAACmnicbVFNb9NAEN24fJTw0QSOcFg1qsQhWHaDWpB6qOAAqJcgNW1REkXr9dhZZb1r7Y5RXcv/gT/Fj+A3cIU7a6eHtmGkXT3Nm9k3+ybKpbAYBL863ta9+w8ebj/qPn7y9NlOr//8zOrCcJhwLbW5iJgFKRRMUKCEi9wAyyIJ59HqY8OffwdjhVanWOYwz1iqRCI4Q5da9BazGBLX275UMZOVqQFQdWXSqK4Cf//9kAb+aNTc4ajuzhAucaP4zUzqtO7u3WANxHUV1YveIPCDNugmCK/B4Hj327j/4+RovOh3BrNY8yIDhVwya6dhkOPcqaHgEtwEhYWc8RVLYeqgYhnYedWK1nTPZWKaaOOOQtpmb3ZULLO2zCJXmTFc2rtck/wfNy0weTevhMoLBMXXQkkhKWra2EpjYYCjLB1g3Ag3K+VLZhhHZ/4tFRSrq/UvGiRFZJgpq0TgMNdWNFsRKh3GbiOm3ZH1czdNpk2+bAjOJK+7ztbwromb4GzfDw/8t1+dv5/IOrbJS7JLXpOQHJJj8pmMyYRw8pP8Jn/IX++V98H74p2sS73Odc8Lciu803/si9KX</latexit>

w1
<latexit sha1_base64="COapl/sdFPPJrahwwPVDqoV/7GU=">AAACT3icbZDNTuMwFIWdDr8dBsrADhYRFdIsUJUgBLNEmgUsQaKA1HYqx7lprTq2Zd8AJcpy3mO2sJ39vANLnoQdwkm74O9Klj6dc+17fSItuMUgePRqX2Zm5+YXFutfl74trzRWv59blRkGbaaEMpcRtSC4hDZyFHCpDdA0EnARjX6V/sUVGMuVPMOxhl5KB5InnFF0Ur+x3kW4weqd3EBc5Nf9sOg3mkErqMr/COEUmoebf5Z+/1v7f9Jf9ZrdWLEsBYlMUGs7YaCxl1ODnAko6t3MgqZsRAfQcShpCraXV2MLf9spsZ8o445Ev1Jf38hpau04jVxnSnFo33ul+JnXyTD52cu51BmCZJNBSSZ8VH6ZhR9zAwzF2AFlhrtdfTakhjJ0ib2Zgnx0O/lFSYJHhppxnnDc0cryMkouBzsxMGWqYG1Lu21SZfSwNBgVrKi7WMP3IX6E891WuN/aO3X5HpFJLZANskV+kJAckENyTE5ImzByS/6SO3LvPXhP3nNt2lrzprBG3lRt8QV9orgv</latexit>

⇥ b
<latexit sha1_base64="D4FdYbbBfh/yXoNhfuguCRKIUvI=">AAAClHicbVHbitswEFXc29a9JS30pS9mw0Khwdib0t3CPiwspYVCSaHZC0kIsjx2RCRZSONS1/gf+kf9jn5AX9tfqOzsw262AxKHc2Y0ozOJFtxiFP3qebdu37l7b+e+/+Dho8dP+oOnp7YoDYMpK0RhzhNqQXAFU+Qo4FwboDIRcJasT1r97CsYywv1BSsNC0lzxTPOKDpq2b+Yp5C52u6lmhpZ5QZANbXJk6aOwv23oyAKx+P2jseNvzdH+Ibb2a/8K7SBtKmTZtkfRmHURXATxJdgeLx7MRn8+Hg0WQ56w3lasFKCQiaotbM40rhwbZAzAY0/Ly1oytY0h5mDikqwi7pr2gR7jkmDrDDuKAw69mpFTaW1lUxcpqS4sttaS/5Pm5WYHS5qrnSJoNimUVaKAIugNTRIuQGGonKAMsPdrAFbUUMZOtuvdUG+/r75RYsETww1VZ1xHOnC8nYfXOWj1O3CdNuxoXbTyMLoVSswKljjO1vjbRNvgtP9MH4Tvv7s/H1PNrFDXpBd8pLE5IAckw9kQqaEkZ/kN/lD/nrPvSPvxHu3SfV6lzXPyLXwPv0DbsHPww==</latexit>

<latexit sha1_base64="ZEhmNtmWna9lR+bGQjd+LwCuntg=">AAAChXicbVFNj9MwEHXDxy7lY1s4crG2QuJQomS7WkBCYiUOILgUie4uaqvKcSapVdux7AkiRPkP/BR+DRIn+Ck46R7YLiPZepo3M2/8nBgpHEbRr15w4+at23v7d/p3791/cDAYPjxzRWk5zHghC3uRMAdSaJihQAkXxgJTiYTzZPOm5c+/gHWi0J+wMrBULNciE5yhT60G7xcpZL63m1Qzq6rcAuimtnnS1FF49HJMo3Ayae940vQXCF9xt3iBQoFbDUZRGHVBr4P4EoxODz9Ph98/vJquhr3RIi14qUAjl8y5eRwZXPrBKLgEL1Y6MIxvWA5zDzXzKsu6U2/oE59JaVZYfzTSLvtvR82Uc5VKfKViuHa7XJv8HzcvMXuxrIU2JYLmW6GslBQL2jpIU2GBo6w8YNwKvyvla2YZR+/zFRUUm2/bV7RIisQyW9WZwLEpnGg/QOh8nHrzbfcdLjR+G1VYs24JziRv+t7WeNfE6+DsKIxPwuOP3t+3ZBv75DE5JE9JTJ6TU/KOTMmMcPKD/CS/yZ9gL3gWHAcn29Kgd9nziFyJ4PVfIMbJhA==</latexit>

x2
<latexit sha1_base64="+QD6lLYrlFcju7OsE+tVFe8fWe4=">AAACPHicbVA9SwNBEN3z2/gZLbU4DIJFCHciahmw0FLRfEASw95mL1myt7vszonxSGlpqz/E3v9hbye21u7lUhjjwMDjvRnmzQsUZwY8792ZmZ2bX1hcWs6trK6tb2zmt6pGxprQCpFc6nqADeVM0Aow4LSuNMVRwGkt6J+leu2OasOkuIGBoq0IdwULGcFgqev79mF7s+CVvFG508Afg0J593H19nX77bKddwrNjiRxRAUQjo1p+J6CVoI1MMLpMNeMDVWY9HGXNiwUOKKmlYy8Dt19y3TcUGrbAtwR+3sjwZExgyiwkxGGnvmrpeR/WiOG8LSVMKFioIJkh8KYuyDd9HG3wzQlwAcWYKKZ9eqSHtaYgI1n4gqw/kP2RYo4CzTWgyRkUFTSsDQ3JrrFDiVSj1I0JWXdRFKrXioQzMkwZ2P1/4Y4DaqHJf+4dHRl8z1HWS2hHbSHDpCPTlAZXaBLVEEEddETekYvzpvz4Xw6X9nojDPe2UYT5Xz/ALfesek=</latexit>

w2
<latexit sha1_base64="bS4yJRC3gCcytIHoJK4H5zT95/g=">AAACT3icbZDPThsxEMa9KZQQoA1Nb3BYNULigKJdVAFHJA5wpBIBpCREXu9sYsVrW/YsbVjtkffgClfuvEOPfZLeKrybHPjTkSz99H1jz/iLtOAWg+CPV/uwsPhxqb7cWFld+/S5uf7l3KrMMOgyJZS5jKgFwSV0kaOAS22AppGAi2hyVPoX12AsV/IMpxoGKR1JnnBG0UnD5tc+wi+s3skNxEX+c7hbDJvtoBNU5b+HcA7tw83b1avH1tPpcN1r92PFshQkMkGt7YWBxkFODXImoGj0MwuasgkdQc+hpCnYQV6NLfwtp8R+oow7Ev1KfXkjp6m10zRynSnFsX3rleL/vF6GycEg51JnCJLNBiWZ8FH5ZRZ+zA0wFFMHlBnudvXZmBrK0CX2agryyc3sFyUJHhlqpnnCcUcry8souRztxMCUqYK1He22SZXR49JgVLCi4WIN34b4Hs53O+Fe5/sPl+8xmVWdbJBvZJuEZJ8ckhNySrqEkRtyR+7Jg/fb++v9q81ba94cWuRV1ZafAX95uDA=</latexit>
How to minimize? (Automatic Differentiation)

f
<latexit sha1_base64="51r33id42qoQpOCb5sc2SQuMEtU=">AAACOnicbVA9SwNBEN3zM8avREubI0GwkHAnohYWAQsFmwhGIzHI3mYuWbK3u+zOCfHIL7Cx0H/jH7G1E1t/gHuJhV8DA4/3Zpg3L9KCWwyCF29qemZ2br6wUFxcWl5ZLZXXLqxKDYMmU0KZVkQtCC6hiRwFtLQBmkQCLqPBUa5f3oKxXMlzHGroJLQnecwZRUedxTelalALxuX/BeEXqNYrV43yw+lh46bsVa+7iqUJSGSCWtsOA42djBrkTMCoeJ1a0JQNaA/aDkqagO1kY6cjf9MxXT9WxrVEf8x+38hoYu0widxkQrFvf2s5+Z/WTjE+6GRc6hRBssmhOBU+Kj9/2+9yAwzF0AHKDHdefdanhjJ04fy4gnxwN/kiR4JHhpphFnPc1sryPDUue9tdYMqMM7Q17dwkyuh+LjAq2KjoYg1/h/gXXOzUwr3a7pnL95hMqkA2SIVskZDskzo5IQ3SJIwAuSeP5Ml79l69N+99Mjrlfe2skx/lfXwCPmCwlg==</latexit>

z
<latexit sha1_base64="C+JEy0i/mGSP1ACrQVI0dU4eCek=">AAACXXicbVDLSgMxFE3Hd31VXbhwEyyiggwzIupGKLjQpYJVoS0lk8m0oZkkJHeEOszer3GrP+BH6MZfMdN24etCyOGce7nnnkgLbiEI3ive1PTM7Nz8QnVxaXlltba2fmtVZihrUiWUuY+IZYJL1gQOgt1rw0gaCXYXDc5L/e6BGcuVvIGhZp2U9CRPOCXgqG5tux0pEdth6r68X+AznOx9px6L/W6tHvjBqPBfEE5AvbGbvfnmY/2qu1apt2NFs5RJoIJY2woDDZ2cGOBUsKLazizThA5Ij7UclCRltpOPjinwjmNinCjjngQ8Yr9P5CS1pTnXmRLo299aSf6ntTJITjs5lzoDJul4UZIJDAqXyeCYG0ZBDB0g1HDnFdM+MYSCy+/HFuCDx/EVJRI8MsQM84TDgVaWl8Fy2TuIGVVmFLP1tXOTKqP7pUCJoEXVxRr+DvEvuD30w2P/6Nrle4HGNY+20DbaQyE6QQ10ia5QE1H0hJ7RC3qtfHoz3pK3Mm71KpOZDfSjvM0vGAC71Q==</latexit>

h = f (z) h
<latexit sha1_base64="w0y6IvRoNAJvnv0COAjMii6XFzU=">AAACR3icbVDLSgMxFM3UV62vqks3g0VwIWVGRN2IggtdVrBVaEfJZDJtaCaJyR1hHPofbtWvceFa/Ap34tJM24WvCyGHc+7lnntCxZkBz3tzShOTU9Mz5dnK3PzC4lJ1eaVlZKoJbRLJpb4MsaGcCdoEBpxeKk1xEnJ6EfaPC/3ilmrDpDiHTNEgwV3BYkYwWOqqE0oemSyxX94bXFdrXt0blvsX+GNQO3xRT6+Vg5vG9bJT60SSpAkVQDg2pu17CoIca2CE00GlkxqqMOnjLm1bKHBCTZAPbQ/cDctEbiy1fQLcIft9IseJKbzZzgRDz/zWCvI/rZ1CvB/kTKgUqCCjRXHKXZBukYEbMU0J8MwCTDSzXl3SwxoTsEn92AKsfze6okCchRrrLI8ZbClpWBEhE92tiBKph4GaurJuEqlVrxAI5mRQsbH6v0P8C1rbdX+3vnPm1Y5O0KjKaA2to03koz10hE5RAzURQRrdowf06Dw7786H8zlqLTnjmVX0o0rOF/IdtyE=</latexit>

<latexit sha1_base64="8r8R7LlOnESwMqyVzXt31ZGJXcI=">AAACR3icbVDLSgMxFM3UV62vqks3g0VwIWVGRN2IBRe6rGBroa0lk8m0oZkkJneEOvQ/3Kpf48K1+BXuxKWZjgtfF0IO59zLPfcEijMDnvfqFKamZ2bnivOlhcWl5ZXy6lrTyEQT2iCSS90KsKGcCdoABpy2lKY4Dji9DIYnmX55Q7VhUlzASNFujPuCRYxgsNRVJ5A8NKPYfuntuFeueFVvUu5f4H+ByvGzenwpHV3Xe6tOpRNKksRUAOHYmLbvKeimWAMjnI5LncRQhckQ92nbQoFjarrpxPbY3bJM6EZS2yfAnbDfJ1Icm8yb7YwxDMxvLSP/09oJRIfdlAmVABUkXxQl3AXpZhm4IdOUAB9ZgIlm1qtLBlhjAjapH1uADW/zKzLEWaCxHqURgx0lDcsiZKK/E1Ii9SRQU1XWTSy1GmQCwZyMSzZW/3eIf0Fzt+rvV/fOvUrtFOVVRBtoE20jHx2gGjpDddRABGl0h+7Rg/PkvDnvzkfeWnC+ZtbRjyo4nxNKtzM=</latexit>

@s
<latexit sha1_base64="5pp5Zj8sufCFxucogNzyV8Hn83I=">AAACd3icbVFNTxQxGO4OqLh+LXj0QONGw4FsZgxBDySSeJAjJixgdjabTued3WY7bdO+Y1ib+TMc+C1c9eZ/4Gzizc4sCQK+SdMnz/N+9WlmpHAYx7860crqg4eP1h53nzx99vxFb33j2OnKchhyLbU9zZgDKRQMUaCEU2OBlZmEk2z+qdFPvoF1QqsjXBgYl2yqRCE4w0BNenspwhm2fXwmK6h9WljGfWqYRcEkdfUNTjMtc7cow+VndV1Pev14ELdB74PkGvT3P15ewNffV4eT9U4/zTWvSlDIJXNulMQGx77pzyXU3bRyYBifsymMAlSsBDf27Xo1fROYnBbahqOQtuy/FZ6VrtkuZJYMZ+6u1pD/00YVFh/GXihTISi+HFRUkqKmjWc0FxY4ykUAjFsRdqV8xoJNGJy9NQXF/PvyFQ2SIrPMLnwhcNtoJxrLhZpu58C1bT/ADUzYptTWzBqBM8nrbrA1uWvifXD8bpDsDna+BH8/k2WskVfkNdkiCXlP9skBOSRDwsk5uSQ/yM/On2gzehttLVOjznXNS3IrouQv5PnKBQ==</latexit>

@h
<latexit sha1_base64="pSbQF8Miw56tgZ5on8xVPlM0Wzc=">AAAChHicbVHLbhMxFHUGaEt4pbBkYxEhsYiiGQqUFVRiAeyKRNqiTBR5PHcSKx7bsu8gUsu/w5bf4AdYsIEfYM0az6QL+riS5aNz7svHhZHCYZr+7CXXrt/Y2t652b91+87de4Pd+0dON5bDhGup7UnBHEihYIICJZwYC6wuJBwXqzetfvwZrBNafcS1gVnNFkpUgjOM1HzwPkf4gl0fX8gGgs8ry7jPDbMomKR5oWXp1nW8/DKEq4XTEMJ8MEzHaRf0MsjOwPDg9fev8Onvn8P5bm+Yl5o3NSjkkjk3zVKDM9/25xJCP28cGMZXbAHTCBWrwc18t2ugjyNT0krbeBTSjv2/wrPatdvFzJrh0l3UWvIqbdpg9XLmhTINguKbQVUjKWraGkhLYYGjXEfAuBVxV8qXLHqG0eZzU1CsTjevaJEUhWV27SuBI6OdaP0XajEqgWvb/YYbm7hNra1ZtgJnkod+tDW7aOJlcPR0nL0YP/sQ/X1LNrFDHpJH5AnJyD45IO/IIZkQTr6RH+QX+Z1sJaNkL3m+SU16ZzUPyLlIXv0DSu3PGg==</latexit>

@s @s @h
<latexit sha1_base64="Qepk+stdO3Q91ukg46qZLK69qmA=">AAAC23icjVFNb9QwEHVSKGX52pYjF4sVEofVKkGl9IJaiQMci8S2RZto5TiTXWud2LInFdvIp94QV+6c+C38Bf4DZ8442UrQLUiMZPnpvRnP+E2mpbAYRd+DcOPGzc1bW7d7d+7eu/+gv71zbFVtOIy5ksqcZsyCFBWMUaCEU22AlZmEk2zxqtVPzsBYoap3uNSQlmxWiUJwhp6a9r8kCB+we6fJZA2uSQrDeJNoZlAwSa37jZNMydwuS381587Rl/Q/k+fOrWWuqf/s4ab9QTSKuqDXQXwJBocH377C+58/jqbbwSDJFa9LqJBLZu0kjjSmTfs+l+B6SW1BM75gM5h4WLESbNp0Fjj6xDM5LZTxp0LasX9WNKy07XQ+s2Q4t+taS/5Nm9RY7KeNqHSNUPFVo6KWFBVt90JzYYCjXHrAuBF+VsrnzHuGfntXuqBYnK9+0SIpMsPMsikEDrWyol2rqGbDHLgy3ZLtSPtpSmX0vBU4k9z1vK3xuonXwfGzUbw32n3r/X1NVrFFHpHH5CmJyQtySN6QIzImPNgMhsHzYC9Mw4vwY/hplRoGlzUPyZUIP/8CcizxUw==</latexit>

Chain rule =
@z @h @z @z @h
Downstream Local Upstream
Gradient Gradient Gradient
How to minimize? (Automatic Differentiation)

x1
<latexit sha1_base64="Y1MfsLqoTt9iL03xQrfEyz0C1Wk=">AAACPHicbVA9SwNBEN2L3/EraqnFYRAsQrgTUcuAhZaKJhGSGPY2e8mSvd1ld06MR0pLW/0h9vkf9nZia+1eLoVGBwYe780wb16gODPgeW9ObmZ2bn5hcSm/vLK6tl7Y2KwZGWtCq0RyqW8CbChnglaBAac3SlMcBZzWg/5pqtfvqDZMimsYKNqKcFewkBEMlrq6b/vtQtEre+Ny/wJ/AoqVnceV29et0UV7wyk2O5LEERVAODam4XsKWgnWwAinw3wzNlRh0sdd2rBQ4IiaVjL2OnT3LNNxQ6ltC3DH7M+NBEfGDKLATkYYemZaS8n/tEYM4UkrYULFQAXJDoUxd0G66eNuh2lKgA8swEQz69UlPawxARvPryvA+g/ZFyniLNBYD5KQQUlJw9LcmOiWOpRIPU7RlJV1E0mteqlAMCfDvI3Vnw7xL6gdlP2j8uGlzfcMZbWIttEu2kc+OkYVdI4uUBUR1EVP6Bm9OCPn3flwPrPRnDPZ2UK/yvn6BrYIseg=</latexit>

+
<latexit sha1_base64="hSo+8nFIIzdpadb5Q6VOnv+Q6mI=">AAACgHicbVHLjtMwFHXDayivFpZsrKmQRqIKyRQxA2IxEkIgsSkSnRnUVpXj3KRWbceybxAhyj/wH3wNS+BncNJZMB2uZOvonPvycWKkcBhFv3rBtes3bt7au92/c/fe/QeD4cNTV5SWw4wXsrDnCXMghYYZCpRwbiwwlUg4SzZvWv3sC1gnCv0JKwNLxXItMsEZemo1eLtIIfO1XaeaWVXlFkA3tc2Tpo7Cw5djGoWTSXvHk6a/QPiKu8lPV4NRFEZd0KsgvgCjk/3P0+H3D6+nq2FvtEgLXirQyCVzbh5HBpe+Jwouwc8pHRjGNyyHuYeaKXDLuhvc0CeeSWlWWH800o79t6JmyrlKJT5TMVy7Xa0l/6fNS8yOl7XQpkTQfDsoKyXFgrbm0VRY4CgrDxi3wu9K+ZpZxtFbfGkKis237StaJEVima3qTODYFE603gudj1Pvu+1+woXGb6MKa9atwJnkTd/bGu+aeBWcHobxi/D5R+/vO7KNPfKY7JMDEpMjckLekymZEU5+kJ/kN/kTBMFB8CyIt6lB76LmEbkUwau/nLHG/w==</latexit>

h1 p1 loss
<latexit sha1_base64="0aMS+62nMB/pVmVWoZAvPgOQQf8=">AAACUHicbZDPThsxEMZnU0ohpZAU9QSHFVGlHlC0ixDtEakHOFKpAaQkjbzObGLFa1v2bNV0tcc+SK/lyJ2H4NY36Y16NyDxbyTLn77P9ox/iZHCURT9DRovll4uv1pZbb5ee7O+0Wq/PXU6txx7XEttzxPmUAqFPRIk8dxYZFki8SyZfa7ys+9ondDqK80NDjM2USIVnJG3Rq13A8IfVL9TJDLHspiO4nLU6kTdqK7wqYhvRedw+9fat8vNq5NRO+gMxprnGSrikjnXjyNDw4JZElxi2RzkDg3jMzbBvpeKZeiGRd23DN97Zxym2vqlKKzd+zcKljk3zxJ/MmM0dY+zynwu6+eUfhoWQpmcUPFFozSXIemwghGOhUVOcu4F41b4WUM+ZZZx8sgedCEx+7n4RaWkSCyz8yIVtGu0ExVLoSa7Y+Ta1mRd1/hpMm3NtAo4k7xseqzxY4hPxeleNz7o7n/xfI9gUSuwBTvwAWL4CIdwDCfQAw4F/IY/cBFcB/+Cm0awOHq3wyY8qEbzP1gGuJc=</latexit>

<latexit sha1_base64="/WnQCs7ketQzMhW2Yz5e6u/ygwU=">AAACWHicbVBNa9tAEB0raRI7beqkxxIQMYEegpFCaXIpMeTQHlOIk4BtzGo9shevdre7o1JX6NSf0lOu7e/oOaS/pis5h3wNLDzem5k3+xIjhaMoum0EK6sv1tY3mq3Nl6+2Xre3dy6czi3HPtdS26uEOZRCYZ8ESbwyFlmWSLxM5qeVfvkNrRNandPC4ChjUyVSwRl5atzeHRJ+p3pPkcgcy6ImCqmdK8txuxN1o7rCpyC+A52Tv+bXTevj17PxdqMznGieZ6iIS+bcII4MjQpmSXCJZWuYOzSMz9kUBx4qlqEbFbV/Ge57ZhKm2vqnKKzZ+xMFy5xbZInvzBjN3GOtIp/TBjmlx6NCKJMTKr40SnMZkg6rUMKJsMhJLjxg3Ap/a8hnzDJOProHLiTmP5a/qJAUiWV2UaSCDox2ospUqOnBBLm2dcKua/w1mbZmVgmcSV62fKzx4xCfgovDbvyh+/5L1Ol9gmVtwFvYg3cQwxH04DOcQR84/IRr+A1/Gv8CCNaD5rI1aNzNvIEHFez8Byncu6o=</latexit>

<latexit sha1_base64="4UhpAX1AVR8YjJNAOx3goAoXdmg=">AAACUHicbZDBThsxEIZnQ0tpaCEB9dTLigipBxTtIkQ59IDEoZW4pFIDVEkUeZ3ZxIrXtuxZ1LDad+AVuLZvw4034Ua9m1Yq0JEs//p/2zP+EiOFoyi6CxorL16uvlp73Vx/83Zjs9XeOnM6txz7XEttLxLmUAqFfRIk8cJYZFki8TyZn1T5+SVaJ7T6RguDo4xNlUgFZ+StcevdkPAH1e8UicyxLMw4LsetTtSN6gqfi/iP6BzvfO+1r08/9cbtoDOcaJ5nqIhL5twgjgyNCmZJcIllc5g7NIzP2RQHXiqWoRsVdd8y3PXOJEy19UtRWLv/3ihY5twiS/zJjNHMPc0q83/ZIKf0aFQIZXJCxZeN0lyGpMMKRjgRFjnJhReMW+FnDfmMWcbJI3vUhcT8avmLSkmRWGYXRSpoz2gnKpZCTfcmyLWtybqu8dNk2ppZFXAmedn0WOOnEJ+Ls/1ufNg9+Or5foZlrcF72IEPEMNHOIYv0IM+cCjgBn7Cr+A2uA8eGsHy6N8dtuFRNZq/AWReuAM=</latexit>


<latexit sha1_base64="ZEhmNtmWna9lR+bGQjd+LwCuntg=">AAAChXicbVFNj9MwEHXDxy7lY1s4crG2QuJQomS7WkBCYiUOILgUie4uaqvKcSapVdux7AkiRPkP/BR+DRIn+Ck46R7YLiPZepo3M2/8nBgpHEbRr15w4+at23v7d/p3791/cDAYPjxzRWk5zHghC3uRMAdSaJihQAkXxgJTiYTzZPOm5c+/gHWi0J+wMrBULNciE5yhT60G7xcpZL63m1Qzq6rcAuimtnnS1FF49HJMo3Ayae940vQXCF9xt3iBQoFbDUZRGHVBr4P4EoxODz9Ph98/vJquhr3RIi14qUAjl8y5eRwZXPrBKLgEL1Y6MIxvWA5zDzXzKsu6U2/oE59JaVZYfzTSLvtvR82Uc5VKfKViuHa7XJv8HzcvMXuxrIU2JYLmW6GslBQL2jpIU2GBo6w8YNwKvyvla2YZR+/zFRUUm2/bV7RIisQyW9WZwLEpnGg/QOh8nHrzbfcdLjR+G1VYs24JziRv+t7WeNfE6+DsKIxPwuOP3t+3ZBv75DE5JE9JTJ6TU/KOTMmMcPKD/CS/yZ9gL3gWHAcn29Kgd9nziFyJ4PVfIMbJhA==</latexit>

sigmoid log
<latexit sha1_base64="lUzUAnzIFb5iwZmg1DHP9+oSaps=">AAACo3icbVFNa9tAEF2rX6n7ZbfHXkSMoQcjpLgkLeQQ6KGFXNwSJym2MKvVSF68H2J3VKIK/Yf+st77P3pt6UrOIYk7sMvjvZmd2XlJIbjFMPzV8+7df/Dw0d7j/pOnz56/GAxfnltdGgZzpoU2lwm1ILiCOXIUcFkYoDIRcJFsPrT6xTcwlmt1hlUBsaS54hlnFB21GqyXKWSutnuppkZWuQFQTW3ypKnD4OD9xA+D6bS9o2nTXyJc4U5yx9aW51LztGn64xtpBtKmTprVYBQGYRf+Loiuwehk/+ts+OP0eLYa9kbLVLNSgkImqLWLKCwwdm2RMwFulNJCQdmG5rBwUFEJNq67po0/dkzqZ9q4o9Dv2JsVNZXWVjJxmZLi2t7VWvJ/2qLE7F1cc1WUCIptG2Wl8FH77X79lBtgKCoHKDPczeqzNTWUoXPhVhfkm+/bX7RI8MRQU9UZx0mhLW/t4SqfpM4a05llg8JNI7Up1q3AqGBN3601urvEXXB+EESHwdvPbr8fyTb2yGuyT96QiByRE/KJzMicMPKT/CZ/yF9v7J16X7yzbarXu655RW6FF/8DZm3XKw==</latexit> <latexit sha1_base64="Xyx7hozIAJqxe7UkbLyu9FAGh4k=">AAACmnicbVFNb9NAEN24fJTw0QSOcFg1qsQhWHaDWpB6qOAAqJcgNW1REkXr9dhZZb1r7Y5RXcv/gT/Fj+A3cIU7a6eHtmGkXT3Nm9k3+ybKpbAYBL863ta9+w8ebj/qPn7y9NlOr//8zOrCcJhwLbW5iJgFKRRMUKCEi9wAyyIJ59HqY8OffwdjhVanWOYwz1iqRCI4Q5da9BazGBLX275UMZOVqQFQdWXSqK4Cf//9kAb+aNTc4ajuzhAucaP4zUzqtO7u3WANxHUV1YveIPCDNugmCK/B4Hj327j/4+RovOh3BrNY8yIDhVwya6dhkOPcqaHgEtwEhYWc8RVLYeqgYhnYedWK1nTPZWKaaOOOQtpmb3ZULLO2zCJXmTFc2rtck/wfNy0weTevhMoLBMXXQkkhKWra2EpjYYCjLB1g3Ag3K+VLZhhHZ/4tFRSrq/UvGiRFZJgpq0TgMNdWNFsRKh3GbiOm3ZH1czdNpk2+bAjOJK+7ztbwromb4GzfDw/8t1+dv5/IOrbJS7JLXpOQHJJj8pmMyYRw8pP8Jn/IX++V98H74p2sS73Odc8Lciu803/si9KX</latexit>

w1
<latexit sha1_base64="COapl/sdFPPJrahwwPVDqoV/7GU=">AAACT3icbZDNTuMwFIWdDr8dBsrADhYRFdIsUJUgBLNEmgUsQaKA1HYqx7lprTq2Zd8AJcpy3mO2sJ39vANLnoQdwkm74O9Klj6dc+17fSItuMUgePRqX2Zm5+YXFutfl74trzRWv59blRkGbaaEMpcRtSC4hDZyFHCpDdA0EnARjX6V/sUVGMuVPMOxhl5KB5InnFF0Ur+x3kW4weqd3EBc5Nf9sOg3mkErqMr/COEUmoebf5Z+/1v7f9Jf9ZrdWLEsBYlMUGs7YaCxl1ODnAko6t3MgqZsRAfQcShpCraXV2MLf9spsZ8o445Ev1Jf38hpau04jVxnSnFo33ul+JnXyTD52cu51BmCZJNBSSZ8VH6ZhR9zAwzF2AFlhrtdfTakhjJ0ib2Zgnx0O/lFSYJHhppxnnDc0cryMkouBzsxMGWqYG1Lu21SZfSwNBgVrKi7WMP3IX6E891WuN/aO3X5HpFJLZANskV+kJAckENyTE5ImzByS/6SO3LvPXhP3nNt2lrzprBG3lRt8QV9orgv</latexit>

⇥ b Gradient Flow Direction


<latexit sha1_base64="D4FdYbbBfh/yXoNhfuguCRKIUvI=">AAAClHicbVHbitswEFXc29a9JS30pS9mw0Khwdib0t3CPiwspYVCSaHZC0kIsjx2RCRZSONS1/gf+kf9jn5AX9tfqOzsw262AxKHc2Y0ozOJFtxiFP3qebdu37l7b+e+/+Dho8dP+oOnp7YoDYMpK0RhzhNqQXAFU+Qo4FwboDIRcJasT1r97CsYywv1BSsNC0lzxTPOKDpq2b+Yp5C52u6lmhpZ5QZANbXJk6aOwv23oyAKx+P2jseNvzdH+Ibb2a/8K7SBtKmTZtkfRmHURXATxJdgeLx7MRn8+Hg0WQ56w3lasFKCQiaotbM40rhwbZAzAY0/Ly1oytY0h5mDikqwi7pr2gR7jkmDrDDuKAw69mpFTaW1lUxcpqS4sttaS/5Pm5WYHS5qrnSJoNimUVaKAIugNTRIuQGGonKAMsPdrAFbUUMZOtuvdUG+/r75RYsETww1VZ1xHOnC8nYfXOWj1O3CdNuxoXbTyMLoVSswKljjO1vjbRNvgtP9MH4Tvv7s/H1PNrFDXpBd8pLE5IAckw9kQqaEkZ/kN/lD/nrPvSPvxHu3SfV6lzXPyLXwPv0DbsHPww==</latexit>

<latexit sha1_base64="ZEhmNtmWna9lR+bGQjd+LwCuntg=">AAAChXicbVFNj9MwEHXDxy7lY1s4crG2QuJQomS7WkBCYiUOILgUie4uaqvKcSapVdux7AkiRPkP/BR+DRIn+Ck46R7YLiPZepo3M2/8nBgpHEbRr15w4+at23v7d/p3791/cDAYPjxzRWk5zHghC3uRMAdSaJihQAkXxgJTiYTzZPOm5c+/gHWi0J+wMrBULNciE5yhT60G7xcpZL63m1Qzq6rcAuimtnnS1FF49HJMo3Ayae940vQXCF9xt3iBQoFbDUZRGHVBr4P4EoxODz9Ph98/vJquhr3RIi14qUAjl8y5eRwZXPrBKLgEL1Y6MIxvWA5zDzXzKsu6U2/oE59JaVZYfzTSLvtvR82Uc5VKfKViuHa7XJv8HzcvMXuxrIU2JYLmW6GslBQL2jpIU2GBo6w8YNwKvyvla2YZR+/zFRUUm2/bV7RIisQyW9WZwLEpnGg/QOh8nHrzbfcdLjR+G1VYs24JziRv+t7WeNfE6+DsKIxPwuOP3t+3ZBv75DE5JE9JTJ6TU/KOTMmMcPKD/CS/yZ9gL3gWHAcn29Kgd9nziFyJ4PVfIMbJhA==</latexit>

x2
<latexit sha1_base64="+QD6lLYrlFcju7OsE+tVFe8fWe4=">AAACPHicbVA9SwNBEN3z2/gZLbU4DIJFCHciahmw0FLRfEASw95mL1myt7vszonxSGlpqz/E3v9hbye21u7lUhjjwMDjvRnmzQsUZwY8792ZmZ2bX1hcWs6trK6tb2zmt6pGxprQCpFc6nqADeVM0Aow4LSuNMVRwGkt6J+leu2OasOkuIGBoq0IdwULGcFgqev79mF7s+CVvFG508Afg0J593H19nX77bKddwrNjiRxRAUQjo1p+J6CVoI1MMLpMNeMDVWY9HGXNiwUOKKmlYy8Dt19y3TcUGrbAtwR+3sjwZExgyiwkxGGnvmrpeR/WiOG8LSVMKFioIJkh8KYuyDd9HG3wzQlwAcWYKKZ9eqSHtaYgI1n4gqw/kP2RYo4CzTWgyRkUFTSsDQ3JrrFDiVSj1I0JWXdRFKrXioQzMkwZ2P1/4Y4DaqHJf+4dHRl8z1HWS2hHbSHDpCPTlAZXaBLVEEEddETekYvzpvz4Xw6X9nojDPe2UYT5Xz/ALfesek=</latexit>

w2
<latexit sha1_base64="bS4yJRC3gCcytIHoJK4H5zT95/g=">AAACT3icbZDPThsxEMa9KZQQoA1Nb3BYNULigKJdVAFHJA5wpBIBpCREXu9sYsVrW/YsbVjtkffgClfuvEOPfZLeKrybHPjTkSz99H1jz/iLtOAWg+CPV/uwsPhxqb7cWFld+/S5uf7l3KrMMOgyJZS5jKgFwSV0kaOAS22AppGAi2hyVPoX12AsV/IMpxoGKR1JnnBG0UnD5tc+wi+s3skNxEX+c7hbDJvtoBNU5b+HcA7tw83b1avH1tPpcN1r92PFshQkMkGt7YWBxkFODXImoGj0MwuasgkdQc+hpCnYQV6NLfwtp8R+oow7Ev1KfXkjp6m10zRynSnFsX3rleL/vF6GycEg51JnCJLNBiWZ8FH5ZRZ+zA0wFFMHlBnudvXZmBrK0CX2agryyc3sFyUJHhlqpnnCcUcry8souRztxMCUqYK1He22SZXR49JgVLCi4WIN34b4Hs53O+Fe5/sPl+8xmVWdbJBvZJuEZJ8ckhNySrqEkRtyR+7Jg/fb++v9q81ba94cWuRV1ZafAX95uDA=</latexit>
Example: Feedforward Neural Network
Example: Feedforward Neural Network
Example: Feedforward Neural Network
Example: Feedforward Neural Network
Example: Feedforward Neural Network
Example: Feedforward Neural Network
Example: Feedforward Neural Network
Example: Feedforward Neural Network
Summary
Larger Networks
Increase number of nodes

Increase number of hidden layers


Larger Networks
$12 million or ~300 V100 GPUs for 3 months

Microsoft Research Blog. Oct 6, 2021.


Larger Networks

The larger the better?

No -> overfitting (layman term – memorize training data only, poor


performance when testing data is used)

What the reason of overfitting? One of the reason is we have too many
parameters.

Why do I say so?

You might also like