[ML/AI] Learning note on Neural Net work (Coursera)

 For me the most important lesson should be the last one on week 5, and one image can show what we are doning by trainning a Nearal Network : Looking for parameters which minimize the cost function.

Image from Coursera

Steps are :

1. initialize the weights (thetas) randomly 
2. Forwardpropagation: compute from X to Y
3. Cost function
4. Backprobagation : compute deltas ( partial derivatives) dCost_function/dTheta
5. Gradient checking to confirm dCost_function/dTheta is correct, then disable this check
6. Gradient Decent or built-in optimization function to minimize the Cost Function with thetas

It's very similar to a Curve Calibration  which the theta is ZC and target is not minimize but find the zero NPV with Newton raphson.

So what's the difference : 

Newton raphson VS Gradient Decent

1. objectivf  :
1. N.R. is for reaching the ZERO POINT
2. G.D. is for reaching the MIN POINT (bottom)
2. User's input 
1. N.R. just need partial derivitives (decent to zero point, then next iteration)
2. G.D. need a step lenght (decent a step lenght, then next iteration)
3. Termination condition
1. Both them should define a MAX iteration number as termination condition
2. N.R. could have a tolorence for how closing to ZERO
3. G.D. could have a tolorence for how much minimized comparing to last iteration.