Neural Network

Residual Unit:

$y_l = H(x_l) + F(x_l, W_l)$
$x_{l+1} = f(y_l)$

$x_l$ and $x_l+1$ is the input and output of $l$-th residual unit
$f$ is a RELU function
$h(x_l) = x_l$ is an identity mapping

Log loss:
$L(y, p(y, x)) = -log(p(y\ |\ x))$
$f(x) =\dfrac{1}{ 1+e^{-w^Tx}}$
$y \in \{0, 1\}$

$p(Y=1\ |\ x) = f(x)$
$p(Y=0\ |\ x) = 1 - f(x)$

似然函数
$l(w) =\prod_{i=0}^N p(y_i\ |\ x_i)^{y_i} [1 - p(y_i\ |\ x_i)]^{(1 - y_i)}$

取对数
$L(w) =\sum_{i=0}^N (y_ilog(p(y_i\ |\ x_i) )+ (1 - y_i)log(1 - p(y_i\ |\ x_i))$

最大化似然函数 <=> 最小化负似然函数
$L^{'} (w, x) = \sum_{i=0}^N (-y_ilog(p(y_i\ |\ x_i) ) - (1 - y_i)log(1 - p(y_i\ |\ x_i))$

加L1 正则
$J(w, x) = \sum_{i=0}^N (-y_ilog(p(y_i\ |\ x_i) ) - (1 - y_i)log(1 - p(y_i\ |\ x_i)) + \dfrac{1}{2}r ||w||_1$

加L2正则
$J (w, x) = \sum_{i=0}^N (-y_ilog(p(y_i\ |\ x_i) ) - (1 - y_i)log(1 - p(y_i\ |\ x_i)) + \dfrac{1}{2}r ||w||^2$

$$p(y \ |\ x, w) =$$
$$x = \dfrac{-b \pm \sqrt{b^2 - 4ac}}{2a}$$