Midterm

David Wallace Croft

Neural Net Mathematics
Richard M. Golden, Ph.D.
U.T. Dallas

2005-02-28

Problem 1

Objective Function
$c(w) = μ_{1} * {(o_{1} - r_{1})}^{2} + μ_{2} * {(o_{2} - r_{2})}^{2}$
Response
$r_{k} = w^{T} * s_{k}$
Weight Update Rule
$w (t + 1) = w (t) + Δ w$
Gradient Descent
$Δ w = - η * \nabla c(w)$
Error
$e_{k} = o_{k} - r_{k}$
Error Squared
$f_{k} = e_{k}^{2}$
Vectors
$μ = [\begin{matrix} μ_{1} \\ μ_{2} \end{matrix}], e = [\begin{matrix} e_{1} \\ e_{2} \end{matrix}], f = [\begin{matrix} f_{1} \\ f_{2} \end{matrix}], o = [\begin{matrix} o_{1} \\ o_{2} \end{matrix}], r = [\begin{matrix} r_{1} \\ r_{2} \end{matrix}]$
Objective Function Revised
$c(w) = μ^{T} * f = μ^{T} * (e ⊙ e)$
Chain Rule
$\nabla c(w) = {c'(w)}^{T} = {[c'(f) * f'(e) * e'(r) * r'(w)]}^{T}$
First Term
$c'(f) = (μ^{T} * f)' = μ^{T}$
Second Term
$f'(e) = (e ⊙ e)' = 2 * diag (e)$
Third Term
$e'(r) = (o - r)' = - I$
Fourth Term
$r'(w) = ([\begin{matrix} s_{1}^{T} \\ s_{2}^{T} \end{matrix}] * w)' = [\begin{matrix} s_{1}^{T} \\ s_{2}^{T} \end{matrix}]$
Combined Terms
$\nabla c(w) = {[[μ^{T}] * [2 * diag (e)] * [- I] * [\begin{matrix} s_{1}^{T} \\ s_{2}^{T} \end{matrix}]]}^{T}$
Reduce
$\nabla c(w) = {[-2 * [\begin{matrix} μ_{1} * e_{1} & μ_{2} * e_{2} \end{matrix}] * [\begin{matrix} s_{1}^{T} \\ s_{2}^{T} \end{matrix}]]}^{T}$
Transpose
$\nabla c(w) = -2 * [\begin{matrix} s_{1} & s_{2} \end{matrix}] * [\begin{matrix} μ_{1} * e_{1} \\ μ_{2} * e_{2} \end{matrix}]$
Reduce More
$\nabla c(w) = -2 * [\begin{matrix} μ_{1} * (o_{1} - r_{1}) * s_{1} + μ_{2} * (o_{2} - r_{2}) * s_{2} \end{matrix}]$
Weight Update Rule Final
$w (t + 1) = w (t) + 2 * η * [\begin{matrix} μ_{1} * (o_{1} - r_{1}) * s_{1} + μ_{2} * (o_{2} - r_{2}) * s_{2} \end{matrix}]$

Problem 2

Taylor Series
$f (t + Δ t) = Σ_{k = 0}^{\infty} \frac{{Δ t}^{k}}{k!} f^{(k)} (t)$
Taylor Series for a Function of a Vector
$c (w + Δ w) = c (w) + {\nabla c (w)}^{T} * Δ w + \frac{1}{2} * Δ w^{T} * \nabla^{2} c (w) * Δ w + ...$
Function Delta
$Δ c (w) = c (w + Δ w) - c (w) ≃ {\nabla c (w)}^{T} * Δ w + \frac{1}{2} * Δ w^{T} * \nabla^{2} c (w) * Δ w$
Minimize the Function Delta
$Δ c' (Δ w) ≃ {\nabla c (w)}^{T} + Δ w^{T} * \nabla^{2} c (w) = 0$
Solve for the Weight Delta, Step 1
$Δ w^{T} * \nabla^{2} c (w) = - {\nabla c (w)}^{T}$
Solve for the Weight Delta, Step 2
$\nabla^{2} c (w) * Δ w = - \nabla c (w)$
Solve for the Weight Delta, Step 3
$Δ w = - {[\nabla^{2} c (w)]}^{-1} * \nabla c (w)$
Newton-Raphson Descent
$w (t + 1) = w (t) - η * {[\nabla^{2} c (w)]}^{-1} * \nabla c (w)$
Hessian (Derivation)
$\nabla^{2} c (w) = \frac{\partial^{2} c (w)}{\partial w \partial w} = \frac{\partial c' (w)}{\partial w} = \frac{\partial {[\nabla c (w)]}^{T}}{\partial w}$
From Problem 1
$\nabla c(w) = {[-2 * [\begin{matrix} μ_{1} * e_{1} & μ_{2} * e_{2} \end{matrix}] * [\begin{matrix} s_{1}^{T} \\ s_{2}^{T} \end{matrix}]]}^{T}$
Define y
$y = [\begin{matrix} μ_{1} * e_{1} \\ μ_{2} * e_{2} \end{matrix}]$
Define F
$F = [\begin{matrix} s_{1}^{T} \\ s_{2}^{T} \end{matrix}]$
Derivative of c in terms of y and F
$c' (w) = -2 * y^{T} * F$
Identity from Marlow p216
$(y^{T} * F)' = (y^{T} \otimes I_{s}) * F' (w) + F^{T} * y' (w)$
Hessian in terms of y and F
$\nabla^{2} c (w) = -2 * F^{T} * y' (e) * e' (r) * r' (w)$
Hessian (more)
$\nabla^{2} c (w) = -2 * F^{T} * diag (μ) * (- I) * F$
Hessian (more 2)
$\nabla^{2} c (w) = 2 * F^{T} * diag (μ) * F$
Hessian (more 3)
$\nabla^{2} c (w) = 2 * [\begin{matrix} s_{1} & s_{2} \end{matrix}] * diag (μ) * {[\begin{matrix} s_{1} & s_{2} \end{matrix}]}^{T}$
Newton-Raphson Descent requires the inverse of the Hessian. The inverse of the Hessian exists if the Hessian is positive definite [Haykin p151]. The Hessian is positive definite if the input signal vectors (s) span the d-dimensional real vector space [Golden p367].

Problem 3

Input
$h_{t} = {[s_{t}^{T}, o_{t - 1}, 1]}^{T}$
Response at time t
$r_{t} = ⅇ^{- {‖ w - h_{t} ‖}^{2}}$
Objective Function
$l_{n} (w) = (\frac{1}{n}) Σ_{t = 1}^{n} [μ_{t} * {(o_{t} - r_{t})}^{2} + λ * {(r_{t} - r_{t - 1})}^{2}]$
Semantic Interpretation
The first part of the objective function minimizes the mean square error between observable desired responses and actual responses. The second part of the objective function minimizes the differences between the current response and the immediately preceding response. This acts like a temporal smoothing function, eliminating high frequency components.
Observable Error
$m_{t} = μ_{t} * (o_{t} - r_{t})$
Observable Errors Over Observation Time
$m = {[\begin{matrix} m_{1} & m_{2} & ... & m_{n} \end{matrix}]}^{T} = μ ⊙ (o - r)$
Zero or One Squared
$μ_{t}^{2} = μ_{t}$
Square Error
$a = m^{T} * m$
Change in Response
$c_{t} = r_{t} - r_{t - 1}$
Changes in Response Over Observation Time
$c = {[\begin{matrix} c_{1} & c_{2} & ... & c_{n} \end{matrix}]}^{T}$
Sum of Square of Changes
$b = c^{T} * c$
Objective Function revised
$l_{n} (w) = (\frac{1}{n}) * [a + λ * b]$
Derivative
$l_{n}' (w) = (\frac{1}{n}) * [a' (w) + λ * b' (w)]$
Difference
$f_{t} = w - h_{t}$
Negative Distance
$k_{t} = - f_{t}^{T} * f_{t}$
Derivative of the First Part
$a' (w) = a' (m) * m' (r) * r' (k) * k' (f) * f' (w)$
First Term
$a' (m) = 2 * m^{T} = 2 * {[μ ⊙ (o - r)]}^{T}$
Diagonal Matrices
$m' (r) * r' (k) = - diag (μ) * diag (r) = - diag (μ ⊙ r)$
First Three Terms
$a' (m) * m' (r) * r' (k) = -2 * m^{T} * diag (μ ⊙ r) = -2 * {[μ ⊙ (o - r) ⊙ μ ⊙ r]}^{T} = -2 * {[μ ⊙ (o - r) ⊙ r]}^{T}$
Vector f
$f = vec (F) = vec ([\begin{matrix} f_{1} & f_{2} & ... & f_{n} \end{matrix}]) = {[\begin{matrix} f_{1}^{T} & f_{2}^{T} & ... & f_{n}^{T} \end{matrix}]}^{T}$
Fourth Term
$k' (f) = -2 * {[\begin{matrix} f_{1}^{T} & 0^{T} & 0^{T} & ... & 0^{T} \\ 0^{T} & f_{2}^{T} & 0^{T} & ... & 0^{T} \\ ... & ... & ... & ... & ... \\ 0^{T} & 0^{T} & 0^{T} & ... & f_{n}^{T} \end{matrix}]}_{n x (n * (d+2))}$
Fifth Term
$f' (w) = {[\begin{matrix} I_{d + 2} \\ I_{d + 2} \\ ... \\ I_{d + 2} \end{matrix}]}_{(n * (d+2)) x (d+2)} = 1_{n} \otimes I_{d + 2}$
Fourth and Fifth Terms Combined
$k' (f) * f' (w) = -2 * {[\begin{matrix} f_{1}^{T} \\ f_{2}^{T} \\ ... \\ f_{n}^{T} \end{matrix}]}_{n x (d+2)}$
First Part
$a' (w) = a' (m) * m' (r) * r' (k) * k' (f) * f' (w) = 4 * {[μ ⊙ (o - r) ⊙ r]}^{T} * {[\begin{matrix} f_{1}^{T} \\ f_{2}^{T} \\ ... \\ f_{n}^{T} \end{matrix}]}_{n x (d+2)}$
Derivative of the Second Part
$b' (w) = b' (c) * c' (r) * r' (k) * k' (f) * f' (w)$
Second Part, First Term
$b' (c) = 2 * c^{T}$
Temporal Smoothing Term
$c' (r) = [\begin{matrix} 1 & 0 & 0 & 0 & ... & 0 & 0 & 0 \\ -1 & 1 & 0 & 0 & ... & 0 & 0 & 0 \\ 0 & -1 & 1 & 0 & ... & 0 & 0 & 0 \\ ... & ... & ... & ... & ... & ... & ... & ... \\ 0 & 0 & 0 & 0 & ... & 0 & -1 & 1 \end{matrix}]$
Two Terms
$c' (r) * r' (k) = [\begin{matrix} 1 & 0 & 0 & 0 & ... & 0 & 0 & 0 \\ -1 & 1 & 0 & 0 & ... & 0 & 0 & 0 \\ 0 & -1 & 1 & 0 & ... & 0 & 0 & 0 \\ ... & ... & ... & ... & ... & ... & ... & ... \\ 0 & 0 & 0 & 0 & ... & 0 & -1 & 1 \end{matrix}] * diag (r) = [\begin{matrix} r_{1} & 0 & 0 & 0 & ... & 0 & 0 & 0 \\ {- r}_{1} & r_{2} & 0 & 0 & ... & 0 & 0 & 0 \\ 0 & {- r}_{2} & r_{3} & 0 & ... & 0 & 0 & 0 \\ ... & ... & ... & ... & ... & ... & ... & ... \\ 0 & 0 & 0 & 0 & ... & 0 & {- r}_{n - 1} & r_{n} \end{matrix}]$
Second Part, All Terms
$b' (c) * c' (r) * r' (k) * k' (f) * f' (w) = 2 * c^{T} * [\begin{matrix} r_{1} & 0 & 0 & 0 & ... & 0 & 0 & 0 \\ {- r}_{1} & r_{2} & 0 & 0 & ... & 0 & 0 & 0 \\ 0 & {- r}_{2} & r_{3} & 0 & ... & 0 & 0 & 0 \\ ... & ... & ... & ... & ... & ... & ... & ... \\ 0 & 0 & 0 & 0 & ... & 0 & {- r}_{n - 1} & r_{n} \end{matrix}] * -2 * {[\begin{matrix} f_{1}^{T} \\ f_{2}^{T} \\ ... \\ f_{n}^{T} \end{matrix}]}_{n x (d+2)}$
Second Part
$b' (c) * c' (r) * r' (k) * k' (f) * f' (w) = -4 * c^{T} * [\begin{matrix} r_{1} & 0 & 0 & 0 & ... & 0 & 0 & 0 \\ {- r}_{1} & r_{2} & 0 & 0 & ... & 0 & 0 & 0 \\ 0 & {- r}_{2} & r_{3} & 0 & ... & 0 & 0 & 0 \\ ... & ... & ... & ... & ... & ... & ... & ... \\ 0 & 0 & 0 & 0 & ... & 0 & {- r}_{n - 1} & r_{n} \end{matrix}] * {[\begin{matrix} f_{1}^{T} \\ f_{2}^{T} \\ ... \\ f_{n}^{T} \end{matrix}]}_{n x (d+2)}$
Both Parts
$l_{n}' (w) = \frac{4}{n} * {[μ ⊙ (o - r) ⊙ r]}^{T} * {[\begin{matrix} f_{1}^{T} \\ f_{2}^{T} \\ ... \\ f_{n}^{T} \end{matrix}]}_{n x (d+2)} - \frac{4 * λ}{n} * c^{T} * [\begin{matrix} r_{1} & 0 & 0 & 0 & ... & 0 & 0 & 0 \\ {- r}_{1} & r_{2} & 0 & 0 & ... & 0 & 0 & 0 \\ 0 & {- r}_{2} & r_{3} & 0 & ... & 0 & 0 & 0 \\ ... & ... & ... & ... & ... & ... & ... & ... \\ 0 & 0 & 0 & 0 & ... & 0 & {- r}_{n - 1} & r_{n} \end{matrix}] * {[\begin{matrix} f_{1}^{T} \\ f_{2}^{T} \\ ... \\ f_{n}^{T} \end{matrix}]}_{n x (d+2)}$
Both Parts Again
$l_{n}' (w) = \frac{4}{n} * [{[μ ⊙ (o - r) ⊙ r]}^{T} - λ * c^{T} * [\begin{matrix} r_{1} & 0 & 0 & 0 & ... & 0 & 0 & 0 \\ {- r}_{1} & r_{2} & 0 & 0 & ... & 0 & 0 & 0 \\ 0 & {- r}_{2} & r_{3} & 0 & ... & 0 & 0 & 0 \\ ... & ... & ... & ... & ... & ... & ... & ... \\ 0 & 0 & 0 & 0 & ... & 0 & {- r}_{n - 1} & r_{n} \end{matrix}]] * {[\begin{matrix} f_{1}^{T} \\ f_{2}^{T} \\ ... \\ f_{n}^{T} \end{matrix}]}_{n x (d+2)}$
Gradient is the Transpose
$\nabla l_{n} (w) = \frac{4}{n} * {[\begin{matrix} w - h_{1} & w - h_{2} & ... & w - h_{n} \end{matrix}]}_{(d+2) x n} * [[μ ⊙ (o - r) ⊙ r] - λ * [\begin{matrix} r_{1} * (- r_{0} + 2 * r_{1} + - r_{2}) \\ r_{2} * (- r_{1} + 2 * r_{2} + - r_{3}) \\ ... \\ r_{n - 1} * (- r_{n - 2} + 2 * r_{n - 1} + - r_{n}) \\ r_{n} * (r_{n} - r_{n - 1}) \end{matrix}]]$
Weight Update Rule
$w (t + 1) = w (t) - η * \nabla l_{n} (w)$

Problem 4

Yes. "If f is twice-differentiable at every [vector x which is an element of] D then f is twice-differentiable on D and is a twice-differentiable function" [Marlow p198]. For any input in the domain, the second derivative of the objective function exists.

Problem 5

Newton-Raphson Descent
$w (t + 1) = w (t) - η * {[\nabla^{2} l_{n} (w)]}^{-1} * \nabla l_{n} (w)$
Hessian
$\nabla^{2} l_{n} (w) = \frac{\partial \nabla l_{n} (w)}{\partial w}$
Identity from Marlow p216
$(F_{(d + 2) x n} * z)' (w) = F * z' (w) + (I_{d + 2} \otimes z^{T}) * F' (w)$
Define z
$z = μ ⊙ (o - r) ⊙ r$
Derivative of z with Respect to r
$z' (r) = diag (μ ⊙ (o - 2 * r))$
Derivative of z with Respect to w
$z' (w) = z' (r) * r' (k) * k' (f) * f' (w) = diag (μ ⊙ (o - 2 * r)) * diag (r) * -2 * F^{T} = -2 * diag (μ ⊙ (o - 2 * r) ⊙ r) * F^{T}$
Identity from Marlow p211
$F' (w) = Σ_{t = 1}^{n} (f_{t}' (w) \otimes e_{t})$
Derivative of f at time t
$f_{t}' (w) = I_{d + 2}$
Derivative of F
$F' (w) = {[I_{d + 2} \otimes 1_{n}]}_{[(d + 2) * n] x (d + 2)}$
Combining Terms
$(I_{d + 2} \otimes z^{T}) * F' (w) = (I_{d + 2} \otimes z^{T}) * {[I_{d + 2} \otimes 1_{n}]}_{[(d + 2) * n] x (d + 2)} = (Σ_{t = 1}^{n} z_{t}) * I_{d + 2}$
First Part
$(F_{(d + 2) x n} * z)' (w) = -2 * F * diag (μ ⊙ (o - 2 * r) ⊙ r) * F^{T} + (Σ_{t = 1}^{n} [μ_{t} * (o_{t} - r_{t}) * r_{t}]) * I_{d + 2}$
Define p
$p = [\begin{matrix} r_{1} * (- r_{0} + 2 * r_{1} + - r_{2}) \\ r_{2} * (- r_{1} + 2 * r_{2} + - r_{3}) \\ ... \\ r_{n - 1} * (- r_{n - 2} + 2 * r_{n - 1} + - r_{n}) \\ r_{n} * (r_{n} - r_{n - 1}) \end{matrix}]$
Derivative of p with respect to r
$p' (r) = [\begin{matrix} - r_{0} + 4 * r_{1} - r_{2} & - r_{1} & 0 & 0 & ... & 0 & 0 & 0 & 0 \\ - r_{2} & - r_{1} + 4 * r_{2} - r_{3} & - r_{2} & 0 & ... & 0 & 0 & 0 & 0 \\ ... & ... & ... & ... & ... & ... & ... & ... & ... \\ 0 & 0 & 0 & 0 & ... & 0 & - r_{n - 1} & - r_{n - 2} + 4 * r_{n - 1} - r_{n} & - r_{n - 1} \\ 0 & 0 & 0 & 0 & ... & 0 & 0 & - r_{n} & - r_{n - 1} + 2 * r_{n} \end{matrix}]$
Derivative of F times p
$(F_{(d + 2) x n} * p)' (w) = F * p' (w) + (I_{d + 2} \otimes p^{T}) * F' (w)$
Second Part
$(F_{(d + 2) x n} * p)' (w) = -2 * F * p' (r) * diag (r) * F^{T} + (Σ_{t = 1}^{n} p_{t}) * I_{d + 2}$
Hessian from Both Parts
$\nabla^{2} l_{n} (w) = \frac{4}{n} * ([-2 * F * diag (μ ⊙ (o - 2 * r) ⊙ r) * F^{T} + (Σ_{t = 1}^{n} [μ_{t} * (o_{t} - r_{t}) * r_{t}]) * I_{d + 2}] - λ * [-2 * F * p' (r) * diag (r) * F^{T} + (Σ_{t = 1}^{n} p_{t}) * I_{d + 2}])$

Problem 6

Gradient Descent Weight Update Rule
$w (t + 1) = w (t) - η * \nabla Q(w)$

References

Golden, Richard M., Mathematical Methods for Neural Network Analysis and Design, MIT Press, 1996.
Haykin, Simon, Neural Networks: A Comprehensive Foundation, 2nd Ed., Prentice Hall, 1999.
Marlow, W. H., Mathematics for Operations Research, Dover Publications, 1978.