Scientific Computing Seminar

Date and Place: Thursdays in Room 32-349. For detailed dates see below!

Content

In the Scientific Computing Seminar we host talks of guests and members of the SciComp team as well as students of mathematics, computer science and engineering. Everbody interested in the topics is welcome.

List of Talks

Event Information:

  • Thu
    13
    Dec
    2018

    SC Seminar: Avraam Chatzimichailidis

    9:00SC Seminar Room 32-349

    Avraam Chatzimichailidis, Fraunhofer ITWM

    Title:
    Second Order Methods applied to Deep Neural Networks

    Abstract:

    Optimizing deep neural networks involves finding a good enough minimum of a highly nonlinear and
    nonconvex function. State of the art first order methods suffer from pathological curvature of the loss
    landscape and successful convergence relies on the right metaparameter tweaking. Extending the optimizer
    to second order eliminates these problems, at the cost of having to compute the inverse Hessian of the
    deep neural network, which takes O(N^3).

    The R-operator allows efficient Hessian-vector-product computation of DNNs in O(N), without having to
    store the whole Hessian. Combining this operator together with the Lanczos algorithm, an iterative eigenvalue
    solver, allows for an efficient computation of eigenvalues in DNNs.

    A framework is built that is able to visualize the loss landscape of DNNs together with the
    trajectory taken by the optimizer. This is done by performing a PCA over the network parameters
    at different points of the trajectory and choosing the two directions in parameter space with the most
    variance.