tag:blogger.com,1999:blog-11295132.post112655666912795280..comments2015-12-23T12:32:50.863-08:00Comments on A Neighborhood of Infinity: AdjointsDan Piponihttps://plus.google.com/107913314994758123748noreply@blogger.comBlogger2125tag:blogger.com,1999:blog-11295132.post-1128901168715792522005-10-09T16:39:00.000-07:002005-10-09T16:39:00.000-07:00This is just reverse mode AD. But I've never seen ...This is just reverse mode AD. But I've never seen anyone else factor reverse mode AD as the product of two program transformations: forward AD followed by the adjoint. The adjoint step on its own is useful for applications besides differentiating.<BR/><BR/>My description of loops is correct, as you say, for fixed loops. For iterate-until-fixed-point, the implementations I have seen simply 'record' the forward pass and 'replay' it backwards with the same number of iterations on the reverse pass (eg. by building a tree that represents the unrolled loop during the forward run). So I don't see that my description is off. However, it seems to me that you're talking about something more interesting than this simple minded approach. Maybe you have a reference. If you don't post back here I might have to email you...sigfpehttp://www.blogger.com/profile/08096190433222340957noreply@blogger.comtag:blogger.com,1999:blog-11295132.post-1128814478587743392005-10-08T16:34:00.000-07:002005-10-08T16:34:00.000-07:00This really is just reverse-mode AD. With f:R^n->R...This really is just reverse-mode AD. With f:R^n->R^m, and J the Jacobian of f at x, forward-mode AD maps (x,x')->(f(x),J*x'), which is just the push-forward. Reverse-mode AD maps y'->J^T*y'. But note that if f is linear then f(x')=J*x', or we might just say f=J. So in this case, forward-mode AD gives the reasonably useless (x,x')->(f(x),f(x')) and reverse-mode AD gives simply the adjoint of f, which is in this case just a fancy way of saying the transpose of J.<BR/><BR/>Also ... your description of AD is a little off WRT loops. As described reverse-mode wouldn't work for iterate-to-fixedpoint loops, which are not uncommon (ahem) in numeric code. What you give works only for loops that could be in principle unrolled away, eg loops for a fixed number of iterations, or loops over the indices of an array.<BR/><BR/>There is a special transformation rule for iterate-to-fixedpoint loops, but it does not ammount to just reversing them, it is slightly more complicated. Feynman found and published the rule as his senior thesis (!) but the notation he uses is very specific so you wouldn't recognize this this is what he was doing unless you already know it.<BR/>--<BR/>Barak Pearlmutter, barak@cs.nuim.ieBarak Pearlmutterhttp://www.blogger.com/profile/01327547613085010573noreply@blogger.com