3 AD songs

"Optimality"

I'm sittin' in front of the computer screen.
Newtons second iteration is what I've just seen.
It's not quite the progress that I would expect
from a code such as mine - no doubt it must be perfect!
Just the facts are not supportive and I wonder...

My linear solver is state of the art.
It does not get better wherever I start.
For differentiation is there anything else?
Perturbing the inputs - can't imagine this fails.
I pick a small epsilon and I wonder...

I wonder how but I still give it a try.
The next change in step size is bound to fly
'cause all I'd like to see is simply optimality.
Epsilon, in fact, appears to be rather small.
A factor of ten should improve it all
'cause all I'd like to see is nearly optimality.

A. D.A.D. A.D.A.D.A. D.A.D. A.D.A.D.A. D.A.D.A.D

A few hours later my talk's getting rude.
The sole thing descending seems to be my mood.
How can guessing the Hessian only take this much time?
N squared function runs appear to be the crime.
The facts support this thesis and I wonder...

Isolation due to KKT
Isolation - why not simply drop feasability?

The guy next door's been saying again and again:
An adjoint Lagrangian might relieve my pain.
Though I don't quite believe him, I surrender

I wonder how but I still give it a try:
Gradients and Hessians in the blink of an eye.
Still all I'd like to see is simply optimality.
Epsilon itself has finally disappeared.
Reverse mode AD works no matter how weird,
and I'm about to see local optimality

Yes, I wonder, I wonder...

I wonder how but I still give it a try:
Gradients and Hessians in the blink of an eye.
Still all I'd like to see...
I really need to see...
Now I can finally see my cherished optimality :-))))

"AD version 1"

AD, AD, automatic? Well just may be!
Categorically we state
Never ever do truncate
Cancellation won't arise

AD, AD, automatic? Well just may be!
When you preprocess the code
Or just simply overload
Slopes and adjoints will be fine

AD, who can claim that it's his baby
IBM and Microsoft are fools
There's no intellectual property
Just old differentiation rules

AD, AD, Automatic? Well just may be!
Note that gradients are cheap
Though there' many reals to keep
Try checkpointing otherwise

AD, AD, who knows where to go from here?

AD, AD, will it stay or disappear?

"AD version 2"

AD, yesterday derivatives meant pain
AD, hand coding drove everyone insane
Now the tedium is gone
Differentiation is fun
Monster codes here we come
Soon the magic is done
AD, it is true, I love you

AD, nowadays derivatives mean gain
AD, first or higher order just the same
With predictable cost
And no accuracy lost
Not for sine nor for cos
It's no longer a toss
AD tops the scene, rules supreme