suggest-learning-rate

suggest-learning-rate parameters current-lr gradient-normprocedure

Suggests a new learning rate from the gradient norm: lower for large norms, higher for very small ones. Prints the suggestion.

(import (except nanograd-diagnostics compute-gradient-norm))

(define monitor (make-gradient-monitor))

;; In the training loop, after backward!
(record-step! monitor step (parameters model))

;; After training
(diagnose-training monitor)

chickadee » nanograd » suggest-learning-rate

Identifier search