chickadee » nanograd » make-step-decay-schedule

make-step-decay-schedule init-lr decay-rate decay-steps #!key (warmup-steps 0)procedure

Multiplies the learning rate by decay-rate every decay-steps steps, after an optional warmup:

lr = init-lr * decay-rate ^ floor(step / decay-steps)