Influence Functions & Preconditioners¶

Influence functions can be roughly conceptualized as a preconditioned gradient similarity search, with formula \(g_q H^{-1} g_t^\top\) where \(g_q\) is the query gradient, \(g_t\) is a training gradient, and \(H\) is a Hessian (approximation). Many Hessian approximations are used, and hyperparameters such as the inversion damping factor are also usually incorporated. In some cases, heuristics such as gradient unit normalization are also applied.

Preconditioners¶

Preconditioner	CLI
Gradient autocorrelation (second moment)	`bergson hessian <run_path> --method autocorrelation`
KFAC	`bergson hessian <run_path> --method kfac`
TKFAC	`bergson hessian <run_path> --method tkfac`
EK-FAC	`bergson hessian <run_path> --method kfac --ev_correction` (end-to-end pipeline: `bergson ekfac`)
Shampoo	`bergson hessian <run_path> --method shampoo`
Optimizer state (Adam / Adafactor)	`--optimizer_state <path>` on `build` / `reduce` / `score`