@cottenio

New AI Paper: working on a pre-print for the Frustration mechanism. It's like mixing save-scumming from video games with the power of GPUs. Instead of transient noise during epochs (like Dropout), the engine waits until an improved epoch emerges, then immediately damages the network, then forces the next epoch to improve on the damaged state. So unlike pruning, where we try to reduce the number of neurons post-training, this is an active blind attack on the network between each successful iteration. Surprising results! It rescues certain degenerate topologies (especially where the vanishing gradient problem harms backpropagation), but even still looks useful in networks with skip connections (like ResMLP) for providing "kicks" out of local optima. Lots more to do. Paper & code: https://github.com/tcotten-scrypted/persistent-stochastic-ablation-mlp

1 reply

1 recast

1 reaction