(Weight) Sparsity in Deep Learning

 SOTA* Overview

*[Submitted on 31 Jan 2021]

Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networks

https://arxiv.org/abs/2102.00554

The growing energy and performance costs of deep learning have driven the community to reduce the size of neural networks by selectively pruning components. Similarly to their biological counterparts, sparse networks generalize just as well, if not better than, the original dense networks. Sparsity can reduce the memory footprint of regular networks to fit mobile devices, as well as shorten training time for ever growing networks. In this paper, we survey prior work on sparsity in deep learning and provide an extensive tutorial of sparsification for both inference and training. We describe approaches to remove and add elements of neural networks, different training strategies to achieve model sparsity, and mechanisms to exploit sparsity in practice. Our work distills ideas from more than 300 research papers and provides guidance to practitioners who wish to utilize sparsity today, as well as to researchers whose goal is to push the frontier forward. We include the necessary background on mathematical methods in sparsification, describe phenomena such as early structure adaptation, the intricate relations between sparsity and the training process, and show techniques for achieving acceleration on real hardware. We also define a metric of pruned parameter efficiency that could serve as a baseline for comparison of different sparse networks. We close by speculating on how sparsity can improve future workloads and outline major open problems in the field.

download pdf:

https://arxiv.org/pdf/2102.00554

a tutorial offered by the paper's authors from ETH Zürich 

Sparsity in Deep Learning Tutorial

the promised video link that was not added, a talk by Torsten Hoefler

https://www.youtube.com/watch?v=H7-p3OWPpEI

!!Hint!!

The "paper" is more like a pamphlet, with 90 pages in total. It might be better to start with the authors' tutorial or talk, which is only about 2hr long.

Use the paper for reference to details and reviews after the tutorial/talk session.

Early Paper: SSL

https://proceedings.neurips.cc/paper/2016/file/41bfd20a38bb1b0bec75acf0845530a7-Paper.pdf

Learning Structured Sparsity in Deep Neural Networks 

High demand for computation resources severely hinders deployment of large-scale Deep Neural Networks (DNN) in resource constrained devices. In this work, we propose a Structured Sparsity Learning (SSL) method to regularize the structures (i.e., filters, channels, filter shapes, and layer depth) of DNNs. SSL can: (1) learn a compact structure from a bigger DNN to reduce computation cost; (2) obtain a hardware-friendly structured sparsity of DNN to efficiently accelerate the DNN’s evaluation. Experimental results show that SSL achieves on average 5.1× a