What is sparse neural network?
We define a sparse neural network as a network in which only a percentage of the possible connections exists. You can imagine a fully connected layer with some of the connections missing.
What is dense network?
A dense network is a network in which the number of links of each node is close to the maximal number of nodes. Each node is linked to almost all other nodes. The total connected case in which exactly each node is linked to each other node is called a completely connected network.
What makes a neural network sparse or dense?
The usage of these terms in the context of neural networks is similar to their usage in other fields. In the context of NNs, things that may be described as sparse or dense include the activations of units within a particular layer, the weights, and the data.
Which is an example of a pre defined sparse network?
The overall density of these pre-defined sparse networks is set such that for different values of x, different networks have the same number of weights. For example, a FC network with x =14 has the same number of weights (around 11000) as a 50% dense network with x =28, a 22% dense network with x =56, and a 10% dense network with x =112.
What’s the difference between sparse and dense NNS?
In the context of NNs, things that may be described as sparse or dense include the activations of units within a particular layer, the weights, and the data. One could also talk about “sparse connectivity”, which refers to the situation where only a small subset of units are connected to each other.
Why is sparsity important in a neural network?
Beyond weight sparsity, the high sparsity levels found in sparse attention and ReLU activations are sufficient to tilt the scales in favor of sparsity on a wider range of platforms. As algorithms, software and hardware improve, sparsity will bite off a greater and greater fraction of the problem space.