Hard-Swish Activation

An efficient activation function designed to approximate Swish-like behavior at lower computational cost.

Hard-Swish is often used in mobile and efficient network architectures to reduce computational cost. It approximates the smooth behavior of Swish through a simpler piecewise formulation. This makes it possible to strike a better balance between performance and hardware efficiency. It is a good example of how activation functions can be chosen with strong engineering constraints in mind.