What is an Activation Function?

What is an Activation Function?

Approach: Explain the introduction of non-linearity to the network.

A function applied to the output of a neural network layer. Without non-linear activation functions, a neural network with any number of layers would behave like a single-layer linear regression model.

Follow-up: What are the benefits of ReLU?

Key concepts: sigmoid, relu, non-linearity.