Sigmoid

19.11.2020 | Standard Functions/Activation

Contents/Index

@Sigmoid
ReLU

Sigmoid functions are a class of functions with an S-shaped curve. A common example is the logistic function given as $$\sigma(x) = \frac{1}{1 + e^{-x}}$$ This function is plotted in Figure 1. Note that we have $\sigma(0) = 0.5$, that is $$\sigma(0) = \frac{1}{1 + e^{-0}} = \frac{1}{2}$$ and that we have $$x \in [-\infty,0) \Rightarrow 0 \lt \sigma(x) \lt 0.5$$ and $$x \in (0,\infty] \rightarrow 0.5 \lt \sigma(x) \lt 1.0$$

The following Python3 code has been used for the plot:

import matplotlib.pyplot as plt import math def f(x): return 1 / (1.0 + math.exp(-x)) xs = [float(x) / 100.0 for x in range(-1000,1000)] ys = [f(y) for y in xs] plt.plot(xs,ys) plt.show()

The Derivative of the Sigmoid

The sigmoid function has a quite straightforward derivative: $$\sigma(x)' = \sigma(x)(1 - \sigma(x))$$ We can prove this as follows. We have that $$\frac{d}{dx} \frac{1}{x} = - \frac{1}{x^2}$$ And we have that $$\frac{d}{dx} e^{kx} = ke^{kx}$$ We use the chain rule to obtain $$\sigma(x)' = - \frac{1}{(1 + e^{-x})^2} (-e^{-x}) = \frac{1}{1 + e^{-x}} \cdot \frac{e^{-x}}{1 + e^{-x}}$$ We have that $$1 - \frac{1}{1 + e^{-x}} = \frac{1 + e^{-x}}{1 + e^{-x}} - \frac{1}{1 + e^{-x}} = \frac{e^{-x}}{1 + e^{-x}}$$ We substitute twice to obtain $$\sigma(x)' = \sigma(x)(1 - \sigma(x))$$

Negative input for Sigmoid

Another straight forward thing about the sigmoid is that we have that $$\sigma(-x) = 1 - \sigma(x)$$ Again we can prove it. We have as seen in the above that $$1 - \frac{1}{1 + e^{-x}} = \frac{e^{-x}}{1 + e^{-x}}$$ We have that $$\frac{1 + e^{-x}}{e^{-x}} = \frac{1}{e^{-x}} + 1 = e^x + 1$$ Hence we have that $$1 - \frac{1}{1 + e^{-x}} = \frac{1}{1 + e^x}$$