pathterminuspages/machine learning/aboutcontactabout me

Sigmoid

19.11.2020 | Standard Functions/Activation

Contents/Index

@Sigmoid
ReLU

Sigmoid functions are a class of functions with an S-shaped curve. A common example is the logistic function given as $$ \sigma(x) = \frac{1}{1 + e^{-x}} $$ This function is plotted in Figure 1. Note that we have $\sigma(0) = 0.5$, that is $$ \sigma(0) = \frac{1}{1 + e^{-0}} = \frac{1}{2} $$ and that we have $$ x \in [-\infty,0) \Rightarrow 0 \lt \sigma(x) \lt 0.5 $$ and $$ x \in (0,\infty] \rightarrow 0.5 \lt \sigma(x) \lt 1.0 $$

Figure 1: Plot of the sigmoid logistic function

The following Python3 code has been used for the plot:

import matplotlib.pyplot as plt import math def f(x): return 1 / (1.0 + math.exp(-x)) xs = [float(x) / 100.0 for x in range(-1000,1000)] ys = [f(y) for y in xs] plt.plot(xs,ys) plt.show()

The Derivative of the Sigmoid

The sigmoid function has a quite straightforward derivative: $$ \sigma(x)' = \sigma(x)(1 - \sigma(x)) $$ We can prove this as follows. We have that $$ \frac{d}{dx} \frac{1}{x} = - \frac{1}{x^2} $$ And we have that $$ \frac{d}{dx} e^{kx} = ke^{kx} $$ We use the chain rule to obtain $$ \sigma(x)' = - \frac{1}{(1 + e^{-x})^2} (-e^{-x}) = \frac{1}{1 + e^{-x}} \cdot \frac{e^{-x}}{1 + e^{-x}} $$ We have that $$ 1 - \frac{1}{1 + e^{-x}} = \frac{1 + e^{-x}}{1 + e^{-x}} - \frac{1}{1 + e^{-x}} = \frac{e^{-x}}{1 + e^{-x}} $$ We substitute twice to obtain $$ \sigma(x)' = \sigma(x)(1 - \sigma(x)) $$

Negative input for Sigmoid

Another straight forward thing about the sigmoid is that we have that $$ \sigma(-x) = 1 - \sigma(x) $$ Again we can prove it. We have as seen in the above that $$ 1 - \frac{1}{1 + e^{-x}} = \frac{e^{-x}}{1 + e^{-x}} $$ We have that $$ \frac{1 + e^{-x}}{e^{-x}} = \frac{1}{e^{-x}} + 1 = e^x + 1 $$ Hence we have that $$ 1 - \frac{1}{1 + e^{-x}} = \frac{1}{1 + e^x} $$

CommentsGuest Name:Comment: