HUIJZER.XYZ

The logit and logistic functions

2020-11-04

Linear regression works on real numbers R\mathbb{R}, that is, the input and output are in R\mathbb{R}. For probabilities, this is problematic because the linear regression will happily give a probability of 934-934, where we know that probabilities should always lie between 00 and 11. This is only by definition, but it is an useful definition in practice. Informally, the logistic function has been designed to convert values from real numbers to probabilities and the logit function is the inverse.

  1. Logistic
  2. Logit

Logistic

The logistic function converts values from (,)(-\infty, \infty) to (0,1)(0, 1):

logistic(x)=11+ex. \text{logistic}(x) = \frac{1}{1 + e^{-x}}.
logistic(x) = 1 / (1 + exp(-x))

We can visualise this with

using Gadfly

plot(y = [logistic], xmin = [-6], xmax = [6], 
	Geom.line, Stat.func, Guide.xlabel("x")
)

Some people advise to remember the following numbers by heart.

logistic(3)0.05,logistic(1)14,logistic(1)34,andlogistic(3)0.95. \begin{aligned} \text{logistic}(-3) &\approx 0.05, \\ \text{logistic}(-1) &\approx \tfrac{1}{4}, \\ \text{logistic}(1) &\approx \tfrac{3}{4}, \: \text{and} \\ \text{logistic}(3) &\approx 0.95. \end{aligned}

since

logistic(-3) = 0.04742587317756678
logistic(-1) = 0.2689414213699951
logistic(1) = 0.7310585786300049
logistic(3) = 0.9525741268224334

Logit

The inverse of the logistic function is the logit function,

logit(x)=log(x1x). \text{logit}(x) = \log(\frac{x}{1 - x}).
logit(x) = log(x / (1 - x))

This function goes from (0,1)(0, 1) to (,)(- \infty, \infty).

plot(y = [logit], xmin = [0], xmax = [1],
	Geom.line, Stat.func, Guide.xlabel("x")
)