Generalised Linear Models (GLM)
Generalised Linear Models (GLMs) are a family of models which include many common distributions such as Gaussian, Bernoulli and Multinomial.
The Exponential Family
The exponential family serves as a starting point for GLMs.
The exponential family is defined as $$ p(y; \eta) = b(y) \ \exp{\left( \eta^T\ T(y) - a(\eta) \right)} $$
- Natural (Canonical) parameter : $\eta$
- Sufficient statistic : $T(y)$
- Log Partition function : $a(\eta)$
$T(y)$ is often chosen to be $T(y) = y$
$e^{-a(\eta)}$ serves as a normalisation constant, and ensures the distribution sums to $1$
Family / Set Of Distributions
A family or set of distributions is defined by $(T, a, b)$ and parameterized by $\eta$
Bernoulli as Exponential Family
The Bernoulli distribution is in the exponential family. It is a distribution over $y \in {0,1}$, with mean $\phi$. Varying $\phi$ gives different Bernoulli distributions.
The Bernoulli distribution can be shown to be an exponential family distribution by manipulating the equation into the form $p = b \exp \left( \eta T - a \right)$ $$ \begin{aligned} p(y; \phi) &= \phi^y (1 - \phi)^{1-y} \cr &= \exp \left( y\log\phi + (1-y) \log(1-\phi) \right) \cr &= \underbrace{1 \vphantom{\frac{}{}} }_b \exp {\large(} \underbrace{\log \frac{\phi}{1-\phi} }_\eta \ \underbrace{y \vphantom{\frac{}{}} }_T + \underbrace{ \log(1-\phi) \vphantom{\frac{}{}} }_a {\large)} \end{aligned} $$
Gaussian as Exponential Family
Similarly for the Gaussian distribution $$ \begin{aligned} p(y; \mu) &= \frac{1}{\sqrt{2\pi}} \left(-\frac{1}{2} (y-\mu)^2 \right) \cr &= \underbrace{\frac{1}{\sqrt{2\pi}} \exp (-\frac{1}{2} y^2)}_b \cdot \exp ( \underbrace{\mu \vphantom{\frac{}{}} }_n \underbrace{y \vphantom{\frac{}{}} }_T - \underbrace{\frac{1}{2} \mu^2}_a ) \end{aligned} $$
Other Distributions in the Exponential Family
There are many other common distributions such as:
- Multinomial, Poisson
- Gamma, Exponential
- Beta, Dirichlet
Constructing GLMs
Consider a classification/regression problem where we want to predict $y$ as a function of $x$
If we make three assumptions
1 . Given $x$ and $\theta$, $y$ is some exponential family distribution with parameter $\eta$ $$ y\ |\ x;\theta \sim \text{ExpFamily}(\eta) $$ 2 . Given $x$, predict $\mathbb{E}[T(y)]$ $$ \text{Want } h(x) = \mathbb{E} [ T(y) \ | \ x] $$ 3 . Natural parameter $\eta$ and inputs $x$ are linearly related $$ \eta = \theta^T x $$
Many different/common types of distributions can be modelled as GLMs, and GLMs possess desirable properties such as ease of learning.
Canonical Response Function : Distribution’s mean as function of the natural parameter $$ g(\eta) = \mathbb{E}[T(y); \eta] $$ Canonical Link Function : Inverse of the response function $$ g^{-1} $$
Ordinary Least Squares
Ordinary least squares can be derived as a GLM with the following properties:
- Target variable $y$ is Continuous
- $y$ given $x$ is modelled as Gaussian $\mathcal{N}(\mu, \sigma^2)$
Formulate the hypothesis function from assumption 2 (Predict $\mathbb{E}[T(y)]$ given $x$) $$ h_\theta(x) = \mathbb{E}[ y\ |\ x;\theta] $$ Because it is Gaussian $$ h_\theta(x) = \mu $$ From the formulation of the Gaussian as an exponential family distribution, $\mu = \eta$ $$ \begin{aligned} h_\theta(x) &= \mu \cr &= \eta \end{aligned} $$ Finally from assumption 3 ($\eta = \theta^T x$) $$ \begin{aligned} h_\theta(x) &= \eta \cr &= \theta^T x \end{aligned} $$
The canonical response function of the Gaussian distribution is the identity function.
Logistic Regression
Logistic regression can also be formulated as a GLM.
$y \in {0, 1 }$
Choose Bernoulli family to model $y\ |\ x$
The hypothesis function is $$ \begin{aligned} h_\theta(x) &= \mathbb{E}[y\ |\ x;\theta] \cr &= \phi \vphantom{\frac{1}{1}} \cr &= \frac{1}{(1 + e^{-\eta})} \cr &= \frac{1}{(1+e^{-\theta^Tx})} \end{aligned} $$
The canonical response function of the Bernoulli distribution is the logistic function.
Softmax Regression
- Classification problem with multiple classes/categories
- Model as a multinomial distribution
Express multinomial as an exponential family distribution
Parameterise Multinomial
To parameterise over $k$ outcomes, we need to use $k-1$ parameters to prevent redundancy
(The sum of all outcomes must be $1$)
Define $T(y) \in \mathbb{R}^{k-1}$ as $$ \begin{aligned} T(1) &= { 1, 0, 0, \dots, 0 } \cr T(2) &= { 0, 1, 0, \dots, 0 } \cr T(k-1) &= { 0, 0, 0, \dots, 1 } \cr T(k) &= { 0, 0, 0, \dots, 0 } \end{aligned} $$ Define an indicator function: $$ \left\{ \begin{aligned} &1 &&\text{if True} \cr &0 &&\text{otherwise} \end{aligned} \right. $$
$$ (T(y))_i = 1 \{ y=i \} $$
Which gives $$ \begin{aligned} \mathbb{E} [ (T(y))_i ] &= P(y=i) \cr &= \quad \phi_i \end{aligned} $$ Therefore the probability $$ \begin{aligned} p(y; \phi) &= \phi_1^{1\{y=1\}} \phi_2^{1\{y=2\}} \dots \phi_{k-1}^{1\{y=k-1\}} \phi_k^{1-\sum_{i=1}^{k-1}1\{y=i\}} \cr &= \phi_1^{(T(y))_1} \phi_2^{(T(y))_2} \dots \phi_k^{1-\sum_{i=1}^{k-1}1(T(y))_i} \cr &= \exp\left( (T(y))_i \log(\phi_1) + \dots + ((1 - \Sigma_{i=1}^{k-1}(T(y))_i)) \log(\phi_k) \right) \end{aligned} $$
Fits the exponential family form $$ p(y;\theta) = \underbrace{1 \vphantom{\frac{1}{1}} }_b \exp {\LARGE(} \underbrace{ (T(y))_i \log(\frac{\phi_1}{\phi_k}) + \dots + (T(y))_{k-1} \log(\frac{\phi_{k-1}}{\phi_k}) }_\eta + \underbrace{ \log(\phi_k) \vphantom{\frac{1}{1}} }_a {\LARGE)} $$
With:
Link Function : $\eta_i = \log \frac{\phi_i}{\phi_k}$
Response Function : $\phi_i = \frac{e^{n_i}}{\sum_{j=1}^{k} e^{n_j}}$
This mapping from $\eta \rightarrow \phi$ is also called the softmax function
Softmax regression is a generalisation of logistic regression
It outputs estimated probabilities for every value of $i = 1, \dots, k$