第5講回帰分析

変数選択

赤池情報量規準 (AIC)

AICの値が小さなモデルの方がよい．

\begin{aligned} A I C & = - 2 {最大対数尤度} + 2 (パラメタ数) \\ = n {\log \frac{2 π S_{e}}{n} + 1} + 2 (p + 2) \end{aligned}

$\begin{align} AIC &= -2\{\text{最大対数尤度}\} + 2(\text{パラメタ数}) \\ &= n \left\{ \log\frac{2\pi S_e}{n} + 1 \right\} + 2(p + 2) \end{align}$

S_{e} = \sum_{i = 1}^{n} (y_{i} - {\hat{y}}_{i})^{2}

$\begin{equation} S_e = \sum_{i=1}^{n}(y_i - \hat{y}_i)^2 \end{equation}$

次の線形回帰モデルを考える．

y = X β + ϵ

$\begin{equation} \bm{y} = X \bm{\beta} + \bm{\epsilon} \end{equation}$

ϵ \sim N_{n} (0, σ^{2} I_{n})

$\begin{equation} \bm{\epsilon} \sim N_n(\bm{0}, \sigma^2 I_n) \end{equation}$

β = (β_{0}, β_{1}, \dots, β_{p})^{T}

$\begin{equation} \bm{\beta} = (\beta_0, \beta_1, \ldots, \beta_p)^T \end{equation}$

このとき，

y \sim N_{n} (X β, σ^{2} I_{n})

$\begin{equation} \bm{y} \sim N_n(X \bm{\beta}, \sigma^2 I_n) \end{equation}$

となる．ここで，

y

$\bm{y}$ と

X

$X$ が観測値として得られたとすると，尤度関数は，

\begin{aligned} L (\hat{β}, {\hat{σ}}^{2}) & \equiv f (y ∣ X; β, σ^{2}) \\ = \frac{1}{(2 π)^{\frac{n}{2}} {| σ^{2} I_{n} |}^{\frac{1}{2}}} \exp [- \frac{1}{2} (y - X β)^{T} (σ^{2} I_{n})^{- 1} (y - X β)] \\ = \frac{1}{(2 π σ^{2})^{\frac{n}{2}}} \exp [- \frac{1}{2 σ^{2}} (y - X β)^{T} (y - X β)] \end{aligned}

$\begin{align} L(\hat{\bm{\beta}}, \hat{\sigma}^2) &\equiv f(\bm{y} \mid X; \bm{\beta}, \sigma^2) \\ &= \frac{1}{(2\pi)^{\frac{n}{2}}\left|\sigma^2 I_n\right|^{\frac{1}{2}}}\exp\left[-\frac{1}{2}(\bm{y} - X \bm{\beta})^T(\sigma^2 I_n)^{-1}(\bm{y} - X \bm{\beta})\right] \\ &= \frac{1}{(2\pi\sigma^2)^{\frac{n}{2}}}\exp\left[-\frac{1}{2\sigma^2}(\bm{y} - X \bm{\beta})^T(\bm{y} - X \bm{\beta})\right] \end{align}$

対数尤度関数は，

\begin{aligned} l (\hat{β}, {\hat{σ}}^{2}) & \equiv \log L (\hat{β}, {\hat{σ}}^{2}) \\ = - \frac{n}{2} \log (2 π σ^{2}) - \frac{1}{2 σ^{2}} (y - X β)^{T} (y - X β) \end{aligned}

$\begin{align} l(\hat{\bm{\beta}}, \hat{\sigma}^2) &\equiv \log L(\hat{\bm{\beta}}, \hat{\sigma}^2) \\ &= -\frac{n}{2} \log(2\pi\sigma^2) - \frac{1}{2\sigma^2}(\bm{y} - X \bm{\beta})^T(\bm{y} - X \bm{\beta}) \end{align}$

β

$\beta$ と

σ^{2}

$\sigma^2$ で偏微分して

0

$\bm{0}$ とおくと，

\frac{\partial}{\partial β} l (\hat{β}, {\hat{σ}}^{2}) = \frac{1}{σ^{2}} (X^{T} y - X^{T} X β) = 0

$\begin{equation} \frac{\partial}{\partial \bm{\beta}}l(\hat{\bm{\beta}}, \hat{\sigma}^2) = \frac{1}{\sigma^2}(X^T\bm{y} - X^TX\bm{\beta}) = \bm{0} \end{equation}$

\frac{\partial}{\partial σ^{2}} l (\hat{β}, {\hat{σ}}^{2}) = - \frac{n}{2 σ^{2}} + \frac{1}{σ^{4}} (y - X β)^{T} (y - X β) = 0

$\begin{equation} \frac{\partial}{\partial \sigma^2}l(\hat{\bm{\beta}}, \hat{\sigma}^2) = -\frac{n}{2\sigma ^2} + \frac{1}{\sigma ^4}(\bm{y} - X \bm{\beta})^T(\bm{y} - X \bm{\beta}) = \bm{0} \end{equation}$

よって，

β

$\beta$ と

σ^{2}

$\sigma^2$ の最尤推定値は，

\hat{β} = (X^{T} X)^{- 1} X^{T} y

$\begin{equation} \hat{\bm{\beta}} = (X^T X)^{-1}X^T \bm{y} \end{equation}$

{\hat{σ}}^{2} = \frac{1}{n} (y - X \hat{β})^{T} (y - X \hat{β}) = \frac{S_{e}}{n}

$\begin{equation} \hat{\sigma}^2 = \frac{1}{n}(\bm{y} - X \hat{\bm{\beta}})^T(\bm{y} - X \hat{\bm{\beta}}) = \frac{S_e}{n} \end{equation}$

これらより，最大対数尤度は，

\begin{aligned} l (\hat{β}, {\hat{σ}}^{2}) & = - \frac{n}{2} \log (2 π {\hat{σ}}^{2}) - \frac{(y - X \hat{β})^{T} (y - X \hat{β})}{2 {\hat{σ}}^{2}} \\ = - \frac{n}{2} \log \frac{2 π S_{e}}{n} - \frac{n}{2} \end{aligned}

$\begin{align} l(\hat{\bm{\beta}}, \hat{\sigma}^2) &= -\frac{n}{2}\log(2 \pi \hat{\sigma}^2) - \frac{(\bm{y} - X \hat{\bm{\beta}})^T(\bm{y} - X \hat{\bm{\beta}})}{2\hat{\sigma}^2} \\ &= -\frac{n}{2}\log\frac{2 \pi S_e}{n} - \frac{n}{2} \end{align}$

よって，

\begin{aligned} A I C & = - 2 {l (\hat{β}, {\hat{σ}}^{2}) - (p + 2)} \\ = n \log \frac{2 π S_{e}}{n} + n + 2 (p + 2) \end{aligned}

$\begin{align} AIC &= -2\left\{l(\hat{\bm{\beta}}, \hat{\sigma}^2) - (p + 2) \right\} \\ &= n \log\frac{2 \pi S_e}{n} + n + 2(p + 2) \end{align}$

第5講 回帰分析

変数選択

赤池情報量規準 (AIC)

第5講回帰分析