R - Feature selection - Model Generation (Best Subset and Stepwise)

About

This article talks about the first step of feature selection in R that is the models generation.

Once the models are generated, you can select the best model with one of this approach:

Articles Related

Steps

Function regsubsets

Best subset

Best subset regression looks through all possible regression models of all different subset sizes and looks for the best of each size. And so produces a sequence of models which is the best subset for each particular size.

best subset is quite aggressive looking at all possible subsets.

A function in leaps called regsubsets do best subset modeling.

# Load the library
library(leaps)
myPathOfModel=regsubsets(Response~.,data=myDataFrame, nvmax=8)
# By default it only goes up to subsets of size 8, you can change it with the parameter nvmax

Forward Stepwise

Statistics - Forward and Backward Stepwise (Selection|Regression)

Forward stepwise is a greedy algorithm. It produces a nested sequence of models as each time you just add the variable that improves the set the most.

The models selected are nested because each new model includes all the variables that were before plus one new one.

The regsubsets with the method=“forward” option

library(leaps)
myPathOfModel=regsubsets(response~.,data=myDataFrame,nvmax=10,method="forward")

Summary

myPathOfModel.summary=summary(myPathOfModel)
myPathOfModel.summary

Summary will summarize the best subset models:

for the best subset of size 1, it puts 1 star next to the variable that's in the best subset of size 1.
for the best subset of size 2, it puts 2 stars next to the variables that are in the best subset of size 2
…
for the best subset of size n, it puts n stars next to the variables that are in the best subset of size n

Subset selection object
Call: regsubsets.formula(Response ~ ., data = myDataFrame)
3 Variables  (and intercept)
           Forced in Forced out
Var1           FALSE      FALSE
Var2           FALSE      FALSE
Var3           FALSE      FALSE
1 subsets of each size up to 8
Selection Algorithm: exhaustive
         Var1  Var2 Var3
1  ( 1 ) " "   " "  "*"   
2  ( 1 ) "*"   "*"  " "

The subsets don't have to be nested in a best subset method whereas in forward stepwise, they are.

Variables

names(myPathOfModel.summary)

[1] "which"  "rsq"    "rss"    "adjr2"  "cp"     "bic"    "outmat" "obj"

Like for each models, the best subset models has the following variables:

You use this data to R - Feature Selection - Indirect Model Selection them

Get the regression coefficient vector for a model

coef(myPathOfModel,10)

(Intercept)        Var1         Var2        ....         Varn     
 162.5354420   -2.1686501    6.9180175    5.7732246   -0.1300798