This article talks about the first step of feature selection in R that is the models generation.
Once the models are generated, you can select the best model with one of this approach:
Best subset regression looks through all possible regression models of all different subset sizes and looks for the best of each size. And so produces a sequence of models which is the best subset for each particular size.
best subset is quite aggressive looking at all possible subsets.
A function in leaps called regsubsets do best subset modeling.
# Load the library
library(leaps)
myPathOfModel=regsubsets(Response~.,data=myDataFrame, nvmax=8)
# By default it only goes up to subsets of size 8, you can change it with the parameter nvmax
Statistics - Forward and Backward Stepwise (Selection|Regression)
Forward stepwise is a greedy algorithm. It produces a nested sequence of models as each time you just add the variable that improves the set the most.
The models selected are nested because each new model includes all the variables that were before plus one new one.
The regsubsets with the method=“forward” option
library(leaps)
myPathOfModel=regsubsets(response~.,data=myDataFrame,nvmax=10,method="forward")
myPathOfModel.summary=summary(myPathOfModel)
myPathOfModel.summary
Summary will summarize the best subset models:
Subset selection object
Call: regsubsets.formula(Response ~ ., data = myDataFrame)
3 Variables (and intercept)
Forced in Forced out
Var1 FALSE FALSE
Var2 FALSE FALSE
Var3 FALSE FALSE
1 subsets of each size up to 8
Selection Algorithm: exhaustive
Var1 Var2 Var3
1 ( 1 ) " " " " "*"
2 ( 1 ) "*" "*" " "
The subsets don't have to be nested in a best subset method whereas in forward stepwise, they are.
names(myPathOfModel.summary)
[1] "which" "rsq" "rss" "adjr2" "cp" "bic" "outmat" "obj"
Like for each models, the best subset models has the following variables:
You use this data to R - Feature Selection - Indirect Model Selection them
coef(myPathOfModel,10)
(Intercept) Var1 Var2 .... Varn
162.5354420 -2.1686501 6.9180175 5.7732246 -0.1300798