Embedded Feature Selection Using Probabilistic Model-Based Optimization

Shota Saito, Shinichi Shirakawa, Youhei Akimoto: “Embedded Feature Selection Using Probabilistic Model-Based Optimization”, Student Workshop in Genetic and Evolutionary Computation Conference 2018 (GECCO 2018) , Kyoto, Japan, 15th-19th July (2018). (Best student paper nominated)


In machine learning, feature selection is a commonly used technique for improving the predictive performance and interpretability of a trained model. Feature selection techniques are classified into three approaches: the filter, wrapper, and embedded approaches. The embedded approach performs the feature selection process during the model training and achieves a good balance between performance and computational cost in general. In the paper, we propose a novel embedded feature selection method using probabilistic model-based evolutionary optimization. We introduce the multivariate Bernoulli distribution, which determines the selection of features, and we optimize its parameters during the training. The distribution parameter update rule is the same as that of the population-based incremental learning (PBIL), but we simultaneously update the parameters of the machine learning model using an ordinary gradient descent method. This method can be easily implemented into non-linear models, such as neural networks. Moreover, we incorporate the penalty term into the objective function to control the number of selected feature. We apply the proposed method with the neural network model to the feature selection of three classification problems. The proposed method achieves competitive performance and reasonable computational cost compared with conventional feature selection methods.



    author = {Saito, Shota and Shirakawa, Shinichi and Akimoto, Youhei},
    title = {Embedded Feature Selection Using Probabilistic Model-based Optimization},
    booktitle = {Proceedings of the Genetic and Evolutionary Computation Conference Companion},
    series = {GECCO '18},
    year = {2018},
    isbn = {978-1-4503-5764-7},
    location = {Kyoto, Japan},
    pages = {1922--1925},
    numpages = {4},
    url = {http://doi.acm.org/10.1145/3205651.3208227},
    doi = {10.1145/3205651.3208227},
    acmid = {3208227},
    publisher = {ACM},
    address = {New York, NY, USA},
    keywords = {embedded approach, feature selection, information geometric optimization, natural gradient, neural network},