Hybrid Feature Selection and Feature Extraction Based Ensemble Method in Classification

Abstract

Ensemble methods have been called the most influential development in Data Mining and Machine Learning in the past decade. The idea of ensemble learning is to employ multiple learners and combine their predictions. In this thesis, a novel method is proposed to build an ensemble of classifiers based on feature selection: Random selection, Relief and feature extraction: Principal component analysis method. The feature selection process chooses optimal subset of features according to objective function whereas feature extraction process maps the high dimensional dataset into lower dimensional dataset using the linear combination of original features. These feature selection and extraction method helps to produce diverse as well as accurate set of ensemble classifiers. A comparison of proposed method is made with the Bagging, AdaBoost, feature selection based NN, feature extraction based NN and also with plain NN using 22 benchmark dataset. The result obtained by the proposed method outperformed other algorithms with the following distribution: NN (14 cases), Random-NN (13 cases), Relief-NN (15 cases), PCANN (19 cases), AdaBoost (14 cases), Bagging (15 cases).

Keywords:

Ensemble methods, feature selection, feature extraction, Relief, Principal component analysis, AdaBoost, Bagging, NN, Random-NN, Relief-NN, PCA-NN