The ensemble method is basically a similar kind of random forest where we have n number of models and all the models will contribute to the classification task. In this, basically, we are combining the output of all the predictions through voting and average. So, in this way, the prediction output does not only depend upon a single model. So this will enhance the accuracy of the prediction. To learn about the ensemble methods of deep learning, we first have to understand the basic concepts of this method. So, we are going to learn the basic concepts of the ensemble method approach.

As we also know that all the top leaderboard on the different competitions use mostly ensemble methods to get their prediction value high.

Ensemble Method

Just imagine, we build a machine learning model to either predict the future or target based on one model. But what about the results, if we use multiple models to predict the same target. Obviously, our results will improve somewhat from using multiple models to get the prediction output.

So this is the main idea behind the ensemble method, where we train the multiple models with the same datasets, and later we use all of them together to predict our target.

So, let's understand what will be the benefits of using the above approach.

Just consider the main problems which we face during the model training are bias (under fitting) and variance (overfitting).

What is bias?

We can understand bias from the simple example that when we train any model with the datasets, the model not trying to learn from the datasets during the training period and fail to give the best results as we expect after the model training. So we can say that the bias basically is a difference between the prediction average value and the actual value. If bias is more, then we can directly imagine that error will be high.

What is a variance?

The variance comes after the training starts, which means that when the model training starts then by default there is bias because the machine just starting to learn from the training datasets. The meaning of the variance in simple language is that machine trying to learn more deeply through the datasets itself and fails to generalize on the unseen datasets. The final result comes out that the machine gives the best predictions on the training datasets but not able to give the best results on the unseen datasets.

So, now we understand the basic bias and variance meaning. Now, we are moving to our concept.

We were discussing the benefits of the ensemble approach.

If we train any single model and that model has bias and variance high, then we do not get our predictions on the unseen data. And then obviously we use some methods to overcome the bias and variance but imagine if still there bias and variance not improve then we are not able to do anything. So in that case the ensemble approach can help us.

If we train the multiple models with the same datasets then might be also there is bias and variance high, but we can average the bias and variance, and it can overcome our high bias and variance.

I hope till now we understand what actually ensemble methods do.

What will happen if we train multiple models with the same datasets?

If we think that all models will be the same on the same training datasets. Then it is not going to happen because each building model will be unique. We will train on the same datasets, but we divide the training datasets into small sub train datasets to train each model. But not like that, we divide the training datasets into the number of models we want to train. There are some methods and properties we have to follow to do this. So each building model will be unique from the other.

As we said before, all models will be unique from each other. The ensemble methods will be further classified into two categories.

1. Homogeneous Ensemble

2. Heterogeneous Ensemble

The homogeneous ensemble methods, where we're building different models with the same algorithm. But in the heterogeneous ensemble, where we were building models from different algorithms.

Let's explain some more details about the above two categories.

1. Homogeneous Ensemble Method

The homogeneous ensemble method, where we train the different models with the same algorithm and same training datasets. Even if we were giving training to different models with the same training datasets along with the same algorithm, which means not the models will be the same. The model will still be different. The only things have to remember here that the homogeneous ensemble method meaning is to train different models with the same training algorithm. The method which we used to train the same datasets for the different models is bootstrapping.

For example, Random forest where we create a different type of models with the same algorithm of the decision tree.

2. Heterogeneous Ensemble Method

In the heterogeneous ensemble method, we will train different models with different kinds of machine learning algorithms. But the training datasets will be the same. Same as in the homogeneous approach, all the building models are weak, and they contribute their prediction results to get the stronger output. The stacking approach will come under the heterogeneous ensemble method.

In both the above methods, we studied the weak and strong models. The main concepts about the weak models are that they learn not very well and at last not able to generalize the unseen data. And the strong models are those which can generalize the unseen data very well. So in both above approaches, all models in the ensemble are weak, and they overcome their weak by averaging the results of the predictions.

The method we use to divide the training datasets to train the different models of homogeneous is called Bootstrapping.

So in the next block, we will see what is Bootstrapping, Bagging, and Boosting?

Conclusion

So till now, we have understood the basic idea of the Ensemble method approach and its types. We also studied bias and variance that every single model itself is weak and to overcome bias and variance, we use multiple models. In this doing way, we are averaging the weak. In the next block, we are going to see how we can use bootstrapping methods to divide the single training datasets for the multiple models.