Pattern Recognition course 笔记- combiing, bagging and boosting

仅为个人笔记使用

The basic question

• how to design a combiner?
• how to generate base classifiers?

combination

combiner types

• fixed rules based on crisp labels or confidences[estimated posterior probabilities]
• special trained rules based on classifier condidences
• general trained rules interpreting based-classifier outputs as features
  $$p(f,\varphi |\omega )=p(f|\omega )p(\varphi |\omega )$$

don not need to measure the full parameters in one go,
reduce the calculation

combination —> regularization

combining classifier offen leads to regularization effect

Bagging[ Bootstrap Aggregating]

1. select a training set size $m^{\prime }<m$
2. select at random $n$ subsets of $s^{\prime }$ training objects[originally: bootstrap]
3. train a classifier[originally : decision three]
4. combine [orgina: majority vote]
5. stabilize volatile classifiers

boosting

1. initialize all objects with an euqal weight
2. select a training set size $m^{\prime }<m$ according to the object weights
3. train a weak classifier
4. increase the weights of the erroneously classified objects
5. repeat as long as needed
6. combine
7. inprove performance of weak classifiers

Adaboost

1. sample training set accoring to set of object weights[initialy equal]
2. use it for training imple [weak] classifier ${\omega }_i$
3. classify entire data set, using weights, to get error estimate ${\xi }_i$
4. store classifier weights $a_i=0.5log((1-{\xi }_i)/{\xi }_i)$
5. multiply weights of erroneously classified objects with $exp(a_i)$ and correctly classified objects with $exp(-a_i)$
6. goto 1 as long as needed
7. Final classifier: weighted voting with weights $a_i$