Verbose C4.5 Manual Page

A guide to the verbose output of the C4.5 decision tree generator

This document explains the output of the program C4.5 when it is run with the verbosity level (option v) set to values from 1 to 3.

Verbosity Level 1

To build a decision tree from a set of data items each of which belongs to one of a set of classes, C4.5 proceeds as follows:

1. If all items belong to the same class, the decision tree is a leaf which is labelled with this class.
2. Otherwise, C4.5 attempts to find the best attribute to test in order to divide the data items into subsets, and then builds a subtree from each subset by recursively invoking this procedure for each one.

The best attribute to branch on at each stage is selected by determining the information gain of a split on each of the attributes. If the selection criterion being used is GAIN (option g), the best attribute is that which divides the data items with the highest gain in information, whereas if the GAINRATIO criterion (the default) is being used (and the gain is at least the average gain across all attributes), the best attribute is that with the highest ratio of information gain to potential information. For discrete-valued attributes, a branch corresponding to each value of the attribute is formed, whereas for continuous-valued attributes, a threshold is found, thus forming two branches. If subset tests are being used (option s), branches may be formed corresponding to a subset of values of a discrete attribute being tested.

The verbose output shows the number of items from which a tree is being constructed, as well as the total weight of these items. The weight of an item is the probability that the item would reach this point in the tree and will be less than 1.0 for items with an unknown value of some previously-tested attribute.

Shown for the best attribute is:

cut	- threshold (continuous attributes only)
inf	- the potential information of a split
gain	- the gain in information of a split
val	- the gain or the gain/inf (depending on the selection criterion)

Also shown is the proportion of items at this point in the tree with an unknown value for that attribute. Items with an unknown value for the attribute being tested are distributed across all values in proportion to the relative frequency of these values in the set of items being tested.

If no split gives a gain in information, the set of items is made into a leaf labelled with the most frequent class of items reaching this point in the tree, and the message:

no sensible splits r1/r2

is given, where r1 is the total weight of items reaching this point in the tree, and r2 is the weight of these which don't belong to the class of this leaf.

If a subtree is found to misclassify at least as many items as does replacing the subtree with a leaf, then the subtree is replaced and the following message given:

Collapse tree for n items to leaf c where c is the class assigned to the leaf.

Verbosity Level 2

When determining the best attribute to test, also shown are the threshold (continuous attributes only), information gain and potential information for a split on each of the attributes. If a test on a continuous attribute has no gain or there are insufficient cases with known values of the attribute on which to base a test, appropriate messages are given. (Sufficient here means at least twice MINOBJS, an integer which defaults to 2 but can be set with option m.) The average gain across all attributes is also shown.

If subset tests on discrete attributes are being used, for each attribute being examined, the combinations of attribute values that are made (i.e., at each stage, the combination with highest gain or gain ratio) and the potential info, gain and gain or gain ratio are shown.

Verbosity Level 3

When determining the best attribute to test, also shown is the frequency distribution table showing the total weight of items of each class with:

• each value of the attribute (discrete attributes), or
• values below and above the threshold (contin atts), or
• values in each subset formed so far (subset tests).

Verbosity Level 1

After the entire decision tree has been constructed, C4.5 recursively examines each subtree to determine whether replacing it with a leaf or a branch would be beneficial. (Note: the numbers treated below as counts of items actually refer to the total weight of the items mentioned.)

Each leaf is shown as:

c(r1:r2/r3)

with:

c	- the most frequent class at the leaf
r1	- the number of items at the leaf
r2	- misclassifications at the leaf
r3	- r2 adjusted for additional errors

Each test is shown as:

att:[n1% N=r4tree= r5leaf=r6+ r7br[n2]=r8]

with:

n1	- percentage of egs at this subtree that are misclassified
r4	- the number of items in the subtree
r5	- misclassifications of this subtree
r6	- misclassifications if this was a leaf
r7	- adjustment to r6 for additional errors
n2	- number of the largest branch
r8	- total misclassifications if subtree is replaced by largest branch

If replacing the subtree with a leaf or the largest branch reduces the number of errors, then the subtree is replaced by whichever of these results in the least number of errors.

Verbosity Level 1

In softening the thresholds of tests on continuous attributes (option p), upper and lower bounds for each test are calculated. For each such test, the following are shown:

Base errors	- the number of items misclassified when the threshold has its original value
Items	- the number of items tested (with a known value for this attribute)
se	- the standard deviation of the number of errors

For each of the different attribute values, shown are:

Val	- the attribute value
Errors	- the errors with this value as threshold
+Errors	- Errors - Base errors
+Items	- the number of items between this value and the original threshold
Ratio	- Ratio of +Errors to +Items

The lower and upper bounds are then calculated so that the number of errors with each as threshold would be one standard deviation above the base errors.

c4.5(1)