ANN Training and Final Discriminant

For the training of the ANN, a global weight for all events is used in order to normalise signal yield to the background yield. Additional weights are applied in the MC back-ground samples according to the expected yield. Since tt¯+jets accounts for more than 95% of the total background in the signal regions, only this source of background is used for the training. The training is performed independently in each signal region.

The achieved separation for the ANN can be evaluated in different ways. One way is to use the same method as defined for the input variables. However, using the Receiver Operating Characteristic (ROC) curve is preferred, since the ROC curve definition is in-dependent of the considered binning. To define the ROC curve, the efficiency is first introduced as follows:

ε(x)= R1

x Discriminant R1

−1Discriminant

, (6.23)

where:

ε(x)∀x∈[-1,1]. (6.24)

The ROC curve is obtained plottingεsigversus 1−εbkg, whereεsig andεbkg are calculated using the output discriminant obtained from signal events or background events.

To evaluate the performance of the ANN and the quality of the achieved training, two different tests are considered. These tests require splitting the sample for both signal and background in subsets. The number of subsets can be arbitrary, but given the limited statistics for both signal and background, splitting in more than two subsets is not rec-ommended. Thus, splitting is done according to the even/odd event number. Having two equivalent sets of samples for both signal and background, which are referred to as the training and the testing sample respectively, the following tests are defined:

• The overtraining test: This test is performed to evaluate a given ANN, obtained from the training sample, in the training and in the testing sample. The resulting two classification plots and the corresponding ROC curves are overlayed and com-pared. The test is passed if the ROC curves are equivalent for these two evaluations, otherwise the ANN is overfitted.

• The two-fold validation test: Having a training on the training (testing) sample, the performance of each training is evaluated on the testing (training) sample. The obtained two output classification plots and the corresponding ROC curves are over-layed and compared. The test is passed if the ROC curves are equivalent for these

6.6. ANN Training and Final Discriminant

two evaluations.

The overtraining test is performed, to check if the obtained ANN pattern recognition is biased towards the sample used for the training. The cause of overtraining is usually in-sufficient statistics of the sample used for the training, together with a number of input variables too large to train. This is another reason to use a limited number of input vari-ables to train, especially for regions with lower statistics.

Being forced to split the samples in two subsets in order to perform the overtraining test, two trainings can be made, using both the training and the testing samples, with the ad-vantage of considering all the available statistics. The two different trainings need to be equivalent, and this is exploited through the two-fold validation test. If this test is passed, the two ANN can be used when evaluating the ANN in the analysis, taking into account the even/odd splitting.

Using the ROC curves, a characteristic of the achieved separation power of the ANN is de-fined. By definition, the ROC curve integral is proportional to the latter, so the definition of the indicator is straightforward:

Index =2(AUC)−1, (6.25)

whereAUCis the area under the curve. TheIndexcan be expressed in percentage and is referred to as theFrico-Gini Index. The maximum integral value is 1, which corresponds to Frico-Gini Index of 100%. In the worst case, if the integral is 0.5, the Frico-Gini Index is 0%. The trainings are required to pass the two tests above and maximise the Frico-Gini index at the same time.

The plots for the overtraining and for the two-fold validation tests are shown in Figs.

6.14, 6.15 and 6.16. Plots of the final discriminants and theirS/Bseparation are shown in Fig. 6.17. The number of events available for the training of the ANN is shown in Table 6.8.

≥6j,≥4b ≥6j,3b 5j,≥4b

Signal 2M 2M 1M

Background 1.5M 1.5M 2M

Table 6.8.:Number of events used in the training of the ANN.

6. Analysis Strategy and MVA Techniques

ATLAS: Work in progress

εsig

ATLAS: Work in progress

even-odd odd-even total events

Figure 6.14.:Overtraining and two-fold validation tests for the (5j,≥4b) region.

ATLAS: Work in progress

εsig

ATLAS: Work in progress

even-odd odd-even total events

Figure 6.15.:Overtraining and two-fold validation tests for the (≥6j,3b) region.

ATLAS: Work in progress

εsig

ATLAS: Work in progress

even-odd odd-even total events

Figure 6.16.:Overtraining and two-fold validation tests for the (≥6j,≥4b) region.

6.6. ANN Training and Final Discriminant

220 ATLAS Internal = 13 TeV, 13.2 fb-1

*: normalised to total Bkg.

*: signal normalised to total background

NN output

0.4 Total background

= 125 GeV)

*: normalised to total Bkg.

*: signal normalised to total background

NN output

450 ATLAS Internal = 13 TeV, 13.2 fb-1

*: normalised to total Bkg.

*: signal normalised to total background

NN output

Figure 6.17.:Final ANN discriminants. For each training region both the data-prediction comparison (left) and separation of the normalised plots (right) are shown.

The binning choice is done in order to achieve the best possible separation.

(a) (5j,≥4b). (b) (≥6j,3b). (c) (≥6j,≥4b)

6. Analysis Strategy and MVA Techniques