Most Commonly used Statistical Tests in Machine Learning

 

In Machine Learning field, choosing the correct Statistical Test is crucial for testing hypotheses and drawing conclusions from the data and models. This process helps in determining whether the differences observed in model performance are due to random variations or to underlying patterns in the data.

Statistical tests play a key role in various aspects of machine learning including feature selection, model evaluation, hypothesis testing and comparing algorithms. These Tests are broadly classified into 2 categories—

a) Parametric Tests:

• These Tests require certain assumptions about the population from which the sample is drawn.
• The most common assumption is that the data and models follow some specific distribution, such as the normal distribution.

b) Non-Parametric Tests:

• These Tests do not make assumptions about the underlying distribution of data or models.
• Instead, they rely on the ranks or the signs of the data rather than the actual values.

In this article, we are going to talk about different Statistical Tests that come in the above categories.

PARAMETRIC TESTS

A. Z Test:

• It determines whether there is a significant difference in the means of two samples when the population variance is known and the sample size is large.
• This Test checks whether the sample belongs to same population distribution or not.
• Z Test can be categorized as follows—

1. One-Sample Z-Test: Determines if mean of single sample differs from a known population mean.
2. Two-Sample Z-Test: Compares the means of two independent samples to see if they are from same population.
3. Two-Proportion Z-Test: Compares proportions between two groups

B. T Test:

• It is used to assess if the difference between group means is statistically significant.
• Like Z Test, this test is based on the normal distribution of data.
• It is implemented when the sample data size is less than 30.
• This Test can be categorized as follows—

1. One-sample t-test: Checks if the mean of a single group equals a known mean.
2. Two-sample t-test: Compares the means of two independent groups.
3. Paired t-test: Compares means of the same group measured at different times.

C. ANOVA:

• A statistical method for determining if there are significant differences between the means of three or more groups.
• ANOVA can be categorized as follows—

1. One-Way ANOVA: Used when there is one independent variable with multiple groups.
2. Two-Way ANOVA: Used when there are two independent variables, allowing analysis of main effects and interaction effects between factors.
3. Repeated Measures ANOVA: Used when the same participants are measured multiple times under different conditions, accounting for correlations between repeated measurements

NON-PARAMETRIC TESTS

A. Mann-Whitney U Test:

• Also called as Wilcoxon rank-sum test
• This test is used to compare differences between two independent groups when the dependent variable is either ordinal or continuous but not normally distributed.
• It works by ranking all observations from both groups and evaluating if these ranks differ significantly.

B. Wilcoxon Signed-rank Test:

• Compares two related samples, matched samples or repeated measurements on a single sample to assess whether their population mean ranks differ.
• Instead of focusing on means, it evaluates median differences between paired observations, making it suitable for ordinal or skewed data.

C. Kruskal-Wallis H Test:

• It determines if there are statistically significant differences between the medians of three or more independent groups on an ordinal or continuous dependent variable.
• It serves as the nonparametric alternative to one-way ANOVA, especially when the normality assumption is not met.

D. Chi-Square Test:

• Determines if there is a significant association between two categorical variables.
• Widely applied in hypothesis testing for relationships between variables and for checking how well data fits a theoretical distribution.

HOW TO CHOOSE THE APPROPRIATE STATISTICAL TEST

Choosing the appropriate Statistical Test for Machine Learning is not an easy task, since there are many factors that need to be considered when making an informed decision. Following are the questions that need to be answered that revolve around these factors—

1. What is the type of data?
2. What is the type of models?
3. What is the type of comparison?
4. What is the type of metric?
5. What is the type of hypothesis?

Having answered the above questions, one is able to get the clearer image for the right Statistical Test to choose in Machine Learning.

CONCLUSION

On reaching the end of the article, we got to understand the importance of different Statistical Tests used in Machine Learning. Choosing the appropriate test based on data characteristics is crucial for obtaining valid and reliable results. Each test serves a specific purpose and helps validate different aspects of machine learning models.