5.6 IRL: Randomization Tests

In real life, we don’t often consider every possible permutation (reshuffling of group members) due to the immensely large number of permutations. However, we can randomly reshuffle flights about 1000 times to try to approximate many of those permutations. Such a procedure is called a randomization or permutation procedure. There are three main steps:

1. Hypothesis

Our null hypothesis (a hypothesis that is conservative/not interesting/does not elicit action) is that there is no relationship between time of day and the mean arrival delay.

2. Generate and Calculate

We can generate 1000 new data sets based on randomly reshuffling the labels of day_hour. For each of these data sets, we calculate the difference in mean arrival delay between morning and afternoon groups.

3. Visualize

The histogram below shows the histogram of differences in means if the null hypothesis were true. The vertical line shows the observed difference in means.

Do you think that the mean arrival delay is different for morning and afternoon? Is the observed difference in means likely to have occurred if there were no relationship?

We will return to the ideas of testing hypotheses later in the course.