There are several techniques for testing the overall reliability of a sample, but the most common is the test for likelihood. This is very similar to survey sampling, except, in this case, you’re trying to establish the likelihood that a given fact is true. The test for likelihood is often called a T-test, and the appropriate hypothesis to test is usually “That the sample was drawn in a random order”.
One advantage of the T-test is that it has a fairly clear validation process. For example, to validate the test, you have to change the data to look like the chart, show the charts in a group setting, and decide whether you like the distribution, either distribution-wise or distribution-equivalent (as in the chart for some probability distribution). If you do that, then you can then use the T-test for all subsequent tests. Test for likelihood can be helpful in several contexts.
Where to use the T-test in your Research?
First, it’s a good technique for evaluating large sets of data, because when you have a small sample size, it’s difficult to measure confidence in your results. Test for likelihood helps you understand the total probability you have for an observation of a fact (e.g. that the math is correct) with large sample size.
Second, test for likelihood is sometimes used to look for bias in data, or for methods that need to be validated in a small scale environment (and a small sample size).
Third, test for likelihood can also help to determine if you have a valid sample size by evaluating whether the sample group is “testable”. The method for conducting a T-test is to first make a graph with the sample data as the matrix of differences between the sample and the true distribution. Then perform a simple exercise.
Try to estimate the distribution of the data and try to determine whether the graph shows the data being drawn in random order. If you don’t like the first graph, try to change the data to look like it. Then do another T-test, and make the same analysis, this time using your revised data.
Finally, once you determine that your new data looks like the chart for the distribution of the data, you can try to compare your results to the original chart. If you still don’t like the results, you can repeat the test with different choices of the sample data (e.g. different kinds of random numbers).
[…] Why ANOVA and not T-test […]