Abstract
When both the variance and the N are unequal in a two-group design, the probability of a Type I error shifts from the nominal 5% error rate. The probability is too liberal when the small cell has the larger variance and too conservative when the large cell has the larger variance. We present an algorithm to circumvent the problem when the smaller group has the larger variance and show, by simulation, that the algorithm brings the error rate back to the nominal value without sacrificing the ability to detect true effects.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Box, G. E. P., & Andersen, S. L. (1955). Permutation theory in the derivation of robust criteria and the study of departures from assumption. Journal of the Royal Statistical Society B, 17, 1–34.
Edgington, E. S. (1995). Randomization tests (3rd ed.). New York: Dekker.
Gill, P. M. W. (2007). Efficient calculation of p-values in linear-statistic permutation significance tests. Journal of Statistical Computation & Simulation, 77, 55–61.
Hayes, A. F. (2000). Randomization tests and the equality of variance assumption when comparing group means. Animal Behaviour, 59, 653–656. doi:10.1006/anbe.1999.1366
Ludbrook, J., & Dudley, H. (1998). Why permutation tests are superior to t and F tests in biomedical research. American Statistician, 52, 127–132.
Manly, B. F. J. (1997). Randomization, bootstrap, and Monte Carlo methods in biology (2nd ed.). London: Chapman & Hall.
Mewhort, D. J. K. (2005). A comparison of the randomization test with the F test when error is skewed. Behavior Research Methods, 37, 425–435.
Pagano, M., & Tritchler, D. (1983). On obtaining permutation distributions in polynomial time. Journal of the American Statistical Association, 78, 435–440.
Author information
Authors and Affiliations
Corresponding author
Additional information
This research was supported by Grant AP-130 from the Natural Science and Engineering Research Council of Canada to the first author. The junior authors were supported by NSERC summer research scholarships.
Rights and permissions
About this article
Cite this article
Mewhort, D.J.K., Kelly, M.A. & Johns, B.T. Randomization tests and the unequal-N/unequal-variance problem. Behavior Research Methods 41, 664–667 (2009). https://doi.org/10.3758/BRM.41.3.664
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.3758/BRM.41.3.664