Statistical tests of neutrality of mutations.

Abstract

Abstract Mutations in the genealogy of the sequences in a random sample from a population can be classified as external and internal. External mutations are mutations that occurred in the external branches and internal mutations are mutations that occurred in the internal branches of the genealogy. Under the assumption of selective neutrality, the expected number of external mutations is equal to theta = 4Ne mu, where Ne is the effective population size and mu is the rate of mutation per gene per generation. Interestingly, this expectation is independent of the sample size. The number of external mutations is likely to deviate from its neutral expectation when there is selection while the number of internal mutations is less affected by the presence of selection. Statistical properties of the numbers of external mutations and of internal mutations are studied and their relationships to two commonly used estimates of theta are derived. From these properties, several new statistical tests based on a random sample of DNA sequences from the population are developed for testing the hypothesis that all mutations at a locus are neutral.