The space of phylogenetic mixtures for equivariant models

Abstract

Background

The selection of an evolutionary model to best fit given molecular data is usually a heuristic choice. In his seminal book, J. Felsenstein suggested that certain linear equations satisfied by the expected probabilities of patterns observed at the leaves of a phylogenetic tree could be used for model selection. It remained an open question, however, whether these equations were sufficient to fully characterize the evolutionary model under consideration.

Results

Here we prove that, for most equivariant models of evolution, the space of distributions satisfying these linear equations coincides with the space of distributions arising from mixtures of trees. In other words, we prove that the evolution of an observed multiple sequence alignment can be modeled by a mixture of phylogenetic trees under an equivariant evolutionary model if and only if the distribution of patterns at its columns satisfies the linear equations mentioned above. Moreover, we provide a set of linearly independent equations defining this space of phylogenetic mixtures for each equivariant model and for any number of taxa. Lastly, we use these results to perform a study of identifiability of phylogenetic mixtures.

Conclusions

The space of phylogenetic mixtures under equivariant models is a linear space that fully characterizes the evolutionary model. We provide an explicit algorithm to obtain the equations defining these spaces for a number of models and taxa. Its implementation has proved to be a powerful tool for model selection.

Background

The principal goal of phylogenetics is to reconstruct the ancestral relationships among organisms. Most popular phylogenetic reconstruction methods are based on mathematical models describing the molecular evolution of DNA. In spite of this, there exists no unified framework for model selection and the results are highly dependent on the models and methods used in the analysis (cf. [1]).

In this paper we assume the Darwinian model of evolution proceeding along phylogenetic trees and address the following question: how can the data evolving under a particular model be characterized? In other words, we look for invariants of the DNA patterns which have evolved following a tree (or a mixture of trees, as we will see below) under a particular model. The answer to this question provided in this paper leads to a complete characterization of the evolutionary model and to a novel model selection tool, which is valid for any mixture of trees.

In what follows, we briefly explain the motivation for this work. It has been shown that if the evolution along a phylogenetic tree is described by a particular model, the expected probabilities of nucleotide patterns at the leaves of the tree satisfy certain equalities (see e.g. [2], p.375). Several authors (e.g. [2–4]) pointed out that these equalities could potentially be used to test the fitness of the model of base change. The full set of equations required for viable model selection, however, was unknown. The objective of this work is to fill in this gap and to go a step further into practical aplication by providing an algorithm to compute the required invariants for model selection.

In this work we consider a group of equivariant models ([5, 6]). These models are Markov processes on trees, whose transition matrices satisfy certain symmetries: the Jukes-Cantor model, the Kimura 2 and 3 parameter models, the strand symmetric model, and the general Markov model. Our first important result, Theorem 17, states that if evolution occurs according to trees (or even mixtures of trees) under these equivariant models, then the model of evolution is completely determined by the linear space defined by the aforementioned equalities. By exhaustively studying the group of symmetries of these models, we also give a straightforward combinatorial way of determining the equations of this linear space (see Theorem 22). The implementation of the algorithm producing the equations is available as a package SPIn ([7],http://genome.crg.es/cgi-bin/phylo_mod_sel/AlgModelSelection.pl), which has proved to be a successful tool in evolutionary model selection.

Our main technique consists in proving that the linear space above coincides with the space ${\mathcal{D}}^{\mathcal{M}}$ of phylogenetic mixtures evolving under the model $\mathcal{M}$, i.e. the set of points that are linear combinations of points lying in the phylogenetic varieties $C{V}_T^{\mathcal{M}}$ (see Preliminaries section for specific definitions). In biological words and in the stochastic context, this is the set of vectors of expected pattern frequencies for mixtures of trees evolving under the model $\mathcal{M}$ (not necessarily the same tree topology in the mixture, and not necessarily the same transition matrices when the tree topologies coincide). In phylogenetics, the so-called i.i.d. hypothesis (independent and identically distributed) about the sites of an alignment is prevalent in the most simple models. When the assumption “identically distributed” is replaced it by “distributed according to the same evolutionary model”, one obtains a phylogenetic mixture.

Phylogenetic mixtures are useful in modeling heterogeneous evolutionary processes, e.g. data comprising multiple genes, selected codon positions, or rate variation across sites (e.g. [8]). Among a plethora of applications, they are used in orthology predictions, gene and genome annotations, species tree reconstructions, and drug target identifications.

In addition to the main result, we determine the dimension of these linear spaces and use it to give an upper bound, h₀ (n), on the number of mixtures that should be used in phylogenetic reconstruction on n taxa. This relates to the so-called identifiability problem in phylogenetic mixtures, which can be posed as determining the conditions that guarantee that the model parameters (discrete parameters in the form of tree topologies and the continuous parameters of the root and model distributions) can be recovered from the data. Identifiability is crucial for consistency of the maximum likelihood approaches and, though extensively studied in the phylogenetic context, few results are known (see for instance [9–13]).

In brief, in Theorem 30 we prove that either the tree topologies or the continuous parameters are not generically identifiable for mixtures on more than h₀ (n) trees under equivariant models. Here h₀ (n) is the quotient of the dimension of the linear space ${\mathcal{D}}^{\mathcal{M}}$ (computed in Proposition 20) by the number of free parameters of $\mathcal{M}$ on a trivalent tree plus one. For example, for four taxa and the Jukes-Cantor model (resp. the Kimura 3-parameter model) this result proves that mixtures on three (resp. four) or more taxa are not identifiable (i.e. either the discrete or the continuous parameters cannot be fully identified). A detailed discussion on this subject is provided in the last section.

The main tools used in this work are algebraic geometry and group theory. The reader is referred to [14, 15] for general references on these topics.

Main text

Preliminaries

Phylogenetic trees and Markov models of evolution have been widely used in the literature. In what follows we fix the notation needed to deal with them in our setting.

Let n be a positive integer and denote by [n] the set {1,2,…,n}. A phylogenetic tree T on the set of taxa [n] is a tree (i.e. a connected graph with no loops), whose n leaves are bijectively labeled by [n]. Its vertices represent species or other biological entities and its edges represent evolutionary processes between the vertices.

We allow internal vertices of any degree and if all the internal vertices are of degree 3 we say that the tree is trivalent. We will denote the set of vertices of T by N(T), the set of edges by E(T), and the set of interior nodes by Int(T). A rooted tree is a tree together with a distinguished node r called the root. The root induces an orientation on the edges of T, whereby the root represents the common ancestor to all the species represented in the tree. If e is an edge of a rooted tree T, we write pa(e) and ch(e) for its parent vertex (origin) and its child vertex (end), respectively. Two unrooted phylogenetic trees on the set of taxa [n] are said to have the same tree topology if their labeled graphs have the same topology.

We fix a positive integer k and an ordered set B = {b₁,b₂,…,b _k }. For example, for most applications we take B = {A,C,G,T} to be the set of nucleotides in a DNA sequence. We may think of B as the set of states of a discrete random variable. We call W the complex vector space $W={〈B〉}_{\mathbb{C}}$ spanned by B, so that B is a natural basis of W. For algebraic convenience, we usually work over the complex field and restrict to the stochastic setting when necessary. Vectors in W are thought of as probability distributions on the set of states B if their coordinates are non-negative and sum to one. In this setting the vector $\sum c_i{b}_i$ means that observation b _i occurs with probability c _i . From now on, we will identify vectors in W with their coordinates in the basis B written as a column vector, e.g. we identify ${\sum }_k{b}_k$ with the vector 1 = (1,1,…,1)^t∈ W.

In order to model molecular evolution on a phylogenetic tree T, we consider a Markov process specified by a root distribution, Π∈W, and a collection of transition matrices, A = (A^e)_{e ∈ E (T)}, where each A^eis a k × k-matrix in End(W). The matrices A^erepresent the conditional probabilities of substitution between the states in B from the parent node pa (e) to the child node ch(e) of e. We adopt the convention that the matrices A^e act on W from the right, i.e. a vector ω^tin pa (e) maps to ω^tA^e in ch(e).

Distinct forms of the transition matrices give rise to different evolutionary models. Using the terminology introduced above, we proceed to the definition of evolutionary models used throughout this work.

Definition 1. An (algebraic) evolutionary model$\mathcal{M}$ is specified by giving a vector subspace $W_0\subset W$ such that 1^tΠ ≠ 0 for some Π in W₀, together with a multiplicatively closed vector subspace Mod(for model) of $M_k\left(\mathbb{C}\right)$ containing the identity matrix. We will usually denote such a model by $\mathcal{M}=(W_0,\mathit{\text{Mod}})$. We define the stochastic evolutionary model$s\mathcal{M}=(s{W}_0,\mathit{\text{sMod}})$associated to$\mathcal{M}$ by taking s W₀ = {Π∈W₀:1^tΠ = 1} and sMod = {A∈Mod:A 1 = 1}. The term “stochastic” refers to the fact that, by restricting to the points in the spaces with non-negative real entries, we obtain distributions and Markov matrices. A phylogenetic tree T together with the parameters Π and A = (A^e) _{e ∈ E (T)}is said to evolve under the algebraic evolutionary model$\mathcal{M}$ if Π ∈ W₀, and all matrices A^elie in Mod.

Remark 2. Note that s W₀and sMod are not vector spaces. The condition 1^tΠ ≠ 0 in the above definition means that the sum of the coordinates of Π is not zero. Since vectors in s W₀ with non-negative coordinates represent the probability distributions for the set of observations B, this condition implies no restriction from a biological point of view. Moreover, it ensures that $W_0\cap \{{\sum }_{\mathtt{x}\in B}{\Pi }_{\mathtt{x}}=1\}$ has dimension equal to dim(W₀) − 1. In particular, the simplex of stochastic vectors in W₀will form a semialgebraic set of ${〈B〉}_{\mathbb{R}}$ of dimension equal to dim(W₀)−1 (as expected).

Remark 3. The subspace Mod of substitution matrices is usually required to be multiplicatively closed (as in the definition above) so that when two evolutionary processes are concatenated, the final process is of the same kind. The importance of this requirement is the starting point of [16], where a different approach to the definition of “evolutionary mode” is provided.

Our definition of evolutionary models includes most of the well-known evolutionary models, namely those given in [17] and the equivariant models (see [5, 6]).

Example 4. Let G be a permutation group of B, that is, a group whose elements are permutations of the set B, $G\le {\mathfrak{S}}_k$. Given g ∈ G, write P _g for the k × k-permutation matrix corresponding to g: (P _g )_i,j= 1 if g(j) = i and 0 otherwise. The G-equivariant evolutionary model${\mathcal{M}}_G$ is defined by taking Mod equal to

$$\mathrm{m:m}\left(G\right)=\{A\in {\mathrm{m:m}}_k(\mathbb{C})P_{g}A{P}_g^{-1}=A\text{for all}g\in G\},$$

and W₀ = {Π ∈ W ∣ P _g Π = Π for all g ∈ G}. These subsets are vector subspaces of $M_k\left(\mathbb{C}\right)$ and W, respectively. Moreover, if A₁,A₂ ∈ M(G), then

$$P_g{A}_1{A}_2{P}_g^{-1}=\left(P_g{A}_1{P}_g^{-1}\right)\left(P_g{A}_2{P}_g^{-1}\right)=A_1{A}_2,$$

and A₁A₂ ∈ M (G). Therefore, equivariant models provide a wide family of examples of algebraic evolutionary models in the sense of Definition 1. For example, if B = {A,,̧G T}, it can be seen that the algebraic versions of the Jukes-Cantor model [18], the Kimura models with 2 or 3 parameters [19, 20], the strand symmetric model [21] or the general Markov model [22] are instances of equivariant models:

• if $G={\mathfrak{S}}_4$, then ${\mathcal{M}}_G$ is the algebraic Jukes-Cantor model JC69,

• if G = 〈(A C G T),(A G)〉, then ${\mathcal{M}}_G$ is the algebraic Kimura 2-parameter model K80,

• if G = 〈(A C)(G T),(A G)(C T)〉, then ${\mathcal{M}}_G$ is the algebraic Kimura 3-parameter model K81,

• if G = 〈(A T)(C G)〉, then ${\mathcal{M}}_G$ is known as the strand symmetric model SSM, and

• if G = 〈e〉, then ${\mathcal{M}}_G$ is the general Markov model GMM.

Given an evolutionary model $\mathcal{M}$ and a phylogenetic tree T, we define the space of parameters as

$${\text{Par}}_{\mathcal{M}}\left(T\right)=W_0\times \left(\prod _{e\in E\left(T\right)}\mathit{\text{Mod}}\right).$$

Similarly, we define the space of stochastic parameters associated to T by

$${\text{Par}}_{s\mathcal{M}}\left(T\right)=s{W}_0\times \left(\prod _{e\in E\left(T\right)}\mathit{\text{sMod}}\right).$$

Though artificial at first glance, the use of tensors in the framework that includes the distributions on the set of patterns in B at the leaves of a phylogenetic tree is a natural choice. Indeed, if $p_{{\mathtt{x}}_1{\mathtt{x}}_2\dots {\mathtt{x}}_n}$ denotes the joint probability of observing x₁ at leaf 1, x₂ at leaf 2, and so on, up to x _n at leaf n, then the vector $p=(p_{b_1\dots b_1},p_{b_1{b}_1\dots b_2},\dots ,p_{b_k\dots b_k})$ provides a distribution on the set of patterns in B at the leaves of T, and this can be regarded as the tensor having these coordinates in the natural basis,

$$p=\sum _{{\mathtt{x}}_1\dots {\mathtt{x}}_n\in B}{p}_{{\mathtt{x}}_1\dots {\mathtt{x}}_n}{\mathtt{x}}_1\dots {\mathtt{x}}_n\text{.}$$

This motivates the following definition.

Definition 5. Given a phylogenetic tree T on the set of taxa [n], an [n] - tensor is any element of the tensor power

Given an algebraic evolutionary model $\mathcal{M}$ and a phylogenetic tree T with root r, every Markov process on T (specified by a collection of parameters Π and A = (A _e )_e∈E(T)) gives rise to a tensor in $\mathcal{L}$ in the following way: we consider a parametrization

$${\mathrm{\Psi }}_T^{\mathcal{M}}:{\text{Par}}_{\mathcal{M}}\left(T\right)\to \mathcal{L}$$

(1)

defined by

$${\mathrm{\Psi }}_T^{\mathcal{M}}\left(\Pi ,\mathbf{A}\right)=\sum _{{\mathtt{x}}_i\in B}{p}_{{\mathtt{x}}_1\dots {\mathtt{x}}_n}{\mathtt{x}}_1{\mathtt{x}}_n,$$

where

$$p_{{\mathtt{x}}_1\dots {\mathtt{x}}_n}=\sum _{{\mathtt{x}}_v\in B,v\in \mathit{Int}\left(T\right)}{\Pi }_{{\mathtt{x}}_r}\prod _{e\in E\left(T\right)}{A}_{{\mathtt{x}}_{\mathit{\text{pa(e)}}},{\mathtt{x}}_{\mathit{\text{ch(e)}}}}^e,$$

(2)

x _v denotes the state at the vertex v, pa(e) (resp. ch (e)) is the parent (resp. child) node of e, and Π _x ,x ∈ B, are the coordinates of Π. When restricted to the stochastic matrices and distributions in W₀, this parametrization corresponds to the hidden Markov process on the tree T (the leaves correspond to the observed random variables and the interior nodes to the hidden variables).

The parametrization (1) restricts to another polynomial map ${\varphi }_T^{\mathcal{M}}:{\text{Par}}_{s\mathcal{M}}\left(T\right)\to H$, where $H\subset \mathcal{L}$ is the hyperplane defined by $H=\left\{p\in \mathcal{L}\mid {\sum }_{{\mathtt{x}}_1,\dots ,{\mathtt{x}}_n\in B}{p}_{{\mathtt{x}}_1\dots {\mathtt{x}}_n}=1\right\}$. Because we work in the algebraic setting, the use of the word “stochastic” in this paper is more general than usual, as we only request entries summing to one.

From now on, we will refer to this restriction as the stochastic parametrization${\varphi }_T^{\mathcal{M}}$. It is important to note that when we consider the distributions in s W₀ and the Markov matrices in sMod, its image by ${\varphi }_T^{\mathcal{M}}$ lies in the standard simplex in $\mathcal{L}$ (and thus in H). This in turn implies that the whole image $\text{Im}{\varphi }_T^{\mathcal{M}}$ is contained in H.

We proceed to define the algebraic varieties associated to the parametrization maps defined above. Roughly speaking, algebraic varieties are sets of solutions to systems of polynomial equations (e.g. [14]).

Definition 6. The stochastic phylogenetic variety$V_T^{\mathcal{M}}$associated to a phylogenetic tree T is the smallest algebraic variety containing $\text{Im}{\varphi }_T^{\mathcal{M}}=\left\{{\varphi }_T^{\mathcal{M}}({\Pi }_r,\mathbf{A}):\right.\left.({\Pi }_r,\mathbf{A})\in {\text{Par}}_{s\mathcal{M}}\left(T\right)\right\}$ (in particular, $V_T^{\mathcal{M}}\subset H$).

Similarly, the phylogenetic variety$C{V}_T^{\mathcal{M}}$associated to T is the smallest algebraic variety in $\mathcal{L}$ that contains $\text{Im}{\mathrm{\Psi }}_T^{\mathcal{M}}=\left\{{\mathrm{\Psi }}_T^{\mathcal{M}}({\Pi }_r,\mathbf{A}):({\Pi }_r,\mathbf{A})\in {\text{Par}}_{\mathcal{M}}\left(T\right)\right\}$.

Below we explain the reason for the notation of $C{V}_T^{\mathcal{M}}$, which was adopted from [23].

The reader may note that the position of the root r of T played a role in the above parameterizations. It can be shown, however, that under certain mild assumptions, $\text{Im}{\Psi }_T^{\mathcal{M}}$ and $\text{Im}{\varphi }_T^{\mathcal{M}}$ are independent of the root position in the following sense: if two phylogenetic trees have the same topology as unrooted trees, then the smallest algebraic varieties containing the corresponding image sets are the same. For example, any model $\mathcal{M}=(W_0,\mathit{\text{Mod}})$ satisfying (i) ${\tilde{\Pi }}^t:={\Pi }^{t}A$ belongs to W₀ for all Π ∈ W₀ and all A ∈ Mod, and (ii) $D_{\tilde{\Pi }}^{-1}{A}^t{D}_{\Pi }\in \mathit{\text{Mod}}$ whenever $D_{\tilde{\Pi }}^{-1}$ exists (here D _ω denotes the diagonal matrix with the entries of ω on the diagonal and zeros elsewhere) has this property (in this case, we say the model is root-independent). It is not difficult to check that the equivariant models satisfy these two properties (e.g. adapting the proof of [24] or [25]). For technical reasons, from now on we consider only the evolutionary models satisfying (i) and (ii). Indeed, in this case the notation $C{V}_T^{\mathcal{M}}$ refers to the fact that the phylogenetic variety is just the cone over the stochastic phylogenetic variety (see Figure 1 and the remark below).

Figure 1

On the left, the varieties ${\mathit{V}}_{\mathit{T}}^{\mathcal{M}}$ and ${\mathit{CV}}_{\mathit{T}}^{\mathcal{M}}$ are shown; on the right, the phylogenetic tree described in the proof of Proposition 13 is represented.

Remark 7. Let $\mathcal{M}$ be an evolutionary model satisfying (i) and (ii) above. For $p\in \mathcal{L}$, $p=\sum p_{{\mathtt{x}}_1\dots {\mathtt{x}}_n}{\mathtt{x}}_1\otimes \dots \otimes {\mathtt{x}}_n$, define $\lambda \left(p\right):={\sum }_{{\mathtt{x}}_i\in B}{p}_{{\mathtt{x}}_1\dots {\mathtt{x}}_n}$. Then

$$C{V}_T^{\mathcal{M}}=\left\{p\in \mathcal{L}|p=\lambda (p)q,q\in V_T^{\mathcal{M}}\right\}$$

and $V_T^{\mathcal{M}}=C{V}_T^{\mathcal{M}}\cap H$. This is well known for the general Markov model [23] and can be easily generalized to any model satisfying (i) and (ii).

The space of phylogenetic mixtures

In phylogenetics, the hypothesis that the sites of an alignment are independent and identically distributed is often used. When the assumption “identically distributed” is replaced by “distributed according to the same evolutionary model”, one obtains a phylogenetic mixture. Below, we introduce phylogenetic mixtures from the algebraic point of view (see also [26]).

Definition 8. Fix a set of taxa [n] and an algebraic evolutionary model $\mathcal{M}$. A phylogenetic mixture (on m-classes) or m-mixture is any vector of the form

$$p=\sum _{i=1}^m{\alpha }_i{p}^i,$$

where ${\alpha }_i\in \mathbb{C}$ and $p^i\in \text{Im}\left({\Psi }_{T_i}^{\mathcal{M}}\right)$ for some tree topologies T _i on the set of taxa [n]. As ${\Psi }_{T_i}^{\mathcal{M}}$ is a homogeneous map, phylogenetic mixtures are represented by vectors of the form ${\sum }_{i=1}^m{\check{p}}^i$, where ${\check{p}}^i\in \text{Im}\left({\Psi }_{T_i}^{\mathcal{M}}\right)$. We call ${\mathcal{D}}_{\mathcal{M}}\subset \mathcal{L}$ the space of all phylogenetic mixtures (on any number of classes) under the algebraic evolutionary model $\mathcal{M}$.

As mentioned in the introduction, the tree topologies contained in the mixture can be the same or different. An example of a phylogenetic mixture is the data modeled by the discrete Gamma-rates models (see e.g. [8]).

Restricting matrix rows to sum to one requires restricting the phylogenetic mixtures to the points of the form

$$q=\sum _{i=1}^m{\alpha }_i{q}^i\text{where}{q}^i\in \text{Im}\left({\varphi }_{T_i}^{\mathcal{M}}\right),\text{and}\sum _i{\alpha }_i=1.$$

We call ${\mathcal{D}}_{s\mathcal{M}}$ the space of stochastic phylogenetic mixtures.

Remark 9. The phylogenetic variety of a trivalent tree topology contains all phylogenetic varieties of the non-trivalent tree topologies obtained by contracting any of its interior edges. Indeed, the latter are a particular case of the former when the matrices associated to the contracted edges are equal to the identity matrix. It follows that the space of phylogenetic mixtures on the trivalent tree topologies coincides with the space of phylogenetic mixtures on all possible topologies.

The following result was proven by Matsen, Mossel and Steel in [26] for the two state random cluster model but, as proved below, it can be easily generalized to any evolutionary model.

Lemma 10. Given a set of taxa [n] and an algebraic evolutionary model $\mathcal{M}$, the set of all phylogenetic mixtures ${\mathcal{D}}_{\mathcal{M}}$ is a vector subspace of $\mathcal{L}$. Similarly, ${\mathcal{D}}_{s\mathcal{M}}$ is a linear variety and it equals ${\mathcal{D}}_{\mathcal{M}}\cap H$.

Proof.${\mathcal{D}}_{\mathcal{M}}$ is a $\mathbb{C}$-vector space and ${\mathcal{D}}_{s\mathcal{M}}$ is a linear variety by their definition. It follows that ${\mathcal{D}}_{\mathcal{M}}$ is an algebraic variety that contains $\text{Im}{\Psi }_T^{\mathcal{M}}$ for any phylogenetic tree T on the set of taxa [n]. Therefore, it also contains $C{V}_T^{\mathcal{M}}$, and ${\mathcal{D}}_{\mathcal{M}}$ equals the set of points of the form $p=\sum p_i$, where $p_i\in C{V}_{T_i}^{\mathcal{M}}$. Similarly, ${\mathcal{D}}_{s\mathcal{M}}$ is an algebraic variety that contains $\text{Im}{\varphi }_T^{\mathcal{M}}$, so it also contains $V_T^{\mathcal{M}}$ for any phylogenetic tree T. It follows that ${\mathcal{D}}_{s\mathcal{M}}$ is formed by points of type $q=\sum {\alpha }_i{q}_i$, where $q_i\in V_{T_i}^{\mathcal{M}}$ and ${\sum }_i{\alpha }_i=1$.

Now we check that ${\mathcal{D}}_{s\mathcal{M}}={\mathcal{D}}_{\mathcal{M}}\cap H$. Let $q\in {\mathcal{D}}_{s\mathcal{M}}$, so that $q={\sum }_{i=1}^m{\alpha }_i{q}^i$ for some m, $q^i\in V_{T_i}^{\mathcal{M}}$, and $\sum {\alpha }_i=1$. Clearly, $q\in {\mathcal{D}}_{\mathcal{M}}$. Moreover, the sum of coordinates of q, λ(q), satisfies $\lambda \left(q\right)={\sum }_i{\alpha }_i\lambda \left(q^i\right)={\sum }_i{\alpha }_i=1$. Thus, q ∈ H. Conversely, let $p={\sum }_{i=1}^m{p}^i$ with $p^i\in C{V}_{T_i}^{\mathcal{M}}$ for certain tree topologies T _i , and assume that λ (p) = 1. Apply Remark 7 to each pⁱto get pⁱ= λ (pⁱ)q _i for some $q_i\in V_{T_i}^{\mathcal{M}}$. Then

$$p=\sum _i{p}^i=\sum _i\lambda \left(p^i\right)q_i$$

and $1=\lambda \left(p\right)={\sum }_i\lambda \left(p^i\right)\lambda \left(q_i\right)={\sum }_i\lambda \left(p^i\right)$ since each q _i lies on H. This proves that $p\in {\mathcal{D}}_{s\mathcal{M}}$. □

Remark 11. In the proof of the above lemma, we have seen that ${\mathcal{D}}_{\mathcal{M}}$ and ${\mathcal{D}}_{s\mathcal{M}}$ can be alternatively described as the spaces of mixtures obtained from the respective varieties $C{V}_T^{\mathcal{M}}$ and $V_T^{\mathcal{M}}$ (i.e., not only from the images of the parametrization maps).

The space of phylogenetic mixtures for equivariant evolutionary models

This section provides a precise description of the space ${\mathcal{D}}_{\mathcal{M}}$ for the equivariant models $\mathcal{M}$ listed in Example 4 (JC69, K80, K81, SSM, and GMM). First, we recall some definitions and facts of group theory and linear representation theory. From now on, B = {A,C,G,T}, k = 4, $W={〈B〉}_{\mathbb{C}}$, n is fixed and $\mathcal{L}={\otimes }_{\left[n\right]}W$.

Background on representation theory

We introduce some tools in group representation theory needed in the sequel. We refer the reader to [15] as a classical reference for these concepts. Although some of the following results are valid for any permutation group, for simplicity in the exposition we restrict to permutations of four elements (as our applications deal only with the case B = {A C G T}).

Let $G\le {\mathfrak{S}}_4$ be a permutation group. The trivial element in ${\mathfrak{S}}_4$ will be denoted as e. We write ρ _G for the restriction to G of the defining representation $\rho :{\mathfrak{S}}_4\to \mathit{GL}\left(W\right)$ given by the permutations of the basis B of W. This representation induces a G-module structure on W by setting g · x: = ρ(g)(x) ∈ W. In fact, ρ induces a G-module structure on any tensor power ⊗^sW by setting

$$g·\left({\mathtt{x}}_1\otimes \dots \otimes {\mathtt{x}}_s\right):=g·{\mathtt{x}}_1\otimes \dots \otimes g·{\mathtt{x}}_s,$$

(3)

and extending by linearity. From now on, the space $\mathcal{L}={\otimes }^{n}W$ will be implicitly considered as a G-module with this action. We call χ the character associated to the representation ρ _G : G → GL (W), i.e. χ (g) is the trace of the corresponding permutation matrix or, in other words, χ (g) equals the number of fixed elements in B by the permutation g∈G. Then the character associated to the induced representation G → GL (⊗ⁿW) is χⁿ, the n-th power of χ.

We write N₁,…,N _t for the irreducible representations of G and ω₁,…,ω _t for the corresponding irreducible characters, where N₁ and ω₁ will denote the trivial representation and trivial character, respectively. Maschke’s Theorem applied to the action of G described in (3) states that there is a decomposition of ⊗^sW into its isotypic components:

$${\otimes }^{s}W={\oplus }_{i=1}^t\left({\otimes }^{s}W\right)\left[{\omega }_i\right],$$

(4)

where each (⊗^sW)ω _i is isomorphic to a number of copies of the irreducible representation N _i associated to ω _i , $\left({\otimes }^{s}W\right)\left[{\omega }_i\right]\cong N_i\otimes {\mathbb{C}}^{m_i\left(s\right)}$, for some non-negative integer m _i (s) called the multiplicity of⊗^sW relative to ω _i . The isotypic component of $\mathcal{L}$ associated to the trivial representation will be denoted by ${\mathcal{L}}^G$ and it is composed of the n-tensors invariant under the action of G defined in (3). If $\mathcal{M}$ is the equivariant evolutionary model associated to G, ${\mathcal{L}}^G$ will also be denoted as ${\mathcal{L}}^{\mathcal{M}}$. It is easy to prove that $C{V}_T^{\mathcal{M}}\subset {\mathcal{L}}^G$ (see Lemma 4.3 of [5]).

We recall that the set Ω _G = {ω _i }_{i = 1,…,t}of irreducible characters of G forms an orthonormal basis of the space of characters relative to the inner product defined by

$$〈f,h〉:=\frac{1}{\left|G\right|}\sum _{g\in G}f\left(g\right)\overline{h\left(g\right)}\text{.}$$

(5)

We introduce the following notion.

Definition 12. An n - word over B is an ordered sequence X = x₁x₂ … x _n , where every letter is taken from the alphabet B. The set of n-words is equivalent to the cartesian power Bⁿand will be denoted by $\mathcal{B}$.

Words will be denoted in typewritter uppercase font (like X) and their letters in lowercase (like X). Sometimes it will be convenient to identify the [n]-tensors of the form x₁ ⊗ … ⊗ x _n with the n-words X = x₁…x _n . Consequently, we will identify $\mathcal{B}$ with the natural basis of $\mathcal{L}$. Given $\mathtt{X}\in \mathcal{B}$, we will denote by {X} _G = {g X ∣ g ∈ G} the G-orbit of X. We associate a G-invariant tensor, τ{X} _G , to each orbit {X} _G : $\tau {\left\{\mathtt{X}\right\}}_G:={\sum }_{g\in G}g\mathtt{X}$. It is straightforward to see that every G-invariant tensor can be written as a linear combination of the tensors τ{X} _G , $\mathtt{X}\in \mathcal{B}$. On the other hand, the set of different τ{X} _G ’s is linearly independent, since the corresponding G-orbits {X} _G have non-overlapping composition of the elements of $\mathcal{B}$.

Mixtures for equivariant models

For each x ∈ B, we write S _G (x) for the stabiliser of X under the action of G, that is, S _G (x) = {g ∈ G : g·x = x}.

Proposition 13. Let G be a subgroup of ${\mathfrak{S}}_4$ such that S _G (x₀) = {e} for some x₀∈B. Then every tensor of type τ{X} _G , $\mathtt{X}\in \mathcal{B}$, lies in the image of ${\Psi }_T^{{\mathcal{M}}_G}$ for some tree topology T. In particular, ${\mathcal{L}}^G\subset {\mathcal{D}}_{{\mathcal{M}}_G}$.

Proof. For any G-orbit {Y} _G , $\mathtt{Y}\in \mathcal{B}$, write $\tau {\left\{\mathtt{Y}\right\}}_G=y_1\otimes \dots \otimes {\mathtt{Y}}_n+\sum _{g\ne e}g·{\mathtt{Y}}_1\otimes \dots \otimes g·{\mathtt{Y}}_n$. We will explicitly associate a tree topology and parameters (Π,A) to it so that the tensor τ {T} _G is equal to ${\Psi }_T^{{\mathcal{M}}_G}$. To this aim, we denote by B (Y) the set of letters appearing in Y. Then for every z ∈ B (Y), consider the set $L_{\mathtt{z}}^{\mathtt{Y}}=\{i\in [n]:{\mathtt{Y}}_i=\mathtt{z}\}$, so that ${\cup }_{\mathtt{z}\in B\left(\mathtt{Y}\right)}{L}_{\mathtt{z}}^{\mathtt{Y}}=\left[n\right]$.

We construct a tree T on the set of taxa [n] in the following way. We join each taxa in $L_{\mathtt{z}}^{\mathtt{Y}}$ to a common node v _z by an edge. Then each vertex v _z is joined to the root of the tree (we call it r) by an edge that we denote as e(z) (see Figure 1). Now, in the edges joining any v _z with some leaf in $L_{\mathtt{z}}^{\mathtt{Y}}$, we consider the identity matrix, while the matrix in e(z) is defined by taking

$$A_{i,j}^{e\left(\mathtt{z}\right)}=\left\{\begin{array}{ll}1& \text{if}(i,j)=(h·{\mathtt{x}}_0,h·\mathtt{z})\text{for some}h\in G,\\ 0& \text{otherwise.}\end{array}\right.$$

Finally, if c is the cardinality of {x₀} _G , define the distribution at the root Π=(Π _A ,Π _C ,Π _G ,Π _T ) by

$${\Pi }_{\mathtt{z}}=\left\{\begin{array}{ll}\frac{1}{c}& \text{if}\mathtt{z}\in {\left\{{\mathtt{x}}_0\right\}}_G,\\ 0& \text{otherwise.}\end{array}\right.$$

It is straightforward to check that these matrices and the vector Π are G-equivariant, so $(\Pi ,\mathbf{A})\in {\text{Par}}_{{\mathcal{M}}_G}\left(T\right)$. Now, from (2) and the definition of Π, we can write

$$\begin{array}{l}{p}_{{\mathtt{x}}_1\dots {\mathtt{x}}_n}=\sum _{\begin{array}{c}g\in G\\ {\left\{{\mathtt{x}}_{\mathtt{z}}\right\}}_{\mathtt{z}\in B\left(\mathtt{Y}\right)}\subset B\end{array}}{P}_{{\mathtt{x}}_1\dots {\mathtt{x}}_n}(g,{\left\{{\mathtt{x}}_{\mathtt{z}}\right\}}_{\mathtt{z}\in B\left(\mathtt{Y}\right)})\end{array}$$

where

$$\begin{array}{l}{P}_{{\mathtt{x}}_1\dots {\mathtt{x}}_n}(g,{\left\{{\mathtt{x}}_{\mathtt{z}}\right\}}_{\mathtt{z}\in B\left(\mathtt{Y}\right)})={\Pi }_{g·{\mathtt{x}}_0}\prod _{\mathtt{z}\in B\left(Y\right)}\left(A_{g·{\mathtt{x}}_0,{\mathtt{x}}_{\mathtt{z}}}^{e\left(\mathtt{z}\right)}\prod _{j\in L_{{\mathtt{x}}_{\mathtt{z}}}^{\mathtt{Y}}}{\delta }_{{\mathtt{x}}_{\mathtt{z}},{\mathtt{x}}_j}\right)\end{array}$$

(here δ_a,b stands for the Kronecker delta, i.e. δ_a,a= 1, δ_a,b= 0 if a ≠ b). Moreover, from the definition of the matrix A^e(z), we have

$$A_{g·{\mathtt{x}}_0,{\mathtt{x}}_{\mathtt{z}}}^{e\left(\mathtt{z}\right)}=\left\{\begin{array}{ll}1& \text{if}(g·{\mathtt{x}}_0,{\mathtt{x}}_{\mathtt{z}})=(h·{\mathtt{x}}_0,h·\mathtt{z})\text{for some}h\in G,\\ 0& \text{otherwise.}\end{array}\right.$$

The hypothesis S _G (x₀) = {e} ensures that (g·x₀,x _z ) = (h·x₀,h·z) if and only if g = h. From this, it becomes clear that $P_{{\mathtt{x}}_1\dots {\mathtt{x}}_n}(g,{\left\{{\mathtt{x}}_{\mathtt{z}}\right\}}_{\mathtt{z}\in B\left(Y\right)})=0$ unless

• 1. x _z = g · z, for z ∈ B, and

• 2. for each $i\in L_{\mathtt{z}}^{\mathtt{Y}}$, x _i is equal to x _z = g · z,

in which case $P_{{\mathtt{x}}_1\dots {\mathtt{x}}_n}(g,{\left\{{\mathtt{x}}_{\mathtt{z}}\right\}}_{\mathtt{z}\in B\left(Y\right)})={\Pi }_{g·x_0}=\frac{1}{c}$. It follows that

$$p_{{\mathtt{x}}_1\dots {\mathtt{x}}_n}=\left\{\begin{array}{ll}1& \text{if}{\mathtt{x}}_1\dots {\mathtt{x}}_n\in {\left\{\mathtt{Y}\right\}}_G,\\ 0& \text{otherwise,}\end{array}\right.$$

and ${\Psi }_T^{\mathcal{M}}(\Pi ,\mathbf{A})=\tau {\left\{\mathtt{Y}\right\}}_G$. Moreover, as the set of τ{Y} _G , for $\mathtt{Y}\in \mathcal{B}$, generates the vector space ${\mathcal{L}}^G$, the second claim follows. □

Remark 14. The above result is not true if the hypothesis S _G (x₀) = 〈e〉 is removed. For example, if G = 〈(A C G T),(A G)〉 (so that $\mathcal{M}=\mathtt{K80}$), then S _G (A) = S _G (G) = {e,(C T)} and S _G (C) = S _G (T) = {e,(A G)}. In that case, it can be shown that the G-orbit {A C G T} _G is not in $\text{Im}{\Psi }_T^{\mathtt{K80}}$ for any tree topology T with 4 leaves.

Since the above condition on the group holds for $G={\mathfrak{S}}_4$, G = 〈(A T)(C G)〉, and G = 〈(A G),(A C G T)〉, we deduce the following claim.

Corollary 15. If G corresponds to any of the equivariant models K81, SSM or GMM, we have ${\mathcal{L}}^G\subset {\mathcal{D}}_{{\mathcal{M}}_G}$.

In phylogenetics, an invariant of a phylogenetic tree T is an equation satisfied by the expected distributions of patterns at the leaves of T, irrespectively of the continuous parameters of the model $\mathcal{M}$. In the algebraic geometry setting, these are the equations satisfied by all $p\in C{V}_T^{\mathcal{M}}$. Invariants were introduced by Lake (see [27]) and Cavender and Felsenstein (see [28]). A phylogenetic invariant of T is an invariant of T, which is not an invariant of all other phylogenetic trees (under the same model $\mathcal{M}$). Equivalently, f is a phylogenetic invariant of $C{V}_T^{\mathcal{M}}$ if it is an invariant of $C{V}_T^{\mathcal{M}}$ and there exists a tree topology T^′ such that f is not an invariant of $C{V}_{T^{\prime }}^{\mathcal{M}}$. In principle, phylogenetic invariants can be used for tree topology reconstruction purposes.

Remark 16.

• (a) It can be seen that the condition of trivial stabiliser for some element of B given in Proposition 13 guarantees that all the irreducible representations of G will be present in the decomposition of W into its isotypic components. Then, by using the results of [6], it follows that the corresponding equivariant model will have no linear phylogenetic invariants. This fact was already known for the models in the above corollary: see [29] for the GMM, [21] for the SSM and [30] for the K81. Here we provided an alternative proof based on elementary tools of group theory.

• (b) The models JC69 and K80 are known to have linear phylogenetic invariants, but these are the only linear invariants which do not define hyperplanes containing ${\mathcal{L}}^G$, as can be deduced from [3, 30]. In fact, for these two models, the claim of the corollary is still true as stated in the following theorem. Nevertheless, we have not been able to provide a unified proof of this fact because of the different properties of the corresponding groups. There is no description of the space of linear invariants for other equivariant models not listed in Example 4, so we cannot claim that the result below still holds.

Theorem 17. If ${\mathcal{M}}_G$ is one of the equivariant evolutionary models JC69 , K80 , K81 , SSM , or GMM , then the space of phylogenetic mixtures ${\mathcal{D}}_{{\mathcal{M}}_G}$ coincides with ${\mathcal{L}}^G$ , and ${\mathcal{D}}_{s{\mathcal{M}}_G}$ equals ${\mathcal{L}}^G\cap H$ .

This theorem allows to identify the set of all phylogenetic mixtures ${\mathcal{D}}_{{\mathcal{M}}_G}$ with ${\mathcal{L}}^G$, which is a vector subspace of $\mathcal{L}$ whose linear equations are easy to describe. In other words, ${\mathcal{L}}^G$ is the smallest linear space containing the data coming from any mixture of trees evolving under the model ${\mathcal{M}}_G$. One can therefore use ${\mathcal{L}}^G$ to select the most suitable model for the given data. This has been studied in [7].

Proof of Theorem 17. For equivariant models we have that $C{V}_T^{{\mathcal{M}}_G}\subset {\mathcal{L}}^G$ for any tree T. Hence, by Lemma 10 and the definition of ${\mathcal{D}}_{{\mathcal{M}}_G}$, ${\mathcal{D}}_{{\mathcal{M}}_G}$ is a vector subspace of ${\mathcal{L}}^G$.

From Corollary 15, we infer the equality ${\mathcal{L}}^G={\mathcal{D}}_{{\mathcal{M}}_G}$ for the models K81, SSM and GMM. For the other two models, JC69 and K80, it remains to prove that there does not exist any hyperplane Π containing ${\mathcal{D}}_{{\mathcal{M}}_G}$ and not containing ${\mathcal{L}}^G$. If such a hyperplane existed, then it would contain all the points of $C{V}_T^{{\mathcal{M}}_G}$ for any tree topology T. It suffices to prove that for these models there are no homogeneous linear polynomials vanishing on all tree topologies, except for the linear equations vanishing on ${\mathcal{L}}^G$. This has been seen in Remark 16(b).

The equality ${\mathcal{D}}_{s{\mathcal{M}}_G}={\mathcal{L}}^G\cap H$ follows immediately from Lemma 10 and the first assertion in the statement of this theorem.

Remark 18. We are indebted to one of the referees of this paper for pointing out that the preceeding result, as well as the second part of Proposition 13, can also be inferred from Proposition 4.9 of [5]: under the assumption that the stabiliser of some state is trivial, Draisma and Kuttler show that the star tree is the smallest algebraic variety containing the tensors τ{X} _G , for pure tensors X(that is, tensors of rank 1). It follows that the set of mixtures on the star tree equals the space ${\mathcal{L}}^G$.

Remark 19. It is not difficult to check that for $\mathcal{M}=$K81, SSM or GMM, ${\mathcal{D}}_{\mathcal{M}}$ coincides with the space of mixtures on the star tree (see also [26], where the same result is proven for a 2-state model). On the contrary, this is not true for JC69 and K80 models because in this case the star tree lies in a smaller linear space as a consequence of the existence of phylogenetic linear equations (see Remark 16(b)).

Equations for the space ${\mathcal{L}}^{\mathit{G}}$

Our goal here is to compute the dimension of ${\mathcal{L}}^G$ for the groups associated to the equivariant models listed in Definition 4, and to list a set of independent linear equations defining this space.

Proposition 20. Using the notations above,

• (i) $\text{dim}{\mathcal{L}}^{\mathtt{\text{SSM}}}=2^{2n-1}$,

• (ii) $dim{\mathcal{L}}^{\mathtt{K81}}=4^{n-1}$,

• (iii) $dim{\mathcal{L}}^{\mathtt{K80}}=2^{2n-3}+2^{n-2}$, and

• (iv) $dim{\mathcal{L}}^{\mathtt{JC69}}=\frac{2^{2n-3}+1}{3}+2^{n-2}$.

Proof. Let $\mathcal{M}$ be any equivariant model. By definition, we know that ${\mathcal{L}}^G$ is the isotypic component of ⊗ⁿW associated to the trivial representation (⊗ⁿW)[ω₁]. Since the dimension of the trivial representation is one, it follows that the dimension of ${\mathcal{L}}^{\mathcal{M}}$ is precisely the multiplicity m₁ (n), i.e. the number of times the trivial representation appears in the decomposition of ⊗ⁿW into isotypic components. This multiplicity m₁ (n) equals (see (5))

$$\begin{array}{l}〈{\chi }^n,{\omega }_1〉=\frac{1}{\left|G\right|}\sum _{g\in G}{\chi }^n\left(g\right){\omega }_1\left(g\right).\end{array}$$

The proof ends by grouping the elements of G in the conjugacy classes of G for SSM, K81, K80, or JC69. Recall that the conjugacy classes of a group G are the disjoint sets of the form C (g) = {h^{− 1}gh : h ∈ G}. If C₁,…,C _s are the conjugacy classes for G, write $\mathcal{C}\left(G\right)=\left(\right|C_1|,\dots ,|C_s\left|\right)$ for the s-tuple of their cardinalities, so that $\sum _{i=1}^s\left|C_i\right|=\left|G\right|$. Recall that χⁿ(g₁) = χⁿ(g₂) whenever g₁ and g₂ lie in the same conjugacy class, so we can represent χⁿby an s-tuple ${\chi }_{\mathcal{C}\left(G\right)}^n=(t_1,\dots ,t_s)$, where t _i = χⁿ(g) for any g ∈ C _i . Thus, we have $m_1\left(n\right)=\frac{1}{\left|G\right|}\sum _{i=1}^s{\chi }^n\left(g_i\right)\left|C_i\right|$, where g _i is any element in the conjugacy class c _i . The result for $\mathcal{M}=\mathtt{\text{SSM}}$, K81, K80, or JC69 follows by applying the Table 1.

Table 1 Details of the conjugacy classes of some permutation groups needed in the proof of Proposition 20

□

Our next goal is to provide a set of independent linear equations for ${\mathcal{L}}^G$. Before stating the main result, let us introduce some useful notation.

Notation 21. We consider the following subsets of $\mathcal{B}=B^n$:

$$\begin{array}{ll}{\mathcal{B}}_0& =\{\mathtt{A}\dots \mathtt{A},\mathtt{C}\dots \mathtt{C},\mathtt{G}\dots \mathtt{G},\mathtt{T}\dots \mathtt{T}\},\\ {\mathcal{B}}_{\mathtt{A}\mathtt{C}\mid \mathtt{G}\mathtt{T}}& ={\{\mathtt{A},\mathtt{C}\}}^n\cup {\{\mathtt{G},\mathtt{T}\}}^n,\\ {\mathcal{B}}_{\mathtt{A}\mathtt{G}\mid \mathtt{C}\mathtt{T}}& ={\{\mathtt{A},\mathtt{G}\}}^n\cup {\{\mathtt{C},\mathtt{T}\}}^n,\\ {\mathcal{B}}_{\mathtt{A}\mathtt{T}\mid \mathtt{C}\mathtt{G}}& ={\{\mathtt{A},\mathtt{T}\}}^n\cup {\{\mathtt{C},\mathtt{G}\}}^n,\text{and}\\ {\mathcal{B}}_2& ={\mathcal{B}}_{\mathtt{A}\mathtt{C}\mid \mathtt{G}\mathtt{T}}\cup {\mathcal{B}}_{\mathtt{A}\mathtt{G}\mid \mathtt{C}\mathtt{T}}\cup {\mathcal{B}}_{\mathtt{A}\mathtt{T}\mid \mathtt{C}\mathtt{G}}.\end{array}$$

The set ${\mathcal{B}}_0$ is composed of all n-words with only one letter and it is contained in ${\mathcal{B}}_{\mathtt{A}\mathtt{C}\mid \mathtt{G}\mathtt{T}}$, ${\mathcal{B}}_{\mathtt{A}\mathtt{G}\mid \mathtt{C}\mathtt{T}}$, and ${\mathcal{B}}_{\mathtt{A}\mathtt{T}\mid \mathtt{C}\mathtt{G}}$. Similarly, ${\mathcal{B}}_2$ is composed of all n-words with two letters at most. It is straightforward to check that $\left|{\mathcal{B}}_{\mathtt{A}\mathtt{C}\mid \mathtt{G}\mathtt{T}}\right|=\left|{\mathcal{B}}_{\mathtt{A}\mathtt{G}\mid \mathtt{C}\mathtt{T}}\right|=\left|{\mathcal{B}}_{\mathtt{A}\mathtt{T}\mid \mathtt{C}\mathtt{G}}\right|=2^{n+1}$ and $\left|{\mathcal{B}}_2\right|=3·2^{n+1}-8$.

We adopt multiplicative notation for n-words, for instance, we write C^lfor the word $\underset{l}{\underbracket{\mathtt{C}\dots \mathtt{C}}}$, and (A^l)(G^m)x_{l + m + 1}…x _n for $\underset{l}{\underbracket{\mathtt{A}\dots \mathtt{A}}}\underset{m}{\underbracket{\mathtt{G}\dots \mathtt{G}}}{x}_{l+m+1}\dots x_n$, where x_{l + m + 1},…,x _n represent any letters.

The main result of this section is the following:

Theorem 22. A set of linearly independent equations${\mathbb{E}}^{\mathcal{M}}$defining${\mathcal{L}}^{\mathcal{M}}$for$\mathcal{M}$=JC69 , K80 , K81 , or SSM is given by

${\mathbb{E}}^{\mathtt{\text{SSM}}}$ : equations p _X= p_{(A T)(CG)X} for all $\mathtt{X}\in \mathcal{B}$ with x₁ ∈ {A,C};

${\mathbb{E}}^{\mathtt{K81}}$ : the equations in ${\mathbb{E}}^{\mathtt{\text{SSM}}}$, and the equations {p _X= p_(AT)(CG)X} for all $\mathtt{X}\in \mathcal{B}$ with x₁ = A;

${\mathbb{E}}^{\mathrm{K80}}$ : the equations in ${\mathbb{E}}^{\mathtt{K81}}$, plus the equations {p _X = p_{( A )( G ) X} } for all $\mathtt{X}\in \mathcal{B}\backslash {\mathcal{B}}_{\mathtt{A}\mathtt{C}\mid \mathtt{G}\mathtt{T}}$ having x₁ = A and satisfying the following condition: if T appears in X, then there is a C in a preceding position;

${\mathbb{E}}^{\mathrm{JC69}}$ : the equations in ${\mathbb{E}}^{\mathrm{K80}}$, together with the equations {p _X = p_{( A T ) X} } for all $\mathtt{X}\in {\mathcal{B}}_{\mathtt{A}\mathtt{C}\mid \mathtt{G}\mathtt{T}}\backslash {\mathcal{B}}_0$ of the form (A^l)(C^m)x_{l + m + 1}…x _n ; plus the equations {p _X = p_{( A C ) X} } and {p _X = p_{( A T ) X} } for all $\mathtt{X}\in \mathcal{B}\backslash {\mathcal{B}}_2$ of the form (A^l)(C^m)x_{l + m + 1}…x _n and satisfying the condition: if T appears in X, then there is a G in a preceding position.

The number of equations added in each case is 2^{2 n − 1}for SSM, 2^{2 n − 2} for K81, 2^{2n − 3}− 2ⁿ⁻² for K80, and $2^{n-1}-1+2(\frac{2^{2n-3}+1}{3}-2^{n-2})$ for JC69.

Before proving this theorem, we explain how these sets of equations were obtained. Notice that a system of linear equations of ${\mathcal{L}}^G$ is given by

$$\begin{array}{l}\left\{p_{g\mathtt{X}}=p_{\mathtt{X}}\mid g\in G,\mathtt{X}\in \mathcal{B}\right\}.\end{array}$$

The role played by the G-orbits on $\mathcal{B}$ becomes apparent. Indeed, the idea is to relate the equations to the orbits of a subgroup of G. To this aim, let H be a subgroup of G and write H \ G = {Hg:g ∈ G} for the set of right cosets of H in G. We consider a transversal of H \ G, i.e. a collection {g₁,…,g_[G:H]} such that $G={\bigsqcup }_{i=1}^{[G:H]}H{g}_i$. Then the orbit of any $\mathtt{X}\in \mathcal{B}$ can be decomposed as

$${\left\{\mathtt{X}\right\}}_G=\bigcup _{i=1,\dots ,[G:H]}{\left\{g_i\mathtt{X}\right\}}_H.$$

(6)

This decomposition establishes the connection between the G-orbits and the H-orbits. In order to obtain a system of equations for ${\mathcal{L}}^G$, once ${\mathbb{E}}^H$ has been computed, it is enough to add the equations involving the permutations in a transversal {g₁ = e,g₂,…,g_[G:H]} of H \ G:

$$\left.\begin{array}{l}{p}_{\mathtt{X}}=p_{g_2\mathtt{X}}\\ p_{\mathtt{X}}=p_{g_3\mathtt{X}}\\ \dots \\ p_{\mathtt{X}}=p_{g_{[G:H]}\mathtt{X}}\end{array}\right\}\text{for all}\mathtt{X}\in \mathcal{B}.$$

Notice that the union in (6) is not necessarily disjoint as it may happen that {g _i X} _H = {g _j X} _H for i ≠ j. In this case, the equality $p_{g_j\mathtt{X}}=p_{g_j\mathtt{X}}$ already holds in the space ${\mathcal{L}}^H$ and does not provide any new restriction. In order to avoid this situation and obtain a minimal set of equations for ${\mathcal{L}}^G$, we request the special conditions on the $\mathtt{X}\in \mathcal{B}$ in the statement of the theorem.

Proof. For each model $\mathcal{M}$, we prove that the corresponding equations are linearly independent and there are as many equations as the codimension of ${\mathcal{L}}^{\mathcal{M}}$. By Proposition 20, the codimension of ${\mathcal{L}}^{\mathcal{M}}$ is 2^{2 n − 1}for SSM, 3· 4^{n − 1}for K81, 7·2²ⁿ⁻³− 2^{n − 2}for K80, and $4^n-\frac{2^{2n-3}+1}{3}-2^{n-2}$ for JC69. In the sequel, we refer to the groups by the name of the equivariant model associated to them.

SSM: As SSM is the group {e,(A T)(C G)}, a set of equations for SSM is {p _X = p_{( A T)( C G ) X} }. Fixing x₁in {A,C} we obtain 2²ⁿ⁻¹linearly independent equations (equations involving different coordinates). The codimension of ${\mathcal{L}}^{\mathtt{\text{SSM}}}$ is equal to 2²ⁿ⁻¹, which coincides with the number of equations given, and thus this set of equations defines ${\mathcal{L}}^{\mathtt{\text{SSM}}}$.

K81: Since a transversal of SSM\K81 is {e,(A C)(G T)}, the hyperplanes p _X = p_{( A T )( C G ) X} contain ${\mathcal{L}}^{\mathtt{K81}}$ but not ${\mathcal{L}}^{\mathtt{\text{SSM}}}$. Moreover, using (6) we see that the orbit {X} _K81 decomposes into the disjoint union of {X} _SSM and {(A C)(G T)X} _SSM for any $\mathtt{X}\in \mathcal{B}$. Therefore, the equations given for K81 involve different coordinates than those in ${\mathbb{E}}^{\mathtt{\text{SSM}}}$. Requiring x₁ = A, we obtain 4ⁿ⁻¹linearly independent new equations. Thus ${\mathbb{E}}^{\mathtt{K81}}$ defines the space ${\mathcal{L}}^{\mathtt{K81}}$ because the number of linearly independent equations provided, 2²ⁿ⁻¹ + 4ⁿ⁻¹= 3·4ⁿ⁻¹, coincides with the codimension of ${\mathcal{L}}^{\mathtt{K81}}$.

K80: The set {e,(A G)} is a transversal of K81\K80. In order to show that the equations provided are linearly independent to those of ${\mathbb{E}}^{\mathtt{K81}}$, we apply (6) to this transversal to obtain {X} _K80 = {X} _K81 ∪{(A G)X} _K81 . If $\mathtt{X}\notin {\mathcal{B}}_{\mathtt{A}\mathtt{G}\mid \mathtt{C}\mathtt{T}}$, then {(A G)X} _K81 and {X} _K81 are disjoint, so each equation p _X = p_{( A )( G ) X} is linearly independent from ${\mathbb{E}}^{\mathtt{K81}}$. The set $\mathcal{B}\backslash {\mathcal{B}}_{\mathtt{A}\mathtt{G}\mid \mathtt{C}\mathtt{T}}$ has cardinal 4ⁿ− 2^{n + 1}and, if $\mathtt{X}\in \mathcal{B}\backslash {\mathcal{B}}_{\mathtt{A}\mathtt{G}\mid \mathtt{C}\mathtt{T}}$, each orbit {X} _K80 has cardinality 8. Therefore, the number of different orbits for $\mathtt{X}\in \mathcal{B}\backslash {\mathcal{B}}_{\mathtt{A}\mathtt{G}\mid \mathtt{C}\mathtt{T}}$ is (4ⁿ− 2^{n + 1})/8 = 2^{2 n − 3}−2^{n − 2}. Moreover, the choice of X’s in $\mathcal{B}\backslash {\mathcal{B}}_{\mathtt{A}\mathtt{G}\mid \mathtt{C}\mathtt{T}}$ with x₁ = A and satisfying “if T appears in X, there is a C in a preceding position” guarantees that we take only one element in each {X} _K80 , and thus we are adding exactly one equation for each of these X^′, and thus we are adding exactly one equation for each of these X^′s. Overall, there are 3·4^{n − 1} + (2^{2 n − 3}−2^{n − 2}) = 7 · 2^{2 n − 3}− 2^{n − 2}linearly independent equations in ${\mathbb{E}}^{\mathrm{K80}}$. This number coincides with the codimension of ${\mathcal{L}}^{\mathrm{K80}}$ and these equations define ${\mathcal{L}}^{\mathrm{K80}}$.

JC69: A transversal of K80\JC69 is {e,(A C),(A T)}, therefore (6) applies to give ${\left\{\mathtt{X}\right\}}_{\mathrm{JC69}}={\left\{\mathtt{X}\right\}}_{\mathrm{K80}}\cup {\left\{\right(\mathtt{A}\mathtt{C}\left)\mathtt{X}\right\}}_{\mathrm{K80}}\cup {\left\{\right(\mathtt{A}\mathtt{T}\left)\mathtt{X}\right\}}_{\mathrm{K80}}$. Summing up, there are if $\mathtt{X}\in {\mathcal{B}}_{\mathtt{A}\mathtt{C}\mid \mathtt{G}\mathtt{T}}\backslash {\mathcal{B}}_0$, then {(A C)X} _K80 = {X} _K80 and {X} _JC69 is the disjoint union of {X} _K80 and {(A T)X} _K80 . As such, each equation p _X = p_{( A T ) X} is linearly independent from ${\mathbb{E}}^{\mathrm{K80}}$. Moreover, if $\mathtt{X}\in {\mathcal{B}}_{\mathtt{A}\mathtt{C}\mid \mathtt{G}\mathtt{T}}\backslash {\mathcal{B}}_0$ is of the form (A^l)(C^m)x_{l + m + 1}…x _n , we have 2^{n − 1}− 1 such equations and they are linearly independent.

if $\mathtt{X}\in \mathcal{B}\backslash {\mathcal{B}}_2$ then the three orbits {(A C)X} _K80 , {(A T)X} _K80 , and {X} _K80 have 8 elements each and are disjoint. Therefore, for these X’s, each equation of type {p _X = p_{( A C ) X} } or {p _X = p_{( A T ) X} } is linearly independent from ${\mathbb{E}}^{\mathrm{K80}}.$ Moreover, as $\mathcal{B}\backslash {\mathcal{B}}_2$ has cardinal 4ⁿ− 3 · 2^{n + 1} + 8 and is covered by these orbits, we have $\frac{4^n-3·2^{n+1}+8}{24}=\frac{1}{3}(2^{2n-3}+1)-2^{n-2}$ different orbits. The restriction to the elements of the form (A^l)(C^m) x_{l + m + 1}… x _n and satisfying that “if T appears in X, there is some G in a preceding position” guarantees that the equations are written only only once for each orbit.

$$7·2^{2n-3}-2^{n-2}+\left(2^{n-1}-1+2\left(\frac{1}{3}(2^{2n-3}+1)-2^{n-2}\right)\right)$$

linearly independent equations in ${\mathbb{E}}^{\mathrm{JC69}}$ that contain ${\mathcal{L}}^{\mathrm{JC69}}$. As this number is equal to the codimension $4^n-\frac{2^{2n-3}+1}{3}-2^{n-2}$ of ${\mathcal{L}}^{\mathrm{JC69}}$, the proof is complete.

All the equalities among orbits used in this proof are summarized in Table 2.

Table 2 Equalities among orbits used in the proof of Theorem 22

Remark 23. The sets of equations of Theorem 22 has been successfully used in [7] for model selection. Although the dimensions of these linear spaces are exponential in n, in practice it is not necessary to consider the full set of equations, but only those containing the patterns observed in the data. This is crucial for the applicability of the method, since the number of different columns in an alignment is really small compared to the dimension of these spaces.

Example 24. As an example, we compute a minimal system of equations for SSM, K81, K80, and JC69 in the case of 3 leaves.

Equations for${\mathcal{L}}^{\mathtt{\text{SSM}}}$: ${\mathbb{E}}^{\mathtt{\text{SSM}}}$ is composed of the following equations:

$$\begin{array}{l}{p}_{\mathtt{A}\mathtt{A}\mathtt{A}}=p_{\mathtt{T}\mathtt{T}\mathtt{T}},p_{\mathtt{A}\mathtt{A}\mathtt{C}}=p_{\mathtt{T}\mathtt{T}\mathtt{G}},p_{\mathtt{A}\mathtt{A}\mathtt{G}}=p_{\mathtt{T}\mathtt{T}\mathtt{C}},\\ p_{\mathtt{A}\mathtt{A}\mathtt{T}}=p_{\mathtt{T}\mathtt{T}\mathtt{A}},p_{\mathtt{A}\mathtt{C}\mathtt{A}}=p_{\mathtt{T}\mathtt{G}\mathtt{T}},p_{\mathtt{A}\mathtt{C}\mathtt{C}}=p_{\mathtt{T}\mathtt{G}\mathtt{G}},\\ p_{\mathtt{A}\mathtt{C}\mathtt{G}}=p_{\mathtt{T}\mathtt{G}\mathtt{C}},p_{\mathtt{A}\mathtt{C}\mathtt{T}}=p_{\mathtt{T}\mathtt{G}\mathtt{A}},p_{\mathtt{A}\mathtt{G}\mathtt{A}}=p_{\mathtt{T}\mathtt{C}\mathtt{T}},\\ p_{\mathtt{A}\mathtt{G}\mathtt{C}}=p_{\mathtt{T}\mathtt{C}\mathtt{G}},p_{\mathtt{A}\mathtt{G}\mathtt{G}}=p_{\mathtt{T}\mathtt{C}\mathtt{C}},p_{\mathtt{A}\mathtt{G}\mathtt{T}}=p_{\mathtt{T}\mathtt{C}\mathtt{A}},\\ p_{\mathtt{A}\mathtt{T}\mathtt{A}}=p_{\mathtt{T}\mathtt{A}\mathtt{T}},p_{\mathtt{A}\mathtt{T}\mathtt{C}}=p_{\mathtt{T}\mathtt{A}\mathtt{G}},p_{\mathtt{A}\mathtt{T}\mathtt{G}}=p_{\mathtt{T}\mathtt{A}\mathtt{C}},\\ p_{\mathtt{A}\mathtt{T}\mathtt{T}}=p_{\mathtt{T}\mathtt{A}\mathtt{A}},p_{\mathtt{C}\mathtt{A}\mathtt{A}}=p_{\mathtt{G}\mathtt{T}\mathtt{T}},p_{\mathtt{C}\mathtt{A}\mathtt{C}}=p_{\mathtt{G}\mathtt{T}\mathtt{G}},\\ p_{\mathtt{C}\mathtt{A}\mathtt{G}}=p_{\mathtt{G}\mathtt{T}\mathtt{C}},p_{\mathtt{C}\mathtt{A}\mathtt{T}}=p_{\mathtt{G}\mathtt{T}\mathtt{A}},p_{\mathtt{C}\mathtt{C}\mathtt{A}}=p_{\mathtt{G}\mathtt{G}\mathtt{T}},\\ p_{\mathtt{C}\mathtt{C}\mathtt{C}}=p_{\mathtt{G}\mathtt{G}\mathtt{G}},p_{\mathtt{C}\mathtt{C}\mathtt{G}}=p_{\mathtt{G}\mathtt{G}\mathtt{C}},p_{\mathtt{C}\mathtt{C}\mathtt{T}}=p_{\mathtt{G}\mathtt{G}\mathtt{A}},\\ p_{\mathtt{C}\mathtt{G}\mathtt{A}}=p_{\mathtt{G}\mathtt{C}\mathtt{T}},p_{\mathtt{C}\mathtt{G}\mathtt{C}}=p_{\mathtt{G}\mathtt{C}\mathtt{G}},p_{\mathtt{C}\mathtt{G}\mathtt{G}}=p_{\mathtt{G}\mathtt{C}\mathtt{C}},\\ p_{\mathtt{C}\mathtt{G}\mathtt{T}}=p_{\mathtt{G}\mathtt{C}\mathtt{A}},p_{\mathtt{C}\mathtt{T}\mathtt{A}}=p_{\mathtt{G}\mathtt{A}\mathtt{T}},p_{\mathtt{C}\mathtt{T}\mathtt{C}}=p_{\mathtt{G}\mathtt{A}\mathtt{G}},\\ p_{\mathtt{C}\mathtt{T}\mathtt{G}}=p_{\mathtt{G}\mathtt{A}\mathtt{C}},p_{\mathtt{C}\mathtt{T}\mathtt{T}}=p_{\mathtt{G}\mathtt{A}\mathtt{A}}.\end{array}$$

Equations for${\mathcal{L}}^{\mathtt{K81}}$: ${\mathbb{E}}^{\mathtt{K81}}$ is formed by ${\mathbb{E}}^{\mathtt{\text{SSM}}}$ and

$$\begin{array}{l}{p}_{\mathtt{A}\mathtt{A}\mathtt{A}}=p_{\mathtt{C}\mathtt{C}\mathtt{C}},p_{\mathtt{A}\mathtt{A}\mathtt{C}}=p_{\mathtt{C}\mathtt{C}\mathtt{A}},p_{\mathtt{A}\mathtt{A}\mathtt{G}}=p_{\mathtt{C}\mathtt{C}\mathtt{T}},\\ p_{\mathtt{A}\mathtt{A}\mathtt{T}}=p_{\mathtt{C}\mathtt{C}\mathtt{G}},p_{\mathtt{A}\mathtt{C}\mathtt{A}}=p_{\mathtt{C}\mathtt{A}\mathtt{C}},p_{\mathtt{A}\mathtt{C}\mathtt{C}}=p_{\mathtt{C}\mathtt{A}\mathtt{A}},\\ p_{\mathtt{A}\mathtt{C}\mathtt{G}}=p_{\mathtt{C}\mathtt{A}\mathtt{T}},p_{\mathtt{A}\mathtt{C}\mathtt{T}}=p_{\mathtt{C}\mathtt{A}\mathtt{G}},p_{\mathtt{A}\mathtt{G}\mathtt{A}}=p_{\mathtt{C}\mathtt{T}\mathtt{C}},\\ p_{\mathtt{A}\mathtt{G}\mathtt{C}}=p_{\mathtt{C}\mathtt{T}\mathtt{A}},p_{\mathtt{A}\mathtt{G}\mathtt{G}}=p_{\mathtt{C}\mathtt{T}\mathtt{T}},p_{\mathtt{A}\mathtt{G}\mathtt{T}}=p_{\mathtt{C}\mathtt{T}\mathtt{G}},\\ p_{\mathtt{A}\mathtt{T}\mathtt{A}}=p_{\mathtt{C}\mathtt{G}\mathtt{C}},p_{\mathtt{A}\mathtt{T}\mathtt{C}}=p_{\mathtt{C}\mathtt{G}\mathtt{A}},p_{\mathtt{A}\mathtt{T}\mathtt{G}}=p_{\mathtt{C}\mathtt{G}\mathtt{T}},\\ p_{\mathtt{A}\mathtt{T}\mathtt{T}}=p_{\mathtt{C}\mathtt{G}\mathtt{G}}.\end{array}$$

Equations for${\mathcal{L}}^{\mathrm{K80}}$: ${\mathbb{E}}^{\mathrm{K80}}$ is formed by ${\mathbb{E}}^{\mathtt{K81}}$ and

$$\begin{array}{l}{p}_{\mathtt{A}\mathtt{A}\mathtt{G}}=p_{\mathtt{G}\mathtt{A}\mathtt{A}},p_{\mathtt{A}\mathtt{C}\mathtt{G}}=p_{\mathtt{G}\mathtt{C}\mathtt{A}},p_{\mathtt{A}\mathtt{C}\mathtt{T}}=p_{\mathtt{G}\mathtt{C}\mathtt{T}},\\ p_{\mathtt{A}\mathtt{G}\mathtt{A}}=p_{\mathtt{G}\mathtt{A}\mathtt{G}},p_{\mathtt{A}\mathtt{G}\mathtt{C}}=p_{\mathtt{G}\mathtt{A}\mathtt{C}},p_{\mathtt{A}\mathtt{G}\mathtt{G}}=p_{\mathtt{G}\mathtt{A}\mathtt{A}}.\end{array}$$

Equations for${\mathcal{L}}^{\mathrm{JC69}}$: ${\mathbb{E}}^{\mathrm{JC69}}$ is formed by ${\mathbb{E}}^{\mathrm{K80}}$ and

$$\begin{array}{l}{p}_{\mathtt{A}\mathtt{A}\mathtt{C}}=p_{\mathtt{T}\mathtt{T}\mathtt{C}},p_{\mathtt{A}\mathtt{C}\mathtt{A}}=p_{\mathtt{T}\mathtt{C}\mathtt{T}},p_{\mathtt{A}\mathtt{C}\mathtt{C}}=p_{\mathtt{T}\mathtt{C}\mathtt{C}},\\ p_{\mathtt{A}\mathtt{C}\mathtt{G}}=p_{\mathtt{C}\mathtt{A}\mathtt{G}},p_{\mathtt{A}\mathtt{C}\mathtt{G}}=p_{\mathtt{T}\mathtt{C}\mathtt{G}}.\end{array}$$

Identifiability of phylogenetic mixtures

In this section we study the identifiability of phylogenetic mixtures. To this end, we use projective algebraic varieties and techniques from algebraic geometry. It is not our intention to give the reader a background on these tools, so we refer to the algebraic geometry book [14] and, more specifically, to [10] for the usage of these techniques in the study of phylogenetic mixtures.

There is a natural isomorphism between the points lying in the hyperplane H considered above, $H=\{p=(p_{\mathtt{A}\dots \mathtt{A}},\dots ,p_{\mathtt{T}\dots \mathtt{T}})\in \mathcal{L}:\sum p_{x_1\dots x_n}=1\}$, and the open affine subset $\{p=[p_{\mathtt{A}\dots \mathtt{A}}:\cdots :p_{\mathtt{T}\dots \mathtt{T}}]:\sum p_{x_1\dots x_n}\ne 0\}$ of ${\mathbb{P}}^{4^n-1}=\mathbb{P}\left(\mathcal{L}\right)$. We use the notation [p _{A … A} :⋯:p _{T … T} ] for projective coordinates (in contrast to $(p_{\mathtt{A}\dots \mathtt{A}},\dots ,p_{\mathtt{T}\dots \mathtt{T}})$ used for affine coordinates). The projective phylogenetic variety$\mathbb{P}{V}_T^{\mathcal{M}}$ associated to a phylogenetic tree T is the projective closure in $\mathbb{P}\left(\mathcal{L}\right)$ of the image of the stochastic parameterization ${\varphi }_T^{\mathcal{M}}$ defined above. That is, it is the smallest projective variety in $\mathbb{P}\left(\mathcal{L}\right)$ containing $\text{Im}{\varphi }_T^{\mathcal{M}}$ via the above isomorphism.

In what follows, we explain the relationship between this new variety and $C{V}_T^{\mathcal{M}}$ and $V_T^{\mathcal{M}}$. By Remark 7, it becomes clear that $C{V}_T^{\mathcal{M}}$ equals the affine cone over the projective phylogenetic variety $\mathbb{P}{V}_T^{\mathcal{M}}$ (for the general Markov model, see also [23], Proposition 1). This implies that $dimC{V}_T^{\mathcal{M}}=dim\mathbb{P}{V}_T^{\mathcal{M}}+1$, and if $p=(p_{\mathtt{A}\mathtt{\dots }\mathtt{A}},\dots ,p_{\mathtt{T}\mathtt{\text{…}}\mathtt{T}})$ belongs to $C{V}_T^{\mathcal{M}}$, then q := p _{A … T} :⋯:p _{T … T} belongs to $\mathbb{P}{V}_T^{\mathcal{M}}$. Moreover, if $\lambda :=\sum p_{x_1\dots x_n}$ is not zero, then $(\frac{p_{\mathtt{A}\dots \mathtt{A}}}{\lambda },\dots ,\frac{p_{\mathtt{T}\dots \mathtt{T}}}{\lambda })$ is a point in the affine stochastic phylogenetic variety $V_T^{\mathcal{M}}$.

Before defining identifiability of mixtures, we consider the following construction of projective algebraic varieties.

Definition 25. Given two projective varieties $X,Y\subset {\mathbb{P}}^m$, the join of X and Y, X ∨ Y, is the smallest variety in ${\mathbb{P}}^m$ containing all lines $\overline{\mathit{xy}}$ with x ∈ X, y∈Y, and x ≠ y(see [14], 8.1 for details). Similarly, one defines the join of projective varieties$X_1,\dots ,X_h\subset {\mathbb{P}}^m$, ∨i = 1h X _i , as the smallest subvariety in ${\mathbb{P}}^m$ containing all the linear varieties spanned by $x_1,\dots ,x_h$ with x _i ∈ X _i and x _i ≠ x _j . It is known that

$$\begin{array}{l}dim\left({\vee }_{i=1}^h{X}_i\right)\le min\{\sum _{i=1}^{h}dim(X_i)+h-1,m\}.\end{array}$$

The right hand side of this inequality is usually known as the expected dimension of ∨i = 1h X _i .

For instance, if we consider the join ${\vee }_{i=1}^h\mathbb{P}{V}_{T_i}^{\mathcal{M}}$ for certain tree topologies T _i on the leaf set [n] and a given evolutionary model $\mathcal{M}$, then there is a (dominant rational) map

$$\mathbb{P}{V}_{T_1}^{\mathcal{M}}\times \mathbb{P}{V}_{T_2}^{\mathcal{M}}\times \times \mathbb{P}{V}_{T_h}^{\mathcal{M}}\times {\mathbb{P}}^{h-1}--\to {\vee }_{i=1}^h\mathbb{P}{V}_{T_i}^{\mathcal{M}}\subset \mathbb{P}\left(\mathcal{L}\right),$$

(7)

which is the projective closure of the parameterization ${\varphi }_{T_1}\vee \dots \vee {\varphi }_{T_h}$ defined by

$$\begin{array}{lll}\mathit{Pa}{r}_{s\mathcal{M}}\left(T_1\right)\times \dots \times \mathit{Pa}{r}_{s\mathcal{M}}\left(T_h\right)\times \mathrm{\Omega }& \to & \mathcal{L}\\ \hfill \left({\xi }_1,\dots ,{\xi }_h,\mathbf{a}\right)& \mapsto & \sum _j{a}_i{\varphi }_{T_i}^{\mathcal{M}}\left({\xi }_i\right).\end{array}$$

Here, $\mathrm{\Omega }=\{\mathbf{a}=(a_1,\dots ,a_h)\mid \sum _i{a}_i=1\}$ is isomorphic to an affine open subset of ${\mathbb{P}}^{h-1}$. In this setting, an h-mixture on {T₁,…,T _h } corresponds to a point in the variety ${\vee }_{i=1}^h\mathbb{P}{V}_{T_i}^{\mathcal{M}}$. We will use this algebraic variety to study the identifiability of phylogenetic mixtures.

When considering unmixed models $\mathcal{M}$ on trivalent trees on n taxa, generic identifiability of the tree topology is equivalent to the projective varieties $\mathbb{P}{V}_T^{\mathcal{M}}$ and $\mathbb{P}{V}_{T^{\prime }}^{\mathcal{M}}$ being different when T ≠ T^′(see [31]). The identifiability of the continuous parameters must take into account the possibility of permuting the labels of the states at the interior nodes, as such permutations give rise to the same joint distribution at the leaves. In the language of algebraic geometry, generic identifiability of the continuous parameters of the model implies that the map ${\varphi }_T^{\mathcal{M}}$ is generically finite (i.e. the preimage of a generic point is a finite number of points; see [31]). In this case, the fiber dimension Theorem ([14], Theorem 11.12) applies and we have that $dim\mathbb{P}{V}_T^{\mathcal{M}}$ is equal to the number of stochastic parameters of the model, $dim\mathit{Pa}{r}_{s\mathcal{M}}\left(T\right)$. Therefore, if the continuous parameters are generically identifiable for the unmixed trees under $\mathcal{M}$, then the dimension of the variety $\mathbb{P}{V}_T^{\mathcal{M}}$ is the same for all trivalent tree topologies on n taxa. This dimension is denoted by $s_{\mathcal{M}}\left(n\right)$.

Example 26. The tree topologies and the continuous parameters are generically identifiable for the unmixed equivariant models JC69, K80, K81, SSM, and GMM on trees with any number of leaves (see [9] and [6], Corollary 3.9).

From now on we only consider trees without nodes of degree 2, so that the number of free stochastic parameters on a phylogenetic tree on n taxa under $\mathcal{M}$ is $\le s_{\mathcal{M}}\left(n\right)$.

We recall the definition of generic identifiability of the tree topologies on h-mixtures (see [10]).

Definition 27. The tree topologies on h-mixtures under $\mathcal{M}$ are generically identifiable if for any set of trivalent tree topologies {T₁,…,T _h } and a generic choice of $({\xi }_1,\dots {\xi }_h,\mathbf{a})\in \mathit{Pa}{r}_{s\mathcal{M}}\left(T_1\right)\times \dots \times \mathit{Pa}{r}_{s\mathcal{M}}\left(T_h\right)\times \mathrm{\Omega }$, the equality

$${\varphi }_{T_1}\vee \dots \vee {\varphi }_{T_h}({\xi }_1,\dots {\xi }_h,\mathbf{a})={\varphi }_{T_1^{\prime }}\vee \dots \vee {\varphi }_{T_h^{\prime }}({\xi }_1^{\prime },\dots {\xi }_h^{\prime },{\mathbf{a}}^{\prime }),$$

for tree topologies $\{T_1^{\prime },\dots ,T_h^{\prime }\}$ and $({\xi }_1^{\prime },\dots {\xi }_h^{\prime },{\mathbf{a}}^{\prime })\in \mathit{Pa}{r}_{s\mathcal{M}}\left(T_1^{\prime }\right)\times \dots \times \mathit{Pa}{r}_{s\mathcal{M}}\left(T_h^{\prime }\right)\times \mathrm{\Omega }$ implies

$$\{T_1,\dots ,T_h\}=\{T_1{,}^{\prime }\dots ,T_h^{\prime }\}.$$

In terms of algebraic varieties this is equivalent to saying that the variety ${\vee }_{i=1}^h\mathbb{P}{V}_{T_i}^{\mathcal{M}}$ is not contained in ${\vee }_{i=1}^h\mathbb{P}{V}_{T_i^{\prime }}^{\mathcal{M}}$ and vice versa.

The tree topologies are the discrete parameters of h-mixtures. When considering the continuous parameters of h-mixtures, the above mentioned label-swapping can be disregarded. We give the following definition according to [12].

Definition 28. The continuous parameters of h-mixtures on $T_1,\dots ,T_h$ under an evolutionary model $\mathcal{M}$ are generically identifiable if, for a generic choice of stochastic parameters $({\xi }_1,\dots ,{\xi }_h,\mathbf{a})$, the equality

$${\varphi }_{T_1}\vee \dots \vee {\varphi }_{T_h}({\xi }_1,\dots {\xi }_h,\mathbf{a})={\varphi }_{T_1}\vee \dots \vee {\varphi }_{T_h}({\xi }_1^{\prime },\dots {\xi }_h^{\prime },{\mathbf{a}}^{\prime })$$

for stochastic parameters $\left({\xi }_1^{\prime },\dots ,{\xi }_h^{\prime },{\mathbf{a}}^{\prime }\right)$ implies that there is a permutation $\sigma \in {\mathfrak{S}}_h$ such that $\sigma ·(T_1,\dots ,T_h)=(T_1,\dots ,T_h)$, ${\xi }_i^{\prime }={\xi }_{\sigma \left(i\right)}$, and $a_i^{\prime }=a_{\sigma \left(i\right)}$ for $i=1,\dots r$. In other words, we only allow swapping of the continuous parameters when at least two tree topologies coincide.

Definition 29. An h-mixture under a model $\mathcal{M}$ is said to be identifiable if both its tree topologies and its continuous parameters are generically identifiable.

In terms of algebraic varieties, generic identifiability of continuous parameters on h-mixtures implies that the generic fibers (i.e. preimages of generic points) of the map ${\varphi }_{T_1}\vee \dots \vee {\varphi }_{T_h}$ are finite. In this case, the fiber dimension theorem applied to (7) (cf. [14], Theorem 11.12) gives

$$\begin{array}{l}dim\left({\vee }_{i=1}^h\mathbb{P}{V}_{T_i}\right)=\sum _{i=1}^{h}dim\left(\mathbb{P}{V}_{T_i}\right)+h-1.\end{array}$$

The following result demonstrates the need for careful inspection of identifiability of mixtures with many components (i.e. large values of h).

Theorem 30. Let [n] be a set of taxa and $\mathcal{M}$ be an evolutionary model for which the continuous parameters are generically identifiable on trivalent (unmixed) trees. In addition, let $s_{\mathcal{M}}\left(n\right)$ be the dimension of $\mathbb{P}{V}_T^{\mathcal{M}}$ for any trivalent tree T, and set $h_0\left(n\right):=\frac{dim{\mathcal{D}}_{\mathcal{M}}}{s_{\mathcal{M}}\left(n\right)+1}$. Then the h-mixtures of trees on [n] evolving under $\mathcal{M}$ are not identifiable for h ≥ h₀(n).

Remark 31. Note that, in the above definition of h₀(n), $dim{\mathcal{D}}_{\mathcal{M}}$ also depends on n.

Corollary 32. Let [n] be a set of taxa and $\mathcal{M}$ be one of the equivariant models JC69, K80, K81, SSM, or GMM. Then the phylogenetic h-mixtures under these models are not identifiable for h ≥ h₀(n), where

$$h_0\left(n\right)=\left\{\begin{array}{l}\frac{4^n}{12(2n-3)+4},\text{if}\mathcal{M}=\mathtt{\text{GMM}},\\ \frac{2^{2n-1}}{6(2n-3)+2},\text{if}\mathcal{M}=\mathtt{\text{SSM}},\\ \frac{4^{n-1}}{3(2n-3)+1},\text{if}\mathcal{M}=\mathtt{\text{K81}},\\ \frac{2^{2n-3}+2^{n-2}}{2(2n-3)+1},\text{if}\mathcal{M}=\mathtt{\text{K80}},\\ \frac{2^{2n-3}+3·2^{n-2}+1}{3(2n-2)},\text{if}\mathcal{M}=\mathtt{\text{JC69}}.\end{array}\right.$$

Proof. Theorem 17 shows that ${\mathcal{L}}^{\mathcal{M}}={\mathcal{D}}_{\mathcal{M}}$ and Proposition 20 gives the dimension of ${\mathcal{L}}^{\mathcal{M}}$ in each case. Next, we calculate: s _GMM (n) = 12(2n−3) + 3, s _SSM (n) = 6(2n−3) + 1, s _K81 (n) = 3(2n−3), s _K80 (n) = 2(2n−3), and s _JC69 (n) = 2n−3. Applying Theorem 30, we conclude the proof. □

Example 33. Consider the Kimura 3-parameter model K81 on n = 4 taxa. For any h ≥ 4, phylogenetic h-mixtures are not identifiable by Corollary 32. We are not aware of any result proving that mixtures of 2 or 3 different tree topologies under this model are identifiable (either for the tree parameters or for the continuous parameters).

Example 34. Consider the Jukes-Cantor model JC69 on n = 4 taxa. Then Corollary 32 tells us that for h ≥ 3, h-mixtures are not identifiable. Therefore, for this particular model on four taxa the cases in which the identifiability holds are known: the tree and the continuous parameters are generically identifiable for the unmixed model; the tree parameters are generically identifiable for 2-mixtures ([10], Theorem 10); the continuous parameters are generically identifiable for 2-mixtures on different tree topologies and not identifiable for the same tree topology ([10], Theorem 23); neither the continuous parameters nor the tree topologies are generically identifiable for mixtures with more than two components (Corollary 32).

Proof of Theorem 30. Let $\mathit{edim}\left(h\right):=h{s}_{\mathcal{M}}\left(n\right)+h-1$. Then the variety ${\vee }_{i=1}^h\mathbb{P}{V}_{T_i}$ has dimension ≤ edim(h). Indeed, as ${\vee }_i{\varphi }_{T_i}$ is a parameterization of an open subset of ${\vee }_{i=1}^h\mathbb{P}{V}_{T_i}$, then the dimension of ${\vee }_{i=1}^h\mathbb{P}{V}_{T_i}$ is less than or equal to $\sum dim\mathbb{P}{V}_{T_i}+h-1$. Moreover, the dimension of $\mathbb{P}{V}_{T_i}$ is equal to $s_{\mathcal{M}}\left(n\right)$ if T _i is trivalent (since the continuous parameters for the unmixed models under consideration are generically identifiable) and is less than $s_{\mathcal{M}}\left(n\right)$ for non-trivalent trees. Therefore, $dim\left({\vee }_{i=1}^h\mathbb{P}{V}_{T_i}\right)\le \mathit{edim}\left(h\right)$.

If all T _i are trivalent trees, then $\sum dim\mathbb{P}{V}_{T_i}+h-1=\mathit{edim}\left(h\right)$ and, therefore, $dim\left({\vee }_{i=1}^h\mathbb{P}{V}_{T_i}\right)<\mathit{edim}\left(h\right)$ if and only if $dim\left({\vee }_{i=1}^h\mathbb{P}{V}_{T_i}\right)<\sum dim\mathbb{P}{V}_{T_i}+h-1$. Moreover, by the fiber dimension theorem applied to $\vee {\varphi }_{T_i}$, the equality of dimensions holds if and only if the generic fiber of $\vee {\varphi }_{T_i}$ has dimension 0. In particular, if $dim\left({\vee }_{i=1}^h\mathbb{P}{V}_{T_i}\right)<\mathit{edim}\left(h\right)$, then the continuous parameters of this phylogenetic mixture are not identifiable.

As $h_0\left(n\right)=\frac{dim{\mathcal{D}}_{\mathcal{M}}}{s_{\mathcal{M}}\left(n\right)+1}$, we have that $\mathit{edim}\left(h_0\right(n\left)\right)=h_0\left(n\right)\left(s_{\mathcal{M}}\right(n)+1)-1=dim{\mathcal{D}}_{\mathcal{M}}-1$. Now, we fix an $h\in \mathbb{N}$ with h ≥ h₀(n), so that $\mathit{edim}\left(h\right)\ge dim\left({\mathcal{D}}_{\mathcal{M}}\right)-1$.

There are two possible scenarios:

• (a) For any set of tree topologies $\{T_1,\dots ,T_h\}$, the dimension of ${\vee }_{i=1}^h\mathbb{P}{V}_{T_i}$ is less than $dim\left({\mathcal{D}}_{\mathcal{M}}\right)-1$.

• (b) There exists a set of tree topologies $\{T_1,\dots ,T_h\}$ for which $\text{dim}\left({\vee }_{i=1}^h\mathbb{P}{V}_{T_i}\right)=\text{dim}\left({\mathcal{D}}_{\mathcal{M}}\right)-1.$

Case (a) implies that the dimension of ${\vee }_{i=1}^h\mathbb{P}{V}_{T_i}$ is less than edim(h) for any set of trivalent tree topologies $\{T_1,\dots ,T_h\}$. Based on the conclusions drawn above, this implies that the continuous parameters are not generically identifiable.

In case (b), ${\vee }_{i=1}^h\mathbb{P}{V}_{T_i}$ coincides with $\mathbb{P}\left({\mathcal{D}}_{\mathcal{M}}\right)$. Indeed, ${\vee }_{i=1}^h\mathbb{P}{V}_{T_i}$ is contained in $\mathbb{P}\left({\mathcal{D}}_{\mathcal{M}}\right)$, both varieties are irreducible, and $\text{dim}\left({\vee }_{i=1}^h\mathbb{P}{V}_{T_i}\right)=\text{dim}\left({\mathcal{D}}_{\mathcal{M}}\right)-1=\text{dim}\left(\mathbb{P}\right({\mathcal{D}}_{\mathcal{M}}\left)\right)$, which implies that both varieties coincide. In particular, any h-mixture (which is a point in $\mathbb{P}\left({\mathcal{D}}_{\mathcal{M}}\right)$) would be contained in ${\vee }_{i=1}^h\mathbb{P}{V}_{T_i}$, and therefore the topologies are not generically identifiable.

Remark 35. The negative result of Theorem 30 should be complemented with the following positive result of Rhodes and Sullivant in [12]: if $\mathcal{M}$=GMM and one restricts to h-mixtures on the same trivalent tree topology T, then the tree topology and the continuous parameters are generically identifiable if $h<4^{\frac{n}{4}-1}$.

Conclusions

In this paper, we have dealt with the space of phylogenetic mixtures for evolutionary equivariant models. We have shown that for the case of the Jukes-Cantor model, the Kimura models with two or three parameters, the strand symmetric model and the general Markov model, this linear space is defined by the set of linear equations satisfied by the distributions of the patterns at the leaves of a tree that evolves under that model. It follows that this space completely characterizes the model. The use of tools from group theory and group representation theory played a major role, and allowed us to design a procedure to produce minimal systems of equations for these spaces and for any number of taxa. This procedure has been implemented successfully in a new method for model selection in phylogenetics based on linear invariants (see [7]), which is available online at http://genome.crg.es/cgi-bin/phylo_mod_sel/AlgModelSelection.pl.

In the last part of the paper, we proved new results concerning the identifiability of phylogenetic mixtures. Namely, we provided an upper bound for the number of components (classes) of a mixture so that the identifiability of both the continuous and the discrete parameters is still possible.