Modular Flavor Symmetry on Magnetized Torus

We study the modular invariance in magnetized torus models. Modular invariant flavor model is a recently proposed hypothesis for solving the flavor puzzle, where the flavor symmetry originates from modular invariance. In this framework coupling constants such as Yukawa couplings are also transformed under the flavor symmetry. We show that the low-energy effective theory of magnetized torus models is invariant under a specific subgroup of the modular group. Since Yukawa couplings as well as chiral zero modes transform under the modular group, the above modular subgroup (referred to as modular flavor symmetry) provides a new type of modular invariant flavor models with $D_4 \times \mathbb{Z}_2$, $(\mathbb{Z}_4 \times \mathbb{Z}_2) \rtimes \mathbb{Z}_2$, and $(\mathbb{Z}_8 \times \mathbb{Z}_2) \rtimes \mathbb{Z}_2$. We also find that conventional discrete flavor symmetries which arise in magnetized torus model are non-commutative with the modular flavor symmetry. Combining both two symmetries we obtain a larger flavor symmetry, where the conventional flavor symmetry is a normal subgroup of the whole group.


Introduction
The origin of the flavor structure of the quarks and leptons is a longstanding problem. Discrete flavor symmetry is an attractive candidate answer for the flavor puzzle especially for the neutrino sector. For instance, small θ 13 and large θ 23 might imply the tri-bimaximal mixing [1], and such a characteristic pattern can be originated from discrete symmetry [2,3,4]. For review, see [5,6] and references there in. 1 Modular invariant flavor model is a new hypothesis proposed for solving the flavor puzzle [8,9], which assumes that the action is invariant under the modular group Γ = P SL(Z) = SL(2, Z)/Z 2 . The most distinct feature of this framework is that not only the fields, such as the leptons and the Higgs field, but also the coupling parameters are transformed under the modular group. More precisely they form representation of quotient groups of the modular group: Γ N = Γ/Γ(N). Γ N is called principal congruence subgroup of Γ. The experimental values corresponding to the lepton sectors: the masses of charged leptons, neutrino mass-square differences, three mixing angles, and the CPphase can be reproduced in models with modular symmetries of Γ 2 ∼ = S 3 [10,11,12], Γ 3 ∼ = A 4 [12,13,14,15,16], Γ 4 ∼ = S 4 [17,18], and Γ 5 ∼ = A 5 [19]. Modular symmetry is also applied to other physics beyond the standard model such as leptogenesis and inflation [20,21,22,23], and relationships between generalized CP symmetry [24,25] and the modular symmetry are also pointed out [26,27,28]. Modular symmetry is motivated by string compactifications. So far the modular symmetries were investigated in the heterotic string on orbifolds [29,30,31,32,33], and in the D-brane modes [34,35,36,37]. The situation is different in the case of type II superstring with magnetic flux [38]. Kähler potential of type IIB superstring implies that chiral superfield has modular weights [39]. Zero mode's profiles of bulk fields have also been investigated using the 4-dimensional effective action compactified on torus with magnetic flux [40]. Yukawa couplings are then obtained through the overlap integrals of the zero mode wave functions. These results have been used to investigate the property of the modular transformation for each component [41,42,43,44], and it is found that the Yukawa couplings as well as the chiral zero modes form a representation of the modular group. However it still remains unclear whether the full effective action including the Yukawa term is modular invariant. The purpose of this paper is to study modular invariance of the effective action of the magnetized torus model in a systematic way based on the fundamental generators S and T of the modular transformation. We show that although the effective action is not invariant under the modular group, it is invariant under its specific subgroup. The generators of the Yukawa invariant modular subgroup form a new type of flavor symmetry referred to as modular flavor symmetry, such as Z 2 , D 4 × Z 2 , (Z 4 × Z 2 ) ⋊ Z 2 , and (Z 8 × Z 2 ) ⋊ Z 2 depending on the value of magnetic fluxes. The modular flavor symmetry is non-commutative with conventional discrete flavor symmetries, e.g., ∆ (27), which appear if the greatest common divisor of generation numbers of matter fields g is greater than 1 [45]. Combing these two groups, we obtain a larger flavor symmetry, where the conventional discrete flavor group is a normal subgroup of the whole group. In other words the modular group is interpreted as a subgroup of the automorphism of the conventional flavor group. It may imply the relationships between the modular group and the generalized CP symmetry.
This paper is organized as follows. In section 2, we introduce modular symmetry. In section 3, we review the zero mode profiles of magnetized torus. We show how the wave functions and Yukawa couplings transform under the modular group. In section 4, we study modular transformation of the Yukawa term. We then investigate the modular flavor symmetry as the modular subgroup, under which the Yukawa term is invariant. The group structure of modular flavor symmetry is also analyzed. In section 5 we consider modular transformation and flavor symmetry simultaneously. We will show that they are non-commutative and they form a larger flavor group. Section 6 is devoted to the conclusion.

Modular Symmetry
In this section we introduce modular symmetry [8] and develop our notation.
The action of chiral superfields is determined by two functions: Kähler potential K and superpotential W . Using these two functions, the action is given by where Φ i denotes a chiral superfield and τ is a complex parameter, i.e., modulus. We assume W is a holomorphic function of τ and Φ i , and K is real. Modular symmetry is the invariance of the action under modular transformation. Let γ is an element of SL(2, Z). Modular transformation of τ under γ is given by where a, b, c, d are integers satisfying ad − bc = 1. Since the action of γ and −γ are the same, the modular transformation group Γ is isomorphic to P SL(2, Z) = SL(2, Z)/Z 2 . The modular group is generated by two generators: and they correspond to the SL(2, Z) elements as Thus, modular invariance is equivalent to invariance under these two generators.
In order to construct modular invariant action, we introduce a holomorphic function known as modular form. Modular forms are characterized by two parameters: weight k and level N. The modular group of level N is a subgroup of the modular group given by and modular forms f of weight k and level N are holomorphic functions of τ , which transform as . Let f 1 (τ ) and f 2 (τ ) are modular forms of weight k and level N, then f 1 (τ ) + f 2 (τ ) is also a modular form of weight k and level N. Hence set of the modular forms of weight k and level N forms a vector space. This space is denoted by Mod k . If f (τ ) is a modular form of weight k and level N, f (γτ ) is also a modular form of weight k and level N. This relation holds even if γ / ∈ Γ(N). Hence modular transformation of the modular forms can be written as where f i is the basis of Mod k . Modular forms are classified by the irreducible representations of Γ N . Γ N is called principal congruence subgroup. Γ N is a normal subgroup of Γ. Γ N is a non-abelian finite group if N ≤ 5: Γ 2 = S 3 , Γ 3 = A 4 , Γ 4 = S 4 , Γ 5 = A 5 (and Γ 1 is a trivial group) [9]. Above non-abelian groups have been used for non-abelian flavor symmetries, and this is why modular symmetry is attractive for particle phenomenology.
It is obvious that the product of two modular forms of different weights k 1 and k 2 , and the same level N, is a modular form of weight k 1 + k 2 and level N. Thus, the set of modular forms of all weights and level N forms a ring. It is known as modular ring denoted by To construct modular invariant action, we need modular transformations for chiral superfields. We assume that each chiral superfield Φ i is a modular form of weight k i and level N, which transforms as under the modular group. A modular invariant Kähler potential is given by where Im τ transforms as Im τ → |cτ + d| −2 Im τ under the modular group and it cancels the prefactor of (2.8). This form of the Kähler potential is obtained from dimensional reduction of superstring effective theory. Construction of modular invariant superpotential is more complicated. We expand the superpotential W as We assume the coupling constant In the next section, we consider magnetized torus model. In the following analysis we use canonically normalized chiral fields. As we will see later, the modular invariance of the kinetic term (Kähler potential) is trivial as long as canonically normalized fields are used, while they are not modular forms. The modular invariance of the low energy effective theory is investigated from the Yukawa interaction term (superpotential).

Modular transformation in SYM theory on torus
Let τ is a complex number satisfying Im τ > 0. A lattice L generated by (1, τ ) is defined by L = {n + mτ ∈ C | ∀n, ∀m ∈ Z} A torus is defined by C/L. Since the lattice generated by (1, τ ) and (aτ + b, cτ + d) are equivalent if ad − bc = 1, the modular group is symmetry of a torus. τ is interpreted as the complex structure of torus. Thus natural origin of modular symmetric theories is a higher dimensional theory compactified on a torus or its orbifold. Indeed it is shown that effective action of heterotic orbifolds is modular invariant [33]. In this paper, we study modular invariance of 6-dimensional SYM with SU(N) compactified on a 2 dimensional torus. This model is known as magnetized torus, and it is the low energy effective theory of type IIB superstring [38]. Turning on background magnetic fluxes on the torus, the gauge group is broken to direct product of its subgroup: SU(N) → SU(N 1 ) × ... × SU(N ℓ ). We assume N = N 1 + · · ·+ N ℓ in this paper, i.e., abelian Wilson line. Such backgrounds break not only the gauge group, but also higher dimensional supersymmetry, and 4-dimensional N = 1 super Yang-Mills theory is realized as effective theory. This property is certainly attractive for phenomenological purpose. This model might be the origin of the Standard Model [46,47,48].
To obtain the effective theory, we calculate mode expansion of bulk fields. 4-dimensional chiral superfields originate from the off-diagonal components of the gauginos. After breaking the gauge group, they become bi-fundamental matter fields Φ ij , which transform as (N i ,N j ) under SU(N i ) × SU(N j ). We briefly review the derivation of the zero mode wave function of the Φ ij . We consider the equation of motion for fermionic component of Φ ij . Wave functions of its scalar component is the same as that of the fermion unless 4-dimensional supersymmetry is broken. We also review modular transformation of the zero modes and Yukawa couplings [41,42,43,44].
The 6-dimensional fields Φ is expanded by wave functions on the compact space, We concentrate on the zero mode wave functions since we investigate modular invariance of low energy effective theory. The zero mode equation for the fermionic components of Φ ij is written as z is the complex coordinate of the torus, ζ is Wilson line, and ∂ is the partial derivative in terms of z. m i , m j are integer magnetic fluxes, which are given by The boundary conditions for the wave functions depend on the value of the magnetic flux. They are summarized as the following two equations: for negative M. j runs from 0 to |M| − 1 for the both cases. Thus we have |m i − m j | replicas of zero modes for each Φ ij . This is the origin of the generations of the quarks and the leptons [46,47,48] . ϑ α β (z, τ ) is the Jacobi theta function: Since the Jacobi theta function can not be well-defined if Im τ ≤ 0, ψ + have the normalizable solutions only when M > 0, and ψ − becomes normalizable only when M < 0. Hence chiral theory is realized. Using the area of the torus A, a normalization factor N is calculated as The action of γ on the zero mode wave function is defined as where ad − bc = 1 [49]. It is easily checked that anti-holomorphic part of ψ + and holomorphic part of ψ − are not changed by the modular transformation. Since the Dirac operator includes only∂ for ψ + and ∂ for ψ − , the wave function ψ ′ also satisfies the original zero mode Dirac equation Dψ ′ = 0 for any γ ∈ SL(2, Z). Indeed substituting ψ ′ + to ψ + in (3.2), we obtain The same relation holds for ψ − . However, the boundary conditions (3.4) and (3.5) are not always satisfied. Define new holomorphic function f (z) by The boundary conditions for the wave function are reinterpreted to the conditions for f (z). (3.4) and (3.5) are equivalent to On the other hand, the zero mode wave functions (3.6) and (3.7) imply that Thus the boundary conditions are satisfied only when Mcd and Mab are even. When M is even, these conditions are satisfied for all a, b, c, d, and the action of γ is well defined. When M is odd, the action of γ is not consistent with the boundary conditions if ab or cd is odd. For odd M, however, it is found that a subgroup such that ab and cd are even is consistent with the boundary conditions. This subgroup is called Γ 1,2 [49]: Now we can define modular transformation (or transformation under Γ 1,2 ) of the matter fields. We summarize their results. Let M be a positive integer. Then, the transformation of the wave function under S is given by In the second raw, we use modular transformation of Jacobi theta function and the Poisson resummation formula If M is even, the modular transformation of the wave function under T is given as Since Γ is generated by S and T , we obtain the modular transformation of the chiral zero modes for even M. If M is odd, as shown before, we consider modular transformation of the subgroup Γ 1,2 . Since all the elements of Γ 1,2 are generated by S and T 2 , we consider the modular transformation of the zero modes under T 2 , which is calculated as In the case of negative M, modular transformation is given as the complex conjugate of the one for ψ . We introduce a matrix representation for S and T as M . ρ M (S) and ρ M (T ) are non-commutative with each other, and they generate a non-abelian finite group. If M is odd, we consider T 2 instead of T and its matrix representation is given as The matrix representation for negative M is given as the complex conjugate of the one for positive M: We note that the modular transformation given by ρ M (S) and ρ M (T ) is a unitary transformation among the zero mode wave functions. We consider the modular transformation of the Yukawa couplings. 4-dimensional effective couplings are calculated by overlap integrals among the zero mode wave functions. Yukawa couplings of magnetized torus are given by [40] where we assume that M 1 and M 2 are positive and M 3 is negative for definiteness.
Substituting the zero mode wave functions to (3.27), we obtain Yukawa couplings: where the Kronecker delta is defined modulo M 3 , which means δ k,i+j+M 1 m = 1 if and only if k = i + j + M 1 m mod M 3 . The index i runs from 0 to M 1 − 1, j runs from 0 to M 2 − 1, and k runs from 0 to |M 3 | − 1. From Eq. (3.28), the action of S and T on the Yukawa couplings can be read off as and where we use the fact that When the greatest common divisor of M 1 , M 2 , and |M 3 | is 1, the Yukawa couplings can be written in a simpler form: In this case, modular transformation is given by and Therefore the Yukawa couplings form a representation of the modular group. It is shown here that the modular transformation of the Yukawa couplings is given as a linear combination of the original Yukawa couplings. This is because the Yukawa couplings are given by overlap integral of the zero modes, so that the modular transformation of the Yukawa couplings is given by a tensor product of the modular transformation of each zero mode. Thus they form a representation of the modular group. In fact the modular transformation of the Yukawa couplings given in Eqs. (3.29) and (3.30) is equivalent to the following tensor representation which will be used for the analysis of the modular invariance of the Yukawa term in the next section.
In what follows we ignore overall U(1) phases such as e − πi 4 which appear in the modular transformations for the matter fields and the Yukawa couplings, since it can always be rotated away by field redefinition.

Modular Flavor Symmetry on Magnetized Torus
In the previous section, we show how the modular group acts on the chiral zero modes and Yukawa couplings. Now we are prepared to consider modular invariance of the effective action. The modular transformation of (3.16) and (3.19) are unitary. Thus, modular invariance of the kinetic term is trivial, since the chiral superfield in the kinetic term is canonically normalized. We investigate the modular symmetry of the Yukawa term: where φ j,M k denotes the 4-dimensional chiral field in Eq. (3.1). Using the tensor representation we obtain the general modular transformation of the Yukawa term by g ∈ Γ as Here the overall phases are ignored. We define a subset of the modular transformation which satisfies the following condition . We see that with this definition the modular transformation of the Yukawa term can be regarded as a simultaneous modular transformation of the Yukawa couplings and the chiral fields, where the modular transformation of the 4-dimensional chiral superfield is consistent with that of the wave functions. Using this definition we obtain the Yukawa invariant modular subgroup M for a modular invariant flavor model. Hereafter we refer to the Yukawa invariant modular subgroup M as the modular flavor symmetry. We show that the Yukawa invariant modular subgroup M has the following three independent elements of S 2 , T N and (ST N ) 2 , where N is the least common multiple of the generation numbers of the corresponding zero modes. (T N is well defined since N is always even.) The representations of S 2 and T N are written as For ρ M ((ST N ) 2 ), if M is even and N/M is odd, substituting (4.6), we find Thus Yukawa term is (ST N ) 2 invariant too.
In the case of vanishing Wilson line, the modular symmetry is enhanced. In this case we have Z 2 parity symmetry [45]: (4.9) Substituting the ρ M (S) to (4.3), we find in the second raw, we use (4.4). The Yukawa term is S invariant. Therefore in the case of vanishing Wilson line, the Yukawa invariant modular subgroup M have two independent generators of S and T N . We will see that S can be interpreted as a "square root" of the parity operator in Section 5. where j = 0, 1, 2. The modular transformations of these wave functions are given by (3.16) and (3.19). For M = 3, the matrix representations are given by

Modular flavor symmetry in three-generation model
where ω = e 2πi 3 . We study T 2 instead of T since M 1,2 are odd. For M = |M 3 | = 6, the matrix representations are given by where η = e πi 3 . The Yukawa couplings Y ijk are classified to six values: where Y j is given by Other couplings are prohibited by Z 3 charge of ∆(27) flavor symmetry [45]. A matrix representation of the modular transformation for the 6-component vector (Y i ) is defined as In this basis, ρ Y is exactly the same as the one for the 6-generation chiral zero mode, i.e., ρ Y = ρ 6 . The Yukawa invariant modular subgroup is generated by S 2 , T 6 and (ST 6 ) 2 . These elements satisfy the following relations: (4.17) Thus, they correspond to Z 2 and Z 4 respectively. (ST 6 ) 2 and T 6 are non-commutative, and these three elements generate a non-abelian group. This group has 16 elements. and is found to be isomorphic to Z The irreducible decomposition of the chiral zero modes is given by where the lower index of 1 denotes the eigenvalues of T 6 and (ST 6 ) 2 , and the upper index denotes the eigenvalue of the diagonal Z 2 . Since Z 2 and D 4 are real, irreducible decomposition of the Yukawa couplings are the same as that of ψ j,6 . Table 1 summarizes the irreducible decomposition of each component.

Model without Wilson line
If the Wilson line is set to zero, the Yukawa invariant modular subgroup is enhanced. The Yukawa term is invariant under S for the vanishing Wilson line model and the Yukawa invariant subgroup is enhanced to (Z . The character indices of this group and irreducible representations are summarized in Table 3. This group has eight singlets and six doublets. The 3-generation chiral zero modes are decomposed to three singlets: Representation of D 4 × Z 2 ψ j, 3 1 Table 1: Irreducible decomposition of the chiral zero modes and Yukawa couplings. The upper indices denote the eigenvalue of the diagonal Z 2 and the lower indices denote the eigenvalues of the D 4 generators. Representation of (Z 8 × Z 2 ) ⋊ Z 2 ψ j, 3 1 where the index represents the eigenvalues of T 6 and S; T 6 1 ±j = ±1 ±j and S1 ±j = e j iπ 2 1 ±j . The 6-generation zero modes are decomposed to three doublets: The representation of the Yukawa is the complex conjugate of that of the 6-generation chiral zero modes:6 Table 2 summarizes the irreducible decomposition of each component.

Comments on the possibility of exceptional elements
We see if there is an exceptional element that is not covered by the generators of S 2 , T 6 and (ST 6 ) 2 (S and T 6 for vanishing Wilson line). Since the modular group of {S, T 2 } is finite with the order of 768 = 2 8 × 3, we can numerically check if each modular transformation satisfies the condition (4.2). In our analysis the group elements of the modular transformation are obtained with a specific representation e.g., ρ M , so that the group structure should be defined using the largest representation for definiteness. In this case, we use the definition for the group element of the modular transformation as for concrete calculation. We confirm that there is no other element which keeps the Yukawa term invariant other than the elements covered by S 2 , T 6 and (ST 6 ) 2 (S and T 6 We note that although M is generated by S 2 , T N and (ST N ) 2 (S and T N for vanishing Wilson line), the group structure differs depending on the magnetic fluxes in the model, since the value of N also differs by models. In fact we calculate the group structure for other examples with different magnetic fluxes in Appendix A, and show that various discrete groups appear as modular flavor symmetry, e.g., (Z 2 ×Z 4 )⋊Z 2 for a two-generation model.

Modular Extended Discrete Flavor Symmetry
It is known that the magnetized torus model has discrete flavor symmetry. In this section, we study their relationships and consider the full symmetry group. First we briefly review the conventional discrete flavor symmetry [45]. Suppose that there are chiral zero modes φ j 1 ,M 1 , ..., φ j ℓ ,M ℓ . If the greatest common divisor of the generation numbers, g = g.c.d. (M 1 , ..., M ℓ ), is greater than 1, the theory is invariant under the following two operators: where M k = gJ k , and ω = e 2πi g . C and Z are represented by g × g matrices as These two generators satisfies ZC = ωCZ, and there are three Z g charges in this model. Hence this group is isomorphic to g . We should emphasize that this discrete symmetry is different from the non-abelian symmetry originated from the modular subgroup. The clear difference comes from that the Yukawa couplings are always trivial singlet under the conventional flavor symmetry, but not under the modular transformation.
Let F and M are the conventional flavor group and the Yukawa invariant modular subgroup, respectively. F and M are non-commutative with each other. To see this, we consider 3-generation chiral zero modes for the purpose of illustration. The matrix representation of S 2 for the 3-generation zero modes is given by (4.12). C of ∆ (27) can act on the zero modes too. Their 3-dimensional representations are given by Therefore CS 2 = S 2 C. The sum of the Yukawa invariant modular subgroup and conventional flavor symmetry generates a new group which acts on the effective theory. We use G for denoting this novel group. 2 Our goal of this section is to analyze the structure of G.
As mentioned in the previous section, in order to analyze the group structure we have to use the largest representation given in Eq. (4.23) for all the elements of G. However, since the representation of the Yukawa couplings is trivial for F , i.e., ρ Y (f ) = 1 for f ∈ F , we only need to calculate the algebraic structure for ρ M (the matrix representation for Mgeneration chiral zero mode) in detail. It is convenient to introduce new M × M matrices Z ′ and C ′ as M . These two matrices satisfy the following relations Since ρ M (Z) = Z ′M/g and ρ M (C) = C ′M/g , we obtain We find S 2 F S −2 ⊂ F . If M is even and N/M is odd, Eq. (4.5) becomes We obtain where we note that M/g is always even. 3 T N is commutative with the group elements of F . Using the matrix representation given in (4.6), we obtain Thus we find (5.14) The above two relations hold even if M is odd or N/M is even, i.e., ρ M (T N ) = 1. Thus we find that F is a normal subgroup of G, and G is written as F M. The intersection of F and M is the trivial group, i.e., {e}, since the Yukawa couplings are invariant under F . We conclude G is isomorphic to the semidirect product of F and M: If the Wilson line is set to zero, M is generated by {S, T N }. Using the matrix representation of S given in (3.23) we calculate and we obtain Therefore we find SF S −1 ⊂ mathcalF . In addition, there is a parity symmetry P which acts on the wave functions as This is nothing but S 2 given in Eq. (4.4). Actually the parity operator P is understood as an element of M; P ∈ M. Since Y ijk = Y M 1 −i,M 2 −j,|M 3 |−k for vanishing Wilson line, the action of S 2 on the Yukawa couplings is given as Therefore P is identical to S 2 for the vanishing Wilson line (S 2 as generalization of P for non-vanishing Wilson line). We conclude We consider a concrete example in the following subsection for illustration purposes.

Modular extended flavor symmetry in three-generation model
Here we consider the model of M 1 = M 2 = 3 and M 3 = −6.  Table 4. The following relations can be shown:
These are equivalent to (5.6), (5.7), (5.9), (5.10), (5.13), and (5.14). Thus the conventional flavor group F is the normal subgroup of the novel group G. The intersection of F and M consists only of the identity since the action of F on the Yukawa couplings are always trivial. We conclude G is the semidirect product of F and M: The is the modular extension of the flavor group for this 3-generation model.

Model without Wilson line
Without Wilson line, we have additional generators S. The matrix representation of S is given by We note that P is identical to S 2 since Y jkℓ = Y −i −j −ℓ as we denoted in the previous section. The conjugation by S is given by These are equivalent to (5.18) and (5.19). Therefore G is isomorphic to the semiproduct of F and M: Irreducible decomposition of the 3-generation chiral zero mode is given by a 3-dimensional representation since it is 3 in ∆ (27). The 6-generation chiral zero modes are 6 dimensional representation of G. The Yukawa couplings are decomposed to three 2-dimensional representations, since they are trivial representation in ∆(27).

Conclusion
We have investigated the modular symmetry of the magnetized torus. The modular group is isomorphic to SL(2, Z)/Z 2 and it is an infinite group. For heterotic orbifold, modular group can act on its effective action and it is invariant under the whole group. However, for magnetized torus, the situation is different. When the magnetic fluxes turned on, effective action is no longer invariant under the whole modular group, but is invariant under its specific subgroup M, which we refer as modular flavor symmetry.
We have shown this group consists of S 2 , T N and (ST N ) 2 where N is the least common multiple of the generation numbers in general. These elements are non-commutative and generate non-abelian groups. This group is enhanced for the case of vanishing Wilson line, and the theory (the Yukawa term) becomes S invariant. We show several examples of constructions of this Yukawa invariant subgroups. These subgroups are isomorphic to finite groups, such as D 4 × Z 2 and (Z 8 × Z 2 ) ⋊ Z 2 . We find the group structures depend on the chiral spectrum and we can realize various finite groups as subgroups of the modular group.
It is known that the magnetized torus model has conventional flavor symmetry F . This flavor symmetry includes the parity symmetry in terms of the extra dimension if the Wilson line vanishes. Although the modular group and the conventional flavor group are different, we have found that the parity operator can be interpreted as S 2 in the modular symmetry. We have investigated modular extension of conventional flavor symmetry in detail. They are non-commutative with each other and enlarge the group of the symmetry. We have found the conventional flavor group is a normal subgroup of the novel group G (modular extended flavor group), which is isomorphic to the semidirect product of modular and the conventional flavor group. Our result proposes a novel method for phenomenological model buildings. For instance, the modular flavor symmetry consists of several Z 2 , Z 4 and Z 8 . Such discrete groups are utilized for solving the flavor puzzles [51].
Our result indicates that the Yukawa invariant modular subgroup M can be interpreted as a subgroup of the automorphism of F . It may imply the relationships between generalized CP symmetry and the modular symmetry. The consistency condition of the generalized CP is that generalized CP symmetry should be (outer) automorphism of the favor group [24,25]. Our Yukawa invariant modular subgroup satisfies this condition. Relationships between modular symmetry and generalized CP have also been investigated in [26,27,28].
Our study is based on a field theory analysis of the magnetized torus model which is the low energy effective theory of type II string theory. Taking into account more stringy effects, e.g., vertex operator, local supersymmetry, or Green-Schwartz like anomaly cancellation mechanism, modular properties of fields and couplings may change. Pursuing this possibility is certainly interesting, but it is beyond the scope in the present paper.
and for M = −4, the matrix representations of S and T is given by where complex conjugate are required since M 3 is negative.

Model with Wilson line
First we investigate the model with non-vanishing Wilson line. In this case the Yukawa couplings are classified to 4 values: and these four Y j forms a 4-dimensional representation of the modular group. The Yukawa invariant subgroup is generated by S 2 , T 4 and (ST 4 ) 2 . They satisfy the following equations: Hence they correspond to Z 2 . They are commutative with each others and the group is isomorphic to Z 2 × Z 2 × Z 2 . We also check that there is no extra element which keeps the Yukawa term invariant, but can not be generated by S 2 ,T 4 and (ST 4 ) 2 in the group generated by ρ M (S) and ρ M (T ) which consists of 3072 = 2 10 × 3 elements. Irreducible decomposition of the representations is summarized in Table 5. respectively.

Model without Wilson line
For the vanishing Wilson line model, the Yukawa invariant modular group is enhanced. This group has 16 elements and it contains two Z 2 and one Z 4 . The Z 4 corresponds to S, and the two Z 2 correspond to T 4 and (ST 4 ) 2 . Therefore this group is generated by S and T 4 . They satisfy the following relations: and these mean that the subgroup generated by S and (ST 4 ) 2 is a normal subgroup of the whole group. The group generated by these matrices is isomorphic to (Z . This is the modular symmetry of the Yukawa term without Wilson line. Irreducible decomposition of the representations is summarized in Table 6.

Modular extended discrete flavor symmetry
This model has D 4 flavor symmetry in general, and D 4 ×Z 2 flavor symmetry for vanishing Wilson line model. These D 4 and the Yukawa invariant modular subgroups are noncommutative. As shown in Section 5, we can obtain modular extended flavor symmetry. respectively.
The Yukawa invariant modular subgroup generators are given by Their irreducible decomposition is summarized in Table 7.
It is easy to show that M is commutative with all the generators of F . This is because C −1 and Z −1 are the same as C and Z. Thus, the whole group G is isomorphic to the direct product of F and M. We find G ≃ D 4 × (Z 2 ) 3 . (A.8) Representation of D 4 φ j,2 2 φ j,4 1 ++ ⊕ 1 +− ⊕ 1 −+ ⊕ 1 −− Y j 4 × 1 ++ Table 7: Irreducible representation of the conventional flavor symmetry, Without Wilson line, the modular group is enhanced to (Z 2 × Z 4 ) ⋊ Z 2 . It is generated by S and T 4 . The 10 dimensional representation of S is written as We also have an extra Z 2 symmetry, which acts on the chiral zero modes as ψ j,M → ψ −j,M . This Z 2 is denoted by P and its matrix representation is the same as that of S 2 . The following relations hold: This is the full symmetry of the effective action. Since this group is denoted by (semi)direct product of the groups, its order is 128 = 8 × 16.

A.2 246 model
Here we consider the model with M 1 = 2, M 2 = 4 and M 3 = −6. The matrix representation of the modular transformation is already given in the former subsections. Since g.c.d.(M 1 , M 2 , |M 3 |) = 2, we have D 4 discrete flavor symmetry for nonzero Wilson line models, and D 4 × Z 2 for the vanishing Wilson line. Yukawa couplings are classified to 12 values: The other 3-point couplings are prohibited by Z 2 charge. We obtain 12 dimensional representation of the modular group. This Yukawa term is not invariant under the whole modular group. We construct its subgroup under which the Yukawa term is invariant. If the Wilson line is zero, this subgroup consists of 16 elements. This group is isomorphic to (Z 2 × Z 4 ) ⋊ Z 2 . All elements are commutative with each other. If the Wilson line is not zero, it is not invariant under S, but S 2 , and the group is broken to Z 2 × Z 2 × Z 2 .

A.3 123 model
Here we consider the model of M 1 = 1, M 2 = 2 and M 3 = −3. In this model, there are one 1-generation chiral superfield, one 2-generation chiral superfield, and 3-generation chiral superfield. Their matrix representation of the modular transformation have been given already. In addition, we have 6 Yukawa couplings for general Wilson line case. Their modular transformation is the same as that of 6-dimensional chiral zero mode. If the Wilson line is zero, we have Z 2 parity flavor symmetry. We use 11-dimensional representation to construct the Ykawa invariant modular subgroup: ρ 11 = ρ 2 ⊕ ρ * 3 ⊕ ρ Y = ρ 2 ⊕ ρ * 3 ⊕ ρ 6 . We find that they generate a finite group whose order is 768. The Yukawa invariant modular subgroup is generated by S and T 6 . The subgroup consists of 32 elements. This group is the same as that of the 336 model. This group is isomorphic to (Z 8 × Z 2 ) ⋊ Z 2 . If nonzero Wilson line is turned on, S is no longer an element of the Yukawa invariant modular subgroup. The modular subgroup is broken to D 4 × Z 2 .