The top quark rare decays with flavor violation

In the present study, we investigate the decays of the top quark $t\rightarrow c\gamma$, $t\rightarrow cg$, $t\rightarrow cZ$ and $t\rightarrow ch$. They are extremely rare processes in the Standard Model (SM). As the $U(1)$ extension of the minimal supersymmetric standard model (MSSM), the $U(1)_X$SSM features new superfields such as the right-handed neutrinos and three Higgs singlets. We analyze the effects of different sensitive parameters on the results and make reasonable theorecial predictions, which provides a useful reference for future experimental development. Considering the constraint of the updated experimental data, the numerical results show that the branching ratios of all four processes $t\rightarrow c\gamma,~cg,~cZ,~ch$ can reach the same order of magnitude as their experimental upper limits. Among them, $\tan\beta$ has the most obvious effect on each process and is the main parameter. $g_X$, $g_{YX}$, $\mu$, $M_2$, $\lambda_H$, $M_{U23}^2$ and $M_{Q23}^2$ are important parameters for the processes, and have effects on the numerical results.


I. INTRODUCTION
The top quark was discovered in 1995 by a team of D0 and CDF experiments located at the Fermi National Accelerator Laboratory (Fermilab) in the United States [1,2].This discovery is important for the validation of the SM and the study of particle physics.The study of the nature and behavior of the top quark helps us to understand physical processes such as the origin of the mass of elementary particles, weak and strong interactions.The existence and properties of the top quark have been verified several times in experiments, including the D0 and CDF experiments at Fermilab and the ATLAS [3] and CMS [4] experiments at the Large Hadron Collider (LHC) in Geneva, Switzerland.These experiments investigate aspects of the nature, decay modes and interactions of the top quark through high-energy collision and particle detection techniques.The next generation of LHC will produce top quarks in large quantities.At the upgraded Fermilab, an integrated luminosity of 10f b −1 will produce about 8 × 10 4 top quarks, while at the same luminosity the LHC will produce about 100 times as many [5][6][7].
Top quark decays with flavor violation refer to the decay processes of the top quark that violate the conservation of flavor, specifically the violation of lepton or quark flavor.
While the SM predicts that the top quark predominantly decays into a W boson and a bottom quark, extensions beyond the SM allow for additional decay modes that involve different quarks or leptons [8,9].It is worth noting that specific details about the nature and extent of flavor violation during top quark decays require more in-depth analysis.The study of heavy particle decays via flavor-changing neutral-currents (FCNC) has played an important role in testing the SM and exploring new physics beyond the SM [10][11][12].In the SM branching ratios of the FCNC of the top quarks t → cγ, t → cg, t → cZ, t → ch are highly suppressed, and beyond the detection capabilities of the LHC in the near future [13][14][15].However, exotic mechanisms from new physics can greatly increase these branching ratios of [16], which can be detected in the future.The SM predictions [17] and the latest upper bounds on the branching ratios of t → cγ, t → cg, t → cZ and t → ch at the 95% confidence level (C.L.) [18] are given in Table I.We can find that the current experimental bounds are much higher than the predictions of SM.
As the heaviest elementary particle in the SM with mass on the electroweak scale, the top quark is likely to be more sensitive to new physics.Kinematically, it can reach many FCNC decay modes such as t → cγ, t → cg, t → cZ and t → ch, where h is the lightest CP-even Higgs boson.In the SM, these FCNC decay modes are highly suppressed by the GIM mechanism, with branching ratios typically of the order of 10 −15 − 10 −12 [5][6][7][19][20][21][22][23][24][25], a relatively small order of magnitude.On the other hand, the observation of any such FCNC top-quark decay would be strong evidence of new physics.Therefore, detecting those topquark rare decays at the LHC provides a good window to search for new physics beyond the SM.We learn some theoretical predictions for the branching ratios of top quark rare decays in new physics extensions, such as in supersymmetric (SUSY) models with R-parity conservation.These branching ratios can reach Br(t → cγ) ∼ 10 −6 , Br(t → cg) ∼ 10 In this work, we explore top quark decays with flavor violation under the U(1) X SSM.The µ problem, while in the U(1) X SSM [38] this problem can be alleviated by the S field after vacuum spontaneous breaking.
The outline of this paper is as follows.In Sec.II, we briefly introduce the essential content of the U(1) X SSM, including its superpotential, the general soft breaking terms, the rotations and interactions of the eigenstates "EWSB".In Sec.III, we provide analytical expressions for the branching ratios of the t → cV, ch (V = γ, Z, g) decays in the U(1) X SSM.In Sec.IV, we give the corresponding parameters and numerical analysis.In Sec.V, we present a summary of this article.
In this section, we will provide some overview of U(1) X SSM.U(  [39][40][41].In addition to the MSSM, the field spectrum in U(1) X SSM contains new superfields, which are the right-handed neutrinos νi and the three Higgs singlets η, η, Ŝ.
Through the seesaw mechanism, the lighter neutrinos gain tiny masses at the tree level.The formation of the 5 × 5 mass-squared matrix is due to the mixing of the neutral CP-even parts of H u , H d , η, η, and S. To obtain the 125.25 GeV Higgs particle mass [42,43], loop corrections should be considered.These sneutrinos are decomposed into CP-even sneutrinos and CP-odd sneutrinos, and their mass-squared matrices are both expanded to 6 × 6.
The superpotential in U(1) X SSM is denoted by: In Eq.
The soft SUSY breaking terms of U(1) X SSM are shown as: This effect can be caused by RGEs.A ′ Y µ and A ′ X µ denote the gauge fields of U(1) Y and U(1) X respectively.The form of the covariant derivative of the U(1) X SSM can be written as: We redefine the following [44,45]: Finally, the gauge derivative of U(1) X SSM is transformed into: The g X appearing above is the gauge coupling constant for the U(1) X group.g Y X is the mixed gauge coupling constant for the U(1) Y group and the U(1) X group.
In the U(1) X SSM, the gauge bosons µ and V 3 µ are mixed together at the tree level.The mass matrix of gauge bosons can be found in reference [40].We use two mixing angles θ W and θ ′ W to obtain the mass eigenvalues of the matrix.θ W is the Weinberg angle and θ ′ W is the new mixing angle.We The new mixing angle is defined as: We derive the eigenvalues of the mass-squared matrix of the neutral gauge bosons.One is the zero mass corresponding to the photon and the other two values are Z and Z ′ , The mass matrix for chargino is: This matrix is diagonalized by U and V: The mass matrix for neutrino is: This matrix is diagonalized by U V : In addition, a number of other mass matrices are required in the calculations, all of which can be found in Refs.[40,46].

III. ANALYTICAL FORMULA
In this section, we focus on the theoretical study of the top quark processes t → cγ, t → cg, t → cZ and t → ch with flavor violation under the U(1) X SSM.The relevant Feynman diagrams contributing to t → cγ, t → cg, t → cZ and t → ch in the U(1) X SSM are presented in Fig. 1 and Fig. 2.
In the U(1) X SSM, the flavor violating amplitude corresponding to the decay process ) is written as: In order to better explain how the calculation of the above equation as well as the Feynman diagram in Fig. 1 is done, we take the calculation of Fig. 1(1) as an example.The corresponding amplitude can be written as: (5) FIG. 2: Feynman diagrams for the t → ch process in the U (1) X SSM.
Here, ε µ denotes the polarization vectors of photon and Z boson.u t and u c denote the wave functions of the top and charm quarks, p is the momentum of the top quark, p ′ is the momentum of the charm quark and q is the momentum of the vector boson.m Di , m Dk and m χ ± are the mass eigenvalues from Eq. (11).Correspondingly, and B V D D are the coupling vertices [40,41,47,48].L and R in the subscripts denote the left-handed and right-handed parts, respectively.They can be derived from SARAH.
In the next calculation, we will first solve the Feynman integral, and the formula we use for the integration of the denominator is [49]: Calculating integral in this way can greatly increase the efficiency of numerical calculation in our work.According to Eq.( 15) we get: here we let b to get the final form of the denominator in Eq.( 14) as: This type of substitution is also performed for the numerator of Eq.( 14).We take all the diagrams in Fig. 1 and calculate the Feynman amplitudes for each of t → cγ, t → cg and t → cZ.Finally we perform the calculation of the respective mode squares for the three processes.
In the MSSM, the top quark decay t → ch is flavor-changing, where h is the lightest CP-even Higgs boson.By studying Fig. 2, we find that in addition to the new contribution of the down-type quarks, the mixing between the Higgs doublet and the exotic single-line states η1,2 also affects the t → ch decay channel.We use the following calculation of Fig. 2(4) as an example.The amplitude can be expressed as: Other graphs of the t → ch process can be calculated similarly.
We use dimensional regularization to treat the divergences with d = 4 − 2ǫ and the limit d → 4. To obtain finite results, the divergences are canceled by the modified minimal substraction (MS) scheme.The terms proportional to 1 ǫ − γ E + log(4π) are deleted.Here γ E ≈ 0.5772 is Euler constant.
Based on the above calculations, the branching ratios of the top quark rare decays are respectively: where Γ total =1.42 +0.19 −0.15 GeV [18] is the total decay width of top quark.

IV. NUMERICAL ANALYSIS
In this section, we study the numerical results of flavor violation for the top-quark t → cV, ch processes.According to the latest LHC data [50][51][52][53][54], our values are subject to certain constraints.So we consider a number of individual experimental constraints including: 1.The lightest CP-even Higgs mass is around 125.25 GeV [18,[55][56][57].
2. The updated experimental data show that the mass of the Z ′ boson at the 95% confidence level (CL) [58] satisfies M Z ′ > 5.15 TeV.Eq.( 15) leads to an approximate result of M Z ′ as M Z ′ ≈ g X ξ > 5.15 TeV.

3.
The ratio between M Z ′ and its gauge coupling constant TeV [59,60], so g X is restricted in the region 0< g X 0.85.
5. The limitations for the particle masses accord to the PDG [18] data, and the concrete contents are the following.The neutralino mass is limited to more than 116 GeV, the chargino mass is limited to more than 1000 GeV and the scalar quark mass is greater than 1300 GeV.
The relevant SM input parameters in the numerical program are selected as: 27 GeV, m t =172.69GeV.In conjunction with the above experimental requirements, we obtain a wealth of data and use graphs to analyze and process the data.We generally take the values of new particle masses(M BB ′ , M BL ) near the order of 10 3 GeV, which is around the energy scale of new physics.T λ C and T λ H are trilinear coupling coefficients, which are roughly in the order of magnitude of the mass, and can be varied up or down to the order of 10 2 ∼ 10 4 GeV.
Dii , B S and B µ are all of mass square dimension, and can be up to the order of 10 6 GeV 2 .The dimensionless parameters λ C and λ H are generally taken as numbers less than 1.Considering the above constraints in the previous paragraphs, we use the following parameters: Here, we default to the non-diagonal elements of the mass matrix being set to zero if not otherwise specified.In U(1) X SSM, g Y X is the mixing gauge coupling constant of U(1) Y group and U(1) X group, and it is the parameter beyond MSSM.The mass matrices of neutralino, down type squark and up type squark all contains g Y X .Furthermore, g Y X appears in the vertex, which can enlarge the coupling constant of the vertex.M BB ′ is the mass of the U(1) Y and U(1) X gaugino mixing and presents in the mass matrix of neutralino.tan β appears in almost all the mass matrices of fermions, scalars and Majoranas, and it must be a sensitive parameter.It can affect the masses of particles and vertex couplings by directly affecting v u and v d , M BL is the mass of the new gaugino, and it has influence on the mass matrix of neutralino.λ H relates to the strength of the self-interaction coupling of the Higgs field, which affects the VEV and the Higgs boson mass.
A. The process of t → cγ In order to find out the parameters affecting the top quark flavor violation, some sensitive parameters need to be studied.To show the numerical results clearly, the parameters are set to M 2 Dii = 6 × 10 6 GeV 2 , M 2 Qii = 6 × 10 6 GeV 2 (i=1,2,3), µ = 1000 GeV, M 1 = 1200 GeV.We plot the relationship between Br(t → cγ) and different parameters.
First we plot the one-dimensional diagrams of Br(t → cγ) versus M 2 Q23 , M 2 as shown in Fig. 3.The gray shaded area is the experimental limit satisfied by the Br(t → cγ) process.In Fig. 3 (a) we plot Br(t → cγ) versus M 2 Q23 .Let tan β = 20, M 2 = 1200 GeV, g Y X = 0.2 and λ H = 0.1, the solid line corresponds to g X =0.3 and the dashed line corresponds to g X =0.6.
Overall, both lines show a decreasing trend in the range of 0 − 4 × 10 5 GeV 2 for M 2 Q23 due to the fact that the contribution of the lower-type squarks is canceled by the contribution of the charged Higgs boson at the turning point.Then it is followed by an upward trend, i.e., which means that Br(t → cγ) increases as M 2 Q23 ≥ 4 × 10 5 GeV 2 .From bottom to top in the Fig. 3 (a) Br(t → cγ) increases as the value of g X increases.Fig. 3 (c) is the differential distribution of Fig. 3 (a).From Fig. 3 (c) we find that the trend and pattern of the values in Fig. 3 (a) are more obvious.In Fig. 3 (c), the differential increases linearly, and the speed of variable change is relatively smooth.It shows that M 2 Q23 is a parameter that has influence on Br(t → cγ).
M 2 = 1400 GeV, followed by a slight downward trend.It can be seen that the overall value satisfies this limit and its overall trend is a decreasing function.As the line in the graph goes from bottom to top i.e. as tan β increases Br(t → cγ) also increases gradually.Fig. 3 (d) is the differential distribution of Fig. 3 (b).In Fig. 3 (d) there is a maximum value of differential value appears when M 2 =1400 GeV, at this time Br(t → cγ) is the maximum value.The differential value is negative when M 2 >1600 GeV.That is to say, Br(t → cγ) is decreasing, but the decrease trend is very small.In order to better and more deeply explore the parameter space, as M 2 = 1200 GeV, we scan some parameters randomly, and in the Br(t → cγ) process we scan the following parameters: In Fig. 4(a) we set λ H = 0.1, g Y X = 0.2 to explore the effects of tan β and g X on Br(t → cγ).It is clear from the figure that the value of Br(t → cγ) increases as tan β increases.
When tan β reaches its maximum value of 50, Br(t → cγ) reaches an order of magnitude of 10 −4 , very close to the experimental upper limit.It indicates that tan β is a very important parameter.The value of Br(t → cγ) turns large as g X increases, but the effect of g X on Br(t → cγ) is small and hardly noticeable.In Fig. 4(b) we set tan β = 20 and g X = 0.3 to explore the effects of λ H and g Y X on Br(t → cγ).From the Fig. 4(b) we find that both λ H and g Y X have effects on Br(t → cγ).
The value of Br(t → cγ) decreases with the increase of g Y X , and the smaller the value of g Y X the closer to the upper limit of the Br(t → cγ).There is some slight increase in the value of Br(t → cγ) with the enlaring λ H , but the impact of λ H is relatively small compared to g Y X .In Fig. 4(b), we find the presence of a white area in the upper left corner.This occurs because of the limitation from the masses of Higgs and other particles.gets to the experimental upper limit.g Y X has some minor effect on the results, and it is not apparent enough here.In Fig. 6 (b), we set tan β = 20 to explore the effects of g X and g Y X on Br(t → cg), and it shows that the larger g X is the larger Br(t → cg) is.On the other hand the larger g Y X is the smaller Br(t → cg) is.The Br(t → cg) is maximized when g X = 0.7 and g Y X = 0.01, and the Br(t → cg) is closer to the upper limit of the experiment.
When g Y X tends to zero, the dependence of the branching ratio on g X is strong.

C. The process of t → cZ
The experimental upper bound (5 × 10 −4 ) for the Br(t → cZ) process is of the same order of magnitude as for Br(t → cγ) and Br(t → cg).In this subsection, we still discuss the effects of different parameters on the Br(t → cZ) branching ratio, where we focus on the effects of the parameters Dii and M 2 Qii .We fix the value of the parameters M 1 = 1200 GeV and λ H = 0.1, one-dimensional Fig. 7  The solid and dashed lines run from bottom to top, indicating that tan β is also a sensitive parameter for Br(t → cZ), which increases with tan β.In order to explore the effects of different parameters on Br(t → cZ) in greater depth, let µ = 1000 GeV, M 2 = 1200 GeV, tan β = 20, we will continue to perform randomized scans with different parameters in the range: In Fig. 8 (a) we set M 2 Qii = 6 × 10 6 GeV 2 , M 2 Dii = 6 × 10 6 GeV 2 to explore the effects of g X and g Y X on Br(t → cZ).We can clearly see that Br(t → cZ) has a minimum at g X = 0.3 and g Y X = 0.01.For Br(t → cZ) both g X and g Y X are sensitive parameters.g Y X is a coupling constant that affects the strength of gauge mixing.Further more g X and g Y X make a new contribution to Br(t → cZ) through Z − Z ′ mixing.g X also has a large effect on Br(t → cZ) when g Y X tends to a minimum.Br(t → cZ) increases with enlarging g X and g Y X vice versa.In Fig. 8 (a), the white region appears in the upper right corner.By analysing, we get that it is excluded by the experimental upper limit Br(t → cZ) < 5×10 −4 .In Fig. 8 (b) we set g X = 0.3 and g Y X = 0.2 to explore the effects of M  by setting tan β = 20, λ H = 0.1.In Fig. 10 (b) we explore the effect of g X (0.3 ≤ g X ≤ 0.7) and λ H (0.1 ≤ λ H ≤ 0.4) on Br(t → ch) by setting tan β = 20, g Y X = 0.2.In Fig. 10 (c) we again set g X = 0.3 and g Y X = 0.2 to explore the effect of tan β (10 ≤ tan β ≤ 50) and λ H (0.1 ≤ λ H ≤ 0.4) on Br(t → ch).Combining the three plots in Fig. 10, we can clearly see that all four parameters g X , g Y X , λ H and tan β affect the Br(t → ch), but they do so in different ways.g X and g Y X both present obvious effects on the numerical results.
Br(t → ch) is decreasing function of g X and g Y X .At the time they both take the minimum value, the branching ratio of the t → ch process reaches 10 −4 which is very close to the experimental upper limit.In Fig. 10  the mass matrix, but also dominates the non-diagonal sectors, leading to the above results.

U( 1 )
X SSM is an extension of the MSSM that incorporates an extra U(1) X gauge symmetry.Its local gauge group is SU(3) C ×SU(2) L ×U(1) Y ×U(1) X [35-37].Compared to the MSSM, in the U(1) X SSM we add three new Higgs singlets η, η, Ŝ and three-generation right-handed neutrinos νi .The right-handed neutrinos have the function of both generating tiny mass to light neutrinos via a see-saw mechanism and providing a new dark matter candidate light sneutrino.The presence of right-handed neutrinos, sneutrinos and additional Higgs singlets alleviates the so-called small hierarchy problem arising in the MSSM.MSSM exists

( 1 )
the vacuum expectation value of η produces the Majorana mass of the righthanded neutrino through Y X νη ν.While the right-handed neutrino mixes with the lefthanded neutrino through Y ν νl Ĥu .The vacuum expectation values(VEVs) of the Higgs superfields H u , H d , η, η and S are denoted by v u , v d , v η , v η and v S respectively.Two angles are defined as tan β = v u /v d and tan β η = v η/v η .The explicit forms of the two Higgs doublets and three Higgs singlets are written as:

FIG. 3 :
FIG. 3:The diagrams of Br(t → cγ) affected by different parameters.The gray area is reasonable value range, where Br(t → cγ) is lower than the upper limit.The solid line and dashed line in Fig.3(a) correspond to g X = 0.3 and g X = 0.6.The solid line and dashed line in Fig.3(b)

- 6 FIG. 4 :
FIG. 4: (a) Effects of tan β and g X on Br(t → cγ).The horizontal coordinate indicates the range 5 ≤ tan β ≤ 50 and the vertical coordinate indicates 0.3 ≤ g X ≤ 0.7.(b) Effects of g Y X and λ H on Br(t → cγ).The horizontal coordinate indicates the range 0.01 ≤ g Y X ≤ 0.5 and the vertical coordinate indicates 0.1 ≤ λ H ≤ 0.4.The icons on the right side indicate the colors corresponding to the values of Br(t → cγ).

FIG. 5 :
FIG.5: Br(t → cg) diagrams affected by different parameters.The gray area is reasonable value range, where Br(t → cg) is lower than the upper limit.The solid line and dashed line in Fig.5(a) correspond to µ = 1000 GeV and µ = 1100 GeV.The solid line and dashed line in Fig.5(b) correspond to tan β = 23 and tan β = 25.Fig.5(c) shows the differential distribution of (a) and Fig.5(d) shows the differential distribution of (b).

- 6 FIG. 6 :
FIG. 6: (a) Effects of tan β and g Y X on Br(t → cg).The horizontal coordinate indicates the range 5 ≤ tan β ≤ 50 and the vertical coordinate indicates 0.01 ≤ g Y X ≤ 0.5.(b) Effects of g X and g Y X on Br(t → cg).The horizontal coordinate indicates the range 0.3 ≤ g X ≤ 0.7 and the vertical coordinate indicates 0.01 ≤ g Y X ≤ 0.5.The icons on the right side indicate the colors corresponding to the values of Br(t → cg).

4 × 7 4FIG. 8 : 2
FIG. 8: (a) Effects of g X and g Y X on Br(t → cZ).The horizontal coordinate indicates the range 0.3 ≤ g X ≤ 0.7 and the vertical coordinate indicates 0.01 ≤ g Y X ≤ 0.5.(b) Effects of M 2 Dii and M 2 Qii on Br(t → cZ).The horizontal coordinate indicates the range 10 6 GeV 2 ≤ M 2 Dii ≤ 10 7 GeV 2 and the vertical coordinate indicates 10 6 GeV 2 ≤ M 2 Qii ≤ 10 7 GeV 2 .The icons on the right side indicate the colors corresponding to the values of Br(t → cZ).

FIG. 9 : 2 Q23 = 10 5
FIG.9: Br(t → ch) diagrams affected by different parameters.The gray area is reasonable value range, where Br(t → ch) is lower than the upper limit.As g X = 0.4,The solid line and dashed line in Fig.9(a) correspond to M 1 = 1200 GeV and M 1 = 1600 GeV.Making M 2 Q23 = 10 5 GeV 2 , the solid line and dashed line in Fig.9(b) correspond to tan β = 20 and tan β = 25.Fig.9(c) shows the differential distribution of (a) and Fig.9(d) shows the differential distribution of (b).

FIG. 10 :
FIG. 10: (a) Effects of g X and g Y X on Br(t → ch).The horizontal coordinate indicates the range 0.3 ≤ g X ≤ 0.7 and the vertical coordinate indicates 0.01 ≤ g Y X ≤ 0.5.(b) Effects of g X and λ H on Br(t → ch).The horizontal coordinate indicates the range 0.3 ≤ g X ≤ 0.7 and the vertical coordinate indicates 0.1 ≤ λ H ≤ 0.4.(c) Effects of tan β and λ H on Br(t → ch).The horizontal coordinate indicates the range 5 ≤ tan β ≤ 50 and the vertical coordinate indicates 0.1 ≤ λ H ≤ 0.4.The icons on the right side indicate the colors corresponding to the values of Br(t → cZ).
To summarize, in this work we study the rare decays t → cγ, cg, cZ, ch of the top quark in the U(1) X SSM.Compared to the MSSM, in the U(1) X SSM we add three new Higgs singlets η, η, Ŝ and three generations of right-handed neutrinos νi .Its local gaugegroup is SU(3) C × SU(2) L × U(1) Y × U(1) X .Probing with the U(1) X SSM makes our study

TABLE I :
The SM predictions and experimental bounds on the decays t → cV, ch [40]M are shown in the TableII.In our previous work, we have shown that the U(1) X SSM is anomaly free[40].In the U(1) X SSM, U(1) Y and U(1) X are two Abelian groups.We denote the U(1) Y charge by Y Y .The U(1) X charge by Y X , and the presence of these two Abelian groups gives rise to a new effect that is not found in the MSSM or other SUSY models with only one Abelian gauge group: the gauge kinetic mixing.

2
Qii and M 2 Dii on Br(t → cZ).The values of M 2 Qii and M 2 Dii are both in the range of 10 6 GeV 2 − 10 7 GeV 2 , and through Fig.8 (b) we find that M 2Dii has tiny effect on Br(t → cZ).However, as M 2 Qii increases.In summary, for the t → cZ process, the main sensitive parameters are M 2 , g X , g Y X and the off-diagonal element M 2 U 23 .While the diagonal elements M 2 Qii (i=1,2,3) have an effect but not very sensitive.