Heavy Neutral Leptons in Gauged $U(1)_{L_\mu-L_\tau}$ at Muon Collider

Heavy neutral leptons $N$ are the most appealing candidates to generate the tiny neutrino masses. In this paper, we study the signature of heavy neutral leptons in gauged $U(1)_{L_\mu-L_\tau}$ at a muon collider. Charged under the $U(1)_{L_\mu-L_\tau}$ symmetry, the heavy neutral leptons can be pair produced via the new gauge boson $Z'$ at muon collider as $\mu^+\mu^-\to Z^{\prime *}\to NN$ and $\mu^+\mu^-\to Z^{\prime (*)} \gamma\to NN\gamma$. We then perform a detailed analysis on the lepton number violation signature $\mu^+\mu^-\to NN\to \mu^\pm\mu^\pm W^\mp W^\mp$ and $\mu^+\mu^-\to NN \gamma\to \mu^\pm\mu^\pm W^\mp W^\mp \gamma$ at the 3 TeV muon collider, where the hadronic decays of $W$ boson are treated as fat-jets $J$. These lepton number violation signatures have quite clean backgrounds at the muon collider. Our simulation shows that a wide range of viable parameter space is within the reach of the 3 TeV muon collider. For instance, with new gauge coupling $g'=0.6$ and an integrated luminosity of 1000 fb$^{-1}$, the $\mu^\pm\mu^\pm JJ$ signal could probe $m_{Z'}\lesssim 12.5$ TeV. Meanwhile, if the gauge boson mass satisfies $2 m_N


Introduction
Heavy neutral leptons N are well motivated to explain the origin of tiny neutrino masses.Due to the singlet nature of heavy neutral leptons under the standard model gauge group, we can write a Majorana mass term m N N c N , which results in light Majorana neutrino masses as m ν ∼ m 2 D /m N via the type-I seesaw mechanism [1,2].To generate the sub-eV neutrino masses, the heavy neutral lepton masses should be at very high scale m N ≳ 10 14 GeV if m D is at the electroweak scale, which is far beyond current experimental reach.Alternatively, an electroweak scale m N is usually assumed for phenomenological studies [3], which is then determined by the mixing parameter V ℓN .
To confirm the Majorana nature of neutrinos, lepton number violation signatures are expected.The most sensitive experiment is the neutrinoless double beta decay, which could probe the inverted mass ordering scenario in the next generation experiments [4].On the other hand, the detection of the lepton number violation signature at colliders could unravel the explicit mechanism of tiny neutrino masses [5].For heavy neutral leptons in the type-I seesaw, the distinct signature at the hadron collider is pp → W ± * → ℓ ± N → ℓ ± ℓ ± jj [6].However, the sensitive region of this signature heavily depends on a relatively large mixing parameter V ℓN [7,8], which is usually much higher than the natural seesaw prediction V ℓN ∼ m ν /m N .
Apart from the canonical seesaw there are many theories with extended gauge groups, such as the U (1) B−L model [9] and the left-right symmetric model [10].In these models, the heavy neutral leptons are charged under the extended gauge group, which opens new production mechanisms for the heavy neutral leptons.Search for pair production of heavy neutral leptons via the decay of Z ′ boson in the framework of the left-right symmetric model has been performed recently by the CMS collaboration [11].The region within m N ≲ 1. 4 TeV and m Z ′ ≲ 4. 4 TeV has been excluded by the dimuon channel.Meanwhile, the ATLAS collaboration searches for right-handed W ′ boson decaying to heavy neural lepton N and leptons ℓ, which can exclude the region with m N ≲ 3.8 TeV and m W ′ ≲ 6. 4 TeV [12].
The construction of a multi-TeV muon collider is proposed recently [13,14].Since then, searches for heavy neutral leptons at the multi-TeV muon collider have drawn increasing interest [15][16][17][18][19].One promising signature is µ + µ − → N ν, which could probe the mixing parameter |V µN | 2 ≳ 10 −6 at the 10 TeV muon collider [20][21][22].Due to missing neutrino in the final states, this signal can hardly confirm the Majorana nature of heavy neutral leptons.Another interesting signature is the lepton number violation signal via vector boson scattering process W ± Z/γ → ℓ ± N [23], via associated production process µ + µ − → N W ± ℓ ∓ [24], or at the same-sign muon collider via µ + µ + → W + W + process [25].These studied signatures also require that the mixing parameter V ℓN is not too small.Meanwhile, under the tight constraints from current experimental searches [11,12], the first stage of a 3 TeV muon collider [26] is unpromising to probe heavy neutral leptons in the U (1) B−L and left-right symmetric models.
The multi-TeV muon collider is a perfect machine to test the muon-philic forces.One attractive option is the gauged U (1) Lµ−Lτ model [27][28][29].Compared to the U (1) B−L or left-right symmetry, the U (1) Lµ−Lτ symmetry is less constrained due to the lack of direct couplings to electron and quarks.For instance, there is only a loose constraint from neutrino trident production when m Z ′ ≳ 100 GeV [30].Therefore, this model is extensively studied to explain the anomaly of the muon magnetic moment [31,32], B meson anomaly [33,34], dark matter [35,36], and neutrino masses [37,38].It is shown that a 3 TeV muon collider is powerful enough to probe quite a large parameter space with m Z ′ ≲ 10 TeV [39,40].In this paper, we consider the gauged U (1) Lµ−Lτ model with three heavy neutral leptons N e , N µ , N τ .We then study the pair production of heavy neutral lepton N via Z ′ boson at a 3 TeV muon collider.To test the Majorana nature of neutrinos, we focus on the lepton number violation process The paper is organized as follows.In Section 2, we review the gauged U (1) Lµ−Lτ model with three heavy neutral leptons and discuss relevant experimental constraints.The decay properties of gauge boson Z ′ and heavy neutral lepton N are also considered in this section.In Section 3, we study the pair production of heavy neutral lepton at the 3 TeV muon collider.Analysis of the lepton number violation signatures are performed in Section 4. The conclusion is in Section 5.

The Model
In this paper, we consider the anomaly-free gauged U (1) Lµ−Lτ extension of the type-I seesaw mechanism.Three heavy neutral leptons N e , N µ , N τ with U (1) Lµ−Lτ charge (0, 1, −1) are introduced to generate the tiny neutrino masses.Predicting type C R of the two-zero minor [41,42], the minimal model with only one scalar singlet Φ 1 , which carries U (1) Lµ−Lτ charge +1, is now tightly constrained by the neutrino oscillation data and the sum of light neutrino masses [43,44].So the second scalar singlet Φ 2 with U (1) Lµ−Lτ charge +2 is also employed to reduce the above conflict [45].The Yukawa interactions and mass terms relevant to neutrino masses are given by [46] where L e , L µ , L τ are the lepton doublets, H is the standard model Higgs doublet, and H = iτ 2 H * .After the spontaneous symmetry breaking, we can denote the vacuum expectation values of scalars as Then the Dirac neutrino mass matrix and heavy neutral lepton mass matrix are given by Light neutrino masses are generated via the type-I seesaw formula Without any specific structure of heavy neutral lepton mass matrix M N , the obtained light neutrino mass matrix M ν is a general symmetric matrix, thus can easily fit the neutrino oscillation data [47].Inversely, we can use Eqn.(2.3) to express the mass matrix M N as [48] M where and U is the light neutrino mixing matrix.In this way, the mass matrix M N is determined by the light neutrino oscillation data and the Yukawa coupling y e , y µ , y τ .The mass matrix M N can be diagnalized by a unitarity matrix Ω as Considering the mixing with light neutrinos, the heavy neutral leptons interact with W ± , Z gauge boson and Higgs boson as where the mixing matrix V ≃ M D M −1 N .For electroweak scale heavy neutral lepton, we assume a seesaw induced mixing parameter V ℓN ∼ m ν /m N ∼ 10 −6 , which is allowed by current experimental limits [3].
The interactions of the new boson Z ′ with fermions are In terms of mass eigenstates, the new neutral current of neutral leptons can be rewritten as From Eqn. (2.6) and Eqn.(2.8), it is obvious that the collider phenomenology of heavy neutral leptons depends on the mixing patterns, which makes the results model dependent.In the following collider simulation, we consider a muon-philic heavy neutral lepton N , which couples exclusively to muon [49].We also take the mixing parameter Ω * µN Ω µN − Ω * τ N Ω τ N = 1 in the following calculation, so the general model dependent results can be obtained by a simple rescaling.

Constraints
In this section, we briefly summarize the constraints on the gauged U (1) Lµ−Lτ model.One important motivation of this symmetry is to explain the anomaly of the muon magnetic moment ∆a µ = (251 ± 59) × 10 −11 [31].The new gauge boson Z ′ contributes to ∆a µ at one loop level, which is evaluated as [32] (2.9) The viable parameter space to interpret ∆a µ is shown in Figure 1.For very light m Z ′ , we have ∆a µ ≃ g ′2 /8π 2 , which requires g ′ ∼ 4 × 10 −4 to explain ∆a µ .While in the heavy For the gauge boson Z ′ , one tight constraint is the dimuon production from the inelastic neutrino-nucleus scattering [50].The CCFR experiment has measured the neutrino trident production cross section, which is consistent with the standard model prediction as σ/σ SM = 0.82 ± 0.28 [30].This result has excluded the ∆a µ favored region with m Z ′ ≳ 0.3 GeV.Recently the NA64µ experiment has excluded g ′ ≳ 6 × 10 −4 with m Z ′ ≲ 0.1 GeV [51].A more stringent constraint comes from the e + e − → 4µ searches at BABAR in the mass region of 0.212 < m Z ′ < 10 GeV [52].For Z ′ mass in the range of [5,81] GeV, the LHC experiment also excludes large parameter space by the process pp → Z ′ µ + µ − → 4µ [53].Meanwhile for Z ′ lighter than about 6 MeV, it is disfavored by ∆N eff constraints from big-bang nucleosynthesis (BBN) [54].In the following study, we consider m Z ′ ≳ 100 GeV to avoid these tight constraints.

Decay Properties
Before studying the explicit signatures, we first review the decay properties of new gauge boson Z ′ and heavy neutral lepton N .The partial decay widths of Z ′ are calculated as where ℓ = µ, τ , and we have assumed vanishing masses for ℓ, ν ℓ in the above calculations.
The new gauge boson Z ′ also couples to electron at one loop level.The Z ′ → e + e − decay width is suppressed by the kinetic mixing factor ϵ ≃ g ′ /70, thus is not taken into account in this study.
After the production of heavy neutral lepton N , it decays via the mixing with light neutrino.When the heavy neutral lepton is heavier than the standard model gauge bosons, the two body decays are the dominant modes.The partial decay widths are given by In Figure 2, we show the branching ratio (BR) of gauge boson Z ′ and heavy neutral lepton Neglecting the final states phase space effect, we have BR( for two degenerate heavy neutral leptons.In the heavy m N limit, we have In order to reconstruct the heavy neutral lepton mass, we focus on the decay mode N → ℓ ± W ∓ with W → q q′ in the collider simulation.

Pair Production of Heavy Neutral Lepton
Through mixing with light neutrinos, the pair production of heavy neutral leptons at the muon collider can occur through the s-channel Z exchange and t-channel W exchange.However, the production cross section is proportional to the fourth power of the mixing parameter |V ℓN | 4 .Provided natural seesaw predicted value V ℓN ≲ 10 −6 , the predicted cross section is negligible tiny.The heavy neutral lepton N is charged under the U (1) Lµ−Lτ symmetry, which induces the pair production of heavy neutral lepton via Z ′ at the muon collider.With fixed collision energy, the direct pair production process Meanwhile when m Z ′ < √ s, the new gauge boson Z ′ can be produced on-shell with an associated photon from the initial legs as µ + µ − → Z ′ γ → N N γ, which also requires m N < m Z ′ /2 to make the decay Z ′ → N N kinematically allowed.

Without Associated Photon
We first consider the pair production of heavy neutral leptons without associated photon.The production cross section of µ + µ − → Z ′ * → N N with a center of mass energy √ s is calculated as where Γ Z ′ is the total decay width of Z ′ .The theoretical cross section of µ + µ − → Z ′ * → N N at a 3 TeV muon collider for specific scenarios are shown in Figure 3, where we directly use Eqn 3.1 without considering the initial state radiation effect.It is obvious that when m Z ′ ≃ √ s, the on-shell production of Z ′ can greatly enhance the cross section.The maximum value is approxi- [40], which could over 50 pb.For a light Z ′ , the cross section reduces to which is independent of Z ′ mass.Typically for g ′ = 0.3 and m N = 1000 GeV, we have σ(µ + µ − → Z ′ * → N N ) ≃ 4.7 fb.On the other hand, a heavy Z ′ leads to In this scenario, the cross section is suppressed by the large value of m Z ′ .While single production could probe m N ≲ √ s, the pair production of heavy neutral lepton is kine-  matically allowed when m N < √ s/2.So the 3 TeV muon collider could only probe m N ≲ 1500 GeV.For heavier m N , we need a more energetic muon collider.

With Associated Photon
The photon from initial state radiation carries part of the collision energy, which results in the gauge boson Z ′ produced on-shell when m Z ′ ≲ √ s.In this case, there is a resonance peak in the invariant mass spectrum of the dilepton [39].Here, we study the pair production of heavy neutral leptons with associated photon process as Here, the scenario with off-shell Z ′ contribution is also considered when m In Figure 4, we show the cross section of µ + µ − → N N γ at a 3 TeV muon collider, where the numerical results are calculated by MadGraph.There is also a sharp peak around m Z ′ ≃ √ s.To avoid the soft photon singularity, the detected photon is required to satisfy the following pre-selection cuts When m Z ′ > 2m N , the on-shell production of Z ′ followed by the cascade decay Z ′ → N N can notably enhance the cross section of µ + µ − → N N γ.Using the narrowwidth approximation, the cross section can be expressed as [55] σ(µ Because the branching ratio of Z ′ → N N is independent of the new gauge coupling g ′ , the cross section of µ + µ − → Z ′ γ → N N γ is proportional to e 2 g ′2 , while the cross section of off-shell process µ + µ − → Z ′( * ) γ → N N γ is proportional to e 2 g ′4 .So we observe that the enhancement effect becomes larger when the gauge coupling g ′ is smaller.For g ′ = 0.1, m N = 100 GeV and m Z ′ = 1000 GeV, we have σ(µ ) is less than 0.1 fb, therefore the former process is expected more promising at the muon collider.

Lepton Number Violation Signatures
Pair production of heavy neutral lepton followed by various cascade decay modes could lead to many interesting signatures, such as monolepton, dilepton, and trilepton signals [56].In this paper, we focus on the most distinct lepton number violation same-sign dimuon signal, which also has a much cleaner background compared with the lepton number conserving one.The simulation procedure is as follows.The FeynRules package [57] is used to implement the model.The parton level events are simulated with MadGraph5 aMC@NLO [58].We then use PYTHIA 8 [59] for parton shower.The detector effects are included by employing Delphes 3 [60] with the detector card of the muon collider.The W boson from the heavy neutral lepton decay could be highly boosted, so the two jets from cascade W decay merge into one fat-jet J.We use the Valencia algorithm [61] with R = 1.2 to reconstruct the fat-jets.

Without Associated Photon
The full production process of the same-sign dimuon signature without associated photon is with the hadronic decays of W .The standard model backgrounds come from processes such as 2) The contributions of µ + µ − W + W − and µ + µ − W + W − γ/Z are from lepton charge misidentified, which are suppressed by the misidentification rate 0.1% [15].The W ± W ± W ∓ µ ∓ ν is dominant by the vector boson fusion process, where the two same-sign W bosons decay leptonically.There is also one possible background process µ + µ − → W + W − jj with the light jets mistagged.However, by requiring the light jets masses close to the W mass, this background can be easily suppressed [15].Meanwhile, the t tW ± µ ∓ ν process also contributes to the same-sign dimuon signature, which can be further reduced by cut on the opposite-sign lepton [23].So we do not include contributions of the µ + µ − → W + W − jj and t tW ± µ ∓ ν processes in this paper.We first apply the following pre-selection cuts on the transverse momentum and pseudorapidity of the muon and the fat-jets In Figure 5, we show the distributions of some variables for the µ ± µ ± JJ signal and backgrounds after applying the pre-selection cuts.We have set m N = 1000 GeV, m Z ′ = 2500 GeV and g ′ = 0.6 as the benchmark point of the signal.The cut flow for the µ ± µ ± JJ signature and backgrounds are summarized in Table 1.
The lepton number violation signature is satisfied for events with two same-sign muons.In the single production of the heavy neutral lepton process [23,24], only one boosted W is expected from N decay.To distinguish from the single production and also suppress the background from vector boson fusion processes, we require exact two fat-jets in the final states although tagging one fat-jet has a larger efficiency for the signal.In order to be identified as W bosons, the fat-jets masses are also required in the following range Because there are two heavy neutral leptons in the signal, we reconstruct their masses through the two muons and two fat-jets system by minimizing [15].We require that both the invariant mass of muon and fat-jet m µJ from reconstructed N satisfy 0.8m According to the distributions in Figure 5, we may tighten the cuts on P T (µ ± ) and P T (J) to suppress the background.However, we see from Table 1 that the cut on the invariant mass m µJ is quite efficient to reduce the background.After applying all these cuts, the total cross section of the background is about 1.2 × 10 −3 fb.So with an integrated luminosity of 1000 fb −1 , there are only 1.2 background events, while the signal events is over 10 4 .In this way, the significance will be 116.7 for the benchmark point with 1000 fb −1 data, where the significance is calculated as Here, N S and N B are the event number of signal and background respectively.To reach the 5σ discovery limit, we need only 1.8 fb −1 data.Based on the above analysis, we then explore the sensitivity of the µ ± µ ± JJ signature at the 3 TeV muon collider.The results are shown in Figure 6.There are three free parameters as g ′ , m Z ′ and m N related to the collider phenomenology.With fixed mass relation m N = m Z ′ /3, we could probe m Z ′ approximately in the range of [330,4500] GeV.Around the resonance region m Z ′ ∼ 3000 GeV, the gauge coupling g ′ could be down to about 0.03 with 1000 fb −1 data.We also find that for m N ≲ 300 GeV, the cut efficiency decreases quickly as m N becomes smaller.One main reason is that the heavy neutral leptons are also highly boosted at the 3 TeV muon collider for such light m N , so the muons and W -jets from boosted N decays are mostly non-isolated [63].One may use the substructure based variables as lepton sub-jet fraction (LSF) and lepton mass drop (LMD) [64] to probe the light m N , which is beyond the scope of this study.
For pair production of heavy neutral leptons without associated photon signature, the gauge boson Z ′ does not need to be heavier than 2m N .In the up-right panel of Figure 6, we have fixed m N = 1000 GeV for illustration.The production cross section of µ + µ − → N N   In the down panel, we fix the gauge coupling g ′ = 0.6.The dark-green region corresponds to sensitivity with 1 fb −1 data, while the light-green region corresponds to sensitivity with 1000 fb −1 data.The orange region is excluded by neutrino trident production at CCFR [30].The magenta region is not allowed kinematically.The gray line is the projected sensitivity at HL-LHC [62].The red line is the sensitivity of the 3 TeV muon collider [39].
is approximately a constant for light Z ′ , so g ′ ∼ 0.15 is nearly independent of m Z ′ when Z ′ is lighter than 1 TeV.This channel is most sensitive for large m Z ′ region.For instance, we may probe m Z ′ ∼ 10 TeV with g ′ ≳ 0.5 and m N ∼ 1000 GeV at the 3 TeV muon collider.
Near the resonance region with m Z ′ ∼ √ s, we even can have a promising signature when g ′ ≳ 0.2 with only 1 fb −1 data.
We then fix the gauge coupling g ′ = 0.6 and explore the sensitivity region on the m N − m Z ′ plane.With 1 fb −1 data, a large parameter space near the resonance region with 2 TeV ≲ m Z ′ ≲ 4 TeV can be probed.The most sensitive heavy neutral lepton mass is around 500 GeV.Because a lighter m N leads to smaller acceptance efficiency, while a larger m N has a smaller production cross section.With an integrated luminosity of 1000 fb −1 , the region with m Z ′ ≲ 12 TeV and 150 GeV ≲ m N < 1500 GeV is within the reach of the 3 TeV muon collider.

With Associated Photon
The full production process of the same-sign dimuon signature with associated photon is with the hadronic decays of W .The standard model backgrounds come from processes such as During the simulation, the pre-selection cuts on associated photon in Eqn (3.5) are also applied.Based on the cut-flow in Table 1, we expect that the cross section of the process µ + µ − W + W − γγ/Z with one additional γ/Z gauge boson is about an order of magnitude smaller than the process µ + µ − W + W − γ.Therefore, σ(µ + µ − W + W − γγ/Z) is approximately 10 −5 fb after all cuts, which will not be considered in the analysis of the µ ± µ ± JJγ signal.
The cross sections of the background processes with associated photons are about one order of magnitude smaller than those without associated photons.If we apply exactly the same selection cuts as the previous µ ± µ ± JJ signal, the total cross section of background is expected at the order of 10 −4 fb, which will have about 0.1 background event even with a total of 1000 fb −1 data.Therefore, we can loosen the selection cut to keep more signal events.In the analysis of the µ ± µ ± JJγ signal, we consider that at least one fat-jet is detected in the final states while we further require one detected photon in this signal.In Figure 7, we show the distributions of some variables for the µ ± µ ± JJγ signal and backgrounds after applying the pre-selection cuts.The benchmark point for the µ ± µ ± JJγ signal is the same as the previous µ ± µ ± JJ signal.With quite similar distributions of P T (µ ± ), P T (J), m J and m µJ , we then apply the same selection cuts for these variables as the µ ± µ ± JJ signal.The cut flow for the µ ± µ ± JJγ signature and backgrounds are summarized in Table 2.
Although with one additional photon, the cross section of the µ ± µ ± JJγ signal is in the same order as the µ ± µ ± JJ signal.After all selection cuts, the cross section of the µ ± µ ± JJγ process is 8.86 fb.With a loose cut on the fat-jet number, the total cross section of the background is about 1.5 × 10 −3 fb after all cuts.With an integrated luminosity of 1000 fb −1 , the significance of the µ ± µ ± JJγ signal could be over 90 for the benchmark point.Meanwhile, 2.9 fb −1 data is enough to discover the benchmark point at the 5σ level.
Besides the invariant mass of dilepton m ℓ + ℓ − (ℓ = µ, τ ), there are two more pathways to confirm the on-shell production of gauge boson Z ′ in the µ ± µ ± JJγ signature.For the two-body process µ + µ − → Z ′ γ, the photon energy is connected to the gauge boson mass as [39] For the benchmark point with m Z ′ = 2500 GeV at the 3 TeV muon collider, the peak value of E γ is about 458 GeV.The sharp bump above the background is clearly shown in the down-left panel of Figure 7. On the other hand, we can also measure m Z ′ through the invariant mass of two reconstructed heavy neutral leptons m N N , which also has a resonance near m Z ′ .From the distribution of m N N in the down-right panel of Figure 7, the background process µ + µ − W + W − γ and W ± W ± W ∓ γµ ∓ ν have peaks around 3 TeV and 1.5 TeV respectively.Anyway, with less than two background events after all cuts, the backgrounds have little impact on the distribution of m N N for the signal.The sensitivity of the µ ± µ ± JJγ signature at the 3 TeV muon collider is explored with the cuts in Table 2, and Figure 8 depicts the results.The up-left panel shows the sensitivity region with fixed mass relation m N = m Z ′ /3, so the decay channel Z ′ → N N is always allowed.In this way, we might probe m Z ′ in the range of [340, 4300] GeV.Compared with the µ ± µ ± JJ signature, we find that this signal is more promising approximately in the region of [400, 3000] GeV.For example, with m Z ′ = 2000 GeV, the µ ± µ ± JJγ signature could probe g ′ ≳ 0.02, while the µ ± µ ± JJ signature could only reach g ′ ∼ 0.1.Near m Z ′ ∼ √ s = 3 TeV, we even can reveal g ′ ≳ 0.006.When m Z ′ is larger than the collision energy √ s, the associated photon with the off-shell contribution of Z ′ makes the µ ± µ ± JJγ signature less detectable than the µ ± µ ± JJ signal.
The up-right panel of Figure 8 shows the sensitivity region with fixed m N = 1000 GeV.When m Z ′ ≤ 2m N , the decay channel Z ′ → N N is not allowed, so production of N N γ is through the off-shell Z ′ .With nearly a constant cross section, the µ ± µ ± JJγ signature could test g ′ ≳ 0.26 for m Z ′ ≤ 2000 GeV, which is clearly less promising than the µ ± µ ± JJ signature.However, once the decay mode Z ′ → N N is open, the cross section of N N γ is enhanced, so the µ ± µ ± JJγ signal becomes quite promising.According to our simulation, we find that this signature could probe g ′ ≳ 0.15 with only 1 fb −1 data when 2000 GeV ≲ m Z ′ ≲ √ s.The sensitivity region in the m N − m Z ′ with fixed g ′ = 0.6 is shown in the down panel of Figure 8.With 1 fb −1 data, a large portion of the parameter space with 2m N ≲ m Z ′ ≲ √ s could be tested.With a similar reason as the µ ± µ ± JJ signature, the most sensitive m N of the µ ± µ ± JJγ signature is also approximately 500 GeV.Outside the above resonance Z ′ region, the significance of the µ ± µ ± JJγ signature decreases due to the associated photon.Although less promising than the µ ± µ ± JJ signature, the µ ± µ ± JJγ signal could probe the region with m Z ′ ≲ 7.6 TeV and 120 GeV ≲ m N ≲ 1470 GeV at the 3 TeV muon collider with an integrated luminosity of 1000 fb −1 .

Conclusion
Heavy neutral leptons N are introduced in the type-I seesaw mechanism to generate tiny neutrino masses.While electroweak scale N is extensively studied at colliders, a relatively large mixing parameter V ℓN is usually required.In this paper, we consider the gauged U (1) Lµ−Lτ extension of the type-I seesaw mechanism, where three heavy neutral lepton N e , N ν , N τ with U (1) Lµ−Lτ charge (0, 1, −1) are introduced.Charged under the new gauged U (1) Lµ−Lτ symmetry, there are new production channels of the heavy neutral lepton at colliders.We then investigate the lepton number violation signatures of heavy neutral lepton N in the gauged U (1) Lµ−Lτ at the 3 TeV muon collider.
Mediated by the new gauge boson Z ′ , the heavy neutral lepton N can be pair produced at the muon collider via the process µ + µ − → Z ′ * → N N and µ + µ − → Z ′( * ) γ → N N γ.Cross sections of these processes are determined by the new gauge coupling g ′ , gauge boson mass m Z ′ and heavy neutral lepton mass m N , which are independent of the mixing parameter V ℓN .While both processes have a maximum cross section at m Z ′ ≃ √ s, the former one is more promising when m Z ′ > √ s.For lighter new gauge boson satisfying 2m N < m Z ′ < √ s, Z ′ is produced on-shell with associated photon, and the decay mode Z ′ → N N is also allowed.In this way, the cross section of µ + µ − → N N γ can be enhanced.
Cascade decays of heavy neutral lepton can raise various interesting signatures.Provided the Majorana nature of heavy neutral lepton in the seesaw model, we focus on the lepton number violation signatures in this study.For illustration, we further assume that the heavy neutral lepton N preferentially couples to muon via mixing with light neutrinos.So the dominant decay mode of heavy neutral lepton is N → µ ± W ∓ .The hadronic decays of W boson are considered to reconstruct m N , which are treated as one fat-jet J.The explicit signatures are µ + µ − → N N → µ ± µ ± JJ and µ + µ − → N N γ → µ ± µ ± JJγ.
With relatively clean backgrounds, a cut-based analysis is then performed, which indicates that a large part of the viable parameter space is within the reach of the 3 TeV muon collider.For example, the µ ± µ ± JJ could probe m Z ′ in the range of [330,4500] GeV with fixed mass relation m N = m Z ′ /3 and an integrated luminosity of 1000 fb −1 .Around the resonance region m Z ′ ∼ √ s, the gauge coupling g ′ could be down to about 0.03.We can probe g ′ ≳ 0.15 with m N = 1000 GeV and light Z ′ .On the other hand, this signal could probe m Z ′ ≲ 12.5 TeV with g ′ = 0.6.Meanwhile, if the gauge boson mass satisfies 2m N < m Z ′ < √ s, the µ ± µ ± JJγ signature would be more promising than the µ ± µ ± JJ signature.It is also notable that around the resonance region m Z ′ ∼ √ s, both signatures could probe sizable parameter space even with only 1 fb −1 data for g ′ = 0.6.

Figure 3 .
Figure 3. Cross section of µ + µ − → Z ′ * → N N at a 3 TeV muon collider.We have fixed m N = 100 GeV in the left panel and m Z ′ = 2000 GeV in the right panel.

Figure 4 .
Figure 4. Cross section of µ + µ − → N N γ at a 3 TeV muon collider.We have fixed m N = 100 GeV in the left panel and m Z ′ = 2000 GeV in the right panel.

Figure 5 .
Figure 5. Normalized distribution of transverse momentum of muon P T (µ ± ) (up-left panel), transverse momentum of fat-jet P T (J) (up-right panel), fat-jet mass m J (down-left panel), and invariant mass of muon and fat-jet m µJ (down-right panel) for the µ ± µ ± JJ signature and corresponding backgrounds.

6 m N > s / 2 μFigure 6 .
Figure 6.Sensitivity of the µ ± µ ± JJ signature at the 3 TeV muon collider.In the up-left panel, we fix the mass relation with m N = m Z ′ /3.In the up-right panel, we set m N = 1000 GeV.In the down panel, we fix the gauge coupling g ′ = 0.6.The dark-green region corresponds to sensitivity with 1 fb −1 data, while the light-green region corresponds to sensitivity with 1000 fb −1 data.The orange region is excluded by neutrino trident production at CCFR[30].The magenta region is not allowed kinematically.The gray line is the projected sensitivity at HL-LHC[62].The red line is the sensitivity of the 3 TeV muon collider[39].

Figure 7 .
Figure 7. Normalized distribution of transverse momentum of muon P T (µ ± ) (up-left panel), transverse momentum of fat-jet P T (J) (up-right panel), fat-jet mass m J (middle-left panel), and invariant mass of muon and fat-jet m µJ (middle-right panel), energy of photon E γ (down-left panel) and invariant mass of reconstructed heavy neutral lepton pair m N N (down-right panel) for the µ ± µ ± JJγ signature and corresponding backgrounds.

Table 1 .
Cut flow table for the µ ± µ ± JJ signal at the 3 TeV muon collider and corresponding backgrounds.

Table 2 .
Cut flow table for the µ ± µ ± JJγ signal at the 3 TeV muon collider and corresponding backgrounds.