Holographic operator product expansion of loop operators in the ${{ \mathcal N} {\bf{=4}}\sim {\boldsymbol {S O(N)}}}$ super Yang-Mills theory^*

The holographic duality between the maximally supersymmetric Yang-Mills theory (SYM) with the $S U(N)$ gauge group and Type IIB string theory on the $AdS_{5}\times S^{5}$ background is the most studied example of the AdS/CFT correspondence [1]. The vacuum expectation values of Wilson loops are natural observables in gauge theories, and they are also calculable from the AdS side. In the string theory description, a Wilson loop ^① in the fundamental representation is related to a fundamental string with the worldsheet ending on the AdS boundary along the contour of this Wilson loop [2, 3]. The on-shell action, with the boundary terms from the Legendre transformation [4], yields the prediction for the vacuum expectation value (vev) of this Wilson loop at large N and large 't Hooft coupling $\lambda\equiv g_{{\rm YM}}^{2}N$, when the classical string theory becomes a good approximation with large string tension and small curvature. This holographic prediction matches the field theory results in the large N and λ limit. The field theory results were obtained based on the conjecture that the computations can be reduced to the ones in the Gaussian matrix model [5]. Later, this conjecture was proved using supersymmetric localization [6]. This match provided a highly non-trivial check of the AdS/CFT conjecture since the vev of a Wilson loop is a non-trival function of λ and N. Higher-rank Wilson loops in gauge theories are dual to D-branes carrying electric flux on the their worldvolume [7−10]. When the rank of the representation is sufficiently high, the back reaction from the D-branes must be considered. A Wilson loop in the higher-rank representation with mixed symmetries is dual to a certain bubbling supergravity solution [11−13]. We will not discuss such supergravity solutions in this paper.

Specifically, half-BPS circular Wilson loops in the rank-k symmetric representation of the gauge group correspond to a D3-brane with the $AdS_{2}\times S^{2}$ worldvolume and k units of fundamental string charge [7]. Half-BPS circular Wilson loops in the rank-k anti-symmetric representation of the gauge group have a bulk description in terms of the $AdS_{2}\times S^{4}$ D5-brane with k units of fundamental string charge [8]. These D-branes are $1/2$-BPS and preserve the same $S O(2,1)\times S O(3)\times S O(5)$ isometries. While a 't Hooft loop, which is the magnetic dual of a Wilson loop, can be obtained using S-duality in $\mathcal{N}=4$ SYM. A general $SL(2, \mathbf{Z})$ transformation maps a Wilson loop to a Wilson-'t Hooft (WH) loop [14]. It was proposed in [15] that a WH loop in symmetric representations of both the gauge group and its Goddard-Nuyts-Olive (GNO) dual group [16] (the Langlands dual group) is dual to a D3-brane carrying both F-string and D-string charges. More details on such WH loops will be provided later in this section.

A circular Wilson loop can be expanded in a series of local operators with different conformal dimensions, when the probing distance is much larger than the radius of this loop. Half-BPS chiral primary operators (CPOs) are an important class of operators with protected dimensions appearing in this operator product expansion (OPE). The OPE coefficient can be extracted from the correlation function of a Wilson loop and local operators [17]. In the large N and λ limit, the correlation function of a Wilson loop in the fundamental representation with a CPO can be derived by calculating the coupling of the supergravity modes dual to this CPO to the string worldsheet [17]. Similar procedure can be used to compute the correlator of a higher rank Wilson loop with a CPO using D3$_{k}$ and D5$_{k}$ branes and replacing the string worldsheet by the brane worldvolume [18]. These results were confirmed by the field theory side using the matrix model [18, 19]. The reduction to this matrix model computations was later confirmed by supersymmetric localization [20].

The $\mathcal{N}=4$ SYM theory with the gauge group $S O(N)$ has some features different from the $S U(N)$ theory. For odd N, the group is non-simply-laced, and the S-dual theory has the gauge algebra $sp(\dfrac{N-1}2)$ [16]. In this case, the gauge algebras before and after the S-duality transformation are different. This is distinct from the S-duality transformation of the theory with the gauge group $S U(N)$. For even N, the group $S O(N)$ is simply-laced and the dual theory still has the gauge algebra $spin(N)$. Another notable feature regarding Wilson loops in $S O(N)$ theories is the presence of Wilson loops in spinor representations.

In the string theory, ${\mathcal{N}}=4~ S O(N)$ SYM can be realized as the low energy effective theory of coincident D3-branes atop a suitable O3 plane. Based on this, Witten proposed that the ${\mathcal{N}}=4~ S O(N)$ SYM is holographically dual to the string theory on the $AdS_{5}\times\mathbf{RP}^{5}$ orientifold [21]. The five-dimensional real projective space $\mathbf{RP}^{5}$ is obtained by the five-dimensional sphere $S^{5}$ by identifying antipodal points, $\mathbf{RP}^5=S^5/\mathbf {Z}_2$. This correspondence was recently studied in [22]. It has been demonstrated that the expectation value of the Wilson loop in the spinor representation of the gauge group, calculated through supersymmetric localization [22, 23], precisely matches the result obtained from the D5-brane, with its worldvolume including the $\mathbf{RP}^{4}$ subspace of $\mathbf{RP}^{5}$. The holographic descriptions of Wilson loops in the fundamental, symmetric, and anti-symmetric representations were also studied, and the holographic predictions of their vevs exactly matched the results of supersymmetric localization [22,23]. In this study, we compute the correlation functions of Wilson(-'t Hooft) loops with CPOs of $\mathcal{N}=4$SYM with $S O(N)$gauge symmetry. The considered line operators include the following:

● Half-BPS circular Wilson loops in the fundamental representation of the Lie algebra $g=spin(N)$, $W_\Box$.

● Half-BPS circular Wilson loops in the k-th anti-symmetric representation of g, $W_{A_k}$.

● Half-BPS circular Wilson loops in the spinor representation of g, $W_{sp}$.

● Special half-BPS circular WH loops. Recall that WH loops [14] are labelled by $(\lambda_{\rm elec.}, \lambda_{\rm mag.})\in \Lambda_w\times \Lambda_{mw}$ with the identification

$$\begin{equation} (\lambda_{\rm elec.}, \lambda_{\rm mag.}) \sim (w\lambda_{\rm elec.}, w\lambda_{\rm mag.}), w\in W. \end{equation} \tag{ 1 }$$

Here, $\Lambda_w$ and $\Lambda_{mw}$ are the weight lattices of g and $^Lg$, respectively, $^L g$ is the GNO dual group [16] of g ^② , and W is the Weyl group of g and $^Lg$. We focus on the case in which the W-orbit $[\lambda_{\rm elec.}]$ corresponds to the n-th symmetric representation of g and the W-orbit $[\lambda_{\rm mag.}]$ corresponds to the m-th symmetric representation of $^Lg$. We label these WH loops by ${WH}_{S_n, S_m}$'s.

The paper is organized as follows. In Sections II and III, we briefly review the dual string description of the ${\mathcal{N}}=4 SO(N)$ theory and the half-BPS CPOs with their gravity duals. In Sections IV, V, VI, and VII, we compute the OPE coefficients of these CPOs in the OPE expansion of the Wilson loops in the fundamental representation, the WH loops in the symmetric representation, the Wilson loops in the anti-fundamental representation, and the Wilson loops in the spinor representation, respectively. The final section lists our conclusions and provides a discussion. In Appendix A, we briefly discuss the coefficient of the bulk-to-boundary propagator of a certain mode in $AdS_5$.

Four-dimensional $\mathcal{N}=4$ SYM with the gauge group $S O(N)$ is dual to the Type IIB superstring theory on the $AdS_5\times \mathbf{RP}^5$ background with Ramond-Ramond (RR) $5$-form fluxes $F_5$ [21]. We also choose "discrete torsion'' of the RR $2$-form $B_{RR}$. We will describe this discrete torsion later. In the large N and large 't Hooft coupling limit, the IIB supergravity on $AdS_5\times \mathbf{RP}^5$ is a good approximation of this superstring theory. We set the radius of $AdS_5$, $L_{AdS_5}$ to$1$; then, the metric of $AdS_5\times \mathbf{RP}^5$ is

$$\begin{equation} {\rm d}s^2={\rm d}s_{AdS_5}^2+{\rm d}s_{\mathbf{RP}^5}^2\,. \end{equation} \tag{ 2 }$$

The RR $5$-form flux is

$$\begin{equation} F_5=4 (\omega_5+\tilde{\omega}_5)\,, \end{equation} \tag{ 3 }$$

where $\omega_5$ and $\tilde{\omega}_5$ are the volume forms on $AdS_5$ and $\mathbf{RP}^5$ with unit radius, respectively.

From $L_{AdS_5}=1$, one obtains that [22] in the large N limit,

$$\begin{equation} 4\pi g_s N \alpha^\prime=1\,, \end{equation} \tag{ 4 }$$

which leads to

$$\begin{equation} \alpha^\prime=\sqrt{\frac2{\lambda}}\,, \end{equation} \tag{ 5 }$$

by using the relation $g_{\rm YM}^2=8\pi g_s$ in the $S O(N)$ case [22] and the definition of the 't Hooft coupling $\lambda\equiv g^2_{\rm YM}N$.

The discrete torsions for the Neveu-Schwarz $2$-form $B_{NS}$ and the RR $2$-form $B_{RR}$ are defined through

$$\begin{align} {\rm e}^{2\pi {\rm i}\theta_{NS}} & \equiv \exp\left({\rm i}\int_{\mathbf{RP}^2}B_{NS}\right)=\pm 1\,, \end{align} \tag{ 6 }$$

$$\begin{align} {\rm e}^{2\pi {\rm i} \theta_{RR}} & \equiv \exp\left({\rm i}\int_{\mathbf{RP}^2}B_{RR}\right)=\pm 1\,. \end{align} \tag{ 7 }$$

where we use $\mathbf{RP}^2$ inside $\mathbf{RP}^5$. When $(\theta_{NS}, \theta_{RR})= (0, 0)$ the gauge group of the dual theory is $S O(2n)$. When $(\theta_{NS}, \theta_{RR})=(0, \dfrac12)$, the gauge group of the dual theory is $S O(2n+1)$.

We plan to compute the correlation functions of half-BPS CPOs and various loop operators. These CPOs are constructed using the six scalar fields $\Phi^i, i=1, \cdots, 6$, which are in the adjoint representation of $S O(N)$ and the vector representation of $S O(6)_R$, the R-symmetry group of this theory. Such CPOs are

$$\begin{equation} \mathcal{O}^I=C^I_{i_1\cdots i_l}{\mathrm Tr}_\Box (\Phi^{i_1}\cdots \Phi^{i_l})\,, \end{equation} \tag{ 8 }$$

with $l\ge 2$. Here, the trace is taken in the fundamental representation of $S O(N)$, and $C^I$ is in the traceless l-th totally symmetric representation of $S O(6)_R$. We choose $C^I$ to satisfy

$$\begin{equation} C_{i_1\cdots i_l}^I C^{J i_1\cdots i_l}=\delta^{IJ}\,, \end{equation} \tag{ 9 }$$

here, $C^{Ji_1\cdots i_l}$ is defined as $C^{Ji_1\cdots i_l}=\delta^{i_1j_1}\cdots \delta^{i_l j_l}C^J_{j_1\cdots j_l}$. Since $\Phi^i$'s are $N\times N$ anti-symmetric matrices, l should be even for non-vanishing $\mathcal{O}^I$. This constraint is new compared with the case in which the gauge group is $S U(N)$.

For $l\ll N$, the holographic description of $\mathcal{O}^I$ is expressed in terms of fluctuations of the background fields in the IIB supergravity on $AdS_5\times \mathbf{RP}^5$, ^③

$$\begin{array}{*{20}{l}} G_{\underline{mn}}=g_{\underline{mn}}+h_{\underline{mn}}\,, \end{array} \tag{ 10 }$$

$$\begin{array}{*{20}{l}} F_{\underline{m_1\cdots{m}_5}}=f_{\underline{m_1 \cdots m_5}}+\delta f_{\underline{m_1 \cdots m_5}}\,,\qquad \delta f_{\underline{m_1 \cdots m_5}}=5 \nabla_{[\underline{m}_1}a_{\underline{m_2\cdots m_5}]}\,, \end{array} \tag{ 11 }$$

where $g_{\underline{mn}}$ and $f_{\underline{m_1\cdots m_5}}$ are the background fields (2) and (3), and $h_{\underline{mn}}$ and $\delta f_{\underline{m_1\cdots m_5}}$ are fluctuations.

The fluctuations dual to half-BPS CPOs are [24]

$$\begin{align} h_{\underline{\mu\nu}} & =-\frac{6}{5}\Delta s g_{\underline{\mu\nu}}+\frac{4}{\Delta+1}\nabla_{(\underline{\mu}}\nabla_{\underline{\nu})}s\,, \end{align} \tag{ 12 }$$

$$\begin{align} h_{\underline{\alpha\beta}} & =2 \Delta s g_{\underline{\alpha\beta}}\,, \end{align} \tag{ 13 }$$

$$\begin{align} a_{\underline{\mu_1\cdots\mu_4}} & =-4 \epsilon_{\underline{\mu_1\cdots \mu_5}}\nabla^{\underline{\mu_5}} s\,, \end{align} \tag{ 14 }$$

$$\begin{align} a_{\underline{\alpha_1\cdots \alpha_4}} & =4 \sum\limits_I \epsilon_{\underline{\alpha\alpha_1\cdots \alpha_4}} s^I(x)\nabla^{\underline{\alpha}}Y^I(y)\,. \end{align} \tag{ 15 }$$

Here, $s(x, y)=\sum_I s^I(x)Y^I(y)$ with $x, y$ being coordinates in the $AdS_5$ part and $\mathbf{RP}^5$ part, respectively. $(\underline{\mu\nu})$ in (12) assumes the traceless symmetric part. $Y^I(y)$ is the "scalar spherical harmonics'' on $\mathbf{RP}^5$ satisfying,

$$\begin{equation} \nabla^{\underline{\alpha}}\nabla_{\underline{\alpha}}Y^I=-\Delta (\Delta+4) Y^I\,. \end{equation} \tag{ 16 }$$

They are in the $[0, \Delta, 0]$ representation of $S O(6)_R$, and we choose the normalization of $Y^I$ to be the same as the one in [24]. Since $\mathbf{RP}^5=S^5/\mathbf{Z}_2$, locally $Y^I$ is the same as the scalar spherical harmonics on $S^5$. Δ is dual to the conformal dimension of the CPO. For the case at hand, we have $\Delta=l$ since it is protected by supersymmetry. Recall that l should be even. In the supergravity side, this is owing to the fact that the $\mathbf{Z}_2$ projection of the fields on $AdS_5\times S^5$ gives the fields on $AdS_5\times \mathbf{RP}^5$. $\epsilon_{\underline{\mu_1\cdots \mu_5}}$ and $\epsilon_{\underline{\alpha_1\cdots \alpha_5}}$ are the anti-symmetric tensors corresponding to the volume form of $AdS_5$ and $\mathbf{RP}^5$, respectively. The background five-form field strength can then be expressed as

$$\begin{equation} F_{\underline{\mu_1\cdots \mu_5}} =-4\epsilon_{\underline{\mu_1\cdots \mu_5}}\,, \qquad F_{\underline{\alpha_1\cdots \alpha_5}} =-4\epsilon_{\underline{\alpha_1\cdots \alpha_5}}\,. \end{equation} \tag{ 17 }$$

We consider the half-BPS Wilson loop in the $S O(N)$ theory in Euclidean space $\mathbf{R}^4$,

$$\begin{equation} W_\Box[C]=\frac{1}{N} {\mathrm Tr}_\Box \mathcal{P} \exp \left[\oint_C {\rm i}\Big(A_\mu(x) \dot{x}^\mu+{\rm i} |\dot{x}|\Theta_j \Phi^j(x)\Big){\rm d}s\right], \end{equation} \tag{ 18 }$$

where the contour C is $x^\mu(s)=(a \cos s, a\sin s, 0, 0)$, $\dot{x}^\mu=\dfrac{\partial x^\mu}{\partial s}$, and $\Theta^j$ is a constant unit $6$-vector. The trace is taken in the fundamental representation. For the dual description, we use the Euclidean $AdS_5$ ($EAdS_5$) in the Poincarè coordinates, such that the metric is

$$\begin{equation} {\rm d}s^2=\frac{1}{z^2}({\rm d}z^2+{\rm d}x^{\underline{i}} {\rm d}x_{\underline{i}})\,. \end{equation} \tag{ 19 }$$

The action of the fundamental string (F-string) is

$$\begin{equation} S=\frac{1}{2\pi\alpha^\prime} \int {\rm d}^2\sigma \sqrt{{\rm det}g_{\mu\nu}}\,, \end{equation} \tag{ 20 }$$

with the induced metric $g_{\mu\nu}$ being

$$\begin{equation} g_{\mu\nu}=\frac{\partial x^{\underline{\rho}}}{\partial \sigma^\mu} \frac{\partial x^{\underline{\kappa}}}{\partial \sigma^\mu}g_{\underline{\rho\kappa}}\,. \end{equation} \tag{ 21 }$$

As for the F-string solution dual to the circular Wilson loop, we choose the worldsheet coordinates to be $(z, s)$. The corresponding classical F-string solution can be parameterized as [4, 17]

$$\begin{equation} x^1=\sqrt{a^2-z^2}\cos s\,,\qquad x^2=\sqrt{a^2-z^2}\sin s\,,\qquad x^3=x^4=0\,. \end{equation} \tag{ 22 }$$

The worldsheet of this F-string has the topology of $EAdS_2$ and is entirely embedded within the $EAdS_5$ region of the background geometry. ^④

Taking into account the boundary terms from the Legendre transformation [4], the on-shell action of this F-string is given by [4, 17]

$$\begin{equation} S_{F1}=\frac{1}{2\pi \alpha^\prime} (-2\pi)=-\frac1{\alpha^\prime}\,. \end{equation} \tag{ 23 }$$

Using (5), we get [22]

$$\begin{equation} S_{F1}=-\sqrt{\frac{\lambda}{2}}\,. \end{equation} \tag{ 24 }$$

Thus, the holographic prediction for the vev of the Wilson loop is

$$\begin{equation} \langle W_\Box[C] \rangle= \exp\sqrt{\frac{\lambda}{2}}\,, \end{equation} \tag{ 25 }$$

in the large N and large λ limit.

When probing $W_\Box[C]$ from a distance L much larger than its radius a, the operator product expansion (OPE) of $W_\Box [C]$ is

$$\begin{equation} W_\Box [C]=\langle W_\Box[C]\rangle\left(1+\sum\limits_{i, n}C_i^n a^{\Delta_i^n} O_i^n\right)\,, \end{equation} \tag{ 26 }$$

where $\Delta_i^n$ are the conformal weights of the operator $O_i^n$, $O_i^0$ is the i-th primary field, and $O_i^n$'s with $n \gt 0$ are its conformal descends.

To extract the OPE coefficients of the half-BPS CPOs $O^I$ with normalized two-point functions, we can compute the normalized correlation of this Wilson loop and the half-BPS CPO $\mathcal{O}^I$, ^⑤

$$\begin{equation} \langle \langle \mathcal{O}^I(\boldsymbol{x}) \rangle\rangle_{W_\Box[C]}\equiv\frac{\langle W_\Box[C]\mathcal{O}^I(\boldsymbol{x}) \rangle}{\sqrt{{\mathcal{N}_{\mathcal{O}^I}}}\langle W_\Box[C] \rangle }\,, \end{equation} \tag{ 27 }$$

where $\mathcal{N}_{\mathcal{O}^I}$is defined by the two point function of$\mathcal{O}^I$,

$$\begin{equation} \langle \mathcal{O}^I(\mathbf{y})\mathcal{O}^J(\mathbf{z}) \rangle=\frac{\delta^{IJ}\mathcal{N}_{\mathcal{O}^I}}{|\mathbf{y}-\mathbf{z}|^{2\Delta_{\mathcal{O}^I}}}\,. \end{equation} \tag{ 28 }$$

Taking the OPE limit where $L=\sqrt{\boldsymbol{x}^2}\gg a$, we have

$$\begin{equation} \langle \langle \mathcal{O}^I(\boldsymbol{x}) \rangle\rangle_{W_\Box[C]}=C_{\Box, \mathcal{O}}\frac{a^\Delta}{L^{2\Delta}}\,. \end{equation} \tag{ 29 }$$

The goal is to compute $C_{\Box, \mathcal{O}}$ holographically, which is the OPE coefficient of the primary operator $\mathcal{O}^I$ in the expansion (26).

To achieve this, we need to calculate the change in the F-string action owing to the fluctuations of the background fields dual to $\mathcal{O}^I$ [17],

$$\begin{equation} \delta S_{F1}=\frac{1}{2\pi \alpha^\prime} \int {\rm d}^2 \sigma \sqrt{{\rm det}g_{\mu\nu}}\,\frac{1}{2} g^{\mu\nu} \frac{\partial x^{\underline{\rho}}}{\partial \sigma^\mu} \frac{\partial x^{\underline{\kappa}}}{\partial \sigma^\nu}h_{\underline{\rho\kappa}}\,, \end{equation} \tag{ 30 }$$

where $\sigma^\mu$'s are the worldsheet coordinates and $x^{\underline{\rho}}=x^{\underline{\rho}}(\sigma^\mu)$ expresses how the string worldsheet is embedded in the spacetime.

Then, we write $s^I$ as $s^I({\boldsymbol{x}, z})=\int {\rm d}^4\boldsymbol{x}^\prime G_\Delta(\boldsymbol{x}^\prime; \boldsymbol{x}, z) s_0^I(\boldsymbol{x}^\prime)$; here, $s_0^I$ is a source for $\mathcal{O}^I$ on the boundary, and

$$\begin{equation} G_\Delta(\boldsymbol{x}^\prime; \boldsymbol{x}, z)=c \left(\frac{z}{z^2+|\boldsymbol{x}-\boldsymbol{x}^\prime|^2}\right)^\Delta\,, \end{equation} \tag{ 31 }$$

is the boundary-to-bulk propagator with the constant c being ^⑥

$$\begin{equation} c=\frac{\Delta+1}{2^{(3-\Delta)/2} N\sqrt{\Delta}}\,. \end{equation} \tag{ 32 }$$

Then, the correlation function is given by

$$\begin{equation} \langle \langle \mathcal{O}^I(\boldsymbol{x}) \rangle \rangle_{W_\Box[C]}=-\frac{\delta S_{F1}}{\delta s_0^I(\boldsymbol{x})}\bigg|_{s_0^I=0}\,. \end{equation} \tag{ 33 }$$

In the OPE limit, we have

$$\begin{equation} G_\Delta(\boldsymbol{x}^\prime, \boldsymbol{x}, z)\simeq c\frac{z^\Delta}{L^{2\Delta}}, \end{equation} \tag{ 34 }$$

$$\begin{equation} \partial_{\underline{\mu}}s^I\simeq \delta^{z}_{\underline{\mu}}\frac{\Delta}{z}s^I\,, \end{equation} \tag{ 35 }$$

$$\begin{equation} \partial_{\underline{\mu}}\partial_{\underline{\nu}}s^I\simeq \delta^{z}_{\underline{\mu}}\delta^z_{\underline{\nu}}\frac{\Delta(\Delta-1)}{z^2}s^I\,. \end{equation} \tag{ 36 }$$

We use these and the fact that in the Poincarè coordinates

$$\begin{equation} \Gamma^z_{\underline{\mu\nu}}=z g_{\underline{\mu\nu}}-\frac{2}{z} \delta^z_{\underline{\mu}}\delta^z_{\underline{\nu}}\,. \end{equation} \tag{ 37 }$$

Then, from (12), we get

$$\begin{equation} h_{\underline{\mu\nu}}\simeq -2 \Delta g_{\underline{\mu\nu}}s^I Y^I+\frac{4\Delta}{z^2}\delta^z_{\underline{\mu}}\delta^z_{\underline{\nu}}s^I Y^I\,, \end{equation} \tag{ 38 }$$

The induced metric is

$$\begin{equation} g_{ss}=\frac{a^2-z^2}{z^2}\,,\qquad g_{sz}=0\,,\qquad g_{zz}=\frac{a^2}{z^2(a^2-z^2)}\,. \end{equation} \tag{ 39 }$$

We have

$$\begin{equation} {\rm det}(g_{\mu\nu})=\frac{a^2}{z^4}\,. \end{equation} \tag{ 40 }$$

From these, we obtain

$$\begin{equation} g^{\mu\nu}\frac{\partial x^{\underline{\rho}}}{\partial \sigma^\mu} \frac{\partial x^{\underline{\rho}}}{\partial \sigma^\nu} h_{\underline{\rho\kappa}}=-2 \Delta \frac{z^2}{a^2} s^IY^I\,. \end{equation} \tag{ 41 }$$

Then, the variation of the F-string action is

$$\begin{equation} \delta S_{F1}=-\frac{\Delta Y^I}{\pi \alpha^\prime a} \int {\rm d}^2\sigma s^I \,. \end{equation} \tag{ 42 }$$

Using (31), we get

$$\begin{aligned}[b] \langle \langle \mathcal{O}^I(\boldsymbol{x}) \rangle \rangle_{W_\Box[C]} & =-\frac{\delta S_{F1}}{\delta s_0^I(\boldsymbol{x})}\bigg|_{s_0^I=0}\\ & =\frac{\Delta Y^I(y)c}{\pi \alpha^\prime a L^{2\Delta}}\int {\rm d}^2\sigma z^\Delta\\ & =\frac{\Delta Y^I(y)c}{\pi \alpha^\prime a L^{2\Delta}}\int_0^\pi {\rm d}\psi \int_0^a {\rm d}z z^\Delta\\ & =Y^I(y)\frac{2c\Delta}{\alpha^\prime (\Delta+1)}\frac{a^\Delta}{L^{2\Delta}}\,. \end{aligned} \tag{ 43 }$$

Now, using (5) and (32), we obtain

$$\begin{equation} \langle \langle \mathcal{O}^I(\boldsymbol{x}) \rangle \rangle_{W_\Box[C]}=Y^I(y)2^{\Delta/2-1}\frac{\sqrt{\lambda \Delta}}{N}\frac{a^\Delta}{L^{2\Delta}}\,. \end{equation} \tag{ 44 }$$

Thus, the OPE coefficient is ^⑦

$$\begin{equation} \mathcal{C}_{\Box, \mathcal{O}}=Y^I(y)2^{\Delta/2-1}\frac{\sqrt{\lambda \Delta}}{N}\,. \end{equation} \tag{ 45 }$$

We use the convention that the factor $Y^I(y)$ is not included in the OPE coefficient, which leads to

$$\begin{equation} C_{\Box, \mathcal{O}}=2^{\Delta/2-1}\frac{\sqrt{\lambda \Delta}}{N}\,. \end{equation} \tag{ 46 }$$

The above result expressed in terms of $\lambda, N$, and Δ is identical to the result obtained in the $S U(N)$ case [17]. Since the string worldsheet is an $AdS_2$ subspace completely embedded inside the $AdS_5$ part of the background geometry, the change from $S^5$ to $\mathbf{RP}^5$ does not impact the calculation of the coupling between the supergravity modes and the string worldsheet. The relation between $\alpha^\prime$ and λ in the $S O(N)$ case is $\alpha^\prime=\sqrt{2/\lambda}$, which has an extra factor of $\sqrt{2}$, compared with the relation $\alpha^\prime=\sqrt{\lambda}$ in the $S U(N)$case. The coefficient inside the bulk-to-boundary propagator, c, is $c_{SO}=\sqrt{2}c_{SU}$. These two effects cancel each other; thus, the results of the OPE coefficients in terms of $\lambda, N, \Delta$ are identical for both $S O(N)$ and $S U(N)$. However, one should keep in mind that Δ should be even in the case of $S O(N)$.

In this section, we compute the OPE coefficients of half-BPS circular WH loops in the symmetric representation. WH loops appear owing to the worldlines of dyons that carry both electric and magnetic charges of the gauge theory. In this section, we only consider the case in which the dyons are in the n-th symmetric representation of g and the m-th symmetric representation of $^Lg$. ^⑧ When $m=0$, we get the following Wilson loops in the n-th symmetric representation,

$$\begin{equation} W_{S_n}[C]=\frac{1}{{\rm dim} S_n} {\rm Tr}_{S_n} \mathcal{P} \exp \left[\oint_C {\rm i}\Big(A_\mu(x) \dot{x}^\mu+ {\rm i} |\dot{x}|\Theta_j \Phi^j(x)\Big){\rm d}s\right]\,, \end{equation} \tag{ 47 }$$

where $S_n$ denotes the n-th symmetric representation of $S O(N)$ and ${\rm dim }S_n$ denotes its dimensionality.

Non-trivially generalizing the results in [7], it was proposed in [15] that for the $S U(N)$ case, WH loops are dual to D3-branes in $AdS_5\times S^5$. In [22], a D3-brane dual to a Wilson loop in the symmetric representation for the $S O(N)$ case was given. We expect that generalizing the solution in [15] to the $AdS_5\times \mathbf{RP}^5$ case will provide the dual description of WH loops in the symmetric representation, in the $S O(N)$ case.

We start with the coordinate system in $EAdS_5$, such that the metric takes the form [7]

$$\begin{equation} {\rm d}s^2=\frac{1}{z^2}({\rm d}z^2+{\rm d}r_1^2+r_1^2{\rm d}\psi^2+{\rm d}r_2^2+r_2^2{\rm d}\phi^2)\,. \end{equation} \tag{ 48 }$$

The boundary of the $EAdS_5$ is now at $r\to\infty$ and $\eta=0$. In this coordinate, the $AdS_5$ part of the RR $4$-form potential is

$$\begin{equation} C^{AdS}_4=\frac{r_1r_2}{z^4}\,{\rm d}r_1 \wedge {\rm d}\psi \wedge {\rm d}r_2 \wedge {\rm d}\phi\,. \end{equation} \tag{ 49 }$$

We place the WH loop on the boundary at $r_1=a, r_2=0$. We make the following coordinate transformation:

$$\begin{align} r_1 & =\frac{a \cos\eta}{\cosh\rho-\sinh\rho \cos\theta}\,, \end{align} \tag{ 50 }$$

$$\begin{align} r_2 & =\frac{a \sinh\rho \sin\theta}{\cosh\rho-\sinh\rho\cos\theta}\,, \end{align} \tag{ 51 }$$

$$\begin{align} z & =\frac{ a \sin\eta}{\cosh\rho-\sinh\rho\cos\theta}\,. \end{align} \tag{ 52 }$$

The metric on $EAdS_5$ in this coordinate system is

$$\begin{equation} {\rm d}s^2=\frac{1}{\sin^2\eta}\big[{\rm d}\eta^2+\cos^2\eta {\rm d}\psi^2+\sinh^2\rho({\rm d}\theta^2+\sin^2\theta {\rm d}\phi^2)\big]\,. \end{equation} \tag{ 53 }$$

We only consider the case in which the theta angle in the field theory is zero. This corresponds to setting the background RR zero form potential (the axion), $C_0$, to zero. Then, the action of the D3-brane on the $AdS_5\times \mathbf{RP}^5$ background is

$$\begin{equation} S^{\rm D3}=S^{\rm D3}_{\rm DBI}+S^{\rm D3}_{\rm WZ}\,, \end{equation} \tag{ 54 }$$

where

$$\begin{align} S^{\rm D3}_{\rm DBI} & =T_{\rm D3}\int {\rm d}^4\sigma\sqrt{{\rm det} (g+2\pi \alpha^\prime F)}\,, \end{align} \tag{ 55 }$$

$$\begin{align} S^{\rm D3}_{\rm WZ} & =-T_{\rm D3}\int P[C_4]\,. \end{align} \tag{ 56 }$$

Here, g is the induced metric on the D3-brane, F is the electromagnetic field on the D3-brane worldvolume, $P[C_4]$ is the pull-back of $C_4$ to the worldvolume, and the D3-brane tension reads

$$\begin{equation} T_{\rm D3}=\frac{1}{(2\pi)^3\alpha^{\prime2}g_s}=\frac{N}{2\pi^2}\,, \end{equation} \tag{ 57 }$$

where the relations $\alpha^\prime=\sqrt{2/\lambda}$ and $g_{\rm YM}^2=8\pi g_s$ in the $SO(N)$ case have been used.

For the D3-brane dual to the above WH loop, we take the worldvolume coordinates to be $\rho, \psi, \theta, \phi$, and $\eta=\eta(\rho)$ on the worldvolume. We also need to consider the components $F_{\psi\rho}$ and $F_{\theta\phi}$ of the electromagnetic field strength on the D3-brane worldvolume.

The D3-brane solution, obtained by adjusting the solution in [15] to the $SO(N)$ case, is given by

$$\begin{align} \sin\eta & =\kappa^{-1}\sinh \rho,\qquad \kappa=\sqrt{\frac{n^2\lambda}{32 N^2}+\frac{2\pi^2 m^2}{\lambda}}\,, \end{align} \tag{ 58 }$$

$$\begin{align} F_{\psi\rho} & =\frac{in\lambda}{16\pi N\sinh^2\rho}\,,\qquad F_{\theta\phi}=\frac{m \sin\theta}{2}\,. \end{align} \tag{ 59 }$$

Let us introduce a dual 't Hooft coupling ^⑨ , $\tilde{\lambda}=\dfrac{16\pi^2N^2 n_g}{\lambda}$, where $n_g=1$ for $g=spin(2n)$, and $n_g=2$ for $g=spin(2n+1)$. Then, we can express κ as

$$\begin{equation} \kappa=\frac{1}{4N}\sqrt{\frac{n^2\lambda}{2}+\frac{2m^2\tilde{\lambda}^2}{n_g}}\,. \end{equation} \tag{ 60 }$$

Taking into account the boundary terms, the on-shell action of the D3-brane is

$$\begin{equation} S^{\rm D3}=-2 N (\kappa \sqrt{1+\kappa^2}+\sinh^{-1}\kappa)\,. \end{equation} \tag{ 61 }$$

Thus, the holographic prediction of the vacuum expectation value of the WH is

$$\begin{equation} \langle {WH}_{S_n, S_m}[C]\rangle=\exp\big[2N (\kappa \sqrt{1+\kappa^2}+\sinh^{-1}\kappa)\big]\,. \end{equation} \tag{ 62 }$$

When we take $m=0$, this D3-brane solution becomes the same as the one in [22], though in different coordinates. Furthermore, the holographic prediction for $\langle W_{S_n}\rangle$ is consistent with the results from localization [22] in the large λ limit with κ fixed.

Now, we holographically compute the correlator of ${WH}_{S_n, S_m[C]}$ and $\mathcal{O}^I(\mathbf{x})$

$$\begin{equation} \langle \langle \mathcal{O}^I(\mathbf{x})\rangle\rangle_{{WH}_{S_n, S_m}[C]}\equiv \frac{\langle {WH}_{S_n, S_m}[C]\mathcal{O}^I(\mathbf{x}) \rangle}{\sqrt{\mathcal{N}_{\mathcal{O}}}\langle {WH}_{S_n, S_m}[C] \rangle}\,, \end{equation} \tag{ 63 }$$

in the OPE limit $L\gg a$, and extract the OPE coefficient $C_{WH_{S_n, S_m}, \mathcal{O}}$. The change in $S_{\rm DBI}^{\rm D3}$ due to the fluctuations of the background field is

$$\begin{equation} \delta S_{\rm DBI}^{\rm D3}=T_{\rm D3}\int {\rm d}^4\sigma \sqrt{{\rm det}\mathcal{M}} \,\frac12 (\mathcal{M}^{-1})^{\mu\nu} \frac{\partial x^{\underline{\rho}}}{\partial \sigma^\mu} \frac{\partial x^{\underline{\kappa}}}{\partial \sigma^\nu} h_{\underline{\rho}\underline{\kappa}}\,, \end{equation} \tag{ 64 }$$

where we have defined the matrix $\mathcal{M}=g+2\pi \alpha^\prime F$, and $\sigma^\mu$'s are worldvolume coordinates. By using the result of $h_{\underline{\rho \kappa}}$ in the OPE limit given in (38) and the above D3-brane solution, we obtain

$$\begin{aligned}[b] \delta S_{\rm DBI}^{\rm D3}= & 4N \kappa^2 Y^I \int {\rm d}\rho {\rm d}\theta\, \frac{\sin \theta}{\sinh^2\rho}\\ & \times\left(-1-2\kappa^2+\frac{1-\sinh^2 \rho(\kappa^{-2}-\sin^2\theta)}{(\cosh\rho-\sinh\rho\cos\theta)^2}\right)s^I\,. \end{aligned} \tag{ 65 }$$

Now, we compute the change of $S_{\rm WZ}$ due to the fluctuations of the background fields,

$$\begin{equation} \delta S_{\rm WZ}^{\rm D3}=-T_{\rm D3}\int P[a_4]. \end{equation} \tag{ 66 }$$

From (14), we have

$$\begin{equation} a^I_{\underline{\mu \cdots \mu_4}}=-4\Delta z \epsilon_{\underline{\mu \cdots \mu_4} z} s^IY^I=-4 \Delta C_{\underline{\mu \cdots \mu_4}}s^IY^I. \end{equation} \tag{ 67 }$$

Thus,

$$\begin{equation} \delta S_{\rm WZ}^{\rm D3}=4T_{\rm D3}\Delta Y^I \int P[C_4]s^I\,. \end{equation} \tag{ 68 }$$

From the coordinate transformation (50)−(52), we obtain

$$\begin{equation} \delta S_{\rm WZ}^{\rm D3}=8N\Delta \kappa^4 Y^I \int {\rm d}\rho {\rm d}\theta \,\frac{\sin\theta}{\sinh^2\rho}\left(1 - \frac{1}{\kappa^2}\frac{\sinh\rho \cos\theta}{\cosh\rho-\sinh\rho\cos\theta}\right)\,. \end{equation} \tag{ 69 }$$

Then, the total change of the action is

$$\begin{aligned}[b] \delta S^{\rm D3} = & \delta S_{\rm DBI}^{\rm D3}+\delta S_{\rm WZ}^{\rm D3}=-4N\Delta Y^I\int_0^{\sinh^{-1}\kappa} {\rm d}\rho \\ & \times\int_{0}^{\pi}{\rm d}\theta\frac{\sin\theta}{(\cosh\rho-\sinh\rho\cos\theta)^2}s^I\,. \end{aligned} \tag{ 70 }$$

Using $s^I({\boldsymbol{x}, z})=\int {\rm d}^4\boldsymbol{x}^\prime G_\Delta(\boldsymbol{x}^\prime; \boldsymbol{x}, z) s_0^I(\boldsymbol{x}^\prime)$, we can compute $\langle \langle \mathcal{O}(\mathbf{x})\rangle\rangle_{{\rm WH}_{S_n, S_m }[C]}$ as

$$\begin{equation} \langle \langle \mathcal{O}(\mathbf{x})\rangle\rangle_{WH_{S_n, S_m }[C]}=-\frac{\delta S^{\rm D3}}{\delta s^0(\mathbf{x})}\,. \end{equation} \tag{ 71 }$$

In the OPE limit, we have

$$\begin{aligned}[b] \langle \langle \mathcal{O}(\mathbf{x})\rangle\rangle_{WH_{S_n, S_m }[C]}= & \frac{a^\Delta}{L^{2\Delta}} \frac{4N\Delta c Y^I(y)}{\kappa^\Delta}\int_0^{\sinh^{-1}\rho} {\rm d} \rho \\ & \times \int_0^\pi{\rm d}\theta\frac{\sinh^\Delta\rho\sin\theta}{(\cosh\rho-\sinh\rho\cos\theta)^{2+\Delta}}\,. \end{aligned} \tag{ 72 }$$

Taking the two integrals, we get

$$\begin{equation} \mathcal{C}_{WH_{S_n, S_m}, \mathcal{O}}=\frac{2^{(\Delta+3)/2}}{\sqrt{\Delta}}Y^I(y)\sinh(\Delta \sinh^{-1}\kappa)\,. \end{equation} \tag{ 73 }$$

Thus,

$$\begin{aligned}[b] C_{WH_{S_n, S_m}, \mathcal{O}}= & \frac{2^{(\Delta+3)/2}}{\sqrt{\Delta}}\sinh(\Delta \sinh^{-1}\kappa)\\= & \frac{{\rm i} (-1)^{\Delta/2}2^{(\Delta+3)/2}}{\sqrt{\Delta}}V_\Delta({\rm i}\kappa)\,. \end{aligned} \tag{ 74 }$$

Here, $V_n(x)=\sin (n \cos^{-1}x)$ is one type of the Chebyshev polynomials, and we have used the fact that Δ is even.

The result for the Wilson loop ($m=0$) in terms of κ is $\sqrt{2}$ times the results in [18] for the $S U(N)$ case due to the change of c. ^⑩ Here, we provide a brief explanation on this point. Since the worldvolume of the D3-brane is completely inside $AdS_5$, the calculations of the coupling between the supergravity modes and the D3-brane worldvolume for both $S U(N)$ and $S O(N)$ cases are the same. In the $S O(N)$ case, the relation between $\alpha^\prime$ and λ reads $\alpha^\prime=\sqrt{2/\lambda}$, while the relation $g_{\rm YM}^2=8\pi g_s$ in the $S O(N)$ case is also changed compared with the $S U(N)$ case. However, their effects on $T_{\rm D3}$ cancel each other. The relation between $T_{\rm D3}$ and N, i.e., $T_{\rm D3}=N/(2\pi^2)$, is unchanged. Formally, when we express the results in terms of κ and Δ, the only change is from the coefficient of the bulk-to-boundary propagator $c_{SO}=\sqrt{2}c_{SU}$. This leads to the above conclusion about the OPE coefficients. However, the relation between κ and λ changes in the case of $S O(N)$, becoming

$$\begin{equation} \kappa=\frac{n}{4N} \sqrt{\frac{\lambda}{2}}\,, \end{equation} \tag{ 75 }$$

while for the Wilson loop in the n-th symmetric representation of $spin(N)$ in the $S U(N)$ case, the relation reads

$$\begin{equation} \kappa=\frac{n\sqrt{\lambda}}{4N}\,. \end{equation} \tag{ 76 }$$

Hence, the result in terms of λ and Δ in the $S O(N)$ case is not just a constant multiplying the result in the $S U(N)$ case.

Finally, to compare with the result for $C_{\Box, \mathcal{O}}$ in (46), we set $m=0$ in (74) and take the $\kappa\to 0$ limit. Using $\kappa=\dfrac{n}{4N}\sqrt{\lambda/2}$ in this case, we obtain

$$\begin{equation} C_{W_{S_n}, \mathcal{O}}\simeq 2^{(\Delta+3)/2}\sqrt{\Delta}\kappa=2^{\Delta/2-1}\frac{\sqrt{\Delta \lambda}}{N}n, \end{equation} \tag{ 77 }$$

which is just $nC_{\Box, \mathcal{O}}$, as expected.

Let us consider half-BPS circular Wilson loops in the rank-k anti-symmetric representation of the gauge group $S O(N)$,

$$\begin{equation} W_{A_k}[C]=\frac{1}{{\rm dim} A_k} {\rm Tr}_{A_k} \mathcal{P} \exp \left[\oint_C {\rm i}\Big(A_\mu(x) \dot{x}^\mu+ {\rm i} |\dot{x}|\Theta_j \Phi^j(x)\Big){\rm d}s\right]. \end{equation} \tag{ 78 }$$

They have a bulk description in terms of the D5-brane with k units of fundamental string charge. The worldvolume of this D5-brane has topology $AdS_{2}\times S^{4}$. The D5 description of Wilson loops is valid in the large N and large λ limit with $k/N$ fixed.

We can parameterize the unit $S^5$; $\sum_{i=1}^6z_i^2=1$ as

$$\begin{equation} z_1=\cos\theta,\qquad z_{j+1}=\sin\theta w_j, \, j=1,j=2, \cdots, 5, \end{equation} \tag{ 79 }$$

with $\sum_{j=1}^5w_j^2=1$. Then, the metric of the unit $S^5$ can be written as

$$\begin{equation} {\rm d}\Omega^2_5={\rm d}\theta^2+\sin^2\theta {\rm d}\Omega^2_4\,, \end{equation} \tag{ 80 }$$

with ${\rm d}\Omega^2_4$ as the metric of the unit $S^4$.

$\mathbf{RP}^5$ can be obtained from $S^5$ by identifying antipodal points $z_i\sim -z_i$. One way to realize this is to view $\mathbf{RP}^5$ as the upper hemisphere of $S^5$ ($0\leq \theta\leq \pi/2$) with antipodal points on the equator ($\theta=\pi/2$) identified. The metric of $\mathbf{RP}^5$ is thus given by

$$\begin{equation} {\rm d}s_{\mathbf{RP}^5}^2={\rm d}\theta^2+\sin^2\theta {\rm d}s^{\prime 2}_4\,, \qquad 0\leq \theta\leq \pi/2 \end{equation} \tag{ 81 }$$

where ${\rm d} s^{\prime 2}_4={\rm d}\Omega_4^2$ when $\theta \lt \pi/2$, and ${\rm d}s^{\prime 2}_4$ is the metric of $\mathbf{RP}^4$ when $\theta=\pi/2$.

Hence, the metric of $AdS_{5}\times\mathbf{RP}^{5}$ reads

$$\begin{aligned}[b] {\rm d}s^{2}= & \cosh^{2}u({\rm d}\zeta^{2}+\sinh^{2}\zeta{\rm d}\psi^{2})+{\rm d}u^{2}\\ & +\sinh^{2}u({\rm d}\vartheta^{2}+\sin^{2}\vartheta{\rm d}\phi^{2})+{\rm d}\theta^{2}+\sin^{2}\theta{\rm d}s^{\prime 2}_{4}\,, \end{aligned} \tag{ 82 }$$

with the radius of $AdS_{5}$ and $\mathbf{RP}^{5}$ set to $1$. The $AdS_{5}$ part of the above metric is written in the form of an $AdS_{2}\times S^{2}$ fibration for computational convenience, and these coordinates are related to the one in (48) by the following coordinate transformation:

$$\begin{align} r_{1} & =\frac{a\cosh u\sinh\zeta}{\cosh u\cosh\zeta-\cos\vartheta\sinh u}\,, \end{align} \tag{ 83 }$$

$$\begin{align} r_{2} & =\frac{a\sinh u\sin\vartheta}{\cosh u\cosh\zeta-\cos\vartheta\sinh u}\,, \end{align} \tag{ 84 }$$

$$\begin{align} z & =\frac{a}{\cosh u\cosh\zeta-\cos\vartheta\sinh u}\,, \end{align} \tag{ 85 }$$

where a is the radius of the Wilson loop.

Using the $S O(6)_R$ transformation, we set $\Theta^I$in (78) to $\Theta^I=(1, 0, \cdots, 0)$. Then, in $AdS_{5}\times\mathbf{RP}^{5}$, the D5-brane dual to this antisymmetric Wilson loop occupies the $AdS_{2}$ in the above metric with $u=0$ and wraps an $S^{4}$ submanifold of $\mathbf{RP}^{5}$ at a constant polar angle $\theta_{k}$ (on the upper hemisphere of $S^{5}$) [22]. The D5-brane worldvolume is $AdS_{2}\times S^{4}\subset AdS_5\times \mathbf{RP}^{5}$, and its metric reads

$$\begin{equation} {\rm d}\tilde{s}^{2}={\rm d}\zeta^{2}+\sinh^{2}\zeta{\rm d}\psi^{2}+\sin^{2}\theta_k{\rm d}\Omega_{4}^{2}\, . \end{equation} \tag{ 86 }$$

Turning on the worldvolume $U(1)$ gauge field $F_{\psi\zeta}$ to account for the k units of fundamental brane charge, the action of the D5-brane on the ${\rm AdS}_{5}\times\mathbf{RP}^{5}$ background can be written as

$$\begin{equation} S^{{\rm D5}}=S_{{\rm DBI}}^{{\rm D5}}+S_{{\rm WZ}}^{{\rm D5}}\,, \end{equation} \tag{ 87 }$$

where

$$\begin{align} S_{{\rm DBI}}^{{\rm D5}} & =T_{{\rm D5}}\int{\rm d}^{6}\sigma\sqrt{\det(g+2\pi\alpha^{\prime}F)}, \end{align} \tag{ 88 }$$

$$\begin{align} S_{{\rm WZ}}^{{\rm D5}} & =-2\pi\alpha^{\prime}{\rm i}T_{{\rm D5}}\int F\wedge P[C_{(4)}]. \end{align} \tag{ 89 }$$

In the above equations, the tension of the D5-brane reads

$$\begin{equation} T_{{\rm D}5}=\frac{1}{g_{s}(2\pi)^{5}(\alpha^{\prime})^{3}}=\frac{N}{8\pi^{4}}\sqrt{\frac{\lambda}{2}}\,, \end{equation} \tag{ 90 }$$

and the self-dual 4-form potential is [8]

$$\begin{equation} C_{(4)}=4\Big(\frac{u}{8}-\frac{\sinh4u}{32}\Big){\rm d}H_{2}\wedge{\rm d}\Omega_{2}-\Big(\frac{3}{2}\theta-\sin2\theta+\frac{1}{8}\sin4\theta\Big){\rm d}\Omega_{4}\,. \end{equation} \tag{ 91 }$$

Here, ${\rm d}H_2$ is the volume form of the unit $AdS_2$, $\sinh \zeta {\rm d}\zeta \wedge {\rm d}\psi$, ${\rm d}\Omega_2$ is the volume form of the unit $S^2$, $\sin \vartheta {\rm d} \vartheta \wedge {\rm d}\phi$, and ${\rm d}\Omega_4$ is the volume form of the unit $S^4$.

The fact that the flux of the worldvolume gauge field equals k, together with the brane equations of motion, gives rise to the condition [8] ^⑪

$$\begin{equation} \theta_{k}-\sin\theta_{k}\cos\theta_{k}=\pi\frac{k}{N}\,, \end{equation} \tag{ 92 }$$

and the worldvolume gauge field is

$$\begin{equation} F_{\psi\zeta}=\frac{{\rm i} \sqrt{\lambda/2}\sinh\zeta\cos\theta_{k}}{2\pi}\,. \end{equation} \tag{ 93 }$$

The on-shell D5-brane DBI and WZ action are

$$\begin{align} S_{{\rm DBI}}^{{\rm D5}} & =\frac{2N}{3\pi}\sqrt{\frac{\lambda}{2}}\int{\rm d}\zeta\sinh\zeta\sin^{5}\theta_{k}\,. \end{align} \tag{ 94 }$$

$$\begin{align} S_{{\rm WZ}}^{{\rm D5}} & =\frac{4{\rm i} N}{3}\int{\rm d}\zeta\:F_{\psi\zeta}\Big(\frac{3}{2}\theta-\sin2\theta+\frac{1}{8}\sin4\theta\Big)\,, \end{align} \tag{ 95 }$$

Adding appropriate boundary terms [8], the on-shell action for the D5-brane is

$$\begin{equation} S^{\rm D5}=S_{{\rm DBI}}^{\rm D5}+S_{\rm WZ}^{\rm D5}+S_{\rm bdy}^{\rm D5}=-\frac{2N}{3\pi}\sqrt{\frac{\lambda}{2}}\sin^{3}\theta_{k}\,. \end{equation} \tag{ 96 }$$

Thus, the holographic prediction for the expectation value of the Wilson loop in the rank k antisymmetric representation is given by

$$\begin{equation} \langle W_{A_{k}}\rangle=\exp\Big(\frac{2N}{3\pi}\sqrt{\frac{\lambda}{2}}\sin^{3}\theta_{k}\Big)\,. \end{equation} \tag{ 97 }$$

The variation of the DBI part of the action to the first order in the fluctuation $h_{\mu\nu}$ and $h_{\alpha\beta}$ is

$$\begin{aligned}[b] \delta S_{{\rm DBI}}^{{\rm D5}} = & T_{{\rm D5}}\int{\rm d}^{6}\sigma\sqrt{\det(g+2\pi\alpha^{\prime}F)}\left((g+2\pi\alpha^{\prime}F\right)^{-1})^{mn}\\ & \times \frac{1}{2}(h_{\underline{\mu\nu}}\partial_{m}X^{\underline{\mu}}\partial_{n}X^{\underline{\nu}}+h_{\underline{\alpha\beta}}\partial_{m}X^{\underline{\alpha}}\partial_{n}X^{\underline{\beta}}) \\ = & \frac{N}{3\pi}\sqrt{\frac{\lambda}{2}}\sin^{5}\theta_{k}\int{\rm d}\zeta\,\sinh\zeta\\ & \times\Bigg(\frac{-4\Delta}{\cosh^{2}\zeta\sin^{2}\theta_{k}}+8\Delta\Bigg)s^{\Delta}Y^{\Delta,0}(\theta_{k})\,, \end{aligned} \tag{ 98 }$$

where we have used the D5 solution $z=a/\cosh\zeta$, c.f., (85). The variation of the WZ part of the action to the first order in the fluctuation is given by ^⑫

$$\begin{aligned}[b] \delta S_{{\rm WZ}}^{{\rm D5}} = & -2\pi\alpha^{\prime}{\rm i} T_{{\rm D5}}\int F\wedge P[a_{(4)}] \\ = & -2\pi\alpha^{\prime}{\rm i} T_{{\rm D5}}\int{\rm d}\psi{\rm d}\zeta{\rm d}\sigma_{1}{\rm d}\sigma_{2}{\rm d}\sigma_{3}{\rm d}\sigma_{4}\,\mu(\Omega_{4})\,F_{\psi\zeta}\\ & \times4\sin^{4}\theta S^{I}\partial_{\theta}Y^{I} \\ = & \frac{8N}{3\pi}\sqrt{\frac{\lambda}{2}}\cos\theta_{k}\sin^{4}\theta_{k}\int{\rm d}\zeta\,\sinh\zeta s^{\Delta}\partial_{\theta_{k}}Y^{\Delta,0}(\theta_{k})\,, \end{aligned} \tag{ 99 }$$

where the 4-form fluctuation is given by

$$\begin{equation} a_{\sigma_{1}\sigma_{2}\sigma_{3}\sigma_{4}}=4\sin^{4}\theta\,\mu(\Omega_{4})\sum s^{I}\partial_{\theta}Y^{I} \end{equation} \tag{ 100 }$$

with $\sigma_{1},\sigma_{2},\sigma_{3},\sigma_{4}$ being the coordinates on $S^{4}$ and the corresponding measure $\mu(\Omega_{4})$ is

$$\begin{equation} \mu(\Omega_{4})=\sin^{3}\sigma_{1}\sin^{2}\sigma_{2}\sin\sigma_{3}. \end{equation} \tag{ 101 }$$

Thus, the variation of the D5 action to the first order is given by

$$\begin{equation} \delta S^{{\rm D5}}=\delta S_{{\rm DBI}}^{{\rm D5}}+\delta S_{{\rm WZ}}^{{\rm D5}}\,. \end{equation} \tag{ 102 }$$

The normalized correlation function between the Wilson loop and the CPO is evaluated as

$$\begin{equation} \frac{\langle W_{A_k}(\mathcal{C})\mathcal{O}_{\Delta}(L)\rangle}{\sqrt{\mathcal{N}_{\mathcal{O}}}\langle W(\mathcal{C})\rangle}=-\frac{\delta S_{{\rm D5}}}{\delta s_{0}} \end{equation} \tag{ 103 }$$

Recall that

$$\begin{equation} s^{I}(\vec{x},z)=\int{\rm d}^{4}\vec{x}^{\prime}G_{\Delta}(\vec{x}^{\prime},\vec{x},z)s_{0}^{I}(\vec{x})\,, \end{equation} \tag{ 104 }$$

where the bulk-to-boundary propagator

$$\begin{equation} G_{\Delta}(\vec{x}^{\prime},\vec{x},z)=c\left(\frac{z}{z^{2}+|\vec{x}-\vec{x}^{\prime}|^{2}}\right)^{\Delta}\simeq c\frac{z^{\Delta}}{L^{2\Delta}} \end{equation} \tag{ 105 }$$

and the D5 solution $z=a/\cosh\zeta$. The only integral operation one needs to perform is

$$\begin{equation} \int_{0}^{\infty}{\rm d}\zeta\,\frac{\sinh\zeta}{\cosh^{\Delta+2}\zeta}=\frac{1}{\Delta+1}\,. \end{equation} \tag{ 106 }$$

Hence, we obtain

$$\begin{aligned}[b] \frac{\langle W_{A_{k}}\mathcal{O}_{\Delta}(L)\rangle}{\sqrt{\mathcal{N}_{\mathcal{O}}}\langle W_{A_{k}}\rangle}= & \frac{a^{\Delta}}{L^{2\Delta}}\frac{2^{\Delta/2}\mathcal{N}_{\Delta}}{3\pi}\sqrt{\frac{\lambda}{\Delta}}\frac{\Delta+3}{\Delta-1}\sin^{3}\theta_{k}\\ & \times\big[2(\Delta+1)\cos\theta_{k}C_{\Delta-1}^{(2)}-\Delta C_{\Delta}^{(2)}\big]\,, \end{aligned} \tag{ 107 }$$

where we have used the following results for the $S O(5)$ invariant harmonics [18] ^⑬

$$\begin{equation} Y^{\Delta,0}(\theta_{k})=\mathcal{N}_{\Delta}C_{\Delta}^{(2)}(\cos\theta_{k})\,, \end{equation} \tag{ 108 }$$

and

$$\begin{aligned}[b] & \Delta Y^{\Delta, 0}(\theta_{k})+\frac{\cos\theta_{k}}{\sin\theta_{k}}\partial_{\theta_{k}}Y^{\Delta, 0}(\theta_{k})\\= & \mathcal{N}_{\Delta}\Big[\frac{\Delta}{\sin^{2}\theta_{k}}C_{\Delta}^{(2)}-(\Delta+3)\frac{\cos\theta_{k}}{\sin^{2}\theta_{k}}C_{\Delta-1}^{(2)}\Big]\,. \end{aligned} \tag{ 109 }$$

The normalization factor $\mathcal{N}_{\Delta}$ is obtained from

$$\begin{equation} Y^{\Delta, 0}(0)=\mathcal{N}_{\Delta}C_{\Delta}^{(2)}(1)=\mathcal{N}_{\Delta}\frac{(\Delta+3)!}{6\Delta!}\,, \end{equation} \tag{ 110 }$$

yielding

$$\begin{equation} \mathcal{N}_{\Delta}=\frac{6\Delta!}{(\Delta+3)!}Y^{\Delta, 0}(0)\,. \end{equation} \tag{ 111 }$$

We then obtain

$$\begin{aligned}[b] \frac{\langle W_{A_{k}}\mathcal{O}_{\Delta}(L)\rangle}{\sqrt{\mathcal{N}_{\mathcal{O}}}\langle W_{A_{k}}\rangle}= & \frac{a^{\Delta}}{L^{2\Delta}}Y^{\Delta,0}(0)\frac{2^{\Delta/2}}{3\pi}\sqrt{\Delta\lambda}\frac{6(\Delta-2)!}{(\Delta+1)!}\sin^{3}\theta_{k}\\ & \times\big[2(\Delta+1)\cos\theta_{k}C_{\Delta-1}^{(2)}-\Delta C_{\Delta}^{(2)}\big]\,. \end{aligned} \tag{ 112 }$$

Using the recurrence relation [29, 30]

$$\begin{equation} \Delta C^{(\lambda)}_\Delta (x)= 2 (\Delta +\lambda -1) x C^{(\lambda)}_{\Delta -1}(x) -(\Delta +2\lambda -2) C^{(\lambda)}_{\Delta -2}(x)\,, \end{equation} \tag{ 113 }$$

we finally arrive at

$$\begin{equation} C_{W_{A_k}, \mathcal{O}} = \frac{2^{\Delta/2}}{3\pi}\sqrt{\Delta \lambda} \sin^3\theta_k \frac{6(\Delta -2)!}{(\Delta+1)!} C^{(2)}_{\Delta-2}(\cos\theta_k)\,. \end{equation} \tag{ 114 }$$

This $S O(N)$result is identical to that for the $S U(N)$ case obtained in [18]. The D5-brane worldvolume has topology $AdS_2\times S^4$ with $AdS_2$ in the $AdS_5$ part of the background geometry and $S^4$ in the $\mathbf{RP}^5$ part. Since $\theta_k \lt \pi/2$, the $S^4$ we consider in this case is the same as the $S^4$ embedded in $S^5$ determined by $\theta=\theta_k$ in the parameterization given in (79). Thus, the computation of the coupling of the supergravity modes to the $D5$-brane is the same as that for the $S U(N)$ case, although the expression of $T_{D5}$ in terms of λ and N for the $S O(N)$ case is different from that for the $S U(N)$ case.

In the $S O(N)$ case, we have $T_{D5}=\dfrac{N}{8\pi^4}\sqrt{\dfrac{\lambda}{2}}$, while for the $S U(N)$ case the relation is $T_{D5}=\dfrac{N\sqrt{\lambda}}{8\pi^4}$. Taking this change and the relation $c_{SO}=\sqrt{2}c_{SU}$ into account, we arrive at the conclusion that the OPE coefficients in the $S O(N)$ case are the same as the ones in the $S U(N)$ case. Thus, in the $k\ll N$ limit, the relation $C_{A_k, \mathcal{O}}=k C_{\Box, \mathcal{O}}$ remains the same as that in the $S U(N)$ case. This can be obtained from the result $\theta_k^3\sim 3\pi k/2N$ in this limit and

$$\begin{equation} C^{(2)}_{\Delta-2}(1)=\frac{(\Delta+1)!}{6(\Delta-2)!}\,. \end{equation} \tag{ 115 }$$

Now we turn to the half-BPS circular Wilson loop in the spinor representation S of $SO(N)$,

$$\begin{equation} W_{S}[C]=\frac{1}{{\rm dim} S} {\rm Tr}_{S} \mathcal{P} \exp \left[\oint_C {\rm i}\Big(A_\mu(x) \dot{x}^\mu+{\rm i}\Theta_i \Phi^i(x)\Big){\rm d}s\right]. \end{equation} \tag{ 116 }$$

The dual description of this Wilson loop is in terms of the D5-brane whose worldvolume has topology $AdS_2\times \mathbf{RP}^4$ [21]. If we still chose the $\Phi^I$ to be $\Theta^I=(1, 0, \cdots, 0)$, the embedding of the D5-brane is given by $u=0, \theta=\pi/2$ in the coordinates used in the previous section [22]. In this case, the field strength of the worldvolume $U(1)$ gauge field vanishes. Taking into account the boundary terms, the total on-shell action of this D5-brane is

$$\begin{equation} S^{\rm D5}=-\frac{N}{3\pi} \sqrt{\frac{\lambda}{2}}, \end{equation} \tag{ 117 }$$

so the holographic prediction for the expectation value of the Wilson loop in the spinor representation is [22]

$$\begin{equation} \langle W_S \rangle =\exp\left( \frac{N}{3\pi}\sqrt{\frac{\lambda}{2}}\right). \end{equation} \tag{ 118 }$$

As observed in [22], $F_{\psi\zeta}$, given by (93), vanishes when $\theta_k=\pi/2$. A shortcut to compute the OPE coefficient $C_{S, \mathcal{O}}$ using the result obtained in the previous section is by setting $\theta_k=\pi/2$ in $C_{A_k, \mathcal{O}}$ and dividing the result by $2$ to take into account the change of the D5-brane worldvolume from $AdS_2 \times S^4$ into $AdS_2\times \mathbf{RP}^4$,

$$\begin{aligned}[b] C_{S, \mathcal{O}} & =\frac{1}{2}C_{W_{A_k}, \mathcal{O}} \Big |_{\theta_k=\pi/2} = \frac{2^{\Delta/2}}{3\pi}\sqrt{\Delta \lambda} \frac{6(\Delta -2)!}{(\Delta+1)!} C^{(2)}_{\Delta-2}(0) {}\\ & =\frac{(-2)^{\Delta/2-1}\sqrt{\Delta\lambda}}{\pi(\Delta^2-1)}\,. \end{aligned} \tag{ 119 }$$

Here, we have used the fact that, for even Δ,

$$\begin{equation} C_\Delta^{(2)}(0)=(-1)^{\Delta/2}\frac{\Delta+2}{2}, \end{equation} \tag{ 120 }$$

obtained from the following generating function of the Gegenbauer polynomials $C_\Delta^{(\lambda)}(x)$:

$$\begin{equation} \frac{1}{(1-2xt+t^2)^\lambda}=\sum\limits_{\Delta=0}^\infty C_{\Delta}^{(\lambda)}(x)t^\Delta. \end{equation} \tag{ 121 }$$

In this study, we investigated the holographic duality of the ${\mathcal{N}}=4~ S O(N)$ SYM theory and the Type IIB string theory on the $AdS_5\times \mathbf{RP}^5$ background in the large N and λ limit. To this end, we investigated the OPE coefficients of half-BPS circular Wilson loops in various representations. Wilson loops were expanded in terms of local operators when the probing distances were much larger than the sizes of the Wilson loops. The coefficients were extracted from the expansion for the operators we considered. Our focus was on the half-BPS CPOs and their corresponding gravity duals. Specifically, we computed the correlation functions of local CPOs and the Wilson loops in the fundamental representation, the symmetric representation, the anti-symmetric representation, and the spinor representation. We studied the $S O(N)$ Wilson loops in the symmetric/anti-symmetric representations through their dual D3/D5-brane descriptions. The appearance of the Wilson loops in the spinor representation is a new feature in the $SO(N)$ theories. In addition, we discussed the WH loops in the symmetric representation using a D3-brane with both electric and magnetic charges. The $\mathcal{N}=4$ SYM theory with the gauge group $S O(N)$ has some features different from the $S U(N)$ theory. We compared our results with those of the $\mathcal{N}=4~ S U(N)$ SYM theory.

In this appendix, we compute the coefficient c of the bulk-to-boundary propagator of the modes $s^I$. The action for $s^I$, obtained from the full "actual" action of IIB supergravity [31] is [24]

$$\begin{equation} S=\int_{AdS_5} {\rm d}^5 x \sqrt{{\rm det}(g_{AdS_5})} \frac{1}{2} B_I \left[ \partial_\mu s^I \partial^\mu s^I+\Delta (\Delta-4) (s^I)^2\right]\,, \end{equation} \tag{A1}$$

where $B_I$ is given by

$$\begin{equation} B_I=\frac{16}{\kappa^2} \frac{\Delta(\Delta-1)(\Delta+2)}{\Delta+1}z(\Delta)\,, \end{equation} \tag{A2}$$

where κ is the coupling constant of type IIB supergravity, and $z(\Delta)$ is explained below. Using

$$\begin{equation} 2\kappa^2=(2\pi)^7 g^2\alpha^{\prime 4}\,, \end{equation} \tag{A3}$$

and the relations $\alpha^\prime= \sqrt{2/\lambda}$ and $g_s= g_{\rm YM}^2 / (8\pi)$ for the $S O(N)$ case, we obtain

$$\begin{equation} \kappa^2=\frac{(2\pi)^5}{8N^2}\,, \end{equation} \tag{A4}$$

which is same as the one for the $S U(N)$ case. $z(\Delta)$ is defined by

$$\begin{equation} \int_{\mathbf{RP}^5} {\rm d}^5y\sqrt{{\rm det}(g_{\mathbf{RP}^5})} Y^I Y^J =\delta^{IJ}z(\Delta)\,. \end{equation} \tag{A5}$$

The expression for $z(\Delta)$ is

$$\begin{equation} z(\Delta)=\frac{\pi^3}{2^\Delta (\Delta+1)(\Delta+2)}\,, \end{equation}\tag{A6}$$

which equals to half of the result in the $S U(N)$ case since the integration is over $\mathbf{RP}^5=S^5/\mathbf{Z}_2$. Using the above result, we obtain

$$\begin{equation} B_I=\frac{2^{2-\Delta}N^2\Delta (\Delta-1)}{\pi^2 (\Delta+1)^2}\,. \end{equation} \tag{A7}$$

The coefficient of the bulk-to-boundary propagator is

$$\begin{equation} c=\sqrt{\frac{\alpha_0}{B_I}}\,, \end{equation} \tag{A8}$$

where [17]

$$\begin{equation} \alpha_0=\frac{\Delta-1}{2\pi^2}\,, \end{equation} \tag{A9}$$

which is identical for both $S O(N)$ and $S U(N)$ cases. Finally, we obtain

$$\begin{equation} c=\frac{\Delta+1}{2^{(3-\Delta)/2} N\sqrt{\Delta}}\,, \end{equation}\tag{A10}$$

which equals $\sqrt{2}$ times the result for the $S U(N)$ case.