Pilot study of the use of augmented reality (AR) in rock mechanics

Augmented Reality (AR) is a relatively new and exciting technology that provides a real-world view with additional computer-generated enhancements. This paper presents the results from a pilot study on how AR can be used in rock mechanics and introduces cases linked to the use of AR in fields relevant for rock mechanics. To capture the need from the industry, a workshop was arranged and complemented with open-ended qualitative interviews. Nearly 60 people responded and identified several items as being important information to enhance the real-world view: underground excavations, geological information, geomechanical model, rock support and reinforcement, results from analysis, and measurements. Almost all respondents wanted to use AR for visualization of weak zones and structures. Using AR as a communication tool and for navigation and safety aspects was also highlighted. However, the vision is to visualize real-time analysis results in the field underground and to be able to do real-time adjustments of the design. Future tunnel inspections were suggested to be more strategic. The most difficult issue to solve at the moment is localization and positioning for underground excavations. The identified low-hanging fruit for near-future studies is, therefore, visualization for rock slopes.


Introduction
Augmented Reality (AR) is a relatively new and exciting technology that provides a real-world view with additional computer-generated enhancements. AR is a way to view and interact with applications, display information and get inputs from the user. Using sensors, AR devices can also detect information about the environment. The term AR may be used in combination with (or confused with) Virtual Reality (VR), Mixed Reality (MR) or Extended Reality (XR). In this work, all the above terms are of interest and they are covered under the term "augmented reality". Improved safety, a better basis for decision making, and more objective assessments could be the result if the real-world view can be enhanced and improved with important and necessary information from a rock mechanics perspective. Field studies in which the digital model results from numerical modelling or the scanned environment is visible could provide an important contribution to the design process of underground excavations.
The purpose of this pilot study was to investigate AR technology and increase the knowledge of how AR can be used in the field of rock mechanics. The aim is to give examples on how AR can be used from a rock mechanics perspective. This work does not go into detail on other systems and technologies that support AR with information, such as photogrammetry, unmanned aerial vehicle (UAV, or drone), measurement while drilling (MWD), laser scanning, BIM

Methodology
The activities in this work included a literature and case study review, a workshop, and open-ended qualitative interviews. The state-of-the-art review of AR technology is mainly based on experience from research and innovative work within the field. A comprehensive literature review has been conducted using internet searches in Google, online literature databases (Scopus, Web of Science, Google Scholar, etc.) and publications available at Swedish universities. The focus of the case study review was application of AR in underground excavations. Published cases primarily from Sweden as well as international cases were collected as part of the review. Some of the identified cases were further investigated through interviews.
In order to capture the need from industry, a workshop regarding "Augmented reality in rock mechanics" was arranged in Stockholm on February 4, 2020. Nearly 40 participants (consultants, clients, contractors, researchers, manufacturers) attended and actively discussed and responded during the workshop. The workshop included a mix of lectures and group discussions. The discussions were held in both Swedish and English.
A total of 20 open-ended qualitative interviews were conducted with people working as rock mechanics engineers or similar. The respondents were from different areas (mining, tunneling, energy) and in different roles (design, excavation, client/owner, academia).

AR technology
There are currently two different categories of augmented/mixed reality devices, namely mobile devices like phones and tablets, and headsets. One of the most powerful applications of AR/MR technology is the use of digital twins. A digital twin is simply a virtual copy of a real-world object that is in some way connected to the real-world object. For tracking purposes, external GPS systems and/or calibration points are used. Most indoor positional tracking technologies, if not all, offer Application Programming Interface (APIs) or other ways in which an AR/MR application can communicate with other programs and units.
The processing power of headsets and battery life are presently the biggest limitations of AR/MR. Another limitation is the current AR/MR positional tracking, which is done mostly through the visible light spectrum. In a few years, it is likely that AR headsets and mobile phones with depth sensors will be common in the industry. Techniques other than visual light for positional tracking are under development. A summary of state-of-the-art of AR-technology is presented in table 1.

Case studies
At the time of writing, augmented and mixed reality (AR and MR) are used only to a small extent in Sweden in the tunnelling and mining industries. At the same time, it is thought to become a natural part of the industry in the future, and developments within the area are progressing. There are several papers and reports that discuss the future of mines and how AR can be applied and improve learning, safety and production in mining (e.g., [1], [2], [3], [4], [5], [6]). The primary focus of implementing AR has been on rock engineering, the productional aspect, safety and the worker environment in a mine.
For communication purposes, LlamaZoo has developed MineLife, a platform with AR, VR and a desktop application for visualizing planning data. One of their cases is the Galore Creek project in northwest British Columbia, where they developed a solution for better communication between internal and external stakeholders without having to travel to the site [7]. BGC Engineering used MR, VR and holographic images to communicate the environmental reclamation of an Alberta oil sands mine site [8] as well as in the Giant Mine remediation project [9]. AR technology has been used for segment displacement inspection in reinforced concrete segment tunnels [10]. An AR-based system was established by superimposing BIM models onto a real structure. Due to the difficulty of using GPS or location-based tracking technology underground, a calibration point and tracking system was used. Its precision met the requirement of the quality inspection. AR technology was also applied for the metro tunnel across the Changjiang River in China [10]. To diagnose the displacement between two segments, a marker (calibration point) was placed beside one of the segments. A virtual model was then registered over the real displacement. The experiment results show that the AR-based system meets the accuracy and precision requirements for segment displacement inspection.
In 2007, a virtual representation of a longwall mine and transport roadway were developed as modules for training and education in VR at the School of Minerals and Energy Resources Engineering at UNSW in Sydney [11]. Currently, the school has 20 modules available, which can be visualized in a 3D theatre as well as in headsets. Within the SIMS project [12], a VR environment has been developed, consisting of selected parts of a mine. The physical large-scale VR environment is used for education in mining and rock engineering [13] and is located at Luleå University of Technology (LTU). A study visit, see figure 1, was made to the virtual mine in which the user can walk around experiencing a mine to see ongoing activities. Two VR learning systems focusing on training of fracture mapping and rock mass characterization, as well as how to identify rocks and minerals, have been developed at Aalto University, Finland [14]. Other types of VR-simulators are used to train operators in the different parts of the drill-and-blast cycle [15] and for underground inspections [16], [17].
The FARMIN project, with financing from EIT Raw Materials, aims to develop an augmented reality (AR) solution that visualizes 3D geological data and allows exploration and mining professionals to modify models in the field [18]. Simon Fraser University and the Engineering Geology and Resource Geotechnics Research Group has used MR technology on landslides and open pits [19]. The application of VR and MR for data collection from site characterization was investigated by [20]. In their work, they demonstrated the use of a Microsoft HoloLens headset, which contains a short distance 3D scanner useful for data collection and mapping onsite.
Several studies have investigated the possibilities of AR technology in the field of built environments (e.g., [21], [22]). One such case from Sweden is "The Hidden City" in Kiruna [23]. Physically moving historical buildings in the city of Kiruna opens up the potential for innovative approaches in city planning, maintenance and use of technology. The IT firm CGI considered the challenges associated with the move and devised a way for the city to gain a precise view of its underground infrastructure before digging. The technology vision combined GPS-enabled data sources and GIS data from the city, visualized through augmented reality (AR). The use of AR would enable engineers in Kiruna to view concealed or hidden infrastructure, such as underground pipes with centimeter precision (see figure 2).     The ability to observe changes over time could be important for studies regarding deformation, stresses, shotcrete, rock support, etc. It was suggested that AR could be used for forensic analysis after, for example, a fall of ground. The real-world tunnel view combined with selected parts of the computer-generated geomechanical model in 3D (e.g., large-scale structures, geology) would be beneficial during rock mechanics field studies, such as tunnel mapping or monitoring. To have the real-world view integrated with real-time measuring (e.g., smart bolts, photogrammetry, MWD) and monitoring results would increase the understanding of the rock mass as well as provide a good background for decision making. It would also be beneficial to study different layouts and to use the 3D model combined with geomechanical information. In mining, AR could be used to give a better visualization of the location of seismic events.

Results from the workshop
Other applications were given that would benefit rock mechanics work as well as other areas or processes within underground excavation. For example, AR could be used as a tool in IOP Conf. Series: Earth and Environmental Science 833 (2021) 012166 IOP Publishing doi:10.1088/1755-1315/833/1/012166 5 (i) communication and decision-making, (ii) all phases of an underground excavation, (iii) planning pre-investigations, and (iv) safety.

Results from the open-ended qualitative interviews
Relatively few of the respondents had thought of AR as a supporting technology from a rock mechanics perspective. The designers see great potential in using AR in their daily work, while the contractors see less benefit in their role. Based on all the interviews, structures and weak zones seemed to be of greatest interest to visualize in an interactive view. The respondents also see a large benefit in using AR as a communication tool. In the future, the greatest benefit of using AR is to visualize data from field investigations and perform real-time analysis (such as wedge and stress analysis) that would result in an adapted solution for rock reinforcement. Several of the respondents speculate that, in the near future, AR has great potential for focused tunnel inspections. The suggested "low-hanging-fruit" for continued work was the visualization of mapped structures in rock slopes.
A general and schematic view of the information and objects that respondents would like to visualize in AR in the different phases of tunnel design and excavation is presented in figure 4. The AR environment should be continuously improved and updated with more information during subsequent phases. Outside of the general view of what the respondents would like to visualize in a rock mechanics perspective, there were other comments, such as: (i) the ability to visualize seismicity and seismic events in the mines, (ii) the AR technology could be used for learning and improved understanding (e.g., visualizing nearby excavations that have failed or grown in size, such as ore passes, to increase the understanding of observed failures), (iii) use AR for simulation of mining/excavation stages and IOP Conf. Series: Earth and Environmental Science 833 (2021) 012166 IOP Publishing doi:10.1088/1755-1315/833/1/012166 6 steps, and (iv) some respondents commented about the potential of using AR as basis and support in the risk management work.
Beyond what the respondents would like to visualize from a rock mechanics perspective, they commented on the potential of using VR and AR technology as a communication tool for stakeholder engagement and safety issues.

Discussion
Based on the suggested uses of AR in rock mechanics, there are already examples of cases where AR technology is used. AR and VR have been used as a communication tool for stakeholder engagement in some published cases within mining, e.g., [7], [8]. Another request concerned navigation and evacuation underground. For example, Mobilaris [24] is working with real-time 3D visualization that they have implemented in several mines as well as tunnels. The technology is not based on AR as such; however, it is an example of technology that already exists and fulfils the requirements. Ongoing research in the field of enhancing geological information in the real-world view for rock slopes ( [18], [19], [20]) will naturally push forward the development work of AR.
In order to take the next step towards AR, 3D models need to include digital objects that carry their individual information and information about their relationship to other objects in the model. AR systems should be capable of proposing modifications to the existing virtual objects based on later field observations. The 3D model and AR system need to be linked in order to change both models in real time. The future trend of BIM in underground construction is big data analysis, which requires machine learning and computer vision [25].
Relative to BIM and underground construction, it is more difficult to gather data for rock engineering and tunneling. This difficulty is due to fundamental limitations on "seeing into" rock and to the inherent variability of geological materials. Despite these challenges, techniques to directly measure the subsurface are improving, including recent advances in measurement-while-drilling, seismic-while-drilling, ground penetrating radar, tomography, real-time drift and pit scanning, and InSAR. As a result of these advances, subsurface data is now being collected on an unprecedented scale. Machine learning techniques will allow future practitioners to leverage these large quantities of data. As new data becomes available, machine learning models can quickly give design performance predictions and detect anomalous conditions. Such models naturally integrate with AR systems.
Based on the pilot study, the proposed development work should focus on the AR system, the information model (e.g., extended BIM or geomechanical model) and its supporting rock mechanics data. A schematic view of the suggested development work for these three categories is presented in figure 5.  One of the most important aspects of continued development is the acceptance of using this technology within the field of underground excavations. AR technology must be accepted among the different roles and in the different phases of a project. Therefore, the client/owner of underground excavation projects has an important role in the education, promotion, and ultimately, the acceptance and development of AR.

Conclusion
This study has demonstrated that although AR is not yet ready to be fully adopted in rock mechanics and underground excavations, there is great potential and that AR technology will likely become common in the near future. The immersive and data-rich visual experience made possible by AR offers engineers and stakeholders a deeper and more intuitive understanding of underground spaces. Further, the ability to modify designs in an AR system and see analysis results in real time will streamline the iterative design process. The next steps for AR systems are the development of robust positioning, and increasing processing power as well as a longer battery life.
Based on the workshop, interviews, case studies, and assessment of the state-of-the-art of AR technology, the following conclusions are drawn: • In general, the suggested use of AR in rock mechanics concerns visualization of: o underground excavations, computer generated designed tunnel and scanned completed tunnel; o computer generated geomechanical model (geological information, overburden, topography, interpreted stresses, etc.); o results from analytical, empirical and numerical analysis (e.g., wedges, RMR, Q-value and stress tensors); o rock support and reinforcement (position and type of bolts, following changes over time such as crack initiation in shotcrete); and o real-time measuring results (e.g., extensometer, inclinometer, smart bolts, photogrammetry, MWD). • AR and VR are used as communication tools for tunnel inspections, training and education, and rock slope investigations in several published cases in mining and tunneling. • According to suppliers of technology and researchers in the field, the most difficult issue to solve at the moment is localization and positioning through visible light for underground excavations. Suggested near-future work of AR for rock mechanics is primarily on applying AR for investigations of rock slopes in order to avoid problems with positioning underground. More longterm future work should include development regarding visualizing real-time results from analysis and measurements. For that purpose, machine learning and computer vision can play an important role.