Protocols for Image Processing based Underwater Inspection of Infrastructure Elements

Image processing can be an important tool for inspecting underwater infrastructure elements like bridge piers and pile wharves. Underwater inspection often relies on visual descriptions of divers who are not necessarily trained in specifics of structural degradation and the information may often be vague, prone to error or open to significant variation of interpretation. Underwater vehicles, on the other hand can be quite expensive to deal with for such inspections. Additionally, there is now significant encouragement globally towards the deployment of more offshore renewable wind turbines and wave devices and the requirement for underwater inspection can be expected to increase significantly in the coming years. While the merit of image processing based assessment of the condition of underwater structures is understood to a certain degree, there is no existing protocol on such image based methods. This paper discusses and describes an image processing protocol for underwater inspection of structures. A stereo imaging image processing method is considered in this regard and protocols are suggested for image storage, imaging, diving, and inspection. A combined underwater imaging protocol is finally presented which can be used for a variety of situations within a range of image scenes and environmental conditions affecting the imaging conditions. An example of detecting marine growth is presented of a structure in Cork Harbour, Ireland.


Introduction
The monitoring of marine structures is often beset by reduced visibility, limited access, and high costs. Despite these deterrents, the need for underwater inspections is set to intensify globally in the coming decades as the offshore renewable energy sector continues to grow and major oil and gas companies move into deeper and more hostile environments [1]. Underwater inspections are mainly carried out by divers or Remotely Operated Vehicles (ROVs) [2]. ROVs can be deployed to much greater depths and for longer periods than divers, however, they are considerably more expensive, inflexible, and vulnerable to failure. Consequently, their safe operation requires great attention to detail. Underwater inspections performed by divers represent the current state-of-the-art. Divers are capable of covering a wide area and can inspect at a number of scales, from a detailed close-up assessments with the help of specialised equipment if necessary, to broader-scale visual assessments.
The quality of visual inspections largely depends on the ability of divers to observe and objectively record details of defects. This is prone to considerations such as lapses in concentration, level of knowledge, subjectivity, and fatigue, which contribute to the variability and reduced accuracy of visual inspections [3]. Additionally, a diver can expect some loss of judgement in cold conditions and due to inert gas narcosis at lower depths if using air as a breathing medium [4]. To increase reliability of visual inspections, photographs are almost always captured to include in the inspection report. However, these photographs are rarely exploited to their fullest potential in either a qualitative or a quantitative fashion. Moreover, despite having an established role in the inspection process, there has no agreed protocol for image collection and subsequent interpretation [5]. This paper aims to fill that void by specifying the technical requirements for capturing imagery in an underwater infrastructural setting, as well as outlining a set of best practice guidelines for ensuring that the acquired imagery is suited for quantitative analysis. Extracting quantitative information from images using computational techniques has received significant interest in recent years [6][7].
The development of such a protocol is especially useful for maintaining the long-term integrity of an Infrastructure Management System (IMS). A well calibrated IMS is dependent on the quality and consistency of the input information. As offshore structures are being designed with service lives of up to 35 years [8], and possibly longer if they are eligible for requalification programmes at the end of their service lives, it is of great value for the inspection data acquired during the early life stages to be accessible and relatable to the inspection data acquired decades later. This can be difficult to achieve if inspections are carried out in an ad-hoc manner, whereby divers rely on their own intuition when collecting and cataloguing imagery. By adhering to a protocol, the output from inspections is more standardised, and as a result, it becomes easier to compare and track the progression of damage.
The proposed protocol addresses each stage of the imaging pipeline from equipment set-up right through to image archiving. It is developed based on existing literature and the experience of the authors. Special attention is given to stereo based 3D shape recovery systems, which offer significant potential as an inspection tool as they are capable of recovering full 3D shape information. Stereo systems utilise a dual camera set-up to simultaneously photograph a specimen of interest from slightly different viewpoints. They have greater operational complexity than single camera systems as two cameras must be configured and also synchronised.
The following section describes the protocol. Section 3 presents an example of the protocol being implemented as part of an assessment of a submerged pile in Cork Harbour, Ireland. Finally, Section 4 concludes the paper.

Protocol
This protocol describes the procedural method for acquiring underwater images. The protocol addresses: 1) archiving, 2) imaging, 3) diving, and 4) a combined protocol for underwater inspections. format should be retained for archival purposes. RAW images contain minimally processed data from the image sensor. They are capable of storing a greater level of information from a scene (i.e. wider dynamic range and colour gamut) than other image formats. The RAW format differs depending on the camera model and camera manufacturer, although each format contains essentially the same data and metadata. An additional step, if desired, would be to convert the original RAW formats to an open standard and well supported format, namely the Digital Negative (DNG) format, which is a popular and freely available format developed by Adobe Systems™. Since there is a wide range of proprietary RAW formats, it is hard for applications and programs to guarantee future compatibility with them all, especially for some of the lesser known and lesser used RAW formats. Thus, the additional step of converting to the DNG format from the original RAW format would be of particular value to users that have, or expect to have, imagery acquired from a number of devices, and would like to unify the RAW formats into a common format that retains all of the original information.
In the case of video, still frames should be extracted at relevant intervals and saved as JPEG files as analysing every frame is often unnecessary. The intervals are primarily determined by the speed of the camera relative to the subject. If a recording device is moving quite quickly then more frames should be extracted. Generally, extracting three frames per second should be sufficient for most cases. The original video should be stored in its native format.
Photographs and video have accompanying metadata which contain useful information about the content and context of the file. Metadata is automatically embedded into each digital file. It provides information such as the time and date of capture, camera model, exposure information etc. It is vital that the time and date of all contributing cameras are precisely set as this is provides a convenient way for identifying the synchronised stereo image pairs. Some metadata must be manually added such as the baseline distance for stereo imaging, or the camera calibration data. The specific nature of this metadata will vary according to the task. In the case of stereo imaging, the imagery should be marked as coming from the left or right camera.

Imaging Protocol
The first objective of establishing imaging protocols involves specifying the technical requirements of capturing imagery in an underwater setting. These technical requirements deal with underwater conditions, setting up underwater housings, and the choice of camera settings.
Underwater imaging must try to overcome the challenging environmental conditions. Even in perfectly clear water, there is a loss of colour and contrast when the structure is separated from the camera by any significant distance. Water absorbs the red component of light to a greater extent which results in underwater subjects having a blue-green tinge. Colour diminishes with distance so subjects further away will appear indistinct and devoid of colour detail. This issue can be partially offset by following one of two approaches. The first approach involves photographing the damaged region from as close up as possible. Wide-angle lenses facilitate short focusing distances, allowing the damaged region to be in close proximity to the camera(s), at the expense of some radial distortion. The second approach entails the use of artificial lighting in the form of strobes to restore some lost colour. An added complication in this case is the phenomenon of backscatter, where light reflects off particles in the water. Even seemingly clear water is affected by this. The best method for limiting backscatter is positioning the lights away from the axis of the camera lens.
There are refraction issues associated with using cameras enclosed an underwater housing. This will lead to slightly distorted images. This issue can be mitigated in the case of stereo imaging by calibrating the cameras, which accounts for any distortion.
Many modern cameras have simplified the process of choosing the optimum aperture, ISO, and shutter speed settings for a given scene through various automatic exposure modes and the use of through-the-lens (TTL) metering. ISO measures the sensitivity of the image sensor. Higher ISO settings tend to be used in darker situations to amplify the available light, however this comes at a cost of increased noise in the images. The aperture size controls how much light reaches the sensor. A wide aperture allows more light to reach the sensor at the expense of a narrow depth of field, meaning only objects within a confined range will be in focus. A small aperture will have a greater depth of field, however, the resulting image will be dark/under-exposed unless the shutter stays open for an extended period of time to let enough light to reach the sensor.
Keeping the shutter open for too long (i.e. having a slow shutter speed) presents other problems, mainly a high degree of blur. The camera will automatically attempt to counteract the dimly lit underwater conditions by combining a small aperture with a low shutter speed and a high ISO. It is important that the settings remain within certain limits to minimise the impact of these problems. Recommendations of the limits are summarised in Table 1.
It should be noted that the depth of field is also affected by the focal length of the lens. Closer focusing distances will produce a shallower depth of field. Guidelines for choosing the focal length are not discussed as a good choice of focal length is best adjudged by the diver onsite. A small focal length provides a wide field of view, which can be useful in cases where there is a short distance between the camera and subject (possibly due to poor underwater visibility) and where it still wished to photograph a relatively large area of the structure, while long focal lengths give a smaller field of view [9].

ISO
An ISO value of 800 is a good compromise for photographing in dimlylit conditions while still controlling noise/graininess.
The shutter priority mode is recommended as diver-held cameras employed in underwater inspections are prone to shaking, which introduces motion blur. This is a semi-automatic shooting mode that allows the user to specify the shutter speed. The camera then automatically decides the best aperture and ISO sensitivity for the specified shutter speed to get the correct exposure. The shutter speed should not be any slower than 1/15 seconds. The imagery obtained from the cameras should be reviewed at regular intervals. If it is apparent there is too much motion blur present in the images, the shutter speed should be adjusted to a faster setting. Additional artificial lighting will be required if the camera settings exceed any of these ranges. Irrespective of these requirements, artificial lighting will be necessary at greater depths where ambient lighting is not sufficient.

Diving Protocol
The diving protocol addresses the logistical considerations (testing equipment prior to usage, route planning etc), as well as handling lighting and turbidity conditions, which will affect the optimum distance between camera(s) and subject. It is vital that any unnecessary time spent underwater by the diver is kept to a minimum. With this in mind, the diver should be presented with a clear and concise brief outlining the task at hand. A rough idea of the turbidity conditions should be known beforehand. Both lighting and turbidity are crucial factors which affect the underwater visibility and consequently the image quality [10]. Artificial lighting in the form of underwater strobe lights are required in dim lighting conditions, especially if the subject is bumpy/rugged in which case the lights would assist with revealing details that would otherwise remain in shadow.
Water is seldom optimally clear, and the dissolved and suspended matter can reduce visibility by both absorption and scattering of light. While turbidity may not be easily reduced, there are some precautions which can be taken to offset the deleterious effect in relatively high turbid waters. Firstly, caution should be taken in shallow waters to avoid disturbance with the sea/river bed which may unsettle fine sediments through either direct contact or from turbulence created from the ship. Anchoring the vessel downstream away from the inspection site can also prevent additional sediments that would reduce the water clarity. Secondly, the distance between the cameras and the subject under consideration should be reduced. In seas and oceans, the water is generally clear so measures to counteract poor visibility need not be considered.
Choosing the distance between the subject and the diver is a trade off between a number of factors. It will vary depending on the level of visibility, the size of the subject in the scene, and level of detail required. When the underwater visibility is poor, the diver can photograph in close proximity to the subject (up to 30 cm). In the case of stereo imagery, going any closer than 30 cm leads to large perspective differences between the stereo image pairs which can hamper the 3D shape recovery process and result in a myriad of occluded regions [11]. In clear underwater conditions, the diver can photograph from a distance up to 2.5 m before stereo imaging breaks down and the error tolerance becomes unacceptably high [12]. If the object under inspection is quite large (e.g. a wide diameter pile) and it is wished to include the whole structure within an image then photographing from further back will provide better context.
Finally, the baseline distance for stereo imagery will influence the choice of subject-camera distance. While the baseline shift is sometimes fixed at a certain distance according the constraints of the available equipment, it is important to note the effect it has on the accuracy. Theoretically, a wider separation between the cameras results in a lower percentage error in the depth estimation. However, the advantages of having a wide separation are offset by the creation of large perspective differences. The baseline shift should be in the range 10 cm to 30 cm. Additionally, the cameras should be aimed inwards at an angle θ (known as the vergence angle) such that their centrelines intersect approximately at the face of the subject as shown in Figure 1. This is to ensure that the cameras capture as many of the same points in both images as possible. The vergence angle has previously been used to model the error in depth [13], where values in the range 5° -10° were found to provide the lowest errors. All equipment should be checked above water prior to each inspection. The underwater housings should be checked for any signs of leakage by firstly submerging them without their contents. If necessary, the O-rings should be re-lubricated/replaced as per manual instructions. Care should be given to ensure that the time and date of the cameras are precisely set, that there is enough storage capacity in the SD cards and the battery is sufficiently charged. Appropriate settings should be configured for each camera/video recorder ensuring that the shooting modes in both are identical. It is advised to initiate filming immediately before the diver submerges as it is easier to control the simultaneous triggering of both cameras when above water. The captured imagery should be reviewed at regular intervals during the inspection and any adjustments should be made accordingly. Dives which produce sub-standard imagery should be repeated. Necessary props should be prepared such as lighting equipment or calibration tools. Finally, the diver should be familiarised with the blueprint of the structure and identify any components that are of particular interest. A suitable route should then be planned based on this. In cases where the diver cannot photograph a particular component from all sides due to restricted access, he/she should endeavour to photograph as much of it as possible.

Combined Underwater Protocol
Greater care and attention is required for stereo imaging as two cameras must act in unison. A flowchart showing the combined underwater protocol for stereo imaging is shown in Figure 2.

Demonstration of protocol on real world structure
This section applies the protocol during an underwater assessment of a marine growth affected pier located in Cork Harbour, Ireland. The following sub-sections provide an overview of the structure under consideration, the underwater conditions at the site, and the image acquisition procedures.

Structure and Conditions at the Test Site.
The structure under consideration is a long-serving pier in Cork Harbour that is mostly used for recreational and leisure purposes (Figure 3). The structure is affected by marine growth, which is undesirable as it changes the thickness and roughness characteristics of structural members leading to increased hydrodynamic forces. Tracking marine growth thickness is often carried out during underwater inspections so that improved estimates of the hydrodynamic forces acting on the structure can be obtained.  It was a clear and sunny day when the testing was carried out so there was no requirement for additional artificial light sources as ambient light was sufficient in the shallow waters. The turbidity was quite high so the imagery had to be acquired at a close range from the subject.
The stereo system is shown in Figure 4. This system consist of two Canon 600D DSLR cameras, enclosed in underwater cases, which are securely attached to a graduated stereo bar. The graduated stereo bar allows the baseline distance to be easily measured. In this case, the centres of the cameras were separated by 15 cm, which was duly recorded. Knowledge of the baseline is necessary for obtaining the scale factor that converts the dimensions of the reconstructed 3D shape to real world units. The Canon 600D cameras were both configured to simultaneously capture high resolution 18 MP images in RAW and JPEG format at a time interval of two seconds. The cameras were in shutter priority mode in accordance with the protocol and had a shutter speed of 1/20 seconds. The minimum focal length of 18 mm was selected so that as much of the scene could be captured as possible.

Acquired imagery
An image of the marine growth affected pile under consideration is shown in Figure 5.  It may be observed that the high turbidity creates an image lacking in contrast. In this case, the imagery greatly benefits from image pre-processing such as contrast enhancement. The associated reconstruction is shown in Figure 6, which is based on an auto-calibration technique [14]. The shape reconstruction is dependent on the quality of the imagery, which in turn is dependent on the image acquisition practices. As such, the successful results achieved here provide validation for the protocol.

Conclusion
The effort and expense associated with undertaking underwater inspections warrants significant forethought and planning to ensure the every technical aspect of data acquisition is covered. To address this, a comprehensive protocol is developed for the first time that sets out best practices for underwater image acquisition. The protocol provides guidance on archiving and cataloguing the imagery, overcoming challenges such as high turbidity, equipment set-up, and defining appropriate limits for camera parameters. Selecting the most suitable combination of these camera parameters for a given environment involves a trade off between minimising negative image quality factors such as blur and noise, whilst retaining sufficient brightness and ensuring enough of the subject is in focus. Ineffective image acquisition practices directly affect the performance of image algorithms. The protocol was demonstrated on a real world structure in Cork Harbour, Ireland. In spite of the high turbidity, a successful assessment was achieved through careful adherence to the protocol.