Drone-based displacement measurement approach
Drawing inspiration from the balance sensing system of the human inner ear, we proposed a technology that allows for robust and high-precision image de-blurring. Figure 2 depicts the schematic diagram of the proposed approach, which incorporates the utilizing of two 2-D reference markers as receptors within the human ear’s vestibular system. This methodology is specifically designed to achieve image stabilization during drone hovering. Here, our focus centers on the medical perspective of the ear’s structure, and we draw upon this knowledge to construct our displacement measurement system. As it is well researched, the inner ear comprises receptors for both hearing (the cochlea) and balance (the utricle, saccule, and semicircular canals)41, as depicted in Fig. 2a. The vestibular receptors are composed of two distinct components: (i) two otolith organs, namely the utricle and saccule, responsible for measuring linear accelerations, and (ii) three semicircular canals, tasked with measuring angular accelerations. Specifically, rotational motion (angular acceleration) manifests during head rotation, while linear acceleration is perceived during walking, falling (i.e., translations), or head tilts concerning gravity. These receptors transmit vestibular information to the brain, where it is amalgamated into appropriate signals related to the direction and speed of movement, in addition to the head’s position relative to gravito-inertial acceleration. Therefore, the ear encompasses five types of receptor and a total of ten receptors in human ear, providing balance on six axes in three dimensions world.
Taking a cue from the ear’s sense of balance, in this study, we have developed a simple yet efficient approach to achieve stability in measuring deflection, one of the most critical vital signs in bridge inspection. Specifically, we have characterized the geometric structure of the bridge and hovering camera setting to simplify the six degrees-of-freedom (DoF) into four DoF, which forms the foundation for achieving stability in measuring deflection. To achieve this, we have utilized a two-dimensional pattern of a marker, as depicted in Fig. 2b, can emulate the equilibrium sensing function of the human ear, offering a robust image blur compensation and accurate (in-plane) deflection measurement. In this setup, by placing two reference markers on the abutments at both ends of the bridge, through leveraging the human binaural balancing function, the four DoF displacement measurement error, which includes the translation of x, y, z, and the rotation of θ (as shown in Fig. 2c), caused by the drone’s movement can be automatically canceled.
Figure 3a illustrates the optical configuration for bridge deflection measurement via drone aerial photography, accompanies by the definition of the six axes of the drone camera. While the drone is in the airborne state of hovering, even minor variations in its position and attitude can cause simultaneous displacements in the x and y axes (Δx and Δy), magnification along the z-axis (Δs), and rotational adjustments (Δθ). In images acquired before and after the bridge’s deformation by the inspection vehicle, as depicted in Fig. 3b, two reference markers (Mk-A and Mk-B), analogous to the left and right human ears, form a reference line. The model of the bridge, as adopted in this study (Fig. 3c) assumes insignificantly negligible displacements along the y-direction at the fixed points on both ends. For precise image deblurring, markers Mk-A and Mk-B function analogously to the human auditory equilibrium, with a “reference line” connecting the two, as shown in Fig. 3d. A similarity transformation aligns these reference lines in images before and after deformation with an accuracy level of 1/100 camera pixels. The detailed workflow can be referred to in Supplementary Fig. 1. Initially, the efficient maximized cross-correlation (MCC) method42 is employed to extract the center coordinates of markers Mk-A and Mk-B with pixel accuracy. Subsequently, a similarity transformation43 is implemented to align the images with pixel precision to the first frame of video before deformation. This step allows us to achieve reliable alignment of the measurement markers before and after deformation within half of the marker pitch, laying the foundation for further ultra-fine sub-pixel analysis using the SM algorithm24,25. Subsequently, the phase-based SM method is employed to compute the center coordinates of each marker with enhanced 1/100 pixel accuracy40 and the accurate marker pitch in image44. The critical point of the SM method is mentioned in Supplementary Fig. 3. Notably, the SM method has been successfully used in previous studies to measure deflections of bridges using a fixed camera, achieving ultra-high precision on the sub-millimeter accuracy45,46,47. In the present study, we adopt the SM method to extract accurate marker center coordinates, which not only compensates for image blur induced by camera motion but also facilitates highly accurate estimation of deflection. It is imperative to note that the goal of achieving ‘perfect’ image stabilization is defined as rendering zero displacements between reference markers Mk-A and Mk-B. Consequently, an ABC method using a reference line is proposed to facilitate more precise deflection measurements. This approach is simple yet remarkably potent since it eliminates the displacement errors arising from the residual effects of Δx, Δy, Δs, and Δθ induced by image stabilization procedure automatically. Additionally, the drone’s gimbal function significantly reduces out-of-plane rotation (Δα and Δβ) during hover recording (See Supplementary Figs. 5 and 6). Thus, the proposed methodology enables sub-millimeter accuracy in measuring bridge deflections, even with drone aerial photography, attaining comparable results to using fixed cameras and the SM method in previous research.
a Experimental arrangement of drone deflection measurement. b Recorded drone images before and after deformations. Each image contains two reference markers (Mk-A & Mk-B) and one measuring marker (Mk-C). c Our assumption according to a general bridge model. There are two fixed points at each end of the bridge (zero displacement or small enough displacement in the y direction to be negligible). d The key idea of our ABC method using a reference line. The reference line formed by markers Mk-A and Mk-B serves an analogous function to the human balance mechanism of the ear. e The post-deformation image compensation involves a dual application of similarity transform, achieving an exceptional 1/100-pixel accuracy.
Field experiment of bridge deflection measurement
A field experiment was conducted to assess the effectiveness of the proposed methodology using aerial drone photography on a 110-meter-long concrete bridge, as illustrated in Fig. 4a. This bridge, a Druk-Bund structure (hinged in the center) with three spans, was built in 1959 and located more than 20 meter height from the river. This application showcases the proposed drone-based displacement measurement system, as the bridge is situated above a river, making it impossible to find a location to fix the camera as in conventional vision-based displacement measurement methods. The details of experimental setup, marker installation and attachment, and the target bridge can be referred to in Supplementary Figs. 7–10, respectively.
a Schematic diagram of the target bridge. b The specification of the marker with a 200 mm pitch. Four different markers, each 0.5 m square in size, are combined for easy portability. c The record image by the drone camera (Autel EVO-II Pro; 6 K movie) from a distance of 85 m. d The top-view of a scene during measurement test. e The measured results obtained by the developed method (left) and conventional Doppler displacement sensor (right).
The bridge was imaged as a movie using a commercial drone camera while an 8 t test vehicle was traversing the bridge at a speed of 20 km/h. At a distance of 85 m between the aerial drone (Autel Robotics, Autel EVO-II Pro) and the bridge, the 200 mm pitch marker (Fig. 4b) was captured by only 13 pixels in the recorded 6 K (5472 × 3076 pixels) movie (Fig. 4c). The theoretical analysis estimated the achieved measurement resolution to be 0.2 mm, which is deemed sufficient to measure the anticipated displacement caused by the 8 t truck’s passage across the bridge. The proposed methodology therefore achieved a favorable precision in measuring millimeter-scale displacement, and the results were highly consistent with the reference results generated by LDV, as depicted in Fig. 4e. The maximum deflection values obtained using our method and the Doppler sensor were measured as 7.23 mm and 6.44 mm, respectively, yielding a discrepancy of 0.79 mm. Similarly, the maximum deflection values after smoothing obtained using our method and the Doppler sensor were 5.65 mm and 5.13 mm, yielding a discrepancy of 0.52 mm. Furthermore, the side-by-side comparison results indicate that camera motion issues have been effectively resolved (See Supplementary Fig. 11 for intermediate results of drone-based deflection measurement). To demonstrate the repeatability of the proposed approach, we conducted the experiment for multiple times. Furthermore, with the objective of clarifying the limitation of our method, we extended the distance of between the drone and the bridge from 85 m to 100 m. These results can referred to Supplementary Fig. 12. The experimental findings unequivocally demonstrate the aerial photography technique’s efficacy in providing an accurate and practical deflection measurement methodology for common bridge structures, thereby offering a viable solution to the long-standing challenge in the field.
Demonstrations of measurement accuracy verification
In light of the fundamental significance of reproducibility and replicability in scientific research, we carried out additional in-field displacement measurement experiments to establish the reliability of our proposed approach. More precisely, we conducted a series of experiments on a 35-m-long bridge situated at the Fukushima robot test field in Japan, wherein we compared the deflection values obtained by our drone photography-based technology with the displacement values in the y-direction, achieved by controlling a marker with a high-precision linear moving stage. The experimental arrangement is depicted in Fig. 5a, and the experimental scene is shown in Supplementary Fig. 13. In this experiment, the grid pitch of all three moiré markers was 50 mm, and the distance between the drone and the bridge was 34 m. The weather (early Jan. 2023) was sunny, the temperature was 5.2 °C, and the wind speed was 0–6.6 m/s. During the experiment, a measuring marker (Mk-C) was placed near the center of the bridge and longitudinally displaced manually by 1.003 mm, 2.010 mm, 2.999 mm, and 4.967 mm using a precision moving stage (SURUGA SEIKI, KX1250C-R; Unidirectional positioning accuracy: 5b, where the mean value for a one-second interval within the light-orange region is presented. Figure 5c illustrates the correlation between the measured and actual displacements, with an average error of 0.199 mm, surpassing the accuracy of previous studies on drone-based displacement measurements.
a Recorded image by drone camera. b The measured results. c Measurement accuracy.
The two validation experiments have also provided evidence of the exceptional flexibility of our approach, as it enabled the precise measurement of sub-millimeter level displacements for target bridge structures of varying sizes and spans by adjusting the marker pitch (from 50 mm to 200 mm) and the distance (from 30 m to 85 m) between the bridge and drone during image capture. Our approach provides a reliable and innovative methodology for practical bridge displacement measurement with the potential to significantly impact the development of future autonomous bridge inspection systems. On the other hand, our proposed method also has its limitations: (i) Since displacement measurement accuracy is tied to the chosen marker pitch, therefore, measuring minute displacements beyond 1/1000th of the marker pitch proves challenging based on the accuracy of the sampling moiré method. (ii) Our proposed method assumes the fixed girders remain stationary or experience negligible displacements, rendering bridges with unique structure unsuitable for this approach. (iii) The measurements are susceptible to several critical weather-related factors such as strong wind and shimmer for long-distance image shooting under unfavorable weather. For an in-depth discussion, please refer to Supplementary Note 2.
In conclusions, we have demonstrated that displacement measurement of infrastructures at a millimeter scale by using drone photography is feasible. Our method encompasses an ordinary off-the-shelf drone, a set of markers, and a well-designed computational framework for high-precision displacement extraction. As a vision-based system, our proposed approach differs from conventional methods, relying on intensity information. Instead, we characterize the phase information to estimate subtle displacements, providing multiple favorable features, including high accuracy, low computational complexity, and robustness to pixel noise and visual artifacts. Furthermore, to eliminate the errors induced by drone camera motion in displacement measurements, we proposed a simple yet efficient four DoF parameterization scheme taking advantage of the geometry constraints of drone photography of bridges. As a result, we successfully relaxed the requirement of recovering 6 DoF and camera intrinsic parameters. This simplification facilitates computational analysis and enhances displacement measurement stability by suppressing parameter estimation errors.
Through rigorous lab-scale experiments and extensive real-field validations, we have showcased that our approach achieves accuracy on par with conventional displacement measurement sensors, while simultaneously providing superior flexibility, reduced cost, and ease of onsite installation. Our proposed methodology holds great potential for widespread use in a variety of fields, including infrastructures, heavy industry plants, and construction sites, and is primed to make a significant impact in the expansive field of drone inspection48,49. Last but not least, this research holds immense potential for future autonomous inspections, reducing human intervention and improving efficiency. Through this automation, regular and timely inspections van be conducted while maintaining a high level of safety, effectively addressing the challenges posed by aging infrastructures50.