Direct-Visual-Operation Support System for Unmanned Construction

2.1. Proposed System

In the proposed system, a camera robot that includes three servomotors and a stereo camera is used along with a see-through HMD as the display monitor.

There are two types of see-through HMDs: video see-through and optical see-through. The video see-through blocks the user's field-of-view completely and shows the image photographed by the camera instead. In the optical see-through, the image of the micropanel is sent to the eye by reflecting it through a small optical lens and superimposed on the field-of-view in real space. In the proposed system, the optical see-through HMD is used as a support system for the direct visual operation. Thereby, the direct visual operation is performed with small eye movements because the image is observed simultaneously in the construction machine.

By presenting the 3D images acquired by the camera robot to the operator, a stereoscopic effect is generated. The rotation angles of the roll, pitch, and yaw axes of the HMD are acquired per head movement of the wearer by installing gyroscope and acceleration sensors in the HMD. The acquired rotation angle is transmitted to the camera robot. As shown in Fig. (1), a stereo camera is attached to the robot with three servomotors corresponding to the roll, pitch, and yaw, of the HMD. Each servo motor is controlled based on the rotation angle of the pilot's head received from the HMD and moves the camera according to the movement of the pilot's head. By synchronizing the direction of the see-through HMD with that of the installed camera, an image environment approaching the boarding environment is presented.

The rotation angle is transmitted from the sensor installed in the HMD to the camera robot installed in the cockpit. A video image of the stereo camera attached to the robot is transmitted from the camera robot to the see-through HMD. For communicating the rotation angle, a WebSocket that can communicate low-capacity messages at high speed, with low delay, is used [13]. For video communication, WebRTC, which is P2P video communication with low delay, is used [13].

The camera robot is attached to the cockpit of the construction machine, and the remote operator controls the see-through HMD. The operator performs the required work by confirming the camera image while directly performing the visual operation. As a see-through HMD is utilized, large-viewpoint transfer becomes unnecessary. In addition, as the camera robot synchronizes the head of the operator and camera direction, camera switching operation becomes unnecessary.

3.1. Construction of the Proposed System

A remote-control camera robot was constructed using a six-axis manipulator (Lynxmotion AL5D), stereo camera (Zed mini), and microcontroller board (Arduino UNO) available in the market, as depicted in Fig. (2). As the rotation values of the three axes, roll, pitch, and yaw, can be acquired from the see-through HMD, three among the six servomotor axes can be controlled and correlated. A constant value was always set for the remaining three axes, and the robot was fixed to be vertical. The servomotor was controlled using the Arduino UNO board based on the rotation information of the pilot's head received from the optical see-through HMD; Microsoft HoloLens (specifications are listed in Table 1 containing a gyro sensor was used to acquire the rotation angle of the HMD. A remote-control-type backhoe SH 135 X -3 B (Nishio Rent All Co., Ltd.) was used for the experiment.

3.2. Experimental Environment

The stereo-camera robot was installed in the cockpit of the construction machine at eye-line height during boarding operation, and the image system camera was installed in the roof of the cockpit as in the conventional system. Fig. (3) shows the boarding seat after the installation of the camera robot. Fig. (4) depicts an image transferred from the installed stereo-camera. The resolution was reduced to 320×180 per eye to enable the image transfer with low delay, enabling remote operation. The CONDOR 2D system (Nishio Rent All Co., Ltd.) [14] with a display resolution of 1600×900 was used. As shown in Fig. (5),the remote operation was performed in a room from which the construction equipment could not be observed. The block diagram of the experimental setup is displayed in Fig. (6).

3.3. Experimental Procedure

This experiment was performed based on a model task for the work efficiency evaluation of the remote-controlled hydraulic excavator [15]. Excavation by hydraulic excavators is generally affected by the difference in the number of piled buckets [15]. Therefore, for reproducibility, the work of scooping-up drum cans is considered similar to that of scooping-up earth, and sand, because it is difficult to obtain the piled quantity in the bucket each time. As shown in Fig. (7), a drum with handle was initially placed on 50-cm square mats at location one. The drum was required to be moved between four such mats, numbered 1-4, in the following order: 2, 3, 4, 1, at the end of which one cycle was completed and the process again repeated. The experimental duration per operator was set to 30 min with reference to the 3D work guideline [16, 17]. The handle was made larger than the width of the construction machine bucket by several centimeters, as high precision work was required. It was necessary to scoop the drum can by lowering the bucket to the ground each time it was transferred to a mat. The worksite is depicted in Fig. (8). The same work was performed once for each of the conventional 2D system and the proposed system. Before the experiment, the operators were made to practice on both the systems until there was no operator error.

3.4. Evaluation

The average work time required for one cycle is calculated as the work efficiency. The average work error, which is the distance of the drum from the center of the mat, is calculated as the work accuracy.

3.5. Participants

Two vehicle-type construction machine operators with boarding operation qualifications and no experience in unmanned construction participated in the experiment. Participant-1 was a skilled operator who operated daily, whereas participant-2 was unskilled and operated only during experiments for research.

3.6. Results and Discussion

Table 2 lists the average working time for each participant, and Table 3 presents the results of the average work error. The work accuracy and efficiency were impaired for skilled operators (participant-1), whereas both drastically improved for unskilled operators (participant-2).

For participant-1, the average work error with the proposed method was + 1.02 cm (a 21.8% reduction in the work accuracy), and the average work time was +3 min 6 s (49.6% reduction in the work efficiency), compared to the 2D system. The work efficiency of skilled operators may have decreased because of the 430 ms delay in the display in the communication system used in this experiment. In addition, the experiment included scenarios in which experienced operators performed operations through a sense of habit compared to the unskilled worker. Therefore, skilled operators were more likely to be affected by the delay because they cannot easily perceive the effect of environmental changes when high-precision work is required, resulting in unnecessary operations such as frequent stopping, and increased time to return to work because the bucket was moving too much. However, as the conventional 2D system did not sense the delay, work was performed at a stable operating speed. To suppress delay, transfer was carried out by reducing the resolution. Therefore, the resolution of the conventional 2D system was set at 1600×900, whereas that of the proposed system was 320×180, which is 1/25. Fig. (9) displays the transferred image. For skilled operators who can achieve sufficient work efficiency and accuracy in 2D systems, this low resolution can adversely affect the work, causing a slight decrease in the accuracy.

For participant-2, the average work error was -4.65 cm (a 37.8% improvement in the work accuracy), and the average work time was -4 min 55 s (44.2% improvement in the work efficiency). The delay of approximately 430 ms may not have affected the operation considerably because the unskilled operator was operating more slowly than the skilled one. Therefore, the usefulness of the 3D image and the image environment close to the boarding time can be considered to be greater than the effect of the delay for the unskilled operator, enabling smooth work compared to the 2D system. As a result, work efficiency drastically improved, and the same quality of work as that by a skilled operator was realized. The improvement of both items for participant-2 confirmed that the system could be used without any special training.

Previous studies indicated that with a delay of 430 ms, the working time was reduced by approximately 30% compared to that with a delay of 230 ms [18]. Therefore, by suppressing the display delay time to 230 ms or less, the work time can be reduced by approximately 30%. In addition, in the experiment, because communication was performed using Wi-fi, a delay of 430 ms occurred even when the resolution was reduced. From the viewpoint of low delay and high resolution, the use of commercial video transmission equipment is conceivable. For example, Teradek Bolt is a latency-free wireless transmission system for cameras that can achieve uncompressed trans-missions. A significant improvement can be expected by using this.

For the proposed system, the average work error was -1.82 cm (work accuracy improved by 21.4%), and the time was -54 s (work efficiency improved by 10.4%); thus, both parameters exhibited improvement.

In recent years, problems such as shortage of operators for unmanned construction machines have contributed to the development of unmanned construction technology, where the operating environment resembles that at the usual driver's seat, the operation can be performed similar to that in a boarding environment [5]. In addition to monitoring the cameras used for remote control, 3D cameras and omnidirectional cameras are used to increase the visual information. By acquiring not only visual information, but also the sound field, and the vibration and inclination of heavy machinery through sensors and reproducing them in the remote-control seat, field information is provided to the operator, and the operating environment approaches that of the boarding environment. Moreover, 3D and omnidirectional cameras are used to enhance the visual information. Thus, the work efficiency is improved for unmanned construction. Further improvements can be expected by rendering the proposed system closer to the boarding environment.