Technologies like augmented reality in construction are emerging to digitalise the construction industry, making it significantly more effective. Still, one of the oldest industries on Earth comes in next to last in the McKinsey Digitization Index, the first major attempt to measure the ongoing digitalisation of the US economy at sector level. Thus far, the technology industry has been focusing on moving bits of information, rather than atoms of materials. The latter is the essence of construction, which is the localized and labor intensive activity of arranging atoms into useful buildings and infrastructure facilities. But the latest technological developments in sensors, miniaturization, robotics and drones, as well as computing power and smart algorithms, are giving a sense of how we might combine the digital and physical to build cheaper, greener, and more durable buildings in a shorter timeframe.
What if we could have instant access to all the information about a construction site, down to smallest details about every person, tool, and bolt? What if we could always be sure about the final measurements of a beam or that soil volumes in the cuts are close to those of the fills? What if we could always track how fast the supply of materials runs out, and re-order supplies automatically?
All this is achievable with a digital twin -- a concept of having a real-time digital representation of a physical object. Usually, digital data is formed by sensors that continuously monitor changes in environment and report back the updated state in the form of measurements and pictures.
In this article, we will review the use cases and benefits that digital twin technology can give to a construction project team. Then, we will describe the techniques currently in use for the collection of raw image data from construction sites, as well as the methods of processing unstructured data in order to reconstruct a 3D digital representation of a construction site.
What is digital twin technology?
The digital twin technology concept has been around since 2002 and was first used in the fields of astronautics and aerospace, and manufacturing and product lifecycle management. It was used by NASA for the moon exploration mission Apollo 13 and Mars Rover Curiosity.
A digital twin is essentially a link between a real world object and its digital representation that is continuously using data from the sensors. All data comes from sensors located on a physical object; this data is used to establish the representation of a virtual object.
The digital representation is later used for visualization, modeling, analysis, simulation and further planning. This data triggers the feedback loop of decisions and changes in workflows that influence the control processes of the real object system.
Useful application of digital twins in construction
For construction, using digital twins means always having access to as-built and as-designed models, which are constantly synced in real-time. This allows companies to continuously monitor progress against the schedule laid out in a 4D BIM model.
We can even use the model predictive control approach and make decisions based on forward simulation, beginning with the current state of the building. So we can always analyze different paths of actions and estimate their probabilities and corresponding cost functions in order to select the most optimal decision (or adjustment) for what we should do next.
Here possible real-time digital twins applications on construction sites.
1. Automated progress monitoring
Progress monitoring verifies that the completed work is consistent with plans and specifications. A physical site observation is needed in order to verify the reported percentage of work done and determine the stage of the project.
By reconstructing an as-built state of a building or structure we can compare it with an as-planned execution in BIM and take corresponding actions to correct any deviations. This is usually done by reconstructing geometry of a building and registering it to the model coordinate systems, which is later compared to an as-planned model on a shape and object level.
Often data for progress monitoring is collected through the field personnel and can be hugely subjective. For example, the reported percentage of work done can be faster in the beginning and much slower close to the end of the project. People are often initially more optimistic about their progress and the time needed to finish the job.
Hence, having automated means of data collection and comparison means that the resulting model to as-designed BIM models is less liable to human error. Digital twins solve the common construction process problems.
2. As-built vs as-designed models
With a real-time digital twins, it is possible to track changes in an as-built model — daily and hourly. Early detection of any discrepancies can lead to a detailed analysis of historical modeling data, which adds an additional layer of information for any further decision making processes.
The project manager can then reconstruct the steps that led to the error and make changes in the future work schedule in order to prevent any similar mistakes from occurring. They can also detect underperformers and try to fix the cause of the problem earlier in the project, or plan the necessary changes to the budget and timescale of the whole project.
3. Resource planning and logistics
According to the Construction Industry Institute, about 25% of productive time is wasted on unnecessary movement and handling of materials.
Digital twin technology provides automatic resource allocation monitoring and waste tracking, allowing for a predictive and lean approach to resource management. With digital twin technology companies would avoid over-allocation and dynamically predict resource requirements on construction sites, thus avoiding the need to move resources over long distances and improving time management.
4. Safety monitoring
The construction industry is one of the most dangerous sectors in the world. According to the Bureau of Labor Statistics in the United States, more than four thousand construction workers died on-site between 2008 and 2012.
The real-time site reconstruction feature digital twins allows the industry’s companies to track people and hazardous places on a site, so as to prevent inappropriate behavior, usage of unsafe materials, and activity in hazardous zones. A company can develop a system of early notification, letting a construction manager know when a field worker is located in dangerous proximity to working equipment and sending a notification about nearby danger to a worker's wearable device.
Microsoft recently shared a great vision of how AI combined with video cameras and mobile devices can be used to build an extensive safety net for the workplace. See the video:
https://www.youtube.com/watch?v=pL-c00M2CnI
5. Quality assessment
Image-processing algorithms make it possible to check the condition of concrete through a video or photographic image. It is also possible to check for cracks on columns or any material displacement at a construction site. This would trigger additional inspections and thus help to detect possible problems early on.
See an example of how 2D images using 3D scene reconstruction can be used for concrete crack assessments.
Liu, Yu-Fei, et al. "Concrete crack assessment using digital image processing and 3D scene reconstruction." Journal of Computing in Civil Engineering 30.1 (2014)
6. Optimization of equipment usage
Equipment utilization is an important metric that construction firms always want to maximize. Unused machines should be released earlier to the pool so others can use them on other sites where they are needed. With advanced imaging and automatic tracking, it is possible to know how many times each piece of machinery has been used, at what part of the construction site, and on what type of the job.
7. Monitoring and tracking of workers
Some countries impose tough regulations on how to monitor people presence on a construction site. This includes having a digital record of all personnel and their location within the site, so that this information could be used by rescue teams in case of emergency. This monitoring is another digital twins application. Still, it is better to integrate digital twin-based monitoring with an automatic entry and exit registration system, to have a multi-modal data fused into a single analytics system.
Getting Data for Digital Twins
Virtual representations of physical models like construction sites can be obtained by using multi-modal sensor data and algorithms based on computer vision techniques and deep neural networks. Various type of sensors are often used to infer the 3D structures; they include laser scanners (LIDARs), radars, thermal imaging cameras, and standard photo and video cameras. There’s an abundance of means for getting data for digital twins successfully.
As we are more concerned here with the imaging methods that use images from photo and video cameras, let's explore how we can acquire the image input from which we will later reconstruct a digital twin model.
1. Smartphone Cameras
Smartphone prices are continuing to decrease, along with the cost of the components they are built with. Currently the cheapest smartphones have cameras with enough resolution and image quality to reconstruct the walls, the shape of the beams and the equipment within the construction site. As a result this type of data for digital twins is easy to obtain.
For example, it is possible to monitor progress by comparing an as-built reconstructed model with an as-designed BIM model simply by using unordered daily construction photographs taken on various smartphones with different cameras and lighting conditions. This strategy can also be used to track temporary resources, such as personnel, equipment, and materials on an infrastructure construction site.
2. Time-Lapse Cameras
Time-lapse video is a sequence of photos captured at a smaller rate than they are playing, which translates to the time moving faster than on standard video.
This type of videos has been used for construction site recordings and progress monitoring since at least 1989, when old film cameras such as the Standard 8-mm and Super 8-mm were popular. Currently, cameras are cheaper, smaller, and can work from batteries for long hours and weeks in time-lapse mode.
3. Autonomous UAV and Robots
Data for digital twins also comes from Unmanned Aerial Vehicles (UAVs) like drones, or fixed wing planes equipped with high-resolution cameras and sensors; they can be operated in a fully autonomous mode.
With the constantly increasing cost of labor in construction, the preference is for fully-automated methods of data capturing and processing. The mobile industry has produced cheap MEMS (Microelectromechanical systems) based sensors that are finding their way into drones and mobile robots equipped with high-resolution cameras. Drone manufacturers are making specialized versions of drones that are combined with data management platforms, which can automatically perform surveying, mapping, 3D reconstruction, as well as take volumetric measurements.
But with fully-automated data capturing robots there is an even higher need for automated analytics and data processing, since the industry does not have enough time to manually process additional terabytes of video data. We will explore some data processing techniques later in this article.
4. Video Surveillance Cameras
Video surveillance data capturing and processing techniques are similar to time-lapse videos, but with higher frame rates, i.e. normal speed on playback. With the conventional video capturing rate of 30fps we can extract finer details from construction scenes and better track changes in structures and movements in temporal objects.
5. Head-mounted Cameras and Body Cameras
Though such image capturing techniques as head-mounted cameras and body cameras are not widely used in construction today, we can expect that continuous miniaturization and advancement in technologies and AR headsets (which inherently contain cameras) will lead to their wider use on construction sites.
Recent examples show that Microsoft HoloLens is slowly moving to the construction site within the form of a hard hat. It is primarily used to view 3D models over real spaces, which helps to translate a designed structure to spatial representations.
Google recently released Google Clip, a miniaturized, battery-powered, AI-enabled camera, that is targeting the household consumer market. This is just the beginning for specialized hardware combined with on-board AI software (or hardware implementation in ASICS), which will target different markets for real-time image data processing and analysis in specific domains.
Image data processing algorithms for digital twin creation
At this stage, we have raw image data input from various cameras and it is now time to look at the techniques that we can use to build the digital twin representation of a site.
A high rate of generated data demands an even higher rate of data processing and fully-automated pipelines, from data capturing and analysis to knowledge and decisions. Recent advancements in computing power, GPU processing, extensive datasets, and deep learning algorithms have opened the door to automatic data processing.
The rise of self-driving cars, which generate 1-2 GB/sec of data, is pushing companies to find solutions for extracting knowledge and making decisions fast and in real-time, without sending all data to the cloud. A construction site is a contained and controlled environment that generates data at the same rate as self-driving cars, and where we can use the technologies of these cars to better navigate the project according to schedule, using fewer resources and with higher safety standards. With digital twins companies in the industry are at the forefront of technological innovation.
No, we will select image and video processing approaches that would useful for building a digital twin representation of a construction site and automatically extracting knowledge from raw image data.
1. 3D Reconstruction: Conventional Photogrammetry
Photogrammetry is the science of taking measurements from photographs and recovering the exact positions of surface points. It is currently used in many areas — architecture, engineering, manufacturing, quality control, and other. In most cases, the goal of photogrammetry is to reconstruct a 3D scene from given photos by aligning the real world coordinates to image coordinates and finally generating a points cloud.
There are two types of photogrammetry:
- Aerial photography is taken from elevated positions like cranes, aircraft, mountains, etc.
- Close-range photogrammetry is when the camera is located closer to the object and is usually hand-held or mounted on a tripod.
The latter method provides highly accurate site models, but with low spatial resolution, containing about a few hundred scattered points.
2. 3D Reconstruction: Structure from Motion
Structure from motion (SfM) began as a subfield in photogrammetry. It uses the same algorithms for the parts of the pipeline: feature/target tracking, triangulation, camera pose estimation, and bundle adjustment.
Currently, SfM and photogrammetry are used together to create denser 3D representations that are more accurate in measurements than any of the other methods alone.
Here is how the SfM method is used to reconstruct the Colosseum in Rome:
https://www.youtube.com/watch?v=kxtQqYLRaSQ
3. Object Detection and Recognition
Object detection and recognition is usually a cornerstone for robotic applications on construction sites, as robots need to know the location of obstacles for navigation and path planning. It is also an important thing for robots’ manipulators: they need the precise location of the objects to be picked up or moved.
In digital twin modeling, construction companies can use object detection and recognition to create better models of hazardous spaces and to monitor complex machinery on-site.
4. Localization
Simultaneous localization and mapping (SLAM) is a field in robotics. It constitutes the computational problem of constructing or updating a map of an unknown environment, while simultaneously keeping track of an agent's location within it.
Particle filter is one of the algorithms used to solve the localization problem (Image Source Link)
It usually involves a combination of sensors, such as a camera, radar, LIDAR, and inertial measurement unit (IMU), if an agent is moving. The agent can be a robot, a crane, or a head-mounted display a worker is wearing at a construction site.
With a diverse set of images we can localize objects in a world coordinate space.
5. Object Tracking
Object tracking is the recognition of an object’s location, speed, and dynamics at each time-step. It is often used in robotics for tasks of real-time predictive controls, but is also useful for predicting the location of construction workers and equipment on a construction site.
Tracking is also used as part of video-based path planning and can provide capabilities for avoiding obstacles.
Here is a great example of object recognition and tracking taken from a video filmed by street surveillance cameras.
https://www.youtube.com/watch?v=aE1kA0Jy0Xg
Another usage of object tracking is to recognize gestures in human-computer interactions, which can be harnessed for the automatic recognition of workers’ hand signals on a construction site.
Conclusion
We have looked at the concept of the digital twin and how to build a virtual representation of a construction site with advanced data capturing techniques, simply by using cameras and smart post-processing with dense 3D model and structure reconstruction.
The digital twin concept, paired with wearable and mobile devices on a construction site, can help to better represent the as-build project at any point in time. It allows up-to-date information to be fed back to the field so as to decrease the number of errors and reworks.
With continuous localization and tracking of people and equipment it is possible to completely monitor the use of time utilization and dynamically allocate resources in order to decrease time of waiting for free machinery or the inefficient use of expensive equipment.
Moreover, real-time monitoring can push security alerts about hazardous situations right to ta worker’s mobile phone or headset, which increases the safety of a construction site. It is also important to have access to the data about who is present on a construction site and where they are located, in case of emergency.
In this article, we have reviewed techniques and methods without touching upon particular software implementations. Currently, there is no one such system that can get everything working out of the box for all situations.
Nevertheless, there is a number of ongoing developments in the field that are connected to the emergence of Artificial Intelligence and Deep Learning, as well as more powerful GPUs. Fully integrated and innovative solutions that spread from hardware to software, bringing a greater level of efficiency and safety to the construction site, should come right after.
If you already decided to elevate your construction business with AR, or need an extensive consultation on the matter, get in touch with our AR Construction expert.
