Early on a June morning in 2023, my colleagues and I drove down a bumpy dirt road north of Kyiv in Ukraine. The Ukrainian Armed Forces were conducting training exercises nearby, and mortar shells arced through the sky. We arrived at a vast field for a technology demonstration set up by the United Nations. Across the 25-hectare field—that’s about the size of 62 American football fields—the U.N. workers had scattered 50 to 100 inert mines and other ordnance. Our task was to fly our drone over the area and use our machine learning software to detect as many as possible. And we had to turn in our results within 72 hours.
The scale was daunting: The area was 10 times as large as anything we’d attempted before with our drone demining startup, Safe Pro AI. My cofounder Gabriel Steinberg and I used flight-planning software to program a drone to cover the whole area with some overlap, taking photographs the whole time. It ended up taking the drone 5 hours to complete its task, and it came away with more than 15,000 images. Then we raced back to the hotel with the data it had collected and began an all-night coding session.
We were happy to see that our custom machine learning model took only about 2 hours to crunch through all the visual data and identify potential mines and ordnance. But constructing a map for the full area that included the specific coordinates of all the detected mines in under 72 hours was simply not possible with any reasonable computational resources. The following day (which happened to coincide with the short-lived Wagner Group rebellion), we rewrote our algorithms so that our system mapped only the locations where suspected land mines were identified—a more scalable solution for our future work.
In the end we detected 74 mines and ordnance scattered across the surface of that enormous field, and the U.N. deemed our results impressive enough to invite us back for a second round of demonstrations. While we were in Ukraine, we also demonstrated our technology for the State Special Transportation Service, a branch of the Ukrainian military responsible for keeping roads and bridges open.
All our hard work paid off. Today, our technology is being used by several humanitarian nonprofits detecting land mines in Ukraine, including the Norwegian People’s Aid and the HALO Trust, which is the world’s largest nonprofit dedicated to clearing explosives left behind after wars. Those groups are working to make Ukraine’s roads, towns, and agricultural fields safe for the Ukrainian people. Our goal is to make our technology accessible to every humanitarian demining operation, making their jobs safer and more efficient. To that end, we’re deploying and scaling up—first across Ukraine, and soon around the world.
The Scale of the Land-Mine Problem
The remnants of war linger long after conflicts have died down. Today, an estimated 60 countries are still contaminated by mines and unexploded ordnance, according to the 2023 Landmine Monitor report. These dangers include land mines, improvised explosive devices, and shells and artillery that didn’t explode on landing—all together, they’re known as explosive ordnance (EO). More than 4,700 people were killed or wounded by EO in 2022, according to the Landmine Monitor report, and the vast majority of those casualties were civilians. Today, Ukraine is the most contaminated place in the world. About a third of its land—an area the size of Florida—is estimated to contain EO.
In humanitarian mine-clearing work, the typical process for releasing EO-contaminated land back to the community hasn’t changed much over the past 50 years. First a nontechnical survey is conducted where personnel go out to talk with local people about which areas are suspected of being contaminated. Next comes the technical survey, in which personnel use metal detectors, trained dogs, mechanical demining machines, and geophysical methods to identify all the hazards within a mined area. This process is slow, risky, and prone to false positives triggered by cans, screws, or other metal detritus. Once the crew has identified all the potential hazards within an area, a team of explosive-ordnance-disposal specialists either disarm or destroy the explosives.
Unexploded ordnance lies by the road in a Ukrainian town near the war’s front lines. John Moore/Getty Images
Most deminers would agree that it’s not ideal to identify the EO as they walk through the contaminated area; it would be much better to know the lay of the land before they take their first steps. That’s where drones can be literal lifesavers: They take that first look safely from up above, and they can quickly and cheaply cover a large area.
What’s more, the scale of the problem makes artificial intelligence a compelling part of the solution. Imagine if drone imagery was collected for all of Ukraine’s suspected contaminated land: an area of more than 170,000 square kilometers. It takes about 60,000 drone images to cover 1 km 2 at a useful resolution, and we estimate that it takes at minimum 3 minutes for a human expert to analyze a drone image and check for EO. At that rate, it would take more than 500 million person-hours to manually search imagery covering all of Ukraine’s suspected contaminated land for EO. With AI, the task of analyzing this imagery and locating all visible EO in Ukraine will still be a massive endeavor, but it’s within reason.
“Today, our technology is being used by several humanitarian nonprofits detecting land mines in Ukraine.”
Humanitarian demining groups are slow to adopt new technologies because any mistake, including ones caused by unfamiliarity with new tech, can be fatal. But in the last couple of years, drones seem to have reached an inflection point. Many government agencies and nonprofit groups that work on land-mine detection and removal are beginning to integrate drones into their standard procedures. Besides collecting aerial imagery of large areas with suspected hazards, which helps with route planning, the drones are prioritizing areas of clearance, and in some cases, detecting land mines themselves.
After several years of research on this topic during my undergraduate education, in 2020 I cofounded the company now known as Safe Pro AI to push the technology forward and make deployment a reality. My cofounder and I didn’t know at the time that Russia’s full-scale invasion of Ukraine in February 2022 would soon make this work even more vital.
How We Got Started With Drones for Demining
I became interested in land-mine detection while studying geological science as an undergraduate at Binghamton University, New York. Through my work in the Geophysics and Remote Sensing Laboratory run by Timothy de Smet and Alex Nikulin, I got involved in a project to detect the PFM-1, a Russian-made antipersonnel land mine also known as the butterfly mine due to its unique shape and because it’s typically scattered by aircraft or artillery shells. Afghanistan is still contaminated with many of these mines, left behind more than 40 years ago after the Soviet-Afghan War. They’re particularly problematic because they’re mostly made of plastic, with only a few small metal components; to find them with a metal detector requires turning up the equipment’s sensitivity, which leads to more false positives.
In 2019, we trained a machine learning model by scattering inert PFM-1 land mines and collecting visual imagery via drone flights in various environments, including roads, urban areas, grassy fields, and places with taller vegetation. Our resulting model correctly detected 92 percent of PFM-1s in these environments, on average. While we were pleased with its performance, the model could identify only that one type of land mine, and only if they were above ground. Still, this work provided the proof of concept that paved the way for what we’re doing today. In 2020, Steinberg and I founded the Demining Research Community, a nonprofit whose goal is to advance the field of humanitarian mine removal through research in remote sensing, geophysics, and robotics.
Over the next few years, we continued to develop our software and make contacts in the field. At the 2021 Mine Action Innovation Conference in Geneva, we heard about a researcher named John Frucci at Oklahoma State University who directs the OSU Global Consortium for Explosive Hazard Mitigation. In the summer of 2022, we spent two weeks with Frucci at OSU’s explosives range, which has more than 50 types of unexploded ordnance. We used our drones to collect visual training data for many different types of explosives: small antipersonnel mines, larger antitank mines, improvised explosive devices, grenades, and many other dangerous explosive things you never want to encounter.
Our Software Solution for Demining by Drone
To develop our technology for real-world use, Steinberg and I cofounded Safe Pro AI and joined Safe Pro Group, a company that provides drone services and sells protective gear for demining crews. Going into this work, we were aware of many academic proposals for new methods of EO detection that haven’t gotten out of the lab. We wanted to break that paradigm, so we spent a lot of time talking with demining personnel about their needs. Safe Pro Group’s director of operations in Ukraine, Fred Polk, spent more than 200 days last year talking to deminers in Ukraine about the problems they face and the solutions they’d like to see. In light of those conversations, we developed a user-friendly Web application called SpotlightAI. Any authorized person can log on to the website and upload their imagery from a commercial off-the-shelf drone; our system will then run the visual data through our AI model and return a map with all the coordinates of the detected explosive ordnance.
We don’t anticipate that the technology will replace human labor—personnel will still have to go through fields with metal detectors to be sure the drones haven’t missed anything. But the drones can speed up the process of the initial nontechnical survey and can also help demining operators figure out which areas to prioritize. The drone-based maps can also give personnel more situational awareness going into an inherently dangerous situation.
“Drones can be literal lifesavers: They take the first look at a minefield safely from up above.”
The first big test of our technology was in 2022 in Budapest at a Hungarian Explosive Ordnance Disposal test range. At that time, I was at Mount Okmok, a volcano in Alaska’s Aleutian Islands, doing field work on volcanology for my Ph.D., so Steinberg represented Safe Pro AI at that event. He told me via satellite phone that our model detected 20 of the 23 pieces of ordnance, returning the results in under an hour.
After Budapest we made two trips to Ukraine, first to field-test our technology in a real-world minefield environment and then for the 2023 U.N. demonstration previously described. In another trip this past March, we visited minefields in eastern Ukraine that are currently being demined by nonprofit organizations using our SpotlightAI system. We were accompanied by Artem Motorniuk, a Ukrainian software developer who joined Safe Pro Group in 2023. It was incredibly saddening to see the destruction of communities firsthand: Even after the front line has moved, explosive remnants of war still hinder reconstruction. Many people flee, but the ones who stay are faced with difficult decisions. They must balance essential activities such as farming and rebuilding with the risks posed by pursing those activities in areas that might have land mines and explosive ordnance. Seeing the demining operations firsthand reinforced the impact of the work, and listening to the demining operators’ feedback in the field helped us further refine the technology.
4 Ways to Sense Danger
We’ve continued to improve the performance of our model, and it has finally reached a point where it’s almost as good as an expert human in detecting EO on the surface from visual imagery, while performing this task many times faster than any human could. Sometimes it even finds items that are heavily obscured by vegetation. To give it superhuman capabilities to peer under the dirt, we need to bring in other detection modalities. For example, while we originally rejected thermal imaging as a stand-alone detection method, we’re now experimenting with using it in conjunction with visual imaging. The visual--imagery-based machine learning model returns the detection results, but we then add a thermal overlay that can reveal other information—for example, it might show a ground disturbance that suggests a buried object.
The biggest challenge we’re grappling with now is how to detect EO through thick and high vegetation. One strategy I developed is to use the drone imagery to create a 3D map, which is used to estimate the vegetation height and coverage. An algorithm then converts those estimates into a heat map showing how likely it is that the machine learning model can detect EO in each area: For example, it might show a 95 percent detection rate in a flat area with low grass, and only a 5 percent detection rate in a region with trees and bushes. While this approach doesn’t solve the problem posed by vegetation, it gives deminers more context for our results. We’re also incorporating more vegetation imagery into our training data itself to improve the model’s detection rate in such situations.
To offer these services in a scalable way, Safe Pro AI has partnered with Amazon Web Services, which is providing computational resources to deal with large amounts of visual imagery uploaded to SpotlightAI. Drone-based land-mine detection in Ukraine is a problem of scale. An average drone pilot can collect more than 30 hectares (75 acres) of imagery per day, roughly equal to 20,000 images. Each one of these images covers an area of 10 by 20 meters, within which the system must detect a land mine the size of your hand and the color of grass. AWS allows us to utilize extremely powerful computers on demand to process thousands of images a day through our machine learning model to meet the needs of deminers in Ukraine.
What’s Next for Our Humanitarian Demining Work
One obvious way we could improve our technology is by enabling it to detect buried EO, either by visually detecting disturbed earth or using geophysical sensors. In the summer of 2023, our nonprofit experimented with putting ground-penetrating radar, aerial magnetometry, lidar, and thermal sensors on our drones in an attempt to locate buried items.
We found that lidar is useful for detecting trenches that are indicative of ground disturbance, but it can’t detect the buried objects themselves. Thermal imagery can be useful if a buried metal item has a very different thermal signature than the surrounding soil, but we typically see a strong differential only in certain environments and at certain times of day. Magnetometers are the best tools for detecting buried metal targets—they’re the most similar to handheld metal detectors that deminers use. But the magnetic signal gets weaker as the drone gets farther from the ground, decreasing at an exponential rate. So if a drone flies too high, it won’t see the magnetic signatures and won’t detect the objects; but if it flies too low, it may have to navigate through bushes or other terrain obstacles. We’re continuing to experiment with these modalities to develop an intelligent sensor-fusion method to detect as many targets as possible.
Right now, SpotlightAI can detect and identify more than 150 types of EO, and it’s also pretty good at generalization—if it encounters a type of land mine it never saw in its training data, it’s likely to identify it as something worthy of attention. It’s familiar with almost all American and Russian munitions, as well as some Israeli and Italian types, and we can make the model more robust by training it on ordnance from elsewhere. As our company grows, we may want to fine-tune our algorithms to offer more customized solutions for different parts of the world. Our current model is optimized for Ukraine and the types of EO found there, but many other countries are still dealing with contamination. Maybe we’ll eventually have separate models for places such as Angola, Iraq, and Laos.
Our hope is that in the next few years, our technology will become part of the standard procedure for demining teams—we want every team to have a drone that maps out surface contamination before anyone sets foot into a minefield. We hope we can make the world safer for these teams, and significantly speed up the pace of releasing land back to the communities living with remnants of war. The best possible outcome will be if someday our services are no longer needed, because explosive devices are no longer scattered across fields and roads. In the meantime, we’ll work every day to put ourselves out of business.
Reference: https://ift.tt/V2Om8te
No comments:
Post a Comment