New intel program will tap AI to help personnel ‘walk through’ unfamiliar areas before they arrive

The intelligence community’s primary research arm launched a new program to develop software algorithm-based systems that will fuse imagery captured from various altitudes and angles — including from traffic cameras, drones, satellites and other platforms — to build immersive, photorealistic virtual environments of unfamiliar locations across the globe.

Through its new Walk-Through Rendering Images of Varying Altitudes (WRIVA) program, the Intelligence Advanced Research Projects Activity (IARPA) will foster technology to acquaint government officials with potentially dangerous places before they deploy there.

“Imagine if law enforcement, the military, or aid workers could virtually drop themselves into a location and look around and become familiar with it before their arrival. These groups often have to deliver rapid support and lifesaving aid to unfamiliar or dynamic areas. Allowing them to prepare ahead of time helps keep them out of harm’s way when they have to conduct these activities,” IARPA Program Manager Ashwini Deshpande told reporters Friday during a preview of the project and the accompanying broad agency announcement (BAA) for federal funding, prior to its release on Tuesday. 

Inspired by a tween

WRIVA marks the first research project Deshpande is steering for the agency. She was inspired to pitch it after an experience that’s likely familiar to many parents. 

“One day I was trying to explain to my tween where I was going to pick her up when she was hanging out with her friends, but we ran into a problem because I couldn’t describe where I was going to be in a way that she could figure out. And when we went around looking for more information, we discovered that part of the problem could be that the area had recently undergone a lot of changes and we were working with some outdated information,” she recalled.

The solution she and her daughter landed on — picking a different spot to meet — seemed easy. But the entire scenario got Deshpande thinking about similar challenges for government officials.

“For instance, what if law enforcement needs to know where they might be vulnerable when responding to a threat? Or what if we need to respond to needs in an area where the landscape has recently changed due to a bombing or natural disaster?” she said. “Finding a solution is really critical.”

With years of experience as a chief scientist and technical advisor, Deshpande brought the problem to IARPA — a federal research hub that works on projects for the intelligence community (IC) and other agencies — and successfully pitched the project.

“WRIVA is aiming to help users visually see or plan a mission activity. And this will be a game-changer for the IC, military, law enforcement, and also for humanitarian and disaster relief,” she said.

Heaps of images and data from a number of sources about a specific location can be applied to create site models, which can then enable personnel to rehearse missions with better knowledge about where they’ll be operating. But such tools typically require massive amounts of that type of information to work properly. Through WRIVA, Deshpande and her team want to produce software systems to perform site modeling in scenarios where huge volumes of ground-level imagery with reliable metadata are not easily accessible or available.

“Our goal with WRIVA is to leapfrog off of some recent advancements in machine learning and computer vision and to further the technology in the areas of reducing the number of viewpoints that are required to create site models,” she noted. 

Those involved in the project also intend to increase the speed at which site models are completed and approved for use within the federal government. The envisioned outcomes of WRIVA will essentially be algorithms and methodologies that enable officials to rapidly create site models without full 360-degree coverage of a location — and methodologies to repair corrupted imagery.

“If you think about the history of us wanting to familiarize ourselves with an area that we can’t get to, that’s existed for so long. I mean, it goes back to [Greek mythology with] Icarus, and to the time in World War I when we were strapping cameras on the pigeons,” Deshpande said. “But I think the thing that’s going to be really different here is the level of immersiveness that these models will experience.”

Other site modeling capabilities exist within the IC and Defense Department. However, the processes to build such tools are often incredibly slow, tedious and require a lot of lead time, she noted.

“I think the real technological breakthroughs will be the timeliness of the model creation to allow us to respond very quickly, as well as the fact that you won’t need to spend hours and require a ton of expertise to collect the basic imagery to create these site models,” Deshpande said. “This will really allow people to practice almost like they’re in a video game, you know, practicing their scenarios before they have to go and conduct their mission in person.”

A ‘transitionable’ capability

The WRIVA program is slated to be a 42-month research effort with hands-on work beginning in fiscal 2023. 

The locations it will actually be applied for will depend on the nation’s specific needs three-and-a-half years from now, once the envisioned technology is fully developed.

“But I think we are planning on conducting experiments that represent a variety of different scenarios,” Deshpande explained. “One of the areas of concern is improvised housing such as refugee camps — we are planning on doing data collections that emulate scenarios like that. We are also planning on doing data collections that apply towards suburban environments. We have a wide-ranging set of field experiments and challenges for proposers to look forward to.”

So far, IARPA has named MITRE Corp., Johns Hopkins University Applied Physics Laboratory and Massachusetts Institute of Technology (MIT) Lincoln Laboratory as testing and evaluation partners for this effort. 

Building on their broad pursuits to create and refine transformational technologies for disaster relief and humanitarian assistance, MIT officials will work to generate applicable high-fidelity datasets and build infrastructure to assess the performance of capabilities IARPA’s partners produce.

“There’s almost immediate application for technology like this within damage quantification — so you could think just after a disaster when things are damaged, and people are trying to understand what is broken and what needs to be fixed, a tool like this could actually help not only convey that information, but also notably here in the U.S., can expand the available workforce that could come and respond,” MIT Lincoln Laboratory’s Group Leader Adam Norige told reporters during the briefing Friday. 

Research agencies like IARPA often conduct exploratory ventures and don’t play a big part in completely operationalizing the technology they creates — but, considering applications that already exist, those involved in WRIVA are “fairly confident” it will eventually be fielded.

“Our goal is to create a transitionable capability. We have several partners across the DOD and IC who are interested and who are heavily engaged as stakeholders, as members of our government advisory panel and as potential transition partners, and we’re working closely with them even starting now to make sure that their needs are addressed and that they are also putting in the infrastructure necessary to be able to take the WRIVA program and implement it against their own operational needs,” Deshpande said.

“We’ve got a lot of interest from, frankly, several organizations within the military, but we are also working closely with IC elements that more directly support the warfighter,” she noted.