It’s tough being an imagery analyst for the U.S. military: you’re drowning in pictures and drone video, with more pouring in endlessly from the tons of sensors and cameras used on planes, ships and satellites. Sifting through it to find roadside bombs or missile components is a time-consuming challenge. That’s why the Pentagon’s blue sky research arm figures that cameras ought to be able to filter out useless information themselves — so you don’t have to.
Darpa announced yesterday that it’s moving forward in earnest with a program to endow cameras with “visual intelligence.” That’s the ability to process information from visual cues, contextualize its significance, and learn what other visual data is necessary to answer some pre-existing question. Visual-intelligence algorithms are already out there. They can read license plates in traffic or recognized faces (in limited, brighly-lit circumstances). But the programs are still relatively dumb; they simply help collate data that analysts have to go through. Darpa’s program, called Mind’s Eye, seeks to get humans out of the picture. If it works, it could change the world of surveillance overnight.
Following on a March conference for potential contractors, Darpa has given 12 research teams, mostly based at universities, contracts to build these thinking cameras. The initial idea is to mount them on drones for ground surveillance, so robots can take dangerous scouting responsibilities away from troops. In theory, humans wouldn’t be required to instruct the scouts while they wheel around about what pictures to take.
That’s the crucial distinction between Mind’s Eye and every surveillance system the military has. Powerful cameras and sensors, whether they’re the Reaper-mounted Gorgon Stare, with its two-mile-plus field of vision, or the 1.8 gigapixel ARGUS-IS camera for Special Operations helicopters still require a crucial element: You. Even when hooked up to drones, someone needs to tell the cameras what to shoot, and even more people need to mine that data for significance. And “star[ing] at Death TV for hours on end trying to find the single target or see something move” is just “a waste of manpower,” Gen. James Cartwright, the vice chairman of the joint chiefs of staff, recently told an intelligence conference.
So Darpa wants to push artificial intelligence forward in a big way. It envisions its research teams making “novel contributions in visual event learning, new spatiotemporal representations, machine-generated envisionment, visual inspection and grounding of visual concepts.” All that will spot “operationally significant activity and report on that activity so warfighters can focus on important events in a timely manner.” If you’re an imagery-data jockey, you might be free to see a ballgame sometime.
And while all this is clearly a long way away — Darpa didn’t set out a timeline in its announcement — Mind’s Eye would have dramatic privacy implications. After all, military technology typically filters down to law enforcement, given time. Right now, the firehose of data that surveillance cameras give to government analysts acts as de facto privacy protection for individuals caught up in a sprawling surveillance net. But what happens when that firehose becomes a targeted stream? What happens when cameras decide for themselves who to spy on?
For now, Darpa doesn’t intend the images collected by Mind’s Eye to be so extensive. Even if its researchers can develop the visual-intelligence software, it wants to first mount the thinking cameras on robo-scouts like the Army’s Small Unmanned Ground Vehicle, not aboard an airborne drone. The ambition is huge, but the initial scope is small. Still, the mind’s eye has a tendency to wander.