June 30, 2017 • Updated July 5, 2017
Melissa Hanham, Catherine Dill, Jeffrey Lewis, Bo Kim, Dave Schmerler, Joseph Rodgers
Occasional Paper #28
Read the full Occasional Paper #28:
Geo4nonpro.org: A Geospatial Crowd-Sourcing Platform for WMD Verification
EXECUTIVE SUMMARY
Geo4nonpro is a web-based platform for public viewing and annotating satellite imagery. The James Martin Center for Nonproliferation Studies (CNS) sought to harness the power of the crowd to interpret imagery of sites with known or suspected WMD ties. The research team invited a “curated” crowd of experts to annotate and comment on the imagery. The crowd was global in composition, with experience ranging from science and technology to policy and area studies. The Geo4nonpro website has two main components: a web server and an imagery server.
The CNS research team procured and processed batches of imagery for seven sites during the pilot phase of Geo4nonpro in 2016: The Punggye-ri Nuclear Test Site (North Korea), Directorate of Defense Industries (DDI) facilities (Myanmar), the Sinpo South Naval Shipyard (North Korea), the Novaya Zemlya Nuclear Test Site (Russia), the Shahrud Missile Test Facility (Iran), the Korla Missile Test Complex (China), and Kaliningrad missile deployments (Russia). The public and press were able to watch the experiment in real-time on a public-facing website, join in by using the #geo4nonpro hashtag on Twitter, or even apply to become part of the curated crowd.
For each of the cases studies, the CNS research team gave a rationale for the case, described the imagery and the challenge for the crowd, gave CNS expert analysis of the imagery, analyzed the crowd’s response, and drew lessons from the case. Crowd participation ranged from enthusiastic to subdued depending on the site, but across the board, the crowd fared well at accurately identifying objects in satellite imagery for the seven challenges. It remains to be seen whether the crowd is good at drawing overall inferences from these identifications the way an analyst would.
While the project was lauded as a success internally and externally, there were several critical areas that CNS wishes to address in the future, including improving the user interface, asking permission to gather more user data for a robust analysis of the crowd, and exploring whether user-generated data could be useful in machine-learning training algorithms for automated detection of features in satellite imagery of WMD-related sites.
