As spatiotemporal datasets grow, accessing and processing them for analysis and visualization is increasingly the primary bottleneck for their use. Challenges include retrieving, resampling, and analyzing large and often disparately located data. Utilization of large-scale computing resources can be helpful, but may still incur delays due to extensive data transfers, job scheduling, and remote access. And some applications, such as those for public safety, must remain interactive even as data sizes increase. To enable utilization of increasingly massive datasets, it is worthwhile to invest in the creation of workflows that guarantee interactivity, making the broadest set of inquiries possible at minimal cost.

In this work, I present a framework that addresses several common pitfalls of interactive data analysis and visualization. It is comprised of an embedded domain-specific language (EDSL) and associated runtime specifically designed for the interactive exploration of large, remote data ensembles. The EDSL is an extension of JavaScript, which allows users to express a wide range of analyses in a simple and abstract manner. The underlying runtime utilizes a streaming, multiresolution data layout, transparently resolving issues such as remote data access and resampling, and maintaining interactivity through progressive, interruptible computation. This framework enables interactive exploration of massive, remote datasets, such as the 3.5 petabyte 7km NASA GEOS-5 “Nature Run” simulation, for which remote users have previously been able to analyze only offline or at reduced resolution. Most available climate data are stored using legacy file formats that prohibit incremental, multiresolution access. In order for the framework to automatically read these datasets, I developed an on-demand conversion module, currently deployed at Lawrence Livermore National Lab as part of the Earth System Grid Federation (ESGF) platform.

Based on the techniques used for this framework, I also propose a general purpose model to aid creation and evaluation of other interactive workflows for large, remote data. I present the necessary components of such workflows along with important considerations regarding their design and integration, including comprehensive runtime management, effective communication, interruptibility, appropriate data formats, and programming models that facilitate progressive refinement of results.