This event has ended. View the official site or create your own event → Check it out
This event has ended. Create your own
The attendees list includes all authors (even thought they may not be attending), speakers, artists, etc. 

View the full conference website here:
IEEE Cluster 2013 Conference
View analytic
Wednesday, September 25 • 2:00pm - 2:25pm
Fast Data Analysis with Integrated Statistical Metadata in Scientific Datasets

Sign up or log in to save this to your schedule and see who's attending!

Scientific datasets and libraries, such as HDF5, ADIOS, and NetCDF, have been used widely in many data-intensive applications. These libraries have their special file formats and I/O functions to provide efficient access to large datasets. Recent studies have started to utilize indexing, subsetting, and data reorganization to manage the increasingly large datasets. In this work, we present an approach to boost the data analysis performance, namely Fast Analysis with Statistical Metadata (FASM), via data subsetting and integrating a small amount of statistics into the original datasets. The added statistical information illustrates the data shape and provides knowledge of the data distribution; therefore the original I/O libraries can utilize these statistical metadata to perform fast queries and analyses. Various subsetting schemes can affect the access pattern and the I/O performance. We present a comparison study of different subsetting schemes by focusing on three dominant factors, the shape, the concurrency, and the locality. The added statistical metadata slightly increases the original data size, and we evaluate the cost and trade-off as well. This work is the first study that utilizes statistical metadata with various subsetting schemes to perform fast queries and analyses on large datasets. The proposed FASM approach is currently evaluated with the PnetCDF on Lustre file systems, but can also be implemented with other scientific libraries. The FASM can potentially lead to a new dataset design and can have an impact on big data analysis.


Wednesday September 25, 2013 2:00pm - 2:25pm
15th Floor - Circle City 15 (Hilton) 120 W. Market St, Indianapolis, IN