Core Area: User Experience
Underlying all of our work is our ultimate goal: helping people work with information. We bring a user-centered design approach to our work, collaborating closely with users to understand their problems, test solutions, and deliver usable and useful software products.
Research Topics and Products
Threat Stream Generator - Synthetic datasets
Pacific Northwest National Laboratory’s Threat Stream Generator (TSG) provides information analytic software developers with a new approach and better tools for evaluation than ever before. Originally developed for government use, and built upon feedback from a worldwide beta user community, the complete set of TSG's synthetic scenarios, analytic tasks, and complex datasets are available at no cost to users in commercial and educational domains. The key ingredients that enable unbiased and insightful evaluation of analytic software before operational use are scenarios and tasks with known ground truth and realistic supporting data. The Threat Stream suite of tools provides information of a quality that otherwise could only be found in national security or law enforcement agencies.
User-centered evaluation of visual analytics environments
We have approached user-centered evaluations from several perspectives. First, we have developed a reviewing system for the VAST Challenges that combines reviews from end-users and visual analytics researchers to gain different perspectives on the capabilities that are important in these environments. VAST Challenge participants receive these comments and are able to determine if changes should be made to their work.
We took one year of reviews for all the mini challenges (VAST 2009) combined with additional input from professional analysts and analyzed the comments to obtain the most important aspects of the visual analytic environments. We used this information along with additional papers from the literature and constructed an initial set of guidelines for visual analytics environments. Some of these guidelines are borrowed from other related domains such as human computer interaction (HCI), situation awareness (SA), and human-automation.
Developing guidelines for assessing visual analytics environments by Jean Scholtz will be published in Information Visualization in the fall, 2011 edition.
Related Papers
Show all abstractsCollaborative Visualization: Definition, Challenges, and Research Agenda
Isenberg P, N Elmqvist, J Scholtz, D Cernea, KL Ma, and H Hagen. 2011. Collaborative Visualization: Definition, Challenges, and Research Agenda. Information Visualization.
Abstract
Collaborative visualization has emerged as a new research direction which offers the opportunity to reach new audiences and application areas for visualization tools and techniques. Technology now allows us to easily connect and collaborate with one another—in settings as diverse as over networked computers, across mobile devices, or using shared displays such as interactive walls and tabletop surfaces. Any of these collaborative settings carries a set of challenges and opportunities for visualization research. Digital information is already regularly accessed by multiple people together in order to share information, to view it together, to analyze it, or to form decisions. However, research on how to best support collaboration with and around visualizations is still in its infancy and has so far focused only on a small subset of possible application scenarios. The purpose of this article is (1) to provide a clear scope, definition, and overview of the evolving field of collaborative visualization, (2) to help pinpoint the unique focus of collaborative visualization with its specific aspects, challenges, and requirements within the intersection of general computer-supported collaborative work (CSCW) and visualization research, and (3) to draw attention to important future research questions to be addressed by the community. Thus, the goal of the paper is to discuss a research agenda for future work on collaborative visualization, including our vision for how to meet the grand challenge and to urge for a new generation of visualization tools that were designed with collaboration in mind from their very inception.
Developing Qualitative Metrics for Visual Analytic Environments
Scholtz J. 2010. Developing Qualitative Metrics for Visual Analytic Environments. In BELIV '10: Beyond time and errors: novel evaluation methods for Information Visualization, A Workshop of the ACM CHI Conference, April 10-11, 2010, Atlanta, Georgia. Association for Computing Machinery, New York, NY.
Abstract
In this paper, we examine reviews for the entries to the 2009 Visual Analytics Science and Technology (VAST) Challenge. By analyzing these reviews we gained a better understanding of what is important to our reviewers, both visualization researchers and professional analysts. This is a bottom up approach to the development of heuristics to use in the evaluation of visual analytic environments. The meta-analysis and the results are presented in this paper.
Application and Evaluation of Analytic Gaming
Riensche RM, LM Martucci, J Scholtz, and MA Whiting. 2009. Application and Evaluation of Analytic Gaming. In 2009 International Conference on Computational Science and Engineering, August 29-31, 2009, Vancouver, Canada, 4:1169-1173. IEEE Computer Society, Los Alamitos, CA.
Abstract
We describe an "analytic gaming" framework and methodology, and introduce formal methods for evaluation of the analytic gaming process. This process involves conception, development, and playing of games that are informed by predictive models and driven by players. Evaluation of analytic gaming examines both the process of game development and the results of game play exercises.
VAST Contest Dataset Use in Education
Whiting MA, C North, A Endert, J Scholtz, JN Haack, CF Varley, and JJ Thomas. 2009. VAST Contest Dataset Use in Education. In IEEE Symposium on Visual Analytics Science and Technology (VAST 2009), ed. J Stasko and JJ van Wijk, pp. 115 - 122. IEEE, Piscataway, NJ.
Abstract
The IEEE Visual Analytics Science and Technology (VAST) Symposium has held a contest each year since its inception in 2006. These events are designed to provide visual analytics researchers and developers with analytic challenges similar to those encountered by professional information analysts. The VAST contest has had an extended life outside of the symposium, however, as materials are being used in universities and other educational settings, either to help teachers of visual analytics-related classes or for student projects. We describe how we develop VAST contest datasets that results in products that can be used in different settings and review some specific examples of the adoption of the VAST contest materials in the classroom. The examples are drawn from graduate and undergraduate courses at Virginia Tech and from the Visual Analytics "Summer Camp" run by the National Visualization and Analytics Center in 2008. We finish with a brief discussion on evaluation metrics for education.
User-Centered Evaluation of Technosocial Predictive Analysis
Scholtz J, M Whiting. 2009. User-Centered Evaluation of Technosocial Predictive Analysis. Association for the Advancement of Artificial Intelligence 2009
Abstract
In today's technology filled world, it is absolutely essential to show the utility of new software, especially software that brings entirely new capabilities to potential users. In the case of technosocial predictive analytics, researchers are developing software capabilities to augment human reasoning and cognition. Getting acceptance and buy-in from analysts and decision makers will not be an easy task. In this position paper, we discuss an approach we are taking for user-centered evaluation that we believe will result in facilitating the adoption of technosocial predictive software by the intelligence community.