Evaluating Information Visualization in Large Companies: Challenges, Experiences and Recommendations

Authors. Michael Sedlmair, Petra Isenberg, Dominikus Baur, Andreas Butz

Venue. CHI BELIV (2010) Workshop Paper

Type. Workshop Paper

Materials. DOI PDF

Abstract. We examine the process and some implications of evaluating information visualization in a large company setting. While several researchers have addressed the difficulties of evaluating information visualizations with regards to changing data, tasks, and visual encodings, considerably less work has been published on the difficulties of evaluation within specific work contexts. In this paper, we specifically focus on the challenges arising in the context of large companies with several thousand employees. We present a collection of evaluation challenges, discuss our own experiences conducting information visualization evaluation within the context of a large automotive company, and present a set of recommendations derived from our experiences. The set of challenges and recommendations can aid researchers and practitioners in preparing and conducting evaluations of their products within a large company setting.

@inproceedings{10.1145/2110192.2110204,
    title        = {Evaluating Information Visualization in Large Companies: Challenges, Experiences and Recommendations},
    author       = {Sedlmair, Michael and Isenberg, Petra and Baur, Dominikus and Butz, Andreas},
    year         = {2010},
    booktitle    = {Proceedings of the 3rd BELIV'10 Workshop: BEyond Time and Errors: Novel EvaLuation Methods for Information Visualization},
    publisher    = {Association for Computing Machinery},
    series       = {BELIV '10},
    pages        = {79–86},
    doi          = {10.1145/2110192.2110204},
    url          = {https://doi.org/10.1145/2110192.2110204},
    abstract     = {We examine the process and some implications of evaluating information visualization in a large company setting. While several researchers have addressed the difficulties of evaluating information visualizations with regards to changing data, tasks, and visual encodings, considerably less work has been published on the difficulties of evaluation within specific work contexts. In this paper, we specifically focus on the challenges arising in the context of large companies with several thousand employees. We present a collection of evaluation challenges, discuss our own experiences conducting information visualization evaluation within the context of a large automotive company, and present a set of recommendations derived from our experiences. The set of challenges and recommendations can aid researchers and practitioners in preparing and conducting evaluations of their products within a large company setting.},
    numpages     = {8},
    keywords     = {information visualization, company setting, evaluation}
}

Best Paper Award