Axiom-based aggregation functions for calculating variety, novelty, quality and quantity of ideation results

Proposes axioms and matching functions for principled aggregation of quality, quantity, novelty, and variety scores.

Evaluating idea sets from design ideation sessions commonly relies on Shah, Vargas-Hernandez, and Smith's (SVS) four criteria — quality, quantity, novelty, and variety — which help teams and researchers gauge the thoroughness of ideation and compare different ideation methods. The authors argue that the existing methods for aggregating individual idea-level scores into a single set-level score for each criterion are problematic: for example, there is no established way to compute a novelty score for an entire set, no clear method for deciding when two ideas are different enough to count separately, no way to assess how much an individual idea adds to a set's variety, and the standard variance calculation effectively yields only an "average" variety since it is divided by the number of ideas.

To address these gaps, the paper proposes a set of axioms describing the desired behavior of aggregation functions for quality, quantity, variety, and novelty, then defines concrete aggregation functions that satisfy those axioms. The axioms are designed so that resulting scores reward ideation behaviors considered best practice, such as encouraging every idea in a set to meaningfully contribute rather than rewarding sheer volume alone.

The result is a more principled, defensible framework for scoring ideation output, intended to improve both design-team self-assessment and cross-method comparisons in ideation research — building on the same research group's prior applications of the SVS metrics in design education and research.

Cite this paper

Read the full paper

APA

Sorensen, C. D., Ashworth, T. J., Stapleton, T., Mattson, C. A., & Anderson, M. L. (2026). Axiom-based aggregation functions for calculating variety, novelty, quality and quantity of ideation results. Design Science, 12, E9. https://doi.org/10.1017/dsj.2026.10051

MLA

Sorensen, Carl D., et al. "Axiom-Based Aggregation Functions for Calculating Variety, Novelty, Quality and Quantity of Ideation Results." Design Science, vol. 12, 2026, p. E9, doi:10.1017/dsj.2026.10051.

Chicago (Author-Date)

Sorensen, Carl D., Thomas J. Ashworth, Tyler Stapleton, Christopher A. Mattson, and Michael L. Anderson. 2026. "Axiom-Based Aggregation Functions for Calculating Variety, Novelty, Quality and Quantity of Ideation Results." Design Science 12: E9. https://doi.org/10.1017/dsj.2026.10051.

IEEE

C. D. Sorensen, T. J. Ashworth, T. Stapleton, C. A. Mattson, and M. L. Anderson, "Axiom-based aggregation functions for calculating variety, novelty, quality and quantity of ideation results," Design Science, vol. 12, p. E9, 2026, doi: 10.1017/dsj.2026.10051.

BibTeX

bibtex

@article{sorensen2026axiom,
  author  = {Sorensen, Carl D. and Ashworth, Thomas J. and Stapleton, Tyler and Mattson, Christopher A. and Anderson, Michael L.},
  title   = {Axiom-Based Aggregation Functions for Calculating Variety, Novelty, Quality and Quantity of Ideation Results},
  journal = {Design Science},
  volume  = {12},
  pages   = {E9},
  year    = {2026},
  doi     = {10.1017/dsj.2026.10051}
}