Computer Benchmark Evaluation and Design of Experiments, a Case Study

We define “installation tuning” and motivate it as an important area of concern for computer configurations. Examples of installation tuning include, deciding on the paging configuration for a particular workload and partitioning available memory into system and user areas. Installation tuning is complex because available tuning options are often difficult to select due to poor understanding of their effects. Thus, analytic models, so often used in capacity planning studies, rarely exist for these areas. We illustrate the use of statistical models, specifically analysis of variance models, to assess the significance of installation tuning factors. A case study is presented in which a statistical model for response time is developed incorporating the key factors of user load, paging, and main storage configuration. The paper presents an interesting case study of applying the statistical design of experiments to computer performance analysis. General statistical issues pertaining to the design and analysis of computer performance experiments are identified and illustrated in this study. © 1992 IEEE