ASCAR Pilot Bird-of-a-Feather Session at USENIX ATC16 on 6/22/2016

June 20, 2016

We will host a bird-of-a-feather session at USENIX ATC’16 at Denver, CO on Wednesday, June 22, 8:00 pm–9:00 pm, Gold Coin Room.

Carrying out even the simplest form of performance benchmark requires considerable knowledge of statistics and computer systems, and painstakingly following many error-prone steps. In fact, many performance measurements in peer-reviewed publications are flawed. Among many problems, they fall short in one or more of the following requirements: accuracy, precision, comparability, repeatability, and control of overhead. This is a serious problem because poor performance measurements mis-guide system design and optimization. We are designing a collection of algorithms and heuristics to automate the steps that are necessary to produce statistically sound performance measurement results. They cover the collection, storing, analysis, and comparing of performance measurements and can help to reduce human effort and error. We are implementing these methods and will release and manage it as a readily-usable open source software framework. Please come to the BoF session to gain early access to the code and talk about what kind of benchmark tasks you'd like to see automated.

Read more about the ASCAR Pilot Framework or the USENIX ATC'16 BoF session.

SSRC Contact: Li, Yan