If you attended my latest Troubleshooting MySQL Concurrency Issues with Load Testing Tools webinar you learned how I exploit benchmarking tools to replicate locking issues and crashes. But I told you nothing about proper use of these tools: for reliable benchmarks. I did not tell you that for a reason… I am just a Support Engineer and not a benchmarking expert.
And I have a dream. I want to invite world famous MySQL benchmarking experts for a Percona webinar and ask them for their secrets. I have a pretty long list of my own questions, but in order to make 1-hour webinar productive, we need to concentrate on just a few of the hottest ones. Therefore I need your help.
Please add your questions into the comments field of this blog post. Then we will choose the most important and/or most popular of these to discuss at the webinar.
Some benchmarking questions
My list of questions includes:
- What do you benchmark? Are they real applications or artificial tests?
- How do you create tests for a real application workload?
- Do you test hardware?
- Which tools do you use? How? Any tips?
- How do you work with hardware limitations when the test machine is smaller than the production one?
- Which are the most common mistakes for benchmarks?
- We all heard about “Lies, Damned Lies and Benchmarks”. Why do you still benchmark? How are you sure that yours have meaningful results?
- What are the most common mistakes made by those who publish benchmark results for public use?
- What are the best practices for benchmarks?
Please tell me what you think about these questions and add yours!
Can we invite the worlds famous *non-MySQL* benchmarking experts, and have them share their opinion on how people run MySQL benchmarks. That would be more interesting.
Of course! If they want to =) Do you mean someone special?
Neil J. Gunther . I’d like to hear more about how to accurately test scalable systems (throughput, response time distribution and constraints), and have him look at and maybe even constructively criticize benchmarks in MySQL space 🙂
I’d be interested in learning something about how to properly benchmark different DBMSes (MySQL, MongoDB, Cassandra, …) and different storage engines against eachother, especially what benchmarking tools are available for this and how to use them.