To clone it, use:
$ sudo yum install git
$ cd ~
$ git clone https://github.com/Percona-QA/percona-qa.git

reducer.sh was also put directly into this repository (and it is maintained there), so *no* need anymore to separately fetch lp:randgen.

Back to the segfault you keep hitting again and again. Those are ones that we call “qa blockers” – and we tag them on our launchpad with “qablock”. There are several things that you can do with these; 1) ask developers to fix them (or hire Percona to do it for you), or 2) workaround the issue by disabling the particular sql needed to trigger this bug in the sql file you use. This is quite easy to do. Firstly, you need to find out what queries are triggering your issue. Two actions; a) check one of the produced error logs near the end for the “Query: …” statement and b) use gdb to extract all queries that where running at the time of the assertion/crash (we already have scripts for this… see extract_query.gdb and this (including last comment – the variable name changed). Finally, remove or remark these queries in the sql file (and log a bug for the issue seen so it will get fixed in time).

But, for many of these sort of things “we’ve been there, and done that” and all this is already highly automated in percona-qa (except manually disabling qablock sql in the sql files, this is easy/quick to do). Here’s how I suggest going about things; 1) use pquery-run.sh (it uses /dev/shm for nice fast processing), 2) go to the workdir produced by it (as set by you in the $WORKDIR variable in pquery-run.sh) and run ~/percona-qa/pquery-prep-red.sh – this will produce nice testcase reducer scripts ready-to-run … it will auto-extract the query from the error log *and* from the coredump and add these to the end of the pquery sql trace (in the numbered subdirectories) – btw, you could get the to-remark queries from here if that makes sense. 3) Step 3 is optional depending on what you are running: If you are using Percona Server or upstream MySQL, simply run ~/percona-qa/pquery-clean-known.sh. 4) Run ~/percona-qa/pquery-results.sh – see the example in the main text above – based on unique bug text strings, it will ‘classify’ each issue and give you a nice output as to the newly found issues. Don’t be suprised to see only a handful if you are using Percona Server or upstream MySQL and have run the cleaner script pquery-clean-known.sh, as we have logged literally hundreds of bugs over the last months already. Yet, you’re running dual-threaded so you may see new fireworks :). 5) Run the individual reducer scripts. These will reduce your testcase from 30k+ lines to a few hundred, to <5-25, often within the space of an hour/a few hours. Reducer is a powerfull tool. Yet, as you are running two threads, a bit more work may be needed. See how you go (and let us know!). Also, you may like to checkout ~/percona-qa/reproducing_and_simplification.txt which has lots of goodies on how to get testcases to "work" in the first place. Our reproducibility is very high, ~100%, with the occasional "looser" here or there, but in those cases we still have the coredump/stacks from the original trial run and so it's sometimes easy for devs to do a code analysis and find/fix the bug that way.

I hope this answers the questions you had. Looks like a blog post on pquery and it's framework's use would be a good idea too. In short; it is all there already, ready to use. If you have some cool updates, feel free to contribute too. Anyone is free to push to the tree, but please 1) test your changes, 2) be prepared to have changes revoked in case it breaks our integration etc. 🙂

Enjoy!

I am creating a wrapper script like pquery-run.sh. Can you elaborate on how to identify *different* bugs/segmentation faults by using pquery with same main.sql file?

So far I have been able to hit the same segmentation fault again and again with following command:
./pquery-ms –infile=./main.sql –database=test –threads=2 –queries_per_thread=10000 –log_all_queries –log_failed_queries –user=root –port=3306 –addr=127.0.0.1

I am unclear about your statement: “80+ coredumps per hour”. Aren’t they for the same issue?

@Shivam – easy fix; $ export LD_LIBRARY_PATH=${BASEDIR}/lib where ${BASEDIR} is your mysqld’s basedir (or your specific location). Depending on whether your are testing upstream mysqld or Percona Server, you would want to swap between using the pquery (Percona) or pquery-ms (MySQL) binary. Alternatively, see compile information in pquery subdirectory to self-compile pquery, potentially with libs included! As a side note, I may recommend using export LD_PRELOAD=/usr/lib64/libjemalloc.so.1 to use jemalloc (depending on what flavor of mysqld you are testing). Also, you may want to use pquery-run.sh instead – this wrapper script around pquery allows automated pquery runs with all bells and whisles included ($ cd ~; bzr branch lp:percona-qa; vi ~/percona-qa/pquery-run.sh). See reply to Mark above on further steps after pquery-run.sh. In pquery-run.sh you need to set some settings like basedir, pquery or pquery-ms binary etc. See in-script for details.

I am trying to run:

${PQUERY_BIN} –infile=${RUNDIR}/${TRIAL}/${TRIAL}.sql –database=test –threads=${THREADS} –queries_per_thread=${QUERIES_PER_THREAD} –logdir=${RUNDIR}/${TRIAL} –log_all_queries –log_failed_queries –user=root –addr=some_ip –port=3306 >${RUNDIR}/${TRIAL}/pquery.log 2>&1 &

(libperconaserverclient.so.18 and libmysqlclient.so are present in /usr/lib64)

…Yes, I know, the originator of this definition was seriously unclear on the concept of NOT NULL. But it shouldn’t have crashed MySQL.