I recently worked with a customer who had a weird issue: when their MySQL server was started (Percona Server 5.5), if they try to run service mysql start a second time, the init script was not able to detect that an instance was already running. As a result, it tried to start a second instance with the same settings as the first one. Of course this fails and this creates a mess. What was the issue? A missing rule in SELinux. At least it looks like
Summary
If SELinux is set to enforcing and if you are using Percona Server on CentOS/RHEL 6 (other versions could be affected), service mysql start doesn’t work properly and a fix is simple to run:
1 2 3 | # grep mysqld_safe /var/log/audit/audit.log | audit2allow -M mysqld_safe # semodule -i mysqld_safe.pp # service mysql restart |
Other options are:
- Set SELinux to permissive
- Use the CentOS/RHEL standard MySQL init script (note I didn’t extensively check if that could trigger other errors)
How did we see the issue?
That was pretty easy: if an instance is already running and if you run service mysql start again, you should see something like this in the MySQL error log:
1 | 150717 08:47:58 mysqld_safe A mysqld process already exists |
But if you rather see tons of error messages like:
1 2 | 2015-07-17 08:47:05 27065 [ERROR] InnoDB: Unable to lock ./ibdata1, error: 11 2015-07-17 08:47:05 27065 [Note] InnoDB: Check that you do not already have another mysqld process using the same InnoDB data or log files. |
it means that the init script is broken somewhere.
Investigation
When the issue was brought to my attention, I tried to reproduce it on my local box, but with no luck. What was so special in the configuration used by the customer?
The only thing that was slightly out of the ordinary was SELinux which was set to enforcing. Then we set SELinux to permissive, and guess what? service mysql start was now working properly and it didn’t allow 2 concurrent instances to be run!
Next step was to look at the SELinux logs to find any error related to MySQL, and we discovered messages like:
1 2 3 4 | type=SYSCALL msg=audit(1437121845.464:739): arch=c000003e syscall=62 success=no exit=-13 a0=475 a1=0 a2=0 a3=7fff0e954130 items=0 ppid=1 pid=5732 auid=500 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts0 ses=5 comm="mysqld_safe" exe="/bin/bash" subj=unconfined_u:system_r:mysqld_safe_t:s0 key=(null) |
At this point, we knew that a rule was missing for mysqld_safe, we needed to add a new one.
Deeper investigation
Actually what happens is that SELinux prevents this condition to be executed in mysqld_safe:
1 | if kill -0 $PID > /dev/null 2> /dev/null |
and then the script assumes that this means the mysqld process is not running. That’s why a second mysqld is started.
However users of Oracle MySQL will probably never experience that issue, simply because the init script is a bit different: before calling mysqld_safe, the init script tries to ping a potential mysqld instance and if it gets a positive reply or an Access denied error, it concludes that mysqld is already running and it doesn’t invoke mysqld_safe.
The fix
Fortunately, this is quite simple. You can generate the corresponding rule with audit2allow:
1 | grep mysqld_safe /var/log/audit/audit.log | audit2allow -M mysqld_safe |
And after checking the corresponding .te file, we were able to load that new module:
1 | semodule -i mysqld_safe.pp |
After stopping MySQL, you can now use service mysql start normally.
Conclusion
This issue was quite interesting to work on because finding the culprit was not that easy. Also it only triggers when SELinux is enabled and Percona Server is used. Now should the init script of Percona Server be fixed? I’m not sure of the potential problems that could occur if we did so, but of course feel free to leave your feedback in the comments.
Hi! Good stuff.
I’m not a fan of adding custom selinux modules since they tend to become outdated as your Linux distro fixes and changes the selinux rules, and they tend to open up too much of the system (since most people don’t look too closely at what the “audit2allow” script generates).
Instead, follow the recommendations here: https://blogs.oracle.com/jsmyth/entry/selinux_and_mysql and use “semanage” and “restorecon” to label objects with the usual selinux contexts.