Comments on: Auto-bootstrapping an all-down cluster with Percona XtraDB Cluster

By: sanjay

sanjay — Mon, 23 Nov 2015 23:55:21 +0000

Thanks Jay

I have followed steps but when I trying to restart it
service mysql start
ERROR! MySQL (Percona XtraDB Cluster) is not running, but PID file exists

So, I have to manually remove PID file in order to start it.

By: Jay Janssen

Jay Janssen — Mon, 20 Jul 2015 12:02:15 +0000

@Brian — gvwstate.dat is really a best-effort auto-recovery from an all-down situation. If you’re already logged in, you may as well just restart the nodes yourself.

By: Brian Kruger

Brian Kruger — Wed, 15 Jul 2015 17:48:04 +0000

Been playing around with this. Adding some additional help for people if they come across this.

If you do lose your whole cluster, for this to work, all of the nodes need to come back on line that are listed in the gvwstate.dat .. If you manage to not have a machine comeback after a power outage for instance, you can edit gvwstate.dat and remove the dead host uuid and restart.

The big question is with going through this effort, is it easier to just re-bootstrap at that point ?

By: Jay Janssen

Jay Janssen — Mon, 06 Apr 2015 18:19:29 +0000

@morgan — If the nodes cannot recover, then they should eventually timeout and exit. If this happens, you can manually bootstrap one and restart the others normally. During the timeout I would not expect apps to have any access at all.

By: Morgan Jones

Morgan Jones — Mon, 06 Apr 2015 18:09:21 +0000

Jay,

You say that the cluster will recover if all nodes are started and they are able to recover the PRIMARY component. What will happen if they cannot recover the PRIMARY component for some reason? Will the nodes be left running, but not replicating? Will the user be able to access the database?

Thanks,

By: Jay Janssen

Jay Janssen — Sun, 07 Dec 2014 19:08:10 +0000

@Peter — my understanding is that this works by auto-rejoining the last PRIMARY component (if any). The only reason nodes might have different GTIDs is because apply is asynchronous. My understanding is that state transfer will happen normally in that case (and I guess a full SST), but I believe this is after the decision to go primary is reached. In this case, only node(s) with the highest GTID should continue, while the others ST.

By: Peter Zaitsev

Peter Zaitsev — Fri, 05 Dec 2014 22:26:20 +0000

Jay,

I wonder how do we find which node really was discovered to be latest and actually was PRIMARY and gave IST to others (hopefully)

I am testing this and I see:

NODE1:
2014-12-05 17:03:25 2241 [Note] WSREP: promote to primary component

NODE2:
2014-12-05 17:03:25 2194 [Note] WSREP: promote to primary component

What I’m doing is shutting boxes off with 5 seconds delay and I want to ensure the last box down is actually picked so we have indeed latest state

By: Jay Janssen

Jay Janssen — Fri, 05 Dec 2014 12:59:16 +0000

@Antonio — Are you using the latest release? It had some issues prior to that. You’d need to check the logs when it fails to recover, and also confirm that they all have a gvwstate.dat file in their datadirs before the restart.

As for usage cases: it’s enabled by default, but it should gracefully handle auto-recovery the off-chance that you have a full cluster outage. The standard use case is a power failure and then recovery — once all the previous nodes from the last PRIMARY state recover, it should auto-bootstrap itself.

By: Antonio Kang

Antonio Kang — Thu, 04 Dec 2014 18:44:24 +0000

Hi Jay,

I was testing this feature out on my VMs and I having trouble with starting mysql on all 3 of the nodes consistently.

Some times, was able to start mysql on 3 nodes after using the killall command listed in the tutorial but other times, I was not able to start up mysql on the nodes.

Also, I was wondering what scenarios do you recommend using this feature?