I just wondering if we can switch from Master (not the intermediate master) into slave and vice versa ?
thanks

– orchestrator supports planned failovers, via:

orchestrator -c graceful-master-takeover -alias mycluster # Gracefully discard master and promote another (direct child) instance instead, even if everything is running well

This will make replica catch up to master, promote replica, attempt to set up old master below new master (reverse replication).

– orchestrator supports user-initiated non-graceful (panic) failovers. So a user can kick a failover even if orchestrator is confused, or doesn’t see the problem, or is configured to not failover a specific cluster, etc. For master failover:

orchestrator -c force-master-failover -alias mycluster

This promotes a replica without waiting for it to catch up ; just kick the failover process.

orchestrator/raft is now released, and provides HA without MySQL backend HA. In fact, you can run orchestrator/raft on SQLite if you wish. The raft consensus protocol ensures leadership over quorum.

@Marko unfortunately not at this stage. Bringing up VIPs is too much ties to your specific infrastructure. For changing proxy configuration I suggest consul with consul-template (and in the future I’ll provide working examples of integrating orchestrator & consul).

Past contributions included Vagrant config (https://github.com/github/orchestrator/blob/master/Vagrantfile) ; recent contributions include Docker config.

I’m happy if you can further clarify what it is you wish to achieve, and preferably you can open an issue on https://github.com/github/orchestrator/issues?

You are of course free to do anything you like under the terms of the license (Apache 2)

For DBaas services, use drag & drop is user friendly, may I try to extract your front end code to integregate with openstack?

^ what Daniel said; also, for me multi-source is not on the roadmap; I’ve discussed with Daniel & friends the implications for supporting multi-source. Some aspects would be easy wins, others would require quite the change to the codebase.

BTW another solution for you is to only put servers with log_slave_updates on 1st replication tier, and have all the result below them. This ensures you do not lose replicas because orchestrator can rearrange the promotes servers anyway it likes.

> Does the Orchestrator get the old master’s Binlog and merge slave’s Binlog.

I think this would be valuable, especially in environments where there is not the ability or will to go semi-sync

– Otherwise there’s mostly the small-ish-and-yet-blocking limitation of no INSERT...ON DUPLICATE KEY UPDATE in SQLite. At least my limited knowledge of SQLite shows this. Possibly there’s an easy way around this.

> Does the Orchestrator get the old master’s Binlog and merge slave’s Binlog.

It does not. It runs agent-free thus does not merge master binlog entries that were not fetched by replicas. My orientation is to not pursue this, as semi-sync is an available mechanism that fills this need. A nice observations is the the original author of MHA uses (a variant of) semi sync for ensuring lossless failover at Facebook. Orchestrator is not aiming in particular at lossless failover, but with 5.7 semisync you should get just that.

There is expected work on semi-sync support in orchestrator. A contributor is working on that at this time.

> At this moment, it’s required to have a MySQL backend and there is no clear/tested support for having this in high availability (HA) as well. This might change in the future.

You have a few options. To begin with, please note that the orchestrator backend database is likely to be very small, less than a 1GB in size for most installations. Binary logs are going to consume more than the data.
– Setup Galera / XtraDB as backend
– Use NDB Cluster as backend (right now orchestrator strictly created tables as InnoDB)
I have not experimented with either the above options
– I’m working to support (yet to be published & documented) active-active master-master statement based replication. At this time orchestrator is known to not cause collisions and gracefully overwrite and converge data.
Right now I’m already using active-active (writable-writable) setup under HAProxy with first load balancing algorithm. The remaining concern right now is the case of a backend database failover at the time of performing a crash recovery on another cluster, and I’m working to make that failover graceful as well. Semi-sync is likely to play a role here.
Anything failing not-while-performing-failover is good to go.

> One example of what Orchestrator can do is promote a slave if a master is down. It will choose the most up to date slave to be promoted.

It is more than that, actually. You can mark designated servers to be “candidates”, and orchestrator will prefer to promote them on best-effort basis.
To that effect, even if such a candidate server is not the most-up-to-date one, orchestrator will first promote a more fresh replica, converge your candidate instance onto that, and then promote it on top.
There is further ongoing work on further considerations such as conflicting configurations and versions. The most-up-to-date server is not always the one you actually want to promote.
Again, semi-sync can play a major role in this.

AgentAutoDiscover – this actually has to do with orchestrator-agent, which is an altogether different beast and unrelated to auto-discovering your topology. You are likely to keep this variables as false.

> If semi-synchronous replication needs to be used to avoid data loss in case of master failure, this has to be manually added to the hooks as well

There is a contributor working on that right now.

> In order to integrate this in your HA architecture or include in your fail-over processes you still need to manage many aspects manually, which can all be done by using the different hooks available in Orchestrator

Orchestrator tries to be a general-purpose tool which you should be comfortable with deploying regardless of your network/configuration/topology setup. However this pain of “the final steps” of master promotion is shared to everyone. We’re working to suggest a complete failover mechanism that would be generally applicable to many, based on orchestrator and other common open source software.

> The way we understand it right now, one active Orchestrator node will make the decision if a node is down or not. It does check a broken node’s slaves replication state to determine if Orchestrator isn’t the only one losing connectivity (in which it should just do nothing with the production servers).

Correct. Orchestrator analyses the topology as a whole and expects agreement from all servers involved to make sure it is not the only one who doesn’t see right. Read more on: http://code.openark.org/blog/mysql/what-makes-a-mysql-server-failurerecovery-case
The case where an orchestrator node is completely network partitioned is being examined thank to feedback from @gryp.

Thank you for attributing Outbrain, Booking.com and GitHub, who all deserve deep respect for supporting open source in general and the open development of this tool in particular. The people in those companies (DBAs, Sys admins, Devs) have all contributed important feedback and I would like to recognize them for that.
Lastly, the project is open to contributions and there is already a group of awesome contributors, so if you feel like contributing, the door is open!