We work on data recoveries quite often. In many cases, we recover table structures from the .frm files because there was no backup available. There is already a great blog post by my colleague Miguel Ángel Nieto about how we can recover structures from .frm files using MySQL utilities.
This works pretty well and we prefer to run mysqlfrm with the “–server” option to get all possible information from a .frm file. However, this option expects that MySQL is up and running so that mysqlfrm can spawn a new MySQL instance, and run the structure recovery there.
Recently I came across a tool that makes this job easier. The name of tool is dbsake, it’s a collection of command-line tools that perform various DBA related tasks for MySQL. In this blog, we will look at two very useful dbsake commands.
Installation is very easy and straightforward. It’s in an executable python zip archive with all dependencies included.
1 2 3 4 | # curl -s http://get.dbsake.net > dbsake # chmod u+x dbsake # ./dbsake --version dbsake, version 2.1.0 9525896 |
Recovering table structures from MySQL .frm files with dbsake
To recover table structures using dbsake, you need to use the “dbsake frmdump” command, followed by the .frm file path. The frmdump command decodes the MySQL .frm file and provides a “CREATE TABLE” or “CREATE VIEW” statement in the output. The good thing is that it doesn’t require a running a MySQL server instance, and interprets the .frm file according to rules similar to the MySQL server.
Let’s see an example:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 | # ./dbsake frmdump /var/lib/mysql/sakila/staff.frm -- -- Table structure for table `staff` -- Created with MySQL Version 5.6.27 -- CREATE TABLE `staff` ( `staff_id` tinyint(3) unsigned NOT NULL AUTO_INCREMENT, `first_name` varchar(45) NOT NULL, `last_name` varchar(45) NOT NULL, `address_id` smallint(5) unsigned NOT NULL, `picture` blob, `email` varchar(50) DEFAULT NULL, `store_id` tinyint(3) unsigned NOT NULL, `active` tinyint(1) NOT NULL DEFAULT '1', `username` varchar(16) NOT NULL, `password` varchar(40) CHARACTER SET utf8 COLLATE utf8_bin DEFAULT NULL, `last_update` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP, PRIMARY KEY (`staff_id`), KEY `idx_fk_store_id` (`store_id`), KEY `idx_fk_address_id` (`address_id`) ) ENGINE=InnoDB DEFAULT CHARSET=utf8; |
The result looks pretty good, and has recovered the character set and collation information as well. We can also see the MySQL version (5.6.27) retrieved from the .frm file. It is important to mention that the command only decodes the information available in .frm file, which means that it cannot recover InnoDB foreign-key references and AUTO_INCREMENT values. These items are stored outside of the .frm file.
The frmdump command makes the recovery process easy and faster. We can easily script this and recover the structure of a large number of tables. For example, if we need to recover the structure of all tables from a world database, we can do following:
1 2 3 4 5 6 7 8 9 10 11 12 13 | mysql> create database world_recover; # for tbl in `ls -1 /var/lib/mysql/world/*.frm`; do ./dbsake frmdump $tbl | mysql world_recover; done; mysql> show tables from world_recover; +-------------------------+ | Tables_in_world_recover | +-------------------------+ | city | | country | | countrylanguage | +-------------------------+ 3 rows in set (0.00 sec) |
Filter and transform a mysqldump stream with dbsake
It’s a very common requirement to filter one or more tables from a mysqldump full database backup. The “dbsake sieve [options]” command helps us to filter or transform mysqldump output.
Let’s see how to extract a single table from a mysqldump file.
1 2 3 | # mysqldump world > world.sql # cat world.sql | ./dbsake sieve -t world.city > world.city.sql Processed . Output: 1 database(s) 1 table(s) and 0 view(s) |
The “-t” or “–table” option tells the command to only output the table matching the given pattern. It will also show the number of databases, tables or views processed in output.
To extract multiple tables, you can pass “-t db.tbl” multiple times.
1 2 3 4 5 6 | # cat world.sql | ./dbsake sieve -t world.city -t world.country > world.city_country.sql Processed . Output: 1 database(s) 2 table(s) and 0 view(s) # cat world.city_country.sql | grep -i 'create table' CREATE TABLE `city` ( CREATE TABLE `country` ( |
The latest Percona server added the new option “–innodb-optimize-keys” in mysqldump. It changes the way InnoDB tables are dumped, so that secondary keys are created after loading the data, thus taking advantage of InnoDB fast index creation. This is a really great feature in that it helps us to restore data more efficiently than the default incremental rebuild that mysqldump performs.
Using the “dbsake sieve [options]” command, we can transform the regular mysqldump output to take advantage of fast index creation. The “–defer-indexes” option rewrites the output of CREATE TABLE statements, and arranges for secondary indexes to be created after the table data is loaded. Similarly the “–defer-foreign-keys” option can be added to add foreign key constraints after loading table data.
Let’s see an example:
1 2 | # cat world.sql | ./dbsake sieve --defer-indexes --defer-foreign-keys -t world.city > world.city.sql Processed . Output: 1 database(s) 1 table(s) and 0 view(s) |
This means that world.city.sql will have a table structure with the Primary Key first, then will insert statements to load data, and an additional ALTER TABLE statement to create secondary keys when there is at least one secondary index to be added. Foreign keys will also created with secondary indexes.
The original structure of table world.city:
1 2 3 4 5 6 7 8 9 10 | CREATE TABLE `city` ( `ID` int(11) NOT NULL AUTO_INCREMENT, `Name` char(35) NOT NULL DEFAULT '', `CountryCode` char(3) NOT NULL DEFAULT '', `District` char(20) NOT NULL DEFAULT '', `Population` int(11) NOT NULL DEFAULT '0', PRIMARY KEY (`ID`), KEY `CountryCode` (`CountryCode`), CONSTRAINT `city_ibfk_1` FOREIGN KEY (`CountryCode`) REFERENCES `country` (`Code`) ) ENGINE=InnoDB AUTO_INCREMENT=4080 DEFAULT CHARSET=latin1; |
And the transformation done by dbsake:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 | ........... CREATE TABLE `city` ( `ID` int(11) NOT NULL AUTO_INCREMENT, `Name` char(35) NOT NULL DEFAULT '', `CountryCode` char(3) NOT NULL DEFAULT '', `District` char(20) NOT NULL DEFAULT '', `Population` int(11) NOT NULL DEFAULT '0', PRIMARY KEY (`ID`) ) ENGINE=InnoDB AUTO_INCREMENT=4080 DEFAULT CHARSET=latin1; ........... LOCK TABLES `city` WRITE; ........... INSERT INTO `city` VALUES ..................... ........... UNLOCK TABLES; -- -- InnoDB Fast Index Creation (generated by dbsake) -- ALTER TABLE `city` ADD KEY `CountryCode` (`CountryCode`), ADD CONSTRAINT `city_ibfk_1` FOREIGN KEY (`CountryCode`) REFERENCES `country` (`Code`); ........... |
For more sieve command options, please read the online manual.
Conclusion
The dbsake command makes it easier to recover table structures from .frm files. We can also filter and transform the mysqldump output easily without writing a complex awk or sed script. There are some more useful features of this tool that you can read about in the manual.
Hi Alok,
Thanks for sharing the tool.
This tool is not working for me, I am getting below error.
{code}
[root@pgupta]# cat test.sql | ./dbsake sieve -t test.order > order.sql
Unable to determine section type at line 47472: — Temporary view structure for view
view_catalog_simple_visibility{code}
Thanks!
Pramod Gupta
Thanks for this brief discription about the tool.