MySQL Shell is an advanced client tool that has many features and is easy to use. The recent MySQL release (MySQL 8.0.22) has the utility “exportTable()”, which supports exporting the data into a file from a table using MySQL shell. The exported data file can be imported using the utility “importTable()”, which was released in MySQL 8.0.17.
With “exportTable()”, you can export the data into a local server or in any S3-compliant object storage provider. In this blog, I am going to explain how those exportTable() & importTable() utilities are working in local servers and I also did some tests with MySQL 5.7 and MySQL 5.6.
Overview of exportTable() & importTable()
exportTable():
- Introduced in MySQL 8.0.22.
- The utility is used to export the data from the MySQL table into a data file.
- It can be used to export the table to a local server or any S3-compliant object storage provider.
- By default, the data will be exported in the default format for MySQL Shell’s table import utility.
- It supports CSV and TSV files.
- Compression is supported with “gzip” or “zstd”.
- JSON is not supported.
Below is the format of the query that will be used to export the data.
1 | | 43 | root | localhost | NULL | Query | 44 | Sending to client | SELECT SQL_NO_CACHE `<column1>`,`<column2>`,`<column3>`,`<column4>` ... FROM `<db name>`.`<table name>` ORDER BY `<column_pri>` /* mysqlsh exportTable, dumping table `<db name>`.`<table name>`, chunk ID: 1 */ | 39142633 | 0 | |
importTable():
- Introduced in MySQL 8.0.17.
- The utility is used to import the data into a MySQL table from a data file.
- It supports parallelism and a very fast method to import a large data file into the table.
- It is similar to LOAD DATA INFILE.
- By default, the data will be imported in the default format for the MySQL shell table export utility.
- The data file will be chunked and processed with different threads.
- We can control the number of threads for parallel execution.
- “local_infile” is the mandatory variable that should be ON on the server.
1 2 3 4 5 6 7 8 9 10 | MySQL localhost JS > \sql set global local_infile=on; Query OK, 0 rows affected (0.0001 sec) MySQL localhost JS > \sql show global variables like 'local_infile'; +---------------+-------+ | Variable_name | Value | +---------------+-------+ | local_infile | ON | +---------------+-------+ 1 row in set (0.0016 sec) |
Below is the format of the query that will be used to import the data.
1 | | 90 | root | localhost | NULL | Query | 6 | executing | LOAD DATA LOCAL INFILE '/<folder>/<filename>.<extension>' INTO TABLE `<db name>`.`<table name>` FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '\"' ESCAPED BY '\\' LINES STARTING BY '' TERMINATED BY '\n' | |
How to Export the Table Into a File Using util.exportTable()
For testing, I have installed the latest Percona Server for MySQL 8.0.22 and created the table “percona.herc7”. The size of the table is around 2.3G.
1 2 3 4 5 6 7 8 | mysql> select @@version, @@version_comment\G *************************** 1. row *************************** @@version: 8.0.22-13 @@version_comment: Percona Server (GPL), Release '13', Revision '6f7822f' 1 row in set (0.00 sec) [root@mysqlshelldump8 percona]# ls -lrth | grep -i herc7 -rw-r-----. 1 mysql mysql 2.5G Nov 18 21:25 herc7.ibd |
In this section, I am going to address three topics:
- Default output file structure using util.exportTable()
- Modified output file structure using util.exportTable()
- Compressed output using util.exportTable()
Default Output File Structure Using util.exportTable()
I am going to export the table “percona.herc7” with the default option so that the utility will create the output file with default structure.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 | MySQL localhost JS > util.exportTable("percona.herc7", "file:///root/herc7.txt") Preparing data dump for table `percona`.`herc7` Data dump for table `percona`.`herc7` will use column `id` as an index Running data dump using 1 thread. NOTE: Progress information uses estimated values and may not be accurate. Data dump for table `percona`.`herc7` will be written to 1 file 17% (8.25M rows / ~46.39M rows), 905.70K rows/s, 45.74 MB/s 29% (13.64M rows / ~46.39M rows), 924.65K rows/s, 47.23 MB/s 100% (46.54M rows / ~46.39M rows), 869.14K rows/s, 44.76 MB/s Duration: 00:00:52s Data size: 2.39 GB Rows written: 46535194 Bytes written: 2.39 GB Average throughput: 45.18 MB/s The dump can be loaded using: util.importTable("file:///root/herc7.txt", { "characterSet": "utf8mb4", "schema": "percona", "table": "herc7" }) |
The table export is completed and the output seems pretty good. During the execution, the utility prints the progress, data size, execution time, rows written, throughput, etc.
The output from the data file looks like tab-separated:
1 2 3 4 5 6 7 | [root@mysqlshelldump8 ~]# cd /root/mysqlshellexport8/ [root@mysqlshelldump8 mysqlshellexport8]# ls -lrth total 2.3G -rw-r-----. 1 root root 2.3G Nov 18 22:42 herc7.txt [root@mysqlshelldump8 mysqlshellexport8]# cat herc7.txt | tail -n2 47399358 sakthi 2020-11-18 21:21:28 Everything is MySQL 47399359 ram 2020-11-18 21:21:28 I love Python |
Query from processlist:
1 | | 43 | root | localhost | NULL | Query | 44 | Sending to client | SELECT SQL_NO_CACHE `id`,`name`,`sent_time`,`message` FROM `percona`.`herc7` ORDER BY `id` /* mysqlsh exportTable, dumping table `percona`.`herc7`, chunk ID: 1 */ | 39142633 | 0 | |
Note: Make sure to create the needed directories before executing the command. Otherwise, the command will fail.
Modified Output File Structure Using util.exportTable()
In this section, I am going to play with the options and create the output file with a comma separated CSV file. I am going to modify the following options to create the CSV file.
- fieldsOptionallyEnclosed: true,
- fieldsTerminatedBy: “,”
- linesTerminatedBy: “\n”
- fieldsEnclosedBy: ‘”‘
- defaultCharacterSet: “utf8”
- showProgress: true
- dialect: “csv”
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 | MySQL localhost JS > util.exportTable("percona.herc7", "file:///root/mysqlshellexport8/herc7.csv", {defaultCharacterSet: "utf8", showProgress: true, fieldsOptionallyEnclosed: true, fieldsTerminatedBy: ",", linesTerminatedBy: "\n", fieldsEnclosedBy: '"', defaultCharacterSet: "utf8", showProgress: true, dialect: "csv"}) Preparing data dump for table `percona`.`herc7` Data dump for table `percona`.`herc7` will use column `id` as an index Running data dump using 1 thread. NOTE: Progress information uses estimated values and may not be accurate. Data dump for table `percona`.`herc7` will be written to 1 file 93% (43.56M rows / ~46.39M rows), 901.34K rows/s, 51.83 MB/s 95% (44.11M rows / ~46.39M rows), 919.17K rows/s, 52.85 MB/s 100% (46.54M rows / ~46.39M rows), 882.24K rows/s, 50.73 MB/s Duration: 00:00:52s Data size: 2.67 GB Rows written: 46535194 Bytes written: 2.67 GB Average throughput: 50.93 MB/s The dump can be loaded using: util.importTable("file:///root/mysqlshellexport8/herc7.csv", { "characterSet": "utf8", "dialect": "csv", "fieldsEnclosedBy": "\"", "fieldsOptionallyEnclosed": true, "fieldsTerminatedBy": ",", "linesTerminatedBy": "\n", "schema": "percona", "table": "herc7" }) |
The output looks like:
1 2 3 4 5 6 7 | [root@mysqlshelldump8 mysqlshellexport8]# cd /root/mysqlshellexport8/ [root@mysqlshelldump8 mysqlshellexport8]# ls -lrth total 2.5G -rw-r-----. 1 root root 2.5G Nov 18 22:58 herc7.csv [root@mysqlshelldump8 mysqlshellexport8]# cat herc7.csv | head -n2 13,"sakthi","2020-11-18 08:08:42","Everything is MySQL" 14,"ram","2020-11-18 08:09:07","I love Python" |
Compressed Output Using util.exportTable()
exportTable() utility supports two types of compressions ( compression: “gzip” or “zstd”).
- Gzip
- Zstd
Let’s generate the output using the gzip { compression: “gzip” }:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 | MySQL localhost JS > util.exportTable("percona.herc7", "file:///root/mysqlshellexport8/herc7.gz", {compression: "gzip"}) Preparing data dump for table `percona`.`herc7` Data dump for table `percona`.`herc7` will use column `id` as an index Running data dump using 1 thread. NOTE: Progress information uses estimated values and may not be accurate. Data dump for table `percona`.`herc7` will be written to 1 file 82% (38.41M rows / ~46.39M rows), 449.34K rows/s, 23.14 MB/s uncompressed, 1.26 MB/s compressed 85% (39.85M rows / ~46.39M rows), 445.45K rows/s, 22.94 MB/s uncompressed, 1.24 MB/s compressed 100% (46.54M rows / ~46.39M rows), 442.11K rows/s, 22.77 MB/s uncompressed, 1.24 MB/s compressed Duration: 00:01:48s Uncompressed data size: 2.39 GB Compressed data size: 130.39 MB Compression ratio: 18.3 Rows written: 46535194 Bytes written: 130.39 MB Average uncompressed throughput: 22.01 MB/s Average compressed throughput: 1.20 MB/s The dump can be loaded using: util.importTable("file:///root/mysqlshellexport8/herc7.gz", { "characterSet": "utf8mb4", "schema": "percona", "table": "herc7" }) |
Note: The util.importTable() function currently does not support compressed files, the output file needs to be unpacked first.
The output file size is 125M.
1 2 | [root@mysqlshelldump8 mysqlshellexport8]# ls -lrth | grep herc7.gz -rw-r-----. 1 root root 125M Nov 18 23:09 herc7.gz |
At the end of the export command, it prints the below line. It says that we can’t directly import the compressed file using the tableImport() utility, so we need to decompress the file first and then we need to import the file.
How to Import the Data File using util.importTable()
I am going to show a single example using the CSV file, which was already generated from “Modified output using util.exportTable()”.
1 2 3 4 5 6 | root@mysqlshelldump8 mysqlshellexport8]# ls -lrth herc7.csv -rw-r-----. 1 root root 2.5G Nov 18 22:58 herc7.csv [root@mysqlshelldump8 mysqlshellexport8]# cat herc7.csv | head -n2 13,"sakthi","2020-11-18 08:08:42","Everything is MySQL" 14,"ram","2020-11-18 08:09:07","I love Python" |
Before starting the import, I have created an empty table to load the CSV file using the ImportTable utility.
1 2 | MySQL localhost:33060+ ssl JS > \sql create table percona.herc7load like percona.herc7; Query OK, 0 rows affected (0.1410 sec) |
I am going to use the following options to import the data file. Here I would suggest using the same options which were used to export the data.
- schema: “percona”
- table: “herc7load”
- dialect: “csv”
- skipRows: 0
- showProgress: true
- fieldsOptionallyEnclosed: true
- fieldsTerminatedBy: “,”
- linesTerminatedBy: “\n”
- fieldsEnclosedBy: ‘”‘
- threads: 3
- bytesPerChunk: “100M”
1 2 3 4 5 6 7 8 9 10 11 12 13 | MySQL localhost JS > util.importTable("/root/mysqlshellexport8/herc7.csv", {schema: "percona", table: "herc7load", dialect: "csv", skipRows: 0, showProgress: true, fieldsOptionallyEnclosed: true, fieldsTerminatedBy: ",", linesTerminatedBy: "\n",fieldsEnclosedBy: '"',threads: 3, bytesPerChunk: "100M"}) Importing from file '/root/mysqlshellexport8/herc7.csv' to table `percona`.`herc7load` in MySQL Server at /var%2Flib%2Fmysql%2Fmysql.sock using 3 threads [Worker002] herc7.csv: Records: 1769912 Deleted: 0 Skipped: 0 Warnings: 0 [Worker000] herc7.csv: Records: 1769912 Deleted: 0 Skipped: 0 Warnings: 0 ..... ..... [Worker001] herc7.csv: Records: 1135493 Deleted: 0 Skipped: 0 Warnings: 0 [Worker002] herc7.csv: Records: 1739131 Deleted: 0 Skipped: 0 Warnings: 0 [Worker000] herc7.csv: Records: 1739131 Deleted: 0 Skipped: 0 Warnings: 0 100% (2.67 GB / 2.67 GB), 6.73 MB/s File '/root/mysqlshellexport8/herc7.csv' (2.67 GB) was imported in 6 min 7.9999 sec at 7.24 MB/s Total rows affected in percona.herc7load: Records: 46535194 Deleted: 0 Skipped: 0 Warnings: 0 |
Below is the output from the processlist during the process. You can see three LOAD DATA INFILE commands running parallel as I mentioned “thread: 3”. You can define the threads based on your available CPU.
1 2 3 4 5 | | 90 | root | localhost | NULL | Query | 6 | executing | LOAD DATA LOCAL INFILE '/root/mysqlshellexport8/herc7.csv' INTO TABLE `percona`.`herc7load` FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '\"' ESCAPED BY '\\' LINES STARTING BY '' TERMINATED BY '\n' | | 91 | root | localhost | NULL | Query | 6 | executing | LOAD DATA LOCAL INFILE '/root/mysqlshellexport8/herc7.csv' INTO TABLE `percona`.`herc7load` FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '\"' ESCAPED BY '\\' LINES STARTING BY '' TERMINATED BY '\n' | | 92 | root | localhost | NULL | Query | 6 | executing | LOAD DATA LOCAL INFILE '/root/mysqlshellexport8/herc7.csv' INTO TABLE `percona`.`herc7load` FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '\"' ESCAPED BY '\\' LINES STARTING BY '' TERMINATED BY '\n' | |
Will exportTable() and importTable() Work With MYSQL 5.7?
Yes, it is working when I tested with a local server.
1 2 3 4 5 6 7 8 | MySQL localhost percona JS > \sql select @@version, @@version_comment\G *************************** 1. row *************************** @@version: 5.7.31-34 @@version_comment: Percona Server (GPL), Release 34, Revision 2e68637 1 row in set (0.0002 sec) [root@mysqlshelldump57 mysqlshellexport8]# mysqlsh --version mysqlsh Ver 8.0.22 for Linux on x86_64 - for MySQL 8.0.22 (MySQL Community Server (GPL)) |
exportTable():
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 | MySQL localhost percona JS > util.exportTable("percona.herc57", "file:///root/mysqlshellexport7/herc57.txt", {defaultCharacterSet: "utf8"}) Preparing data dump for table `percona`.`herc57` Data dump for table `percona`.`herc57` will use column `id` as an index Running data dump using 1 thread. NOTE: Progress information uses estimated values and may not be accurate. Data dump for table `percona`.`herc57` will be written to 1 file 99% (100.00K rows / ~100.09K rows), 0.00 rows/s, 0.00 B/s Duration: 00:00:00s Data size: 4.84 MB Rows written: 100000 Bytes written: 4.84 MB Average throughput: 4.84 MB/s The dump can be loaded using: util.importTable("file:///root/mysqlshellexport7/herc57.txt", { "characterSet": "utf8", "schema": "percona", "table": "herc57" }) |
importTable()
1 2 3 4 5 6 7 8 9 10 11 12 | MySQL localhost percona JS > util.importTable("/root/mysqlshellexport7/herc57.txt", {schema: "percona", table: "herc57load"}) Importing from file '/root/mysqlshellexport7/herc57.txt' to table `percona`.`herc57load` in MySQL Server at /var%2Flib%2Fmysql%2Fmysql.sock using 1 thread [Worker000] herc57.txt: Records: 100000 Deleted: 0 Skipped: 0 Warnings: 0 100% (4.84 MB / 4.84 MB), 0.00 B/s File '/root/mysqlshellexport7/herc57.txt' (4.84 MB) was imported in 0.4046 sec at 4.84 MB/s Total rows affected in percona.herc57load: Records: 100000 Deleted: 0 Skipped: 0 Warnings: 0 MySQL localhost percona JS > \sql select count(*) from percona.herc57load\G *************************** 1. row *************************** count(*): 100000 1 row in set (0.0382 sec) |
Will exportTable() & importTable() Work With MySQL 5.6?
Yes, it is working when I tested with a local server.
1 2 3 4 5 6 7 8 | MySQL localhost percona JS > \sql select @@version, @@version_comment\G *************************** 1. row *************************** @@version: 5.6.50-90.0 @@version_comment: Percona Server (GPL), Release 90.0, Revision 9dd29a1 1 row in set (0.0002 sec) [root@mysqlshelldump56 percona]# mysqlsh --version mysqlsh Ver 8.0.22 for Linux on x86_64 - for MySQL 8.0.22 (MySQL Community Server (GPL)) |
exportTable():
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 | MySQL localhost percona JS > util.exportTable("percona.herc56", "file:///root/mysqlshellexport56/herc56.txt", {defaultCharacterSet: "utf8"}) Preparing data dump for table `percona`.`herc56` Data dump for table `percona`.`herc56` will use column `id` as an index Running data dump using 1 thread. NOTE: Progress information uses estimated values and may not be accurate. Data dump for table `percona`.`herc56` will be written to 1 file 99% (100.00K rows / ~100.09K rows), 0.00 rows/s, 0.00 B/s Duration: 00:00:00s Data size: 4.84 MB Rows written: 100000 Bytes written: 4.84 MB Average throughput: 4.84 MB/s The dump can be loaded using: util.importTable("file:///root/mysqlshellexport56/herc56.txt", { "characterSet": "utf8", "schema": "percona", "table": "herc56" }) |
importTable()
1 2 3 4 5 6 7 8 9 10 11 12 | MySQL localhost percona JS > util.importTable("/root/mysqlshellexport56/herc56.txt", {schema: "percona", table: "herc56load"}) Importing from file '/root/mysqlshellexport56/herc56.txt' to table `percona`.`herc56load` in MySQL Server at /var%2Flib%2Fmysql%2Fmysql.sock using 1 thread [Worker000] herc56.txt: Records: 100000 Deleted: 0 Skipped: 0 Warnings: 0 100% (4.84 MB / 4.84 MB), 0.00 B/s File '/root/mysqlshellexport56/herc56.txt' (4.84 MB) was imported in 0.3055 sec at 4.84 MB/s Total rows affected in percona.herc56load: Records: 100000 Deleted: 0 Skipped: 0 Warnings: 0 MySQL localhost percona JS > \sql select count(*) from percona.herc56load\G *************************** 1. row *************************** count(*): 100000 1 row in set (0.0284 sec) |
It is good to see that MySQL shell is supported to export and import the data from the table, and you can load the large tables very fast using the importTable utility. In my next blog, I am planning to write about how these utilities support S3.
One advantage of this is, you can check the time to completion of the dump (at least approximately) and the parallel processing.
Good one.
Interesting article!
You mention that exportTable() “can be used to export the table to a local server or any S3-compliant object storage provider”
Do you have an example how to configure this to export into S3 directly? From the docs seems only OCI is supported? thx very much!
can you share the syntax , How to use the export utility table from command line