Comments on: Process MySQL LIMIT & ORDER BY for Performance Optimization https://www.percona.com/blog/mysql-order-by-limit-performance-optimization/ Thu, 03 Aug 2023 20:49:27 +0000 hourly 1 https://wordpress.org/?v=6.5.2 By: Narola MMA https://www.percona.com/blog/mysql-order-by-limit-performance-optimization/#comment-10970941 Tue, 02 Jul 2019 15:08:17 +0000 https://www.percona.com/blog/2006/09/01/order-by-limit-performance-optimization/#comment-10970941 Hi Peter,

Could you suggest improvement that I can do in following query. It’s taking 8 seconds to execute and I am afraid it will take more as table fill up with more data.

EXPLAIN SELECT w.Id, w.PartNumber, w.PartDescription, w.CreatedDate, w.ErpKey, w.IsActive, COALESCE(w.LookupManufacturer.Value, ”) AS
Manufacturer, COALESCE(w.LookupUom.Value, ”) AS Uom, w.IsSerialized AS Serialized, w.IsIMU AS Imu, COALESCE(w.LookupAtaChapter.Value,
”) AS AtaChapter, w.LookupGroup.Value AS Group, w.IsCharge
FROM tab.rfqpart AS w
LEFT JOIN tab.lookupgroup AS w.LookupGroup ON w.RefLookupGroupId = w.LookupGroup.Id
LEFT JOIN tab.lookupatachapter AS w.LookupAtaChapter ON w.RefLookupAtaChapterId = w.LookupAtaChapter.Id
LEFT JOIN tab.lookupuom AS w.LookupUom ON w.RefLookupUomId = w.LookupUom.Id
LEFT JOIN tab.lookupmanufacturer AS w.LookupManufacturer ON w.RefLookupManufacturerId = w.LookupManufacturer.Id
WHERE (w.RfqAccountId = ‘eccd587f-9684-42a4-bae3-5b043fbea1f5’) AND ((w.IsCharge = FALSE) AND (w.IsActive = TRUE))
ORDER BY w.PartNumber
LIMIT 20 OFFSET 10;

Possible_Keys: IX_AccountId_IsCharge_IsActive
Key: IX_AccountId_IsCharge_IsActive
Key_Length: 124
ref: cost,const,const
rows: 570677
filtered: 100
Extra: Using temporary; Using filesort

I would appreciate any help you may render.

]]>
By: sballa https://www.percona.com/blog/mysql-order-by-limit-performance-optimization/#comment-10969870 Fri, 26 Oct 2018 22:35:14 +0000 https://www.percona.com/blog/2006/09/01/order-by-limit-performance-optimization/#comment-10969870 Adding index on (col1, col2, col3) helped! Thanks.

]]>
By: sballa https://www.percona.com/blog/mysql-order-by-limit-performance-optimization/#comment-10969868 Fri, 26 Oct 2018 17:08:05 +0000 https://www.percona.com/blog/2006/09/01/order-by-limit-performance-optimization/#comment-10969868 We were facing performance issues with querying a InnoDB table in MySql 5.6 instance. This table has about 72 million rows, has only 28 columns. There are indexes on the columns used in where and order by clauses of the query.

In the following query, col1: String (6 characters length), col2: Date, col3: Time

Indexes: idx_col1col2 on (col1, col2) and idx_col2col3 on (col2, col3)

select * from force index (idx_col1col2) force index for order by (idx_col2col3) where col1= and col2 >= order by col2 desc, col3 desc limit 1000;

The above query was taking about 30 seconds to a minute to return results. As you can see we used force index hints for where as well as order by.

Could you please advise.

]]>
By: Whiskyjan https://www.percona.com/blog/mysql-order-by-limit-performance-optimization/#comment-10969363 Tue, 10 Jul 2018 10:34:41 +0000 https://www.percona.com/blog/2006/09/01/order-by-limit-performance-optimization/#comment-10969363 Thanks for this good informations!

]]>
By: madjeremy https://www.percona.com/blog/mysql-order-by-limit-performance-optimization/#comment-10967480 Wed, 21 Dec 2016 21:29:27 +0000 https://www.percona.com/blog/2006/09/01/order-by-limit-performance-optimization/#comment-10967480 What about an ORDER BY on a SUM()ed column with a LIMIT on a table that has 18 million rows?
What would you recommend on something like that?

]]>
By: Waka Waka https://www.percona.com/blog/mysql-order-by-limit-performance-optimization/#comment-10302420 Fri, 19 Dec 2014 07:25:28 +0000 https://www.percona.com/blog/2006/09/01/order-by-limit-performance-optimization/#comment-10302420 waka waka

]]>
By: Thomas https://www.percona.com/blog/mysql-order-by-limit-performance-optimization/#comment-8681023 Mon, 08 Sep 2014 21:50:00 +0000 https://www.percona.com/blog/2006/09/01/order-by-limit-performance-optimization/#comment-8681023 In the software for the directory, all categories are written the same.
1 of the categories is continuing to fail when selected, although 100% of the sub-categories and listings in the parent category are available.
I continue to get the following error in response to the attempt at accessing MySQL:

[09-07-2014 07:25:38pm] Fatal Error: MySQL error.Query:SELECT id, friendly_url, IF(date_update!='0000-00-00 00:00:00',date_update,date) AS date, priority FROM pmd_listings WHERE status='active' ORDER BY id ASC LIMIT 180000, 10000Error: (3) Error writing file ‘/var/db/mysql-tmp/MYnTfXMV’ (Errcode: 28) in /home/citydir/public_html/prevpmd/includes/class_database.php on line 132

[09-07-2014 07:27:38pm] Fatal Error: MySQL error.Query:SELECT SQL_CALC_FOUND_ROWS l.* FROM pmd_listings l INNER JOIN pmd_listings_categories lc ON l.id=lc.list_id WHERE lc.cat_id=192 AND l.status='active' ORDER BY priority DESC LIMIT 0,10Error: (126) Incorrect key file for table ‘/var/db/mysql-tmp/#sql_e66b1_0.MYI’; try to repair it in /home/citydir/public_html/prevpmd/includes/class_database.php on line 132

[09-07-2014 07:28:55pm] Fatal Error: MySQL error.Query:SELECT id, friendly_url, IF(date_update!='0000-00-00 00:00:00',date_update,date) AS date, priority FROM pmd_listings WHERE status='active' ORDER BY id ASC LIMIT 1710000, 10000Error: (3) Error writing file ‘/var/db/mysql-tmp/MYeKsGGZ’ (Errcode: 28) in /home/citydir/public_html/prevpmd/includes/class_database.php on line 132

I can’t get a response from the developer who wrote the software, so I’m left to my own devices in finding a resolve.

I would appreciate any help you may render.

]]>
By: Gaurav https://www.percona.com/blog/mysql-order-by-limit-performance-optimization/#comment-4083497 Thu, 02 Jan 2014 09:47:39 +0000 https://www.percona.com/blog/2006/09/01/order-by-limit-performance-optimization/#comment-4083497 We’ve 1 query written in our legacy system. (using MySQL 5.5)
Now, with growing data – the below mentioned query taking huge time.
In our system, we’ve somewhat 200,00,00,000 (2 billion rows) approx 650 GB of data.
Table is partitioned with respect to every day. (which means above query is fetching data from 30 partitions).

=> Allocate 16GB to innodb_buffer_pool_size.
=> Index is there on START_TIME

Query-1
SELECT * FROM ( SELECT a,b,c,d,e,f,g,h,i,j,k,l FROM TEST WHERE START_TIME between ‘2013-11-14 00:00:01’ and ‘2013-12-14 23:59:59’ ORDER BY START_TIME DESC) as TEST_DATA LIMIT 10000;

Above Query => Means selecting all the columns for all the data between 1 month and performing sorting and at last show 10000 records to end user.

Now, my doubt goes: Query-2
SELECT a,b,c,d,e,f,g,h,i,j,k,l FROM TEST WHERE START_TIME between ‘2013-11-14 00:00:01’ and ‘2013-12-14 23:59:59’ ORDER BY START_TIME DESC limit 10000;

Above Query => selecting all the columns from 1 month of data and perform sorting and display the result as soon as 10000 records sorted. (No sorting and buffering of all records).

With Query-1 and Query-2 ->
1> Does these 2 queries will display different result set? Or same?
2> Any other impact on performance?

Because in Query-1, we doing sorting on all records and then display 10k
whereas in Query-2, we doing display 10k sorted records.

Thanks a lot for your help.

]]>
By: Eric Chiang https://www.percona.com/blog/mysql-order-by-limit-performance-optimization/#comment-3298925 Wed, 06 Nov 2013 18:48:19 +0000 https://www.percona.com/blog/2006/09/01/order-by-limit-performance-optimization/#comment-3298925 Super helpful and clear explanation! I was able to use the “sort by column in leading table” and “force index” approaches to avoid a file sort. Yay!

]]>
By: Laurent https://www.percona.com/blog/mysql-order-by-limit-performance-optimization/#comment-1409949 Fri, 05 Apr 2013 14:52:11 +0000 https://www.percona.com/blog/2006/09/01/order-by-limit-performance-optimization/#comment-1409949 Hello everybody,

We tried the solution base on query see in older post :
SELECT * FROM sites WHERE category_id=5 ORDER BY date_created DESC LIMIT 10

changed to

select * FROM (SELECT * FROM sites WHERE category_id=5 ORDER BY date_created DESC) as my_table LIMIT 10

It works for us !

]]>
By: Veselin Kulov https://www.percona.com/blog/mysql-order-by-limit-performance-optimization/#comment-1088438 Thu, 01 Nov 2012 16:02:43 +0000 https://www.percona.com/blog/2006/09/01/order-by-limit-performance-optimization/#comment-1088438 I made a simple test case comparing performance of the LIMIT approach vs the JOIN workaround. You can see the results here: http://devoluk.com/sql-pagination.html

]]>
By: Cupidvogel https://www.percona.com/blog/mysql-order-by-limit-performance-optimization/#comment-1033666 Wed, 19 Sep 2012 07:48:00 +0000 https://www.percona.com/blog/2006/09/01/order-by-limit-performance-optimization/#comment-1033666 Thanks.

]]>
By: Humble Pie https://www.percona.com/blog/mysql-order-by-limit-performance-optimization/#comment-1033664 Wed, 19 Sep 2012 07:45:43 +0000 https://www.percona.com/blog/2006/09/01/order-by-limit-performance-optimization/#comment-1033664 @Cupidvogel:

You should create another index in this case for category_id. There is always tradeoff between performance and space. Space will always remain cheap compared to the performance gain.

]]>
By: Cupidvogel https://www.percona.com/blog/mysql-order-by-limit-performance-optimization/#comment-1032902 Tue, 18 Sep 2012 17:54:19 +0000 https://www.percona.com/blog/2006/09/01/order-by-limit-performance-optimization/#comment-1032902 Peter, in the 2nd example where the query was “SELECT * FROM sites WHERE category_id=5 ORDER BY date_created DESC LIMIT 10”, you suggested that using an index on (date_created,category_id) might fasten the look-up. But if you use this index, queries won’t be able to use the index on category_id because it comes 2nd in the compound index. So won’t this compensate the gain earned by using the compound index?

]]>
By: Harald Beckser https://www.percona.com/blog/mysql-order-by-limit-performance-optimization/#comment-1007799 Mon, 27 Aug 2012 14:31:31 +0000 https://www.percona.com/blog/2006/09/01/order-by-limit-performance-optimization/#comment-1007799 Good Article. Thank you for that!

]]>
By: Chris https://www.percona.com/blog/mysql-order-by-limit-performance-optimization/#comment-949607 Tue, 12 Jun 2012 22:45:12 +0000 https://www.percona.com/blog/2006/09/01/order-by-limit-performance-optimization/#comment-949607 Thank you so much for this great post! It was so much of a huge help and just exactly what I needed to know and understand right now. It is always great how you can present a topic so logically and make it look so easy and understandable even for not yet highly experienced people like me! 🙂

]]>
By: Rbk https://www.percona.com/blog/mysql-order-by-limit-performance-optimization/#comment-856194 Tue, 13 Dec 2011 11:27:29 +0000 https://www.percona.com/blog/2006/09/01/order-by-limit-performance-optimization/#comment-856194 Hi Peter,

I have one question regarding your comment

“Sort by column in leading table if you have JOIN with ORDER BY … LIMIT you should try hard to have sorting column(s) to be in the leading table. If ORDER BY is going by field from the table which is not first in the join order index can’t be used.”

I have a similar situation where I can not have sorting columns in leading table; in such case please suggest any work around.

Thanks in anticipation.
Rbk

]]>
By: james https://www.percona.com/blog/mysql-order-by-limit-performance-optimization/#comment-805582 Mon, 25 Apr 2011 09:56:21 +0000 https://www.percona.com/blog/2006/09/01/order-by-limit-performance-optimization/#comment-805582 Hello Peter,

I am going through your article its found pretty much inserting. I need one help as i am displaying my property query which is displaying properly and ordering the result by following 4 column

p.is_move_up DESC,REPLACE(IFNULL(p.is_feature,””),”2015-12-31″,””) DESC,
p.listing_order DESC,p.propid DESC

In order site, properties are been displayed in this order: Moveup paid, Paid featured, paid, trial then expired. However, we want to show within each group the properties random and make them rotate. Now,
the properties do not rotate so it is not fair for users to see always the same properties in same order. So, we need the rotation to be within same group not mixing all.

Query :-

SELECT DISTINCT(p.propid),p.category_id,p.people_sleep,p.bedrooms,p.bathrooms,p.children,
p.airport,p.features,p.pets,p.smoke, p.wheelchair,p.prop_title,p.property_type,prop_date,
display_calender,other_url_calender,is_feature
FROM property p
LEFT JOIN category ON (p.category_id=category.id)
WHERE p.active IN(“yes”,”d”) AND p.is_feature >= SYSDATE()
ORDER BY p.is_move_up DESC,REPLACE(IFNULL(p.is_feature,””),”2015-12-31″,””) DESC,
p.listing_order DESC,p.propid DESC

Can you please help me into this.

]]>
By: Dieter https://www.percona.com/blog/mysql-order-by-limit-performance-optimization/#comment-804211 Mon, 11 Apr 2011 09:29:07 +0000 https://www.percona.com/blog/2006/09/01/order-by-limit-performance-optimization/#comment-804211 Thanks for the quick response. VEry Nice

]]>
By: Peter https://www.percona.com/blog/mysql-order-by-limit-performance-optimization/#comment-799672 Tue, 22 Feb 2011 20:11:22 +0000 https://www.percona.com/blog/2006/09/01/order-by-limit-performance-optimization/#comment-799672 Thanks for this artikel!! Was what I was looking for.

]]>