Comments on: PostgreSQL on ARM-based AWS EC2 Instances: Is It Any Good?

By: Mark Callaghan

Mark Callaghan — Sat, 06 Feb 2021 17:17:10 +0000

In reply to @FranckPachot. I look forward to more interesting perf comparisons given the new diversity in CPU (x86 is joined by ARM, I haven't been near a POWER server since forever, although I had a PowerPC Mac long ago). It can be hard to do OSS on POWER given the lack of access to such HW. It seemed like IBM could have done more to promote that in their cloud. Fortunately we have the OSUOSL that hosts HW for OSS projects, including ARM and POWER servers. I hope that HW is used by the PG community. https://osuosl.org/services/

By: @FranckPachot

@FranckPachot — Sat, 06 Feb 2021 10:25:10 +0000

In reply to Mark Callaghan. Thank you Sergey and Mark :) I'm currently comparing x86 and POWER platforms for PostgreSQL for a customer. The result really depends on what is tested. One is faster with pgio shared_buffers hits, the other when it is filesystem cache hits. And with pgbench, one faster with the "simple" protocol, the other with prepared statements. So if we want to know which one is better for the applications, the only answer is: it depends ;) It would be very interesting to see if the Graviton2 shows general patterns in all cases.

By: Mark Callaghan

Mark Callaghan — Fri, 05 Feb 2021 16:30:07 +0000

In reply to Sergey Kuzmichev. I am also a fan of Frank's work, and will read that pgbench post soon

By: Sergey Kuzmichev

Sergey Kuzmichev — Tue, 02 Feb 2021 08:06:43 +0000

In reply to @FranckPachot. Hi Frank, a big fan of your work, and I've been following Kevin for a long time since I worked with Oracle databases. Yes, I read your blog post back when it was out, and while working on this test. It was one of the reasons I discarded pgbench for a more realistic sysbench-tpcc workload. While absolutely not perfect, it's not exactly tpc-b either. Performing a round of testing with pgio (and using EBS for storage, for that matter, to nullify storage differences) is something I haven't managed to do yet, but it's definitely on my list.

By: @FranckPachot

@FranckPachot — Sat, 30 Jan 2021 09:21:08 +0000

Hi, this is the default pgbench workload (TPC-B with simple query protocol). It is full of context switches and network calls and then do not really measure the CPU component. I tried to explain this here on an example: https://franckpachot.medium.com/do-you-know-what-you-are-measuring-with-pgbench-d8692a33e3d6 but probably.
A pgio or all plpgsql workload may give a better idea on the processor performance.

By: Krunal Bauskar

Krunal Bauskar — Wed, 27 Jan 2021 04:13:58 +0000

In reply to Mark Cotner. For higher concurrency there is active discussion on pgsql forum [1]. The said proposed patch helps improve scalability of #pgsqlonarm. [if you have tested it on platform other than graviton, kunpeng, apple-m1 can you post your result with the said patch on the pgsql thread for more reference]. [1] https://www.postgresql.org/message-id/CAB10pyamDkTFWU_BVGeEVmkc8=EhgCjr6QBk02SCdJtKpHkdFw@mail.gmail.com

By: krunal bauskar

krunal bauskar — Wed, 27 Jan 2021 04:07:22 +0000

vCPU vs Physical core is valid but then given 2 different architectures it is difficult to get machines
with exact same configuration.

x86 has hyper threading, turbo mode, viz optimization and disabling them in favor on ARM (since arm doesn’t have it)
may not be right.

—————-

I would suggest we look at it from different perspective.

Let’s keep the cost constant (and other resources like storage and memory) and let’s allow the compute power to vary.
(ARM being cheaper will get more compute resources).

This would give a fair idea that for given cost how much more TPS/USD can ARM get back.

—————–

This model is tagged as #cpm (cost performance model) and is widely being used for mysql on arm evaluation.
You can read more about it here https://mysqlonarm.github.io/CPM/

By: Jobin Augustine

Jobin Augustine — Tue, 26 Jan 2021 14:54:54 +0000

@Mark, It is an interesting finding that you see difference gains with smaller instances.
There are few reports that the code generation by different compilers are sub optimal for ARM processors at this stage. We should expect some performance gains as the compiler infrastructure improves. So more than PostgreSQL version I would expect change in performance due to compilers used, It is not just about PostgreSQL binaries but other libraries present in the OS aswell.

By: Jobin Augustine

Jobin Augustine — Tue, 26 Jan 2021 14:44:19 +0000

In reply to Ed Spiridonov. Yes, That will be interesting comparison. I am expecting some benchmarks from community as there are many more cloud vendors offering AMD based instances for better value propositions. Else we have plan to do that as well.

By: Mark Cotner

Mark Cotner — Mon, 25 Jan 2021 18:55:37 +0000

Just completed a bank of internal testing. Was happy to see you are seeing a performance improvement with ARM based instances. I’m showing the slight opposite on smaller instances(large) using Pg 12.x. Perhaps the Pg folks saw this coming and have optimizations in 13, or you see greater dividends when you go larger. Either way, performance does drop off in the higher end of concurrency and when you do something other than simple straight joins. Price performance is still a hit, however. You’d be hard pressed to find a scenario where 25% price difference is justified on Intel IMO.

By: Ed Spiridonov

Ed Spiridonov — Sun, 24 Jan 2021 08:09:32 +0000

Could you add m5ad.8xlarge or something like that to the comparsion?

By: Jobin Augustine

Jobin Augustine — Sun, 24 Jan 2021 04:30:13 +0000

In reply to Yuriy Safris. Thank you @Yuriy. Yes, This is more of a value comparison between 2 instance types : What we pay for and what we get. The point you raised is very significant for cloud instances : in m6gd.8xlarge instance, the vCPUs are in a 1 to 1 relation with physical cores. When there are more vCPUs than physical cores, the chance of "noisy neighbor" increases. As we know, some of the AWS instance types are notorious for noisy neighbor problems and unexpected CPU "steal"s in the wrong times.

By: Sergey Kuzmichev

Sergey Kuzmichev — Sat, 23 Jan 2021 19:40:00 +0000

In reply to Mark Callaghan. Yes, running with EBS would likely make this particular difference go away. However, EBS could also become a bottleneck, somewhat skewing the testing results. Cloudwatch data showed that m5d pushed more IOPS to its volumes, though, and I agree with an HN comment that having something like a comparative bonnie++ reports would make things clearer. Still, it's an interesting caveat for m6gd family that it has fewer but larger local drives across the range. Instances with local drives are a pretty exotic setup for common databases anyway, I think, though they do make for an interesting way to make IO capacity much larger. Testing with EBS and running some tests with pgio (to get another viewpoint) are definitely among the missing pieces here. The 16/32 cores point is of course also very valid, although it's difficult to get two really equal instances, unless it's something like an m4->m5 evolutionary change. As for the pricing, I think it will be interesting to see offers of the other major cloud providers when (or if) they introduce ARM-based instances.

By: Mark Callaghan

Mark Callaghan — Sat, 23 Jan 2021 18:16:46 +0000

Interesting analysis. I assume that EBS is the best way to keep storage the same. Even when using the same instance type with ephemeral storage across instance types I wonder if there is variance — are you getting a device for yourself, or sharing it with others, and possibly getting extra IO.

For the Intel has 16 cores vs ARM has 32 cores, in the end the comparison is about price/performance and if a real Intel core costs more than 2X a real ARM, then users will move to ARM. But this makes me wonder whether that price difference is a result of ARM being that much better, AWS marking up Intel cores, or AWS marking down ARM cores.

By: anaconda

anaconda — Sat, 23 Jan 2021 15:49:22 +0000

Compare against zen2 real cores, and ARM will loose.

By: chessmango

chessmango — Sat, 23 Jan 2021 13:14:05 +0000

In reply to Yuriy Safris. Not sure this will necessarily be the case, though valid point anyway. One method could be to drop one instance size down for the Graviton2 variant, and disable SMT on the other: https://aws.amazon.com/blogs/compute/disabling-intel-hyper-threading-technology-on-amazon-linux/ (guide is for Amazon's own distro, but same principle applies)

By: Aituar

Aituar — Sat, 23 Jan 2021 10:37:11 +0000

I agree with Yuriy Safris.
Also, the measurements themselves without specifying the error are meaningless.
Example.
221436 + -30000 will already intersect with the value 288867 + -30000. That is, these 2 measurements will be almost equal to each other.

By: Hareesh H

Hareesh H — Fri, 22 Jan 2021 22:26:35 +0000

Difference in cpu cache sizes of graviton and intel processors could be the reason of better performance on intel on io bound loads. I have observed similar results between amd and intel processors

By: Yuriy Safris

Yuriy Safris — Fri, 22 Jan 2021 19:51:31 +0000

One note for comparison:
m6gd.8xlarge Virtual CPUs : 32 – these are 32 physical cores
m5d.8xlarge Virtual CPUs : 32 – these are 32 virtual threads or 16 physical cores
Thus, you are comparing 32 physical cores against 16.
Considering that the competitors were selected on the basis of comparable value, the comparison can be considered quite correct.
But it should be borne in mind that with an equal number of cores, the solution with Graviton2 will be much slower.