Comments on: Foreign Data Wrappers in PostgreSQL and a closer look at postgres_fdw https://www.percona.com/blog/foreign-data-wrappers-postgresql-postgres_fdw/ Fri, 22 Dec 2023 00:37:47 +0000 hourly 1 https://wordpress.org/?v=6.5.2 By: Florian https://www.percona.com/blog/foreign-data-wrappers-postgresql-postgres_fdw/#comment-10972264 Tue, 26 May 2020 08:41:35 +0000 https://www.percona.com/blog/?p=51967#comment-10972264 Hello Jobin,

thank you for this great article of yours! I already learned a lot from it but have a question.

My database I am trying to connect to has a SSL certificate. What do I have to chnage in the code to connect to the Server? Is it Step 8 I have to alter and if yes, what do I need to do?

Thank you for any help!

Hav a great week,
Florian

]]>
By: Rohit Tawde https://www.percona.com/blog/foreign-data-wrappers-postgresql-postgres_fdw/#comment-10972239 Mon, 18 May 2020 11:56:14 +0000 https://www.percona.com/blog/?p=51967#comment-10972239 Great article Jobin,

I am getting below error while importing foreign schema.

SQLSTATE[08001]: could not translate host name to address: Name or service not known.

I already tested below things :
– I can able to connect to both local and foreign db with psql command
– Can able to connect to my DB instance using ‘nc -v hosturl’

What else could be possible reasons ?

]]>
By: Rajni Baliyan https://www.percona.com/blog/foreign-data-wrappers-postgresql-postgres_fdw/#comment-10971742 Mon, 27 Jan 2020 09:28:06 +0000 https://www.percona.com/blog/?p=51967#comment-10971742 After inserting record in Oracle , execute commit command and select data in PG.
SQL> insert ;
SQL> commit;

]]>
By: Jobin Augustine https://www.percona.com/blog/foreign-data-wrappers-postgresql-postgres_fdw/#comment-10971556 Thu, 19 Dec 2019 08:39:12 +0000 https://www.percona.com/blog/?p=51967#comment-10971556 Hi Muhammad,
Every record in the remote database will be available though FDW. oracle_fdw is widely used for huge migrations.
if you have any reproducible problem like you mentioned, appreciate filing a bug with oracle_fdw project : https://github.com/laurenz/oracle_fdw/issues

]]>
By: Muhammad ali https://www.percona.com/blog/foreign-data-wrappers-postgresql-postgres_fdw/#comment-10971549 Wed, 18 Dec 2019 18:33:50 +0000 https://www.percona.com/blog/?p=51967#comment-10971549 Hello brother.
Can you guide me about oracle_fdw?
let consider i fetch a table “abc” which has 50 rows from oracle into Postgresql using oracle_fdw.
Now i insert into foreign table, it works smoothly and data is synced with oracle.
But when i insert a row in “abc” table from oracle, let say now there are 51 rows in oracle “abc” table,
It doesnt reflects in same postgresql foreign table i.e: the postgres foreign table still shows 50 rows.

Please correct me If I’m wrong or tell me if the above statements are true.
Thanks.

]]>
By: Jobin Augustine https://www.percona.com/blog/foreign-data-wrappers-postgresql-postgres_fdw/#comment-10971264 Mon, 07 Oct 2019 18:42:53 +0000 https://www.percona.com/blog/?p=51967#comment-10971264 Hi Manoj,
FDW is different and dblink is different. Hope you are discussing about FDW.
use regular psql utility in the sever machine and try to connect to foreign database with using the same credentials as you specified in the FDW. if that works, database sever on the same server should be able to connect.

]]>
By: Manoj Jain https://www.percona.com/blog/foreign-data-wrappers-postgresql-postgres_fdw/#comment-10971258 Sun, 06 Oct 2019 11:22:08 +0000 https://www.percona.com/blog/?p=51967#comment-10971258 Thanks Jobin for such a nice article.

I am facing issue while creating DB link between two databases dev_abc_db (source db) and dev_xyz_db (destination db) present on the same machine. And the owner of both database is same user (dev_user).
Here is what I am doing.

1. CREATE SERVER IF NOT EXISTS resdb FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host ‘dev.abc.com’, dbname ‘dev_abc_db’);
2. CREATE USER MAPPING FOR dev_user SERVER resdb OPTIONS (user ‘dev_user’, password ‘abc’);
3. GRANT USAGE ON FOREIGN SERVER resdb TO dev_user;

I have created a view in my destination database which uses resdb (created in point 1) to fetch the data from source database. While executing view I am getting below error.

SQL Error [2F003]: ERROR: password is required
Detail: Non-superuser cannot connect if the server does not request a password.
Hint: Target server’s authentication method must be changed.

Is it must to have super user role to use DB link ?

]]>
By: inteligencia360 https://www.percona.com/blog/foreign-data-wrappers-postgresql-postgres_fdw/#comment-10971204 Tue, 17 Sep 2019 02:04:13 +0000 https://www.percona.com/blog/?p=51967#comment-10971204 Great, Perfect, Thank so much

]]>
By: Jobin Augustine https://www.percona.com/blog/foreign-data-wrappers-postgresql-postgres_fdw/#comment-10971130 Tue, 20 Aug 2019 09:55:12 +0000 https://www.percona.com/blog/?p=51967#comment-10971130 Both are independent databases. So databases won’t do it automatically. It is Architects and DBAs responsibility to ensure that all database schema are in sync in the appropriate change window. You may use external scripting to assist you.

]]>
By: Jobin Augustine https://www.percona.com/blog/foreign-data-wrappers-postgresql-postgres_fdw/#comment-10971129 Tue, 20 Aug 2019 09:48:04 +0000 https://www.percona.com/blog/?p=51967#comment-10971129 If I understand correctly, A query with Union can be best suitable for you.

]]>
By: Naushad Ahmad https://www.percona.com/blog/foreign-data-wrappers-postgresql-postgres_fdw/#comment-10971124 Mon, 19 Aug 2019 12:38:27 +0000 https://www.percona.com/blog/?p=51967#comment-10971124 Hi Sir
Great article !!!

But I am stuck at a point when the app database changes in source then how is it going to be updated in the destination schema. For example: if we add a table, or drop a column—our foreign tables will need updating too. And in case it is happening frequently then how can I trigger the change in my destination as well. Kindly explain how can I achieve this.

]]>
By: Scott https://www.percona.com/blog/foreign-data-wrappers-postgresql-postgres_fdw/#comment-10970927 Wed, 26 Jun 2019 16:11:20 +0000 https://www.percona.com/blog/?p=51967#comment-10970927 Jobin, great article. I currently have a postgres RDS database in AWS. I have imported schemas and foreign tables from AWS Redshift. The RDS instance contains daily transnational data the is moved to Redshift monthly, but the reporting needs require the data from RDS and the data from Redshift to be merged. The schemas in RDS match the foreign schema from Redshift. How would you suggest queries be handled such that all joins both locally and foreign are pushed down to their respective database engine and the results then merged and returned? The number of joins and filters are completely dynamic in nature.

]]>
By: Jobin Augustine https://www.percona.com/blog/foreign-data-wrappers-postgresql-postgres_fdw/#comment-10970636 Tue, 16 Apr 2019 09:00:55 +0000 https://www.percona.com/blog/?p=51967#comment-10970636 Since everything need to go though the fdw interface, A slight performance hit related to CPU is expected. However I don’t have numbers now. Thank you for giving me the need to have a benchmark.

]]>
By: Nam Le https://www.percona.com/blog/foreign-data-wrappers-postgresql-postgres_fdw/#comment-10970331 Thu, 21 Feb 2019 04:48:22 +0000 https://www.percona.com/blog/?p=51967#comment-10970331 Thank you for your greate article. I have a question:
Does the query performance decrease when we map one or more tables from one database to many other databases?
Ex:
Database A has some table:
– Employee
– Department
– Organization
– …
And I want to mapping some tables (Emp, Dept, Org) to database B, C, D, and more.
Is it decrease query performance? or have any trouble?

]]>
By: Vijay https://www.percona.com/blog/foreign-data-wrappers-postgresql-postgres_fdw/#comment-10970327 Wed, 20 Feb 2019 07:16:38 +0000 https://www.percona.com/blog/?p=51967#comment-10970327 FYI, the pg11 setup too does not parallelize the union all queries via FDW.
postgres_fdw operates through declared cursors, and declared cursors inhibit parallel query. This doesn’t change in v11, see https://www.postgresql.org/docs/11/when-can-parallel-query-be-used.html

https://postgrespro.com/list/id/CAE7uO5hVKdUGJHsBbV8KKXfEWJQ4SCynRMbgG3vgwjQh+HTz7A@mail.gmail.com

]]>
By: Vijay https://www.percona.com/blog/foreign-data-wrappers-postgresql-postgres_fdw/#comment-10970075 Wed, 19 Dec 2018 11:57:13 +0000 https://www.percona.com/blog/?p=51967#comment-10970075 Thanks Jobin.
We do have plans to move to pg11.

one more query though.

coz now we are in a distributed setup, there may be cases of servers having issues and we run a pg_terminate_backend(pid) on the remote server or on the mgmt server (from the above setup)

now pg_terminate_backend is (like kill -9?) so if we kill a query like FETCH from c1 on the shard or
pg_terminate_backend(pid) on the mgmt server (from where we make the remote query) it may forget about the query on the remote server, but remote server query is already triggered and will keep running until it returns the rows and then figures out the tcp connection is broken ?
i mean are we at the risk of creating sql orphans ? coz the parent is dead and child keep waiting for parent
if i am wrong about this 🙂 then does a pg_cancel_backend(pid) cascade to foriegn server and kill all the remote queries before killing itself?

i mean
A -> FDW -> B

kill on A (will it kill on B too) ?
kill on B( will it signal A that B is dead and you should abort)

]]>
By: Jobin Augustine https://www.percona.com/blog/foreign-data-wrappers-postgresql-postgres_fdw/#comment-10970067 Mon, 17 Dec 2018 03:36:34 +0000 https://www.percona.com/blog/?p=51967#comment-10970067 Hi Raja,
FDW acts as a client for a remote server. So the performance mainly depends on how fast remote server can answer the queries from the PostgreSQL side. So the benchmarking of FDW layer alone may not be relevant because the overhead in that layer is very minimal.

]]>
By: Jobin Augustine https://www.percona.com/blog/foreign-data-wrappers-postgresql-postgres_fdw/#comment-10970066 Mon, 17 Dec 2018 03:27:00 +0000 https://www.percona.com/blog/?p=51967#comment-10970066 Hi Vijay,
Great to see your experiment in setting up a shared cluster using FDW.
Parallel scanning of all shards was a known limitation till PostgreSQL 10 and it is addressed in PostgreSQL 11. You may try the same with PostgreSQL 11.

]]>
By: Vijay https://www.percona.com/blog/foreign-data-wrappers-postgresql-postgres_fdw/#comment-10970063 Sat, 15 Dec 2018 19:10:46 +0000 https://www.percona.com/blog/?p=51967#comment-10970063 Hey Jobin,

Awesome writeup.
I use a setup to simulate sharded architecture as following
server a
server b
server c
server d

each shard has FDW mapping talking to each other server.
i.e
server a -> (via fdw) -> can query (server b, server c, server d)
and so on

this worked great for our usecase where we had data in one shard server a has no direct relation to server b and so on.
now for queries we has a view which does a query across all the shards
create view from_all_shards as
select * from remote_server_a.tablea
union all
select * from remote_server_b.tablea
union all
select * from remote_server_c.tablea
union all
select * from remote_server_d.tablea

so even though the queries seem to be making isolated query to shards, i am not able to have parallel queries (even via parallel setup in pg10)
i mean if all query take 10s, if takes 40 s to return.

in simple table setups parallel workers come into play, but in FDW i did not see the case?
Am i doing things correctly?

]]>
By: Raja Kt https://www.percona.com/blog/foreign-data-wrappers-postgresql-postgres_fdw/#comment-10969805 Fri, 12 Oct 2018 06:58:13 +0000 https://www.percona.com/blog/?p=51967#comment-10969805 Any benchmark with fdw implementation with Mariadb ( Spider+innodb) or (Spider+MyRocks)?

]]>