Documentation

Setting up CDC for Postgres on RDS

To set up Change Data Capture (CDC) on Postgres you need to enable/configure Logical Replication. Before you start, please keep in mind:

  • Supported Postgres versions: 9.4.9+ and 9.5.4+

  • You can only set up CDC for Postgres on the RDS master instance.

  • The process requires a master instance reboot.

  • CDC for Postgres requires additional log storage. Though unlikely, failed replication due to a lost connection can cause logs to remain on the server. If logs do accumulate on the server, drop and recreate the replication slot. Notify Alooma if this happens so that we can recover any possible gaps in the data.

  • In order to replicate DELETE operations, the REPLICA IDENTITY parameter of each table being replicated must be set to DEFAULT, FULL, or BY INDEX (or anything but NOTHING).

Note

You cannot have multiple Postgres CDC inputs referencing the same replication slot.

You can set up CDC for Postgres on RDS in just a few steps:

Step 1- Create and configure a Parameter Group
  1. Go to your Amazon RDS Dashboard. If you already have a Parameter Group, skip to step 4.

  2. In the sidebar, click Parameter Groups and then Create parameter group. Select your Postgres version (postgres9.6 in this example) and provide a name and description:

    psql_screenshot_0.png
  3. Click Create.

  4. In the Parameter Groups, for the Parameter Group you created, find "logical_replication" and click Edit Parameters.

  5. Check the box at the bottom and change the value for rds.logical_replication to 1.

    psql_screenshot_1.png
  6. Click Save changes.

Step 2 - Modify the Database instance
  1. Go to your Amazon RDS dashboard.

  2. From DB instances, choose your instance.

  3. Click on the Instance Actions drop-down and choose Modify. The Modify DB instance dialog will appear.

  4. Scroll to Database Options and change DB Parameter group to the group you created.

  5. Scroll to the bottom and click Continue to apply your changes.

  6. In the Scheduling of Modifications screen that appears, select Apply immediately.

    psql_screenshot_2.png
  7. Click Modify DB Instance.

Note

This will require you to reboot the instance.

Step 3 - Reboot the instance

From the main RDS page:

  1. In the navigation pane, choose Instances, and then select the your Instance.

  2. From the Instance Actions menu, choose Reboot.

Step 4 - Create a dedicated Logical Replication slot

Note

You must create a dedicated replication slot for Alooma. If any other processes use this replication slot, you risk losing data.

Perform the following steps in your master database.

  1. Log into postgresSql as a superuser and, using the test_decoding plugin, create a logical replication slot named: alooma_replication_slot

    Copy
    SELECT pg_create_logical_replication_slot('alooma_replication_slot', 'test_decoding');
  2. Grant the necessary replication permission to your user to enable reading the logical replication slot:

    Copy
    GRANT rds_replication TO <user>;
  3. Grant SELECT permission on all tables in the schema to your user:

    Copy
    GRANT SELECT ON ALL TABLES IN SCHEMA <schema> TO <user>;
  4. Log in as the user and verify you can read from the logical replication slot:

    Copy
    SELECT count(*) FROM pg_logical_slot_peek_changes('alooma_replication_slot', null, null);

And that's it! You're all set to finish connecting to Postgres.

Search results

    No results found