MongoDB real-time synchronization

This article provides implementation ideas, configuration examples, and verification methods for MongoDB real-time synchronization scenarios, which can be used as a reference for actual project configuration.

Real-time synchronization of data from the result table to the target table in mongoDB

Case

Synchronize the data in cdc_source to cdc_target in real time in mongoDB

Premise

• Enable MongoDB replica set mode

solution

Create a Flink SQL job type and add the following dependencies

• flink-sql-connector-mongodb-cdc-3.2.1.jar download
• mongodb-driver-core-5.2.1.jar download
• flink-connector-mongodb-1.2.0-1.18.jar download
• mongodb-driver-sync-5.2.1.jar download

CREATE TABLE from_table(
    _id STRING,
    name STRING,
    age INT,
    city STRING,
    PRIMARY KEY (_id) NOT ENFORCED
) WITH (
    'connector'='mongodb-cdc',
    'hosts'='localhost:27017',
    'username' = 'root',
    'password' = 'root123',
    'collection'='t_source',
    'database'='isxcode_db'
);

CREATE TABLE target_table(
    _id STRING,
    name STRING,
    age INT,
    city STRING,
    PRIMARY KEY (_id) NOT ENFORCED
) WITH (
    'connector'='mongodb',
    'uri'='mongodb://root:root123@localhost:27017',
    'collection'='t_sink',
    'database'='isxcode_db'
); 

INSERT INTO target_table select * from from_table;