Kafka sync to Doris

This article provides implementation ideas, configuration examples and verification methods for Kafka synchronization Doris scenarios, which can be used as a reference for actual project configuration.

Case

Synchronize kafka(3.1.2) to doris(2.0.13) database

solution

20250415180208

The sample sql is as follows:

CREATE TABLE from_table
(
  username string,
  age int 
) WITH (
      'connector' = 'kafka',
      'topic' = 'ispong-topic',
      'properties.bootstrap.servers' = '127.0.0.1:9092',
      'scan.startup.mode' = 'latest-offset',
      'format' = 'json',          
      'json.fail-on-missing-field' = 'false',
      'json.ignore-parse-errors' = 'true'
);

CREATE TABLE target_table
(
    username string,
    age      int
) WITH (
      'connector' = 'doris',
      'fenodes' = '127.0.0.1:9083',
      'table.identifier' = 'ispong_db.users',
      'username' = 'admin',
      'password' = '',
      'sink.enable.batch-mode' = 'true',
      'sink.buffer-flush.max-bytes' = '50MB'
  );

INSERT INTO target_table (username, age)
select username, age
from from_table;