HBase synchronization to Hive

This article provides implementation ideas, configuration examples, and verification methods for HBase synchronization with Hive scenarios, which can be used as a reference for actual project configuration.

Case

Synchronize the HBase2.1.0 version database to the hive3.1.3 database

Premise

Native flink only supports hive2.3.0 or above.

Replace the hadoop-HDFS-client-3.3.5.Jar package in the lib directory of zhiqingyun-agent with the currently dependent hadoop version For example: if the current user hadoop version is 3.0.0, you need to delete the default 3.3.5 package and download the dependency package of 3.0.0

rm ~/zhiqingyun-agent/lib/hadoop-hdfs-client-3.3.5.jar   
cd ~/zhiqingyun-agent/lib/  
wget https://zhiqingyun-saas.isxcode.com/tools/open/file/hadoop-hdfs-client-3.0.0.jar

solution

Create a Flink SQL job type and add the following dependencies Download the corresponding HBase, hive dependent versions and hadoop corresponding versions

20250325145253

The sample sql is as follows:

CREATE CATALOG my_hive_catalog WITH (
  'type' = 'hive',
  'hive-version' = '3.1.3', 
  'default-database' = 'ispong_db2',
  'hive-conf-dir' = '/data/cloudera/parcels/CDH/lib/hive/conf'
);

USE CATALOG my_hive_catalog;
DROP table hTable;

CREATE TABLE hTable (
   rowkey string,
   cf1 ROW<username STRING, age STRING>
) WITH (
  'connector' = 'hbase-2.2',
  'table-name' = 'ispong_namespace:users',
  'zookeeper.quorum' = '127.0.0.1:2181'
);

INSERT INTO users (username)
SELECT cf1.username FROM hTable;