至轻云

Plug-in debugging

This article describes the development, debugging, or maintenance requirements related to plug-in debugging and is applicable to project development and delivery personnel.

Debugging custom plugins locally

Prerequisite: Create a Standalone type computing cluster Shut down the computing cluster node service through the console. Do not use the interface to recheck the status. Otherwise, submitting a job will verify that the computing cluster is unavailable.

cd ~/zhiqingyun-agent/bin
bash stop.sh

Start the computing cluster node code through idea

20241206175513

Specify the break point at the code and get the value of args

spark-yun/agent/src/main/java/com/isxcode/star/agent/run/impl/StandaloneAgentService.java

20241212183949

Through the interface, click the job run button to trigger job submission

Take the debugging data synchronization plug-in as an example

spark-yun/plugins/spark-data-sync-JDBC-plugin/src/main/java/com/isxcode/star/plugin/dataSync/JDBC/Execute.java

Change the args variable in the plug-in to the args value obtained at the code breakpoint in advance

Or use arthas to get args

curl -O https://arthas.aliyun.com/arthas-boot.jar
java -jar arthas-boot.jar
watch org.apache.spark.launcher.SparkLauncher addAppArgs '{params,returnObj,throwExp}'  -n 5  -x 3

[arthas@4480]$ watch org.apache.spark.launcher.SparkLauncher addAppArgs '{params,returnObj,throwExp}'  -n 5  -x 3 
Press Q or Ctrl+C to abort.
Affect(class count: 1 , method count: 2) cost in 150 ms, listenerId: 1
method=org.apache.spark.launcher.SparkLauncher.addAppArgs location=AtExit
ts=2025-11-13 21:07:30.084; [cost=1.232365ms] result=@ArrayList[
    @Object[][
        @String[][
            @String[eyJjb250YWluZXJQb3J0IjowLCJmdW5jSW5mb0xpc3QiOltdLCJzcGFya0NvbmZpZyI6eyJzcGFyay5leGVjdXRvci5tZW1vcnkiOiIyZyIsInNwYXJrLnNxbC5wYXJxdWV0LndyaXRlTGVnYWN5Rm9ybWF0IjoidHJ1ZSIsInNwYXJrLmRyaXZlci5tZW1vcnkiOiIxZyIsInNwYXJrLnNxbC5wYXJxdWV0LmRhdGV0aW1lUmViYXNlTW9kZUluV3JpdGUiOiJMRUdBQ1kiLCJzcGFyay5kcml2ZXIuY29yZXMiOiIxIiwic3Bhcmsuc3FsLnN0b3JlQXNzaWdubWVudFBvbGljeSI6IkxFR0FDWSIsInNwYXJrLmV4ZWN1dG9yLmNvcmVzIjoiMSIsImhpdmUubWV0YXN0b3JlLnVyaXMiOiJ0aHJpZnQ6Ly8xMDYuMTUuNzcuNDk6OTA4MyIsInNwYXJrLnNxbC5wYXJxdWV0LmludDk2UmViYXNlTW9kZUluUmVhZCI6IkxFR0FDWSIsInNwYXJrLnNxbC5wYXJxdWV0LmRhdGV0aW1lUmViYXNlTW9kZUluUmVhZCI6IkxFR0FDWSIsInNwYXJrLnNxbC5sZWdhY3kudGltZVBhcnNlclBvbGljeSI6IkxFR0FDWSIsInNwYXJrLnNxbC5wYXJxdWV0LmludDk2UmViYXNlTW9kZUluV3JpdGUiOiJMRUdBQ1kiLCJzcGFyay5zcWwuYXV0b0Jyb2FkY2FzdEpvaW5UaHJlc2hvbGQiOiItMSIsInNwYXJrLmV4ZWN1dG9yLmluc3RhbmNlcyI6IjEiLCJzcGFyay5leGVjdXRvci5leHRyYUphdmFPcHRpb25zIjoiLURmaWxlLmVuY29kaW5nPXV0Zi04Iiwic3BhcmsuY29yZXMubWF4IjoiMSIsInNwYXJrLmRyaXZlci5leHRyYUphdmFPcHRpb25zIjoiLURmaWxlLmVuY29kaW5nPXV0Zi04Iiwic3Bhcmsuc3FsLnBhcnF1ZXQuZW5hYmxlVmVjdG9yaXplZFJlYWRlciI6ImZhbHNlIn0sInN5bmNSdWxlIjp7Im51bUNvbmN1cnJlbmN5IjoxLCJudW1QYXJ0aXRpb25zIjoxLCJzZXRNb2RlIjoiU0lNUExFIiwic3FsQ29uZmlnSnNvbiI6Im51bGwifSwic3luY1dvcmtDb25maWciOnsiY29sdW1uTWFwIjpbeyJzb3VyY2UiOiJ1c2VuYW1lIiwidGFyZ2V0IjoidXNlcm5hbWUifV0sIm92ZXJNb2RlIjoiT1ZFUldSSVRFIiwicGFydGl0aW9uQ29sdW1uIjoidXNlbmFtZSIsInF1ZXJ5Q29uZGl0aW9uIjoidXNlbmFtZSA9ICcxJyIsInNvdXJjZURCSWQiOiJzeV8xOTg4OTAwNjgyOTI2MTk0Njg4Iiwic291cmNlREJUeXBlIjoiSElWRSIsInNvdXJjZURhdGFiYXNlIjp7ImRiVGFibGUiOiJ1c2VyczEiLCJkcml2ZXIiOiJvcmcuYXBhY2hlLmhpdmUuamRiYy5IaXZlRHJpdmVyIiwicGFzc3dvcmQiOiIiLCJ1cmwiOiJqZGJjOmhpdmUyOi8vMTA2LjE1Ljc3LjQ5OjEwMDAwL2RlZmF1bHQiLCJ1c2VyIjoicm9vdCJ9LCJzb3VyY2VUYWJsZSI6InVzZXJzMSIsInNvdXJjZVRhYmxlQ29sdW1uIjpbeyJjb2RlIjoidXNlbmFtZSIsInNxbCI6IiIsInR5cGUiOiJTVFJJTkcifV0sInRhcmdldERCSWQiOiJzeV8xOTg4OTAwNTE0NTU4NDQzNTIwIiwidGFyZ2V0REJUeXBlIjoiTVlTUUwiLCJ0YXJnZXREYXRhYmFzZSI6eyJkYlRhYmxlIjoidGVzdF9tb2RlbCIsImRyaXZlciI6ImNvbS5teXNxbC5jai5qZGJjLkRyaXZlciIsInBhc3N3b3JkIjoiaXNwb25nMTIzIiwidXJsIjoiamRiYzpteXNxbDovLzQ3LjkyLjE2OC4xMTY6MzAxMDIvaXNwb25nX2RiIiwidXNlciI6ImlzcG9uZyJ9LCJ0dHlwZSI6IlZBUkNIQVIoMjAwKSJ9XSwid29ya0lkIjoic3lfMTk4ODkwNDgwNjU3NzE0MzgwOCJ9fQ==],
        ],

public static void main(String[] args) {
        
    // args = new String[]{"eyJjb250YWluZXJQb3J0IjowLCJmdW5jSW5mb0xpc3QiOltdLCJzcGFya0NvbmZpZyI6eyJzcGFyay5leGVjdXRvci5tZW1vcnkiOiIyZyIsInNwYXJrLnNxbC5wYXJxdWV0LndyaXRlTGVnYWN5Rm9ybWF0IjoidHJ1ZSIsInNwYXJrLmRyaXZlci5tZW1vcnkiOiIxZyIsInNwYXJrLnNxbC5wYXJxdWV0LmRhdGV0aW1lUmViYXNlTW9kZUluV3JpdGUiOiJMRUdBQ1kiLCJzcGFyay5kcml2ZXIuY29yZXMiOiIxIiwic3Bhcmsuc3FsLnN0b3JlQXNzaWdubWVudFBvbGljeSI6IkxFR0FDWSIsInNwYXJrLmV4ZWN1dG9yLmNvcmVzIjoiMSIsImhpdmUubWV0YXN0b3JlLnVyaXMiOiJ0aHJpZnQ6Ly8xMDYuMTUuNzcuNDk6OTA4MyIsInNwYXJrLnNxbC5wYXJxdWV0LmludDk2UmViYXNlTW9kZUluUmVhZCI6IkxFR0FDWSIsInNwYXJrLnNxbC5wYXJxdWV0LmRhdGV0aW1lUmViYXNlTW9kZUluUmVhZCI6IkxFR0FDWSIsInNwYXJrLnNxbC5sZWdhY3kudGltZVBhcnNlclBvbGljeSI6IkxFR0FDWSIsInNwYXJrLnNxbC5wYXJxdWV0LmludDk2UmViYXNlTW9kZUluV3JpdGUiOiJMRUdBQ1kiLCJzcGFyay5zcWwuYXV0b0Jyb2FkY2FzdEpvaW5UaHJlc2hvbGQiOiItMSIsInNwYXJrLmV4ZWN1dG9yLmluc3RhbmNlcyI6IjEiLCJzcGFyay5leGVjdXRvci5leHRyYUphdmFPcHRpb25zIjoiLURmaWxlLmVuY29kaW5nPXV0Zi04Iiwic3BhcmsuY29yZXMubWF4IjoiMSIsInNwYXJrLmRyaXZlci5leHRyYUphdmFPcHRpb25zIjoiLURmaWxlLmVuY29kaW5nPXV0Zi04Iiwic3Bhcmsuc3FsLnBhcnF1ZXQuZW5hYmxlVmVjdG9yaXplZFJlYWRlciI6ImZhbHNlIn0sInN5bmNSdWxlIjp7Im51bUNvbmN1cnJlbmN5IjoxLCJudW1QYXJ0aXRpb25zIjoxLCJzZXRNb2RlIjoiU0lNUExFIiwic3FsQ29uZmlnSnNvbiI6Im51bGwifSwic3luY1dvcmtDb25maWciOnsiY29sdW1uTWFwIjpbeyJzb3VyY2UiOiJ1c2VuYW1lIiwidGFyZ2V0IjoidXNlcm5hbWUifV0sIm92ZXJNb2RlIjoiT1ZFUldSSVRFIiwicGFydGl0aW9uQ29sdW1uIjoidXNlbmFtZSIsInF1ZXJ5Q29uZGl0aW9uIjoidXNlbmFtZSA9ICcxJyIsInNvdXJjZURCSWQiOiJzeV8xOTg4OTAwNjgyOTI2MTk0Njg4Iiwic291cmNlREJUeXBlIjoiSElWRSIsInNvdXJjZURhdGFiYXNlIjp7ImRiVGFibGUiOiJ1c2VyczEiLCJkcml2ZXIiOiJvcmcuYXBhY2hlLmhpdmUuamRiYy5IaXZlRHJpdmVyIiwicGFzc3dvcmQiOiIiLCJ1cmwiOiJqZGJjOmhpdmUyOi8vMTA2LjE1Ljc3LjQ5OjEwMDAwL2RlZmF1bHQiLCJ1c2VyIjoicm9vdCJ9LCJzb3VyY2VUYWJsZSI6InVzZXJzMSIsInNvdXJjZVRhYmxlQ29sdW1uIjpbeyJjb2RlIjoidXNlbmFtZSIsInNxbCI6IiIsInR5cGUiOiJTVFJJTkcifV0sInRhcmdldERCSWQiOiJzeV8xOTg4OTAwNTE0NTU4NDQzNTIwIiwidGFyZ2V0REJUeXBlIjoiTVlTUUwiLCJ0YXJnZXREYXRhYmFzZSI6eyJkYlRhYmxlIjoidGVzdF9tb2RlbCIsImRyaXZlciI6ImNvbS5teXNxbC5jai5qZGJjLkRyaXZlciIsInBhc3N3b3JkIjoiaXNwb25nMTIzIiwidXJsIjoiamRiYzpteXNxbDovLzQ3LjkyLjE2OC4xMTY6MzAxMDIvaXNwb25nX2RiIiwidXNlciI6ImlzcG9uZyJ9LCJ0YXJnZXInVzZXJuYW1lIiwidHlwZSI6IlZBUkNIQVIoMjAwKSJ9XSwid29ya0lkIjoic3lfMTk4ODkwNDgwNjU3NzE0MzgwOCJ9fQ=="}
    args=new String[]{"代码断点处获取的args值"};
}

Comment out the configuration in sparkConf and use spark.master to set it to local[*]


public static SparkConf initSparkConf(Map<String, String> sparkConfig) {
    
//    SparkConf conf = new SparkConf();
//    if (sparkConfig != null) {
//        for (Map.Entry<String, String> entry : sparkConfig.entrySet()) {
//            conf.set(entry.getKey(), entry.getValue());
//        }
//    }
//    return conf;
    
    SparkConf conf = new SparkConf();
    conf.set("spark.master", "local[*]");
    return conf;
}

Introduce missing dependencies

For example, external dependencies such as database drivers

vim plugins/spark-data-sync-jdbc-plugin/build.gradle

implementation group: 'com.microsoft.sqlserver', name: 'mssql-jdbc', version: '12.4.2.jre8'

// https://mvnrepository.com/artifact/com.mysql/mysql-connector-j
implementation("com.mysql:mysql-connector-j:8.0.31")

Start debugging the plugin

20241206175947