本地调试自定义插件
前提：创建一个standalone类型的计算集群
通过控制台关闭计算集群节点服务，不要使用界面重新检测状态，否则提交作业会校验计算集群不可用。
cd ~/zhiqingyun-agent/bin
bash stop.sh
通过idea启动计算集群节点代码
指定代码处打断点,获取args的值
spark-yun/spark-yun-agent/src/main/java/com/isxcode/star/agent/run/impl/StandaloneAgentService.java
通过界面，点击作业运行按钮，触发作业提交
以调试数据同步插件为例
spark-yun/spark-yun-plugins/spark-data-sync-jdbc-plugin/src/main/java/com/isxcode/star/plugin/dataSync/jdbc/Execute.java
将插件中的args变量，提前改成代码断点处获取的args值
或者使用arthas获取args
curl -O https://arthas.aliyun.com/arthas-boot.jar
java -jar arthas-boot.jar
watch org.apache.spark.launcher.SparkLauncher addAppArgs '{params,returnObj,throwExp}' -n 5 -x 3
[arthas@4480]$ watch org.apache.spark.launcher.SparkLauncher addAppArgs '{params,returnObj,throwExp}' -n 5 -x 3
Press Q or Ctrl+C to abort.
Affect(class count: 1 , method count: 2) cost in 150 ms, listenerId: 1
method=org.apache.spark.launcher.SparkLauncher.addAppArgs location=AtExit
ts=2025-11-13 21:07:30.084; [cost=1.232365ms] result=@ArrayList[
@Object[][
@String[][
@String[eyJjb250YWluZXJQb3J0IjowLCJmdW5jSW5mb0xpc3QiOltdLCJzcGFya0NvbmZpZyI6eyJzcGFyay5leGVjdXRvci5tZW1vcnkiOiIyZyIsInNwYXJrLnNxbC5wYXJxdWV0LndyaXRlTGVnYWN5Rm9ybWF0IjoidHJ1ZSIsInNwYXJrLmRyaXZlci5tZW1vcnkiOiIxZyIsInNwYXJrLnNxbC5wYXJxdWV0LmRhdGV0aW1lUmViYXNlTW9kZUluV3JpdGUiOiJMRUdBQ1kiLCJzcGFyay5kcml2ZXIuY29yZXMiOiIxIiwic3Bhcmsuc3FsLnN0b3JlQXNzaWdubWVudFBvbGljeSI6IkxFR0FDWSIsInNwYXJrLmV4ZWN1dG9yLmNvcmVzIjoiMSIsImhpdmUubWV0YXN0b3JlLnVyaXMiOiJ0aHJpZnQ6Ly8xMDYuMTUuNzcuNDk6OTA4MyIsInNwYXJrLnNxbC5wYXJxdWV0LmludDk2UmViYXNlTW9kZUluUmVhZCI6IkxFR0FDWSIsInNwYXJrLnNxbC5wYXJxdWV0LmRhdGV0aW1lUmViYXNlTW9kZUluUmVhZCI6IkxFR0FDWSIsInNwYXJrLnNxbC5sZWdhY3kudGltZVBhcnNlclBvbGljeSI6IkxFR0FDWSIsInNwYXJrLnNxbC5wYXJxdWV0LmludDk2UmViYXNlTW9kZUluV3JpdGUiOiJMRUdBQ1kiLCJzcGFyay5zcWwuYXV0b0Jyb2FkY2FzdEpvaW5UaHJlc2hvbGQiOiItMSIsInNwYXJrLmV4ZWN1dG9yLmluc3RhbmNlcyI6IjEiLCJzcGFyay5leGVjdXRvci5leHRyYUphdmFPcHRpb25zIjoiLURmaWxlLmVuY29kaW5nPXV0Zi04Iiwic3BhcmsuY29yZXMubWF4IjoiMSIsInNwYXJrLmRyaXZlci5leHRyYUphdmFPcHRpb25zIjoiLURmaWxlLmVuY29kaW5nPXV0Zi04Iiwic3Bhcmsuc3FsLnBhcnF1ZXQuZW5hYmxlVmVjdG9yaXplZFJlYWRlciI6ImZhbHNlIn0sInN5bmNSdWxlIjp7Im51bUNvbmN1cnJlbmN5IjoxLCJudW1QYXJ0aXRpb25zIjoxLCJzZXRNb2RlIjoiU0lNUExFIiwic3FsQ29uZmlnSnNvbiI6Im51bGwifSwic3luY1dvcmtDb25maWciOnsiY29sdW1uTWFwIjpbeyJzb3VyY2UiOiJ1c2VuYW1lIiwidGFyZ2V0IjoidXNlcm5hbWUifV0sIm92ZXJNb2RlIjoiT1ZFUldSSVRFIiwicGFydGl0aW9uQ29sdW1uIjoidXNlbmFtZSIsInF1ZXJ5Q29uZGl0aW9uIjoidXNlbmFtZSA9ICcxJyIsInNvdXJjZURCSWQiOiJzeV8xOTg4OTAwNjgyOTI2MTk0Njg4Iiwic291cmNlREJUeXBlIjoiSElWRSIsInNvdXJjZURhdGFiYXNlIjp7ImRiVGFibGUiOiJ1c2VyczEiLCJkcml2ZXIiOiJvcmcuYXBhY2hlLmhpdmUuamRiYy5IaXZlRHJpdmVyIiwicGFzc3dvcmQiOiIiLCJ1cmwiOiJqZGJjOmhpdmUyOi8vMTA2LjE1Ljc3LjQ5OjEwMDAwL2RlZmF1bHQiLCJ1c2VyIjoicm9vdCJ9LCJzb3VyY2VUYWJsZSI6InVzZXJzMSIsInNvdXJjZVRhYmxlQ29sdW1uIjpbeyJjb2RlIjoidXNlbmFtZSIsInNxbCI6IiIsInR5cGUiOiJTVFJJTkcifV0sInRhcmdldERCSWQiOiJzeV8xOTg4OTAwNTE0NTU4NDQzNTIwIiwidGFyZ2V0REJUeXBlIjoiTVlTUUwiLCJ0YXJnZXREYXRhYmFzZSI6eyJkYlRhYmxlIjoidGVzdF9tb2RlbCIsImRyaXZlciI6ImNvbS5teXNxbC5jai5qZGJjLkRyaXZlciIsInBhc3N3b3JkIjoiaXNwb25nMTIzIiwidXJsIjoiamRiYzpteXNxbDovLzQ3LjkyLjE2OC4xMTY6MzAxMDIvaXNwb25nX2RiIiwidXNlciI6ImlzcG9uZyJ9LCJ0dHlwZSI6IlZBUkNIQVIoMjAwKSJ9XSwid29ya0lkIjoic3lfMTk4ODkwNDgwNjU3NzE0MzgwOCJ9fQ==],
],
public static void main(String[] args) {
// args = new String[]{"eyJjb250YWluZXJQb3J0IjowLCJmdW5jSW5mb0xpc3QiOltdLCJzcGFya0NvbmZpZyI6eyJzcGFyay5leGVjdXRvci5tZW1vcnkiOiIyZyIsInNwYXJrLnNxbC5wYXJxdWV0LndyaXRlTGVnYWN5Rm9ybWF0IjoidHJ1ZSIsInNwYXJrLmRyaXZlci5tZW1vcnkiOiIxZyIsInNwYXJrLnNxbC5wYXJxdWV0LmRhdGV0aW1lUmViYXNlTW9kZUluV3JpdGUiOiJMRUdBQ1kiLCJzcGFyay5kcml2ZXIuY29yZXMiOiIxIiwic3Bhcmsuc3FsLnN0b3JlQXNzaWdubWVudFBvbGljeSI6IkxFR0FDWSIsInNwYXJrLmV4ZWN1dG9yLmNvcmVzIjoiMSIsImhpdmUubWV0YXN0b3JlLnVyaXMiOiJ0aHJpZnQ6Ly8xMDYuMTUuNzcuNDk6OTA4MyIsInNwYXJrLnNxbC5wYXJxdWV0LmludDk2UmViYXNlTW9kZUluUmVhZCI6IkxFR0FDWSIsInNwYXJrLnNxbC5wYXJxdWV0LmRhdGV0aW1lUmViYXNlTW9kZUluUmVhZCI6IkxFR0FDWSIsInNwYXJrLnNxbC5sZWdhY3kudGltZVBhcnNlclBvbGljeSI6IkxFR0FDWSIsInNwYXJrLnNxbC5wYXJxdWV0LmludDk2UmViYXNlTW9kZUluV3JpdGUiOiJMRUdBQ1kiLCJzcGFyay5zcWwuYXV0b0Jyb2FkY2FzdEpvaW5UaHJlc2hvbGQiOiItMSIsInNwYXJrLmV4ZWN1dG9yLmluc3RhbmNlcyI6IjEiLCJzcGFyay5leGVjdXRvci5leHRyYUphdmFPcHRpb25zIjoiLURmaWxlLmVuY29kaW5nPXV0Zi04Iiwic3BhcmsuY29yZXMubWF4IjoiMSIsInNwYXJrLmRyaXZlci5leHRyYUphdmFPcHRpb25zIjoiLURmaWxlLmVuY29kaW5nPXV0Zi04Iiwic3Bhcmsuc3FsLnBhcnF1ZXQuZW5hYmxlVmVjdG9yaXplZFJlYWRlciI6ImZhbHNlIn0sInN5bmNSdWxlIjp7Im51bUNvbmN1cnJlbmN5IjoxLCJudW1QYXJ0aXRpb25zIjoxLCJzZXRNb2RlIjoiU0lNUExFIiwic3FsQ29uZmlnSnNvbiI6Im51bGwifSwic3luY1dvcmtDb25maWciOnsiY29sdW1uTWFwIjpbeyJzb3VyY2UiOiJ1c2VuYW1lIiwidGFyZ2V0IjoidXNlcm5hbWUifV0sIm92ZXJNb2RlIjoiT1ZFUldSSVRFIiwicGFydGl0aW9uQ29sdW1uIjoidXNlbmFtZSIsInF1ZXJ5Q29uZGl0aW9uIjoidXNlbmFtZSA9ICcxJyIsInNvdXJjZURCSWQiOiJzeV8xOTg4OTAwNjgyOTI2MTk0Njg4Iiwic291cmNlREJUeXBlIjoiSElWRSIsInNvdXJjZURhdGFiYXNlIjp7ImRiVGFibGUiOiJ1c2VyczEiLCJkcml2ZXIiOiJvcmcuYXBhY2hlLmhpdmUuamRiYy5IaXZlRHJpdmVyIiwicGFzc3dvcmQiOiIiLCJ1cmwiOiJqZGJjOmhpdmUyOi8vMTA2LjE1Ljc3LjQ5OjEwMDAwL2RlZmF1bHQiLCJ1c2VyIjoicm9vdCJ9LCJzb3VyY2VUYWJsZSI6InVzZXJzMSIsInNvdXJjZVRhYmxlQ29sdW1uIjpbeyJjb2RlIjoidXNlbmFtZSIsInNxbCI6IiIsInR5cGUiOiJTVFJJTkcifV0sInRhcmdldERCSWQiOiJzeV8xOTg4OTAwNTE0NTU4NDQzNTIwIiwidGFyZ2V0REJUeXBlIjoiTVlTUUwiLCJ0YXJnZXREYXRhYmFzZSI6eyJkYlRhYmxlIjoidGVzdF9tb2RlbCIsImRyaXZlciI6ImNvbS5teXNxbC5jai5qZGJjLkRyaXZlciIsInBhc3N3b3JkIjoiaXNwb25nMTIzIiwidXJsIjoiamRiYzpteXNxbDovLzQ3LjkyLjE2OC4xMTY6MzAxMDIvaXNwb25nX2RiIiwidXNlciI6ImlzcG9uZyJ9LCJ0YXJnZXInVzZXJuYW1lIiwidHlwZSI6IlZBUkNIQVIoMjAwKSJ9XSwid29ya0lkIjoic3lfMTk4ODkwNDgwNjU3NzE0MzgwOCJ9fQ=="}
args=new String[]{"代码断点处获取的args值"};
}
将sparkConf中的配置注释掉，改用
spark.master设置为local[*]
public static SparkConf initSparkConf(Map<String, String> sparkConfig) {
// SparkConf conf = new SparkConf();
// if (sparkConfig != null) {
// for (Map.Entry<String, String> entry : sparkConfig.entrySet()) {
// conf.set(entry.getKey(), entry.getValue());
// }
// }
// return conf;
SparkConf conf = new SparkConf();
conf.set("spark.master", "local[*]");
return conf;
}
引入缺少的依赖
比如数据库的驱动等外部依赖
vim spark-yun-plugins/spark-data-sync-jdbc-plugin/build.gradle
implementation group: 'com.microsoft.sqlserver', name: 'mssql-jdbc', version: '12.4.2.jre8'
// https://mvnrepository.com/artifact/com.mysql/mysql-connector-j
implementation("com.mysql:mysql-connector-j:8.0.31")
开始调试插件
