Debugging a custom job locally

This article describes the development, commissioning and maintenance requirements related to job commissioning, applicable to the project development and delivery process.

Case: Parse local JSON file data

Download demo code

git clone https://github.com/ispong/spark-job-template.git

Create JSON data file

vim /Users/ispong/Downloads/test.json

{"age": 13,"username": "张三"}
{"age": 14,"username": "李四"}

write code

vim spark-job-template/src/main/java/com/isxcode/star/job/Execute.java

package com.isxcode.zhiqingyun.job;

import org.apache.spark.SparkConf;
import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Row;
import org.apache.spark.sql.SparkSession;

public class Execute {

    public static void main(String[] args) {

        try (SparkSession sparkSession = initSparkSession()) {
            Dataset<Row> dataset = sparkSession.read().json("/Users/ispong/Downloads/test.json");
            dataset.show();
            dataset.collectAsList().forEach(e -> {
                for (int i = 0; i < e.size(); i++) {
                    System.out.println(e.get(i));
                }
            });
        }
    }

    /**
     * 初始化本地sparkSession.
     */
    public static SparkSession initSparkSession() {

        SparkSession.Builder sparkSessionBuilder = SparkSession.builder();
        SparkConf conf = new SparkConf();
        conf.set("spark.master", "local[*]");
        return sparkSessionBuilder.config(conf).getOrCreate();
    }
}

Summary

在spark的config配置中，使用以下代码即可实现本地调试

conf.set("spark.master", "local[*]");