Resource Center

This article introduces the usage scenarios, operating procedures and key configurations of the Resource Center function to help users complete daily management and business operations.

Query Resource Center

The resource center is the file management center of the Zhiqingyun platform The resource center includes four file types: jobs, functions, dependencies, and Excel. Supports resource type and remark search Supports operations such as file upload, download, deletion, notes, etc.

Resource Type Description

• Job: used for custom job, managing custom job Jar package, reference address: https://github.com/ispong/spark-job-template
• Function: used for Function Warehouse, managing custom function Jar packages, reference address: https://github.com/ispong/spark-function-template
• Dependencies: Basic dependency Jar packages required for job running, reference address: https://mvnrepository.com/
• Excel: used for Excel synchronization job, the Excel file will be exported to the specified target table

Upload resources

Click the Upload File button, select the file type, drag the file into the upload box, and click the OK button

• Type: required, resource types include: job, function, dependency, Excel
• Resource: required, resource file that needs to be uploaded
• Remarks: Optional

Use resource files

Demonstration use case: Parse uploaded JSON file Create a new JSON file

vim test.json

Note: JSON content, refer to the following format

{"age": 13,"username": "张三"}
{"age": 14,"username": "李四"}

To upload resource files, use the dependency type. Resource files of this type need to be added with the .jar suffix.

In the job configuration, add test.json dependency

Execute jobs in Standalone

Resource file address: {zhiqingyun-agent-path}/file/{fileId}.jar Example: /Users/ispong/zhiqingyun-agent/file/sy_1880173934985502720.jar

zhiqingyun-agent-path: The default address is ~ under the home directory, that is /home/xxx or /rootfileId: resource file id

CREATE TEMPORARY VIEW jsonTable
USING org.apache.spark.sql.json
OPTIONS (
  path '/Users/ispong/zhiqingyun-agent/file/sy_1880173934985502720.jar'
);

SELECT * FROM jsonTable

Execute jobs in Kubernetes

Resource file address: file:///opt/spark/jars/{fileId}.jar Example: file:///opt/spark/jars/sy_1880173934985502720.jar

fileId: resource file id

CREATE TEMPORARY VIEW jsonTable
USING org.apache.spark.sql.json
OPTIONS (
  path 'file:///opt/spark/jars/sy_1880173934985502720.jar'
);

SELECT * FROM jsonTable

Execute jobs in YARN

Resource file address: Files need to be uploaded to HDFS in advance /isxcode/sy_1880173934985502720.jar

zhiqingyun-agent-path: The default address is ~ under the home directory, that is /home/xxx or /rootfileId: resource file id

hadoop fs -put {zhiqingyun-agent-path}/file/{fileId}.jar /isxcode/
hadoop fs -put /home/zhiqingyun/zhiqingyun-agent/file/sy_1880173934985502720.jar /isxcode/

CREATE TEMPORARY VIEW jsonTable
USING org.apache.spark.sql.json
OPTIONS (
  path '/isxcode/sy_1880173934985502720.jar'
);

SELECT * FROM jsonTable