ChunJun

Overview

ChunJun task type for executing ChunJun programs. For ChunJun nodes, the worker will execute ${CHUNJUN_HOME}/bin/start-chunjun to analyze the input json file.

Create Task

  • Click Project Management -> Project Name -> Workflow Definition, and click the Create Workflow button to enter the DAG editing page.
  • Drag the ChunJun - 图1 from the toolbar to the drawing board.

Task Parameters

ParameterDescription
Custom templateCustom the content of the ChunJun node’s json profile.
jsonjson configuration file for ChunJun synchronization.
Custom parametersIt is a user-defined parameter, and will replace the content with ${variable} in the script.
Deploy modeExecute chunjun task mode, eg local standalone.
Option ParametersSupport such as -confProp “{\”flink.checkpoint.interval\”:60000}”
Predecessor taskSelecting a predecessor task for the current task will set the selected predecessor task as upstream of the current task.

Task Example

This example demonstrates importing data from Hive into MySQL.

Configuring the ChunJun environment in DolphinScheduler

If you are using the ChunJun task type in a production environment, it is necessary to configure the required environment first. The configuration file is as follows: /dolphinscheduler/conf/env/dolphinscheduler_env.sh.

chunjun_task01

After the environment has been configured, DolphinScheduler needs to be restarted.

Configuring ChunJun Task Node

As the data to be read from Hive, a custom json is required, refer to: Hive Json Template.

After writing the required json file, you can configure the node content by following the steps in the diagram below.

chunjun_task02

View run results

chunjun_task03

Note

Before execute ${CHUNJUN_HOME}/bin/start-chunjun, need to change the shell ${CHUNJUN_HOME}/bin/start-chunjun, remove ‘&’ in order to run in front.

such as:

  1. nohup $JAVA_RUN -cp $JAR_DIR $CLASS_NAME $@ &

update to following:

  1. nohup $JAVA_RUN -cp $JAR_DIR $CLASS_NAME $@