datax二次开发，非常详细[源码下载-项目搭建-打包发布]

最新推荐文章于 2025-04-23 17:22:18 发布
置顶 最新推荐文章于 2025-04-23 17:22:18 发布
阅读量4.4k 收藏 9
点赞数 2
分类专栏： 数据迁移 datax二次开发 datax打包发布
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/u014738571/article/details/108097370
版权
本文详细介绍了如何在DataX框架中自定义开发MongoDB Reader插件，以解决读取文档和数组类型数据的问题。通过修改源码，将未识别的Document和List类型转换为JSON字符串，确保数据能被正确处理。

摘要生成于 C知道 ，由 DeepSeek-R1 满血版支持， 前往体验 >

前序

参考文档：
datax 官网中的DataX插件开发宝典

一、环境准备

  • jdk1.8 及以上
  • idea 工具
  • git 工具

一、项目搭建

1、用idea下载源码
打开idea 选择 File->New -> Project from Version Control…
在这里插入图片描述
github地址：https://github.com/alibaba/DataX.git
在URL:填写gihub地址并Clone
在这里插入图片描述
2.等待加载相应文件…
3.Engine.java 启动文件
修改main方法，增加两行代码
//设置系统参数，参数值：datax编译后的目录
System.setProperty(“datax.home”, “D:\work\DataX\target\datax\datax”);
//设置启动参数：job的json文件的路径，和其他参数
String[] datxArgs = {"-job", “D:\work\DataX\core\src\main\job\mongo-file.json”, “-mode”, “standalone”, “-jobid”, “-1”};

   public static void main(String[] args) throws Exception {
       System.setProperty("datax.home", "D:\\work\\DataX\\target\\datax\\datax");  //datax编译后的目录
       String[] datxArgs = {"-job", "D:\\work\\DataX\\core\\src\\main\\job\\mongo-file.json", "-mode", "standalone", "-jobid", "-1"};  //自己的json文件路径
       int exitCode = 0;
       try {
           Engine.entry(datxArgs);
       } catch (Throwable e) {
           exitCode = 1;
           LOG.error("\n\n经DataX智能分析,该任务最可能的错误原因是:\n" + ExceptionTracker.trace(e));

           if (e instanceof DataXException) {
               DataXException tempException = (DataXException) e;
               ErrorCode errorCode = tempException.getErrorCode();
               if (errorCode instanceof FrameworkErrorCode) {
                   FrameworkErrorCode tempErrorCode = (FrameworkErrorCode) errorCode;
                   exitCode = tempErrorCode.toExitValue();
               }
           }

           System.exit(exitCode);
       }
       System.exit(exitCode);
   }

4.检查DataX 父目录下面pom.xml，注释掉无用的插件模块

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
   <modelVersion>4.0.0</modelVersion>

   <groupId>com.alibaba.datax</groupId>
   <artifactId>datax-all</artifactId>
   <version>0.0.1-SNAPSHOT</version>
   <dependencies>
       <dependency>
           <groupId>org.hamcrest</groupId>
           <artifactId>hamcrest-core</artifactId>
           <version>1.3</version>
       </dependency>
   </dependencies>

   <name>datax-all</name>
   <packaging>pom</packaging>

   <properties>
       <jdk-version>1.8</jdk-version>
       <datax-project-version>0.0.1-SNAPSHOT</datax-project-version>
       <commons-lang3-version>3.3.2</commons-lang3-version>
       <commons-configuration-version>1.10</commons-configuration-version>
       <commons-cli-version>1.2</commons-cli-version>
       <fastjson-version>1.1.46.sec01</fastjson-version>
       <guava-version>16.0.1</guava-version>
       <diamond.version>3.7.2.1-SNAPSHOT</diamond.version>

       <!--slf4j 1.7.10 和 logback-classic 1.0.13 是好基友 -->
       <slf4j-api-version>1.7.10</slf4j-api-version>
       <logback-classic-version>1.0.13</logback-classic-version>
       <commons-io-version>2.4</commons-io-version>
       <junit-version>4.11</junit-version>
       <tddl.version>5.1.22-1</tddl.version>
       <swift-version>1.0.0</swift-version>

       <project-sourceEncoding>UTF-8</project-sourceEncoding>
       <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
       <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
       <maven.compiler.encoding>UTF-8</maven.compiler.encoding>
   </properties>

   <modules>
       <module>common</module>
       <module>core</module>
       <module>transformer</module>

       <!-- reader -->
<!--        <module>mysqlreader</module>-->
<!--        <module>drdsreader</module>-->
<!--        <module>sqlserverreader</module>-->
<!--        <module>postgresqlreader</module>-->
<!--        <module>oraclereader</module>-->
<!--        <module>odpsreader</module>-->
<!--        <module>otsreader</module>-->
<!--        <module>otsstreamreader</module>-->
<!--        <module>txtfilereader</module>-->
<!--        <module>hdfsreader</module>-->
<!--        <module>streamreader</module>-->
<!--        <module>ossreader</module>-->
<!--        <module>ftpreader</module>-->
       <module>mongodbreader</module>
<!--        <module>rdbmsreader</module>-->
<!--        <module>hbase11xreader</module>-->
<!--        <module>hbase094xreader</module>-->
<!--        <module>tsdbreader</module>-->
<!--        <module>opentsdbreader</module>-->
<!--        <module>cassandrareader</module>-->
<!--        <module>gdbreader</module>-->

       <!-- writer -->
<!--        <module>mysqlwriter</module>-->
<!--        <module>drdswriter</module>-->
<!--        <module>odpswriter</module>-->
       <module>txtfilewriter</module>
<!--        <module>ftpwriter</module>-->
       <module>hdfswriter</module>
       <module>streamwriter</module>
<!--        <module>otswriter</module>-->
<!--        <module>oraclewriter</module>-->
<!--        <module>sqlserverwriter</module>-->
<!--        <module>postgresqlwriter</module>-->
<!--        <module>osswriter</module>-->
<!--        <module>mongodbwriter</module>-->
<!--        <module>adswriter</module>-->
<!--        <module>ocswriter</module>-->
<!--        <module>rdbmswriter</module>-->
<!--        <module>hbase11xwriter</module>-->
<!--        <module>hbase094xwriter</module>-->
<!--        <module>hbase11xsqlwriter</module>-->
<!--        <module>hbase11xsqlreader</module>-->
<!--        <module>elasticsearchwriter</module>-->
<!--        <module>tsdbwriter</module>-->
<!--        <module>adbpgwriter</module>-->
<!--        <module>gdbwriter</module>-->
<!--        <module>cassandrawriter</module>-->
<!--        <module>clickhousewriter</module>-->
       <!-- common support module -->
<!--        <module>plugin-rdbms-util</module>-->
<!--        <module>plugin-unstructured-storage-util</module>-->
<!--        <module>hbase20xsqlreader</module>-->
<!--        <module>hbase20xsqlwriter</module>-->
   </modules>

   <dependencyManagement>
       <dependencies>
           <dependency>
               <groupId>org.apache.commons</groupId>
               <artifactId>commons-lang3</artifactId>
               <version>${commons-lang3-version}</version>
           </dependency>
           <dependency>
               <groupId>com.alibaba</groupId>
               <artifactId>fastjson</artifactId>
               <version>${fastjson-version}</version>
           </dependency>
           <!--<dependency>
               <groupId>com.google.guava</groupId>
               <artifactId>guava</artifactId>
               <version>${guava-version}</version>
           </dependency>-->
           <dependency>
               <groupId>commons-io</groupId>
               <artifactId>commons-io</artifactId>
               <version>${commons-io-version}</version>
           </dependency>
           <dependency>
               <groupId>org.slf4j</groupId>
               <artifactId>slf4j-api</artifactId>
               <version>${slf4j-api-version}</version>
           </dependency>
           <dependency>
               <groupId>ch.qos.logback</groupId>
               <artifactId>logback-classic</artifactId>
               <version>${logback-classic-version}</version>
           </dependency>

           <dependency>
               <groupId>com.taobao.tddl</groupId>
               <artifactId>tddl-client</artifactId>
               <version>${tddl.version}</version>
               <exclusions>
                   <exclusion>
                       <groupId>com.google.guava</groupId>
                       <artifactId>guava</artifactId>
                   </exclusion>
                   <exclusion>
                       <groupId>com.taobao.diamond</groupId>
                       <artifactId>diamond-client</artifactId>
                   </exclusion>
               </exclusions>
           </dependency>

           <dependency>
               <groupId>com.taobao.diamond</groupId>
               <artifactId>diamond-client</artifactId>
               <version>${diamond.version}</version>
           </dependency>

           <dependency>
               <groupId>com.alibaba.search.swift</groupId>
               <artifactId>swift_client</artifactId>
               <version>${swift-version}</version>
           </dependency>

           <dependency>
               <groupId>junit</groupId>
               <artifactId>junit</artifactId>
               <version>${junit-version}</version>
           </dependency>

           <dependency>
               <groupId>org.mockito</groupId>
               <artifactId>mockito-all</artifactId>
               <version>1.9.5</version>
               <scope>test</scope>
           </dependency>
       </dependencies>
   </dependencyManagement>

   <repositories>
       <repository>
           <id>central</id>
           <name>Nexus aliyun</name>
           <url>https://maven.aliyun.com/repository/central</url>
           <releases>
               <enabled>true</enabled>
           </releases>
           <snapshots>
               <enabled>true</enabled>
           </snapshots>
       </repository>
   </repositories>

   <pluginRepositories>
       <pluginRepository>
           <id>central</id>
           <name>Nexus aliyun</name>
           <url>https://maven.aliyun.com/repository/central</url>
           <releases>
               <enabled>true</enabled>
           </releases>
           <snapshots>
               <enabled>true</enabled>
           </snapshots>
       </pluginRepository>
   </pluginRepositories>

   <build>
       <plugins>
           <plugin>
               <artifactId>maven-assembly-plugin</artifactId>
               <configuration>
                   <finalName>datax</finalName>
                   <descriptors>
                       <descriptor>package.xml</descriptor>
                   </descriptors>
               </configuration>
               <executions>
                   <execution>
                       <id>make-assembly</id>
                       <phase>package</phase>
                   </execution>
               </executions>
           </plugin>
           <plugin>
               <groupId>org.apache.maven.plugins</groupId>
               <artifactId>maven-compiler-plugin</artifactId>
               <version>2.3.2</version>
               <configuration>
                   <source>${jdk-version}</source>
                   <target>${jdk-version}</target>
                   <encoding>${project-sourceEncoding}</encoding>
               </configuration>
           </plugin>
       </plugins>
   </build>
</project>

5.调bug
我的版本3.0
存在bug: ClickhouseWriter.java 中引用的类(import ru.yandex.clickhouse.ClickHouseTuple;)无效

import ru.yandex.clickhouse.ClickHouseTuple;

解决办法：直接删除该行即可

6.打包发布：
DataX使用assembly打包，assembly的使用方法请咨询谷哥或者度娘。打包命令如下：

mvn clean package -DskipTests assembly:assembly

编译后结果

D:\ProgramFiles\Java\jdk1.8.0_241\bin\java.exe -Dmaven.multiModuleProjectDirectory=D:\work\DataX -Dmaven.home=D:\ProgramFiles\JetBrains\IntelliJIDEA2020.1.2\plugins\maven\lib\maven3 -Dclassworlds.conf=D:\ProgramFiles\JetBrains\IntelliJIDEA2020.1.2\plugins\maven\lib\maven3\bin\m2.conf -Dmaven.ext.class.path=D:\ProgramFiles\JetBrains\IntelliJIDEA2020.1.2\plugins\maven\lib\maven-event-listener.jar -javaagent:D:\ProgramFiles\JetBrains\IntelliJIDEA2020.1.2\lib\idea_rt.jar=60517:D:\ProgramFiles\JetBrains\IntelliJIDEA2020.1.2\bin -Dfile.encoding=UTF-8 -classpath D:\ProgramFiles\JetBrains\IntelliJIDEA2020.1.2\plugins\maven\lib\maven3\boot\plexus-classworlds-2.6.0.jar;D:\ProgramFiles\JetBrains\IntelliJIDEA2020.1.2\plugins\maven\lib\maven3\boot\plexus-classworlds.license org.codehaus.classworlds.Launcher -Didea.version2020.1.3 -s D:\work\conf\settings.xml -Dmaven.repo.local=D:\work\repository -DskipTests=true clean package -DskipTests assembly:assembly
[INFO] Scanning for projects...
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Build Order:
[INFO] 
[INFO] datax-all                                                          [pom]
[INFO] datax-common                                                       [jar]
[INFO] datax-transformer                                                  [jar]
[INFO] datax-core                                                         [jar]
[INFO] plugin-unstructured-storage-util                                   [jar]
[INFO] mongodbreader                                                      [jar]
[INFO] txtfilewriter                                                      [jar]
[INFO] hdfswriter                                                         [jar]
[INFO] streamwriter                                                       [jar]
[INFO] plugin-rdbms-util                                                  [jar]
[INFO] hbase20xsqlreader                                                  [jar]
[INFO] hbase20xsqlwriter                                                  [jar]
[INFO] 
[INFO] --------------------< com.alibaba.datax:datax-all >---------------------
[INFO] Building datax-all 0.0.1-SNAPSHOT                                 [1/12]
[INFO] --------------------------------[ pom ]---------------------------------
[INFO] 
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ datax-all ---
[INFO] Deleting D:\work\DataX\target
[INFO] 
[INFO] -------------------< com.alibaba.datax:datax-common >-------------------
[INFO] Building datax-common 0.0.1-SNAPSHOT                              [2/12]
[INFO] --------------------------------[ jar ]---------------------------------
[INFO] 
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ datax-common ---
[INFO] Deleting D:\work\DataX\common\target
[INFO] 
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ datax-common ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory D:\work\DataX\common\src\main\resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ datax-common ---
[INFO] Compiling 39 source files to D:\work\DataX\common\target\classes
[INFO] 
[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ datax-common ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory D:\work\DataX\common\src\test\resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ datax-common ---
[INFO] No sources to compile
[INFO] 
[INFO] --- maven-surefire-plugin:2.12.4:test (default-test) @ datax-common ---
[INFO] Tests are skipped.
[INFO] 
[INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ datax-common ---
[INFO] Building jar: D:\work\DataX\common\target\datax-common-0.0.1-SNAPSHOT.jar
[INFO] 
[INFO] ----------------< com.alibaba.datax:datax-transformer >-----------------
[INFO] Building datax-transformer 0.0.1-SNAPSHOT                         [3/12]
[INFO] --------------------------------[ jar ]---------------------------------
[INFO] 
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ datax-transformer ---
[INFO] Deleting D:\work\DataX\transformer\target
[INFO] 
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ datax-transformer ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory D:\work\DataX\transformer\src\main\resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ datax-transformer ---
[INFO] Compiling 2 source files to D:\work\DataX\transformer\target\classes
[INFO] 
[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ datax-transformer ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory D:\work\DataX\transformer\src\test\resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ datax-transformer ---
[INFO] No sources to compile
[INFO] 
[INFO] --- maven-surefire-plugin:2.12.4:test (default-test) @ datax-transformer ---
[INFO] Tests are skipped.
[INFO] 
[INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ datax-transformer ---
[INFO] Building jar: D:\work\DataX\transformer\target\datax-transformer-0.0.1-SNAPSHOT.jar
[INFO] 
[INFO] --- maven-assembly-plugin:2.2-beta-5:single (dwzip) @ datax-transformer ---
[INFO] Reading assembly descriptor: src/main/assembly/package.xml
[INFO] Copying files to D:\work\DataX\transformer\target\datax
[WARNING] Assembly file: D:\work\DataX\transformer\target\datax is not a regular file (it may be a directory). It cannot be attached to the project build for installation or deployment.
[INFO] 
[INFO] --------------------< com.alibaba.datax:datax-core >--------------------
[INFO] Building datax-core 0.0.1-SNAPSHOT                                [4/12]
[INFO] --------------------------------[ jar ]---------------------------------
[INFO] 
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ datax-core ---
[INFO] Deleting D:\work\DataX\core\target
[INFO] 
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ datax-core ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory D:\work\DataX\core\src\main\resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ datax-core ---
[INFO] Compiling 70 source files to D:\work\DataX\core\target\classes
[INFO] 
[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ datax-core ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory D:\work\DataX\core\src\test\resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ datax-core ---
[INFO] No sources to compile
[INFO] 
[INFO] --- maven-surefire-plugin:2.12.4:test (default-test) @ datax-core ---
[INFO] Tests are skipped.
[INFO] 
[INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ datax-core ---
[INFO] Building jar: D:\work\DataX\core\target\datax-core-0.0.1-SNAPSHOT.jar
[INFO] 
[INFO] --- maven-assembly-plugin:2.2-beta-5:single (default) @ datax-core ---
[INFO] Reading assembly descriptor: src/main/assembly/package.xml
[INFO] Copying files to D:\work\DataX\core\target\datax
[WARNING] Assembly file: D:\work\DataX\core\target\datax is not a regular file (it may be a directory). It cannot be attached to the project build for installation or deployment.
[INFO] 
[INFO] ---------< com.alibaba.datax:plugin-unstructured-storage-util >---------
[INFO] Building plugin-unstructured-storage-util 0.0.1-SNAPSHOT          [5/12]
[INFO] --------------------------------[ jar ]---------------------------------
[INFO] 
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ plugin-unstructured-storage-util ---
[INFO] Deleting D:\work\DataX\plugin-unstructured-storage-util\target
[INFO] 
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ plugin-unstructured-storage-util ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory D:\work\DataX\plugin-unstructured-storage-util\src\main\resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ plugin-unstructured-storage-util ---
[INFO] Compiling 13 source files to D:\work\DataX\plugin-unstructured-storage-util\target\classes
[INFO] 
[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ plugin-unstructured-storage-util ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory D:\work\DataX\plugin-unstructured-storage-util\src\test\resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ plugin-unstructured-storage-util ---
[INFO] No sources to compile
[INFO] 
[INFO] --- maven-surefire-plugin:2.12.4:test (default-test) @ plugin-unstructured-storage-util ---
[INFO] Tests are skipped.
[INFO] 
[INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ plugin-unstructured-storage-util ---
[INFO] Building jar: D:\work\DataX\plugin-unstructured-storage-util\target\plugin-unstructured-storage-util-0.0.1-SNAPSHOT.jar
[INFO] 
[INFO] ------------------< com.alibaba.datax:mongodbreader >-------------------
[INFO] Building mongodbreader 0.0.1-SNAPSHOT                             [6/12]
[INFO] --------------------------------[ jar ]---------------------------------
[INFO] 
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ mongodbreader ---
[INFO] Deleting D:\work\DataX\mongodbreader\target
[INFO] 
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ mongodbreader ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 2 resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ mongodbreader ---
[INFO] Compiling 5 source files to D:\work\DataX\mongodbreader\target\classes
[INFO] 
[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ mongodbreader ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory D:\work\DataX\mongodbreader\src\test\resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ mongodbreader ---
[INFO] No sources to compile
[INFO] 
[INFO] --- maven-surefire-plugin:2.12.4:test (default-test) @ mongodbreader ---
[INFO] Tests are skipped.
[INFO] 
[INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ mongodbreader ---
[INFO] Building jar: D:\work\DataX\mongodbreader\target\mongodbreader-0.0.1-SNAPSHOT.jar
[INFO] 
[INFO] --- maven-assembly-plugin:2.2-beta-5:single (dwzip) @ mongodbreader ---
[INFO] Reading assembly descriptor: src/main/assembly/package.xml
[INFO] Copying files to D:\work\DataX\mongodbreader\target\datax
[WARNING] Assembly file: D:\work\DataX\mongodbreader\target\datax is not a regular file (it may be a directory). It cannot be attached to the project build for installation or deployment.
[INFO] 
[INFO] ------------------< com.alibaba.datax:txtfilewriter >-------------------
[INFO] Building txtfilewriter 0.0.1-SNAPSHOT                             [7/12]
[INFO] --------------------------------[ jar ]---------------------------------
[INFO] 
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ txtfilewriter ---
[INFO] Deleting D:\work\DataX\txtfilewriter\target
[INFO] 
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ txtfilewriter ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 2 resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ txtfilewriter ---
[INFO] Compiling 3 source files to D:\work\DataX\txtfilewriter\target\classes
[INFO] 
[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ txtfilewriter ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory D:\work\DataX\txtfilewriter\src\test\resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ txtfilewriter ---
[INFO] No sources to compile
[INFO] 
[INFO] --- maven-surefire-plugin:2.12.4:test (default-test) @ txtfilewriter ---
[INFO] Tests are skipped.
[INFO] 
[INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ txtfilewriter ---
[INFO] Building jar: D:\work\DataX\txtfilewriter\target\txtfilewriter-0.0.1-SNAPSHOT.jar
[INFO] 
[INFO] --- maven-assembly-plugin:2.2-beta-5:single (dwzip) @ txtfilewriter ---
[INFO] Reading assembly descriptor: src/main/assembly/package.xml
[INFO] Copying files to D:\work\DataX\txtfilewriter\target\datax
[WARNING] Assembly file: D:\work\DataX\txtfilewriter\target\datax is not a regular file (it may be a directory). It cannot be attached to the project build for installation or deployment.
[INFO] 
[INFO] --------------------< com.alibaba.datax:hdfswriter >--------------------
[INFO] Building hdfswriter 0.0.1-SNAPSHOT                                [8/12]
[INFO] --------------------------------[ jar ]---------------------------------
[INFO] 
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ hdfswriter ---
[INFO] Deleting D:\work\DataX\hdfswriter\target
[INFO] 
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ hdfswriter ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 2 resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ hdfswriter ---
[INFO] Compiling 6 source files to D:\work\DataX\hdfswriter\target\classes
[INFO] 
[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ hdfswriter ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory D:\work\DataX\hdfswriter\src\test\resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ hdfswriter ---
[INFO] No sources to compile
[INFO] 
[INFO] --- maven-surefire-plugin:2.12.4:test (default-test) @ hdfswriter ---
[INFO] Tests are skipped.
[INFO] 
[INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ hdfswriter ---
[INFO] Building jar: D:\work\DataX\hdfswriter\target\hdfswriter-0.0.1-SNAPSHOT.jar
[INFO] 
[INFO] --- maven-assembly-plugin:2.2-beta-5:single (dwzip) @ hdfswriter ---
[INFO] Reading assembly descriptor: src/main/assembly/package.xml
Downloading from alimaven: http://maven.aliyun.com/nexus/content/groups/public/eigenbase/eigenbase-properties/1.1.4/eigenbase-properties-1.1.4.pom
Downloading from alimaven: http://maven.aliyun.com/nexus/content/groups/public/org/pentaho/pentaho-aggdesigner-algorithm/5.1.5-jhyde/pentaho-aggdesigner-algorithm-5.1.5-jhyde.pom
[INFO] Copying files to D:\work\DataX\hdfswriter\target\datax
[WARNING] Assembly file: D:\work\DataX\hdfswriter\target\datax is not a regular file (it may be a directory). It cannot be attached to the project build for installation or deployment.
[INFO] 
[INFO] -------------------< com.alibaba.datax:streamwriter >-------------------
[INFO] Building streamwriter 0.0.1-SNAPSHOT                              [9/12]
[INFO] --------------------------------[ jar ]---------------------------------
[INFO] 
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ streamwriter ---
[INFO] Deleting D:\work\DataX\streamwriter\target
[INFO] 
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ streamwriter ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 2 resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ streamwriter ---
[INFO] Compiling 3 source files to D:\work\DataX\streamwriter\target\classes
[INFO] 
[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ streamwriter ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory D:\work\DataX\streamwriter\src\test\resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ streamwriter ---
[INFO] No sources to compile
[INFO] 
[INFO] --- maven-surefire-plugin:2.12.4:test (default-test) @ streamwriter ---
[INFO] Tests are skipped.
[INFO] 
[INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ streamwriter ---
[INFO] Building jar: D:\work\DataX\streamwriter\target\streamwriter-0.0.1-SNAPSHOT.jar
[INFO] 
[INFO] --- maven-assembly-plugin:2.2-beta-5:single (dwzip) @ streamwriter ---
[INFO] Reading assembly descriptor: src/main/assembly/package.xml
[INFO] Copying files to D:\work\DataX\streamwriter\target\datax
[WARNING] Assembly file: D:\work\DataX\streamwriter\target\datax is not a regular file (it may be a directory). It cannot be attached to the project build for installation or deployment.
[INFO] 
[INFO] ----------------< com.alibaba.datax:plugin-rdbms-util >-----------------
[INFO] Building plugin-rdbms-util 0.0.1-SNAPSHOT                        [10/12]
[INFO] --------------------------------[ jar ]---------------------------------
[INFO] 
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ plugin-rdbms-util ---
[INFO] Deleting D:\work\DataX\plugin-rdbms-util\target
[INFO] 
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ plugin-rdbms-util ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory D:\work\DataX\plugin-rdbms-util\src\main\resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ plugin-rdbms-util ---
[INFO] Compiling 24 source files to D:\work\DataX\plugin-rdbms-util\target\classes
[INFO] 
[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ plugin-rdbms-util ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory D:\work\DataX\plugin-rdbms-util\src\test\resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ plugin-rdbms-util ---
[INFO] No sources to compile
[INFO] 
[INFO] --- maven-surefire-plugin:2.12.4:test (default-test) @ plugin-rdbms-util ---
[INFO] Tests are skipped.
[INFO] 
[INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ plugin-rdbms-util ---
[INFO] Building jar: D:\work\DataX\plugin-rdbms-util\target\plugin-rdbms-util-0.0.1-SNAPSHOT.jar
[INFO] 
[INFO] ----------------< com.alibaba.datax:hbase20xsqlreader >-----------------
[INFO] Building hbase20xsqlreader 0.0.1-SNAPSHOT                        [11/12]
[INFO] --------------------------------[ jar ]---------------------------------
[INFO] 
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ hbase20xsqlreader ---
[INFO] Deleting D:\work\DataX\hbase20xsqlreader\target
[INFO] 
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ hbase20xsqlreader ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 0 resource
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ hbase20xsqlreader ---
[INFO] Compiling 6 source files to D:\work\DataX\hbase20xsqlreader\target\classes
[INFO] 
[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ hbase20xsqlreader ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory D:\work\DataX\hbase20xsqlreader\src\test\resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ hbase20xsqlreader ---
[INFO] No sources to compile
[INFO] 
[INFO] --- maven-surefire-plugin:2.12.4:test (default-test) @ hbase20xsqlreader ---
[INFO] Tests are skipped.
[INFO] 
[INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ hbase20xsqlreader ---
[INFO] Building jar: D:\work\DataX\hbase20xsqlreader\target\hbase20xsqlreader-0.0.1-SNAPSHOT.jar
[INFO] 
[INFO] --- maven-assembly-plugin:2.2-beta-5:single (dwzip) @ hbase20xsqlreader ---
[INFO] Reading assembly descriptor: src/main/assembly/package.xml
[INFO] Copying files to D:\work\DataX\hbase20xsqlreader\target\datax
[WARNING] Assembly file: D:\work\DataX\hbase20xsqlreader\target\datax is not a regular file (it may be a directory). It cannot be attached to the project build for installation or deployment.
[INFO] 
[INFO] ----------------< com.alibaba.datax:hbase20xsqlwriter >-----------------
[INFO] Building hbase20xsqlwriter 0.0.1-SNAPSHOT                        [12/12]
[INFO] --------------------------------[ jar ]---------------------------------
[INFO] 
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ hbase20xsqlwriter ---
[INFO] Deleting D:\work\DataX\hbase20xsqlwriter\target
[INFO] 
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ hbase20xsqlwriter ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 0 resource
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ hbase20xsqlwriter ---
[INFO] Compiling 7 source files to D:\work\DataX\hbase20xsqlwriter\target\classes
[INFO] 
[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ hbase20xsqlwriter ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory D:\work\DataX\hbase20xsqlwriter\src\test\resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ hbase20xsqlwriter ---
[INFO] No sources to compile
[INFO] 
[INFO] --- maven-surefire-plugin:2.12.4:test (default-test) @ hbase20xsqlwriter ---
[INFO] Tests are skipped.
[INFO] 
[INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ hbase20xsqlwriter ---
[INFO] Building jar: D:\work\DataX\hbase20xsqlwriter\target\hbase20xsqlwriter-0.0.1-SNAPSHOT.jar
[INFO] 
[INFO] --- maven-assembly-plugin:2.2-beta-5:single (dwzip) @ hbase20xsqlwriter ---
[INFO] Reading assembly descriptor: src/main/assembly/package.xml
[INFO] Copying files to D:\work\DataX\hbase20xsqlwriter\target\datax
[WARNING] Assembly file: D:\work\DataX\hbase20xsqlwriter\target\datax is not a regular file (it may be a directory). It cannot be attached to the project build for installation or deployment.
[INFO] 
[INFO] --------------------< com.alibaba.datax:datax-all >---------------------
[INFO] Building datax-all 0.0.1-SNAPSHOT                                [13/12]
[INFO] --------------------------------[ pom ]---------------------------------
[INFO] 
[INFO] >>> maven-assembly-plugin:2.2-beta-5:assembly (default-cli) > package @ datax-all >>>
[INFO] 
[INFO] >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
[INFO] Forking datax-common 0.0.1-SNAPSHOT
[INFO] >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
[INFO] 
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ datax-common ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory D:\work\DataX\common\src\main\resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ datax-common ---
[INFO] Nothing to compile - all classes are up to date
[INFO] 
[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ datax-common ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory D:\work\DataX\common\src\test\resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ datax-common ---
[INFO] No sources to compile
[INFO] 
[INFO] --- maven-surefire-plugin:2.12.4:test (default-test) @ datax-common ---
[INFO] Tests are skipped.
[INFO] 
[INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ datax-common ---
[INFO] 
[INFO] >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
[INFO] Forking datax-transformer 0.0.1-SNAPSHOT
[INFO] >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
[INFO] 
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ datax-transformer ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory D:\work\DataX\transformer\src\main\resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ datax-transformer ---
[INFO] Nothing to compile - all classes are up to date
[INFO] 
[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ datax-transformer ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory D:\work\DataX\transformer\src\test\resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ datax-transformer ---
[INFO] No sources to compile
[INFO] 
[INFO] --- maven-surefire-plugin:2.12.4:test (default-test) @ datax-transformer ---
[INFO] Tests are skipped.
[INFO] 
[INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ datax-transformer ---
[INFO] 
[INFO] --- maven-assembly-plugin:2.2-beta-5:single (dwzip) @ datax-transformer ---
[INFO] Reading assembly descriptor: src/main/assembly/package.xml
[INFO] Copying files to D:\work\DataX\transformer\target\datax
[WARNING] Assembly file: D:\work\DataX\transformer\target\datax is not a regular file (it may be a directory). It cannot be attached to the project build for installation or deployment.
[INFO] 
[INFO] >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
[INFO] Forking datax-core 0.0.1-SNAPSHOT
[INFO] >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
[INFO] 
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ datax-core ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory D:\work\DataX\core\src\main\resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ datax-core ---
[INFO] Nothing to compile - all classes are up to date
[INFO] 
[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ datax-core ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory D:\work\DataX\core\src\test\resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ datax-core ---
[INFO] No sources to compile
[INFO] 
[INFO] --- maven-surefire-plugin:2.12.4:test (default-test) @ datax-core ---
[INFO] Tests are skipped.
[INFO] 
[INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ datax-core ---
[INFO] 
[INFO] --- maven-assembly-plugin:2.2-beta-5:single (default) @ datax-core ---
[INFO] Reading assembly descriptor: src/main/assembly/package.xml
[INFO] Copying files to D:\work\DataX\core\target\datax
[WARNING] Assembly file: D:\work\DataX\core\target\datax is not a regular file (it may be a directory). It cannot be attached to the project build for installation or deployment.
[INFO] 
[INFO] >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
[INFO] Forking plugin-unstructured-storage-util 0.0.1-SNAPSHOT
[INFO] >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
[INFO] 
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ plugin-unstructured-storage-util ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory D:\work\DataX\plugin-unstructured-storage-util\src\main\resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ plugin-unstructured-storage-util ---
[INFO] Nothing to compile - all classes are up to date
[INFO] 
[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ plugin-unstructured-storage-util ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory D:\work\DataX\plugin-unstructured-storage-util\src\test\resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ plugin-unstructured-storage-util ---
[INFO] No sources to compile
[INFO] 
[INFO] --- maven-surefire-plugin:2.12.4:test (default-test) @ plugin-unstructured-storage-util ---
[INFO] Tests are skipped.
[INFO] 
[INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ plugin-unstructured-storage-util ---
[INFO] 
[INFO] >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
[INFO] Forking mongodbreader 0.0.1-SNAPSHOT
[INFO] >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
[INFO] 
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ mongodbreader ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 2 resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ mongodbreader ---
[INFO] Nothing to compile - all classes are up to date
[INFO] 
[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ mongodbreader ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory D:\work\DataX\mongodbreader\src\test\resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ mongodbreader ---
[INFO] No sources to compile
[INFO] 
[INFO] --- maven-surefire-plugin:2.12.4:test (default-test) @ mongodbreader ---
[INFO] Tests are skipped.
[INFO] 
[INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ mongodbreader ---
[INFO] 
[INFO] --- maven-assembly-plugin:2.2-beta-5:single (dwzip) @ mongodbreader ---
[INFO] Reading assembly descriptor: src/main/assembly/package.xml
[INFO] Copying files to D:\work\DataX\mongodbreader\target\datax
[WARNING] Assembly file: D:\work\DataX\mongodbreader\target\datax is not a regular file (it may be a directory). It cannot be attached to the project build for installation or deployment.
[INFO] 
[INFO] >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
[INFO] Forking txtfilewriter 0.0.1-SNAPSHOT
[INFO] >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
[INFO] 
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ txtfilewriter ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 2 resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ txtfilewriter ---
[INFO] Nothing to compile - all classes are up to date
[INFO] 
[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ txtfilewriter ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory D:\work\DataX\txtfilewriter\src\test\resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ txtfilewriter ---
[INFO] No sources to compile
[INFO] 
[INFO] --- maven-surefire-plugin:2.12.4:test (default-test) @ txtfilewriter ---
[INFO] Tests are skipped.
[INFO] 
[INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ txtfilewriter ---
[INFO] 
[INFO] --- maven-assembly-plugin:2.2-beta-5:single (dwzip) @ txtfilewriter ---
[INFO] Reading assembly descriptor: src/main/assembly/package.xml
[INFO] Copying files to D:\work\DataX\txtfilewriter\target\datax
[WARNING] Assembly file: D:\work\DataX\txtfilewriter\target\datax is not a regular file (it may be a directory). It cannot be attached to the project build for installation or deployment.
[INFO] 
[INFO] >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
[INFO] Forking hdfswriter 0.0.1-SNAPSHOT
[INFO] >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
[INFO] 
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ hdfswriter ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 2 resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ hdfswriter ---
[INFO] Nothing to compile - all classes are up to date
[INFO] 
[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ hdfswriter ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory D:\work\DataX\hdfswriter\src\test\resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ hdfswriter ---
[INFO] No sources to compile
[INFO] 
[INFO] --- maven-surefire-plugin:2.12.4:test (default-test) @ hdfswriter ---
[INFO] Tests are skipped.
[INFO] 
[INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ hdfswriter ---
[INFO] 
[INFO] --- maven-assembly-plugin:2.2-beta-5:single (dwzip) @ hdfswriter ---
[INFO] Reading assembly descriptor: src/main/assembly/package.xml
[INFO] Copying files to D:\work\DataX\hdfswriter\target\datax
[WARNING] Assembly file: D:\work\DataX\hdfswriter\target\datax is not a regular file (it may be a directory). It cannot be attached to the project build for installation or deployment.
[INFO] 
[INFO] >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
[INFO] Forking streamwriter 0.0.1-SNAPSHOT
[INFO] >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
[INFO] 
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ streamwriter ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 2 resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ streamwriter ---
[INFO] Nothing to compile - all classes are up to date
[INFO] 
[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ streamwriter ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory D:\work\DataX\streamwriter\src\test\resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ streamwriter ---
[INFO] No sources to compile
[INFO] 
[INFO] --- maven-surefire-plugin:2.12.4:test (default-test) @ streamwriter ---
[INFO] Tests are skipped.
[INFO] 
[INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ streamwriter ---
[INFO] 
[INFO] --- maven-assembly-plugin:2.2-beta-5:single (dwzip) @ streamwriter ---
[INFO] Reading assembly descriptor: src/main/assembly/package.xml
[INFO] Copying files to D:\work\DataX\streamwriter\target\datax
[WARNING] Assembly file: D:\work\DataX\streamwriter\target\datax is not a regular file (it may be a directory). It cannot be attached to the project build for installation or deployment.
[INFO] 
[INFO] >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
[INFO] Forking plugin-rdbms-util 0.0.1-SNAPSHOT
[INFO] >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
[INFO] 
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ plugin-rdbms-util ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory D:\work\DataX\plugin-rdbms-util\src\main\resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ plugin-rdbms-util ---
[INFO] Nothing to compile - all classes are up to date
[INFO] 
[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ plugin-rdbms-util ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory D:\work\DataX\plugin-rdbms-util\src\test\resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ plugin-rdbms-util ---
[INFO] No sources to compile
[INFO] 
[INFO] --- maven-surefire-plugin:2.12.4:test (default-test) @ plugin-rdbms-util ---
[INFO] Tests are skipped.
[INFO] 
[INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ plugin-rdbms-util ---
[INFO] 
[INFO] >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
[INFO] Forking hbase20xsqlreader 0.0.1-SNAPSHOT
[INFO] >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
[INFO] 
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ hbase20xsqlreader ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 0 resource
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ hbase20xsqlreader ---
[INFO] Nothing to compile - all classes are up to date
[INFO] 
[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ hbase20xsqlreader ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory D:\work\DataX\hbase20xsqlreader\src\test\resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ hbase20xsqlreader ---
[INFO] No sources to compile
[INFO] 
[INFO] --- maven-surefire-plugin:2.12.4:test (default-test) @ hbase20xsqlreader ---
[INFO] Tests are skipped.
[INFO] 
[INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ hbase20xsqlreader ---
[INFO] 
[INFO] --- maven-assembly-plugin:2.2-beta-5:single (dwzip) @ hbase20xsqlreader ---
[INFO] Reading assembly descriptor: src/main/assembly/package.xml
[INFO] Copying files to D:\work\DataX\hbase20xsqlreader\target\datax
[WARNING] Assembly file: D:\work\DataX\hbase20xsqlreader\target\datax is not a regular file (it may be a directory). It cannot be attached to the project build for installation or deployment.
[INFO] 
[INFO] >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
[INFO] Forking hbase20xsqlwriter 0.0.1-SNAPSHOT
[INFO] >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
[INFO] 
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ hbase20xsqlwriter ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 0 resource
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ hbase20xsqlwriter ---
[INFO] Nothing to compile - all classes are up to date
[INFO] 
[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ hbase20xsqlwriter ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory D:\work\DataX\hbase20xsqlwriter\src\test\resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ hbase20xsqlwriter ---
[INFO] No sources to compile
[INFO] 
[INFO] --- maven-surefire-plugin:2.12.4:test (default-test) @ hbase20xsqlwriter ---
[INFO] Tests are skipped.
[INFO] 
[INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ hbase20xsqlwriter ---
[INFO] 
[INFO] --- maven-assembly-plugin:2.2-beta-5:single (dwzip) @ hbase20xsqlwriter ---
[INFO] Reading assembly descriptor: src/main/assembly/package.xml
[INFO] Copying files to D:\work\DataX\hbase20xsqlwriter\target\datax
[WARNING] Assembly file: D:\work\DataX\hbase20xsqlwriter\target\datax is not a regular file (it may be a directory). It cannot be attached to the project build for installation or deployment.
[INFO] 
[INFO] <<< maven-assembly-plugin:2.2-beta-5:assembly (default-cli) < package @ datax-all <<<
[INFO] 
[INFO] 
[INFO] --- maven-assembly-plugin:2.2-beta-5:assembly (default-cli) @ datax-all ---
[INFO] Reading assembly descriptor: package.xml
[INFO] datax/lib\commons-io-2.4.jar already added, skipping
[INFO] datax/lib\commons-lang3-3.3.2.jar already added, skipping
[INFO] datax/lib\commons-math3-3.1.1.jar already added, skipping
[INFO] datax/lib\datax-common-0.0.1-SNAPSHOT.jar already added, skipping
[INFO] datax/lib\datax-transformer-0.0.1-SNAPSHOT.jar already added, skipping
[INFO] datax/lib\fastjson-1.1.46.sec01.jar already added, skipping
[INFO] datax/lib\hamcrest-core-1.3.jar already added, skipping
[INFO] datax/lib\logback-classic-1.0.13.jar already added, skipping
[INFO] datax/lib\logback-core-1.0.13.jar already added, skipping
[INFO] datax/lib\slf4j-api-1.7.10.jar already added, skipping
[INFO] Building tar : D:\work\DataX\target\datax.tar.gz
[INFO] datax/lib\commons-io-2.4.jar already added, skipping
[INFO] datax/lib\commons-lang3-3.3.2.jar already added, skipping
[INFO] datax/lib\commons-math3-3.1.1.jar already added, skipping
[INFO] datax/lib\datax-common-0.0.1-SNAPSHOT.jar already added, skipping
[INFO] datax/lib\datax-transformer-0.0.1-SNAPSHOT.jar already added, skipping
[INFO] datax/lib\fastjson-1.1.46.sec01.jar already added, skipping
[INFO] datax/lib\hamcrest-core-1.3.jar already added, skipping
[INFO] datax/lib\logback-classic-1.0.13.jar already added, skipping
[INFO] datax/lib\logback-core-1.0.13.jar already added, skipping
[INFO] datax/lib\slf4j-api-1.7.10.jar already added, skipping
[INFO] datax/lib\commons-io-2.4.jar already added, skipping
[INFO] datax/lib\commons-lang3-3.3.2.jar already added, skipping
[INFO] datax/lib\commons-math3-3.1.1.jar already added, skipping
[INFO] datax/lib\datax-common-0.0.1-SNAPSHOT.jar already added, skipping
[INFO] datax/lib\datax-transformer-0.0.1-SNAPSHOT.jar already added, skipping
[INFO] datax/lib\fastjson-1.1.46.sec01.jar already added, skipping
[INFO] datax/lib\hamcrest-core-1.3.jar already added, skipping
[INFO] datax/lib\logback-classic-1.0.13.jar already added, skipping
[INFO] datax/lib\logback-core-1.0.13.jar already added, skipping
[INFO] datax/lib\slf4j-api-1.7.10.jar already added, skipping
[INFO] Copying files to D:\work\DataX\target\datax
[INFO] datax/lib\commons-io-2.4.jar already added, skipping
[INFO] datax/lib\commons-lang3-3.3.2.jar already added, skipping
[INFO] datax/lib\commons-math3-3.1.1.jar already added, skipping
[INFO] datax/lib\datax-common-0.0.1-SNAPSHOT.jar already added, skipping
[INFO] datax/lib\datax-transformer-0.0.1-SNAPSHOT.jar already added, skipping
[INFO] datax/lib\fastjson-1.1.46.sec01.jar already added, skipping
[INFO] datax/lib\hamcrest-core-1.3.jar already added, skipping
[INFO] datax/lib\logback-classic-1.0.13.jar already added, skipping
[INFO] datax/lib\logback-core-1.0.13.jar already added, skipping
[INFO] datax/lib\slf4j-api-1.7.10.jar already added, skipping
[WARNING] Assembly file: D:\work\DataX\target\datax is not a regular file (it may be a directory). It cannot be attached to the project build for installation or deployment.
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for datax-all 0.0.1-SNAPSHOT:
[INFO] 
[INFO] datax-all .......................................... SUCCESS [01:36 min]
[INFO] datax-common ....................................... SUCCESS [  1.939 s]
[INFO] datax-transformer .................................. SUCCESS [  2.208 s]
[INFO] datax-core ......................................... SUCCESS [  5.849 s]
[INFO] plugin-unstructured-storage-util ................... SUCCESS [  1.479 s]
[INFO] mongodbreader ...................................... SUCCESS [  9.187 s]
[INFO] txtfilewriter ...................................... SUCCESS [  8.526 s]
[INFO] hdfswriter ......................................... SUCCESS [ 32.463 s]
[INFO] streamwriter ....................................... SUCCESS [  1.248 s]
[INFO] plugin-rdbms-util .................................. SUCCESS [  0.907 s]
[INFO] hbase20xsqlreader .................................. SUCCESS [  2.312 s]
[INFO] hbase20xsqlwriter .................................. SUCCESS [  1.865 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  02:45 min
[INFO] Finished at: 2020-08-19T15:00:06+08:00
[INFO] ------------------------------------------------------------------------

可能会有报红，不用管即可
找到[WARNING] Assembly file: D:\work\DataX\target\datax is not a regular file (it may be a directory). It cannot be attached to the project build for installation or deployment.
该目录就是System.setProperty(“datax.home”, “D:\work\DataX\target\datax\datax”);中设置的路径
接下来就可以进行自己代码的开发了

二、项目开发

先说下我的需求：由于mongodbreader读取时Document和Arraay类型没有识别，而是直接存入的obj的输出字符串，这个类型我们的数据库不能识别。所以需要特殊转换，目标转换成json类型。

我扩展的是mongodbreader项目，由于图简单，没有在其基础上进行扩展，而是直接简单粗暴的修改源码，我需要将未识别的参数类型Document、List类型转换成josn字符串类型。

分析
在这里插入图片描述
目录清晰，直接修改MongoDBReader类即可
MongoDBReader源码

package com.alibaba.datax.plugin.reader.mongodbreader;

import java.util.ArrayList;
import java.util.Arrays;
import java.util.Date;
import java.util.Iterator;
import java.util.List;

import com.alibaba.datax.common.element.BoolColumn;
import com.alibaba.datax.common.element.DateColumn;
import com.alibaba.datax.common.element.DoubleColumn;
import com.alibaba.datax.common.element.LongColumn;
import com.alibaba.datax.common.element.Record;
import com.alibaba.datax.common.element.StringColumn;
import com.alibaba.datax.common.exception.DataXException;
import com.alibaba.datax.common.plugin.RecordSender;
import com.alibaba.datax.common.spi.Reader;
import com.alibaba.datax.common.util.Configuration;
import com.alibaba.datax.plugin.reader.mongodbreader.util.CollectionSplitUtil;
import com.alibaba.datax.plugin.reader.mongodbreader.util.MongoUtil;
import com.alibaba.fastjson.JSON;
import com.alibaba.fastjson.JSONArray;
import com.alibaba.fastjson.JSONObject;

import com.google.common.base.Joiner;
import com.google.common.base.Strings;
import com.mongodb.MongoClient;
import com.mongodb.client.MongoCollection;
import com.mongodb.client.MongoCursor;
import com.mongodb.client.MongoDatabase;
import org.bson.Document;
import org.bson.types.ObjectId;

/**
 * Created by jianying.wcj on 2015/3/19 0019.
 * Modified by mingyan.zc on 2016/6/13.
 * Modified by mingyan.zc on 2017/7/5.
 */
public class MongoDBReader extends Reader {

    public static class Job extends Reader.Job {

        private Configuration originalConfig = null;

        private MongoClient mongoClient;

        private String userName = null;
        private String password = null;

        @Override
        public List<Configuration> split(int adviceNumber) {
            return CollectionSplitUtil.doSplit(originalConfig,adviceNumber,mongoClient);
        }

        @Override
        public void init() {
            this.originalConfig = super.getPluginJobConf();
            this.userName = originalConfig.getString(KeyConstant.MONGO_USER_NAME, originalConfig.getString(KeyConstant.MONGO_USERNAME));
            this.password = originalConfig.getString(KeyConstant.MONGO_USER_PASSWORD, originalConfig.getString(KeyConstant.MONGO_PASSWORD));
            String database =  originalConfig.getString(KeyConstant.MONGO_DB_NAME, originalConfig.getString(KeyConstant.MONGO_DATABASE));
            String authDb =  originalConfig.getString(KeyConstant.MONGO_AUTHDB, database);
            if(!Strings.isNullOrEmpty(this.userName) && !Strings.isNullOrEmpty(this.password)) {
                this.mongoClient = MongoUtil.initCredentialMongoClient(originalConfig,userName,password,authDb);
            } else {
                this.mongoClient = MongoUtil.initMongoClient(originalConfig);
            }
        }

        @Override
        public void destroy() {

        }
    }


    public static class Task extends Reader.Task {

        private Configuration readerSliceConfig;

        private MongoClient mongoClient;

        private String userName = null;
        private String password = null;

        private String authDb = null;
        private String database = null;
        private String collection = null;

        private String query = null;

        private JSONArray mongodbColumnMeta = null;
        private Object lowerBound = null;
        private Object upperBound = null;
        private boolean isObjectId = true;

        @Override
        public void startRead(RecordSender recordSender) {

            if(lowerBound== null || upperBound == null ||
                mongoClient == null || database == null ||
                collection == null  || mongodbColumnMeta == null) {
                throw DataXException.asDataXException(MongoDBReaderErrorCode.ILLEGAL_VALUE,
                    MongoDBReaderErrorCode.ILLEGAL_VALUE.getDescription());
            }
            MongoDatabase db = mongoClient.getDatabase(database);
            MongoCollection col = db.getCollection(this.collection);

            MongoCursor<Document> dbCursor = null;
            Document filter = new Document();
            if (lowerBound.equals("min")) {
                if (!upperBound.equals("max")) {
                    filter.append(KeyConstant.MONGO_PRIMARY_ID, new Document("$lt", isObjectId ? new ObjectId(upperBound.toString()) : upperBound));
                }
            } else if (upperBound.equals("max")) {
                filter.append(KeyConstant.MONGO_PRIMARY_ID, new Document("$gte", isObjectId ? new ObjectId(lowerBound.toString()) : lowerBound));
            } else {
                filter.append(KeyConstant.MONGO_PRIMARY_ID, new Document("$gte", isObjectId ? new ObjectId(lowerBound.toString()) : lowerBound).append("$lt", isObjectId ? new ObjectId(upperBound.toString()) : upperBound));
            }
            if(!Strings.isNullOrEmpty(query)) {
                Document queryFilter = Document.parse(query);
                filter = new Document("$and", Arrays.asList(filter, queryFilter));
            }
            dbCursor = col.find(filter).iterator();
            while (dbCursor.hasNext()) {
                Document item = dbCursor.next();
                Record record = recordSender.createRecord();
                Iterator columnItera = mongodbColumnMeta.iterator();
                while (columnItera.hasNext()) {
                    JSONObject column = (JSONObject)columnItera.next();
                    Object tempCol = item.get(column.getString(KeyConstant.COLUMN_NAME));
                    if (tempCol == null) {
                        if (KeyConstant.isDocumentType(column.getString(KeyConstant.COLUMN_TYPE))) {
                            String[] name = column.getString(KeyConstant.COLUMN_NAME).split("\\.");
                            if (name.length > 1) {
                                Object obj;
                                Document nestedDocument = item;
                                for (String str : name) {
                                    obj = nestedDocument.get(str);
                                    if (obj instanceof Document) {
                                        nestedDocument = (Document) obj;
                                    }
                                }

                                if (null != nestedDocument) {
                                    Document doc = nestedDocument;
                                    tempCol = doc.get(name[name.length - 1]);
                                }
                            }
                        }
                    }
                    if (tempCol == null) {
                        //continue; 这个不能直接continue会导致record到目的端错位
                        record.addColumn(new StringColumn(null));
                    }else if (tempCol instanceof Double) {
                        //TODO deal with Double.isNaN()
                        record.addColumn(new DoubleColumn((Double) tempCol));
                    } else if (tempCol instanceof Boolean) {
                        record.addColumn(new BoolColumn((Boolean) tempCol));
                    } else if (tempCol instanceof Date) {
                        record.addColumn(new DateColumn((Date) tempCol));
                    } else if (tempCol instanceof Integer) {
                        record.addColumn(new LongColumn((Integer) tempCol));
                    }else if (tempCol instanceof Long) {
                        record.addColumn(new LongColumn((Long) tempCol));
                    } else {
                        if(KeyConstant.isArrayType(column.getString(KeyConstant.COLUMN_TYPE))) {
                            String splitter = column.getString(KeyConstant.COLUMN_SPLITTER);
                            if(Strings.isNullOrEmpty(splitter)) {
                                throw DataXException.asDataXException(MongoDBReaderErrorCode.ILLEGAL_VALUE,
                                    MongoDBReaderErrorCode.ILLEGAL_VALUE.getDescription());
                            } else {
                                ArrayList array = (ArrayList)tempCol;
                                String tempArrayStr = Joiner.on(splitter).join(array);
                                record.addColumn(new StringColumn(tempArrayStr));
                            }
                        } else {
                            record.addColumn(new StringColumn(tempCol.toString()));
                        }
                    }
                }
                recordSender.sendToWriter(record);
            }
        }

        @Override
        public void init() {
            this.readerSliceConfig = super.getPluginJobConf();
            this.userName = readerSliceConfig.getString(KeyConstant.MONGO_USER_NAME, readerSliceConfig.getString(KeyConstant.MONGO_USERNAME));
            this.password = readerSliceConfig.getString(KeyConstant.MONGO_USER_PASSWORD, readerSliceConfig.getString(KeyConstant.MONGO_PASSWORD));
            this.database = readerSliceConfig.getString(KeyConstant.MONGO_DB_NAME, readerSliceConfig.getString(KeyConstant.MONGO_DATABASE));
            this.authDb = readerSliceConfig.getString(KeyConstant.MONGO_AUTHDB, this.database);
            if(!Strings.isNullOrEmpty(userName) && !Strings.isNullOrEmpty(password)) {
                mongoClient = MongoUtil.initCredentialMongoClient(readerSliceConfig,userName,password,authDb);
            } else {
                mongoClient = MongoUtil.initMongoClient(readerSliceConfig);
            }

            this.collection = readerSliceConfig.getString(KeyConstant.MONGO_COLLECTION_NAME);
            this.query = readerSliceConfig.getString(KeyConstant.MONGO_QUERY);
            this.mongodbColumnMeta = JSON.parseArray(readerSliceConfig.getString(KeyConstant.MONGO_COLUMN));
            this.lowerBound = readerSliceConfig.get(KeyConstant.LOWER_BOUND);
            this.upperBound = readerSliceConfig.get(KeyConstant.UPPER_BOUND);
            this.isObjectId = readerSliceConfig.getBool(KeyConstant.IS_OBJECTID);
        }

        @Override
        public void destroy() {

        }

    }
}

通过上面代码找到类型判断逻辑如下

                    ...
                    else if (tempCol instanceof Double) {
                        //TODO deal with Double.isNaN()
                        record.addColumn(new DoubleColumn((Double) tempCol));
                    } else if (tempCol instanceof Boolean) {
                        record.addColumn(new BoolColumn((Boolean) tempCol));
                    }...

增加一下逻辑

                   else if (tempCol instanceof Document || tempCol instanceof List) {
                        record.addColumn(new StringColumn(JSONObject.toJSONString(tempCol)));
                    }

重新打包发布：
由于不会部分插件方式打包，所以只能先打包mongodbreader然后将其
打包mongodbreader模块：
mvn clean package -pl mongodbreader -DskipTests assembly:assembly
如果其他模块，直接将mongodbreader配置成对应的模块名（pom.xml中 <module></module>内名称）即可。

谢谢，阅读！

datax到hive数据全部为空_datax二次开发
weixin_39670267的博客
12-20 826
一、datax插件hbase12xwriter开发查看datax源码，发现有hbase094xwriter和hbase11xwriter源码，说明datax支持hbase写入，再查看测试和生产环境使用的hbase版本是：hbase-1.2.0-cdh5.8.4自己写一个hbase12xwriter插件包开发流程：1、搭建项目模块moduledatax-all项目上右击->New->ot...
本地虚拟机从零搭建部署datax-web
Stone.鱼儿的博客
12-30 2862
码云项目地址：datax-web: DataX集成可视化页面，选择数据源即可一键生成数据同步任务，支持批量创建RDBMS数据同步任务，集成开源调度系统，支持分布式、增量同步数据、实时查看运行日志、监控执行器资源、KILL运行进程、数据源信息加密等。 通过项目中的安装部署说明书：遇到很多坑，接下来我从头详细的记录我是如何搭建的和所遇到的坑解决方案。 datax-web部署方案 一、依赖资源 1、datax：datax-web的核心资源包 2、jdk 3、Maven（编译工具） 4、python（推荐.
【maven打包失败】
weixin_37701609的博客
10-28 581
maven打包出现错误
datax源码在线下载
12-17
datax源码在线下载datax源码在线下载datax源码在线下载
如何使用 Postman 发送和测试 WebSocket
最新发布
gdjnrc_com的博客
04-23 2235
WebSocket 促进客户端和服务器之间通过单个持久连接进行实时、双向通信。 需要使用 Postman 建立 WebSocket 连接吗？ 请查看我们简洁的循序渐进指南！
DataX源码...
04-13
Datax源码；异构数据源同步；大数据；alibaba开源框架
DataX插件开发指南
11-03
淘宝： DataX插件开发指南 说明了。datax插件开发的方法，用java语言实现
Datax
lz13419607299的博客
07-07 743
datax
datax 源码 data-x-master
08-28
datax 源码，可以直接修改读取写出源码，并编译
电商日志项目(一)
The Fifth Leaf 的博客
04-30 1593
浏览器信息分析以及地域信息分析其实就是在用户基本信息分析的基础上添加浏览器和地域这个维度信息，其中浏览器信息我们可以通过浏览器的window.navigator.userAgent来进行分析，地域信息可以通过nginx服务器来收集用户的ip地址来进行分析，也就是说pageview事件也可以满足这两个模块的分析。本身作为数据同步框架，将不同数据源的同步抽象为从源头数据源读取数据的Reader插件，以及向目标端写入数据的Writer插件，理论上DataX框架可以支持任意数据源类型的数据同步工作。
5.1数据采集通道搭建之业务数据采集通道搭建
Mmj666的博客
07-02 965
业务采集通道的搭建
datax Jar.zip
07-30
datax是很好的数据同步工具，但是网上大多数基于python指令来运行。本资源是基于ali的datax项目源码改编，直接打包成core和depend两个jar包。可以直接添加到java项目中,通过方法Engine.run("datax-plugin路径",".json配置文件地址")来执行datax迁移任务。
databus搭建文档记录
python_xep的博客
03-18 641
最近公司因需要同步oracle数据到mysql ，调研了Datax 对于大数据量的同步代价有些大。开源的databus 需要对源码做二次开发，才可以使用，前期我们搭建后，用自带的person表做了测试。确认可行后研发更改了源码。 准备工作： 1.配制gradle 和 java 2.ojdbc6-11.2.0.2.0.jar放到如下目录： databus-master/sandbox-repo/com/oracle/ojdbc6/11.2.0.2.0/ 更改defaultEnvironment.gradl
大数据技术之DataX （一）DataX插件开发
大数据指北
10-12 8295
文章目录一、背景二、基于java的本地测试datax2.1 github上下载datax的源代码2.2 datax代码导入idea三、docker安装南大通用数据库GBase和GBase 8a3.1 docker安装Gbase 8a3.2 docker安装Gbase 8s三、南大通用数据库GBase 8s To GBase 8a3.1 GBase 8s的reader读插件开发（writer同理）四、南大通用gbase可视化工具4.1 南大通用GBase 8a的连接配置4.2 南大通用GBase 8s的
DATAX源码初探
jibin6693923的博客
09-14 819
​DataX 是阿里开源的一个离线数据同步工具，目前在阿里巴巴集团内以及国内许多大厂都被广泛使用。DataX 实现了包括 MySQL、Oracle、SqlServer、Postgre、HDFS、Hive、HBase、TableStore(OTS)、MaxCompute(ODPS)、Hologres、DRDS 等各种异构数据源之间高效的数据同步功能。
Datax及Datax-web 下载使用
四维大脑的博客
08-01 2924
datax部署，datax连接mysql8，datax-web部署
DataX安装
qq_44005449的博客
07-16 2195
Linux服务器默认安装Python2.7，默认版本即可。
datax-web源码idea启动
wangzq2213的博客
02-27 770
datax-web 在idea源码运行 及问题解决
大数据学习——dataX工具部署和源码编译
qifengsunny的博客
03-28 2988
为了做大数据项目数据抽取工作，开始学习dataX，尝试比较dataX（版本 3.0）与sqoop的功能与性能差异。以下教程包括： 下载dataX工具及简要使用、下载dataX源码尝试编译及遇到的问题、尝试改造开源项目dataX 一、下载dataX工具及简要使用 gitHub下载（在下面的README里，Quick Start栏中）：https://github.com/alibaba/Dat...
DataX项目插件源码二次开发后，如何使用IntelliJ IDEA编译、打包、部署
06-12
如果您想要使用 IntelliJ IDEA 对 DataX 的插件源码进行二次开发，您可以按照以下步骤进行编译、打包和部署： 1. 打开 IntelliJ IDEA，选择 File -> Open，然后选择 DataX 的插件源码目录。 2. 在 IntelliJ IDEA 中打开 Maven Projects 窗口，然后双击插件的 pom.xml 文件。这将加载插件的所有依赖项。 3. 对插件进行修改和调试。 4. 执行以下命令进行编译打包： ``` mvn clean package assembly:assembly -DskipTests ``` 这将生成一个包含所有必需文件的 jar 包，并将其保存在 target 目录中。 5. 复制生成的 jar 包到 DataX 的插件目录中。您可以按照上一篇回答中的步骤将 jar 包复制到插件目录中。 6. 启动 DataX 并使用您修改的插件进行数据同步。 ``` python bin/datax.py job/job.json ``` 希望这些步骤能够帮助您成功使用 IntelliJ IDEA 对 DataX 的插件源码进行编译、打包和部署。
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值