DataX vs 腾讯云COS对象存储 -> StarRocks集群
本文将介绍使用DataX读出Cos的Orc文件往StarRocks里面写。
需求: 需要将腾讯云cos上84TB的数据, 同步到StarRocks某个大表。正常每个分区数据量20~30亿，600GB。
工具：DataX
插件：hdfsreader、starrockswriter
对象存储COS：非融合
- hdfsreader：https://cloud.tencent.com/document/product/436/43654
- starrockswriter：https://docs.mirrorship.cn/zh/docs/loading/DataX-starrocks-writer
DataX
这里我使用的datax版本是 DataX (DATAX-OPENSOURCE-3.0)
[svccnetlhs@HOST datax]<231211 17:17:11>$ tree bin/ conf/ bin/ ├── datax.py ├── dxprof.py └── perftrace.py conf/ ├── core.json └── logback.xml 0 directories, 5 files [svccnetlhs@HOST datax]<231211 17:18:52>$ /bin/python3 python3 python3.6 python3.6m [svccnetlhs@HOST datax]<231211 17:18:52>$ /bin/python3 bin/datax.py DataX (DATAX-OPENSOURCE-3.0), From Alibaba ! Copyright (C) 2010-2017, Alibaba Group. All Rights Reserved. Usage: datax.py [options] job-url-or-path Options: -h, --help show this help message and exit Product Env Options: Normal user use these options to set jvm parameters, job runtime mode etc. Make sure these options can be used in Product Env. -j <jvm parameters>, --jvm=<jvm parameters> Set jvm parameters if necessary. --jobid=<job unique id> Set job unique id when running by Distribute/Local Mode. -m <job runtime mode>, --mode=<job runtime mode> Set job runtime mode such as: standalone, local, distribute. Default mode is standalone. -p <parameter used in job config>, --params=<parameter used in job config> Set job parameter, eg: the source tableName you want to set it by command, then you can use like this: -p"-DtableName=your-table-name", if you have mutiple parameters: -p"-DtableName=your-table-name -DcolumnName=your-column-name".Note: you should config in you job tableName with ${tableName}. -r <parameter used in view job config[reader] template>, --reader=<parameter used in view job config[reader] template> View job config[reader] template, eg: mysqlreader,streamreader -w <parameter used in view job config[writer] template>, --writer=<parameter used in view job config[writer] template> View job config[writer] template, eg: mysqlwriter,streamwriter Develop/Debug Options: Developer use these options to trace more details of DataX. -d, --debug Set to remote debug mode. --loglevel=<log level> Set log level such as: debug, info, all etc. [svccnetlhs@HOST datax]<231211 17:19:06>$
DataX (HdfsReader) 插件
[svccnetlhs@HOST datax]<231211 17:23:29>$ ls bin conf job lib log log_perf plugin script tmp [svccnetlhs@HOST datax]<231211 17:23:29>$ [svccnetlhs@HOST datax]<231211 17:23:30>$ cd plugin/ [svccnetlhs@HOST plugin]<231211 17:23:32>$ ls reader writer [svccnetlhs@HOST plugin]<231211 17:23:32>$ cd reader/ [svccnetlhs@HOST reader]<231211 17:23:36>$ ls cassandrareader datahubreader ftpreader hbase094xreader hbase11xsqlreader hdfsreader loghubreader mysqlreader odpsreader oraclereader otsreader postgresqlreader sqlserverreader streamreader tsdbreader clickhousereader drdsreader gdbreader hbase11xreader hbase20xsqlreader kingbaseesreader mongodbreader oceanbasev10reader opentsdbreader ossreader otsstreamreader rdbmsreader starrocksreader tdenginereader txtfilereader [svccnetlhs@HOST reader]<231211 17:23:37>$ cd hdfsreader/ [svccnetlhs@HOST hdfsreader]<231211 17:23:39>$ ls hdfsreader-0.0.1-SNAPSHOT.jar libs plugin_job_template.json plugin.json [svccnetlhs@HOST hdfsreader]<231211 17:23:40>$ [svccnetlhs@HOST hdfsreader]<231211 17:23:42>$ pwd /home/svccnetlhs/chengken/starrocks/datax/plugin/reader/hdfsreader [svccnetlhs@HOST hdfsreader]<231211 17:23:43>$ [svccnetlhs@HOST hdfsreader]<231211 17:23:44>$ cd libs/ [svccnetlhs@HOST libs]<231211 17:23:54>$ ls activation-1.1.jar commons-beanutils-1.9.2.jar curator-recipes-2.7.1.jar hadoop-mapreduce-client-core-2.7.1.jar httpclient-4.1.2.jar jetty-util-6.1.26.jar parquet-hadoop-bundle-1.6.0rc3.jar aircompressor-0.3.jar commons-beanutils-core-1.8.0.jar datanucleus-api-jdo-3.2.6.jar hadoop-yarn-api-2.7.1.jar httpcore-4.1.2.jar jline-2.12.jar pentaho-aggdesigner-algorithm-5.1.5-jhyde.jar annotations-2.0.3.jar commons-cli-1.2.jar datanucleus-core-3.2.10.jar hadoop-yarn-common-2.7.1.jar jackson-core-asl-1.9.13.jar jpam-1.1.jar plugin-unstructured-storage-util-0.0.1-SNAPSHOT.jar ant-1.9.1.jar commons-codec-1.4.jar datanucleus-rdbms-3.2.9.jar hadoop-yarn-server-applicationhistoryservice-2.6.0.jar jackson-jaxrs-1.9.13.jar jsch-0.1.42.jar protobuf-java-2.5.0.jar ant-launcher-1.9.1.jar commons-collections-3.2.1.jar datax-common-0.0.1-SNAPSHOT.jar hadoop-yarn-server-common-2.6.0.jar jackson-mapper-asl-1.9.13.jar jsp-api-2.1.jar servlet-api-2.5.jar antlr-2.7.7.jar commons-compiler-2.7.6.jar derby-10.11.1.1.jar hadoop-yarn-server-resourcemanager-2.6.0.jar jackson-xc-1.9.13.jar jsr305-3.0.0.jar slf4j-api-1.7.10.jar antlr-runtime-3.4.jar commons-compress-1.4.1.jar eigenbase-properties-1.1.4.jar hadoop-yarn-server-web-proxy-2.6.0.jar janino-2.7.6.jar jta-1.1.jar slf4j-log4j12-1.7.10.jar aopalliance-1.0.jar commons-configuration-1.6.jar fastjson2-2.0.23.jar hamcrest-core-1.3.jar javacsv-2.0.jar leveldbjni-all-1.8.jar snappy-java-1.0.4.1.jar apache-curator-2.6.0.pom commons-daemon-1.0.13.jar geronimo-annotation_1.0_spec-1.1.1.jar hive-ant-1.1.1.jar javax.inject-1.jar libfb303-0.9.2.jar ST4-4.0.4.jar apacheds-i18n-2.0.0-M15.jar commons-dbcp-1.4.jar geronimo-jaspic_1.0_spec-1.0.jar hive-cli-1.1.1.jar java-xmlbuilder-0.4.jar libthrift-0.9.2.jar stax-api-1.0.1.jar apacheds-kerberos-codec-2.0.0-M15.jar commons-digester-1.8.jar geronimo-jta_1.1_spec-1.1.1.jar hive-common-1.1.1.jar jaxb-api-2.2.2.jar log4j-1.2.17.jar stax-api-1.0-2.jar apache-log4j-extras-1.2.17.jar commons-httpclient-3.1.jar groovy-all-2.1.6.jar hive-exec-1.1.1.jar jaxb-impl-2.2.3-1.jar log4j-api-2.17.1.jar stringtemplate-3.2.1.jar api-asn1-api-1.0.0-M20.jar commons-io-2.4.jar gson-2.2.4.jar hive-hcatalog-core-1.1.1.jar jdo-api-3.0.1.jar log4j-core-2.17.1.jar velocity-1.5.jar api-util-1.0.0-M20.jar commons-lang-2.6.jar guava-11.0.2.jar hive-metastore-1.1.1.jar jersey-client-1.9.jar logback-classic-1.0.13.jar xercesImpl-2.9.1.jar asm-3.1.jar commons-lang3-3.3.2.jar guice-3.0.jar hive-serde-1.1.1.jar jersey-core-1.9.jar logback-core-1.0.13.jar xml-apis-1.3.04.jar asm-commons-3.1.jar commons-logging-1.1.3.jar guice-servlet-3.0.jar hive-service-1.1.1.jar jersey-guice-1.9.jar lzo-core-1.0.5.jar xmlenc-0.52.jar asm-tree-3.1.jar commons-math3-3.1.1.jar hadoop-aliyun-2.7.2.jar hive-shims-0.20S-1.1.1.jar jersey-json-1.9.jar mail-1.4.1.jar xz-1.0.jar avro-1.7.4.jar commons-net-3.1.jar hadoop-annotations-2.7.1.jar hive-shims-0.23-1.1.1.jar jersey-server-1.9.jar netty-3.6.2.Final.jar zookeeper-3.4.6.jar bonecp-0.8.0.RELEASE.jar commons-pool-1.5.4.jar hadoop-auth-2.7.1.jar hive-shims-1.1.1.jar jets3t-0.9.0.jar netty-all-4.0.23.Final.jar calcite-avatica-1.0.0-incubating.jar cos_api-bundle-5.6.137.2.jar hadoop-common-2.7.1.jar hive-shims-common-1.1.1.jar jettison-1.1.jar opencsv-2.3.jar calcite-core-1.0.0-incubating.jar curator-client-2.7.1.jar hadoop-cos-3.1.0-8.3.2.jar hive-shims-scheduler-1.1.1.jar jetty-6.1.26.jar oro-2.0.8.jar calcite-linq4j-1.0.0-incubating.jar curator-framework-2.6.0.jar hadoop-hdfs-2.7.1.jar htrace-core-3.1.0-incubating.jar jetty-all-7.6.0.v20120127.jar paranamer-2.3.jar [svccnetlhs@HOST libs]<231211 17:23:55>$
DataX (StarRocksWriter) 插件
[svccnetlhs@HOST datax]<231211 17:25:07>$ ls bin conf job lib log log_perf plugin script tmp [svccnetlhs@HOST datax]<231211 17:25:08>$ cd plugin/ [svccnetlhs@HOST plugin]<231211 17:25:11>$ ls reader writer [svccnetlhs@HOST plugin]<231211 17:25:11>$ cd writer/ [svccnetlhs@HOST writer]<231211 17:25:13>$ ls adbpgwriter clickhousewriter doriswriter ftpwriter hbase11xsqlwriter hdfswriter kuduwriter mysqlwriter ocswriter oscarwriter postgresqlwriter sqlserverwriter tdenginewriter adswriter databendwriter drdswriter gdbwriter hbase11xwriter hologresjdbcwriter loghubwriter neo4jwriter odpswriter osswriter rdbmswriter starrockswriter tsdbwriter cassandrawriter datahubwriter elasticsearchwriter hbase094xwriter hbase20xsqlwriter kingbaseeswriter mongodbwriter oceanbasev10writer oraclewriter otswriter selectdbwriter streamwriter txtfilewriter [svccnetlhs@HOST writer]<231211 17:25:13>$ cd starrockswriter/ [svccnetlhs@HOST starrockswriter]<231211 17:25:15>$ ls libs plugin_job_template.json plugin.json starrockswriter-1.1.0.jar [svccnetlhs@HOST starrockswriter]<231211 17:25:16>$ ls libs/ commons-codec-1.9.jar commons-io-2.4.jar commons-logging-1.1.1.jar datax-common-0.0.1-SNAPSHOT.jar fastjson2-2.0.23.jar hamcrest-core-1.3.jar httpcore-4.4.6.jar logback-core-1.0.13.jar plugin-rdbms-util-0.0.1-SNAPSHOT.jar commons-collections-3.0.jar commons-lang3-3.3.2.jar commons-math3-3.1.1.jar druid-1.0.15.jar guava-r05.jar httpclient-4.5.3.jar logback-classic-1.0.13.jar mysql-connector-java-5.1.46.jar slf4j-api-1.7.10.jar [svccnetlhs@HOST starrockswriter]<231211 17:25:21>$
注: 两个datax插件在文件开头可以进行下载。
DataX JSON
众所周知，DataX的是基于数据抽取、数据转换和数据加载三个步骤来实现数据流的搬迁。
Datax设计理念：
Datax框架设计：
Datax工作流程：
连接JSON:
模板1：
{ “content”: [ { “reader”: { “name”: “hdfsreader”, “parameter”: { “column”: [ { /************************************/ “name”: “ts”, /************************************/ “type”: “string”, /************************************/ “value”: “2023-11-14” /************************************/ }, /************************************/ { /************************************/ “index”: 0, /************************************/ “name”: “local_id”, /************************************/ “type”: “string” /************************************/ }, /************************************/ { /****1.由于cos文件中没有ts这个字段***/ “index”: 1, /****这里我则使用value指定一个固定值*/ “name”: “encrypted_imei”, /****value=2023-11-14代表当前path****/ “type”: “string” /****的分区数据, ********************/ }, /****此值在脚本中属于动态传参********/ { /************************************/ “index”: 2, /****2.这里其他的字段使用了index*****/ “name”: “encrypted_idfa”, /****下标的形式取到每个字段的值******/ “type”: “string” /************************************/ }, /************************************/ { /************************************/ “index”: 3, /************************************/ “name”: “encrypted_mac”, /************************************/ “type”: “string” /************************************/ }, /************************************/ { /************************************/ “index”: 4, /************************************/ “name”: “encrypted_android_id”, /************************************/ “type”: “string” /************************************/ } /************************************/ ], “defaultFS”: “cosn: //桶名/”, “encoding”: “UTF-8”, “fieldDelimiter”: ",", “fileType”: “orc”, “hadoopConfig”: { “fs.cosn.impl”: “org.apache.hadoop.fs.CosFileSystem”, “fs.cosn.tmp.dir”: “本地临时路径（随便）”, “fs.cosn.userinfo.region”: “ap-guangzhou”, “fs.cosn.userinfo.secretId”: "", “fs.cosn.userinfo.secretKey”: "" }, “path”: "/sam/sam_dwd_user_action_cos_d/20231114/part-00011*" } }, “writer”: { “name”: “starrockswriter”, “parameter”: { “column”: [ “ts”, /******************************************/ “local_id”, /******************************************/ “encrypted_imei”, /****StarRocks需要接收的字段名*************/ “encrypted_idfa”, /******************************************/ “encrypted_mac”, /******************************************/ “encrypted_android_id” /******************************************/ ], “database”: “StarRocks库名”, “jdbcUrl”: “jdbc: mysql: //StarRocksFE_IP:9030/”, “loadProps”: { “max_filter_ratio”: 1 }, “loadUrl”: [ “StarRocksFE_IP:8030”, “StarRocksFE_IP:8030”, “StarRocksFE_IP:8030” ], “password”: “StarRocks密码”, “postSql”: [ ], “preSql”: [ ], “table”: “StarRocks表名”, “username”: “StarRocks用户” } } } ], “setting”: { “speed”: { “byte”: -1, /********channel调整为3，不限速**********/ “channel”: 3 /*********************************************/ } } }
模板2：
{ "job": { "setting": { "speed": { "channel":3 }, "errorLimit": {} }, "content": [{ "reader": { "name": "hdfsreader", "parameter": { "path": "/sam/sam_dwd_user_action_cos_d/20231114/part-*", "defaultFS": "cosn://*********/", "column": [ {"name":"ts","type":"string","value":"2023-11-14"}, {"name":"import_ds_","type":"string","index":0}, {"name":"unique_action_id","type":"string","index":1}, {"name":"action_time","type":"string","index":2}, {"name":"report_time","type":"string","index":3}, {"name":"action_type","type":"string","index":4}, {"name":"ka_id","type":"string","index":5}, {"name":"action_session_id","type":"string","index":6}, {"name":"uuid","type":"string","index":7}, {"name":"wx_app_id","type":"string","index":8}, {"name":"wx_open_id","type":"string","index":9}, {"name":"wx_union_id","type":"string","index":10}, {"name":"external_user_id","type":"string","index":11}, {"name":"merber_id","type":"string","index":12}, {"name":"local_id","type":"string","index":13}, {"name":"encrypted_imei","type":"string","index":14}, {"name":"encrypted_idfa","type":"string","index":15}, {"name":"encrypted_mac","type":"string","index":16}, {"name":"encrypted_android_id","type":"string","index":17}, {"name":"encrypted_qq","type":"string","index":18}, {"name":"encrypted_phone","type":"string","index":19}, {"name":"encrypting_algorithm","type":"string","index":20}, {"name":"chan_id","type":"string","index":21}, {"name":"chan_refer_app_id","type":"string","index":22}, {"name":"chan_shop_id","type":"string","index":23}, {"name":"chan_shop_name","type":"string","index":24}, {"name":"client_type","type":"string","index":25}, {"name":"client_name","type":"string","index":26}, {"name":"client_version","type":"string","index":27}, {"name":"sdk_version","type":"string","index":28}, {"name":"device_model","type":"string","index":29}, {"name":"ip","type":"string","index":30}, {"name":"user_agent","type":"string","index":31}, {"name":"page_path","type":"string","index":32}, {"name":"page_name","type":"string","index":33}, {"name":"referrer","type":"string","index":34}, {"name":"address","type":"string","index":35}, {"name":"city","type":"string","index":36}, {"name":"province","type":"string","index":37}, {"name":"country","type":"string","index":38}, {"name":"latitude","type":"string","index":39}, {"name":"longitude","type":"string","index":40}, {"name":"json_properties","type":"string","index":41}, {"name":"fdate","type":"string","index":42}, {"name":"tag_id","type":"string","index":43}, {"name":"tag_name","type":"string","index":44}, {"name":"chan_custom_id","type":"string","index":45}, {"name":"etl_load_time","type":"string","index":46}, {"name":"event_name","type":"string","index":47} ], "fileType": "orc", "encoding": "UTF-8", "hadoopConfig": { "fs.cosn.impl": "org.apache.hadoop.fs.CosFileSystem", "fs.cosn.userinfo.region": "ap-guangzhou", "fs.cosn.tmp.dir": "/u/chengken/starrocks/sam/data", "fs.cosn.userinfo.secretId": "***************", "fs.cosn.userinfo.secretKey": "**************", "fs.cosn.read.ahead.block.size": 1048576, "fs.cosn.read.ahead.queue.size": 2 }, "fieldDelimiter": "," } }, "writer": { "name": "starrockswriter", "parameter": { "maxBatchRows":"5000000", "maxBatchSize":"5368709120", "username": "cndlopsns", "password": "lizhenghua1.", "database": "ods", "table": "buckets_tmp_20231205__sams_cos", "column": [ "ts", "import_ds_", "unique_action_id", "action_time", "report_time", "action_type", "ka_id", "action_session_id", "uuid", "wx_app_id", "wx_open_id", "wx_union_id", "external_user_id", "merber_id", "local_id", "encrypted_imei", "encrypted_idfa", "encrypted_mac", "encrypted_android_id", "encrypted_qq", "encrypted_phone", "encrypting_algorithm", "chan_id", "chan_refer_app_id", "chan_shop_id", "chan_shop_name", "client_type", "client_name", "client_version", "sdk_version", "device_model", "ip", "user_agent", "page_path", "page_name", "referrer", "address", "city", "province", "country", "latitude", "longitude", "json_properties", "fdate", "tag_id", "tag_name", "chan_custom_id", "etl_load_time", "event_name" ], "preSql": [], "postSql": [], "jdbcUrl": "jdbc:mysql://***********:9030/ods?useCursorFetch=true&tinyInt1isBit=false&query_timeout=36000&useUnicode=true&characterEncoding=utf8", "loadUrl": ["****1:8030","****2:8030","****3:8030"], "loadProps": {"max_filter_ratio":1} } } }] } }
启动
启动并顺利读到上游数据文件，然后异步写入StarRocks。
/usr/bin/python2.7 /home/hadoop/datax/bin/datax.py --jvm="-Xms1G -Xmx4G" /home/hadoop/datax/cos-starrocks1.json
Run: /usr/bin/python2.7 /u/chengken/datax/bin/datax.py --jvm="-Xms16g -Xmx16g" /home/svccnetlhs/chengken/starrocks/json/20231114_sam_dwd_user_action_cos__1702288181.json Output_____________ DataX (DATAX-OPENSOURCE-3.0), From Alibaba ! Copyright (C) 2010-2017, Alibaba Group. All Rights Reserved. 2023-12-11 17:49:54.918 [main] INFO MessageSource - JVM TimeZone: GMT+08:00, Locale: zh_CN 2023-12-11 17:49:54.920 [main] INFO MessageSource - use Locale: zh_CN timeZone: sun.util.calendar.ZoneInfo[id="GMT+08:00",offset=28800000,dstSavings=0,useDaylight=false,transitions=0,lastRule=null] 2023-12-11 17:49:54.945 [main] INFO VMInfo - VMInfo# operatingSystem class => sun.management.OperatingSystemImpl 2023-12-11 17:49:54.950 [main] INFO Engine - the machine info => osInfo: Linux amd64 4.18.0-348.7.1.el8_5.x86_64 jvmInfo: Oracle Corporation 1.8 25.112-b15 cpu num: 16 totalPhysicalMemory: -0.00G freePhysicalMemory: -0.00G maxFileDescriptorCount: -1 currentOpenFileDescriptorCount: -1 GC Names [PS MarkSweep, PS Scavenge] MEMORY_NAME | allocation_size | init_size PS Eden Space | 4,096.00MB | 4,096.00MB Code Cache | 240.00MB | 2.44MB Compressed Class Space | 1,024.00MB | 0.00MB PS Survivor Space | 682.50MB | 682.50MB PS Old Gen | 10,923.00MB | 10,923.00MB Metaspace | -0.00MB | 0.00MB 2023-12-11 17:49:54.966 [main] INFO Engine - { "setting":{ "speed":{ "channel":3 }, "errorLimit":{ } }, "content":[ { "reader":{ "name":"hdfsreader", "parameter":{ "path":"/sam/sam_dwd_user_action_cos_d/20231114/part-*", "defaultFS":"cosn://*********/", "column":[ { "name":"ts", "type":"string", "value":"2023-11-14" }, { "name":"import_ds_", "type":"string", "index":0 }, { "name":"unique_action_id", "type":"string", "index":1 }, { "name":"action_time", "type":"string", "index":2 }, { "name":"report_time", "type":"string", "index":3 }, { "name":"action_type", "type":"string", "index":4 }, { "name":"ka_id", "type":"string", "index":5 }, { "name":"action_session_id", "type":"string", "index":6 }, { "name":"uuid", "type":"string", "index":7 }, { "name":"wx_app_id", "type":"string", "index":8 }, { "name":"wx_open_id", "type":"string", "index":9 }, { "name":"wx_union_id", "type":"string", "index":10 }, { "name":"external_user_id", "type":"string", "index":11 }, { "name":"merber_id", "type":"string", "index":12 }, { "name":"local_id", "type":"string", "index":13 }, { "name":"encrypted_imei", "type":"string", "index":14 }, { "name":"encrypted_idfa", "type":"string", "index":15 }, { "name":"encrypted_mac", "type":"string", "index":16 }, { "name":"encrypted_android_id", "type":"string", "index":17 }, { "name":"encrypted_qq", "type":"string", "index":18 }, { "name":"encrypted_phone", "type":"string", "index":19 }, { "name":"encrypting_algorithm", "type":"string", "index":20 }, { "name":"chan_id", "type":"string", "index":21 }, { "name":"chan_refer_app_id", "type":"string", "index":22 }, { "name":"chan_shop_id", "type":"string", "index":23 }, { "name":"chan_shop_name", "type":"string", "index":24 }, { "name":"client_type", "type":"string", "index":25 }, { "name":"client_name", "type":"string", "index":26 }, { "name":"client_version", "type":"string", "index":27 }, { "name":"sdk_version", "type":"string", "index":28 }, { "name":"device_model", "type":"string", "index":29 }, { "name":"ip", "type":"string", "index":30 }, { "name":"user_agent", "type":"string", "index":31 }, { "name":"page_path", "type":"string", "index":32 }, { "name":"page_name", "type":"string", "index":33 }, { "name":"referrer", "type":"string", "index":34 }, { "name":"address", "type":"string", "index":35 }, { "name":"city", "type":"string", "index":36 }, { "name":"province", "type":"string", "index":37 }, { "name":"country", "type":"string", "index":38 }, { "name":"latitude", "type":"string", "index":39 }, { "name":"longitude", "type":"string", "index":40 }, { "name":"json_properties", "type":"string", "index":41 }, { "name":"fdate", "type":"string", "index":42 }, { "name":"tag_id", "type":"string", "index":43 }, { "name":"tag_name", "type":"string", "index":44 }, { "name":"chan_custom_id", "type":"string", "index":45 }, { "name":"etl_load_time", "type":"string", "index":46 }, { "name":"event_name", "type":"string", "index":47 } ], "fileType":"orc", "encoding":"UTF-8", "hadoopConfig":{ "fs.cosn.impl":"org.apache.hadoop.fs.CosFileSystem", "fs.cosn.userinfo.region":"ap-guangzhou", "fs.cosn.tmp.dir":"/u/chengken/starrocks/sam/data", "fs.cosn.userinfo.secretId":"**********", "fs.cosn.userinfo.secretKey":"****************", "fs.cosn.read.ahead.block.size":1048576, "fs.cosn.read.ahead.queue.size":2 }, "fieldDelimiter":"," } }, "writer":{ "name":"starrockswriter", "parameter":{ "maxBatchRows":"5000000", "maxBatchSize":"5368709120", "username":"cndlopsns", "password":"************", "database":"ods", "table":"buckets_tmp_20231205__sams_cos", "column":[ "ts", "import_ds_", "unique_action_id", "action_time", "report_time", "action_type", "ka_id", "action_session_id", "uuid", "wx_app_id", "wx_open_id", "wx_union_id", "external_user_id", "merber_id", "local_id", "encrypted_imei", "encrypted_idfa", "encrypted_mac", "encrypted_android_id", "encrypted_qq", "encrypted_phone", "encrypting_algorithm", "chan_id", "chan_refer_app_id", "chan_shop_id", "chan_shop_name", "client_type", "client_name", "client_version", "sdk_version", "device_model", "ip", "user_agent", "page_path", "page_name", "referrer", "address", "city", "province", "country", "latitude", "longitude", "json_properties", "fdate", "tag_id", "tag_name", "chan_custom_id", "etl_load_time", "event_name" ], "preSql":[ ], "postSql":[ ], "jdbcUrl":"jdbc:mysql://192.168.1.121:9030/ods?useCursorFetch=true&tinyInt1isBit=false&query_timeout=36000&useUnicode=true&characterEncoding=utf8", "loadUrl":[ "192.168.1.121:8030", "192.168.1.122:8030", "192.168.1.123:8030" ], "loadProps":{ "max_filter_ratio":1 } } } } ] } 2023-12-11 17:49:54.983 [main] INFO PerfTrace - PerfTrace traceId=job_-1, isEnable=true 2023-12-11 17:49:54.983 [main] INFO JobContainer - DataX jobContainer starts job. 2023-12-11 17:49:54.984 [main] INFO JobContainer - Set jobId = 0 2023-12-11 17:49:54.994 [job-0] INFO HdfsReader$Job - init() begin... 2023-12-11 17:49:55.250 [job-0] INFO HdfsReader$Job - hadoopConfig details:{"classLoader":{"URLs":["file:/u/chengken/datax/plugin/reader/hdfsreader/hdfsreader-0.0.1-SNAPSHOT.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jaxb-impl-2.2.3-1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-hcatalog-core-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-auth-2.7.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/calcite-linq4j-1.0.0-incubating.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-metastore-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jta-1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/activation-1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/aircompressor-0.3.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/aopalliance-1.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-configuration-1.6.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-yarn-server-web-proxy-2.6.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/guice-servlet-3.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-cli-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/calcite-avatica-1.0.0-incubating.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/xz-1.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/parquet-hadoop-bundle-1.6.0rc3.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jetty-all-7.6.0.v20120127.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-httpclient-3.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/stax-api-1.0-2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/apacheds-i18n-2.0.0-M15.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/log4j-api-2.17.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/mail-1.4.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jaxb-api-2.2.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-annotations-2.7.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/geronimo-jta_1.1_spec-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/libthrift-0.9.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/oro-2.0.8.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-hdfs-2.7.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jackson-jaxrs-1.9.13.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/xercesImpl-2.9.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-logging-1.1.3.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-serde-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-shims-common-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/curator-recipes-2.7.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/opencsv-2.3.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jersey-json-1.9.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-service-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/ant-1.9.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/avro-1.7.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-shims-0.20S-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/log4j-1.2.17.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/httpclient-4.1.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-cli-1.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/xml-apis-1.3.04.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-digester-1.8.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/servlet-api-2.5.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/lzo-core-1.0.5.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/log4j-core-2.17.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/javacsv-2.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/protobuf-java-2.5.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-yarn-server-resourcemanager-2.6.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-yarn-server-common-2.6.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-lang3-3.3.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/xmlenc-0.52.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jackson-xc-1.9.13.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-ant-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/pentaho-aggdesigner-algorithm-5.1.5-jhyde.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/apache-log4j-extras-1.2.17.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jetty-util-6.1.26.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-shims-0.23-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/guava-11.0.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/logback-core-1.0.13.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-shims-scheduler-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-aliyun-2.7.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-dbcp-1.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-compress-1.4.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-yarn-common-2.7.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-shims-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-io-2.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/httpcore-4.1.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-mapreduce-client-core-2.7.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jackson-core-asl-1.9.13.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jersey-core-1.9.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/janino-2.7.6.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jdo-api-3.0.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jsp-api-2.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/annotations-2.0.3.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/velocity-1.5.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-codec-1.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-yarn-server-applicationhistoryservice-2.6.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/netty-3.6.2.Final.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/curator-framework-2.6.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/asm-tree-3.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jetty-6.1.26.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/calcite-core-1.0.0-incubating.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/plugin-unstructured-storage-util-0.0.1-SNAPSHOT.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/datax-common-0.0.1-SNAPSHOT.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-beanutils-core-1.8.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/groovy-all-2.1.6.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-yarn-api-2.7.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jersey-server-1.9.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/datanucleus-api-jdo-3.2.6.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/guice-3.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/datanucleus-rdbms-3.2.9.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/java-xmlbuilder-0.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-common-2.7.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jersey-client-1.9.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jersey-guice-1.9.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/libfb303-0.9.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/paranamer-2.3.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/antlr-runtime-3.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/zookeeper-3.4.6.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/geronimo-annotation_1.0_spec-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/ST4-4.0.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jettison-1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/datanucleus-core-3.2.10.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/api-asn1-api-1.0.0-M20.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/geronimo-jaspic_1.0_spec-1.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/apacheds-kerberos-codec-2.0.0-M15.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hamcrest-core-1.3.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/asm-commons-3.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/ant-launcher-1.9.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-exec-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-collections-3.2.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/asm-3.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/stringtemplate-3.2.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-common-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jline-2.12.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/api-util-1.0.0-M20.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/stax-api-1.0.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-net-3.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/bonecp-0.8.0.RELEASE.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/gson-2.2.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/slf4j-log4j12-1.7.10.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jets3t-0.9.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-lang-2.6.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/eigenbase-properties-1.1.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/antlr-2.7.7.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jsch-0.1.42.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/leveldbjni-all-1.8.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-beanutils-1.9.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/logback-classic-1.0.13.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jpam-1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/snappy-java-1.0.4.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jackson-mapper-asl-1.9.13.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-math3-3.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-pool-1.5.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/netty-all-4.0.23.Final.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-compiler-2.7.6.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-daemon-1.0.13.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/javax.inject-1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/htrace-core-3.1.0-incubating.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jsr305-3.0.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/slf4j-api-1.7.10.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/fastjson2-2.0.23.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/derby-10.11.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/curator-client-2.7.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/cos_api-bundle-5.6.137.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-cos-3.1.0-8.3.2.jar"],"parent":{"URLs":["file:/u/chengken/datax/lib/commons-lang3-3.3.2.jar","file:/u/chengken/datax/lib/logback-core-1.0.13.jar","file:/u/chengken/datax/lib/commons-io-2.4.jar","file:/u/chengken/datax/lib/datax-common-0.0.1-SNAPSHOT.jar","file:/u/chengken/datax/lib/hamcrest-core-1.3.jar","file:/u/chengken/datax/lib/datax-transformer-0.0.1-SNAPSHOT.jar","file:/u/chengken/datax/lib/logback-classic-1.0.13.jar","file:/u/chengken/datax/lib/commons-math3-3.1.1.jar","file:/u/chengken/datax/lib/slf4j-api-1.7.10.jar","file:/u/chengken/datax/lib/fastjson2-2.0.23.jar","file:/u/chengken/datax/lib/commons-logging-1.1.1.jar","file:/u/chengken/datax/lib/httpcore-4.4.13.jar","file:/u/chengken/datax/lib/commons-configuration-1.10.jar","file:/u/chengken/datax/lib/fluent-hc-4.5.jar","file:/u/chengken/datax/lib/commons-cli-1.2.jar","file:/u/chengken/datax/lib/janino-2.5.16.jar","file:/u/chengken/datax/lib/groovy-all-2.1.9.jar","file:/u/chengken/datax/lib/commons-codec-1.11.jar","file:/u/chengken/datax/lib/commons-collections-3.2.1.jar","file:/u/chengken/datax/lib/commons-lang-2.6.jar","file:/u/chengken/datax/lib/httpclient-4.5.13.jar","file:/u/chengken/datax/lib/commons-beanutils-1.9.2.jar","file:/u/chengken/datax/lib/datax-core-0.0.1-SNAPSHOT.jar","file:/home/svccnetlhs/chengken/starrocks/"],"parent":{"URLs":["file:/opt/software/jdk1.8.0_112/jre/lib/ext/jaccess.jar","file:/opt/software/jdk1.8.0_112/jre/lib/ext/sunjce_provider.jar","file:/opt/software/jdk1.8.0_112/jre/lib/ext/sunpkcs11.jar","file:/opt/software/jdk1.8.0_112/jre/lib/ext/localedata.jar","file:/opt/software/jdk1.8.0_112/jre/lib/ext/nashorn.jar","file:/opt/software/jdk1.8.0_112/jre/lib/ext/sunec.jar","file:/opt/software/jdk1.8.0_112/jre/lib/ext/dnsns.jar","file:/opt/software/jdk1.8.0_112/jre/lib/ext/zipfs.jar","file:/opt/software/jdk1.8.0_112/jre/lib/ext/cldrdata.jar","file:/opt/software/jdk1.8.0_112/jre/lib/ext/jfxrt.jar"]}}},"finalParameters":[]} 2023-12-11 17:49:55.250 [job-0] INFO HdfsReader$Job - init() ok and end... 2023-12-11 17:49:55.279 [job-0] INFO JobContainer - jobContainer starts to do prepare ... 2023-12-11 17:49:55.279 [job-0] INFO JobContainer - DataX Reader.Job [hdfsreader] do prepare work . 2023-12-11 17:49:55.279 [job-0] INFO HdfsReader$Job - prepare(), start to getAllFiles... 2023-12-11 17:49:55.279 [job-0] INFO HdfsReader$Job - get HDFS all files in path = [/sam/sam_dwd_user_action_cos_d/20231114/part-*] 十二月 11, 2023 5:49:55 下午 org.apache.hadoop.util.NativeCodeLoader <clinit> 警告: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2023-12-11 17:49:55.527 [job-0] INFO RangerCredentialsClient - begin to init ranger client, impl [] 2023-12-11 17:49:55.776 [job-0] INFO CosNativeFileSystemStore - hadoop cos retry times: 200, cos client retry times: 5 log4j:WARN No appenders could be found for logger (com.qcloud.cos.thirdparty.org.apache.http.client.protocol.RequestAddCookies). log4j:WARN Please initialize the log4j system properly. log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info. 2023-12-11 17:49:56.400 [job-0] INFO CosFileSystem - The cos bucket is the normal bucket. 2023-12-11 17:49:56.418 [job-0] INFO BufferPool - Initialize the buffer pool. 2023-12-11 17:49:56.419 [job-0] INFO BufferPool - fs.cosn.upload.buffer.size is set to -1, so the 'mapped_disk' buffer will be used by default. 2023-12-11 17:49:56.419 [job-0] INFO BufferPool - The type of the upload buffer pool is [MAPPED_DISK]. Buffer size:[-1] 2023-12-11 17:49:56.419 [job-0] INFO BufferPool - tmp dir list 2023-12-11 17:49:56.796 [job-0] INFO CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00000-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading 2023-12-11 17:49:56.953 [job-0] INFO HdfsReader$Job - [cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00000-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc]是[orc]类型的文件, 将该文件加入source files列表 2023-12-11 17:49:57.014 [job-0] INFO CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00001-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading 2023-12-11 17:49:57.192 [job-0] INFO HdfsReader$Job - [cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00001-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc]是[orc]类型的文件, 将该文件加入source files列表 2023-12-11 17:49:57.244 [job-0] INFO CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00002-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading 2023-12-11 17:49:57.392 [job-0] INFO HdfsReader$Job - [cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00002-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc]是[orc]类型的文件, 将该文件加入source files列表 2023-12-11 17:49:57.443 [job-0] INFO CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00003-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading 2023-12-11 17:49:57.586 [job-0] INFO HdfsReader$Job - [cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00003-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc]是[orc]类型的文件, 将该文件加入source files列表 2023-12-11 17:49:57.645 [job-0] INFO CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00004-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading 2023-12-11 17:49:57.782 [job-0] INFO HdfsReader$Job - [cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00004-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc]是[orc]类型的文件, 将该文件加入source files列表 2023-12-11 17:49:57.837 [job-0] INFO CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00005-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading 2023-12-11 17:49:58.007 [job-0] INFO HdfsReader$Job - [cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00005-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc]是[orc]类型的文件, 将该文件加入source files列表 2023-12-11 17:49:58.066 [job-0] INFO CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00006-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading 2023-12-11 17:49:58.233 [job-0] INFO HdfsReader$Job - [cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00006-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc]是[orc]类型的文件, 将该文件加入source files列表 2023-12-11 17:49:58.292 [job-0] INFO CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00007-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading 2023-12-11 17:49:58.444 [job-0] INFO HdfsReader$Job - [cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00007-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc]是[orc]类型的文件, 将该文件加入source files列表 2023-12-11 17:49:58.510 [job-0] INFO CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00008-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading 2023-12-11 17:49:58.694 [job-0] INFO HdfsReader$Job - [cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00008-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc]是[orc]类型的文件, 将该文件加入source files列表 2023-12-11 17:49:58.749 [job-0] INFO CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00009-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading 2023-12-11 17:49:58.975 [job-0] INFO HdfsReader$Job - [cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00009-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc]是[orc]类型的文件, 将该文件加入source files列表 2023-12-11 17:49:59.030 [job-0] INFO CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00010-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading 2023-12-11 17:49:59.181 [job-0] INFO HdfsReader$Job - [cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00010-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc]是[orc]类型的文件, 将该文件加入source files列表 2023-12-11 17:49:59.232 [job-0] INFO CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00011-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading 2023-12-11 17:49:59.393 [job-0] INFO HdfsReader$Job - [cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00011-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc]是[orc]类型的文件, 将该文件加入source files列表 2023-12-11 17:49:59.449 [job-0] INFO CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00012-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading 2023-12-11 17:49:59.602 [job-0] INFO HdfsReader$Job - [cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00012-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc]是[orc]类型的文件, 将该文件加入source files列表 2023-12-11 17:49:59.656 [job-0] INFO CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00013-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading 2023-12-11 17:49:59.836 [job-0] INFO HdfsReader$Job - [cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00013-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc]是[orc]类型的文件, 将该文件加入source files列表 2023-12-11 17:49:59.888 [job-0] INFO CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00014-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading 2023-12-11 17:50:00.038 [job-0] INFO HdfsReader$Job - [cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00014-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc]是[orc]类型的文件, 将该文件加入source files列表 2023-12-11 17:50:00.106 [job-0] INFO CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00015-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading 2023-12-11 17:50:00.325 [job-0] INFO HdfsReader$Job - [cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00015-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc]是[orc]类型的文件, 将该文件加入source files列表 2023-12-11 17:50:00.382 [job-0] INFO CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00016-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading 2023-12-11 17:50:00.551 [job-0] INFO HdfsReader$Job - [cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00016-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc]是[orc]类型的文件, 将该文件加入source files列表 2023-12-11 17:50:00.606 [job-0] INFO CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00017-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading 2023-12-11 17:50:00.769 [job-0] INFO HdfsReader$Job - [cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00017-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc]是[orc]类型的文件, 将该文件加入source files列表 2023-12-11 17:50:00.826 [job-0] INFO CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00018-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading 2023-12-11 17:50:00.969 [job-0] INFO HdfsReader$Job - [cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00018-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc]是[orc]类型的文件, 将该文件加入source files列表 2023-12-11 17:50:01.021 [job-0] INFO CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00019-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading 2023-12-11 17:50:01.165 [job-0] INFO HdfsReader$Job - [cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00019-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc]是[orc]类型的文件, 将该文件加入source files列表 2023-12-11 17:50:01.165 [job-0] INFO HdfsReader$Job - 您即将读取的文件数为: [20], 列表为: [cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00002-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc,cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00018-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc,cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00001-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc,cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00003-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc,cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00017-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc,cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00009-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc,cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00011-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc,cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00015-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc,cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00008-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc,cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00016-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc,cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00010-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc,cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00013-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc,cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00007-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc,cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00012-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc,cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00014-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc,cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00006-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc,cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00000-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc,cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00004-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc,cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00019-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc,cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00005-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc] 2023-12-11 17:50:01.166 [job-0] INFO JobContainer - DataX Writer.Job [starrockswriter] do prepare work . 2023-12-11 17:50:01.168 [job-0] INFO JobContainer - jobContainer starts to do split ... 2023-12-11 17:50:01.168 [job-0] INFO JobContainer - Job set Channel-Number to 3 channels. 2023-12-11 17:50:01.168 [job-0] INFO HdfsReader$Job - split() begin... 2023-12-11 17:50:01.173 [job-0] INFO JobContainer - DataX Reader.Job [hdfsreader] splits to [20] tasks. 2023-12-11 17:50:01.174 [job-0] INFO JobContainer - DataX Writer.Job [starrockswriter] splits to [20] tasks. 2023-12-11 17:50:01.200 [job-0] INFO JobContainer - jobContainer starts to do schedule ... 2023-12-11 17:50:01.213 [job-0] INFO JobContainer - Scheduler starts [1] taskGroups. 2023-12-11 17:50:01.215 [job-0] INFO JobContainer - Running by standalone Mode. 2023-12-11 17:50:01.227 [taskGroup-0] INFO TaskGroupContainer - taskGroupId=[0] start [3] channels for [20] tasks. 2023-12-11 17:50:01.231 [taskGroup-0] INFO Channel - Channel set byte_speed_limit to 209715200. 2023-12-11 17:50:01.232 [taskGroup-0] INFO Channel - Channel set record_speed_limit to -1, No tps activated. 2023-12-11 17:50:01.240 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[17] attemptCount[1] is started 2023-12-11 17:50:01.243 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[0] attemptCount[1] is started 2023-12-11 17:50:01.244 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[8] attemptCount[1] is started 2023-12-11 17:50:01.288 [0-0-17-writer] INFO HostUtils - IP 10.233.76.104 HOSTNAME pose-app-52211-pdc 2023-12-11 17:50:01.322 [0-0-17-reader] INFO HdfsReader$Job - hadoopConfig details:{"classLoader":{"URLs":["file:/u/chengken/datax/plugin/reader/hdfsreader/hdfsreader-0.0.1-SNAPSHOT.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jaxb-impl-2.2.3-1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-hcatalog-core-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-auth-2.7.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/calcite-linq4j-1.0.0-incubating.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-metastore-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jta-1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/activation-1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/aircompressor-0.3.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/aopalliance-1.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-configuration-1.6.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-yarn-server-web-proxy-2.6.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/guice-servlet-3.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-cli-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/calcite-avatica-1.0.0-incubating.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/xz-1.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/parquet-hadoop-bundle-1.6.0rc3.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jetty-all-7.6.0.v20120127.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-httpclient-3.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/stax-api-1.0-2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/apacheds-i18n-2.0.0-M15.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/log4j-api-2.17.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/mail-1.4.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jaxb-api-2.2.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-annotations-2.7.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/geronimo-jta_1.1_spec-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/libthrift-0.9.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/oro-2.0.8.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-hdfs-2.7.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jackson-jaxrs-1.9.13.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/xercesImpl-2.9.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-logging-1.1.3.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-serde-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-shims-common-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/curator-recipes-2.7.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/opencsv-2.3.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jersey-json-1.9.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-service-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/ant-1.9.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/avro-1.7.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-shims-0.20S-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/log4j-1.2.17.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/httpclient-4.1.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-cli-1.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/xml-apis-1.3.04.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-digester-1.8.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/servlet-api-2.5.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/lzo-core-1.0.5.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/log4j-core-2.17.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/javacsv-2.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/protobuf-java-2.5.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-yarn-server-resourcemanager-2.6.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-yarn-server-common-2.6.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-lang3-3.3.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/xmlenc-0.52.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jackson-xc-1.9.13.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-ant-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/pentaho-aggdesigner-algorithm-5.1.5-jhyde.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/apache-log4j-extras-1.2.17.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jetty-util-6.1.26.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-shims-0.23-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/guava-11.0.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/logback-core-1.0.13.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-shims-scheduler-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-aliyun-2.7.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-dbcp-1.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-compress-1.4.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-yarn-common-2.7.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-shims-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-io-2.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/httpcore-4.1.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-mapreduce-client-core-2.7.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jackson-core-asl-1.9.13.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jersey-core-1.9.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/janino-2.7.6.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jdo-api-3.0.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jsp-api-2.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/annotations-2.0.3.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/velocity-1.5.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-codec-1.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-yarn-server-applicationhistoryservice-2.6.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/netty-3.6.2.Final.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/curator-framework-2.6.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/asm-tree-3.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jetty-6.1.26.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/calcite-core-1.0.0-incubating.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/plugin-unstructured-storage-util-0.0.1-SNAPSHOT.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/datax-common-0.0.1-SNAPSHOT.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-beanutils-core-1.8.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/groovy-all-2.1.6.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-yarn-api-2.7.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jersey-server-1.9.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/datanucleus-api-jdo-3.2.6.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/guice-3.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/datanucleus-rdbms-3.2.9.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/java-xmlbuilder-0.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-common-2.7.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jersey-client-1.9.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jersey-guice-1.9.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/libfb303-0.9.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/paranamer-2.3.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/antlr-runtime-3.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/zookeeper-3.4.6.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/geronimo-annotation_1.0_spec-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/ST4-4.0.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jettison-1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/datanucleus-core-3.2.10.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/api-asn1-api-1.0.0-M20.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/geronimo-jaspic_1.0_spec-1.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/apacheds-kerberos-codec-2.0.0-M15.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hamcrest-core-1.3.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/asm-commons-3.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/ant-launcher-1.9.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-exec-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-collections-3.2.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/asm-3.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/stringtemplate-3.2.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-common-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jline-2.12.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/api-util-1.0.0-M20.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/stax-api-1.0.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-net-3.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/bonecp-0.8.0.RELEASE.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/gson-2.2.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/slf4j-log4j12-1.7.10.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jets3t-0.9.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-lang-2.6.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/eigenbase-properties-1.1.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/antlr-2.7.7.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jsch-0.1.42.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/leveldbjni-all-1.8.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-beanutils-1.9.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/logback-classic-1.0.13.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jpam-1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/snappy-java-1.0.4.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jackson-mapper-asl-1.9.13.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-math3-3.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-pool-1.5.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/netty-all-4.0.23.Final.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-compiler-2.7.6.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-daemon-1.0.13.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/javax.inject-1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/htrace-core-3.1.0-incubating.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jsr305-3.0.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/slf4j-api-1.7.10.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/fastjson2-2.0.23.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/derby-10.11.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/curator-client-2.7.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/cos_api-bundle-5.6.137.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-cos-3.1.0-8.3.2.jar"],"parent":{"URLs":["file:/u/chengken/datax/lib/commons-lang3-3.3.2.jar","file:/u/chengken/datax/lib/logback-core-1.0.13.jar","file:/u/chengken/datax/lib/commons-io-2.4.jar","file:/u/chengken/datax/lib/datax-common-0.0.1-SNAPSHOT.jar","file:/u/chengken/datax/lib/hamcrest-core-1.3.jar","file:/u/chengken/datax/lib/datax-transformer-0.0.1-SNAPSHOT.jar","file:/u/chengken/datax/lib/logback-classic-1.0.13.jar","file:/u/chengken/datax/lib/commons-math3-3.1.1.jar","file:/u/chengken/datax/lib/slf4j-api-1.7.10.jar","file:/u/chengken/datax/lib/fastjson2-2.0.23.jar","file:/u/chengken/datax/lib/commons-logging-1.1.1.jar","file:/u/chengken/datax/lib/httpcore-4.4.13.jar","file:/u/chengken/datax/lib/commons-configuration-1.10.jar","file:/u/chengken/datax/lib/fluent-hc-4.5.jar","file:/u/chengken/datax/lib/commons-cli-1.2.jar","file:/u/chengken/datax/lib/janino-2.5.16.jar","file:/u/chengken/datax/lib/groovy-all-2.1.9.jar","file:/u/chengken/datax/lib/commons-codec-1.11.jar","file:/u/chengken/datax/lib/commons-collections-3.2.1.jar","file:/u/chengken/datax/lib/commons-lang-2.6.jar","file:/u/chengken/datax/lib/httpclient-4.5.13.jar","file:/u/chengken/datax/lib/commons-beanutils-1.9.2.jar","file:/u/chengken/datax/lib/datax-core-0.0.1-SNAPSHOT.jar","file:/home/svccnetlhs/chengken/starrocks/"],"parent":{"URLs":["file:/opt/software/jdk1.8.0_112/jre/lib/ext/jaccess.jar","file:/opt/software/jdk1.8.0_112/jre/lib/ext/sunjce_provider.jar","file:/opt/software/jdk1.8.0_112/jre/lib/ext/sunpkcs11.jar","file:/opt/software/jdk1.8.0_112/jre/lib/ext/localedata.jar","file:/opt/software/jdk1.8.0_112/jre/lib/ext/nashorn.jar","file:/opt/software/jdk1.8.0_112/jre/lib/ext/sunec.jar","file:/opt/software/jdk1.8.0_112/jre/lib/ext/dnsns.jar","file:/opt/software/jdk1.8.0_112/jre/lib/ext/zipfs.jar","file:/opt/software/jdk1.8.0_112/jre/lib/ext/cldrdata.jar","file:/opt/software/jdk1.8.0_112/jre/lib/ext/jfxrt.jar"]}}},"finalParameters":["mapreduce.job.end-notification.max.retry.interval","mapreduce.job.end-notification.max.attempts"]} 2023-12-11 17:50:01.324 [0-0-17-reader] INFO Reader$Task - read start 2023-12-11 17:50:01.324 [0-0-17-reader] INFO Reader$Task - reading file : [cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00004-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc] 2023-12-11 17:50:01.324 [0-0-17-reader] INFO HdfsReader$Job - Start Read orcfile [cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00004-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc]. 2023-12-11 17:50:01.326 [0-0-0-reader] INFO HdfsReader$Job - hadoopConfig details:{"classLoader":{"URLs":["file:/u/chengken/datax/plugin/reader/hdfsreader/hdfsreader-0.0.1-SNAPSHOT.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jaxb-impl-2.2.3-1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-hcatalog-core-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-auth-2.7.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/calcite-linq4j-1.0.0-incubating.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-metastore-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jta-1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/activation-1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/aircompressor-0.3.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/aopalliance-1.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-configuration-1.6.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-yarn-server-web-proxy-2.6.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/guice-servlet-3.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-cli-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/calcite-avatica-1.0.0-incubating.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/xz-1.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/parquet-hadoop-bundle-1.6.0rc3.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jetty-all-7.6.0.v20120127.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-httpclient-3.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/stax-api-1.0-2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/apacheds-i18n-2.0.0-M15.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/log4j-api-2.17.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/mail-1.4.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jaxb-api-2.2.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-annotations-2.7.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/geronimo-jta_1.1_spec-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/libthrift-0.9.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/oro-2.0.8.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-hdfs-2.7.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jackson-jaxrs-1.9.13.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/xercesImpl-2.9.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-logging-1.1.3.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-serde-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-shims-common-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/curator-recipes-2.7.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/opencsv-2.3.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jersey-json-1.9.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-service-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/ant-1.9.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/avro-1.7.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-shims-0.20S-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/log4j-1.2.17.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/httpclient-4.1.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-cli-1.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/xml-apis-1.3.04.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-digester-1.8.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/servlet-api-2.5.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/lzo-core-1.0.5.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/log4j-core-2.17.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/javacsv-2.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/protobuf-java-2.5.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-yarn-server-resourcemanager-2.6.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-yarn-server-common-2.6.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-lang3-3.3.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/xmlenc-0.52.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jackson-xc-1.9.13.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-ant-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/pentaho-aggdesigner-algorithm-5.1.5-jhyde.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/apache-log4j-extras-1.2.17.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jetty-util-6.1.26.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-shims-0.23-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/guava-11.0.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/logback-core-1.0.13.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-shims-scheduler-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-aliyun-2.7.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-dbcp-1.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-compress-1.4.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-yarn-common-2.7.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-shims-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-io-2.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/httpcore-4.1.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-mapreduce-client-core-2.7.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jackson-core-asl-1.9.13.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jersey-core-1.9.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/janino-2.7.6.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jdo-api-3.0.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jsp-api-2.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/annotations-2.0.3.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/velocity-1.5.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-codec-1.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-yarn-server-applicationhistoryservice-2.6.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/netty-3.6.2.Final.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/curator-framework-2.6.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/asm-tree-3.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jetty-6.1.26.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/calcite-core-1.0.0-incubating.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/plugin-unstructured-storage-util-0.0.1-SNAPSHOT.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/datax-common-0.0.1-SNAPSHOT.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-beanutils-core-1.8.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/groovy-all-2.1.6.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-yarn-api-2.7.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jersey-server-1.9.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/datanucleus-api-jdo-3.2.6.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/guice-3.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/datanucleus-rdbms-3.2.9.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/java-xmlbuilder-0.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-common-2.7.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jersey-client-1.9.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jersey-guice-1.9.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/libfb303-0.9.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/paranamer-2.3.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/antlr-runtime-3.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/zookeeper-3.4.6.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/geronimo-annotation_1.0_spec-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/ST4-4.0.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jettison-1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/datanucleus-core-3.2.10.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/api-asn1-api-1.0.0-M20.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/geronimo-jaspic_1.0_spec-1.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/apacheds-kerberos-codec-2.0.0-M15.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hamcrest-core-1.3.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/asm-commons-3.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/ant-launcher-1.9.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-exec-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-collections-3.2.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/asm-3.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/stringtemplate-3.2.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-common-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jline-2.12.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/api-util-1.0.0-M20.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/stax-api-1.0.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-net-3.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/bonecp-0.8.0.RELEASE.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/gson-2.2.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/slf4j-log4j12-1.7.10.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jets3t-0.9.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-lang-2.6.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/eigenbase-properties-1.1.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/antlr-2.7.7.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jsch-0.1.42.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/leveldbjni-all-1.8.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-beanutils-1.9.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/logback-classic-1.0.13.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jpam-1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/snappy-java-1.0.4.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jackson-mapper-asl-1.9.13.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-math3-3.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-pool-1.5.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/netty-all-4.0.23.Final.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-compiler-2.7.6.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-daemon-1.0.13.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/javax.inject-1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/htrace-core-3.1.0-incubating.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jsr305-3.0.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/slf4j-api-1.7.10.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/fastjson2-2.0.23.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/derby-10.11.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/curator-client-2.7.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/cos_api-bundle-5.6.137.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-cos-3.1.0-8.3.2.jar"],"parent":{"URLs":["file:/u/chengken/datax/lib/commons-lang3-3.3.2.jar","file:/u/chengken/datax/lib/logback-core-1.0.13.jar","file:/u/chengken/datax/lib/commons-io-2.4.jar","file:/u/chengken/datax/lib/datax-common-0.0.1-SNAPSHOT.jar","file:/u/chengken/datax/lib/hamcrest-core-1.3.jar","file:/u/chengken/datax/lib/datax-transformer-0.0.1-SNAPSHOT.jar","file:/u/chengken/datax/lib/logback-classic-1.0.13.jar","file:/u/chengken/datax/lib/commons-math3-3.1.1.jar","file:/u/chengken/datax/lib/slf4j-api-1.7.10.jar","file:/u/chengken/datax/lib/fastjson2-2.0.23.jar","file:/u/chengken/datax/lib/commons-logging-1.1.1.jar","file:/u/chengken/datax/lib/httpcore-4.4.13.jar","file:/u/chengken/datax/lib/commons-configuration-1.10.jar","file:/u/chengken/datax/lib/fluent-hc-4.5.jar","file:/u/chengken/datax/lib/commons-cli-1.2.jar","file:/u/chengken/datax/lib/janino-2.5.16.jar","file:/u/chengken/datax/lib/groovy-all-2.1.9.jar","file:/u/chengken/datax/lib/commons-codec-1.11.jar","file:/u/chengken/datax/lib/commons-collections-3.2.1.jar","file:/u/chengken/datax/lib/commons-lang-2.6.jar","file:/u/chengken/datax/lib/httpclient-4.5.13.jar","file:/u/chengken/datax/lib/commons-beanutils-1.9.2.jar","file:/u/chengken/datax/lib/datax-core-0.0.1-SNAPSHOT.jar","file:/home/svccnetlhs/chengken/starrocks/"],"parent":{"URLs":["file:/opt/software/jdk1.8.0_112/jre/lib/ext/jaccess.jar","file:/opt/software/jdk1.8.0_112/jre/lib/ext/sunjce_provider.jar","file:/opt/software/jdk1.8.0_112/jre/lib/ext/sunpkcs11.jar","file:/opt/software/jdk1.8.0_112/jre/lib/ext/localedata.jar","file:/opt/software/jdk1.8.0_112/jre/lib/ext/nashorn.jar","file:/opt/software/jdk1.8.0_112/jre/lib/ext/sunec.jar","file:/opt/software/jdk1.8.0_112/jre/lib/ext/dnsns.jar","file:/opt/software/jdk1.8.0_112/jre/lib/ext/zipfs.jar","file:/opt/software/jdk1.8.0_112/jre/lib/ext/cldrdata.jar","file:/opt/software/jdk1.8.0_112/jre/lib/ext/jfxrt.jar"]}}},"finalParameters":["mapreduce.job.end-notification.max.retry.interval","mapreduce.job.end-notification.max.attempts"]} 2023-12-11 17:50:01.326 [0-0-8-reader] INFO HdfsReader$Job - hadoopConfig details:{"classLoader":{"URLs":["file:/u/chengken/datax/plugin/reader/hdfsreader/hdfsreader-0.0.1-SNAPSHOT.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jaxb-impl-2.2.3-1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-hcatalog-core-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-auth-2.7.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/calcite-linq4j-1.0.0-incubating.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-metastore-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jta-1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/activation-1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/aircompressor-0.3.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/aopalliance-1.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-configuration-1.6.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-yarn-server-web-proxy-2.6.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/guice-servlet-3.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-cli-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/calcite-avatica-1.0.0-incubating.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/xz-1.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/parquet-hadoop-bundle-1.6.0rc3.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jetty-all-7.6.0.v20120127.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-httpclient-3.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/stax-api-1.0-2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/apacheds-i18n-2.0.0-M15.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/log4j-api-2.17.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/mail-1.4.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jaxb-api-2.2.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-annotations-2.7.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/geronimo-jta_1.1_spec-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/libthrift-0.9.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/oro-2.0.8.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-hdfs-2.7.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jackson-jaxrs-1.9.13.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/xercesImpl-2.9.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-logging-1.1.3.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-serde-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-shims-common-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/curator-recipes-2.7.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/opencsv-2.3.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jersey-json-1.9.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-service-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/ant-1.9.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/avro-1.7.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-shims-0.20S-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/log4j-1.2.17.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/httpclient-4.1.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-cli-1.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/xml-apis-1.3.04.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-digester-1.8.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/servlet-api-2.5.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/lzo-core-1.0.5.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/log4j-core-2.17.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/javacsv-2.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/protobuf-java-2.5.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-yarn-server-resourcemanager-2.6.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-yarn-server-common-2.6.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-lang3-3.3.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/xmlenc-0.52.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jackson-xc-1.9.13.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-ant-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/pentaho-aggdesigner-algorithm-5.1.5-jhyde.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/apache-log4j-extras-1.2.17.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jetty-util-6.1.26.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-shims-0.23-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/guava-11.0.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/logback-core-1.0.13.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-shims-scheduler-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-aliyun-2.7.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-dbcp-1.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-compress-1.4.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-yarn-common-2.7.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-shims-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-io-2.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/httpcore-4.1.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-mapreduce-client-core-2.7.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jackson-core-asl-1.9.13.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jersey-core-1.9.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/janino-2.7.6.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jdo-api-3.0.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jsp-api-2.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/annotations-2.0.3.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/velocity-1.5.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-codec-1.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-yarn-server-applicationhistoryservice-2.6.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/netty-3.6.2.Final.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/curator-framework-2.6.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/asm-tree-3.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jetty-6.1.26.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/calcite-core-1.0.0-incubating.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/plugin-unstructured-storage-util-0.0.1-SNAPSHOT.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/datax-common-0.0.1-SNAPSHOT.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-beanutils-core-1.8.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/groovy-all-2.1.6.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-yarn-api-2.7.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jersey-server-1.9.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/datanucleus-api-jdo-3.2.6.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/guice-3.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/datanucleus-rdbms-3.2.9.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/java-xmlbuilder-0.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-common-2.7.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jersey-client-1.9.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jersey-guice-1.9.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/libfb303-0.9.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/paranamer-2.3.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/antlr-runtime-3.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/zookeeper-3.4.6.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/geronimo-annotation_1.0_spec-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/ST4-4.0.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jettison-1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/datanucleus-core-3.2.10.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/api-asn1-api-1.0.0-M20.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/geronimo-jaspic_1.0_spec-1.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/apacheds-kerberos-codec-2.0.0-M15.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hamcrest-core-1.3.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/asm-commons-3.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/ant-launcher-1.9.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-exec-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-collections-3.2.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/asm-3.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/stringtemplate-3.2.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-common-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jline-2.12.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/api-util-1.0.0-M20.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/stax-api-1.0.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-net-3.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/bonecp-0.8.0.RELEASE.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/gson-2.2.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/slf4j-log4j12-1.7.10.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jets3t-0.9.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-lang-2.6.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/eigenbase-properties-1.1.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/antlr-2.7.7.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jsch-0.1.42.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/leveldbjni-all-1.8.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-beanutils-1.9.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/logback-classic-1.0.13.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jpam-1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/snappy-java-1.0.4.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jackson-mapper-asl-1.9.13.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-math3-3.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-pool-1.5.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/netty-all-4.0.23.Final.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-compiler-2.7.6.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-daemon-1.0.13.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/javax.inject-1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/htrace-core-3.1.0-incubating.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jsr305-3.0.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/slf4j-api-1.7.10.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/fastjson2-2.0.23.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/derby-10.11.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/curator-client-2.7.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/cos_api-bundle-5.6.137.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-cos-3.1.0-8.3.2.jar"],"parent":{"URLs":["file:/u/chengken/datax/lib/commons-lang3-3.3.2.jar","file:/u/chengken/datax/lib/logback-core-1.0.13.jar","file:/u/chengken/datax/lib/commons-io-2.4.jar","file:/u/chengken/datax/lib/datax-common-0.0.1-SNAPSHOT.jar","file:/u/chengken/datax/lib/hamcrest-core-1.3.jar","file:/u/chengken/datax/lib/datax-transformer-0.0.1-SNAPSHOT.jar","file:/u/chengken/datax/lib/logback-classic-1.0.13.jar","file:/u/chengken/datax/lib/commons-math3-3.1.1.jar","file:/u/chengken/datax/lib/slf4j-api-1.7.10.jar","file:/u/chengken/datax/lib/fastjson2-2.0.23.jar","file:/u/chengken/datax/lib/commons-logging-1.1.1.jar","file:/u/chengken/datax/lib/httpcore-4.4.13.jar","file:/u/chengken/datax/lib/commons-configuration-1.10.jar","file:/u/chengken/datax/lib/fluent-hc-4.5.jar","file:/u/chengken/datax/lib/commons-cli-1.2.jar","file:/u/chengken/datax/lib/janino-2.5.16.jar","file:/u/chengken/datax/lib/groovy-all-2.1.9.jar","file:/u/chengken/datax/lib/commons-codec-1.11.jar","file:/u/chengken/datax/lib/commons-collections-3.2.1.jar","file:/u/chengken/datax/lib/commons-lang-2.6.jar","file:/u/chengken/datax/lib/httpclient-4.5.13.jar","file:/u/chengken/datax/lib/commons-beanutils-1.9.2.jar","file:/u/chengken/datax/lib/datax-core-0.0.1-SNAPSHOT.jar","file:/home/svccnetlhs/chengken/starrocks/"],"parent":{"URLs":["file:/opt/software/jdk1.8.0_112/jre/lib/ext/jaccess.jar","file:/opt/software/jdk1.8.0_112/jre/lib/ext/sunjce_provider.jar","file:/opt/software/jdk1.8.0_112/jre/lib/ext/sunpkcs11.jar","file:/opt/software/jdk1.8.0_112/jre/lib/ext/localedata.jar","file:/opt/software/jdk1.8.0_112/jre/lib/ext/nashorn.jar","file:/opt/software/jdk1.8.0_112/jre/lib/ext/sunec.jar","file:/opt/software/jdk1.8.0_112/jre/lib/ext/dnsns.jar","file:/opt/software/jdk1.8.0_112/jre/lib/ext/zipfs.jar","file:/opt/software/jdk1.8.0_112/jre/lib/ext/cldrdata.jar","file:/opt/software/jdk1.8.0_112/jre/lib/ext/jfxrt.jar"]}}},"finalParameters":["mapreduce.job.end-notification.max.retry.interval","mapreduce.job.end-notification.max.attempts"]} 2023-12-11 17:50:01.327 [0-0-0-reader] INFO Reader$Task - read start 2023-12-11 17:50:01.327 [0-0-0-reader] INFO Reader$Task - reading file : [cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00002-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc] 2023-12-11 17:50:01.328 [0-0-0-reader] INFO HdfsReader$Job - Start Read orcfile [cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00002-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc]. 2023-12-11 17:50:01.328 [0-0-8-reader] INFO Reader$Task - read start 2023-12-11 17:50:01.328 [0-0-8-reader] INFO Reader$Task - reading file : [cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00008-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc] 2023-12-11 17:50:01.328 [0-0-8-reader] INFO HdfsReader$Job - Start Read orcfile [cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00008-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc]. 十二月 11, 2023 5:50:01 下午 org.apache.hadoop.hive.ql.log.PerfLogger PerfLogBegin 信息: <PERFLOG method=OrcGetSplits from=org.apache.hadoop.hive.ql.io.orc.ReaderImpl> 十二月 11, 2023 5:50:01 下午 org.apache.hadoop.hive.ql.log.PerfLogger PerfLogBegin 信息: <PERFLOG method=OrcGetSplits from=org.apache.hadoop.hive.ql.io.orc.ReaderImpl> 十二月 11, 2023 5:50:01 下午 org.apache.hadoop.hive.ql.log.PerfLogger PerfLogBegin 信息: <PERFLOG method=OrcGetSplits from=org.apache.hadoop.hive.ql.io.orc.ReaderImpl> 十二月 11, 2023 5:50:01 下午 org.apache.hadoop.conf.Configuration warnOnceIfDeprecated 信息: mapred.input.dir is deprecated. Instead, use mapreduce.input.fileinputformat.inputdir 2023-12-11 17:50:01.646 [ORC_GET_SPLITS #1] INFO CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00004-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading 2023-12-11 17:50:01.776 [ORC_GET_SPLITS #1] INFO CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00002-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading 2023-12-11 17:50:01.778 [ORC_GET_SPLITS #1] INFO CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00008-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading 十二月 11, 2023 5:50:02 下午 org.apache.hadoop.hive.ql.io.orc.OrcInputFormat generateSplitsInfo 信息: FooterCacheHitRatio: 0/1 十二月 11, 2023 5:50:02 下午 org.apache.hadoop.hive.ql.log.PerfLogger PerfLogEnd 信息: </PERFLOG method=OrcGetSplits start=1702288201425 end=1702288202185 duration=760 from=org.apache.hadoop.hive.ql.io.orc.ReaderImpl> 2023-12-11 17:50:02.288 [0-0-17-reader] INFO CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00004-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading 十二月 11, 2023 5:50:02 下午 org.apache.hadoop.hive.ql.io.orc.OrcInputFormat generateSplitsInfo 信息: FooterCacheHitRatio: 0/1 十二月 11, 2023 5:50:02 下午 org.apache.hadoop.hive.ql.log.PerfLogger PerfLogEnd 信息: </PERFLOG method=OrcGetSplits start=1702288201425 end=1702288202296 duration=871 from=org.apache.hadoop.hive.ql.io.orc.ReaderImpl> 2023-12-11 17:50:02.369 [0-0-8-reader] INFO CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00008-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading 十二月 11, 2023 5:50:02 下午 org.apache.hadoop.hive.ql.io.orc.OrcInputFormat generateSplitsInfo 信息: FooterCacheHitRatio: 0/1 十二月 11, 2023 5:50:02 下午 org.apache.hadoop.hive.ql.log.PerfLogger PerfLogEnd 信息: </PERFLOG method=OrcGetSplits start=1702288201425 end=1702288202433 duration=1008 from=org.apache.hadoop.hive.ql.io.orc.ReaderImpl> 2023-12-11 17:50:02.492 [0-0-0-reader] INFO CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00002-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading 十二月 11, 2023 5:50:02 下午 org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger <init> 信息: min key = null, max key = {originalTxn: 0, bucket: -1, row: 302079} 十二月 11, 2023 5:50:02 下午 org.apache.hadoop.hive.ql.io.orc.ReaderImpl rowsOptions 信息: Reading ORC rows from cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00004-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc with {include: null, offset: 3, length: 9223372036854775807} 2023-12-11 17:50:02.647 [0-0-17-reader] INFO CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00004-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading 十二月 11, 2023 5:50:02 下午 org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger <init> 信息: min key = null, max key = {originalTxn: 0, bucket: -1, row: 302079} 十二月 11, 2023 5:50:02 下午 org.apache.hadoop.hive.ql.io.orc.ReaderImpl rowsOptions 信息: Reading ORC rows from cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00008-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc with {include: null, offset: 3, length: 9223372036854775807} 2023-12-11 17:50:02.784 [0-0-8-reader] INFO CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00008-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading 十二月 11, 2023 5:50:02 下午 org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger <init> 信息: min key = null, max key = {originalTxn: 0, bucket: -1, row: 302079} 十二月 11, 2023 5:50:02 下午 org.apache.hadoop.hive.ql.io.orc.ReaderImpl rowsOptions 信息: Reading ORC rows from cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00002-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc with {include: null, offset: 3, length: 9223372036854775807} 2023-12-11 17:50:02.867 [0-0-0-reader] INFO CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00002-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading 2023-12-11 17:50:11.235 [job-0] INFO StandAloneJobContainerCommunicator - Total 0 records, 0 bytes | Speed 0B/s, 0 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 0.000s | All Task WaitReaderTime 0.000s | Percentage 0.00% 2023-12-11 17:50:21.236 [job-0] INFO StandAloneJobContainerCommunicator - Total 297312 records, 558177719 bytes | Speed 53.23MB/s, 29731 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 0.023s | All Task WaitReaderTime 25.333s | Percentage 0.00% 2023-12-11 17:50:30.385 [0-0-8-reader] INFO CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00008-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading 十二月 11, 2023 5:50:30 下午 org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger <init> 信息: min key = {originalTxn: 0, bucket: -1, row: 302079}, max key = {originalTxn: 0, bucket: -1, row: 599039} 十二月 11, 2023 5:50:30 下午 org.apache.hadoop.hive.ql.io.orc.ReaderImpl rowsOptions 信息: Reading ORC rows from cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00008-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc with {include: null, offset: 156410867, length: 9223372036854775807} 2023-12-11 17:50:30.731 [0-0-8-reader] INFO CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00008-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading 2023-12-11 17:50:30.907 [0-0-17-reader] INFO CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00004-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading 2023-12-11 17:50:31.114 [0-0-0-reader] INFO CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00002-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading 十二月 11, 2023 5:50:31 下午 org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger <init> 信息: min key = {originalTxn: 0, bucket: -1, row: 302079}, max key = {originalTxn: 0, bucket: -1, row: 599039} 十二月 11, 2023 5:50:31 下午 org.apache.hadoop.hive.ql.io.orc.ReaderImpl rowsOptions 信息: Reading ORC rows from cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00004-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc with {include: null, offset: 156011050, length: 9223372036854775807} 2023-12-11 17:50:31.237 [job-0] INFO StandAloneJobContainerCommunicator - Total 598944 records, 1125524739 bytes | Speed 54.11MB/s, 30163 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 0.040s | All Task WaitReaderTime 43.878s | Percentage 0.00% 2023-12-11 17:50:31.280 [0-0-17-reader] INFO CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00004-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading 十二月 11, 2023 5:50:31 下午 org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger <init> 信息: min key = {originalTxn: 0, bucket: -1, row: 302079}, max key = {originalTxn: 0, bucket: -1, row: 604159} 十二月 11, 2023 5:50:31 下午 org.apache.hadoop.hive.ql.io.orc.ReaderImpl rowsOptions 信息: Reading ORC rows from cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00002-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc with {include: null, offset: 156411995, length: 9223372036854775807} 2023-12-11 17:50:31.457 [0-0-0-reader] INFO CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00002-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading 2023-12-11 17:50:41.238 [job-0] INFO StandAloneJobContainerCommunicator - Total 906144 records, 1702990600 bytes | Speed 55.07MB/s, 30720 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 0.056s | All Task WaitReaderTime 62.147s | Percentage 0.00% 2023-12-11 17:50:51.241 [job-0] INFO StandAloneJobContainerCommunicator - Total 1203104 records, 2261515086 bytes | Speed 53.27MB/s, 29696 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 0.070s | All Task WaitReaderTime 92.372s | Percentage 0.00% 2023-12-11 17:50:56.098 [0-0-8-reader] INFO CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00008-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading 十二月 11, 2023 5:50:56 下午 org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger <init> 信息: min key = {originalTxn: 0, bucket: -1, row: 599039}, max key = {originalTxn: 0, bucket: -1, row: 803839} 十二月 11, 2023 5:50:56 下午 org.apache.hadoop.hive.ql.io.orc.ReaderImpl rowsOptions 信息: Reading ORC rows from cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00008-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc with {include: null, offset: 311140141, length: 9223372036854775807} 2023-12-11 17:50:56.469 [0-0-8-reader] INFO CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00008-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading 2023-12-11 17:50:57.564 [0-0-0-reader] INFO CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00002-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading 十二月 11, 2023 5:50:57 下午 org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger <init> 信息: min key = {originalTxn: 0, bucket: -1, row: 604159}, max key = {originalTxn: 0, bucket: -1, row: 803839} 十二月 11, 2023 5:50:57 下午 org.apache.hadoop.hive.ql.io.orc.ReaderImpl rowsOptions 信息: Reading ORC rows from cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00002-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc with {include: null, offset: 312563739, length: 9223372036854775807} 2023-12-11 17:50:57.971 [0-0-0-reader] INFO CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00002-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading 2023-12-11 17:50:58.447 [0-0-17-reader] INFO CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00004-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading 十二月 11, 2023 5:50:58 下午 org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger <init> 信息: min key = {originalTxn: 0, bucket: -1, row: 599039}, max key = {originalTxn: 0, bucket: -1, row: 803839} 十二月 11, 2023 5:50:58 下午 org.apache.hadoop.hive.ql.io.orc.ReaderImpl rowsOptions 信息: Reading ORC rows from cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00004-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc with {include: null, offset: 310368886, length: 9223372036854775807} 2023-12-11 17:50:58.782 [0-0-17-reader] INFO CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00004-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading 2023-12-11 17:51:01.242 [job-0] INFO StandAloneJobContainerCommunicator - Total 1703232 records, 3203376993 bytes | Speed 89.82MB/s, 50012 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 0.093s | All Task WaitReaderTime 127.642s | Percentage 0.00% 2023-12-11 17:51:11.242 [job-0] INFO StandAloneJobContainerCommunicator - Total 1829984 records, 3440997771 bytes | Speed 22.66MB/s, 12675 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 0.101s | All Task WaitReaderTime 138.648s | Percentage 0.00% 2023-12-11 17:51:15.910 [0-0-8-reader] INFO CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00008-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading 十二月 11, 2023 5:51:16 下午 org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger <init> 信息: min key = {originalTxn: 0, bucket: -1, row: 803839}, max key = {originalTxn: 0, bucket: -1, row: 1100799} 十二月 11, 2023 5:51:16 下午 org.apache.hadoop.hive.ql.io.orc.ReaderImpl rowsOptions 信息: Reading ORC rows from cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00008-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc with {include: null, offset: 417009783, length: 9223372036854775807} 2023-12-11 17:51:16.305 [0-0-8-reader] INFO CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00008-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading 2023-12-11 17:51:17.688 [0-0-0-reader] INFO CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00002-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading 十二月 11, 2023 5:51:17 下午 org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger <init> 信息: min key = {originalTxn: 0, bucket: -1, row: 803839}, max key = {originalTxn: 0, bucket: -1, row: 1100799} 十二月 11, 2023 5:51:17 下午 org.apache.hadoop.hive.ql.io.orc.ReaderImpl rowsOptions 信息: Reading ORC rows from cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00002-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc with {include: null, offset: 415835058, length: 9223372036854775807} 2023-12-11 17:51:18.017 [0-0-0-reader] INFO CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00002-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading 2023-12-11 17:51:18.695 [0-0-17-reader] INFO CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00004-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading 十二月 11, 2023 5:51:18 下午 org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger <init> 信息: min key = {originalTxn: 0, bucket: -1, row: 803839}, max key = {originalTxn: 0, bucket: -1, row: 1105919} 十二月 11, 2023 5:51:18 下午 org.apache.hadoop.hive.ql.io.orc.ReaderImpl rowsOptions 信息: Reading ORC rows from cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00004-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc with {include: null, offset: 415904892, length: 9223372036854775807} 2023-12-11 17:51:19.066 [0-0-17-reader] INFO CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00004-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading 2023-12-11 17:51:21.243 [job-0] INFO StandAloneJobContainerCommunicator - Total 2342208 records, 4403928517 bytes | Speed 91.83MB/s, 51222 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 0.126s | All Task WaitReaderTime 178.482s | Percentage 0.00% 2023-12-11 17:51:31.245 [job-0] INFO StandAloneJobContainerCommunicator - Total 2466496 records, 4638515258 bytes | Speed 22.37MB/s, 12428 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 0.132s | All Task WaitReaderTime 190.077s | Percentage 0.00% 2023-12-11 17:51:41.246 [job-0] INFO StandAloneJobContainerCommunicator - Total 2967264 records, 5579213090 bytes | Speed 89.71MB/s, 50076 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 0.156s | All Task WaitReaderTime 229.517s | Percentage 0.00% 2023-12-11 17:51:41.359 [0-0-8-reader] INFO CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00008-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading 十二月 11, 2023 5:51:41 下午 org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger <init> 信息: min key = {originalTxn: 0, bucket: -1, row: 1100799}, max key = {originalTxn: 0, bucket: -1, row: 1300479} 十二月 11, 2023 5:51:41 下午 org.apache.hadoop.hive.ql.io.orc.ReaderImpl rowsOptions 信息: Reading ORC rows from cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00008-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc with {include: null, offset: 571347891, length: 9223372036854775807} 2023-12-11 17:51:41.747 [0-0-8-reader] INFO CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00008-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading 2023-12-11 17:51:41.998 [0-0-0-reader] INFO CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00002-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading 十二月 11, 2023 5:51:42 下午 org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger <init> 信息: min key = {originalTxn: 0, bucket: -1, row: 1100799}, max key = {originalTxn: 0, bucket: -1, row: 1300479} 十二月 11, 2023 5:51:42 下午 org.apache.hadoop.hive.ql.io.orc.ReaderImpl rowsOptions 信息: Reading ORC rows from cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00002-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc with {include: null, offset: 570444727, length: 9223372036854775807} 2023-12-11 17:51:42.390 [0-0-0-reader] INFO CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00002-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading 2023-12-11 17:51:44.650 [0-0-17-reader] INFO CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00004-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading 十二月 11, 2023 5:51:44 下午 org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger <init> 信息: min key = {originalTxn: 0, bucket: -1, row: 1105919}, max key = {originalTxn: 0, bucket: -1, row: 1305599} 十二月 11, 2023 5:51:44 下午 org.apache.hadoop.hive.ql.io.orc.ReaderImpl rowsOptions 信息: Reading ORC rows from cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00004-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc with {include: null, offset: 572342605, length: 9223372036854775807} 2023-12-11 17:51:44.978 [0-0-17-reader] INFO CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00004-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading 2023-12-11 17:51:51.248 [job-0] INFO StandAloneJobContainerCommunicator - Total 3307424 records, 6219717845 bytes | Speed 61.08MB/s, 34016 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 0.172s | All Task WaitReaderTime 246.926s | Percentage 0.00% 2023-12-11 17:52:01.250 [job-0] INFO StandAloneJobContainerCommunicator - Total 3614624 records, 6797953036 bytes | Speed 55.14MB/s, 30720 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 0.187s | All Task WaitReaderTime 279.975s | Percentage 0.00% 2023-12-11 17:52:02.446 [0-0-8-reader] INFO CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00008-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading 十二月 11, 2023 5:52:02 下午 org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger <init> 信息: min key = {originalTxn: 0, bucket: -1, row: 1300479}, max key = {originalTxn: 0, bucket: -1, row: 1597439} 十二月 11, 2023 5:52:02 下午 org.apache.hadoop.hive.ql.io.orc.ReaderImpl rowsOptions 信息: Reading ORC rows from cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00008-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc with {include: null, offset: 674573070, length: 9223372036854775807} 2023-12-11 17:52:02.859 [0-0-8-reader] INFO CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00008-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading 2023-12-11 17:52:03.369 [0-0-0-reader] INFO CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00002-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading 十二月 11, 2023 5:52:03 下午 org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger <init> 信息: min key = {originalTxn: 0, bucket: -1, row: 1300479}, max key = {originalTxn: 0, bucket: -1, row: 1597439} 十二月 11, 2023 5:52:03 下午 org.apache.hadoop.hive.ql.io.orc.ReaderImpl rowsOptions 信息: Reading ORC rows from cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00002-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc with {include: null, offset: 673856234, length: 9223372036854775807} 2023-12-11 17:52:03.751 [0-0-0-reader] INFO CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00002-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading 2023-12-11 17:52:04.567 [0-0-17-reader] INFO CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00004-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading 十二月 11, 2023 5:52:04 下午 org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger <init> 信息: min key = {originalTxn: 0, bucket: -1, row: 1305599}, max key = {originalTxn: 0, bucket: -1, row: 1602559} 十二月 11, 2023 5:52:04 下午 org.apache.hadoop.hive.ql.io.orc.ReaderImpl rowsOptions 信息: Reading ORC rows from cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00004-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc with {include: null, offset: 675641495, length: 9223372036854775807} 2023-12-11 17:52:04.881 [0-0-17-reader] INFO CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00004-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading 2023-12-11 17:52:11.251 [job-0] INFO StandAloneJobContainerCommunicator - Total 3906464 records, 7347149221 bytes | Speed 52.38MB/s, 29184 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 0.202s | All Task WaitReaderTime 297.463s | Percentage 0.00% 2023-12-11 17:52:21.252 [job-0] INFO StandAloneJobContainerCommunicator - Total 4198304 records, 7896134390 bytes | Speed 52.36MB/s, 29184 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 0.215s | All Task WaitReaderTime 331.832s | Percentage 0.00% ...
StarRocks数据顺利加载：
[Mon Dec 11 09:56:57 2023]:['default_cluster:cndlopsns']>[192.168.1.121]:[ods] [ADHOC用户集群] > show partitions from ods.buckets_tmp_20231205__sams_cos; +-------------+---------------+----------------+---------------------+--------------------+--------+--------------+----------------------------------------------------------------------------+-----------------+---------+----------------+---------------+---------------------+--------------------------+----------+------------+ | PartitionId | PartitionName | VisibleVersion | VisibleVersionTime | VisibleVersionHash | State | PartitionKey | Range | DistributionKey | Buckets | ReplicationNum | StorageMedium | CooldownTime | LastConsistencyCheckTime | DataSize | IsInMemory | +-------------+---------------+----------------+---------------------+--------------------+--------+--------------+----------------------------------------------------------------------------+-----------------+---------+----------------+---------------+---------------------+--------------------------+----------+------------+ | 513180658 | p20231114 | 1 | 2023-12-11 09:49:41 | 0 | NORMAL | ts | [types: [DATE]; keys: [2023-11-14]; ..types: [DATE]; keys: [2023-11-15]; ) | uuid | 196 | 2 | HDD | 9999-12-31 15:59:59 | NULL | 16.3GB | false | | 504724038 | p20231115 | 1 | 2023-12-06 13:47:20 | 0 | NORMAL | ts | [types: [DATE]; keys: [2023-11-15]; ..types: [DATE]; keys: [2023-11-16]; ) | uuid | 196 | 2 | HDD | 9999-12-31 15:59:59 | NULL | .000 | false | | 504722860 | p20231116 | 1 | 2023-12-06 13:47:20 | 0 | NORMAL | ts | [types: [DATE]; keys: [2023-11-16]; ..types: [DATE]; keys: [2023-11-17]; ) | uuid | 196 | 2 | HDD | 9999-12-31 15:59:59 | NULL | .000 | false | | 504721682 | p20231206 | 1 | 2023-12-06 13:47:20 | 0 | NORMAL | ts | [types: [DATE]; keys: [2023-12-06]; ..types: [DATE]; keys: [2023-12-07]; ) | uuid | 196 | 2 | HDD | 9999-12-31 15:59:59 | NULL | .000 | false | | 504722271 | p20231207 | 1 | 2023-12-06 13:47:20 | 0 | NORMAL | ts | [types: [DATE]; keys: [2023-12-07]; ..types: [DATE]; keys: [2023-12-08]; ) | uuid | 196 | 2 | HDD | 9999-12-31 15:59:59 | NULL | .000 | false | | 504720504 | p20231208 | 1 | 2023-12-06 13:47:20 | 0 | NORMAL | ts | [types: [DATE]; keys: [2023-12-08]; ..types: [DATE]; keys: [2023-12-09]; ) | uuid | 196 | 2 | HDD | 9999-12-31 15:59:59 | NULL | .000 | false | | 504721093 | p20231209 | 1 | 2023-12-06 13:47:20 | 0 | NORMAL | ts | [types: [DATE]; keys: [2023-12-09]; ..types: [DATE]; keys: [2023-12-10]; ) | uuid | 196 | 2 | HDD | 9999-12-31 15:59:59 | NULL | .000 | false | | 505174943 | p20231210 | 1 | 2023-12-07 00:14:47 | 0 | NORMAL | ts | [types: [DATE]; keys: [2023-12-10]; ..types: [DATE]; keys: [2023-12-11]; ) | uuid | 196 | 2 | HDD | 9999-12-31 15:59:59 | NULL | .000 | false | | 506968779 | p20231211 | 1 | 2023-12-08 00:04:57 | 0 | NORMAL | ts | [types: [DATE]; keys: [2023-12-11]; ..types: [DATE]; keys: [2023-12-12]; ) | uuid | 196 | 2 | HDD | 9999-12-31 15:59:59 | NULL | .000 | false | | 508936021 | p20231212 | 1 | 2023-12-09 00:11:09 | 0 | NORMAL | ts | [types: [DATE]; keys: [2023-12-12]; ..types: [DATE]; keys: [2023-12-13]; ) | uuid | 196 | 2 | HDD | 9999-12-31 15:59:59 | NULL | .000 | false | | 510079030 | p20231213 | 1 | 2023-12-10 00:05:38 | 0 | NORMAL | ts | [types: [DATE]; keys: [2023-12-13]; ..types: [DATE]; keys: [2023-12-14]; ) | uuid | 196 | 2 | HDD | 9999-12-31 15:59:59 | NULL | .000 | false | | 510795860 | p20231214 | 1 | 2023-12-11 00:07:15 | 0 | NORMAL | ts | [types: [DATE]; keys: [2023-12-14]; ..types: [DATE]; keys: [2023-12-15]; ) | uuid | 196 | 2 | HDD | 9999-12-31 15:59:59 | NULL | .000 | false | +-------------+---------------+----------------+---------------------+--------------------+--------+--------------+----------------------------------------------------------------------------+-----------------+---------+----------------+---------------+---------------------+--------------------------+----------+------------+ 12 rows in set (0.002 sec) [Mon Dec 11 09:56:57 2023]:['default_cluster:cndlopsns']>[192.168.1.121]:[ods] [ADHOC用户集群] >
查询正常：
[Mon Dec 11 09:58:20 2023]:['default_cluster:cndlopsns']>[192.168.1.121]:[ods] [ADHOC用户集群] > select * from ods.buckets_tmp_20231205__sams_cos where ts='2023-11-14' limit 2; +------------+------------------+------------+--------------------------------------+---------------+---------------+----------------------+----------+---------------------------------------------------+-----------+------------+-------------+------------------+-------------------+----------+--------------------------------------+----------------+---------------+----------------------+--------------+-----------------+----------------------+---------+-------------------+--------------------------+--------------------------------------------------------------------------------------------------------------------+--------------+-------------+----------------+-------------+--------------+------+------------+--------------+-----------+----------+---------+------+----------+---------+----------+-----------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------+----------------+---------------+-----------+--------------+------------+ | ts | uuid | import_ds_ | unique_action_id | action_time | report_time | action_type | ka_id | action_session_id | wx_app_id | wx_open_id | wx_union_id | external_user_id | merber_id | local_id | encrypted_imei | encrypted_idfa | encrypted_mac | encrypted_android_id | encrypted_qq | encrypted_phone | encrypting_algorithm | chan_id | chan_refer_app_id | chan_shop_id | chan_shop_name | client_type | client_name | client_version | sdk_version | device_model | ip | user_agent | page_path | page_name | referrer | address | city | province | country | latitude | longitude | json_properties | fdate | chan_custom_id | etl_load_time | tag_id | tag_name | event_name | +------------+------------------+------------+--------------------------------------+---------------+---------------+----------------------+----------+---------------------------------------------------+-----------+------------+-------------+------------------+-------------------+----------+--------------------------------------+----------------+---------------+----------------------+--------------+-----------------+----------------------+---------+-------------------+--------------------------+--------------------------------------------------------------------------------------------------------------------+--------------+-------------+----------------+-------------+--------------+------+------------+--------------+-----------+----------+---------+------+----------+---------+----------+-----------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------+----------------+---------------+-----------+--------------+------------+ | 2023-11-14 | 8325 | 2023111412 | ed55491f-da41-4976-8601-49c4405d994f | 1699936160765 | 1699936132584 | element | 10001042 | 1699936154050a1b33cce-f249-4112-8db2-288f1f0f6703 | NULL | NULL | NULL | 274012627 | 10742100529761108 | NULL | 9090 | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | 9991,6758,4834,6119,9996 | 苏州木渎DC | sams-app-sdk | NULL | NULL | NULL | NULL | NULL | NULL | HomeFragment | 首页 | NULL | NULL | NULL | NULL | NULL | NULL | NULL | ***** | 20231114 | default | NULL | advantage | 普通会员 | NULL | | 2023-11-14 | 8614 | 2023111412 | f30f3672-1bad-478c-ab19-16b736b850ef | 1699936161032 | 1699936133093 | expose_sku_component | 10001042 | 1699936154050a1b33cce-f249-4112-8db2-288f1f0f6703 | NULL | NULL | NULL | 274012627 | 10742100529761108 | NULL | 9090 | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | 9991,6758,4834,6119,9996 | 苏州木渎DC | sams-app-sdk | NULL | NULL | NULL | NULL | NULL | NULL | HomeFragment | 首页 | NULL | NULL | NULL | NULL | NULL | NULL | NULL | ***** | 20231114 | default | NULL | advantage | 普通会员 | NULL | +------------+------------------+------------+--------------------------------------+---------------+---------------+----------------------+----------+---------------------------------------------------+-----------+------------+-------------+------------------+-------------------+----------+--------------------------------------+----------------+---------------+----------------------+--------------+-----------------+----------------------+---------+-------------------+--------------------------+--------------------------------------------------------------------------------------------------------------------+--------------+-------------+----------------+-------------+--------------+------+------------+--------------+-----------+----------+---------+------+----------+---------+----------+-----------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------+----------------+---------------+-----------+--------------+------------+ 2 rows in set (1.660 sec)
速率不断太慢，Datax也就这样了：
2023-12-11 17:52:11.251 [job-0] INFO StandAloneJobContainerCommunicator - Total 3906464 records, 7347149221 bytes | Speed 52.38MB/s, 29184 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 0.202s | All Task WaitReaderTime 297.463s | Percentage 0.00% 2023-12-11 17:52:21.252 [job-0] INFO StandAloneJobContainerCommunicator - Total 4198304 records, 7896134390 bytes | Speed 52.36MB/s, 29184 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 0.215s | All Task WaitReaderTime 331.832s | Percentage 0.00%
备注
之前有伙伴问到，为什么我的json里面字段用到“index”
"column": [ { "name": "ts", "type": "string", "value": "2023-11-14" }, { "name": "import_ds_", "type": "string", "index": 0 }, { "name": "unique_action_id", "type": "string", "index": 1 }, { "name": "action_time", "type": "string", "index": 2 },
.... ]
这个地方经过尝试，原有的column1，column2，column3...的方式测试行不通， 因为我有个ts的字段需要造值。但如果使用
{ "name": "ts", "type": "string", "value": "2023-11-14" }, { "name": "import_ds_", "type": "string", },
....
这种方式， 则抛出：由于您配置了type, 则至少需要配置 index 或 value，这是一件令人头疼的事。
[svccnetlhs@HOST log]<231211 18:10:01>$ /usr/bin/python2.7 /u/chengken/datax/bin/datax.py --jvm="-Xms1G -Xmx4G" /home/svccnetlhs/chengken/starrocks/json/20231114_sam_dwd_user_action_cos__1702288181.json DataX (DATAX-OPENSOURCE-3.0), From Alibaba ! Copyright (C) 2010-2017, Alibaba Group. All Rights Reserved. 2023-12-11 18:10:30.213 [main] INFO MessageSource - JVM TimeZone: GMT+08:00, Locale: zh_CN 2023-12-11 18:10:30.215 [main] INFO MessageSource - use Locale: zh_CN timeZone: sun.util.calendar.ZoneInfo[id="GMT+08:00",offset=28800000,dstSavings=0,useDaylight=false,transitions=0,lastRule=null] 2023-12-11 18:10:30.226 [main] INFO VMInfo - VMInfo# operatingSystem class => sun.management.OperatingSystemImpl 2023-12-11 18:10:30.231 [main] INFO Engine - the machine info => osInfo: Linux amd64 4.18.0-348.7.1.el8_5.x86_64 jvmInfo: Oracle Corporation 1.8 25.112-b15 cpu num: 16 totalPhysicalMemory: -0.00G freePhysicalMemory: -0.00G maxFileDescriptorCount: -1 currentOpenFileDescriptorCount: -1 GC Names [PS MarkSweep, PS Scavenge] MEMORY_NAME | allocation_size | init_size PS Eden Space | 1,280.00MB | 256.00MB Code Cache | 240.00MB | 2.44MB Compressed Class Space | 1,024.00MB | 0.00MB PS Survivor Space | 42.50MB | 42.50MB PS Old Gen | 2,731.00MB | 683.00MB Metaspace | -0.00MB | 0.00MB 2023-12-11 18:10:30.258 [main] INFO PerfTrace - PerfTrace traceId=job_-1, isEnable=true 2023-12-11 18:10:30.258 [main] INFO JobContainer - DataX jobContainer starts job. 2023-12-11 18:10:30.259 [main] INFO JobContainer - Set jobId = 0 2023-12-11 18:10:30.269 [job-0] INFO HdfsReader$Job - init() begin... 2023-12-11 18:10:30.274 [job-0] ERROR JobContainer - Exception when job run com.alibaba.datax.common.exception.DataXException: Code:[HdfsReader-06], Description:[没有 Index]. - 由于您配置了type, 则至少需要配置 index 或 value at com.alibaba.datax.common.exception.DataXException.asDataXException(DataXException.java:30) ~[datax-common-0.0.1-SNAPSHOT.jar:na] at com.alibaba.datax.plugin.reader.hdfsreader.HdfsReader$Job.validateColumns(HdfsReader.java:150) ~[hdfsreader-0.0.1-SNAPSHOT.jar:na] at com.alibaba.datax.plugin.reader.hdfsreader.HdfsReader$Job.validate(HdfsReader.java:111) ~[hdfsreader-0.0.1-SNAPSHOT.jar:na] at com.alibaba.datax.plugin.reader.hdfsreader.HdfsReader$Job.init(HdfsReader.java:50) ~[hdfsreader-0.0.1-SNAPSHOT.jar:na] at com.alibaba.datax.core.job.JobContainer.initJobReader(JobContainer.java:673) ~[datax-core-0.0.1-SNAPSHOT.jar:na] at com.alibaba.datax.core.job.JobContainer.init(JobContainer.java:303) ~[datax-core-0.0.1-SNAPSHOT.jar:na] at com.alibaba.datax.core.job.JobContainer.start(JobContainer.java:113) ~[datax-core-0.0.1-SNAPSHOT.jar:na] at com.alibaba.datax.core.Engine.start(Engine.java:86) [datax-core-0.0.1-SNAPSHOT.jar:na] at com.alibaba.datax.core.Engine.entry(Engine.java:168) [datax-core-0.0.1-SNAPSHOT.jar:na] at com.alibaba.datax.core.Engine.main(Engine.java:201) [datax-core-0.0.1-SNAPSHOT.jar:na] 2023-12-11 18:10:30.277 [job-0] INFO StandAloneJobContainerCommunicator - Total 0 records, 0 bytes | Speed 0B/s, 0 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 0.000s | All Task WaitReaderTime 0.000s | Percentage 0.00% 2023-12-11 18:10:30.279 [job-0] ERROR Engine - 经DataX智能分析,该任务最可能的错误原因是: com.alibaba.datax.common.exception.DataXException: Code:[HdfsReader-06], Description:[没有 Index]. - 由于您配置了type, 则至少需要配置 index 或 value at com.alibaba.datax.common.exception.DataXException.asDataXException(DataXException.java:30) at com.alibaba.datax.plugin.reader.hdfsreader.HdfsReader$Job.validateColumns(HdfsReader.java:150) at com.alibaba.datax.plugin.reader.hdfsreader.HdfsReader$Job.validate(HdfsReader.java:111) at com.alibaba.datax.plugin.reader.hdfsreader.HdfsReader$Job.init(HdfsReader.java:50) at com.alibaba.datax.core.job.JobContainer.initJobReader(JobContainer.java:673) at com.alibaba.datax.core.job.JobContainer.init(JobContainer.java:303) at com.alibaba.datax.core.job.JobContainer.start(JobContainer.java:113) at com.alibaba.datax.core.Engine.start(Engine.java:86) at com.alibaba.datax.core.Engine.entry(Engine.java:168) at com.alibaba.datax.core.Engine.main(Engine.java:201)
那么我是如何取得每个字段精准的index？
这里我用到了 orc-tools-1.8.0-uber.jar 这个包把orc里面的字段先解析出来，
下载：https://repo1.maven.org/maven2/org/apache/orc/orc-tools/
java -jar orc-tools-1.8.0-uber.jar meta <ORC文件>
成功解析orc文件的元数据字段信息，Type: struct：代表的就是字段列与下标顺序。
log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.Shell). log4j:WARN Please initialize the log4j system properly. log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info. Processing data file part-01291-c9231ae0-6186-4b4a-83dc-c95521cf2b8d-c000 [length: 290734159] Structure for part-01291-c9231ae0-6186-4b4a-83dc-c95521cf2b8d-c000 File Version: 0.12 with ORC_14 by ORC Java 1.6.14 Rows: 849920 Compression: SNAPPY Compression size: 131072 Calendar: Julian/Gregorian Type: struct<import_ds_:int,unique_action_id:string,action_time:bigint,report_time:bigint,action_type:string,ka_id:bigint,action_session_id:string,uuid:string,wx_app_id:string,wx_open_id:string,wx_union_id:string,external_user_id:string,merber_id:string,local_id:string,encrypted_imei:string,encrypted_idfa:string,encrypted_mac:string,encrypted_android_id:string,encrypted_qq:string,encrypted_phone:string,encrypting_algorithm:string,chan_id:string,chan_refer_app_id:string,chan_shop_id:string,chan_shop_name:string,client_type:string,client_name:string,client_version:string,sdk_version:string,device_model:string,ip:string,user_agent:string,page_path:string,page_name:string,referrer:string,address:string,city:string,province:string,country:string,latitude:string,longitude:string,json_properties:string,fdate:string,tag_id:string,tag_name:string,chan_custom_id:string,etl_load_time:string> Stripe Statistics:
完。