The Teradata Connector for Hadoop (TDCH) is a map-reduce application that supports high-performance parallel bi-directional data movement between Teradata systems and various Hadoop ecosystem components.

Overview

The Teradata Connector for Hadoop (Command Line Edition) is freely available and provides the following capabilities:

o    End-user tool with its own CLI (Command Line Interface).

o    Designed and implemented for the Hadoop user audience. 

o    Provides a Java API, enabling integration by 3rd parties as part of of an end-user tool. Hadoop vendors such as Hortonworks, Cloudera, IBM and MapR use TDCH's Java API in their respective Sqoop implementations, which are distributed and supported by the Hadoop vendors themselves. There is a Java API document available upon request.

o    Includes an installation script which setups up TDCH such that it can be launched remotely by Teradata Studio's Smart Loader for Hadoop and Teradata DataMover. For more information about these products see: 

·         Smart Loader for Hadoop article

·         Teradata Studio 14.02 now available article 

Need Help? 

For more detailed information on the Teradata Connector for Hadoop, please see the attached Tutorial document as well as the README file in the appropriate TDCH download packages.  The download packages are for use on commodity hardware.  For Teradata Hadoop Appliance hardware, it will be distributed with the appliance.  TDCH is supported by Teradata CS in certain situations where the user is a Teradata customer.

Teradata Connector for Hadoop 1.5.1 is now available.

For more information about Hadoop Product Management (PM), Teradata employees can go to Teradata Connections Hadoop PM.

Discussion
thedba 10 comments Joined 03/10
09 May 2013

I get this error when trying to run an import from teradata to HDFS .. any idea how to fix this ?

13/05/09 17:29:40 INFO tool.TeradataImportTool: TeradataImportTool starts at 1368120580654

java.io.FileNotFoundException: File -url does not exist.

at org.apache.hadoop.util.GenericOptionsParser.validateFiles(GenericOptionsParser.java:379)

at org.apache.hadoop.util.GenericOptionsParser.processGeneralOptions(GenericOptionsParser.java:275)

at org.apache.hadoop.util.GenericOptionsParser.parseGeneralOptions(GenericOptionsParser.java:413)

at org.apache.hadoop.util.GenericOptionsParser.<init>(GenericOptionsParser.java:164)

at org.apache.hadoop.util.GenericOptionsParser.<init>(GenericOptionsParser.java:147)

at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:59)

at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)

at com.teradata.hadoop.tool.TeradataImportTool.main(TeradataImportTool.java:369)

13/05/09 17:29:41 INFO tool.TeradataImportTool: job completed with exit code 10000
 
Command:
hadoop com.teradata.hadoop.tool.TeradataImportTool -libjars $LIB_JARS -url jdbc:teradata://myserver/database=mydb -username user -password password -jobtype hdfs -sourcetable example1_td -nummappers 1 -separator ',' -targetpaths /user/mapred/ex1_hdfs -method split.by.hash -splitbycolumn c1

thedba 10 comments Joined 03/10
09 May 2013

Please igore the above, it is resolved.

dd2teradata 5 comments Joined 10/09
30 May 2013

I got the following error message when running the command to import data from Teradata to Hadoop.

mapred@sdll6060:/> hadoop com.teradata.hadoop.tool.TeradataImportTool -classname com.teradata.jdbc.TeraDriver -url jdbc:teradata://pitd/DATABASE=hadoop_user1 -username hadoop_user1 -password hadoop_user1 -jobtype hdfs -fileformat textfile -method split.by.hash -separator "," -sourcetable sales_transaction -targetpaths /user/davidd/sales_transaction

13/05/30 13:51:06 INFO tool.TeradataImportTool: TeradataImportTool starts at 1369947066274

13/05/30 13:51:06 INFO mapreduce.TeradataInputProcessor: job setup starts at 1369947066893

13/05/30 13:51:08 INFO mapreduce.TeradataInputProcessor: job setup ends at 1369947068339

13/05/30 13:51:08 INFO mapreduce.TeradataInputProcessor: job setup time is 1s

com.teradata.hadoop.exception.TeradataHadoopSQLException: com.teradata.jdbc.jdbc_4.util.JDBCException: [Teradata Database] [TeraJDBC 14.10.00.21] [Error 6706] [SQLState HY000] The string contains an untranslatable character.

        at com.teradata.jdbc.jdbc_4.util.ErrorFactory.makeDatabaseSQLException(ErrorFactory.java:307)

        at com.teradata.jdbc.jdbc_4.statemachine.ReceiveInitSubState.action(ReceiveInitSubState.java:108)

        at com.teradata.jdbc.jdbc_4.statemachine.StatementReceiveState.subStateMachine(StatementReceiveState.java:321)

        at com.teradata.jdbc.jdbc_4.statemachine.StatementReceiveState.action(StatementReceiveState.java:202)

        at com.teradata.jdbc.jdbc_4.statemachine.StatementController.runBody(StatementController.java:123)

        at com.teradata.jdbc.jdbc_4.statemachine.StatementController.run(StatementController.java:114)

        at com.teradata.jdbc.jdbc_4.TDStatement.executeStatement(TDStatement.java:381)

        at com.teradata.jdbc.jdbc_4.TDStatement.executeStatement(TDStatement.java:323)

        at com.teradata.jdbc.jdbc_4.TDStatement.doNonPrepExecuteQuery(TDStatement.java:311)

        at com.teradata.jdbc.jdbc_4.TDStatement.executeQuery(TDStatement.java:1087)

        at com.teradata.hadoop.db.TeradataConnection.getColumnDescsForTable(TeradataConnection.java:966)

        at com.teradata.hadoop.mapreduce.TeradataInputProcessor.setupSchemaMapping(TeradataInputProcessor.java:215)

        at com.teradata.hadoop.mapreduce.TeradataSplitByHashInputProcessor.setupSchemaMapping(TeradataSplitByHashInputProcessor.java:85)

        at com.teradata.hadoop.mapreduce.TeradataInputProcessor.setup(TeradataInputProcessor.java:53)

        at com.teradata.hadoop.mapreduce.TeradataSplitByHashInputProcessor.setup(TeradataSplitByHashInputProcessor.java:51)

        at com.teradata.hadoop.job.TeradataImportJob.runJob(TeradataImportJob.java:86)

        at com.teradata.hadoop.tool.TeradataJobRunner.runImportJob(TeradataJobRunner.java:119)

        at com.teradata.hadoop.tool.TeradataImportTool.run(TeradataImportTool.java:39)

        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)

        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)

        at com.teradata.hadoop.tool.TeradataImportTool.main(TeradataImportTool.java:369)

 

        at com.teradata.hadoop.mapreduce.TeradataInputProcessor.setupSchemaMapping(TeradataInputProcessor.java:217)

        at com.teradata.hadoop.mapreduce.TeradataSplitByHashInputProcessor.setupSchemaMapping(TeradataSplitByHashInputProcessor.java:85)

        at com.teradata.hadoop.mapreduce.TeradataInputProcessor.setup(TeradataInputProcessor.java:53)

        at com.teradata.hadoop.mapreduce.TeradataSplitByHashInputProcessor.setup(TeradataSplitByHashInputProcessor.java:51)

        at com.teradata.hadoop.job.TeradataImportJob.runJob(TeradataImportJob.java:86)

        at com.teradata.hadoop.tool.TeradataJobRunner.runImportJob(TeradataJobRunner.java:119)

        at com.teradata.hadoop.tool.TeradataImportTool.run(TeradataImportTool.java:39)

        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)

        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)

        at com.teradata.hadoop.tool.TeradataImportTool.main(TeradataImportTool.java:369)

13/05/30 13:51:08 INFO tool.TeradataImportTool: job completed with exit code 10000

mapred@sdll6060:/>

 
The command is:
hadoop com.teradata.hadoop.tool.TeradataImportTool -classname com.teradata.jdbc.TeraDriver -url jdbc:teradata://pitd/DATABASE=hadoop_user1 -username hadoop_user1 -password hadoop_user1 -jobtype hdfs -fileformat textfile -method split.by.hash -separator "," -sourcetable sales_transaction -targetpaths /user/davidd/sales_transaction
 

31 May 2013

My requirement is to process huge volumes of data in hadoop (HDFS), do trasnformations and copy the transformed data to Teradata.
Can you please share the user manual to move data from HDFS to Teradata using Java API or Sqoop? Thanks in adavance.
 

hcnguyen 10 comments Joined 10/12
05 Aug 2013

Manikandan:
Have you seen the Teradata Connector for Hadoop Tutorial that is attached?  Sqoop is open source so you can get more information here:  http://sqoop.apache.org/
Have you tried using the Teradata Connector for Hadoop (Command Line Edition)?   The Teradata Connector for Hadoop (Sqoop Integration Edition) is designed for hadoop distribution vendors.  You can download the product with manuals from the hadoop distribution's website.
 

araghava 1 comment Joined 10/12
03 Jun 2013

Hi ,
I am trying to understand the TDCH better and wanted to know if I would be able to load a non-empty table in Teradata from a hive table using the utility? or would I have to stage it in an empty table first( because the method is fastload) and then  load to final target.

Thanks,
Anand
 

dd2teradata 5 comments Joined 10/09
04 Jun 2013

Problem:com.teradata.hadoop.exception.TeradataHadoopSQLException: com.teradata.jdbc.jdbc_4.util.JDBCException: [Teradata Database] [TeraJDBC 14.10.00.21] [Error 6706] [SQLState HY000] The string contains an untranslatable character.
Issue resolved by setting the internal flag of the Teradata 104 to true.
Flag value explained by Mark Li:
There is a flag “acceptreplacementCharacters” in the database, and its meaning is:
AcceptReplacementCharacters - Allow invalid characters to converted to the replacement character, rather than rejected.           
     FALSE (default): do not accept the replacement character, return error.                                          
     TRUE           : accept the replacement character.     
 
A tpareset is required.
Also, either run as "hdfs" (su hdfs) or make sure mapred has write access to output target folder of '/user/mapred' and it has to exist.  Teradata Hadoop VM template did not seem to create this folder, but there was a folder of '/mapred'.  You have to create it yourself and give proper write permission.  To list and check for folder content do:
hadoop fs -ls /user
or 
hadoop fs -ls /
 

thedba 10 comments Joined 03/10
10 Jun 2013

I was unable to find any document around the "-conf <conf file>" option below.

13/06/10 19:51:26 INFO tool.TeradataExportTool: TeradataExportTool starts at 1370919086995

hadoop jar teradata-hadoop-connector.jar

    com.teradata.hadoop.tool.TeradataExportTool

    [-conf <conf file>] (optional)

 

Can you please point me to place which has more info or provide me with more here ?

 

Thanks !

Mark_tdch 7 comments Joined 06/13
12 Jun 2013

Hi, I'm Mark, from Teradata Connector Sustaining Team.
"-conf" option is used for hadoop command exclusively. User can set their hadoop options in a file, and use "-conf" to specify this file to pass all of his options to hadoop. You can refer to hadoop manual for more information.
 
Thanks!

pankajparashar 2 comments Joined 05/13
13 Jun 2013

Hi,
I have been looking at ways to do a batch data export from TD to HADOOP.
We used sqoop connector but the perfromance was not as good as native FastExport to a falt file.
Are there any performance benchmarks available for the sqoop connector?
Thanks!

Mark_tdch 7 comments Joined 06/13
16 Jun 2013

Hi,
I wonder if you use the teradata connector or use sqoop to connect database directly. And if you use the connector, there are different import method(we call transferration from TD to HADOOP as import, from hadoop perspective). Please provide us with your specific sqoop connector command, and table schema for better support.
You can send the info to my email: Mark.Li@teradata.com.
Thanks.

hcnguyen 10 comments Joined 10/12
05 Aug 2013

Pankaj:
Which Sqoop connector are you referring to?  Hortonworks or Cloudera?  Did you try to use TDCH (Command Line Edition)?

VVenkat 8 comments Joined 02/11
17 Jun 2013

 Hi,
I am trying to import data using Teradata Studio to TD 13.0 vm instance, followed the steps for install for the connector I was able to connect to Hadoop  default DB  however I get the following error when I import data to TD 13.0 .. appreciate any insight..
Exception in thread "main" java.lang.NoClassDefFoundError: com/teradata/hadoop/tool/TeradataExportTool
Caused by: java.lang.ClassNotFoundException: com.teradata.hadoop.tool.TeradataExportTool
 at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
 at java.security.AccessController.doPrivileged(Native Method)
 at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
 at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
 at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
 at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
Could not find the main class: com.teradata.hadoop.tool.TeradataExportTool.  Program will exit.
Thanks
Venkat

Mark_tdch 7 comments Joined 06/13
17 Jun 2013

Did you specify connector jar file using "jar" option in the command? Could you show me you whole hadoop command?

VVenkat 8 comments Joined 02/11
18 Jun 2013

I was trying with smart loader for hadoop feature in Teradata Studio 14 not using command line. I apologise I should have posted this on the Studio forum instead, which I just did @ http://developer.teradata.com/tools/articles/smart-loader-for-hadoop

bhusabh 3 comments Joined 06/13
21 Jun 2013

"There is a Java API document available upon request." Can you please let me know whom to contact for getting this documentation also i have used cloudera sqoop teradata connector, it lacks many features that we can use with "Teradata Connector for Hadoop CLI" . 
In nutshell if we want to use Teradata Connector for Hadoop in java wrappers using java API what approach can we follow and where we can get documentation or example codes

hcnguyen 10 comments Joined 10/12
05 Aug 2013

Abhishek,
Please provide your email address.  The Cloudera Connector for Teradata doesn't use TDCH.  The new Cloudera Connector Powered by Teradata does.  It should be out on Cloudera's website soon.
 

bhusabh 3 comments Joined 06/13
21 Jun 2013

Hau,
My email address is abhishek.bhushan.verma@gmail.com

bhusabh 3 comments Joined 06/13
25 Jun 2013

Hau,
I did not recieve any mail from you about the documentations, Can you please send the same
Thanks
Abhishek

hcnguyen 10 comments Joined 10/12
05 Aug 2013

Abhishek,
Please check your email.  I have sent you an email to discuss.

nancyjean 7 comments Joined 02/13
01 Jul 2013

Hi,
 
Can you please explain the command line edition of TDCH. According to the manual... where should i extract the jar file? in hadoop or teradata?
 

nancyjean 7 comments Joined 02/13
01 Jul 2013

Hi i could not find any tar.gz file... all i could get was  a jar file and xml files.

Mark_tdch 7 comments Joined 06/13
01 Jul 2013

Hi, TDCH is published with tar.gz file. What you've got may be the files after extraction. Please confirm the jar file name and xml files from the README.

nancyjean 7 comments Joined 02/13
01 Jul 2013

Hi, Can you please tell which TDCH will support HDP1.1.1.16?
Which connector should i go for?

nancyjean 7 comments Joined 02/13
01 Jul 2013

Hi,
 
These are the files I am having,
teradata-connector-1.0.6.jar
             a) teradata-export-properties.xml
             b) teradata-import-properties.xml

Mark_tdch 7 comments Joined 06/13
02 Jul 2013

Recently, we have an investigation of the following usage in hadoop to find if we should imporove our product to support following cases. Has anybody has the requirement to run a hive job on the hadoop slave node(not hive node), through the other tools or applicatons? like templeton.  

hcnguyen 10 comments Joined 10/12
05 Aug 2013

TDCH has not been certified with HDP 1.1.1.16.  We have certified with HDP 1.1.0.17.  The next certification in the pipeline is HDP 1.3.

pankajparashar 2 comments Joined 05/13
10 Jul 2013

Hi Hau,
SQOOP TD Connector I had used was from Cloudera - version 1.0.5.
I have not tried TDCH command line yet. Is that recommended over sqoop with TD connector? Is that expected to be faster? If yes, why?

hcnguyen 10 comments Joined 10/12
05 Aug 2013

There is a new version, Cloudera Connector Powered by Teradata, that you may want to try out...it takes advantage of TDCH.  It's available on Cloudera's website for download.
It's a matter of preference if you prefer to use Sqoop Command Line with the Cloudera Connector Powered by Teradata or Command Line with TDCH.  Since the Cloudera Connector Powered by Teradata uses TDCH, the performance is similar to TDCH command line.
-Hau

hcnguyen 10 comments Joined 10/12
05 Aug 2013

I would like to know who is using TDCH, and what stage people are in with respect to their deployment.
Please send me an email (hau.nguyen@teradata.com) and let me know. Please indicate customer name if you are a Teradata customer.
Thanks,
-Hau

31 Jul 2013

I have a couple of situations
1. Teradata 13.10.6 and HDP 1.0.X
2. Teradata 13.10.6 and HDP 1.3
 
Which connectors should i be using which would suit the above versions?
 
What is the difference between the command line edition and the sqoop integration edition?
Please help.

hcnguyen 10 comments Joined 10/12
05 Aug 2013

The Sqoop Integration Edition is for Hadoop distributions to use to integrate with Sqoop.  For example, Hortonworks has used it to create the "Hortonworks Connector for Teradata".  Cloudera has used it to create the "Cloudera Connector Powered by Teradata".  The products use the Sqoop Command Line.
 
The Teradata Connector for Hadoop (Command Line Edition), doesn't use Sqoop Command Line, just command line.
 
Teradata Connector for Hadoop is currently certified with HDP 1.1.0.17 and HDP 1.3.
 
-Hau

jackliu 1 comment Joined 09/13
23 Sep 2013

Test it in Hadoop 2.1.0, failed on incompatible interface.
Exception:

java.lang.IncompatibleClassChangeError: Found interface org.apache.hadoop.mapreduce.JobContext, but class was expected

        at com.teradata.hadoop.mapreduce.TeradataInputFormat.getSplits(TeradataInputFormat.java:131)

        at org.apache.hadoop.mapreduce.JobSubmitter.writeNewSplits(JobSubmitter.java:476)

        at org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:493)

        at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:390)

        at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1268)

 

Could you please share wether there is any time plan to support hadoop2+?
 

sguthi0910 4 comments Joined 10/13
31 Oct 2013

hi..I'm getting the following error message when running the command to import data from Teradata to Hadoop.
 
13/10/31 15:37:25 INFO tool.TeradataImportTool: TeradataImportTool starts at 1383248245608

13/10/31 15:37:25 INFO mapreduce.TeradataInputProcessor: job setup starts at 1383248245829

13/10/31 15:37:28 INFO db.TeradataConnection: CREATE MULTISET TABLE "_033728", DATABLOCKSIZE = 130048 BYTES, NO FALLBACK, NO BEFORE JOURNAL, NO AFTER JOURNAL, CHECKSUM = DEFAULT ("c1" CHAR(2) CHARACTER SET LATIN NOT CASESPECIFIC NULL, "c2" CHAR(4) CHARACTER SET LATIN NOT CASESPECIFIC NOT NULL, "c3" VARCHAR(10) CHARACTER SET LATIN NOT CASESPECIFIC NULL, "c4" VARCHAR(10) CHARACTER SET LATIN NOT CASESPECIFIC NULL, "c5" DECIMAL(5, 2) NULL, "c6" DECIMAL(5, 2) NULL, "c7" DECIMAL(5, 2) NULL, "c8" DECIMAL(5, 2) NULL, "c9" DECIMAL(5, 2) NULL, "c10" VARCHAR(10) CHARACTER SET LATIN NOT CASESPECIFIC NULL, "c11" DATE NULL, "c12" VARCHAR(5) CHARACTER SET LATIN NOT CASESPECIFIC NULL, "c13" DATE NULL, "c14" INTEGER NULL, "c15" INTEGER NULL, "c16" INTEGER NULL, "c17" VARCHAR(7) CHARACTER SET LATIN NOT CASESPECIFIC NULL, "c18" BYTEINT NULL, "c19" VARCHAR(10) CHARACTER SET LATIN NOT CASESPECIFIC NOT NULL, "c20" VARCHAR(25) CHARACTER SET LATIN NOT CASESPECIFIC NULL, "c21" VARCHAR(10) CHARACTER SET LATIN NOT CASESPECIFIC NULL, "c22" VARCHAR(10) CHARACTER SET LATIN NOT CASESPECIFIC NULL, "c23" CHAR(1) CHARACTER SET LATIN NOT CASESPECIFIC NULL, "c24" DECIMAL(5, 2) NULL, "c25" DECIMAL(5, 2) NULL, "c26" DECIMAL(5, 2) NULL, "c27" DECIMAL(5, 2) NULL, "c28" DECIMAL(5, 2) NULL, "c29" DATE NULL, "c30" VARCHAR(5) CHARACTER SET LATIN NOT CASESPECIFIC NULL, "c31" VARCHAR(8) CHARACTER SET LATIN NOT CASESPECIFIC NULL, "c32" VARCHAR(60) CHARACTER SET LATIN NOT CASESPECIFIC NULL, "c33" VARCHAR(13) CHARACTER SET LATIN NOT CASESPECIFIC NULL, "c34" VARCHAR(10) CHARACTER SET LATIN NOT CASESPECIFIC NULL, "c35" VARCHAR(10) CHARACTER SET LATIN NOT CASESPECIFIC NULL, "c36" VARCHAR(10) CHARACTER SET LATIN NOT CASESPECIFIC NULL, "c37" VARCHAR(4) CHARACTER SET LATIN NOT CASESPECIFIC NULL, "c38" CHAR(1) CHARACTER SET LATIN NOT CASESPECIFIC NOT NULL, "c39" VARCHAR(9) CHARACTER SET LATIN NOT CASESPECIFIC NULL, "c40" VARCHAR(21) CHARACTER SET LATIN NOT CASESPECIFIC NULL, "c41" VARCHAR(40) CHARACTER SET LATIN NOT CASESPECIFIC NULL, "c42" VARCHAR(30) CHARACTER SET LATIN NOT CASESPECIFIC NULL, "c43" VARCHAR(10) CHARACTER SET LATIN NOT CASESPECIFIC NULL, "c44" DATE NULL, "c45" VARCHAR(5) CHARACTER SET LATIN NOT CASESPECIFIC NULL, "c46" SMALLINT NULL, "c47" VARCHAR(10) CHARACTER SET LATIN NOT CASESPECIFIC NULL, "c48" DATE NULL, "c49" VARCHAR(5) CHARACTER SET LATIN NOT CASESPECIFIC NULL, "c50" VARCHAR(10) CHARACTER SET LATIN NOT CASESPECIFIC NULL, "c51" BYTEINT NULL, "c52" DECIMAL(11, 2) NULL, "c53" DECIMAL(11, 2) NULL, "c54" BYTEINT NULL, "c55" DECIMAL(11, 2) NULL, "c56" DECIMAL(11, 2) NULL, "c57" VARCHAR(15) CHARACTER SET LATIN NOT CASESPECIFIC NULL, "c58" VARCHAR(4) CHARACTER SET LATIN NOT CASESPECIFIC NULL, "c59" VARCHAR(8) CHARACTER SET LATIN NOT CASESPECIFIC NULL, "c60" VARCHAR(40) CHARACTER SET LATIN NOT CASESPECIFIC NULL, "c61" VARCHAR(4) CHARACTER SET LATIN NOT CASESPECIFIC NULL, "c62" DECIMAL(12, 2) NOT NULL, "c63" DECIMAL(11, 2) NULL, "c64" CHAR(1) CHARACTER SET LATIN NOT CASESPECIFIC NULL, "c65" VARCHAR(8) CHARACTER SET LATIN NOT CASESPECIFIC NULL, "c66" VARCHAR(10) CHARACTER SET LATIN NOT CASESPECIFIC NULL, "c67" CHAR(2) CHARACTER SET LATIN NOT CASESPECIFIC NULL, "c68" VARCHAR(10) CHARACTER SET LATIN NOT CASESPECIFIC NULL, "c69" VARCHAR(10) CHARACTER SET LATIN NOT CASESPECIFIC NULL, "c70" DECIMAL(9, 2) NULL, "c71" VARCHAR(8) CHARACTER SET LATIN NOT CASESPECIFIC NULL, "c72" CHAR(1) CHARACTER SET LATIN NOT CASESPECIFIC NULL, "c73" VARCHAR(10) CHARACTER SET LATIN NOT CASESPECIFIC NULL, "c74" CHAR(1) CHARACTER SET LATIN NOT CASESPECIFIC NULL, "c75" VARCHAR(10) CHARACTER SET LATIN NOT CASESPECIFIC NULL, "c76" VARCHAR(10) CHARACTER SET LATIN NOT CASESPECIFIC NULL, "c77" VARCHAR(7) CHARACTER SET LATIN NOT CASESPECIFIC NULL, "c78" DATE NULL, "c79" VARCHAR(5) CHARACTER SET LATIN NOT CASESPECIFIC NULL, "c80" DATE NULL, "c81" VARCHAR(5) CHARACTER SET LATIN NOT CASESPECIFIC NULL, "c82" VARCHAR(10) CHARACTER SET LATIN NOT CASESPECIFIC NULL, "c83" DATE NULL, "c84" VARCHAR(5) CHARACTER SET LATIN NOT CASESPECIFIC NULL, "c85" VARCHAR(30) CHARACTER SET LATIN NOT CASESPECIFIC NULL, "c86" VARCHAR(8) CHARACTER SET LATIN NOT CASESPECIFIC NULL, "c87" VARCHAR(10) CHARACTER SET LATIN NOT CASESPECIFIC NULL, "c88" CHAR(1) CHARACTER SET LATIN NOT CASESPECIFIC NULL, "c89" VARCHAR(4) CHARACTER SET LATIN NOT CASESPECIFIC NULL, "c90" VARCHAR(10) CHARACTER SET LATIN NOT CASESPECIFIC NULL, "c91" BYTEINT NULL, "c92" DECIMAL(3, 1) NULL, "c93" DECIMAL(11, 2) NULL, "c94" DECIMAL(11, 2) NULL, "c95" VARCHAR(5) CHARACTER SET LATIN NOT CASESPECIFIC NULL, "c96" DATE NULL, "c97" INTEGER NULL, "c98" DATE NULL, "c99" VARCHAR(4) CHARACTER SET LATIN NOT CASESPECIFIC NULL, "c100" VARCHAR(8) CHARACTER SET LATIN NOT CASESPECIFIC NULL, "c101" VARCHAR(10) CHARACTER SET LATIN NOT CASESPECIFIC NULL, "c102" CHAR(2) CHARACTER SET LATIN NOT CASESPECIFIC NULL, "c103" CHAR(2) CHARACTER SET LATIN NOT CASESPECIFIC NULL, "c104" VARCHAR(4) CHARACTER SET LATIN NOT CASESPECIFIC NULL, "c105" VARCHAR(4) CHARACTER SET LATIN NOT CASESPECIFIC NULL, "c106" VARCHAR(30) CHARACTER SET LATIN NOT CASESPECIFIC NULL, "c107" DECIMAL(5, 1) NULL, "TDIN_PARTID" INTEGER NULL) PRIMARY INDEX ("TDIN_PARTID") PARTITION BY "TDIN_PARTID"

13/10/31 15:37:31 INFO mapreduce.TeradataSplitByPartitionInputProcessor: stage table "_033728" is created

13/10/31 15:37:31 INFO db.TeradataConnection: CREATE VIEW "V_033728" ("c1","c2","c3","c4","c5","c6","c7","c8","c9","c10","c11","c12","c13","c14","c15","c16","c17","c18","c19","c20","c21","c22","c23","c24","c25","c26","c27","c28","c29","c30","c31","c32","c33","c34","c35","c36","c37","c38","c39","c40","c41","c42","c43","c44","c45","c46","c47","c48","c49","c50","c51","c52","c53","c54","c55","c56","c57","c58","c59","c60","c61","c62","c63","c64","c65","c66","c67","c68","c69","c70","c71","c72","c73","c74","c75","c76","c77","c78","c79","c80","c81","c82","c83","c84","c85","c86","c87","c88","c89","c90","c91","c92","c93","c94","c95","c96","c97","c98","c99","c100","c101","c102","c103","c104","c105","c106","c107") AS SELECT top 10 * FROM wo_header1

13/10/31 15:37:31 INFO mapreduce.TeradataInputProcessor: job setup ends at 1383248251454

13/10/31 15:37:31 INFO mapreduce.TeradataInputProcessor: job setup time is 5s

13/10/31 15:37:31 ERROR tool.TeradataImportTool: com.teradata.hadoop.exception.TeradataHadoopSQLException: com.teradata.jdbc.jdbc_4.util.JDBCException: [Teradata Database] [TeraJDBC 14.00.00.01] [Error 3524] [SQLState 42000] The user does not have CREATE VIEW access to database tedw.

        at com.teradata.jdbc.jdbc_4.util.ErrorFactory.makeDatabaseSQLException(ErrorFactory.java:307)

        at com.teradata.jdbc.jdbc_4.statemachine.ReceiveInitSubState.action(ReceiveInitSubState.java:102)

        at com.teradata.jdbc.jdbc_4.statemachine.StatementReceiveState.subStateMachine(StatementReceiveState.java:298)

        at com.teradata.jdbc.jdbc_4.statemachine.StatementReceiveState.action(StatementReceiveState.java:179)

        at com.teradata.jdbc.jdbc_4.statemachine.StatementController.runBody(StatementController.java:120)

        at com.teradata.jdbc.jdbc_4.statemachine.StatementController.run(StatementController.java:111)

        at com.teradata.jdbc.jdbc_4.TDStatement.executeStatement(TDStatement.java:372)

        at com.teradata.jdbc.jdbc_4.TDStatement.executeStatement(TDStatement.java:314)

        at com.teradata.jdbc.jdbc_4.TDStatement.doNonPrepExecute(TDStatement.java:277)

        at com.teradata.jdbc.jdbc_4.TDStatement.execute(TDStatement.java:1087)

        at com.teradata.hadoop.db.TeradataConnection.executeDDL(TeradataConnection.java:374)

        at com.teradata.hadoop.db.TeradataConnection.createView(TeradataConnection.java:421)

        at com.teradata.hadoop.mapreduce.TeradataSplitByPartitionInputProcessor.setupDatabaseEnvironment(TeradataSplitByPartitionInputProcessor.java:296)

        at com.teradata.hadoop.mapreduce.TeradataInputProcessor.setup(TeradataInputProcessor.java:57)

        at com.teradata.hadoop.mapreduce.TeradataSplitByPartitionInputProcessor.setup(TeradataSplitByPartitionInputProcessor.java:67)

        at com.teradata.hadoop.job.TeradataImportJob.runJob(TeradataImportJob.java:86)

        at com.teradata.hadoop.tool.TeradataJobRunner.runImportJob(TeradataJobRunner.java:119)

        at com.teradata.hadoop.tool.TeradataImportTool.run(TeradataImportTool.java:41)

        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)

        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)

        at com.teradata.hadoop.tool.TeradataImportTool.main(TeradataImportTool.java:392)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

        at java.lang.reflect.Method.invoke(Method.java:597)

        at org.apache.hadoop.util.RunJar.main(RunJar.java:156)

 

        at com.teradata.hadoop.mapreduce.TeradataSplitByPartitionInputProcessor.setupDatabaseEnvironment(TeradataSplitByPartitionInputProcessor.java:304)

        at com.teradata.hadoop.mapreduce.TeradataInputProcessor.setup(TeradataInputProcessor.java:57)

        at com.teradata.hadoop.mapreduce.TeradataSplitByPartitionInputProcessor.setup(TeradataSplitByPartitionInputProcessor.java:67)

        at com.teradata.hadoop.job.TeradataImportJob.runJob(TeradataImportJob.java:86)

        at com.teradata.hadoop.tool.TeradataJobRunner.runImportJob(TeradataJobRunner.java:119)

        at com.teradata.hadoop.tool.TeradataImportTool.run(TeradataImportTool.java:41)

        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)

        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)

        at com.teradata.hadoop.tool.TeradataImportTool.main(TeradataImportTool.java:392)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

        at java.lang.reflect.Method.invoke(Method.java:597)

        at org.apache.hadoop.util.RunJar.main(RunJar.java:156)

 

13/10/31 15:37:31 INFO tool.TeradataImportTool: job completed with exit code 10000

 

 

 

 

Command I used is:

 

hadoop jar $USERLIBPATH com.teradata.hadoop.tool.TeradataImportTool -classname com.teradata.jdbc.TeraDriver -url "jdbc:teradata://XYZ/DATABASE=dbc" -username uname -password pswd -jobtype hdfs -fileformat textfile -sourcequery 'SELECT top 10 * FROM wo_header1'  -method split.by.partition -separator "," -targetpaths /user/hdfs/WO_Header1

thedba 10 comments Joined 03/10
31 Oct 2013

Ask your DBA to run "grant create view on tedw to uname; "

sguthi0910 4 comments Joined 10/13
04 Nov 2013

Hi. Is there any workaround to run the command without having create view access for the user? I tried using an userID which had create view access, but it is throwing me error " The user does not have CREATE TABLE access to database dbc". Does the userID requires both create table and create view access for the database to run this command? 

thedba 10 comments Joined 03/10
04 Nov 2013

Since you are using a sourcequery you will need create view and create table access. I think you should set the database to something other than DBC, probably tedw if you have create table and create view access.
Also if you are just trying to play with the tool instead of a sourcequery provide a source table, that way you should not need CT or CV access. 
Finally I think you should read this document Teradata Connector for Hadoop Tutorial - Teradata Developer ...
and if you are not familiar with how Teradata works, it would be helpful talk to your DBA and review this document with the DBA.

sguthi0910 4 comments Joined 10/13
06 Nov 2013

Hi,
Thank you thedba.
I got permissions from my user and I'm able to import the table to hdfs using teradata connector(version 1.0.9). Now I'm trying to import the data from teradata directly into hive table. I'm using the following exports and command.

Export commands

export HADOOP_HOME=/usr/lib/hadoop

 

export HIVE_HOME=/usr/lib/hive

 

export USERLIBTDCH=/usr/lib/tdch/teradata-connector-1.0.9.jar

 

export HADOOP_CLASSPATH=$HIVE_HOME/lib/hive-metastore-0.9.0.15.jar:$HIVE_HOME/lib/libthrift-0.7.0.jar:$HIVE_HOME/lib/hive-exec-0.9.0.15.jar:$HIVE_HOME/lib/libfb303-0.7.0.jar:$HIVE_HOME/lib/jdo2-api-2.3-ec.jar:$HIVE_HOME/conf:$HIVE_HOME/lib/slf4j-api-1.6.1.jar:$HIVE_HOME/lib/antlr-runtime-3.0.1.jar:$HIVE_HOME/lib/datanucleus-core-3.0.9.jar:$HIVE_HOME/lib/datanucleus-rdbms-3.0.8.jar:$HIVE_HOME/lib/datanucleus-connectionpool-2.0.3.jar:$HIVE_HOME/lib/mysql-connector-java.jar:$HIVE_HOME/lib/commons-dbcp-1.4.jar:$HIVE_HOME/lib/commons-pool-1.5.4.jar:$HIVE_HOME/lib/hive-cli-0.9.0.15.jar:$HIVE_HOME/lib/hive-builtins-0.9.0.15.jar

 

export HIVE_LIB_JARS=$HIVE_HOME/lib/hive-metastore-0.9.0.15.jar,$HIVE_HOME/lib/libthrift-0.7.0.jar,$HIVE_HOME/lib/hive-exec-0.9.0.15.jar,$HIVE_HOME/lib/libfb303-0.7.0.jar,$HIVE_HOME/lib/jdo2-api-2.3-ec.jar,$HIVE_HOME/lib/slf4j-api-1.6.1.jar,$HIVE_HOME/lib/hive-cli-0.9.0.15.jar,$HIVE_HOME/lib/hive-builtins-0.9.0.15.jar

 

Command for importing to Hive

hadoop jar $USERLIBTDCH com.teradata.hadoop.tool.TeradataImportTool -libjars $HIVE_LIB_JARS -classname com.teradata.jdbc.TeraDriver -url jdbc:teradata://XYZ/DATABASE=dbc -username uname -password pswd -jobtype hive -fileformat textfile -method split.by.hash -sourcetable dsstable -targettable dsstable

 

Error

 

13/11/06 15:12:43 ERROR tool.TeradataImportTool: com.teradata.hadoop.exception.TeradataHadoopException: Import Hive table's column schema is missing

        at com.teradata.hive.job.TeradataHiveImportJob.beforeJob(TeradataHiveImportJob.java:149)

        at com.teradata.hadoop.tool.TeradataJobRunner.runImportJob(TeradataJobRunner.java:118)

        at com.teradata.hadoop.tool.TeradataImportTool.run(TeradataImportTool.java:41)

        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)

        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)

        at com.teradata.hadoop.tool.TeradataImportTool.main(TeradataImportTool.java:392)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

        at java.lang.reflect.Method.invoke(Method.java:597)

        at org.apache.hadoop.util.RunJar.main(RunJar.java:156)

 

 

Do I need to create the table schema in hive before i import the table into hive? I read in the userguide of this connector that the command will create the table in the hive if the table does not exist. Is there any issue with my command?

 

 

 
 

sourabhpotnis 1 comment Joined 01/14
27 Jan 2014

Hi,
I am getting similar error like @thedba when i am trying to export data from HDFS to Teradata:

14/01/27 05:46:18 ERROR tool.TeradataExportTool: java.io.FileNotFoundException: File -username does not exist.

        at org.apache.hadoop.util.GenericOptionsParser.validateFiles(GenericOptionsParser.java:397)

        at org.apache.hadoop.util.GenericOptionsParser.processGeneralOptions(GenericOptionsParser.java:288)

        at org.apache.hadoop.util.GenericOptionsParser.parseGeneralOptions(GenericOptionsParser.java:431)

        at org.apache.hadoop.util.GenericOptionsParser.<init>(GenericOptionsParser.java:170)

        at org.apache.hadoop.util.GenericOptionsParser.<init>(GenericOptionsParser.java:153)

        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:64)

        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)

        at com.teradata.hadoop.tool.TeradataExportTool.main(TeradataExportTool.java:439)

 

14/01/27 05:46:18 INFO tool.TeradataExportTool: job completed with exit code 10000

 

Following is my command:

hadoop com.teradata.hadoop.tool.TeradataExportTool -libjars $LIB_JARS -url jdbc:teradata://Server_name/database=DB_name  -username user -password pwd -jobtype hdfs -sourcepaths /user/example2_hdfs/01 -nummappers 1 -separator ',' -targettable test1

Mark_tdch 7 comments Joined 06/13
27 Jan 2014

It looks like the parameter "-username" is treated as file name. Can you provide the value of $LIB_JARS, Server_name and DB_name for me to check if there's special characters?
#echo $LIB_JARS

Mark_tdch 7 comments Joined 06/13
27 Jan 2014

For hive table's schema missing issue, TDCH supports two ways to specify hive table's schema. If hive table exists, TDCH will get schema from the metastore, or "-sourcetableschema" parameter should be specified if you want the hive table to be created automatically.

KeithJones 5 comments Joined 07/10
03 Feb 2014

I'm a Teradata DBA trying to help some Hadoop users with the TDCH tool and it is new to me.  I see there is an internal.fastload option for invoking a distributed fastload which I'm guessing is using the TPT API to load data from Hadoop to Teradata across multiple Hadoop nodes.  However there appears to be no corresponding method for exporting from Teradata to Hadoop.  I would expect there to be an "internal.fastexport" option enabling exports distributed across muliple Hadoop nodes, similar to the way the Aster Data Connector for Teradata works.  The split.by.value, split.by.hash and split.by.partition options all seem inefficient on the Teradata side, each in its own way.  Are there plans to add an option to TDCH that uses the TPT API for pulling data into Hadoop?

thedba 10 comments Joined 03/10
04 Feb 2014

I second the request made by @KeithJones, adding a internal.fastexport like feature to TDCH for moving data out of Teradata would greatly simplify flow of data in that direction.

amala 1 comment Joined 02/10
13 Feb 2014

Hi
This is my first time using Teradata Import Tool. I downloaded the jar file and put it in sqoop folder and tried to import a teradata table to hadoop as given below but I get  [SQLState 28000] The UserId, Password or Account is invalid error eventhough everything I gave is correct and I'm able to login to SQL Assistant using the same credentials. Can anyone please advise as it is urgent for me to fix this issue.
export USERLIBTDCH=/usr/lib/sqoop/teradata-connector-1.1.1-hadoop200.jar
hadoop jar $USERLIBTDCH com.teradata.hadoop.tool.TeradataImportTool -classname com.teradata.jdbc.TeraDriver -url jdbc:teradata://a.b.c.net/DATABASE=XYZ-username ROOT -password XXXX -jobtype hdfs -fileformat textfile -method split.by.hash -separator "," -sourcetable ICDW_REG_QRY -targetpaths /user/ABC/TD_REG_QRY
 
14/02/13 00:36:28 ERROR tool.TeradataImportTool: com.teradata.hadoop.exception.TeradataHadoopException: com.teradata.jdbc.jdbc_4.util.JDBCException: [Teradata Database] [TeraJDBC 14.00.00.39] [Error 8017] [SQLState 28000] The UserId, Password or Account is invalid.
Thanks
 

RyanBerti 1 comment Joined 07/13
21 Mar 2014

Hi KeithJones and thedba,
 
TDCH does not use the TPT API under the covers; rather it uses JDBC fastload for import to Teradata and standard JDBC driver for export from Teradata. We have support for JDBC fastexport on our long term road map, but at this point we don't have a date / release version in which this feature will be available. Thanks
 
Hi amala
Looks like your login credentials for the DBS specified by the '-url' option are invalid? Can you verify that you're able to login to that DBS with your credentials using an application like BTEQ? Thanks

nirmal_hbti 2 comments Joined 04/14
10 Apr 2014

Hi All,
I am using teradata-connector-1.2.1 with the following versions:
->Teradata 14.0
->HDP 2.0(Hadoop 2.1.0.2.0.5.0-67)
->Hive 0.11
I am trying to import data from teradata tables to Hive.
 
/*Created table and inserted a row in teradata*/
CREATE MULTISET TABLE example3_td ( c1 INT,c2 VARCHAR(100));
INSERT INTO example3_td VALUES (3,'bar');

/*Created similar table in Hive*/ 

CREATE TABLE example3_hive (h1 INT, h2 STRING) STORED AS RCFILE;

 
Following are the set of commands I used after following the documentation:
export HIVE_HOME=/home/nirmal/hadoop/hive-0.11.0

export USERLIBTDCH=/home/nirmal/hadoop/TeradataConnectors/teradata-connector-1.2.1/lib/teradata-connector-1.2.1.jar

 

export HADOOP_CLASSPATH=$HIVE_HOME/lib/hive-metastore-0.11.0.jar:$HIVE_HOME/lib/libthrift-0.9.0.jar:$HIVE_HOME/lib/hive-exec-0.11.0.jar:$HIVE_HOME/lib/libfb303-0.9.0.jar:$HIVE_HOME/lib/jdo2-api-2.3-ec.jar:$HIVE_HOME/conf:$HIVE_HOME/lib/slf4j-api-1.6.1.jar:$HIVE_HOME/lib/antlr-runtime-3.4.jar:$HIVE_HOME/lib/datanucleus-core-3.0.9.jar:$HIVE_HOME/lib/datanucleus-rdbms-3.0.8.jar:$HIVE_HOME/lib/datanucleus-api-jdo-3.0.7.jar:$HIVE_HOME/lib/commons-dbcp-1.4.jar:$HIVE_HOME/lib/commons-pool-1.5.4.jar:$HIVE_HOME/lib/hive-cli-0.11.0.jar

 

export HIVE_LIB_JARS=$HIVE_HOME/lib/hive-metastore-0.11.0.jar,$HIVE_HOME/lib/libthrift-0.9.0.jar,$HIVE_HOME/lib/hive-exec-0.11.0.jar,$HIVE_HOME/lib/libfb303-0.9.0.jar,$HIVE_HOME/lib/jdo2-api-2.3-ec.jar,$HIVE_HOME/lib/slf4j-api-1.6.1.jar,$HIVE_HOME/lib/hive-cli-0.11.0.jar

 
hadoop jar $USERLIBTDCH com.teradata.hadoop.tool.TeradataImportTool -libjars $HIVE_LIB_JARS  -classname com.teradata.jdbc.TeraDriver -url jdbc:teradata://192.168.199.129/DATABASE=airlinesuser -username airlinesuser -password airlinesuser -jobtype hive -fileformat rcfile -sourcetable example3_td -nummappers 1 -targettable example3_hive

 

radata://192.168.199.129/DATABASE=airlinesuser -username airlinesuser -password airlinesuser -jobtype hive -fileformat rcfile -sourcetable example3_td -nummappers 1 -targettable example3_hive

14/04/10 21:14:56 INFO tool.TeradataImportTool: TeradataImportTool starts at 1397144696332

14/04/10 21:14:57 WARN conf.Configuration: mapred.map.tasks.speculative.execution is deprecated. Instead, use mapreduce.map.speculative

14/04/10 21:14:57 WARN conf.Configuration: mapred.max.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.maxsize

14/04/10 21:14:57 WARN conf.Configuration: mapred.min.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize

14/04/10 21:14:57 WARN conf.Configuration: mapred.min.split.size.per.rack is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize.per.rack

14/04/10 21:14:57 WARN conf.Configuration: mapred.min.split.size.per.node is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize.per.node

14/04/10 21:14:57 WARN conf.Configuration: mapred.reduce.tasks is deprecated. Instead, use mapreduce.job.reduces

14/04/10 21:14:57 WARN conf.Configuration: mapred.reduce.tasks.speculative.execution is deprecated. Instead, use mapreduce.reduce.speculative

14/04/10 21:14:57 WARN conf.Configuration: org.apache.hadoop.hive.conf.LoopingByteArrayInputStream@3eafdb52:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval;  Ignoring.

14/04/10 21:14:57 WARN conf.Configuration: org.apache.hadoop.hive.conf.LoopingByteArrayInputStream@3eafdb52:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.

14/04/10 21:14:57 ERROR tool.TeradataImportTool: com.teradata.hadoop.exception.TeradataHadoopException: Hive current user directory not exists

at com.teradata.hive.job.TeradataHiveImportJob.beforeJob(TeradataHiveImportJob.java:148)

at com.teradata.hadoop.tool.TeradataJobRunner.runImportJob(TeradataJobRunner.java:118)

at com.teradata.hadoop.tool.TeradataImportTool.run(TeradataImportTool.java:41)

at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)

at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)

at com.teradata.hadoop.tool.TeradataImportTool.main(TeradataImportTool.java:464)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:606)

at org.apache.hadoop.util.RunJar.main(RunJar.java:212)

 

14/04/10 21:14:57 INFO tool.TeradataImportTool: job completed with exit code 25008

nirmal@nirmal-Vostro-3560 ~/hadoop/hive-0.12.0 $ 
I Hive I see the table being created in the default database/schema:
hive> show tables;

OK

example3_hive

Time taken: 5.831 seconds, Fetched: 1 row(s)

hive> 
Kindly help me if I am missing something.
Thanks,
-Nirmal
 

suman01 4 comments Joined 07/12
02 May 2014

Teradata DB: 14.10
Sqoop Version: 1.4
CDH4
Cloudera Connector powered by Teradata 1.2c4
I'm using Sqoop Teradata connector, getting the follwing error. Please anyone can provide details. Thanks

ERROR tool.ImportTool: Encountered IOException running import job: com.teradata.hadoop.exception.TeradataHadoopException: Malformed \uxxxx encoding

        at com.teradata.hadoop.utils.TeradataUnicodeCharacterConverter.fromEncodedUnicode(TeradataUnicodeCharacterConverter.java:90)

        at com.teradata.hadoop.db.TeradataConfiguration.setInputEscapedByString(TeradataConfiguration.java:540)

        at com.cloudera.connector.teradata.imports.BaseImportJob.configureInputFormat(BaseImportJob.java:74)

        at com.cloudera.connector.teradata.imports.TableImportJob.configureInputFormat(TableImportJob.java:32)

        at org.apache.sqoop.mapreduce.ImportJobBase.runImport(ImportJobBase.java:239)

        at com.cloudera.connector.teradata.TeradataManager.importTable(TeradataManager.java:273)

        at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:413)

        at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:506)

        at org.apache.sqoop.Sqoop.run(Sqoop.java:147)

        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)

        at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183)

        at org.apache.sqoop.Sqoop.runTool(Sqoop.java:222)

        at org.apache.sqoop.Sqoop.runTool(Sqoop.java:231)

        at org.apache.sqoop.Sqoop.main(Sqoop.java:240)

purna88 1 comment Joined 05/14
25 May 2014

 ERROR sqoop.Sqoop: Got exception running Sqoop: java.lang.RuntimeException: Could not load db driver class: com.teradata.jdbc.TeraDriver
java.lang.RuntimeException: Could not load db driver class: com.teradata.jdbc.TeraDriver
        at org.apache.sqoop.manager.SqlManager.makeConnection(SqlManager.java:636)
        at org.apache.sqoop.manager.GenericJdbcManager.getConnection(GenericJdbcManager.java:52)
        at org.apache.sqoop.manager.SqlManager.execute(SqlManager.java:525)
        at org.apache.sqoop.manager.SqlManager.execute(SqlManager.java:548)
        at org.apache.sqoop.manager.SqlManager.getColumnTypesForRawQuery(SqlManager.java:191)
        at org.apache.sqoop.manager.SqlManager.getColumnTypes(SqlManager.java:175)
        at org.apache.sqoop.manager.ConnManager.getColumnTypes(ConnManager.java:262)
        at org.apache.sqoop.orm.ClassWriter.getColumnTypes(ClassWriter.java:1236)
        at org.apache.sqoop.orm.ClassWriter.generate(ClassWriter.java:1061)
        at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:82)
        at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:390)
        at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:476)
        at org.apache.sqoop.Sqoop.run(Sqoop.java:145)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
        at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:181)
        at org.apache.sqoop.Sqoop.runTool(Sqoop.java:220)
        at org.apache.sqoop.Sqoop.runTool(Sqoop.java:229)
        at org.apache.sqoop.Sqoop.main(Sqoop.java:238)
 

31 May 2014

Hi Hau Nguyen
I have a couple of questions regarding the Hadoop connectors:

  1. What connector do I need to integrate the Hadoop and Teradata using an ETL tool like Talend or Informatica PowerCenter?
  2. Can I use all of the 3 connectors side-by-side and deploy it in a sandbox environment?

Regards,
Joseph

05 Jun 2014

Guys, I need help regarding my inquery.

ariff 1 comment Joined 10/13
09 Jun 2014

Joseph - Talend and PowerCenter have their own methods for connecting to Hadoop and Teradata. Neither uses TDCH. You would need to work with each solution to figure out how they connect to Hadoop.
Since neither Talend or PowerCenter use TDCH, having TDCH on the same system as the Talend and PowerCenter connector should be possible.

10 Jun 2014

Thanks Ariff

kashwath 3 comments Joined 10/13
20 Jun 2014

Hi,
I get the following error when invoking TDCH from oozie. How ever the same command runs fine in commandline. What am i doing wrong?
 

Jun 19, 2014 4:54:38 PM com.teradata.hadoop.tool.TeradataImportTool main
INFO: TeradataImportTool starts at 1403211278263
Jun 19, 2014 4:54:38 PM com.teradata.hadoop.tool.TeradataImportTool main
SEVERE: java.lang.NoClassDefFoundError: org.apache.log4j.LogManager (initialization failure)
at java.lang.J9VMInternals.initialize(J9VMInternals.java:140)
at org.slf4j.impl.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:66)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:270)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:281)
at org.apache.hadoop.security.authentication.util.KerberosName.<clinit>(KerberosName.java:42)
at java.lang.J9VMInternals.initializeImpl(Native Method)
at java.lang.J9VMInternals.initialize(J9VMInternals.java:200)
at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:227)
at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:216)
at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:671)
at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:573)
at org.apache.hadoop.mapreduce.task.JobContextImpl.<init>(JobContextImpl.java:72)
at org.apache.hadoop.mapreduce.Job.<init>(Job.java:166)
at com.teradata.hadoop.tool.TeradataImportTool.run(TeradataImportTool.java:39)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
at com.teradata.hadoop.tool.TeradataImportTool.main(TeradataImportTool.java:464)

Jun 19, 2014 4:54:38 PM com.teradata.hadoop.tool.TeradataImportTool main
INFO: job completed with exit code 10000
Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.ShellMain], exit code [1]
Command that i used:
hadoop com.teradata.hadoop.tool.TeradataImportTool -url jdbc:teradata://TDDEV/DATABASE=XXXXXX -username XXXXXX -password XXXXXX -jobtype hive -fileformat textfile -sourcequery "select * from tablename" -targettable TMP_exp -method split.by.partition -hiveconf /biginsights/hive/conf/hive-site.xml -stagedatabase DB_WRK_HDOOP -stagetablename abcd

jimirwin 3 comments Joined 03/14
24 Jun 2014

I downloaded TDCH 1.3 and used it successfully to import from TD to Hive, and from TD to HDFS (csv).  However, when I try to import from TD to HDFS (Avro), I get the following exception in the MapReduce jobs:
Error: Found interface org.apache.hadoop.mapreduce.TaskAttemptContext, but class was expected
 
I'm using Hortonworks 2.1.  The log file is large, but there is no exception stack trace.  This is a portion of the log, which is repeated for each of the task attempts:
2014-06-24 15:29:05,324 FATAL [IPC Server handler 0 on 48577] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Task: attempt_1403546838254_0017_m_000000_1 - exited : Found interface org.apache.hadoop.mapreduce.TaskAttemptContext, but class was expected
2014-06-24 15:29:05,324 INFO [IPC Server handler 0 on 48577] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Diagnostics report from attempt_1403546838254_0017_m_000000_1: Error: Found interface org.apache.hadoop.mapreduce.TaskAttemptContext, but class was expected
2014-06-24 15:29:05,326 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1403546838254_0017_m_000000_1: Error: Found interface org.apache.hadoop.mapreduce.TaskAttemptContext, but class was expected
2014-06-24 15:29:05,329 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1403546838254_0017_m_000000_1 TaskAttempt Transitioned from RUNNING to FAIL_CONTAINER_CLEANUP
2014-06-24 15:29:05,330 INFO [ContainerLauncher #7] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_CLEANUP for container container_1403546838254_0017_01_000006 taskAttempt attempt_1403546838254_0017_m_000000_1
2014-06-24 15:29:05,330 INFO [ContainerLauncher #7] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: KILLING attempt_1403546838254_0017_m_000000_1
2014-06-24 15:29:05,333 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1403546838254_0017_m_000000_1 TaskAttempt Transitioned from FAIL_CONTAINER_CLEANUP to FAIL_TASK_CLEANUP
2014-06-24 15:29:05,334 INFO [CommitterEvent Processor #4] org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler: Processing the event EventType: TASK_ABORT
2014-06-24 15:29:05,344 WARN [CommitterEvent Processor #4] org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: Could not delete hdfs://hdp2.jri.revelytix.com:8020/user/jirwin/td_avro/_temporary/1/_temporary/attempt_1403546838254_0017_m_000000_1
2014-06-24 15:29:05,345 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1403546838254_0017_m_000000_1 TaskAttempt Transitioned from FAIL_TASK_CLEANUP to FAILED
2014-06-24 15:29:05,345 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved hdp2.jri.revelytix.com to /default-rack
2014-06-24 15:29:05,345 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1403546838254_0017_m_000000_2 TaskAttempt Transitioned from NEW to UNASSIGNED
2014-06-24 15:29:05,346 INFO [Thread-51] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Added attempt_1403546838254_0017_m_000000_2 to list of failed maps
2014-06-24 15:29:06,236 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before Scheduling: PendingReds:0 ScheduledMaps:2 ScheduledReds:0 AssignedMaps:2 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:6 ContRel:2 HostLocal:2 RackLocal:0
2014-06-24 15:29:06,248 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() for application_1403546838254_0017: ask=1 release= 0 newContainers=0 finishedContainers=1 resourcelimit=<memory:0, vCores:0> knownNMs=1
2014-06-24 15:29:06,249 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: Update the blacklist for application_1403546838254_0017: blacklistAdditions=0 blacklistRemovals=1
2014-06-24 15:29:06,249 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_1403546838254_0017_01_000005
2014-06-24 15:29:06,250 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1403546838254_0017_m_000001_1: Container killed by the ApplicationMaster.
Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
 

jimirwin 3 comments Joined 03/14
24 Jun 2014

With respect to my previous message, it appears that the problem is that HDP 2.1 is distributed with Hive 0.13, which does not have the same Avro jars as previous versions of Hive. 
In order to impprt/export Avro format using TDCH 1.3 on HDP 2.1, it is necessary to obtain the avro-1.7.4.jar and avro-mapred-1.7.4-hadoop2.jar from a prior version of Hive, such as Hive 0.12.  After obtaining those jar files, and adding them to the HADOOP_CLASSPATH and the -libjars, I was able to run TDCH 1.3 with Avro successfully.
By contrast, using the jars from the HDP 2.1 Hive 0.13, TDCH produces the following exception on the task:
2014-06-24 15:48:16,384 FATAL [main] org.apache.hadoop.mapred.YarnChild: Error running child : java.lang.IncompatibleClassChangeError: Found interface org.apache.hadoop.mapreduce.TaskAttemptContext, but class was expected
    at org.apache.avro.mapreduce.AvroKeyOutputFormat.getRecordWriter(AvroKeyOutputFormat.java:87)
    at com.teradata.connector.hdfs.HdfsAvroOutputFormat.getRecordWriter(HdfsAvroOutputFormat.java:44)
    at com.teradata.connector.common.ConnectorOutputFormat$ConnectorFileRecordWriter.<init>(ConnectorOutputFormat.java:84)
    at com.teradata.connector.common.ConnectorOutputFormat.getRecordWriter(ConnectorOutputFormat.java:33)
    at org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.<init>(MapTask.java:624)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:744)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1557)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
 

jimirwin 3 comments Joined 03/14
25 Jun 2014

I have a new issue with TDCH 1.3 command line edition and HCatalog.
 
Every time I attempt to import from TD into HCatalog, I get the following exception.  I've tried various distributions (CDH4.5, CDH5.0 GA, and HDP2.1), and they all get the same exception.  I've previously used TDCH 1.1 on CDH5.0 GA, with no problems.
 
14/06/25 15:18:27 INFO tool.ConnectorImportTool: ConnectorImportTool starts at 1403723907881
14/06/25 15:18:28 INFO common.ConnectorPlugin: load plugins in file:/tmp/hadoop-jirwin/hadoop-unjar3619433998773300551/teradata.connector.plugins.xml
14/06/25 15:18:28 INFO processor.TeradataInputProcessor: input preprocessor com.teradata.connector.teradata.processor.TeradataSplitByAmpProcessor starts at:  1403723908764
14/06/25 15:18:29 INFO utils.TeradataUtils: the input database product is Teradata
14/06/25 15:18:29 INFO utils.TeradataUtils: the input database version is 14.0
14/06/25 15:18:29 INFO utils.TeradataUtils: the jdbc driver version is 14.0
14/06/25 15:18:29 INFO processor.TeradataInputProcessor: the teradata connector for hadoop version is: 1.3
14/06/25 15:18:29 INFO processor.TeradataInputProcessor: input jdbc properties are jdbc:teradata://192.168.11.200/database=vmtest
14/06/25 15:18:29 INFO processor.TeradataInputProcessor: the number of mappers are 2
14/06/25 15:18:29 INFO processor.TeradataInputProcessor: input preprocessor com.teradata.connector.teradata.processor.TeradataSplitByAmpProcessor ends at:  1403723909868
14/06/25 15:18:29 INFO processor.TeradataInputProcessor: the total elapsed time of input preprocessor com.teradata.connector.teradata.processor.TeradataSplitByAmpProcessor is: 1s
14/06/25 15:18:30 WARN conf.HiveConf: DEPRECATED: Configuration property hive.metastore.local no longer has any effect. Make sure to provide a valid value for hive.metastore.uris if you are connecting to a remote metastore.
14/06/25 15:18:30 INFO hive.metastore: Trying to connect to metastore with URI thrift://hdp2.jri.revelytix.com:9083
14/06/25 15:18:30 INFO hive.metastore: Connected to metastore.
14/06/25 15:18:30 WARN conf.HiveConf: DEPRECATED: Configuration property hive.metastore.local no longer has any effect. Make sure to provide a valid value for hive.metastore.uris if you are connecting to a remote metastore.
14/06/25 15:18:30 INFO Configuration.deprecation: mapred.output.dir is deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir
14/06/25 15:18:31 INFO processor.TeradataInputProcessor: input postprocessor com.teradata.connector.teradata.processor.TeradataSplitByAmpProcessor starts at:  1403723911542
14/06/25 15:18:31 INFO processor.TeradataInputProcessor: input postprocessor com.teradata.connector.teradata.processor.TeradataSplitByAmpProcessor ends at:  1403723911542
14/06/25 15:18:31 INFO processor.TeradataInputProcessor: the total elapsed time of input postprocessor com.teradata.connector.teradata.processor.TeradataSplitByAmpProcessor is: 0s
14/06/25 15:18:31 INFO tool.ConnectorImportTool: com.teradata.connector.common.exception.ConnectorException: java.lang.NullPointerException
    at org.apache.hcatalog.data.schema.HCatSchema.get(HCatSchema.java:99)
    at com.teradata.connector.hcat.utils.HCatSchemaUtils.getTargetFieldsTypeName(HCatSchemaUtils.java:37)
    at com.teradata.connector.hcat.processor.HCatOutputProcessor.outputPreProcessor(HCatOutputProcessor.java:70)
    at com.teradata.connector.common.tool.ConnectorJobRunner.runJob(ConnectorJobRunner.java:88)
    at com.teradata.connector.common.tool.ConnectorJobRunner.runJob(ConnectorJobRunner.java:48)
    at com.teradata.connector.common.tool.ConnectorImportTool.run(ConnectorImportTool.java:57)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
    at com.teradata.connector.common.tool.ConnectorImportTool.main(ConnectorImportTool.java:694)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
    at com.teradata.connector.common.tool.ConnectorJobRunner.runJob(ConnectorJobRunner.java:103)
    at com.teradata.connector.common.tool.ConnectorJobRunner.runJob(ConnectorJobRunner.java:48)
    at com.teradata.connector.common.tool.ConnectorImportTool.run(ConnectorImportTool.java:57)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
    at com.teradata.connector.common.tool.ConnectorImportTool.main(ConnectorImportTool.java:694)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
 

sj186043 1 comment Joined 07/14
07 Sep 2014

I am getting the following error while importing data from Teradata to Hadoop. Can anybody  help please?
 
java.io.FileNotFoundException: File -url does not exist.
at org.apache.hadoop.util.GenericOptionsParser.validateFiles(GenericOptionsParser.java:379)
at org.apache.hadoop.util.GenericOptionsParser.processGeneralOptions(GenericOptionsParser.java:275)
at org.apache.hadoop.util.GenericOptionsParser.parseGeneralOptions(GenericOptionsParser.java:413)
at org.apache.hadoop.util.GenericOptionsParser.<init>(GenericOptionsParser.java:164)
at org.apache.hadoop.util.GenericOptionsParser.<init>(GenericOptionsParser.java:147)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:59)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
at com.teradata.hadoop.tool.TeradataImportTool.main(TeradataImportTool.java:369)
13/05/09 17:29:41 INFO tool.TeradataImportTool: job completed with exit code 10000

Command:
hadoop com.teradata.hadoop.tool.TeradataImportTool -libjars $LIB_JARS -url jdbc:teradata://myserver/database=mydb -username user -password password -jobtype hdfs -sourcetable example1_td -nummappers 1 -separator ',' -targetpaths /user/mapred/ex1_hdfs -method split.by.hash -splitbycolumn c1

ahsan.khan 4 comments Joined 07/14
09 Oct 2014

@araghava
@hcnguyen
Yes, TDCH once installed on Hadoop allows data transfers to and from TD. I used HDP2.1 with TD 15 and Teradata Studio 15. But first I installed TDCH in HDP2.1 /usr/lib directory and run ./configureOozie there. More details here
http://ahsannabi.wordpress.com/2014/09/16/free-hadoop-teradata-integration/

sshekhard 1 comment Joined 11/14
04 Nov 2014

Hi All,
Can anybody help me with the link for Java API for TDCH. Please let me know.
 
Regards
Shekhar

mc186036 3 comments Joined 09/12
04 Dec 2014

Hi All,
We have 24nodes 2650 model running 1GB link.
We might be  having Teradata Hadoop by first quarter of 2015.
Our concern is, do we have any confirmed benchmark on how  much time it will take to copy data, let’s say 100GB from hadoop to teradata or from teradata to hadoop using different tools.
 Hadoop has 10GB link but our old model of 2650 has only 1GB link, can it accommodate big data transfer and vice versa?
Can you send us any sample data showing how much  time it got loaded from each other during heavy hours using 100GB data?
We need to answer our client’s question.
 
Thank you.

21 Jan 2015

Hi All,

We have a usecase for bidirectional data transfer between Teradata Aster and Hortonworks Data Platform. Please guide me with the following.

1)Will TDCH be sufficient ? I have had a quick glance through the above attached tutorial, but did not find any mention of Aster. 
2) If TDCH is not the answer, then what is the best way to acheive the above said usecase ?
Regards,
Anil

 

bs185059 4 comments Joined 02/11
15 Feb 2015

Hi Anil,
Bidirectional exchange between Teradata Aster and Hadoop can be done using "QueryGrid Aster-to-Hadoop" (formerly known as "Aster SQL-H"). The Aster SQL-MR functions load_from_hcatalog and load_to_hcatalog are an integral part of the Aster platform.
Hope this helps
Beat

Uttamn 1 comment Joined 10/14
20 Feb 2015

Does Teradata connector for hadoop also works for  Aster  connectivity?Looking for Aster 5.0 and HDP 2.2 connectivity.

Uttam Nayak

caimantian 1 comment Joined 01/15
25 Feb 2015

Hi All,
I am using teradata-connector-1.3.3 with the following versions:
->Teradata 15.0
->HDP 2.1
I am trying to import data from teradata tables to HDFS.
 
I create a table in the Teradata database.

CREATE MULTISET TABLE martinpoc.example3_td3 ,NO FALLBACK ,

NO BEFORE JOURNAL,

NO AFTER JOURNAL,

CHECKSUM = DEFAULT,

DEFAULT MERGEBLOCKRATIO

(

c1 INTEGER,

c2 VARCHAR(100) CHARACTER SET UNICODE NOT CASESPECIFIC)

PRIMARY INDEX ( c1 );
I insert a data with Chinese word into this table.

INSERT INTO martinpoc.example3_td3(c1,C2) VALUES ('1','蔡先生');
 
I run the command call the teradata connector to export data from teredata to HDFS

hadoop com.teradata.hadoop.tool.TeradataImportTool -libjars $LIB_JARS -url jdbc:teradata://192.168.65.132/CHARSET=UTF8,database=martinpoc -username martinpoc -password martin -jobtype hdfs -sourcetable example3_td3 -separator ',' -targetpaths /user/martin/example3_td3 -method split.by.hash -splitbycolumn c1
 
I success to export the data from teradata to HDFS, but I found the Chinese word in the HDFS is weird. (I already using the CHARSET=UTF8 in the JDBC Url)
 
Do you have the same experience and how to resolve it. Thanks.
 
Kindly Thanks and Best Regard
Martin
 

<p>Martin</p>

mc186036 3 comments Joined 09/12
16 Apr 2015

Hi All,
Can anyone assist on this error?
java.lang.Exception: Cannot connect to WebHDFS, check host name and port.
 at com.teradata.datatools.hadoop.connectivity.HadoopConnection.open(HadoopConnection.java:129)
 at com.teradata.datatools.hadoop.connectivity.HadoopConnectionFactory.createConnection(HadoopConnectionFactory.java:30)
 at org.eclipse.datatools.connectivity.internal.ConnectionFactoryProvider.createConnection(ConnectionFactoryProvider.java:83)
 at org.eclipse.datatools.connectivity.internal.ConnectionProfile.createConnection(ConnectionProfile.java:359)
 at org.eclipse.datatools.connectivity.ui.PingJob.createTestConnection(PingJob.java:76)
 at org.eclipse.datatools.connectivity.ui.PingJob.run(PingJob.java:59)
 at org.eclipse.core.internal.jobs.Worker.run(Worker.java:54)
 
Previously, teradata-connector 1.3.3 is working but after few day, it did work anymore. We cannot hook at hadoop file system and teradata.
I have updated the connectoer to 1.3.4 ---->>>>TeradataConnectorForHadoopVersion":"1.3.4" but still not working.
here is the log also from the edgnode where cteradata-connector and studio were installed:
Server returned HTTP response code: 403 for URL: http://gvlhdmpap02:50070/webhdfs/v1/?op=LISTSTATUS&user.name=hdfs
 15:14:03.045 Teradata Datatools [Worker-6] ERROR hadoop - Could not read HDFS
 
Please assist
Thank you.

mc186036 3 comments Joined 09/12
16 Apr 2015

this is resolved.
there was  failover testing and the standby namenode becomes active.
since there is a new version of the teradata-connector, 1.3.4, I have updated it. and rerun oozie script pointing to new active node and it works

senguru78 4 comments Joined 05/15
13 May 2015

Can you please help me, where to findout Teradata connector for Hadoop Java API documentation?
Thank you

heldtrl 1 comment Joined 02/08
15 Jun 2015

The tutorial says the Teradata JDBC URL should point to an MPP system name (and not a single node).  
1) How do I specify an MPP system name in the url?
2) What are the corresponding entries in the Hadoop hosts files I need to reference the MPP nodes?
3) Which Hadoop nodes need these entries in their hosts files?  Master nodes?  Data nodes? Edge nodes?  All three?
Thanks!

TDThrottle 11 comments Joined 11/11
05 Jul 2015

Hadoop to Teradata data movement using TDCH:
While using TDCH (JDBC-FastLoad), if the operation is interrupted for some reason, it must be restarted from the beginning. I have faced this issue quite frequently while large record set is loaded using JDBC FastLoad since "Checkpoint" feature is not available in JDBC connectivity. This feature is not introduced even in recent JDBC 15.x releases.
Considering this:
1. TDCH (Hadoop-to-Teradata) is a scalable Extract-Load solution to move data in regular interval for large IDW system
2. What is the internal latency involved in Teradata while using TDCH/QueryGrid, more specific to AWT usage
Thanks!!

atandon 1 comment Joined 05/15
14 Oct 2015

Teradata Connector for Hadoop (Command Line Edition): Hadoop 2.x: 
Just curious to understand if this version (1.4.1) of TDCH supports Hadoop 2.7, as I am intended to use in MapR 5.0 which as per my knowldge works with Hadoop 2.7.x?
Please advise. Thanks.

ap25912 3 comments Joined 01/14
27 Nov 2015

Can someone point me to link for latest documentation of Teradata Connector for Hadoop 1.4.1 (Command Line Edition) . The documentation above seems very old (Apr 2013).
I am looking to understand what support is provided in new version of connector for incremental imports from teradata to hadoop.
 
Thanks,
Anand

ap25912 3 comments Joined 01/14
23 Dec 2015

HI,
Issue with importing data from teradata table which has delimiter , \r, \n in description column text which is VARCHAR(100).
As a work around we are using -enclosedby '"' parameter bring data into HDFS, and then processing it to remove \n characters if present in column text.
Sqoop has an option as below, which drop specified delimiter, \n, \r from columns if present in column text.

--hive-drop-import-delims	Drops \n, \r, and \01 from string fields when importing to Hive

Not sure, how do I get attention of teradata to resolve this one, if it is included in next release of TDCH it will save lot of processing and data can be straight imported to hive table.
 
Thanks,
Anand
 
 

ap25912 3 comments Joined 01/14
23 Dec 2015

Thanks, I can now see that documentation has been updated and is relavant to v1.4
Cheers!!

frank101 1 comment Joined 02/16
07 Mar 2016

Anand,
You mentioned that hive-drop-import-delims has been solved in TDCH v1.4 which I could not find it. Could you please point me the right direction. Thanks. 

sribmsce 1 comment Joined 03/12
16 Mar 2016

Hi,
I am also facing this issue with delimiter \r and \n and don't find any solution for it.
TDCH does not support the feature  --hive-drop-import-delims yet as far i tested.
I have TDCH v1.4 installed.
Even the sqoop command doesn't support this feauture for Teradata. It works for other databases though. The only option i found is to generate sequence file from TDCH.
However, i need to export this file again to Oracle and Sqoop can export only text files.
Please let me know if any suggestions.
 
Thanks,
Sridhar
 
 

rlobera 6 comments Joined 01/15
05 May 2016

Hello  i need help please
the problem is that version of teradata,
    InfoKey    InfoData
    LANGUAGE SUPPORT MODE    Standard
    RELEASE    12.00.03.33
    VERSION    12.00.03.38
 
and read, the prerequisites is : Teradata Database 13.0
and hadoop is apache 2.7, what can i do for i use sqoop?
thanks
René
What can i do?

chanchalg 1 comment Joined 04/16
05 May 2016

Hi
In TDCH command password for db is mentioned in command itself.
So is there any other way so that we don't need to specify password in command ? (i'm trying to hide password)
can we set pass in env variable for any file ?

indra91 4 comments Joined 06/16
21 Jun 2016

Hello,
As I can see from the TDCH(Version 1.4) user guide it creates multiple JDBC connection to create connection and export data from a Teradata database and that is does not uses Teradata fastexport utitily to extract the data but I would like to know how FastExport extracts the data and how it is different from the method TDCH for the same.

  • It would be very helpful if someone could highlight the main architectural difference between the two methods(TDCH and fastexport for extraction)
  • Which should be a better option(TDCH vs Fastexport utiltiy) to extract huge volume of data(in TB's) from Teradata tables.
  • Whether the TDCH comes with a free license or it comes along with teradata installation

 

indra91 4 comments Joined 06/16
22 Jun 2016

Hi,
I would like to know if the TDCH  for teradata supports the following
 

  1.  Hbase as a target?
  2. Also while loading into Hadoop whether it provided any support for the various compression formats like bzip,snappy?
  3. Whether it provides any feature for streaming data migration between Teradata and Hadoop systems?

Would be very grateful if someone could throw some light into the queries posted above.

smdorval 3 comments Joined 06/15
22 Jun 2016

indra91,
Do you have a support contract with Teradata for your TD system?  If so, you should open up incidents for help instead of asking on this article.
Best regards,
Sean
 

indra91 4 comments Joined 06/16
22 Jun 2016

Hi Sean,
I am not sure whether I have a support contract with Teradata,as I can see this is a forum for discussion and many queries have been asked and answered through this forum.I could not find this info on the user guide and tutorial available with TDCH.It will be helpful if somebody can provide some insight.

smdorval 3 comments Joined 06/15
22 Jun 2016

indra91,
Here are some answers to your questions.
1.  TDCH 1.4.4 (latest release) supports many different methods for data export from Teradata.  You can see these in section 1.4.5 of the tutorial.  TDCH 1.5 (which is scheduled to release Q3 of this year) will support FastExport.
2.  TDCH is free to use.
3.  HBase is not supported as a target.
4.  Compression is not supported at the moment.
Hope this helps.
Best regards,
Sean

indra91 4 comments Joined 06/16
22 Jun 2016

Hi Sean,
Thanks a lot for your inputs it was really helpful.For a use case where we need to offload huge volume of data (in TB)between
Teradata systems and hadoop could you suggest which would be a faster option to extract the data native fastexport utility or the TDCH?

smdorval 3 comments Joined 06/15
23 Jun 2016

indra91,
I would suggest waiting for the TDCH 1.5 release and using the FastExport within TDCH which will give you the best results.
Best regards,
Sean

pratik.raj 1 comment Joined 05/16
03 Jul 2016

I got following error while importing table from teradata to hadoop:-

16/07/03 09:52:15 INFO mapreduce.TeradataInputProcessor: job setup ends at 1467564735483

16/07/03 09:52:15 INFO mapreduce.TeradataInputProcessor: job setup time is 10s

16/07/03 09:52:15 ERROR tool.TeradataImportTool: com.teradata.hadoop.exception.TeradataHadoopException: java.sql.SQLException: [Teradata JDBC Driver] [TeraJDBC 15.00.00.20] [Error 1277] [SQLState 08S01] Login timeout for Connection to 153.65.248.183 Sun Jul 03 09:52:15 PDT 2016 socket orig=X.X.X.X cid=65efb4be sess=0 java.net.SocketTimeoutException: connect timed out  at java.net.PlainSocketImpl.socketConnect(Native Method)  at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351)  at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213)  at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200)  at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)  at java.net.Socket.connect(Socket.java:529)  at com.teradata.jdbc.jdbc_4.io.TDNetworkIOIF$ConnectThread.run(TDNetworkIOIF.java:1216)

        at com.teradata.jdbc.jdbc_4.util.ErrorFactory.makeDriverJDBCException(ErrorFactory.java:94)

        at com.teradata.jdbc.jdbc_4.util.ErrorFactory.makeDriverJDBCException(ErrorFactory.java:69)

        at com.teradata.jdbc.jdbc_4.util.ErrorFactory.makeIoJDBCException(ErrorFactory.java:207)

        at com.teradata.jdbc.jdbc_4.util.ErrorAnalyzer.analyzeIoError(ErrorAnalyzer.java:59)

        at com.teradata.jdbc.jdbc_4.io.TDNetworkIOIF.createSocketConnection(TDNetworkIOIF.java:154)

        at com.teradata.jdbc.jdbc_4.io.TDNetworkIOIF.<init>(TDNetworkIOIF.java:133)

        at com.teradata.jdbc.jdbc.GenericTeradataConnection.getIO(GenericTeradataConnection.java:113)

        at com.teradata.jdbc.jdbc.GenericLogonController.run(GenericLogonController.java:98)

        at com.teradata.jdbc.jdbc_4.TDSession.<init>(TDSession.java:205)

        at com.teradata.jdbc.jdk6.JDK6_SQL_Connection.<init>(JDK6_SQL_Connection.java:35)

        at com.teradata.jdbc.jdk6.JDK6ConnectionFactory.constructSQLConnection(JDK6ConnectionFactory.java:25)

        at com.teradata.jdbc.jdbc.ConnectionFactory.createConnection(ConnectionFactory.java:179)

        at com.teradata.jdbc.jdbc.ConnectionFactory.createConnection(ConnectionFactory.java:169)

        at com.teradata.jdbc.TeraDriver.doConnect(TeraDriver.java:232)

        at com.teradata.jdbc.TeraDriver.connect(TeraDriver.java:158)

        at java.sql.DriverManager.getConnection(DriverManager.java:582)

        at java.sql.DriverManager.getConnection(DriverManager.java:185)

        at com.teradata.hadoop.db.TeradataConnection.connect(TeradataConnection.java:336)

        at com.teradata.hadoop.mapreduce.TeradataProcessorBase.openConnection(TeradataProcessorBase.java:77)

        at com.teradata.hadoop.mapreduce.TeradataInputProcessor.openConnection(TeradataInputProcessor.java:108)

        at com.teradata.hadoop.mapreduce.TeradataInputProcessor.setup(TeradataInputProcessor.java:51)

        at com.teradata.hadoop.mapreduce.TeradataSplitByHashInputProcessor.setup(TeradataSplitByHashInputProcessor.java:51)

        at com.teradata.hadoop.job.TeradataHdfsFileImportJob.runJob(TeradataHdfsFileImportJob.java:207)

        at com.teradata.hadoop.tool.TeradataJobRunner.runImportJob(TeradataJobRunner.java:119)

        at com.teradata.hadoop.tool.TeradataImportTool.run(TeradataImportTool.java:41)

        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)

        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)

        at com.teradata.hadoop.tool.TeradataImportTool.main(TeradataImportTool.java:464)

Caused by: java.net.SocketTimeoutException: connect timed out

        at java.net.PlainSocketImpl.socketConnect(Native Method)

        at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351)

        at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213)

        at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200)

        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)

        at java.net.Socket.connect(Socket.java:529)

        at com.teradata.jdbc.jdbc_4.io.TDNetworkIOIF$ConnectThread.run(TDNetworkIOIF.java:1216)

 

        at com.teradata.hadoop.mapreduce.TeradataProcessorBase.openConnection(TeradataProcessorBase.java:80)

        at com.teradata.hadoop.mapreduce.TeradataInputProcessor.openConnection(TeradataInputProcessor.java:108)

        at com.teradata.hadoop.mapreduce.TeradataInputProcessor.setup(TeradataInputProcessor.java:51)

        at com.teradata.hadoop.mapreduce.TeradataSplitByHashInputProcessor.setup(TeradataSplitByHashInputProcessor.java:51)

        at com.teradata.hadoop.job.TeradataHdfsFileImportJob.runJob(TeradataHdfsFileImportJob.java:207)

        at com.teradata.hadoop.tool.TeradataJobRunner.runImportJob(TeradataJobRunner.java:119)

        at com.teradata.hadoop.tool.TeradataImportTool.run(TeradataImportTool.java:41)

        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)

        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)

        at com.teradata.hadoop.tool.TeradataImportTool.main(TeradataImportTool.java:464)

 

 

I think I am not able to connect Teradata database. I checked I am able to ping the URL.

jebeckford1020 1 comment Joined 02/16
18 Jul 2016

Hi I've downloaded TDCH 1.4.4.  I'm trying to connect an IBM BI 3.1 (Hadoop 2.2) cluster to a TD 14.10 database.  I have read through the TDCH 1.4 tutorial that you included in this article but I noticed it doesn't include any details on the Java API.  Can I get a copy of the Java API documentation sent to me?  Also, can you tell me whether leveraging the Java API will still allow me to use TD Wallet?  Or is TD Wallet only compatible with the command line version of TDCH?
 
Thanks,
Jonathan

07 Sep 2016

Hi, I am trying to Import/Export From/To teradata to Hive using TDCH , But getting below issue with CHAR,VARCHAR and DATE data types not supported in TDCH. As an alternate if we change those data types to string then no problem seen.

 

INFO tool.ConnectorExportTool: ConnectorExportTool starts at 1473252737445

16/09/07 14:52:17 INFO common.ConnectorPlugin: load plugins in file:/tmp/hadoop-unjar6516039745100009834/teradata.connector.plugins.xml

16/09/07 14:52:18 INFO hive.metastore: Trying to connect to metastore with URI thrift://el3207.bc:9083

16/09/07 14:52:18 INFO hive.metastore: Connected to metastore.

16/09/07 14:52:18 INFO processor.TeradataOutputProcessor: output postprocessor com.teradata.connector.teradata.processor.TeradataBatchInsertProcessor starts at:  1473252738715

16/09/07 14:52:19 INFO processor.TeradataOutputProcessor: output postprocessor com.teradata.connector.teradata.processor.TeradataBatchInsertProcessor ends at:  1473252738715

16/09/07 14:52:19 INFO processor.TeradataOutputProcessor: the total elapsed time of output postprocessor com.teradata.connector.teradata.processor.TeradataBatchInsertProcessor is: 0s

16/09/07 14:52:19 INFO tool.ConnectorExportTool: com.teradata.connector.common.exception.ConnectorException: CHAR(6) Field data type is not supported

        at com.teradata.connector.common.tool.ConnectorJobRunner.runJob(ConnectorJobRunner.java:140)

        at com.teradata.connector.common.tool.ConnectorExportTool.run(ConnectorExportTool.java:62)

        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)

        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)

        at com.teradata.connector.common.tool.ConnectorExportTool.main(ConnectorExportTool.java:780)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

        at java.lang.reflect.Method.invoke(Method.java:498)

        at org.apache.hadoop.util.RunJar.run(RunJar.java:221)

        at org.apache.hadoop.util.RunJar.main(RunJar.java:136)

 

16/09/07 14:52:19 INFO tool.ConnectorExportTool: job completed with exit code 14006

 

If anyone has solution for this, please share.

 

Thanks

 

You must sign in to leave a comment.