Teradata Studio provides a Smart Loader for Hadoop feature that allows users to transfer data from Teradata to Hadoop, Hadoop to Teradata, and Hadoop to Aster. When transfering between Teradata and Hadoop, the Hadoop Smart Loader uses the Teradata Connector for Hadoop MapReduce Java classes as the underlying technology for data movement. It requires the HCatalog metadata layer to browse the Hadoop objects and uses Oozie workflow to manage the data transfer.  Currently, the Smart Loader for Hadoop feature in Teradata Studio is certified to use the Teradata Connector for Hadoop (TDCH) version 1.3.4, and the Hortonworks and Cloudera distributions of Hadoop. The Teradata Connector for Hadoop needs to be installed on the Hadoop System.

                  

NOTE: You must have the Teradata Connector for Hadoop (TDCH) installed on your Hadoop system. You can download the TDCH version 1.3.4  on the Developer Exchange Download site. You must also download the Configure Oozie script and run it on your Hadoop system. Refer to the Readme on the Teradata Studio download page for instructions on running the Configure Oozie script.

For Hadoop to Aster data transfers, the Smart Loader for Hadoop uses the Aster Map Reduce Function, load_from_hcatalog. The data transfer is initiated from the Aster Database to remotely access the Hadoop System, via SQL-H, and pull the data across.

With bi-directional data loading, users can easily perform ad hoc data transfer between their Teradata, Aster, and Hadoop systems. The Hadoop Smart Loader can be invoked by drag and drop of a table between the Transfer View and Data Source Explorer or by selecting a table in the Data Source Explorer and choosing the option Data>Export Data... or Data>Load Data.... This will invoke the Data Transfer Wizard for you to select the Source or Destination Type.

               

Create Hadoop Connection Profile.

You can create connections to your Hadoop System using the Connection Profile Wizard. The wizard is invoked from the Data Source Explorer by right clicking on the Database Connections folder or selecting the 'New Connection Profile' button, , from the Data Source Explorer toolbar.

          

There are two options for creating Hadoop Connection Profiles:

  • Hadoop Generic System - The Hadoop Generic System profile supports migrating Hadoop connections from Studio releases prior to Studio 15.10. It is also used to support Cloudera Hadoop connections.  Hadoop Generic System connections are created using the WebHCat protocol to connect and discover database and tables information. Enter the WebHDFS, WebHDFS Port number, and System Username. This connection requires that the ConfigureOozie script is run on the Hadoop System.

             

  • Hadoop Hortonworks - The Hadoop Hortonworks connection profile provides additional options for connecting to Hortonworks Hadoop systems. It is based on the desired functionality between Studio and your Hadoop System: Knox Gateway (Secure connection), TDCH (Teradata data transfers), JDBC (creating and running SQL), or SQL-H (Hadoop to Aster data transfers). Note that the Knox Gateway option also supports JDBC connections. Click next to enter the Host name, Port number, User name, and Password, if required. The TDCH option is equivalent to the Hadoop Generic System connection profile described above and requires the ConfigureOozie script to be run on the Hadoop System.

                        

Once you have your Hadoop connection profile, you can browse the Hadoop database and table objects in the Data Source Explorer.

You can also run HiveQL SQL commands against your Hadoop system if you have configured a Hadoop JDBC or Knox connection profile.

Transfer Tables between Teradata and Hadoop

Before invoking the Hadoop Transfer Wizard (aka Smart Loader for Hadoop), switch to the Data Transfer perspective and choose your Hadoop connection profile in the Transfer View.

There are two ways to invoke the Hadoop Transfer Wizard. One way to transfer a table from Teradata to Hadoop is to drag a Teradata table from the Data Source Explorer and drop it on a Hadoop database in the Transfer View. You can also transfer from Hadoop to Teradata by dragging the table from the Hadoop system in the Transfer View and dropping on a Teradata Database in the DSE.

    

The other way is from the Data Transfer Wizard. Choose the Teradata table from the Data Source Explorer, right click and choose Data>Export.... This will invoke the Data Transfer Wizard. Choose Hadoop as the Destination Type and click Launch. NOTE: You can also choose Data>Load... and Source Type as Hadoop to transfer data from Hadoop to Teradata.

    

This will launch the Hadoop Transfer wizard for you to choose the destination Hadoop system for the transfer. You can transfer the Teradata table as a 'New Table' or the data to an existing table in Hadoop.

    

From either drag and drop or from the Data Transfer Wizard, the Hadoop Transfer wizard will next prompt the user to choose the file options and column mappings. The Hadoop Transfer wizard will attempt to choose Hadoop columns types based on the source Teradata column data types. The user can override the destination column type by selecting a new column type from the drop down list. You can also choose to filter out columns you don't want in the destination Hadoop table by unchecking the column. If you are transfering to an existing Hadoop table, you will need to map the source columns to the destination columns. You will also be given an option to append or replace the table data.

    

Press finish to complete the Hadoop data transfer and submit a Data Transfer job to perform the data copy. As with data load, the status of the Data Transfer job is displayed in the Transfer Progress View. When the job has completed, an entry is placed in the Transfer History View.

Transfer Tables from Hadoop to Aster

There are two ways to invoke the Hadoop to Aster Transfer Wizard. One way is to drag a Hadoop table from the Data Source Explorer and drop it on an Aster Database in the Transfer View. You can also invoke the wizard by selecting the Aster Tables folder in the Data Source Explorer, right click and choosing Aster>Data Transfer... option. This will invoke the Data Transfer Wizard for you to select Hadoop as the Source Type.

    

Press the Launch button to launch the Hadoop Table to Aster Wizard. Choose the Hadoop Connection Profile to locate the database and table to transer. Next it will display the columns and column types of the Hadoop table. You can filter out columns and select whether the column can contain nulls and whether it is unique. The Hadoop Table to Aster Wizard will only create Aster Fact tables.

Press finish to complete the Hadoop data transfer and submit a Data Transfer job to perform the data copy. As with data load, the status of the Data Transfer job is displayed in the Transfer Progress View. When the job has completed, an entry is placed in the Transfer History View.

Hadoop Data Transfer Job

A transfer job is created to transfer the data to and from Teradata and Hadoop. You can view the progress of the transfer job in the Transfer Progress View of the Data Transfer perspective. NOTE: With the Oozie workflow, the status of the job is not available until the job has finished. Once the job is complete, an entry is placed in the Transfer History and displayed in the Transfer History View.

Select the entry in the Transfer History and click on the Show Job Output toolbar button to view the output from the Hadoop job transfer.

                     

Help

Teradata Studio provides Help information. Click on Help>Help Contents in the main toolbar. You can also get additional information on the Hadoop Transfer Wizard by clicking the question mark, '?' at the lower left hand corner of the wizard page.

     

Conclusion

Teradata Studio Hadoop Smart Loader provides an ad hoc data movement tool to transfer data between Teradata and Hadoop. It provides a point and click GUI where no scripting is required. You can download Teradata Studio and the Teradata Connector for Hadoop on the Teradata Download site. For more information about other Teradata Studio features, refer to the article called Teradata Studio.

 

Discussion
dd2teradata 5 comments Joined 10/09
11 Jun 2013

Excellent step by step instructions.  Easy to follow with great explanations.
Initially I could not locate the Import from Hadoop since I clicked on the table name in the Teradata explorer.  You will only see the Export to Hadoop... from Data tab.  Make sure your mouse is at the Table folder and not individual table name so that the Import from Hadoop... will be seen from Teradata tab.
Happy computing...

dd2teradata 5 comments Joined 10/09
11 Jun 2013

Currently the hadoop transfer does not support Kanji enabled Teradata database.  Well, most of my test data are in Unicode and/or Multi-bytes character object name (table, user) and the data contain in the table.  Hope there will be some work around or support for dealing with Unicode data and objects such as Multi-bytes characters.
 
I will try to use the Teradata Connector for Hadoop by calling the com.teradata.hadoop.tool.TeradataExportTool and com.teradata.hadoop.tool.TeradataImportTool to see if it can handle Multi-bytes object names and data.
Happy learning always...

Chuckbert 37 comments Joined 07/09
11 Jun 2013

I have been able to transfer data both ways between Teradata and Hadoop that contain non-ASCII characters. When Teradata Studio sets up its transfer jobs, it uses the CHARSET parameter defined in the Connection Profile in the JDBC URL that the Connector job on Hadoop uses. If the CHARSET is set to the default value of ASCII, the non-ASCII characters will not transfer.
I was not able to transfer a table whose name contains Kanji characters. The Connector's Limitations section says Kanji is not supported.

Chuckbert 37 comments Joined 07/09
11 Jun 2013

UTF8 is the CHARSET value that I used.

VVenkat 8 comments Joined 02/11
18 Jun 2013

 Hi,
I am trying to import data using Teradata Studio to TD 13.0 vm instance, followed the steps for install for the connector I was able to connect to Hadoop  default DB  however I get the following error when I import data to TD 13.0 .. appreciate any insight..
Exception in thread "main" java.lang.NoClassDefFoundError: com/teradata/hadoop/tool/TeradataExportTool
Caused by: java.lang.ClassNotFoundException: com.teradata.hadoop.tool.TeradataExportTool
 at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
 at java.security.AccessController.doPrivileged(Native Method)
 at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
 at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
 at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
 at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
Could not find the main class: com.teradata.hadoop.tool.TeradataExportTool.  Program will exit.
Thanks
Venkat

fgrimmer 660 comments Joined 04/09
18 Jun 2013

@VVenkat, You need to install the Teradata Connector for Hadoop (TDCH) (Studio Edition). You can download the connector on the Download Page (at http://downloads.teradata.com/download/connectivity/teradata-connector-for-hadoop-studio-edition). The connector is installed on your Hadoop system. Refer to the Readme inside the TDCH. Here are the main steps from the Readme for installing the TDCH on your Hadoop system:
4.2 Installation

 1. After unzipping Teradata Connector for Hadoop tar.gz package, copy the following files into Hadoop's lib directory:
/usr/lib/hadoop/lib

  a) teradata-connector-1.0.6.jar
  b) tdgssconfig.jar
  c) terajdbc4.jar

2. Copy the following files into Hadoop's conf directory:
/etc/hadoop/conf

   a) teradata-export-properties.xml
   b) teradata-import-properties.xml

3. Modify mapred.child.java.opts property value in mapred-site.xml and add /etc/hadoop/conf
   -Djava.security.egd=file:/dev/./urandom

4. Modify the following OS networking values on the name node (recommended - Optional)
   /proc/sys/net/core/somaxconn to 1024
   /proc/sys/net/ipv4/tcp_max_syn_backlog to 2048
 

VVenkat 8 comments Joined 02/11
18 Jun 2013

Thank you very much that did resolve the above issue as one of the jars was missing, my import exports now run without error however  hang at the mapreduce step as shown below, I have updated the mapred-site.xml as mentioned above..
13/06/18 09:09:14 INFO mapreduce.TeradataInputProcessor: input batch size is 1000013/06/18 09:09:14 INFO mapreduce.TeradataInputProcessor: input number of mappers are 2 13/06/18 09:09:14 INFO mapreduce.TeradataInputProcessor: job setup ends at 1371571754830 13/06/18 09:09:14 INFO mapreduce.TeradataInputProcessor: job setup time is 3s13/06/18 09:09:28 INFO mapred.JobClient: Running job: job_201306180704_000313/06/18 09:09:29 INFO mapred.JobClient: map 0% reduce 0% ..
pl advise
Thanks
V

fgrimmer 660 comments Joined 04/09
18 Jun 2013

@VVenkat, So exports to Hadoop work but import from Hadoop does not? What version of Teradata are you importing to?

VVenkat 8 comments Joined 02/11
18 Jun 2013

I am trying to export and import from and to Teradata 13.0 both jobs hang at the above step and do not complete..Should I be runnig against a higher ver of TD?

fgrimmer 660 comments Joined 04/09
18 Jun 2013

@VVenkat, No. It is supported on Teradata Database 13.0 and up. I have sent your issue over to TDCH support and hope to hear quickly.  Can you confirm what version of Hortonworks HDP you are running?

VVenkat 8 comments Joined 02/11
18 Jun 2013

Hortonworks HDP 1.2.0.1.3.0.0-107
Thanks for the follow up.

fgrimmer 660 comments Joined 04/09
19 Jun 2013

@VVenkat, The supported versions of HDP are: HDP 1.1.0.17 and HDP 1.0.0.12. Can you install one of these supported versions?
 

VVenkat 8 comments Joined 02/11
19 Jun 2013

I am using the latest sandbox from HW I was not able to find the older version that has the required HDP I will continue look.. however would appreciate any alternatives/suggestions.
Thanks

fgrimmer 660 comments Joined 04/09
19 Jun 2013

There is a mapper output log that might show information on what is going on. The log file is located in slave node: <hadoop_install_dir>/logs/userlogs/job_201306180704_0003/.

VVenkat 8 comments Joined 02/11
20 Jun 2013

Above issue is resolved.
Folowing updates to the /etc/hadoop/conf/mapred-site.xml fixed the issue
replaced the values of _HOST@EXAMPLE.COM with sandbox:50300
I was able to export / Import to and from Hadoop to Teradata after this which is very cool!. Thanks  much for the detailed steps and support
Venkat
 

nancyjean 7 comments Joined 02/13
01 Jul 2013

Hi,
I have HDP 1.1.1.16 version. Which teradata-hadoop connector can i use?

fgrimmer 660 comments Joined 04/09
02 Jul 2013

@nancyjean, The supported versions of HDP are: HDP 1.1.0.17 and HDP 1.0.0.12. Can you install one of these supported versions?

sguthi0910 4 comments Joined 10/13
05 Nov 2013

Hi,
I have installed the teradata studio and connected to a teradata database with my credentials. I do not see any tables in the Data Source Explorer on the left. So I'm not able to proceed further in selecting the table and do export to hadoop. What kind of permissions does my ID need to view the tables in the explorer? I'm able to run the SQL queries on the tables of the database and view the results on the Result set view. But I dont see any tables in the explorer.

fgrimmer 660 comments Joined 04/09
05 Nov 2013

@sguthi0910, Can you locate the Teradata database in the Data Source Explorer for your connection profile, that contains the table? Set focus (select) the Teradata View tab then select your Teradata Database in the Data Source Explorer. Do you see the list of tables in the Teradata View tab? Can you double click on your table, in the Teradata View tab, and does it show the details for the table?

06 Nov 2013

Fgrimmer,
I do not see "Hadoop Transfer" option in "open perspective". Only below options,
 
- Database Development
- Teradata Administrator(Default ) 
I am using Terdata Studio version  14.10.0.201308041204.  Any suggestion what am I missing?
 
Regards,
Ravi

Ravi Singh

fgrimmer 660 comments Joined 04/09
06 Nov 2013

Please download 4.10.01 version of Studio. It was temporarily removed from 14.10.00.

Qionglinnewbie 3 comments Joined 09/13
14 Dec 2013

Hi Fgrimmer,
smart loader's hadoop profile doesn't accept my webhdfs hostname (192.168.1.9) and port (50070/50075).
I'm trying to connect Teradata studio 14.10.1 to Hortonworks sandbox 2.0 in virtual machine. The IP address 192.168.1.9 is given by the sandbox vm to access my hadoop cluster.
On vm sandbox, I have
- add teradata-connector-1.06.jar
- and teradata-export-properties.xml & teradata-import-properties.xml
- set dfs.webhdfs.enabled to true under hdfs-site.xml and restart
Any help please. Thanks!

fgrimmer 660 comments Joined 04/09
16 Dec 2013

@Qionglinnewbie, As mentioned above, the procedure has changed for Studio 14.10.01. You must download a new version of TDCH (1.0.9a or greater), as well as run the ConfigureOozie script. Please refer to the instructions in the Readme on the Teradata Studio download page (http://downloads.teradata.com/download/tools/teradata-studio).

Qionglinnewbie 3 comments Joined 09/13
16 Dec 2013

Fgrimmer,
Sorry for the mistake, actually I installed the last teradata connector for hadoop, version 1.1. I saw also latter that I missed out the oozie configure script. Thanks for point me out. One thing about Oozie, can you tell me under which folder should it run, /usr/lib/hadoop/lib or elsewhere? I tried a few places without success. Thank you for your help.

fgrimmer 660 comments Joined 04/09
16 Dec 2013

You would download the ConfigureOozie script to your HDP system and run it as root. It is creating the scripts so probably run in a subdirectory so you can see what it created.

esrkreddy 2 comments Joined 09/13
01 Apr 2014

Is Cludera Hadoop distribution supported?

fgrimmer 660 comments Joined 04/09
01 Apr 2014

Sairamakrishna, Cloudera is not a supported configuration of Hadoop for Teradata Studio at this time.

umerbaig 4 comments Joined 12/12
07 Apr 2014

How to check system user name and WebHDFS Host Name on hadoop for connectivity? I have 2 nodes cluster.

esrkreddy 2 comments Joined 09/13
07 Apr 2014

Is HDP 2.x yet supported?

harristd 5 comments Joined 07/12
07 Apr 2014

Umer, you can find the WebHDFS host name on the system under $HADOOP_INSTALL/hadoop/conf/hdfs-site.xml.  $HADOOP_INSTALL is typically /etc.  Look for the dfs.http.address property for the hostname. 
You can check hadoop users also on the system by running "hadoop fs -ls /user" and it will list the user names that are set up.

fgrimmer 660 comments Joined 04/09
07 Apr 2014

Sairamakrishna, No, HDP 2.x in not yet supported. We are working on certifying with it now and hope to have a release soon.

umerbaig 4 comments Joined 12/12
18 Apr 2014

I have sucessfully transfer data from hadoop to teradata and vice through studio but in another enviornment I have done same thing but my transfer progress view dissapear before 1 min and give unknown error.
using hadoop 1.3.3

fgrimmer 660 comments Joined 04/09
18 Apr 2014

@umerbaig, I noticed you posted this same question to several articles, but I will try to answer your question here. What version of Teradata Studio are you running? You mention that the Transfer Progress View disappears. Does the view disappear or just the transfer job entry? The transfer job entry will move from the Transfer Progress View to the Transfer History View when it completes. Can you open the Transfer History View and see an entry for your attempted transfer? If the entry is there, select it and click the 'Job Details' button to see the status output from the transfer.

28 May 2014

Hi @fgrimmer
I always get the following error whenever I try to perform Smart Loader from Hadoop to Teradata:
05/28 21:05:41  INFO TeradataHiveUtils: Using hive-site.xml: hdfs://sandbox:8020/teradata/oozieworkflows/hive-site.xml
05/28 21:05:41  INFO HiveMetaStoreClient: Trying to connect to metastore with URI thrift://sandbox:9083
05/28 21:05:41  INFO HiveMetaStoreClient: Waiting 1 seconds before next connection attempt.
05/28 21:05:42  INFO HiveMetaStoreClient: Connected to metastore.
05/28 21:05:43  INFO TeradataOutputProcessor: job setup starts at 1401336343706
05/28 21:06:04  INFO TeradataOutputProcessor: job setup ends at 1401336364968
05/28 21:06:04  INFO TeradataOutputProcessor: job setup time is 21s
05/28 21:06:04 ERROR TeradataExportTool: com.teradata.hadoop.exception.TeradataHadoopException: com.teradata.jdbc.jdbc_4.util.JDBCException: [Teradata JDBC Driver] [TeraJDBC 14.00.00.01] [Error 1000] [SQLState 08S01] Login failure for Connection to localtdcop1 Wed May 28 21:06:04 PDT 2014 socket orig=localtdcop1 cid=508de972 sess=0 java.net.UnknownHostException: localtdcop1  at java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method)  at java.net.InetAddress$1.lookupAllHostAddr(InetAddress.java:849)  at java.net.InetAddress.getAddressFromNameService(InetAddress.java:1202)  at java.net.InetAddress.getAllByName0(InetAddress.java:1153)  at java.net.InetAddress.getAllByName(InetAddress.java:1083)  at java.net.InetAddress.getAllByName(InetAddress.java:1019)  at com.teradata.jdbc.jdbc_4.io.TDNetworkIOIF$Lookup.<init>(TDNetworkIOIF.java:182)  at com.teradata.jdbc.jdbc_4.io.TDNetworkIOIF.connectToHost(TDNetworkIOIF.java:281)  at com.teradata.jdbc.jdbc_4.io.TDNetworkIOIF.createSocketConnection(TDNetworkIOIF.java:130)  at com.teradata.jdbc.jdbc_4.io.TDNetworkIOIF.<init>(TDNetworkIOIF.java:116)  at com.teradata.jdbc.jdbc_4.TDSession.getIO(TDSession.java:579)  at com.teradata.jdbc.jdbc.GenericLogonController.run(GenericLogonController.java:94)  at com.teradata.jdbc.jdbc_4.TDSession.<init>(TDSession.java:208)  at com.teradata.jdbc.jdk6.JDK6_SQL_Connection.<init>(JDK6_SQL_Connection.java:30)  at com.teradata.jdbc.jdk6.JDK6ConnectionFactory.constructConnection(JDK6ConnectionFactory.java:22)  at com.teradata.jdbc.jdbc.ConnectionFactory.createConnection(ConnectionFactory.java:130)  at com.teradata.jdbc.jdbc.ConnectionFactory.createConnection(ConnectionFactory.java:120)  at com.teradata.jdbc.TeraDriver.doConnect(TeraDriver.java:220)  at com.teradata.jdbc.TeraDriver.connect(TeraDriver.java:153)  at java.sql.DriverManager.getConnection(DriverManager.java:582)  at java.sql.DriverManager.getConnection(DriverManager.java:185)  at com.teradata.hadoop.db.TeradataConnection.connect(TeradataConnection.java:284)  at com.teradata.hadoop.mapreduce.TeradataProcessorBase.openConnection(TeradataProcessorBase.java:68)  at com.teradata.hadoop.mapreduce.TeradataOutputProcessor.openConnection(TeradataOutputProcessor.java:115)  at com.teradata.hadoop.mapreduce.TeradataOutputProcessor.setup(TeradataOutputProcessor.java:54)  at com.teradata.hadoop.mapreduce.TeradataBatchInsertOutputProcessor.setup(TeradataBatchInsertOutputProcessor.java:60)  at com.teradata.hadoop.job.TeradataExportJob.runJob(TeradataExportJob.java:72)  at com.teradata.hadoop.tool.TeradataJobRunner.runExportJob(TeradataJobRunner.java:192)  at com.teradata.hadoop.tool.TeradataExportTool.run(TeradataExportTool.java:41)  at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)  at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)  at com.teradata.hadoop.tool.TeradataExportTool.main(TeradataExportTool.java:439)  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)  at java.lang.reflect.Method.invoke(Method.java:597)  at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:491)  at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)  at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:429)  at org.apache.hadoop.mapred.MapTask.run(MapTask.java:365)  at org.apache.hadoop.mapred.Child$4.run(Child.java:255)  at java.security.AccessController.doPrivileged(Native Method)  at javax.security.auth.Subject.doAs(Subject.java:396)  at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)  at org.apache.hadoop.mapred.Child.main(Child.java:249)
 at com.teradata.jdbc.jdbc_4.util.ErrorFactory.makeDriverJDBCException(ErrorFactory.java:93)
 at com.teradata.jdbc.jdbc_4.util.ErrorFactory.makeDriverJDBCException(ErrorFactory.java:68)
 at com.teradata.jdbc.jdbc_4.util.ErrorFactory.makeIoJDBCException(ErrorFactory.java:206)
 at com.teradata.jdbc.jdbc_4.util.ErrorAnalyzer.analyzeIoError(ErrorAnalyzer.java:61)
 at com.teradata.jdbc.jdbc_4.io.TDNetworkIOIF.createSocketConnection(TDNetworkIOIF.java:137)
 at com.teradata.jdbc.jdbc_4.io.TDNetworkIOIF.<init>(TDNetworkIOIF.java:116)
 at com.teradata.jdbc.jdbc_4.TDSession.getIO(TDSession.java:579)
 at com.teradata.jdbc.jdbc.GenericLogonController.run(GenericLogonController.java:94)
 at com.teradata.jdbc.jdbc_4.TDSession.<init>(TDSession.java:208)
 at com.teradata.jdbc.jdk6.JDK6_SQL_Connection.<init>(JDK6_SQL_Connection.java:30)
 at com.teradata.jdbc.jdk6.JDK6ConnectionFactory.constructConnection(JDK6ConnectionFactory.java:22)
 at com.teradata.jdbc.jdbc.ConnectionFactory.createConnection(ConnectionFactory.java:130)
 at com.teradata.jdbc.jdbc.ConnectionFactory.createConnection(ConnectionFactory.java:120)
 at com.teradata.jdbc.TeraDriver.doConnect(TeraDriver.java:220)
 at com.teradata.jdbc.TeraDriver.connect(TeraDriver.java:153)
 at java.sql.DriverManager.getConnection(DriverManager.java:582)
 at java.sql.DriverManager.getConnection(DriverManager.java:185)
 at com.teradata.hadoop.db.TeradataConnection.connect(TeradataConnection.java:284)
 at com.teradata.hadoop.mapreduce.TeradataProcessorBase.openConnection(TeradataProcessorBase.java:68)
 at com.teradata.hadoop.mapreduce.TeradataOutputProcessor.openConnection(TeradataOutputProcessor.java:115)
 at com.teradata.hadoop.mapreduce.TeradataOutputProcessor.setup(TeradataOutputProcessor.java:54)
 at com.teradata.hadoop.mapreduce.TeradataBatchInsertOutputProcessor.setup(TeradataBatchInsertOutputProcessor.java:60)
 at com.teradata.hadoop.job.TeradataExportJob.runJob(TeradataExportJob.java:72)
 at com.teradata.hadoop.tool.TeradataJobRunner.runExportJob(TeradataJobRunner.java:192)
 at com.teradata.hadoop.tool.TeradataExportTool.run(TeradataExportTool.java:41)
 at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
 at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
 at com.teradata.hadoop.tool.TeradataExportTool.main(TeradataExportTool.java:439)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:597)
 at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:491)
 at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
 at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:429)
 at org.apache.hadoop.mapred.MapTask.run(MapTask.java:365)
 at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
 at java.security.AccessController.doPrivileged(Native Method)
 at javax.security.auth.Subject.doAs(Subject.java:396)
 at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
 at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: java.net.UnknownHostException: localtdcop1
 at java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method)
 at java.net.InetAddress$1.lookupAllHostAddr(InetAddress.java:849)
 at java.net.InetAddress.getAddressFromNameService(InetAddress.java:1202)
 at java.net.InetAddress.getAllByName0(InetAddress.java:1153)
 at java.net.InetAddress.getAllByName(InetAddress.java:1083)
 at java.net.InetAddress.getAllByName(InetAddress.java:1019)
 at com.teradata.jdbc.jdbc_4.io.TDNetworkIOIF$Lookup.<init>(TDNetworkIOIF.java:182)
 at com.teradata.jdbc.jdbc_4.io.TDNetworkIOIF.connectToHost(TDNetworkIOIF.java:281)
 at com.teradata.jdbc.jdbc_4.io.TDNetworkIOIF.createSocketConnection(TDNetworkIOIF.java:130)
 ... 36 more
 at com.teradata.hadoop.mapreduce.TeradataProcessorBase.openConnection(TeradataProcessorBase.java:71)
 at com.teradata.hadoop.mapreduce.TeradataOutputProcessor.openConnection(TeradataOutputProcessor.java:115)
 at com.teradata.hadoop.mapreduce.TeradataOutputProcessor.setup(TeradataOutputProcessor.java:54)
 at com.teradata.hadoop.mapreduce.TeradataBatchInsertOutputProcessor.setup(TeradataBatchInsertOutputProcessor.java:60)
 at com.teradata.hadoop.job.TeradataExportJob.runJob(TeradataExportJob.java:72)
 at com.teradata.hadoop.tool.TeradataJobRunner.runExportJob(TeradataJobRunner.java:192)
 at com.teradata.hadoop.tool.TeradataExportTool.run(TeradataExportTool.java:41)
 at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
 at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
 at com.teradata.hadoop.tool.TeradataExportTool.main(TeradataExportTool.java:439)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:597)
 at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:491)
 at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
 at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:429)
 at org.apache.hadoop.mapred.MapTask.run(MapTask.java:365)
 at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
 at java.security.AccessController.doPrivileged(Native Method)
 at javax.security.auth.Subject.doAs(Subject.java:396)
 at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
 at org.apache.hadoop.mapred.Child.main(Child.java:249)
05/28 21:06:04  INFO TeradataExportTool: job completed with exit code 10000
05/28 21:06:05  INFO Task: Task:attempt_201405281927_0003_m_000000_0 is done. And is in the process of commiting
05/28 21:06:06  INFO Task: Task attempt_201405281927_0003_m_000000_0 is allowed to commit now
05/28 21:06:06  INFO FileOutputCommitter: Saved output of task 'attempt_201405281927_0003_m_000000_0' to hdfs://sandbox:8020/user/root/oozie-oozi/0000008-140528192712135-oozie-oozi-W/java-node--java/output
05/28 21:06:06  INFO Task: Task 'attempt_201405281927_0003_m_000000_0' done.
05/28 21:06:06  INFO TaskLogsTruncater: Initializing logs' truncater with mapRetainSize=-1 and reduceRetainSize=-1
05/28 21:06:06  INFO NativeIO: Initialized cache for UID to User mapping with a cache timeout of 14400 seconds.
05/28 21:06:06  INFO NativeIO: Got UserName mapred for UID 498 from the native implementation
I followed all the instructions in the README file from TDCH for Teradata Studio 1.0.9.
I also add the following properties in core-site.xml
   <property>
      <name>hadoop.proxyuser.oozie.groups</name>
   <value>*</value>
   </property>
   <property>
      <name>hadoop.proxyuser.oozie.hosts</name>
   <value>*</value>
   </property>
   <property>
   <name>hadoop.proxyuser.root.hosts</name>
   <value>*</value>
   </property>
I am using the following:

  • HDP 1.3.7 vm
  • TD 13.0.0.2 express in Windows 7 (32-bit)
  • Teradata Studio 15.0

 

fgrimmer 660 comments Joined 04/09
29 May 2014

Joseph, It looks like you are getting an Unknown Host Exception from the TDCH when trying to connect back to your Teradata Database. Studio constructs the JDBC URL for the Teradata Database from the connection profile Database Server Name and passes it to the TDCH on the Hadoop side. It may be that the name is not known to the Hadoop system. Can you try using the IP address of your Teradata Database for the Database Server Name in your connection profile?

03 Jun 2014

Hi Francine
Just to add from my previous comment, Teradata Studio was able to create a table in Teradata Express 13. But it was not able to load the data.

fgrimmer 660 comments Joined 04/09
03 Jun 2014

Joseph, The problem above is from the TDCH trying to connect back to your Teradata 13.00 Database. Try using the IP address in the Teradata Studio connection profile as the Database Server Name.

fgrimmer 660 comments Joined 04/09
03 Jun 2014

Sorry, the IP address of localtdcop1.

03 Jun 2014

Hi Francine
My local TD Express 13 is using the default localhost IP address of 127.0.0.1
# localhost name resolution is handled within DNS itself.
 127.0.0.1       localhost localtd  localtdcop1 tdadmin
 ::1             localhost
I'll try to use 127.0.0.1 for the TD 13 JDBC connection.

03 Jun 2014

Hi Francine
 
Same output: Studio was able to create the table in TD Express 13 but was not able to load the data from Hadoop even though I use 127.0.0.1 IP address as DSN for the TD Express 13 JDBC connection.
 
Below is the output log:
 
06/03 23:04:47  INFO TeradataHiveUtils: Using hive-site.xml: hdfs://sandbox:8020/teradata/oozieworkflows/hive-site.xml
06/03 23:04:48  INFO HiveMetaStoreClient: Trying to connect to metastore with URI thrift://sandbox:9083
06/03 23:04:48  INFO HiveMetaStoreClient: Waiting 1 seconds before next connection attempt.
06/03 23:04:49  INFO HiveMetaStoreClient: Connected to metastore.
06/03 23:05:00  INFO TeradataOutputProcessor: job setup starts at 1401861900073
06/03 23:05:10  INFO TeradataOutputProcessor: job setup ends at 1401861910245
06/03 23:05:10  INFO TeradataOutputProcessor: job setup time is 10s
06/03 23:05:10 ERROR TeradataExportTool: com.teradata.hadoop.exception.TeradataHadoopException: com.teradata.jdbc.jdbc_4.util.JDBCException: [Teradata JDBC Driver] [TeraJDBC 14.00.00.01] [Error 1000] [SQLState 08S01] Login failure for Connection to 127.0.0.1 Tue Jun 03 23:05:09 PDT 2014 socket orig=127.0.0.1 cid=5a9b8ff9 sess=0 java.net.ConnectException: Connection refused  at java.net.PlainSocketImpl.socketConnect(Native Method)  at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351)  at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213)  at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200)  at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)  at java.net.Socket.connect(Socket.java:529)  at com.teradata.jdbc.jdbc_4.io.TDNetworkIOIF$ConnectThread.run(TDNetworkIOIF.java:968)
 at com.teradata.jdbc.jdbc_4.util.ErrorFactory.makeDriverJDBCException(ErrorFactory.java:93)
 at com.teradata.jdbc.jdbc_4.util.ErrorFactory.makeDriverJDBCException(ErrorFactory.java:68)
 at com.teradata.jdbc.jdbc_4.util.ErrorFactory.makeIoJDBCException(ErrorFactory.java:206)
 at com.teradata.jdbc.jdbc_4.util.ErrorAnalyzer.analyzeIoError(ErrorAnalyzer.java:61)
 at com.teradata.jdbc.jdbc_4.io.TDNetworkIOIF.createSocketConnection(TDNetworkIOIF.java:137)
 at com.teradata.jdbc.jdbc_4.io.TDNetworkIOIF.<init>(TDNetworkIOIF.java:116)
 at com.teradata.jdbc.jdbc_4.TDSession.getIO(TDSession.java:579)
 at com.teradata.jdbc.jdbc.GenericLogonController.run(GenericLogonController.java:94)
 at com.teradata.jdbc.jdbc_4.TDSession.<init>(TDSession.java:208)
 at com.teradata.jdbc.jdk6.JDK6_SQL_Connection.<init>(JDK6_SQL_Connection.java:30)
 at com.teradata.jdbc.jdk6.JDK6ConnectionFactory.constructConnection(JDK6ConnectionFactory.java:22)
 at com.teradata.jdbc.jdbc.ConnectionFactory.createConnection(ConnectionFactory.java:130)
 at com.teradata.jdbc.jdbc.ConnectionFactory.createConnection(ConnectionFactory.java:120)
 at com.teradata.jdbc.TeraDriver.doConnect(TeraDriver.java:220)
 at com.teradata.jdbc.TeraDriver.connect(TeraDriver.java:153)
 at java.sql.DriverManager.getConnection(DriverManager.java:582)
 at java.sql.DriverManager.getConnection(DriverManager.java:185)
 at com.teradata.hadoop.db.TeradataConnection.connect(TeradataConnection.java:284)
 at com.teradata.hadoop.mapreduce.TeradataProcessorBase.openConnection(TeradataProcessorBase.java:68)
 at com.teradata.hadoop.mapreduce.TeradataOutputProcessor.openConnection(TeradataOutputProcessor.java:115)
 at com.teradata.hadoop.mapreduce.TeradataOutputProcessor.setup(TeradataOutputProcessor.java:54)
 at com.teradata.hadoop.mapreduce.TeradataBatchInsertOutputProcessor.setup(TeradataBatchInsertOutputProcessor.java:60)
 at com.teradata.hadoop.job.TeradataExportJob.runJob(TeradataExportJob.java:72)
 at com.teradata.hadoop.tool.TeradataJobRunner.runExportJob(TeradataJobRunner.java:192)
 at com.teradata.hadoop.tool.TeradataExportTool.run(TeradataExportTool.java:41)
 at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
 at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
 at com.teradata.hadoop.tool.TeradataExportTool.main(TeradataExportTool.java:439)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:597)
 at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:491)
 at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
 at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:429)
 at org.apache.hadoop.mapred.MapTask.run(MapTask.java:365)
 at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
 at java.security.AccessController.doPrivileged(Native Method)
 at javax.security.auth.Subject.doAs(Subject.java:396)
 at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
 at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: java.net.ConnectException: Connection refused
 at java.net.PlainSocketImpl.socketConnect(Native Method)
 at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351)
 at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213)
 at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200)
 at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
 at java.net.Socket.connect(Socket.java:529)
 at com.teradata.jdbc.jdbc_4.io.TDNetworkIOIF$ConnectThread.run(TDNetworkIOIF.java:968)
 at com.teradata.hadoop.mapreduce.TeradataProcessorBase.openConnection(TeradataProcessorBase.java:71)
 at com.teradata.hadoop.mapreduce.TeradataOutputProcessor.openConnection(TeradataOutputProcessor.java:115)
 at com.teradata.hadoop.mapreduce.TeradataOutputProcessor.setup(TeradataOutputProcessor.java:54)
 at com.teradata.hadoop.mapreduce.TeradataBatchInsertOutputProcessor.setup(TeradataBatchInsertOutputProcessor.java:60)
 at com.teradata.hadoop.job.TeradataExportJob.runJob(TeradataExportJob.java:72)
 at com.teradata.hadoop.tool.TeradataJobRunner.runExportJob(TeradataJobRunner.java:192)
 at com.teradata.hadoop.tool.TeradataExportTool.run(TeradataExportTool.java:41)
 at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
 at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
 at com.teradata.hadoop.tool.TeradataExportTool.main(TeradataExportTool.java:439)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:597)
 at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:491)
 at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
 at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:429)
 at org.apache.hadoop.mapred.MapTask.run(MapTask.java:365)
 at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
 at java.security.AccessController.doPrivileged(Native Method)
 at javax.security.auth.Subject.doAs(Subject.java:396)
 at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
 at org.apache.hadoop.mapred.Child.main(Child.java:249)
06/03 23:05:10  INFO TeradataExportTool: job completed with exit code 10000
06/03 23:05:11  INFO Task: Task:attempt_201406032249_0001_m_000000_0 is done. And is in the process of commiting
06/03 23:05:12  INFO Task: Task attempt_201406032249_0001_m_000000_0 is allowed to commit now
06/03 23:05:12  INFO FileOutputCommitter: Saved output of task 'attempt_201406032249_0001_m_000000_0' to hdfs://sandbox:8020/user/hive/oozie-oozi/0000003-140603224852558-oozie-oozi-W/java-node--java/output
06/03 23:05:12  INFO Task: Task 'attempt_201406032249_0001_m_000000_0' done.
06/03 23:05:13  INFO TaskLogsTruncater: Initializing logs' truncater with mapRetainSize=-1 and reduceRetainSize=-1
06/03 23:05:13  INFO NativeIO: Initialized cache for UID to User mapping with a cache timeout of 14400 seconds.
06/03 23:05:13  INFO NativeIO: Got UserName mapred for UID 498 from the native implementation

03 Jun 2014

Hi Francine
 
Below is the output log when I try to drag and drop a TD Express 13 table to Hadoop using Studio 15:
 
06/03 23:16:25  INFO TeradataHiveUtils: Using hive-site.xml: hdfs://sandbox:8020/teradata/oozieworkflows/hive-site.xml
06/03 23:16:25  INFO HiveMetaStoreClient: Trying to connect to metastore with URI thrift://sandbox:9083
06/03 23:16:25  INFO HiveMetaStoreClient: Waiting 1 seconds before next connection attempt.
06/03 23:16:26  INFO HiveMetaStoreClient: Connected to metastore.
06/03 23:16:27  INFO TeradataInputProcessor: job setup starts at 1401862587720
06/03 23:16:28  INFO TeradataInputProcessor: job setup ends at 1401862588323
06/03 23:16:28  INFO TeradataInputProcessor: job setup time is 0s
06/03 23:16:28 ERROR TeradataImportTool: com.teradata.hadoop.exception.TeradataHadoopException: com.teradata.jdbc.jdbc_4.util.JDBCException: [Teradata JDBC Driver] [TeraJDBC 14.00.00.01] [Error 1000] [SQLState 08S01] Login failure for Connection to 127.0.0.1 Tue Jun 03 23:16:28 PDT 2014 socket orig=127.0.0.1 cid=7f60c4b0 sess=0 java.net.ConnectException: Connection refused  at java.net.PlainSocketImpl.socketConnect(Native Method)  at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351)  at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213)  at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200)  at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)  at java.net.Socket.connect(Socket.java:529)  at com.teradata.jdbc.jdbc_4.io.TDNetworkIOIF$ConnectThread.run(TDNetworkIOIF.java:968)
 at com.teradata.jdbc.jdbc_4.util.ErrorFactory.makeDriverJDBCException(ErrorFactory.java:93)
 at com.teradata.jdbc.jdbc_4.util.ErrorFactory.makeDriverJDBCException(ErrorFactory.java:68)
 at com.teradata.jdbc.jdbc_4.util.ErrorFactory.makeIoJDBCException(ErrorFactory.java:206)
 at com.teradata.jdbc.jdbc_4.util.ErrorAnalyzer.analyzeIoError(ErrorAnalyzer.java:61)
 at com.teradata.jdbc.jdbc_4.io.TDNetworkIOIF.createSocketConnection(TDNetworkIOIF.java:137)
 at com.teradata.jdbc.jdbc_4.io.TDNetworkIOIF.<init>(TDNetworkIOIF.java:116)
 at com.teradata.jdbc.jdbc_4.TDSession.getIO(TDSession.java:579)
 at com.teradata.jdbc.jdbc.GenericLogonController.run(GenericLogonController.java:94)
 at com.teradata.jdbc.jdbc_4.TDSession.<init>(TDSession.java:208)
 at com.teradata.jdbc.jdk6.JDK6_SQL_Connection.<init>(JDK6_SQL_Connection.java:30)
 at com.teradata.jdbc.jdk6.JDK6ConnectionFactory.constructConnection(JDK6ConnectionFactory.java:22)
 at com.teradata.jdbc.jdbc.ConnectionFactory.createConnection(ConnectionFactory.java:130)
 at com.teradata.jdbc.jdbc.ConnectionFactory.createConnection(ConnectionFactory.java:120)
 at com.teradata.jdbc.TeraDriver.doConnect(TeraDriver.java:220)
 at com.teradata.jdbc.TeraDriver.connect(TeraDriver.java:153)
 at java.sql.DriverManager.getConnection(DriverManager.java:582)
 at java.sql.DriverManager.getConnection(DriverManager.java:185)
 at com.teradata.hadoop.db.TeradataConnection.connect(TeradataConnection.java:284)
 at com.teradata.hadoop.mapreduce.TeradataProcessorBase.openConnection(TeradataProcessorBase.java:68)
 at com.teradata.hadoop.mapreduce.TeradataInputProcessor.openConnection(TeradataInputProcessor.java:108)
 at com.teradata.hadoop.mapreduce.TeradataInputProcessor.setup(TeradataInputProcessor.java:51)
 at com.teradata.hadoop.mapreduce.TeradataSplitByPartitionInputProcessor.setup(TeradataSplitByPartitionInputProcessor.java:67)
 at com.teradata.hive.job.TeradataHiveImportJob.runJob(TeradataHiveImportJob.java:225)
 at com.teradata.hadoop.tool.TeradataJobRunner.runImportJob(TeradataJobRunner.java:119)
 at com.teradata.hadoop.tool.TeradataImportTool.run(TeradataImportTool.java:41)
 at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
 at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
 at com.teradata.hadoop.tool.TeradataImportTool.main(TeradataImportTool.java:392)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:597)
 at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:491)
 at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
 at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:429)
 at org.apache.hadoop.mapred.MapTask.run(MapTask.java:365)
 at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
 at java.security.AccessController.doPrivileged(Native Method)
 at javax.security.auth.Subject.doAs(Subject.java:396)
 at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
 at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: java.net.ConnectException: Connection refused
 at java.net.PlainSocketImpl.socketConnect(Native Method)
 at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351)
 at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213)
 at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200)
 at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
 at java.net.Socket.connect(Socket.java:529)
 at com.teradata.jdbc.jdbc_4.io.TDNetworkIOIF$ConnectThread.run(TDNetworkIOIF.java:968)
 at com.teradata.hadoop.mapreduce.TeradataProcessorBase.openConnection(TeradataProcessorBase.java:71)
 at com.teradata.hadoop.mapreduce.TeradataInputProcessor.openConnection(TeradataInputProcessor.java:108)
 at com.teradata.hadoop.mapreduce.TeradataInputProcessor.setup(TeradataInputProcessor.java:51)
 at com.teradata.hadoop.mapreduce.TeradataSplitByPartitionInputProcessor.setup(TeradataSplitByPartitionInputProcessor.java:67)
 at com.teradata.hive.job.TeradataHiveImportJob.runJob(TeradataHiveImportJob.java:225)
 at com.teradata.hadoop.tool.TeradataJobRunner.runImportJob(TeradataJobRunner.java:119)
 at com.teradata.hadoop.tool.TeradataImportTool.run(TeradataImportTool.java:41)
 at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
 at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
 at com.teradata.hadoop.tool.TeradataImportTool.main(TeradataImportTool.java:392)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:597)
 at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:491)
 at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
 at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:429)
 at org.apache.hadoop.mapred.MapTask.run(MapTask.java:365)
 at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
 at java.security.AccessController.doPrivileged(Native Method)
 at javax.security.auth.Subject.doAs(Subject.java:396)
 at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
 at org.apache.hadoop.mapred.Child.main(Child.java:249)
06/03 23:16:28  INFO TeradataImportTool: job completed with exit code 10000
06/03 23:16:28  INFO Task: Task:attempt_201406032249_0002_m_000000_0 is done. And is in the process of commiting
06/03 23:16:29  INFO Task: Task attempt_201406032249_0002_m_000000_0 is allowed to commit now
06/03 23:16:29  INFO FileOutputCommitter: Saved output of task 'attempt_201406032249_0002_m_000000_0' to hdfs://sandbox:8020/user/hive/oozie-oozi/0000005-140603224852558-oozie-oozi-W/java-node--java/output
06/03 23:16:29  INFO Task: Task 'attempt_201406032249_0002_m_000000_0' done.
06/03 23:16:29  INFO TaskLogsTruncater: Initializing logs' truncater with mapRetainSize=-1 and reduceRetainSize=-1
06/03 23:16:29  INFO NativeIO: Initialized cache for UID to User mapping with a cache timeout of 14400 seconds.
06/03 23:16:29  INFO NativeIO: Got UserName mapred for UID 498 from the native implementation
 
The table was not created.

03 Jun 2014

Hi Francine.
 
localtdcop1 is using the same localhost IP address of 127.0.0.1

04 Jun 2014

Any help guys regarding on my posted issues?

fgrimmer 660 comments Joined 04/09
04 Jun 2014

Joseph, You need the IP address that is accessed from outside the VM so that Hadoop can connect. Find the external IP address for the VM and test connecting Teradata Studio from outside the VM using that IP address. Here is an article that should help: http://developer.teradata.com/database/articles/teradata-express-network-configuration. If that works than use that IP address for your Connection Profile.

04 Jun 2014

Hi Francine
 
TD Express 13 is installed locally in my laptop and Hadoop resides in the VM.

fgrimmer 660 comments Joined 04/09
04 Jun 2014

Joseph, Is TD Express EC2 or VMWare version?

04 Jun 2014

Hi Francine
 
My TD Express 13 is installed locally in my Windows 7 laptop.

04 Jun 2014

Francine, Furthermore, my Hadoop is a VirtualBox VM that is also in my laptop

04 Jun 2014

Hi Francine
 
I was able to load the Hive table to my local TD Express 13 by changing the VirtualBox's network type to Bridge and changing localtdcop1 IP address to my laptop's ethernet IP address.

fgrimmer 660 comments Joined 04/09
05 Jun 2014

Joseph, glad it is working.

05 Jun 2014

Hi Francine
 
I'm happy that I nailed it :) 
May I know if there will be a feature in Studio that I can automate the Smart Load process, kinda-like, generating a script so that I can automate it or call it without using Studio. And, will there be in future versions of Studio that we can browse the contents of a Hive table. That will be cool :)

fgrimmer 660 comments Joined 04/09
05 Jun 2014

Joseph, There currently are not plans to automate or provide a command line interface for Smart Load to Hadoop, but you can re-execute the Smart Load job from the Transfer History. Once you have created the Transfer Job entry for your Smart Load in the Transfer History, select the job entry and click the 're-execute' button in the toolbar. It will bring up the Transfer Hadoop wizard and allow you to append or replace the data.
On another note, we are planning to provide SQL query options for Hadoop.

05 Jun 2014

That's great! I'm looking forward on future versions of Studio.

05 Jun 2014

Hi Francine
 

I have a couple of questions regarding the Hadoop connectors:

  • What connector do I need to integrate the Hadoop and Teradata using an ETL tool like Talend or Informatica PowerCenter?
  • Can I use all of the 3 connectors side-by-side and deploy it in a sandbox environment?

 

Regards,

Joseph

fgrimmer 660 comments Joined 04/09
05 Jun 2014

Joseph, I would like to answer these questions for you but I am not an expert on the Teradata Connectors. Please post your question to the article: http://developer.teradata.com/connectivity/articles/teradata-connector-for-hadoop-now-available or the Extensibility Forum: http://forums.teradata.com/forum/extensibility.

05 Jun 2014

Francine, no worries. I posted a new topic at http://forums.teradata.com/forum/extensibility/about-teradata-hadoop-connectors . thanks for all the help :)

ahsan.khan 4 comments Joined 07/14
09 Oct 2014

I used TD 15 Express VM, TD Studio 15 and HDP 2.1 , and installed TDCH rpm file on Hadoop /usr/lib and it worked. I specified default webHDFS http://sandbox.hortonworks.com and port 50070 and default user. So that should work for you too @umerbaig.
However, whenever I transfer data from Hadoop to TD 15 or from TD 15 to Hadoop, the upper bound on table transfer was always 10 minutes. Sometimes, the full table was not transferred even. I transferred default.sample_07 to TD and financials.accts to Hadoop. The second one was not completely transferred. What should be done @fgrimmer?

fgrimmer 660 comments Joined 04/09
09 Oct 2014

Ahsan, We are looking into this issue.

fgrimmer 660 comments Joined 04/09
09 Oct 2014

Ahsan, Is there an error message that is returned? It would be in the Studio Transfer History view, select the transfer job in the list and click the Job Details toolbar button to see the output from the data transfer job.

kmehra1 2 comments Joined 10/11
27 Jan 2015

Hi fgrimmer,
We are trying to transfer data from hadoop HDP 1.3  VM to Teradata 14.0.3 VM using  TDCH 1.0.9 and TD smart loader 15 and vice-versa.The data load completes succesfully but we are unable to see the logs for the Data Transfer Job.It shows the below error message.
Could not open file /user/hue/Job6408e30e-e58c-475c-b40c-ab3347e9b58e/output.txt
Connection refused: connect

Chuckbert 37 comments Joined 07/09
27 Jan 2015

kmehra1, this is strange. The message you say is shown about not being able to open the output.txt file ("Connection refused: connect") shouldn't happen! The connection being made to read the output file that is being refused is being made in the same way as had been done in setting up the data transfer job.

 

You apparently are running as the user "hue". Is it possible to set up a user that isn't one of the Hadoop services and try the data transfer with that user? That will require adding a directory in the HDFS directory /user with that user's name and making that user the owner of the directory.

 

I'm wondering if there is a restriction connecting to the Web HDFS service as the user "hue" (but such connections had already been made).

 

Are there any clues in the job's logs? To see the log (take a deep breath):

 

In a browser, open the page:

  • http://hostname:11000/oozie/

(where "hostname" is the name of the system running the Oozie service. The "11000" port number is the default, put in the actual port number if it has been changed.)

  • Click on the entry for the job whose output couldn't be read.
  • Click on the second entry in the resulting Job page (the one whose "Type" is "java")
  • Click on the magnifying lens icon in the "Console URL" entry in the resulting Action page
  • Click on the "map" link in the first table of the resulting web page.
  • Click on the first (only?) link in the "Task" column.
  • Click on the "All" link in the "Task Logs" column of the All Task Attemps table.
kmehra1 2 comments Joined 10/11
29 Jan 2015

Hi Chuckbert,
 
 Thanks for the response.I created the new user through web UI for the hadoop snadbox and created the directory with the same name under hdfs in path /user/newuser.With the same user when i tried connecting through Smart loader it is not going through.Its only going through user 'hue'.Let me know if ia mmissing something.
 
Also with the second option i am able to see the all the job logs but this error message is not there.One thing i noticed is this  /user/hue/Job6408e30e-e58c-475c-b40c-ab3347e9b58e/output.txt      path it is picking up from configureoozie.sh script which i executed for configuring oozie on TDCH.
below is the Code from the logs
 
 

Main class        : com.teradata.hadoop.tool.TeradataImportTool

 

Maximum output    : 2048

 

Arguments         :

                    -libjars

                    hdfs://10.12.116.152:8020/teradata/oozieworkflows/oozielib/antlr-runtime-3.4.jar,hdfs://10.12.116.152:8020/teradata/oozieworkflows/oozielib/commons-dbcp-1.4.jar,hdfs://10.12.116.152:8020/teradata/oozieworkflows/oozielib/commons-pool-1.5.4.jar,hdfs://10.12.116.152:8020/teradata/oozieworkflows/oozielib/datanucleus-api-jdo-3.0.7.jar,hdfs://10.12.116.152:8020/teradata/oozieworkflows/oozielib/datanucleus-core-3.0.9.jar,hdfs://10.12.116.152:8020/teradata/oozieworkflows/oozielib/datanucleus-rdbms-3.0.8.jar,hdfs://10.12.116.152:8020/teradata/oozieworkflows/oozielib/hcatalog-core-0.11.0.1.3.0.0-107.jar,hdfs://10.12.116.152:8020/teradata/oozieworkflows/oozielib/hive-cli-0.11.0.1.3.0.0-107.jar,hdfs://10.12.116.152:8020/teradata/oozieworkflows/oozielib/hive-exec-0.11.0.1.3.0.0-107.jar,hdfs://10.12.116.152:8020/teradata/oozieworkflows/oozielib/hive-metastore-0.11.0.1.3.0.0-107.jar,hdfs://10.12.116.152:8020/teradata/oozieworkflows/oozielib/jdo2-api-2.3-ec.jar,hdfs://10.12.116.152:8020/teradata/oozieworkflows/oozielib/libfb303-0.9.0.jar,hdfs://10.12.116.152:8020/teradata/oozieworkflows/oozielib/libthrift-0.9.0.jar,hdfs://10.12.116.152:8020/teradata/oozieworkflows/oozielib/mysql-connector-java.jar,hdfs://10.12.116.152:8020/teradata/oozieworkflows/oozielib/slf4j-api-1.6.1.jar,hdfs://10.12.116.152:8020/teradata/oozieworkflows/oozielib/tdgssconfig.jar,hdfs://10.12.116.152:8020/teradata/oozieworkflows/oozielib/teradata-connector-1.0.9.jar,hdfs://10.12.116.152:8020/teradata/oozieworkflows/oozielib/terajdbc4.jar

                    -jobtype

                    hive

                    -fileformat

                    textfile

                    -separator

                    \u0009

                    -classname

                    com.teradata.jdbc.TeraDriver

                    -url

                    jdbc:teradata://10.12.116.167/CHARSET=UTF8,database=financial

                    -username

                    dbc

                    -password

                    dbc

                    -method

                    split.by.partition

                    -batchsize

                    10000

                    -splitbycolumn

                    bogus

                    -sourcequery

                    SELECT "cust_id","acct_type","acct_nbr","acct_start_date","acct_end_date" FROM "accts"

                    -targettableschema

                    cust_id int,acct_type string,acct_nbr string,acct_start_date string,acct_end_date string

                    -targetdatabase

                    default

                    -targettable

                    accts

                    -hiveconf

                    hdfs://10.12.116.152:8020/teradata/oozieworkflows/hive-site.xml

                    -jobclientoutput

                /user/hue/Jobd2b695d4-b042-4924-9634-8f55a7480072/output.txt

13 Feb 2015

I have a question. Is gpfs (not hdfs) supported? Will SmartLoader work with gpfs.  Thanks, Lana
btw - I did see post from March 13 2014 and Francine asked if IP address was tried. I did try the ip address - it did not help.

fgrimmer 660 comments Joined 04/09
19 Mar 2015

Lana, I forwarded your question to our Hadoop folks and have not heard back so I am assuming that it is NOT supported at this time. Sorry for the late response.

gschasie 9 comments Joined 01/10
31 Mar 2015

Hello fgrimmer,
 
I have the VM loom sandbox 2.1.0, VM TD14.10.3.02 and SqlStudio 14.2.02 up and running. ping is working in all directions.
I have a problem with import/export via smart loader.
If i import a hadoop table to teradata (via drag and drop)  the table is created in teradata. The transfer process starts and never comes to an end until i kill it. After a while the data is visible in the teradata table. The job status as t oexpect is  "Cancelled".
If i try to copy a teradata table to sandbox it also comes not to an end. After kill there is no table in hadoop created.
I can't figure out what is going wrong here.
Please help
br
Guido
 
 
 

gschasie 9 comments Joined 01/10
31 Mar 2015

It is SqlStudio 15.2.02

fgrimmer 660 comments Joined 04/09
31 Mar 2015

Guido,  I am not familiar with the VM loom sandbox. Does it support Oozie? what version of Hadoop, Hive, HCatalog? Did you follow the instructions to install TDCH 1.3.1 and run the OozieConfig script?

gschasie 9 comments Joined 01/10
31 Mar 2015

Hi,
its the Hadoop loom from hortonworks : Teradata_Loom_2.3.0_with_HDP_GA.ova
Loom : Community | version: 2.3.0 | version date: 2014-11-14_15-35-03
Teradata_TDCH 1.3.1
# hadoop fs -cat  /teradata/tdch//services.json
{
        "Distribution":"HDP",
        "DistributionVersion":"2.1",
        "TeradataConnectorForHadoopVersion":"1.3.4",
        "WebHCatalog":"sandbox",
        "WebHCatalogPort":50111,
        "WebHDFS":"sandbox",
        "WebHDFSPort":50070,
        "JobTracker":"sandbox",
        "JobTrackerPort":8050,
        "NameNode":"sandbox",
        "NameNodePort":8020,
        "Oozie":"sandbox",
        "OoziePort":11000,
        "HiveClientMetastorePort":9083
}
 

I followed the instructions
For the Upload of haddop to teradata
Job definition
<!--
Copyright (C) 2013  by Teradata Corporation.  All rights reserved.
-->
<workflow-app xmlns="uri:oozie:workflow:0.2" name="TeradataExportToolByFields-wf">
    <start to="java-node"/>
    <action name="java-node">
        <java>
            <job-tracker>${jobTracker}</job-tracker>
            <name-node>${nameNode}</name-node>
            <configuration>
                <property>
                    <name>mapred.job.queue.name</name>
                    <value>${queueName}</value>
                </property>
                <property>
                    <name>oozie.launcher.mapred.child.java.opts</name>
                    <value>-Djava.security.egd=file:/dev/./urandom</value>
                </property>
            </configuration>
            <main-class>com.teradata.connector.common.tool.ConnectorExportTool</main-class>
            <java-opts>-Xmx1024m -Djava.security.egd=file:/dev/./urandom</java-opts>
            <arg>-libjars</arg>
            <arg>${libjars}</arg>
            <arg>-jobtype</arg>
            <arg>hive</arg>
            <arg>-fileformat</arg>
            <arg>${fileFormat}</arg>
            <arg>-separator</arg>
            <arg>${separator}</arg>
            <arg>-classname</arg>
            <arg>com.teradata.jdbc.TeraDriver</arg>
            <arg>-url</arg>
            <arg>${jdbcURL}</arg>
            <arg>-username</arg>
            <arg>${userName}</arg>
            <arg>-password</arg>
            <arg>${password}</arg>
            <arg>-method</arg>
            <arg>${method}</arg>
            <arg>-batchsize</arg>
            <arg>10000</arg>
            <arg>-sourcedatabase</arg>
            <arg>${sourceTableDatabase}</arg>
            <arg>-sourcetable</arg>
            <arg>${sourceTable}</arg>
            <arg>-sourcefieldnames</arg>
            <arg>${sourceColumnNames}</arg>
            <arg>-targettable</arg>
            <arg>${targetTable}</arg>
            <arg>-targetfieldnames</arg>
            <arg>${targetColumnNames}</arg>
            <arg>-hiveconf</arg>
            <arg>${hiveconf}</arg>
            <arg>-jobclientoutput</arg>
            <arg>${jobOutputFile}</arg>
        </java>
        <ok to="end"/>
        <error to="fail"/>
    </action>
    <kill name="fail">
        <message>Java failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
    </kill>
    <end name="end"/>
</workflow-app>

 
configuration
 
<configuration>
  <property>
    <name>libjars</name>
    <value>hdfs://sandbox:8020/teradata/hadoop/lib/antlr-runtime-3.4.jar,hdfs://sandbox:8020/teradata/hadoop/lib/commons-dbcp-1.4.jar,hdfs://sandbox:8020/teradata/hadoop/lib/commons-pool-1.5.4.jar,hdfs://sandbox:8020/teradata/hadoop/lib/datanucleus-api-jdo-3.2.6.jar,hdfs://sandbox:8020/teradata/hadoop/lib/datanucleus-core-3.2.10.jar,hdfs://sandbox:8020/teradata/hadoop/lib/datanucleus-rdbms-3.2.9.jar,hdfs://sandbox:8020/teradata/hadoop/lib/hive-cli-0.13.0.2.1.1.0-385.jar,hdfs://sandbox:8020/teradata/hadoop/lib/hive-exec-0.13.0.2.1.1.0-385.jar,hdfs://sandbox:8020/teradata/hadoop/lib/hive-hcatalog-core-0.13.0.2.1.1.0-385.jar,hdfs://sandbox:8020/teradata/hadoop/lib/hive-metastore-0.13.0.2.1.1.0-385.jar,hdfs://sandbox:8020/teradata/hadoop/lib/hive-service-0.13.0.2.1.1.0-385.jar,hdfs://sandbox:8020/teradata/hadoop/lib/hive-site.xml,hdfs://sandbox:8020/teradata/hadoop/lib/jdo-api-3.0.1.jar,hdfs://sandbox:8020/teradata/hadoop/lib/jline-0.9.94.jar,hdfs://sandbox:8020/teradata/hadoop/lib/libfb303-0.9.0.jar,hdfs://sandbox:8020/teradata/hadoop/lib/libthrift-0.9.0.jar,hdfs://sandbox:8020/teradata/hadoop/lib/mysql-connector-java.jar,hdfs://sandbox:8020/teradata/hadoop/lib/tez-api-0.4.0.2.1.1.0-385.jar,hdfs://sandbox:8020/teradata/hadoop/lib/tez-mapreduce-0.4.0.2.1.1.0-385.jar,hdfs://sandbox:8020/teradata/hadoop/lib/tez-mapreduce-examples-0.4.0.2.1.1.0-385.jar,hdfs://sandbox:8020/teradata/hadoop/lib/tez-site.xml</value>
  </property>
  <property>
    <name>splitByColumnName</name>
    <value>bogus</value>
  </property>
  <property>
    <name>hiveconf</name>
    <value>hdfs://sandbox:8020/teradata/hadoop/lib/hive-site.xml</value>
  </property>
  <property>
    <name>mapreduce.job.user.name</name>
    <value>guest</value>
  </property>
  <property>
    <name>jobOutputFile</name>
    <value>/user/guest/Job3d4068bf-b4ea-418b-94e2-30f06e4fcf44/output.txt</value>
  </property>
  <property>
    <name>sourceColumnNames</name>
    <value>code,description,total_emp,salary</value>
  </property>
  <property>
    <name>password</name>
    <value>dbc</value>
  </property>
  <property>
    <name>sourceTableDatabase</name>
    <value>default</value>
  </property>
  <property>
    <name>targetTable</name>
    <value>sample_08</value>
  </property>
  <property>
    <name>jdbcURL</name>
    <value>jdbc:teradata://192.168.2.190/CHARSET=UTF8,database=financial</value>
  </property>
  <property>
    <name>targetColumnNames</name>
    <value>"code","description","total_emp","salary"</value>
  </property>
  <property>
    <name>user.name</name>
    <value>guest</value>
  </property>
  <property>
    <name>oozie.libpath</name>
    <value>hdfs://sandbox:8020/teradata/tdch/1.3/lib,hdfs://sandbox:8020/teradata/hadoop/lib</value>
  </property>
  <property>
    <name>queueName</name>
    <value>default</value>
  </property>
  <property>
    <name>userName</name>
    <value>dbc</value>
  </property>
  <property>
    <name>method</name>
    <value>batch.insert</value>
  </property>
  <property>
    <name>nameNode</name>
    <value>hdfs://sandbox:8020</value>
  </property>
  <property>
    <name>sourceTable</name>
    <value>sample_08</value>
  </property>
  <property>
    <name>oozie.wf.application.path</name>
    <value>hdfs://sandbox:8020/teradata/tdch/1.3/oozieworkflows/ooziehadooptoteradata</value>
  </property>
  <property>
    <name>jobTracker</name>
    <value>http://sandbox:8050</value>
  </property>
  <property>
    <name>separator</name>
    <value>\u0009</value>
  </property>
  <property>
    <name>fileFormat</name>
    <value>textfile</value>
  </property>
</configuration>
 
 
log
 
2015-03-31 02:59:00,005  INFO ActionStartXCommand:539 - USER[guest] GROUP[-] TOKEN[] APP[TeradataExportToolByFields-wf] JOB[0000010-150331011433724-oozie-oozi-W] ACTION[0000010-150331011433724-oozie-oozi-W@:start:] Start action [0000010-150331011433724-oozie-oozi-W@:start:] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2015-03-31 02:59:00,005  WARN ActionStartXCommand:542 - USER[guest] GROUP[-] TOKEN[] APP[TeradataExportToolByFields-wf] JOB[0000010-150331011433724-oozie-oozi-W] ACTION[0000010-150331011433724-oozie-oozi-W@:start:] [***0000010-150331011433724-oozie-oozi-W@:start:***]Action status=DONE
2015-03-31 02:59:00,006  WARN ActionStartXCommand:542 - USER[guest] GROUP[-] TOKEN[] APP[TeradataExportToolByFields-wf] JOB[0000010-150331011433724-oozie-oozi-W] ACTION[0000010-150331011433724-oozie-oozi-W@:start:] [***0000010-150331011433724-oozie-oozi-W@:start:***]Action updated in DB!
2015-03-31 02:59:00,024  INFO ActionEndXCommand:539 - USER[guest] GROUP[-] TOKEN[] APP[TeradataExportToolByFields-wf] JOB[0000010-150331011433724-oozie-oozi-W] ACTION[0000010-150331011433724-oozie-oozi-W@:start:] end executor for wf action 0000010-150331011433724-oozie-oozi-W with wf job 0000010-150331011433724-oozie-oozi-W
2015-03-31 02:59:00,113  INFO ActionStartXCommand:539 - USER[guest] GROUP[-] TOKEN[] APP[TeradataExportToolByFields-wf] JOB[0000010-150331011433724-oozie-oozi-W] ACTION[0000010-150331011433724-oozie-oozi-W@java-node] Start action [0000010-150331011433724-oozie-oozi-W@java-node] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2015-03-31 02:59:00,744  WARN JavaActionExecutor:542 - USER[guest] GROUP[-] TOKEN[] APP[TeradataExportToolByFields-wf] JOB[0000010-150331011433724-oozie-oozi-W] ACTION[0000010-150331011433724-oozie-oozi-W@java-node] credentials is null for the action
2015-03-31 02:59:01,849  INFO JavaActionExecutor:539 - USER[guest] GROUP[-] TOKEN[] APP[TeradataExportToolByFields-wf] JOB[0000010-150331011433724-oozie-oozi-W] ACTION[0000010-150331011433724-oozie-oozi-W@java-node] Trying to get job [job_1427789705651_0005], attempt [1]
2015-03-31 02:59:01,872  INFO JavaActionExecutor:539 - USER[guest] GROUP[-] TOKEN[] APP[TeradataExportToolByFields-wf] JOB[0000010-150331011433724-oozie-oozi-W] ACTION[0000010-150331011433724-oozie-oozi-W@java-node] checking action, external ID [job_1427789705651_0005] status [RUNNING]
2015-03-31 02:59:01,874  WARN ActionStartXCommand:542 - USER[guest] GROUP[-] TOKEN[] APP[TeradataExportToolByFields-wf] JOB[0000010-150331011433724-oozie-oozi-W] ACTION[0000010-150331011433724-oozie-oozi-W@java-node] [***0000010-150331011433724-oozie-oozi-W@java-node***]Action status=RUNNING
2015-03-31 02:59:01,874  WARN ActionStartXCommand:542 - USER[guest] GROUP[-] TOKEN[] APP[TeradataExportToolByFields-wf] JOB[0000010-150331011433724-oozie-oozi-W] ACTION[0000010-150331011433724-oozie-oozi-W@java-node] [***0000010-150331011433724-oozie-oozi-W@java-node***]Action updated in DB!
2015-03-31 03:09:55,339  INFO JavaActionExecutor:539 - USER[guest] GROUP[-] TOKEN[] APP[TeradataExportToolByFields-wf] JOB[0000010-150331011433724-oozie-oozi-W] ACTION[0000010-150331011433724-oozie-oozi-W@java-node] Trying to get job [job_1427789705651_0005], attempt [1]
2015-03-31 03:09:55,534  INFO JavaActionExecutor:539 - USER[guest] GROUP[-] TOKEN[] APP[TeradataExportToolByFields-wf] JOB[0000010-150331011433724-oozie-oozi-W] ACTION[0000010-150331011433724-oozie-oozi-W@java-node] checking action, external ID [job_1427789705651_0005] status [RUNNING]
2015-03-31 03:20:55,404  INFO JavaActionExecutor:539 - USER[guest] GROUP[-] TOKEN[] APP[TeradataExportToolByFields-wf] JOB[0000010-150331011433724-oozie-oozi-W] ACTION[0000010-150331011433724-oozie-oozi-W@java-node] Trying to get job [job_1427789705651_0005], attempt [1]
2015-03-31 03:20:55,435  INFO JavaActionExecutor:539 - USER[guest] GROUP[-] TOKEN[] APP[TeradataExportToolByFields-wf] JOB[0000010-150331011433724-oozie-oozi-W] ACTION[0000010-150331011433724-oozie-oozi-W@java-node] checking action, external ID [job_1427789705651_0005] status [RUNNING]
2015-03-31 03:31:55,436  INFO JavaActionExecutor:539 - USER[guest] GROUP[-] TOKEN[] APP[TeradataExportToolByFields-wf] JOB[0000010-150331011433724-oozie-oozi-W] ACTION[0000010-150331011433724-oozie-oozi-W@java-node] Trying to get job [job_1427789705651_0005], attempt [1]
2015-03-31 03:31:55,465  INFO JavaActionExecutor:539 - USER[guest] GROUP[-] TOKEN[] APP[TeradataExportToolByFields-wf] JOB[0000010-150331011433724-oozie-oozi-W] ACTION[0000010-150331011433724-oozie-oozi-W@java-node] checking action, external ID [job_1427789705651_0005] status [RUNNING]
2015-03-31 03:42:55,457  INFO JavaActionExecutor:539 - USER[guest] GROUP[-] TOKEN[] APP[TeradataExportToolByFields-wf] JOB[0000010-150331011433724-oozie-oozi-W] ACTION[0000010-150331011433724-oozie-oozi-W@java-node] Trying to get job [job_1427789705651_0005], attempt [1]
2015-03-31 03:42:55,492  INFO JavaActionExecutor:539 - USER[guest] GROUP[-] TOKEN[] APP[TeradataExportToolByFields-wf] JOB[0000010-150331011433724-oozie-oozi-W] ACTION[0000010-150331011433724-oozie-oozi-W@java-node] checking action, external ID [job_1427789705651_0005] status [RUNNING]
2015-03-31 03:53:55,504  INFO JavaActionExecutor:539 - USER[guest] GROUP[-] TOKEN[] APP[TeradataExportToolByFields-wf] JOB[0000010-150331011433724-oozie-oozi-W] ACTION[0000010-150331011433724-oozie-oozi-W@java-node] Trying to get job [job_1427789705651_0005], attempt [1]
2015-03-31 03:53:55,534  INFO JavaActionExecutor:539 - USER[guest] GROUP[-] TOKEN[] APP[TeradataExportToolByFields-wf] JOB[0000010-150331011433724-oozie-oozi-W] ACTION[0000010-150331011433724-oozie-oozi-W@java-node] checking action, external ID [job_1427789705651_0005] status [RUNNING]
2015-03-31 04:04:55,535  INFO JavaActionExecutor:539 - USER[guest] GROUP[-] TOKEN[] APP[TeradataExportToolByFields-wf] JOB[0000010-150331011433724-oozie-oozi-W] ACTION[0000010-150331011433724-oozie-oozi-W@java-node] Trying to get job [job_1427789705651_0005], attempt [1]
2015-03-31 04:04:55,566  INFO JavaActionExecutor:539 - USER[guest] GROUP[-] TOKEN[] APP[TeradataExportToolByFields-wf] JOB[0000010-150331011433724-oozie-oozi-W] ACTION[0000010-150331011433724-oozie-oozi-W@java-node] checking action, external ID [job_1427789705651_0005] status [RUNNING]
2015-03-31 04:15:46,980  INFO KillXCommand:539 - USER[guest] GROUP[-] TOKEN[] APP[TeradataExportToolByFields-wf] JOB[0000010-150331011433724-oozie-oozi-W] ACTION[-] STARTED WorkflowKillXCommand for jobId=0000010-150331011433724-oozie-oozi-W
2015-03-31 04:15:47,044  WARN CoordActionUpdateXCommand:542 - USER[guest] GROUP[-] TOKEN[] APP[TeradataExportToolByFields-wf] JOB[0000010-150331011433724-oozie-oozi-W] ACTION[-] E1100: Command precondition does not hold before execution, [, coord action is null], Error Code: E1100
2015-03-31 04:15:47,044  INFO KillXCommand:539 - USER[guest] GROUP[-] TOKEN[] APP[TeradataExportToolByFields-wf] JOB[0000010-150331011433724-oozie-oozi-W] ACTION[-] ENDED WorkflowKillXCommand for jobId=0000010-150331011433724-oozie-oozi-W
2015-03-31 04:15:48,621  INFO CallbackServlet:539 - USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000010-150331011433724-oozie-oozi-W] ACTION[0000010-150331011433724-oozie-oozi-W@java-node] callback for action [0000010-150331011433724-oozie-oozi-W@java-node]
2015-03-31 04:15:48,630 ERROR CompletedActionXCommand:536 - USER[-] GROUP[-] TOKEN[] APP[-] JOB[0000010-150331011433724-oozie-oozi-W] ACTION[0000010-150331011433724-oozie-oozi-W@java-node] XException,
org.apache.oozie.command.CommandException: E0800: Action it is not running its in [KILLED] state, action [0000010-150331011433724-oozie-oozi-W@java-node]
    at org.apache.oozie.command.wf.CompletedActionXCommand.eagerVerifyPrecondition(CompletedActionXCommand.java:85)
    at org.apache.oozie.command.XCommand.call(XCommand.java:251)
    at org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:175)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:744)
 
 

gschasie 9 comments Joined 01/10
31 Mar 2015

3/31 02:59:20  INFO Configuration: mapred.input.dir is deprecated. Instead, use mapreduce.input.fileinputformat.inputdir
03/31 02:59:21  INFO HiveUtils: Using hive-site.xml: hdfs://sandbox:8020/teradata/hadoop/lib/hive-site.xml
03/31 02:59:21  INFO HiveMetaStoreClient: Trying to connect to metastore with URI thrift://sandbox.hortonworks.com:9083
03/31 02:59:21  INFO HiveMetaStoreClient: Connected to metastore.
03/31 02:59:21  INFO TeradataOutputProcessor: output preprocessor com.teradata.connector.teradata.processor.TeradataBatchInsertProcessor starts at:  1427795961697
03/31 02:59:22  INFO TeradataUtils: the output database product is Teradata
03/31 02:59:22  INFO TeradataUtils: the output database version is 14.10
03/31 02:59:22  INFO TeradataUtils: the jdbc driver version is 15.0
03/31 02:59:22  INFO TeradataOutputProcessor: the teradata connector for hadoop version is: 1.3.4
03/31 02:59:22  INFO TeradataOutputProcessor: output jdbc properties are jdbc:teradata://192.168.2.190/CHARSET=UTF8,database=financial
03/31 02:59:23  INFO TeradataBatchInsertProcessor: output staging table is not needed
03/31 02:59:23  INFO TeradataOutputProcessor: the number of mappers are 2
03/31 02:59:23  INFO TeradataOutputProcessor: output preprocessor com.teradata.connector.teradata.processor.TeradataBatchInsertProcessor ends at:  1427795963222
03/31 02:59:23  INFO TeradataOutputProcessor: the total elapsed time of output preprocessor com.teradata.connector.teradata.processor.TeradataBatchInsertProcessor is: 1s
03/31 02:59:23  INFO RMProxy: Connecting to ResourceManager at sandbox.hortonworks.com/192.168.2.170:8050
03/31 02:59:23  INFO FileInputFormat: Total input paths to process : 1
03/31 02:59:23  INFO FileInputFormat: Total input paths to process : 1
03/31 02:59:23  INFO CombineFileInputFormat: DEBUG: Terminated node allocation with : CompletedNodes: 1, size left: 0
03/31 02:59:23  INFO JobSubmitter: number of splits:2
03/31 02:59:23  INFO JobSubmitter: Submitting tokens for job: job_1427789705651_0006
03/31 02:59:23  INFO JobSubmitter: Kind: mapreduce.job, Service: 192.168.2.170:37263, Ident: (org.apache.hadoop.mapreduce.security.token.JobTokenIdentifier@4bdc840b)
03/31 02:59:23  INFO JobSubmitter: Kind: RM_DELEGATION_TOKEN, Service: 192.168.2.170:8050, Ident: (owner=guest, renewer=oozie mr token, realUser=oozie, issueDate=1427795941379, maxDate=1428400741379, sequenceNumber=16, masterKeyId=2)
03/31 02:59:24  INFO YarnClientImpl: Submitted application application_1427789705651_0006
03/31 02:59:24  INFO Job: The url to track the job: http://sandbox.hortonworks.com:8088/proxy/application_1427789705651_0006/
03/31 02:59:24  INFO Job: Running job: job_1427789705651_0006
 

gschasie 9 comments Joined 01/10
31 Mar 2015

Transfer Hadoop table default.sample_08 to Teradata database Samples    2015-03-31 12:00:17.416    HADOOP:Hadoop Sandbox    TERADATA:Teradada 14    Canceled    4609162    0    null    Hadoop Sandbox.default.sample_08 TO Teradada 14.SQLJ.sample_08

gschasie 9 comments Joined 01/10
31 Mar 2015

created table by transfer
CREATE MULTISET TABLE financial.sample_08 ,NO FALLBACK ,
     NO BEFORE JOURNAL,
     NO AFTER JOURNAL,
     CHECKSUM = DEFAULT,
     DEFAULT MERGEBLOCKRATIO
     (
      code VARCHAR(2048) CHARACTER SET LATIN CASESPECIFIC,
      description VARCHAR(2048) CHARACTER SET LATIN CASESPECIFIC,
      total_emp INTEGER,
      salary INTEGER)
NO PRIMARY INDEX ;
 
00-0000,All Occupations,135185230,42270
11-0000,Management occupations,6152650,100310
11-1011,Chief executives,301930,160440
11-1021,General and operations managers,1697690,107970
11-1031,Legislators,64650,37980
11-2011,Advertising and promotions managers,36100,94720
11-2021,Marketing managers,166790,118160
11-2022,Sales managers,333910,110390
11-2031,Public relations managers,51730,101220
11-3011,Administrative services managers,246930,79500
11-3021,Computer and information systems managers,276820,118710
11-3031,Financial managers,500590,110640
11-3041,Compensation and benefits managers,38810,93410
11-3042,Training and development managers,29350,93830
11-3049,"Human resources managers, all other",60980,103920
11-3051,Industrial production managers,154030,91200
11-3061,Purchasing managers,67150,94300
11-3071,"Transportation, storage, and distribution managers",96300,84520
11-9011,"Farm, ranch, and other agricultural managers",3410,62400
11-9012,Farmers and ranchers,490,49140
11-9021,Construction managers,220550,89770
11-9031,"Education administrators, preschool and child care center/program",49630,46370
11-9032,"Education administrators, elementary and secondary school",219100,86060
11-9033,"Education administrators, postsecondary",97410,92920
11-9039,"Education administrators, all other",28090,77220
11-9041,Engineering managers,182300,120580
11-9051,Food service managers,196080,50320
11-9061,Funeral directors,25680,58810
11-9071,Gaming managers,3790,73480
11-9081,Lodging managers,32460,52550
11-9111,Medical and health services managers,258130,88750
11-9121,Natural sciences managers,43060,123140
11-9131,Postmasters and mail superintendents,26410,59150
11-9141,"Property, real estate, and community association managers",159700,56250
11-9151,Social and community service managers,117150,60570
11-9199,"Managers, all other",365460,95890
13-0000,Business and financial operations occupations,6135520,64720
13-1011,"Agents and business managers of artists, performers, and athletes",12110,81550
13-1021,"Purchasing agents and buyers, farm products",13010,58760
13-1022,"Wholesale and retail buyers, except farm products",132420,55540
13-1023,"Purchasing agents, except wholesale, retail, and farm products",286990,57630
13-1031,"Claims adjusters, examiners, and investigators",277230,57550
13-1032,"Insurance appraisers, auto damage",11280,53980
13-1041,"Compliance officers, except agriculture, construction, health and safety, and transportation",242270,53760
13-1051,Cost estimators,218400,60320
13-1061,Emergency management specialists,12260,53460
13-1071,"Employment, recruitment, and placement specialists",205800,53870
13-1072,"Compensation, benefits, and job analysis specialists",116250,57060
13-1073,Training and development specialists,206890,54830
13-1079,"Human resources, training, and labor relations specialists, all other",217440,58230
13-1081,Logisticians,98590,68600
13-1111,Management analysts,535850,82920
13-1121,Meeting and convention planners,47960,47500
13-1199,"Business operations specialists, all other",1030320,64990
13-2011,Accountants and auditors,1133580,65840
13-2021,Appraisers and assessors of real estate,66260,53410
13-2031,Budget analysts,62630,68140
13-2041,Credit analysts,74400,64580
13-2051,Financial analysts,236720,84780
13-2052,Personal financial advisors,146690,92970
13-2053,Insurance underwriters,98690,62600
13-2061,Financial examiners,26020,78180
13-2071,Loan counselors,29430,41970
13-2072,Loan officers,321850,63540
13-2081,"Tax examiners, collectors, and revenue agents",66030,53090
13-2082,Tax preparers,63030,35520
13-2099,"Financial specialists, all other",145110,63880
15-0000,Computer and mathematical science occupations,3308260,74500
15-1011,"Computer and information scientists, research",26610,100900
15-1021,Computer programmers,394230,73470
15-1031,"Computer software engineers, applications",494160,87900
15-1032,"Computer software engineers, systems software",381830,94520
15-1041,Computer support specialists,545520,46370
15-1051,Computer systems analysts,489890,78830
15-1061,Database administrators,115770,72900
15-1071,Network and computer systems administrators,327850,69570
15-1081,Network systems and data communications analysts,230410,73830
15-1099,"Computer specialists, all other",191780,76000
15-2011,Actuaries,18220,95980
15-2021,Mathematicians,2770,94960
15-2031,Operations research analysts,60860,74220
15-2041,Statisticians,20680,74790
15-2091,Mathematical technicians,1100,42100
15-2099,"Mathematical scientists, all other",6600,65630
17-0000,Architecture and engineering occupations,2521630,71430
17-1011,"Architects, except landscape and naval",110990,76750
17-1012,Landscape architects,21130,64000
17-1021,Cartographers and photogrammetrists,11690,57980
17-1022,Surveyors,55780,55980
17-2011,Aerospace engineers,67800,93980
17-2021,Agricultural engineers,2640,72850
17-2031,Biomedical engineers,15220,81120
17-2041,Chemical engineers,30970,88760
17-2051,Civil engineers,261360,78560
17-2061,Computer hardware engineers,73370,100180
17-2071,Electrical engineers,154670,85350
17-2072,"Electronics engineers, except computer",139930,88670
17-2081,Environmental engineers,52590,77970
17-2111,"Health and safety engineers, except mining safety engineers and inspectors",25190,73830
17-2112,Industrial engineers,214580,75740
17-2121,Marine engineers and naval architects,6480,77920
17-2131,Materials engineers,24160,84200
17-2141,Mechanical engineers,233610,78200
17-2151,"Mining and geological engineers, including mining safety engineers",6900,79910
17-2161,Nuclear engineers,16640,99750
17-2171,Petroleum engineers,20880,119140
17-2199,"Engineers, all other",169240,89080
17-3011,Architectural and civil drafters,114910,46390
17-3012,Electrical and electronics drafters,32710,53770
17-3013,Mechanical drafters,77070,48600
17-3019,"Drafters, all other",20720,47290
17-3021,Aerospace engineering and operations technicians,8540,56280
17-3022,Civil engineering technicians,88140,45730
17-3023,Electrical and electronic engineering technicians,162330,53990
17-3024,Electro-mechanical technicians,16290,48110
17-3025,Environmental engineering technicians,20740,44440
17-3026,Industrial engineering technicians,72820,50070
17-3027,Mechanical engineering technicians,45770,50040
17-3029,"Engineering technicians, except drafters, all other",73870,56560
17-3031,Surveying and mapping technicians,71920,37500
19-0000,"Life, physical, and social science occupations",1296840,64280
19-1011,Animal scientists,2760,61640
19-1012,Food scientists and technologists,10510,64610
19-1013,Soil and plant scientists,10790,64110
19-1021,Biochemists and biophysicists,22230,88450
19-1022,Microbiologists,15750,70150
19-1023,Zoologists and wildlife biologists,17780,58820
19-1029,"Biological scientists, all other",28290,68030
19-1031,Conservation scientists,15830,60170
19-1032,Foresters,10160,55040
19-1041,Epidemiologists,4370,64500
19-1042,"Medical scientists, except epidemiologists",99750,81870
19-1099,"Life scientists, all other",12030,69020
19-2011,Astronomers,1280,99730
19-2012,Physicists,14810,106440
19-2021,Atmospheric and space scientists,8860,82080
19-2031,Chemists,83080,71070
19-2032,Materials scientists,9650,81600
19-2041,"Environmental scientists and specialists, including health",80120,65280
19-2042,"Geoscientists, except hydrologists and geographers",31260,89300
19-2043,Hydrologists,7590,73540
19-2099,"Physical scientists, all other",22900,91850
19-3011,Economists,12600,90830
19-3021,Market research analysts,230070,67340
19-3022,Survey researchers,21100,42060
19-3031,"Clinical, counseling, and school psychologists",97880,70190
19-3032,Industrial-organizational psychologists,1460,86460
19-3039,"Psychologists, all other",9870,90460
19-3041,Sociologists,4390,75460
19-3051,Urban and regional planners,37120,62400
19-3091,Anthropologists and archeologists,5230,57300
19-3092,Geographers,1120,66830
19-3093,Historians,3700,57180
19-3094,Political scientists,3530,99320
19-3099,"Social scientists and related workers, all other",28680,71730
19-4011,Agricultural and food science technicians,18930,36470
19-4021,Biological technicians,72200,40900
19-4031,Chemical technicians,65830,43710
19-4041,Geological and petroleum technicians,14570,57080
19-4051,Nuclear technicians,6360,66910
19-4061,Social science research assistants,18120,37920
19-4091,"Environmental science and protection technicians, including health",33370,43180
19-4092,Forensic science technicians,11990,52960
19-4093,Forest and conservation technicians,30850,35320
19-4099,"Life, physical, and social science technicians, all other",58070,42910
21-0000,Community and social services occupations,1861750,41790
21-1011,Substance abuse and behavioral disorder counselors,79180,39670
21-1012,"Educational, vocational, and school counselors",243100,53540
21-1013,Marriage and family therapists,24520,46930
21-1014,Mental health counselors,104650,40270
21-1015,Rehabilitation counselors,112700,34600
21-1019,"Counselors, all other",29980,42240
21-1021,"Child, family, and school social workers",274140,43120
21-1022,Medical and public health social workers,131730,47560
21-1023,Mental health and substance abuse social workers,131010,39630
21-1029,"Social workers, all other",68230,48180
21-1091,Health educators,62120,48590
21-1092,Probation officers and correctional treatment specialists,97130,49520
21-1093,Social and human service assistants,332880,29170
21-1099,"Community and social service specialists, all other",107910,39950
21-2011,Clergy,42040,45440
21-2021,"Directors, religious activities and education",14790,40260
21-2099,"Religious workers, all other",5640,32360
23-0000,Legal occupations,1003270,92270
23-1011,Lawyers,553690,124750
23-1021,"Administrative law judges, adjudicators, and hearing officers",13370,80870
23-1022,"Arbitrators, mediators, and conciliators",9570,59650
23-1023,"Judges, magistrate judges, and magistrates",25470,100450
23-2011,Paralegals and legal assistants,253040,48790
23-2091,Court reporters,17930,51960
23-2092,Law clerks,31500,40580
23-2093,"Title examiners, abstractors, and searchers",59390,42090
23-2099,"Legal support workers, all other",39310,56660
25-0000,"Education, training, and library occupations",8451250,48460
25-1011,"Business teachers, postsecondary",69690,77340
25-1021,"Computer science teachers, postsecondary",32520,74050
25-1022,"Mathematical science teachers, postsecondary",45710,68130
25-1031,"Architecture teachers, postsecondary",6430,75450
25-1032,"Engineering teachers, postsecondary",32070,90070
25-1041,"Agricultural sciences teachers, postsecondary",10000,77770
25-1042,"Biological science teachers, postsecondary",51930,83270
25-1043,"Forestry and conservation science teachers, postsecondary",2450,67400
25-1051,"Atmospheric, earth, marine, and space sciences teachers, postsecondary",9650,81470
25-1052,"Chemistry teachers, postsecondary",19950,76310
25-1053,"Environmental science teachers, postsecondary",4870,74610
25-1054,"Physics teachers, postsecondary",12350,81880
25-1061,"Anthropology and archeology teachers, postsecondary",5500,73410
25-1062,"Area, ethnic, and cultural studies teachers, postsecondary",7570,70560
25-1063,"Economics teachers, postsecondary",12540,88330
25-1064,"Geography teachers, postsecondary",4030,67480
25-1065,"Political science teachers, postsecondary",14340,75960
25-1066,"Psychology teachers, postsecondary",31420,69560
25-1067,"Sociology teachers, postsecondary",16440,68900
25-1069,"Social sciences teachers, postsecondary, all other",5720,74720
25-1071,"Health specialties teachers, postsecondary",125100,102000
25-1072,"Nursing instructors and teachers, postsecondary",46890,62660
25-1081,"Education teachers, postsecondary",55880,60080
25-1082,"Library science teachers, postsecondary",3960,61630
25-1111,"Criminal justice and law enforcement teachers, postsecondary",11630,59830
25-1112,"Law teachers, postsecondary",12490,101170
25-1113,"Social work teachers, postsecondary",7930,64680
25-1121,"Art, drama, and music teachers, postsecondary",76810,65030
25-1122,"Communications teachers, postsecondary",24360,63330
25-1123,"English language and literature teachers, postsecondary",62230,63610
25-1124,"Foreign language and literature teachers, postsecondary",26400,65280
25-1125,"History teachers, postsecondary",21020,68360
25-1126,"Philosophy and religion teachers, postsecondary",18370,65140
25-1191,Graduate teaching assistants,124380,31710
25-1192,"Home economics teachers, postsecondary",4820,70420
25-1193,"Recreation and fitness studies teachers, postsecondary",17410,60700
25-1194,"Vocational education teachers, postsecondary",112940,50870
25-1199,"Postsecondary teachers, all other",242780,71320
25-2011,"Preschool teachers, except special education",392170,26610
25-2012,"Kindergarten teachers, except special education",174530,49770
25-2021,"Elementary school teachers, except special education",1544270,52240
25-2022,"Middle school teachers, except special and vocational education",661820,52570
25-2023,"Vocational education teachers, middle school",15720,50150
25-2031,"Secondary school teachers, except special and vocational education",1090490,54390
25-2032,"Vocational education teachers, secondary school",99800,53700
25-2041,"Special education teachers, preschool, kindergarten, and elementary school",226250,52970
25-2042,"Special education teachers, middle school",100650,53540
25-2043,"Special education teachers, secondary school",147210,55050
25-3011,"Adult literacy, remedial education, and GED teachers and instructors",73050,49830
25-3021,Self-enrichment education teachers,163190,40920
25-3099,"Teachers and instructors, all other",574540,40770
25-4011,Archivists,5330,48220
25-4012,Curators,10820,51540
25-4013,Museum technicians and conservators,10200,40750
25-4021,Librarians,151170,54700
25-4031,Library technicians,113510,30130
25-9011,Audio-visual collections specialists,6160,45540
25-9021,Farm and home management advisors,10760,44630
25-9031,Instructional coordinators,122180,59780
25-9041,Teacher assistants,1266900,23560
25-9099,"Education, training, and library workers, all other",99900,40560
27-0000,"Arts, design, entertainment, sports, and media occupations",1804940,50670
27-1011,Art directors,33670,88510
27-1012,Craft artists,5440,32570
27-1013,"Fine artists, including painters, sculptors, and illustrators",9380,48300
27-1014,Multi-media artists and animators,31500,62380
27-1019,"Artists and related workers, all other",8470,55140
27-1021,Commercial and industrial designers,32940,61580
27-1022,Fashion designers,16920,71400
27-1023,Floral designers,57500,24510
27-1024,Graphic designers,209290,46750
27-1025,Interior designers,53290,51020
27-1026,Merchandise displayers and window trimmers,63320,28560
27-1027,Set and exhibit designers,7940,48660
27-1029,"Designers, all other",11160,50130
27-2011,Actors,44360,null
27-2012,Producers and directors,78060,83030
27-2021,Athletes and sports competitors,13960,79460
27-2022,Coaches and scouts,175720,35580
27-2023,"Umpires, referees, and other sports officials",12970,28330
27-2031,Dancers,11370,null
27-2032,Choreographers,13860,41870
27-2041,Music directors and composers,9120,54840
27-2042,Musicians and singers,47030,null
27-2099,"Entertainers and performers, sports and related workers, all other",36190,null
27-3011,Radio and television announcers,37290,40410
27-3012,Public address system and other announcers,8280,37530
27-3021,Broadcast news analysts,6310,70730
27-3022,Reporters and correspondents,50690,44030
27-3031,Public relations specialists,240610,58960
27-3041,Editors,110010,57180
27-3042,Technical writers,47460,64210
27-3043,Writers and authors,44170,64560
27-3091,Interpreters and translators,36610,43130
27-3099,"Media and communication workers, all other",24470,47770
27-4011,Audio and video equipment technicians,45200,41310
27-4012,Broadcast technicians,33550,38070
27-4013,Radio operators,820,38520
27-4014,Sound engineering technicians,16600,53110
27-4021,Photographers,61670,35640
27-4031,"Camera operators, television, video, and motion picture",19270,47710
27-4032,Film and video editors,18720,62500
27-4099,"Media and communication equipment workers, all other",19770,56120
29-0000,Healthcare practitioners and technical occupations,7076800,67890
29-1011,Chiropractors,27050,81340
29-1021,"Dentists, general",85910,154270
29-1022,Oral and maxillofacial surgeons,4760,190420
29-1023,Orthodontists,5500,194930
29-1024,Prosthodontists,370,169810
29-1029,"Dentists, all other specialists",4770,142070
29-1031,Dietitians and nutritionists,53630,51470
29-1041,Optometrists,25970,105200
29-1051,Pharmacists,266410,104260
29-1061,Anesthesiologists,34230,197570
29-1062,Family and general practitioners,106210,161490
29-1063,"Internists, general",46980,176740
29-1064,Obstetricians and gynecologists,19750,192780
29-1065,"Pediatricians, general",29170,153370
29-1066,Psychiatrists,22140,154050
29-1067,Surgeons,47070,206770
29-1069,"Physicians and surgeons, all other",262850,165000
29-1071,Physician assistants,71950,81610
29-1081,Podiatrists,9670,125760
29-1111,Registered nurses,2542760,65130
29-1121,Audiologists,12480,65500
29-1122,Occupational therapists,94800,67920
29-1123,Physical therapists,167300,74410
29-1124,Radiation therapists,14850,75450
29-1125,Recreational therapists,22510,39930
29-1126,Respiratory therapists,103870,53150
29-1127,Speech-language pathologists,107340,66130
29-1129,"Therapists, all other",12960,54750
29-1131,Veterinarians,53110,89450
29-1199,"Health diagnosing and treating practitioners, all other",34890,78540
29-2011,Medical and clinical laboratory technologists,166510,54050
29-2012,Medical and clinical laboratory technicians,149670,37150
29-2021,Dental hygienists,173090,66950
29-2031,Cardiovascular technologists and technicians,48040,48640
29-2032,Diagnostic medical sonographers,48920,62660
29-2033,Nuclear medicine technologists,21200,67480
29-2034,Radiologic technologists and technicians,208570,53230
29-2041,Emergency medical technicians and paramedics,207610,31980
29-2051,Dietetic technicians,24620,27580
29-2052,Pharmacy technicians,324110,28500
29-2053,Psychiatric technicians,54800,32190
29-2054,Respiratory therapy technicians,16210,43670
29-2055,Surgical technologists,89600,40070
29-2056,Veterinary technologists and technicians,78920,29850
29-2061,Licensed practical and licensed vocational nurses,730500,40110
29-2071,Medical records and health information technicians,168650,32960
29-2081,"Opticians, dispensing",59470,35060
29-2091,Orthotists and prosthetists,5490,66060
29-2099,"Health technologists and technicians, all other",74990,41380
29-9011,Occupational health and safety specialists,53250,63030
29-9012,Occupational health and safety technicians,10540,47410
29-9091,Athletic trainers,15070,41620
29-9099,"Healthcare practitioners and technical workers, all other",55750,50510
31-0000,Healthcare support occupations,3779280,26340
31-1011,Home health aides,892410,21440
31-1012,"Nursing aides, orderlies, and attendants",1422720,24620
31-1013,Psychiatric aides,59050,27260
31-2011,Occupational therapist assistants,25610,48440
31-2012,Occupational therapist aides,7410,29580
31-2021,Physical therapist assistants,61820,46300
31-2022,Physical therapist aides,44410,24770
31-9011,Massage therapists,51250,39850
31-9091,Dental assistants,293090,33170
31-9092,Medical assistants,475950,29060
31-9093,Medical equipment preparers,44340,29290
31-9094,Medical transcriptionists,86200,32960
31-9095,Pharmacy aides,53190,21500
31-9096,Veterinary assistants and laboratory animal caretakers,71950,22790
31-9099,"Healthcare support workers, all other",189890,30650
33-0000,Protective service occupations,3128960,40200
33-1011,First-line supervisors/managers of correctional officers,40840,58380
33-1012,First-line supervisors/managers of police and detectives,92840,76820
33-1021,First-line supervisors/managers of fire fighting and prevention workers,53300,70860
33-1099,"First-line supervisors/managers, protective service workers, all other",49310,47700
33-2011,Fire fighters,298900,45700
33-2021,Fire inspectors and investigators,12920,54840
33-2022,Forest fire inspectors and prevention specialists,1580,36400
33-3011,Bailiffs,19290,39090
33-3012,Correctional officers and jailers,428040,41340
33-3021,Detectives and criminal investigators,104480,63840
33-3031,Fish and game wardens,7720,56030
33-3041,Parking enforcement workers,9530,34020
33-3051,Police and sheriff's patrol officers,633710,52810
33-3052,Transit and railroad police,3830,48540
33-9011,Animal control workers,15480,31990
33-9021,Private detectives and investigators,35820,46480
33-9031,Gaming surveillance officers and gaming investigators,9100,31550
33-9032,Security guards,1046760,25840
33-9091,Crossing guards,68530,24290
33-9092,"Lifeguards, ski patrol, and other recreational protective service workers",111560,19930
33-9099,"Protective service workers, all other",85440,32580
35-0000,Food preparation and serving related occupations,11438550,20220
35-1011,Chefs and head cooks,98040,42410
35-1012,First-line supervisors/managers of food preparation and serving workers,805360,30810
35-2011,"Cooks, fast food",559160,17620
35-2012,"Cooks, institution and cafeteria",370920,23260
35-2013,"Cooks, private household",960,31020
35-2014,"Cooks, restaurant",899620,22750
35-2015,"Cooks, short order",168770,20230
35-2019,"Cooks, all other",17340,24770
35-2021,Food preparation workers,880480,19850
35-3011,Bartenders,503420,20460
35-3021,"Combined food preparation and serving workers, including fast food",2708840,17400
35-3022,"Counter attendants, cafeteria, food concession, and coffee shop",527530,18520
35-3031,Waiters and waitresses,2371750,19580
35-3041,"Food servers, nonrestaurant",188390,21190
35-9011,Dining room and cafeteria attendants and bartender helpers,416410,18140
35-9021,Dishwashers,521150,17750
35-9031,"Hosts and hostesses, restaurant, lounge, and coffee shop",349990,18570
35-9099,"Food preparation and serving related workers, all other",50420,21460
37-0000,Building and grounds cleaning and maintenance occupations,4429870,24370
37-1011,First-line supervisors/managers of housekeeping and janitorial workers,183560,36310
37-1012,"First-line supervisors/managers of landscaping, lawn service, and groundskeeping workers",108940,42990
37-2011,"Janitors and cleaners, except maids and housekeeping cleaners",2145320,23500
37-2012,Maids and housekeeping cleaners,917120,20290
37-2019,"Building cleaning workers, all other",13580,28660
37-2021,Pest control workers,63180,31040
37-3011,Landscaping and groundskeeping workers,921900,24860
37-3012,"Pesticide handlers, sprayers, and applicators, vegetation",25060,31210
37-3013,Tree trimmers and pruners,35420,31450
37-3019,"Grounds maintenance workers, all other",15790,27180
39-0000,Personal care and service occupations,3437520,24120
39-1011,Gaming supervisors,26110,46600
39-1012,Slot key persons,15390,28460
39-1021,First-line supervisors/managers of personal service workers,129070,38230
39-2011,Animal trainers,10030,31190
39-2021,Nonfarm animal caretakers,126740,21550
39-3011,Gaming dealers,91130,19890
39-3012,Gaming and sports book writers and runners,16140,22800
39-3019,"Gaming service workers, all other",13910,25290
39-3021,Motion picture projectionists,10200,22700
39-3031,"Ushers, lobby attendants, and ticket takers",106570,19100
39-3091,Amusement and recreation attendants,258820,18930
39-3092,Costume attendants,5120,32150
39-3093,"Locker room, coatroom, and dressing room attendants",18170,21490
39-3099,"Entertainment attendants and related workers, all other",43930,20920
39-4011,Embalmers,8090,39320
39-4021,Funeral attendants,33060,23270
39-5011,Barbers,10330,26610
39-5012,"Hairdressers, hairstylists, and cosmetologists",355990,26660
39-5091,"Makeup artists, theatrical and performance",1930,39020
39-5092,Manicurists and pedicurists,51590,22040
39-5093,Shampooers,15570,18300
39-5094,Skin care specialists,26300,32040
39-6011,Baggage porters and bellhops,49770,23170
39-6012,Concierges,20380,28120
39-6021,Tour guides and escorts,31760,25150
39-6022,Travel guides,4510,33660
39-6031,Flight attendants,99480,39840
39-6032,"Transportation attendants, except flight attendants and baggage porters",21870,22370
39-9011,Child care workers,581670,20350
39-9021,Personal and home care aides,614190,19690
39-9031,Fitness trainers and aerobics instructors,229030,34310
39-9032,Recreation workers,282680,24570
39-9041,Residential advisors,52240,25320
39-9099,"Personal care and service workers, all other",75780,22630
41-0000,Sales and related occupations,14336430,36080
41-1011,First-line supervisors/managers of retail sales workers,1186270,39910
41-1012,First-line supervisors/managers of non-retail sales workers,275390,79870
41-2011,Cashiers,3545610,18880
41-2012,Gaming change persons and booth cashiers,22280,22890
41-2021,Counter and rental clerks,448480,24430
41-2022,Parts salespersons,226530,30850
41-2031,Retail salespersons,4426280,25050
41-3011,Advertising sales agents,161550,53170
41-3021,Insurance sales agents,327780,60440
41-3031,"Securities, commodities, and financial services sales agents",271900,92050
41-3041,Travel agents,86420,32470
41-3099,"Sales representatives, services, all other",569130,59030
41-4011,"Sales representatives, wholesale and manufacturing, technical and scientific products",415120,79260
41-4012,"Sales representatives, wholesale and manufacturing, except technical and scientific products",1493760,61470
41-9011,Demonstrators and product promoters,83540,27150
41-9012,Models,1660,30160
41-9021,Real estate brokers,51390,77240
41-9022,Real estate sales agents,164080,54410
41-9031,Sales engineers,78030,89770
41-9041,Telemarketers,345220,24770
41-9091,"Door-to-door sales workers, news and street vendors, and related workers",9520,27600
41-9099,"Sales and related workers, all other",146480,41810
43-0000,Office and administrative support occupations,23231750,32220
43-1011,First-line supervisors/managers of office and administrative support workers,1404330,48700
43-2011,"Switchboard operators, including answering service",153860,25250
43-2021,Telephone operators,22820,33800
43-2099,"Communications equipment operators, all other",3500,37000
43-3011,Bill and account collectors,408760,32180
43-3021,Billing and posting clerks and machine operators,512120,32120
43-3031,"Bookkeeping, accounting, and auditing clerks",1855010,33800
43-3041,Gaming cage workers,17060,25970
43-3051,Payroll and timekeeping clerks,203210,35500
43-3061,Procurement clerks,79610,35280
43-3071,Tellers,600380,24250
43-4011,Brokerage clerks,68430,41370
43-4021,Correspondence clerks,13450,31280
43-4031,"Court, municipal, and license clerks",115070,35120
43-4041,"Credit authorizers, checkers, and clerks",65020,31980
43-4051,Customer service representatives,2233270,31790
43-4061,"Eligibility interviewers, government programs",112510,39850
43-4071,File clerks,204760,25290
43-4081,"Hotel, motel, and resort desk clerks",230230,20630
43-4111,"Interviewers, except eligibility and loan",224690,29040
43-4121,"Library assistants, clerical",114740,24290
43-4131,Loan interviewers and clerks,212340,33890
43-4141,New accounts clerks,87300,30990
43-4151,Order clerks,248030,29300
43-4161,"Human resources assistants, except payroll and timekeeping",164340,36810
43-4171,Receptionists and information clerks,1097610,25400
43-4181,Reservation and transportation ticket agents and travel clerks,163880,32060
43-4199,All other information and record clerks,215780,34910
43-5011,Cargo and freight agents,85950,38830
43-5021,Couriers and messengers,96110,25100
43-5031,"Police, fire, and ambulance dispatchers",96360,35340
43-5032,"Dispatchers, except police, fire, and ambulance",193210,36560
43-5041,"Meter readers, utilities",44730,34890
43-5051,Postal service clerks,78250,50150
43-5052,Postal service mail carriers,354570,46970
43-5053,"Postal service mail sorters, processors, and processing machine operators",185770,45490
43-5061,"Production, planning, and expediting clerks",281660,42150
43-5071,"Shipping, receiving, and traffic clerks",760950,29180
43-5081,Stock clerks and order fillers,1873390,23140
43-5111,"Weighers, measurers, checkers, and samplers, recordkeeping",72720,28500
43-6011,Executive secretaries and administrative assistants,1491520,42340
43-6012,Legal secretaries,257810,41640
43-6013,Medical secretaries,454500,30800
43-6014,"Secretaries, except legal, medical, and executive",1872070,29990
43-9011,Computer operators,107450,37070
43-9021,Data entry keyers,272810,27110
43-9022,Word processors and typists,128010,32710
43-9031,Desktop publishers,26210,38740
43-9041,Insurance claims and policy processing clerks,237800,34830
43-9051,"Mail clerks and mail machine operators, except postal service",137350,26420
43-9061,"Office clerks, general",2906600,26830
43-9071,"Office machine operators, except computer",79470,27310
43-9081,Proofreaders and copy markers,15300,32240
43-9111,Statistical assistants,16900,36020
43-9199,"Office and administrative support workers, all other",272190,31510
45-0000,"Farming, fishing, and forestry occupations",438490,23560
45-1011,"First-line supervisors/managers of farming, fishing, and forestry workers",20800,42740
45-1012,Farm labor contractors,1110,36640
45-2011,Agricultural inspectors,14340,41330
45-2021,Animal breeders,2080,33070
45-2041,"Graders and sorters, agricultural products",37500,20490
45-2091,Agricultural equipment operators,22110,24490
45-2092,"Farmworkers and laborers, crop, nursery, and greenhouse",242390,19280
45-2093,"Farmworkers, farm and ranch animals",38110,22920
45-2099,"Agricultural workers, all other",7680,27310
45-3011,Fishers and related fishing workers,1110,28460
45-4011,Forest and conservation workers,8280,26110
45-4021,Fallers,7120,34180
45-4022,Logging equipment operators,27010,32780
45-4023,Log graders and scalers,3610,34330
45-4029,"Logging workers, all other",5180,32900
47-0000,Construction and extraction occupations,6548760,42350
47-1011,First-line supervisors/managers of construction trades and extraction workers,577390,61280
47-2011,Boilermakers,20400,53100
47-2021,Brickmasons and blockmasons,106270,47740
47-2022,Stonemasons,18910,40930
47-2031,Carpenters,899920,42940
47-2041,Carpet installers,34390,41330
47-2042,"Floor layers, except carpet, wood, and hard tiles",14250,39580
47-2043,Floor sanders and finishers,8220,35220
47-2044,Tile and marble setters,51210,41870
47-2051,Cement masons and concrete finishers,201730,39000
47-2053,Terrazzo workers and finishers,5550,39300
47-2061,Construction laborers,1020290,32250
47-2071,"Paving, surfacing, and tamping equipment operators",61230,36490
47-2072,Pile-driver operators,4790,51640
47-2073,Operating engineers and other construction equipment operators,398910,43630
47-2081,Drywall and ceiling tile installers,128740,40940
47-2082,Tapers,31850,45900
47-2111,Electricians,633010,49890
47-2121,Glaziers,51730,38990
47-2131,"Insulation workers, floor, ceiling, and wall",28390,34920
47-2132,"Insulation workers, mechanical",30150,41570
47-2141,"Painters, construction and maintenance",250310,36510
47-2142,Paperhangers,4610,39520
47-2151,Pipelayers,54440,36300
47-2152,"Plumbers, pipefitters, and steamfitters",437540,49200
47-2161,Plasterers and stucco masons,43290,40070
47-2171,Reinforcing iron and rebar workers,28620,44380
47-2181,Roofers,120200,37430
47-2211,Sheet metal workers,163480,44310
47-2221,Structural iron and steel workers,68670,47170
47-3011,"Helpers--brickmasons, blockmasons, stonemasons, and tile and marble setters",53300,29610
47-3012,Helpers--carpenters,81260,26390
47-3013,Helpers--electricians,104050,27450
47-3014,"Helpers--painters, paperhangers, plasterers, and stucco masons",19900,24330
47-3015,"Helpers--pipelayers, plumbers, pipefitters, and steamfitters",79870,27260
47-3016,Helpers--roofers,18730,24660
47-3019,"Helpers, construction trades, all other",27210,27130
47-4011,Construction and building inspectors,96000,52160
47-4021,Elevator installers and repairers,25070,67750
47-4031,Fence erectors,25710,30130
47-4041,Hazardous materials removal workers,42500,40290
47-4051,Highway maintenance workers,136420,35040
47-4061,Rail-track laying and maintenance equipment operators,15020,44870
47-4071,Septic tank servicers and sewer pipe cleaners,24730,35550
47-4091,Segmental pavers,1170,28450
47-4099,"Construction and related workers, all other",55820,35170
47-5011,"Derrick operators, oil and gas",23590,41980
47-5012,"Rotary drill operators, oil and gas",27020,54370
47-5013,"Service unit operators, oil, gas, and mining",36850,41320
47-5021,"Earth drillers, except oil and gas",20220,41360
47-5031,"Explosives workers, ordnance handling experts, and blasters",6060,44490
47-5041,Continuous mining machine operators,10920,45570
47-5042,Mine cutting and channeling machine operators,9190,41510
47-5049,"Mining machine operators, all other",4650,43240
47-5051,"Rock splitters, quarry",4210,30160
47-5061,"Roof bolters, mining",4950,45690
47-5071,"Roustabouts, oil and gas",62540,32660
47-5081,Helpers--extraction workers,25550,34030
47-5099,"Extraction workers, all other",7800,41370
49-0000,"Installation, maintenance, and repair occupations",5374850,41230
49-1011,"First-line supervisors/managers of mechanics, installers, and repairers",443840,59160
49-2011,"Computer, automated teller, and office machine repairers",122400,39420
49-2021,Radio mechanics,5440,42530
49-2022,"Telecommunications equipment installers and repairers, except line installers",195170,52650
49-2091,Avionics technicians,18360,49360
49-2092,"Electric motor, power tool, and related repairers",23400,37110
49-2093,"Electrical and electronics installers and repairers, transportation equipment",15860,44940
49-2094,"Electrical and electronics repairers, commercial and industrial equipment",77270,49260
49-2095,"Electrical and electronics repairers, powerhouse, substation, and relay",23180,60700
49-2096,"Electronic equipment installers and repairers, motor vehicles",19980,29770
49-2097,Electronic home entertainment equipment installers and repairers,38680,33830
49-2098,Security and fire alarm systems installers,62720,38630
49-3011,Aircraft mechanics and service technicians,116310,51650
49-3021,Automotive body and related repairers,147200,39950
49-3022,Automotive glass installers and repairers,18330,33180
49-3023,Automotive service technicians and mechanics,649460,37540
49-3031,Bus and truck mechanics and diesel engine specialists,248620,40710
49-3041,Farm equipment mechanics,30240,32850
49-3042,"Mobile heavy equipment mechanics, except engines",125930,44300
49-3043,Rail car repairers,20780,44920
49-3051,Motorboat mechanics,19640,36080
49-3052,Motorcycle mechanics,16850,33490
49-3053,Outdoor power equipment and other small engine mechanics,26440,29970
49-3091,Bicycle repairers,9690,24140
49-3092,Recreational vehicle service technicians,13400,33380
49-3093,Tire repairers and changers,98520,23830
49-9011,Mechanical door repairers,17530,35360
49-9012,"Control and valve installers and repairers, except mechanical door",43900,47260
49-9021,"Heating, air conditioning, and refrigeration mechanics and installers",261610,42240
49-9031,Home appliance repairers,37300,35690
49-9041,Industrial machinery mechanics,280620,45280
49-9042,"Maintenance and repair workers, general",1305170,35630
49-9043,"Maintenance workers, machinery",73650,38610
49-9044,Millwrights,46250,50030
49-9045,"Refractory materials repairers, except brickmasons",2450,41750
49-9051,Electrical power-line installers and repairers,111580,54300
49-9052,Telecommunications line installers and repairers,168050,47330
49-9061,Camera and photographic equipment repairers,3820,37990
49-9062,Medical equipment repairers,34260,44030
49-9063,Musical instrument repairers and tuners,5310,35950
49-9064,Watch repairers,2770,37600
49-9069,"Precision instrument and equipment repairers, all other",12990,50400
49-9091,"Coin, vending, and amusement machine servicers and repairers",41280,30970
49-9092,Commercial divers,2370,54940
49-9093,"Fabric menders, except garment",960,27920
49-9094,Locksmiths and safe repairers,18500,35900
49-9095,Manufactured building and mobile home installers,8290,29460
49-9096,Riggers,13490,42620
49-9097,Signal and track switch repairers,6570,49060
49-9098,"Helpers--installation, maintenance, and repair workers",149350,25670
49-9099,"Installation, maintenance, and repair workers, all other",139100,37220
51-0000,Production occupations,9919120,32320
51-1011,First-line supervisors/managers of production and operating workers,658500,53500
51-2011,"Aircraft structure, surfaces, rigging, and systems assemblers",43330,43600
51-2021,"Coil winders, tapers, and finishers",22160,28720
51-2022,Electrical and electronic equipment assemblers,215230,29410
51-2023,Electromechanical equipment assemblers,62310,30520
51-2031,Engine and other machine assemblers,39270,34900
51-2041,Structural metal fabricators and fitters,111620,33860
51-2091,Fiberglass laminators and fabricators,30890,28680
51-2092,Team assemblers,1131060,27630
51-2093,"Timing device assemblers, adjusters, and calibrators",2700,30710
51-2099,"Assemblers and fabricators, all other",318060,32840
51-3011,Bakers,141130,25020
51-3021,Butchers and meat cutters,128210,29700
51-3022,"Meat, poultry, and fish cutters and trimmers",166150,22400
51-3023,Slaughterers and meat packers,97000,23270
51-3091,"Food and tobacco roasting, baking, and drying machine operators and tenders",17870,28610
51-3092,Food batchmakers,99170,26290
51-3093,Food cooking machine operators and tenders,39300,24110
51-4011,"Computer-controlled machine tool operators, metal and plastic",143030,34520
51-4012,Numerical tool and process control programmers,16990,46360
51-4021,"Extruding and drawing machine setters, operators, and tenders, metal and plastic",92160,31030
51-4022,"Forging machine setters, operators, and tenders, metal and plastic",28800,32370
51-4023,"Rolling machine setters, operators, and tenders, metal and plastic",34970,35410
51-4031,"Cutting, punching, and press machine setters, operators, and tenders, metal and plastic",242970,29420
51-4032,"Drilling and boring machine tool setters, operators, and tenders, metal and plastic",33550,32520
51-4033,"Grinding, lapping, polishing, and buffing machine tool setters, operators, and tenders, metal and plastic",91990,31230
51-4034,"Lathe and turning machine tool setters, operators, and tenders, metal and plastic",56500,34070
51-4035,"Milling and planing machine setters, operators, and tenders, metal and plastic",26220,34450
51-4041,Machinists,419070,37490
51-4051,Metal-refining furnace operators and tenders,19280,37310
51-4052,"Pourers and casters, metal",15320,33510
51-4061,"Model makers, metal and plastic",8990,44700
51-4062,"Patternmakers, metal and plastic",6220,39490
51-4071,Foundry mold and coremakers,15240,30390
51-4072,"Molding, coremaking, and casting machine setters, operators, and tenders, metal and plastic",145760,29270
51-4081,"Multiple machine tool setters, operators, and tenders, metal and plastic",87800,32780
51-4111,Tool and die makers,85610,48180
51-4121,"Welders, cutters, solderers, and brazers",392520,35370
51-4122,"Welding, soldering, and brazing machine setters, operators, and tenders",51840,33700
51-4191,"Heat treating equipment setters, operators, and tenders, metal and plastic",23630,33140
51-4192,"Lay-out workers, metal and plastic",8340,37170
51-4193,"Plating and coating machine setters, operators, and tenders, metal and plastic",40300,30090
51-4194,"Tool grinders, filers, and sharpeners",16410,33880
51-4199,"Metal workers and plastic workers, all other",43690,35570
51-5011,Bindery workers,60560,29100
51-5012,Bookbinders,6150,33970
51-5021,Job printers,42640,35330
51-5022,Prepress technicians and workers,61170,36440
51-5023,Printing machine operators,193510,34150
51-6011,Laundry and dry-cleaning workers,221230,20230
51-6021,"Pressers, textile, garment, and related materials",67500,19860
51-6031,Sewing machine operators,190440,21690
51-6041,Shoe and leather workers and repairers,8170,24130
51-6042,Shoe machine operators and tenders,4910,25420
51-6051,"Sewers, hand",7050,23170
51-6052,"Tailors, dressmakers, and custom sewers",31700,26920
51-6061,Textile bleaching and dyeing machine operators and tenders,16180,24480
51-6062,"Textile cutting machine setters, operators, and tenders",20170,23810
51-6063,"Textile knitting and weaving machine setters, operators, and tenders",30250,25650
51-6064,"Textile winding, twisting, and drawing out machine setters, operators, and tenders",36540,24600
51-6091,"Extruding and forming machine setters, operators, and tenders, synthetic and glass fibers",14440,31800
51-6092,Fabric and apparel patternmakers,7500,42190
51-6093,Upholsterers,39090,30560
51-6099,"Textile, apparel, and furnishings workers, all other",17430,27450
51-7011,Cabinetmakers and bench carpenters,120960,30620
51-7021,Furniture finishers,21630,28410
51-7031,"Model makers, wood",1740,35690
51-7032,"Patternmakers, wood",1930,38720
51-7041,"Sawing machine setters, operators, and tenders, wood",51830,27070
51-7042,"Woodworking machine setters, operators, and tenders, except sawing",88510,25880
51-7099,"Woodworkers, all other",11260,26570
51-8011,Nuclear power reactor operators,4970,73510
51-8012,Power distributors and dispatchers,9820,66070
51-8013,Power plant operators,34700,58340
51-8021,Stationary engineers and boiler operators,39000,50660
51-8031,Water and liquid waste treatment plant and system operators,110300,39950
51-8091,Chemical plant and system operators,44600,52150
51-8092,Gas plant operators,14500,55350
51-8093,"Petroleum pump system operators, refinery operators, and gaugers",45710,54950
51-8099,"Plant and system operators, all other",12370,48590
51-9011,Chemical equipment operators and tenders,52890,45580
51-9012,"Separating, filtering, clarifying, precipitating, and still machine setters, operators, and tenders",41200,38010
51-9021,"Crushing, grinding, and polishing machine setters, operators, and tenders",41270,31360
51-9022,"Grinding and polishing workers, hand",40290,27100
51-9023,"Mixing and blending machine setters, operators, and tenders",140120,32650
51-9031,"Cutters and trimmers, hand",24700,25540
51-9032,"Cutting and slicing machine setters, operators, and tenders",76500,30020
51-9041,"Extruding, forming, pressing, and compacting machine setters, operators, and tenders",85130,30430
51-9051,"Furnace, kiln, oven, drier, and kettle operators and tenders",22950,32950
51-9061,"Inspectors, testers, sorters, samplers, and weighers",467010,33890
51-9071,Jewelers and precious stone and metal workers,24780,35360
51-9081,Dental laboratory technicians,42640,36990
51-9082,Medical appliance technicians,12930,38640
51-9083,Ophthalmic laboratory technicians,32930,29130
51-9111,Packaging and filling machine operators and tenders,357480,26550
51-9121,"Coating, painting, and spraying machine setters, operators, and tenders",103310,29680
51-9122,"Painters, transportation equipment",50310,40300
51-9123,"Painting, coating, and decorating workers",31200,26090
51-9131,Photographic process workers,21040,29410
51-9132,Photographic processing machine operators,49550,22740
51-9141,Semiconductor processors,32230,34170
51-9191,Cementing and gluing machine operators and tenders,19640,29030
51-9192,"Cleaning, washing, and metal pickling equipment operators and tenders",18870,26140
51-9193,Cooling and freezing equipment operators and tenders,9940,27260
51-9194,Etchers and engravers,10760,29400
51-9195,"Molders, shapers, and casters, except metal and plastic",41900,29090
51-9196,"Paper goods machine setters, operators, and tenders",104170,34010
51-9197,Tire builders,21740,38080
51-9198,Helpers--production workers,499870,23320
51-9199,"Production workers, all other",280160,30310
53-0000,Transportation and material moving occupations,9508750,31450
53-1011,Aircraft cargo handling supervisors,4950,46660
53-1021,"First-line supervisors/managers of helpers, laborers, and material movers, hand",186230,44380
53-1031,First-line supervisors/managers of transportation and material-moving machine and vehicle operators,218480,53960
53-2011,"Airline pilots, copilots, and flight engineers",77090,119750
53-2012,Commercial pilots,31250,75500
53-2021,Air traffic controllers,24260,108090
53-2022,Airfield operations specialists,8050,42550
53-3011,"Ambulance drivers and attendants, except emergency medical technicians",21790,23400
53-3021,"Bus drivers, transit and intercity",184160,35700
53-3022,"Bus drivers, school",460100,27060
53-3031,Driver/sales workers,372720,26050
53-3032,"Truck drivers, heavy and tractor-trailer",1672580,38720
53-3033,"Truck drivers, light or delivery services",908960,30260
53-3041,Taxi drivers and chauffeurs,170520,23540
53-3099,"Motor vehicle operators, all other",78610,29650
53-4011,Locomotive engineers,42760,53470
53-4012,Locomotive firers,970,52950
53-4013,"Rail yard engineers, dinkey operators, and hostlers",5480,34850
53-4021,"Railroad brake, signal, and switch operators",24610,49400
53-4031,Railroad conductors and yardmasters,39580,54120
53-4041,Subway and streetcar operators,7430,49330
53-4099,"Rail transportation workers, all other",4660,44200
53-5011,Sailors and marine oilers,32420,35880
53-5021,"Captains, mates, and pilots of water vessels",30600,67730
53-5022,Motorboat operators,3380,36480
53-5031,Ship engineers,11190,66140
53-6011,Bridge and lock tenders,4490,39930
53-6021,Parking lot attendants,136470,20120
53-6031,Service station attendants,84480,20340
53-6041,Traffic technicians,7030,41810
53-6051,Transportation inspectors,24940,59200
53-6099,"Transportation workers, all other",43330,35590
53-7011,Conveyor operators and tenders,41920,30090
53-7021,Crane and tower operators,44490,45430
53-7031,Dredge operators,1910,39040
53-7032,Excavating and loading machine and dragline operators,65160,38180
53-7033,"Loading machine operators, underground mining",3670,44230
53-7041,Hoist and winch operators,2810,41340
53-7051,Industrial truck and tractor operators,620450,30750
53-7061,Cleaners of vehicles and equipment,330850,21700
53-7062,"Laborers and freight, stock, and material movers, hand",2335510,24690
53-7063,Machine feeders and offbearers,144820,26880
53-7064,"Packers and packagers, hand",777630,21100
53-7071,Gas compressor and gas pumping station operators,4050,44410
53-7072,"Pump operators, except wellhead pumpers",9280,41020
53-7073,Wellhead pumpers,17050,39430
53-7081,Refuse and recyclable material collectors,129080,32790
53-7111,Shuttle car operators,3050,42700
53-7121,"Tank car, truck, and ship loaders",12330,39510
53-7199,"Material moving workers, all other",41140,34700

harristd 5 comments Joined 07/12
01 Apr 2015

gschasie,  make sure your sandbox's hosts file has the IP and "sandbox" entry.

Example (fill in actual IP):
0.0.0.0 sandbox  
 
It could be hanging because Studio is telling the sandbox that every service is located at host "sandbox", but that VM might not know itself is the "sandbox"
 
Ryan

gschasie 9 comments Joined 01/10
02 Apr 2015

Hi harristd,
 
i checked the ip & hostname. Reinstalled/reconfigured Terdata Loom with HdP2.1 and follwed the instructions.
Same Result. SqlStudio is running  on PC, Hadoop in VirtualBox, TD in VMWare Player. Ping succeeds ij every direction with sandbox.hortonworks.com also used in configureOozie script.
I can see all tables in Hadoop / TD fro sqlstudio. Seems that the oozie job for cpy hadoop to TD hangs in Running.
thanks.
 

harristd 5 comments Joined 07/12
02 Apr 2015

In the sandbox virtual machine, go to /etc/hosts and add the following line:
 
127.0.0.1     sandbox     sandbox.hortonworks.com
 
That way the virtual machine will know itself is the sandbox.

gschasie 9 comments Joined 01/10
03 Apr 2015

No that didn't solve it.





i think its from oozie coinfiguration or maybe something is missing. 


Here the oozie.log where you see that  the operation is cancelled via sqlstudio, but after that the data is in the table. Seems that somnething between retrieve from hadoop and insert into teradata does not work .


2015-04-03 07:31:15,181&nbsp; WARN JPAService:542 - USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] JPAExecutor [WorkflowActionGetJPAExecutor] ended with an active transaction, rolling back
2015-04-03 07:31:15,203&nbsp; INFO ActionStartXCommand:539 - USER[guest] GROUP[-] TOKEN[] APP[TeradataExportToolByFields-wf] JOB[0000013-150403042259924-oozie-oozi-W] ACTION[0000013-150403042259924-oozie-oozi-W@:start:] Start action [0000013-150403042259924-oozie-oozi-W@:start:] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2015-04-03 07:31:15,203&nbsp; WARN ActionStartXCommand:542 - USER[guest] GROUP[-] TOKEN[] APP[TeradataExportToolByFields-wf] JOB[0000013-150403042259924-oozie-oozi-W] ACTION[0000013-150403042259924-oozie-oozi-W@:start:] [***0000013-150403042259924-oozie-oozi-W@:start:***]Action status=DONE
2015-04-03 07:31:15,203&nbsp; WARN ActionStartXCommand:542 - USER[guest] GROUP[-] TOKEN[] APP[TeradataExportToolByFields-wf] JOB[0000013-150403042259924-oozie-oozi-W] ACTION[0000013-150403042259924-oozie-oozi-W@:start:] [***0000013-150403042259924-oozie-oozi-W@:start:***]Action updated in DB!
2015-04-03 07:31:15,240&nbsp; INFO ActionEndXCommand:539 - USER[guest] GROUP[-] TOKEN[] APP[TeradataExportToolByFields-wf] JOB[0000013-150403042259924-oozie-oozi-W] ACTION[0000013-150403042259924-oozie-oozi-W@:start:] end executor for wf action 0000013-150403042259924-oozie-oozi-W with wf job 0000013-150403042259924-oozie-oozi-W
2015-04-03 07:31:15,267&nbsp; WARN JPAService:542 - USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] JPAExecutor [WorkflowActionGetJPAExecutor] ended with an active transaction, rolling back
2015-04-03 07:31:15,306&nbsp; INFO ActionStartXCommand:539 - USER[guest] GROUP[-] TOKEN[] APP[TeradataExportToolByFields-wf] JOB[0000013-150403042259924-oozie-oozi-W] ACTION[0000013-150403042259924-oozie-oozi-W@java-node] Start action [0000013-150403042259924-oozie-oozi-W@java-node] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2015-04-03 07:31:15,862&nbsp; WARN JavaActionExecutor:542 - USER[guest] GROUP[-] TOKEN[] APP[TeradataExportToolByFields-wf] JOB[0000013-150403042259924-oozie-oozi-W] ACTION[0000013-150403042259924-oozie-oozi-W@java-node] credentials is null for the action
2015-04-03 07:31:16,468&nbsp; WARN JobSubmitter:150 - Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
2015-04-03 07:31:16,474&nbsp; WARN JobSubmitter:259 - No job jar file set.&nbsp; User classes may not be found. See Job or Job#setJar(String).
2015-04-03 07:31:16,839&nbsp; INFO JavaActionExecutor:539 - USER[guest] GROUP[-] TOKEN[] APP[TeradataExportToolByFields-wf] JOB[0000013-150403042259924-oozie-oozi-W] ACTION[0000013-150403042259924-oozie-oozi-W@java-node] Trying to get job [job_1428060213125_0011], attempt [1]
2015-04-03 07:31:16,858&nbsp; INFO JavaActionExecutor:539 - USER[guest] GROUP[-] TOKEN[] APP[TeradataExportToolByFields-wf] JOB[0000013-150403042259924-oozie-oozi-W] ACTION[0000013-150403042259924-oozie-oozi-W@java-node] checking action, external ID [job_1428060213125_0011] status [RUNNING]
2015-04-03 07:31:16,860&nbsp; WARN ActionStartXCommand:542 - USER[guest] GROUP[-] TOKEN[] APP[TeradataExportToolByFields-wf] JOB[0000013-150403042259924-oozie-oozi-W] ACTION[0000013-150403042259924-oozie-oozi-W@java-node] [***0000013-150403042259924-oozie-oozi-W@java-node***]Action status=RUNNING
2015-04-03 07:31:16,860&nbsp; WARN ActionStartXCommand:542 - USER[guest] GROUP[-] TOKEN[] APP[TeradataExportToolByFields-wf] JOB[0000013-150403042259924-oozie-oozi-W] ACTION[0000013-150403042259924-oozie-oozi-W@java-node] [***0000013-150403042259924-oozie-oozi-W@java-node***]Action updated in DB!
2015-04-03 07:31:20,321&nbsp; INFO StatusTransitService$StatusTransitRunnable:539 - USER[-] GROUP[-] Acquired lock for [org.apache.oozie.service.StatusTransitService]
2015-04-03 07:31:20,321&nbsp; INFO StatusTransitService$StatusTransitRunnable:539 - USER[-] GROUP[-] Running coordinator status service from last instance time =&nbsp; 2015-04-03T14:30Z
2015-04-03 07:31:20,323&nbsp; INFO StatusTransitService$StatusTransitRunnable:539 - USER[-] GROUP[-] Running bundle status service from last instance time =&nbsp; 2015-04-03T14:30Z
2015-04-03 07:31:20,324&nbsp; INFO StatusTransitService$StatusTransitRunnable:539 - USER[-] GROUP[-] Released lock for [org.apache.oozie.service.StatusTransitService]
2015-04-03 07:31:21,757&nbsp; INFO PauseTransitService:539 - USER[-] GROUP[-] Acquired lock for [org.apache.oozie.service.PauseTransitService]
2015-04-03 07:31:21,764&nbsp; INFO PauseTransitService:539 - USER[-] GROUP[-] Released lock for [org.apache.oozie.service.PauseTransitService]
2015-04-03 07:31:24,482&nbsp; INFO RecoveryService$RecoveryRunnable:539 - USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] last updated heart beat is 1428071424473
2015-04-03 07:32:18,439&nbsp; INFO KillXCommand:539 - USER[guest] GROUP[-] TOKEN[] APP[TeradataExportToolByFields-wf] JOB[0000013-150403042259924-oozie-oozi-W] ACTION[-] STARTED WorkflowKillXCommand for jobId=0000013-150403042259924-oozie-oozi-W
2015-04-03 07:32:18,487&nbsp; WARN CoordActionUpdateXCommand:542 - USER[guest] GROUP[-] TOKEN[] APP[TeradataExportToolByFields-wf] JOB[0000013-150403042259924-oozie-oozi-W] ACTION[-] E1100: Command precondition does not hold before execution, [, coord action is null], Error Code: E1100
2015-04-03 07:32:18,487&nbsp; INFO KillXCommand:539 - USER[guest] GROUP[-] TOKEN[] APP[TeradataExportToolByFields-wf] JOB[0000013-150403042259924-oozie-oozi-W] ACTION[-] ENDED WorkflowKillXCommand for jobId=0000013-150403042259924-oozie-oozi-W
2015-04-03 07:32:18,488&nbsp; INFO DagEngine:539 - USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] User guest killed the WF job 0000013-150403042259924-oozie-oozi-W
2015-04-03 07:32:20,017&nbsp; INFO CallbackServlet:539 - USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000013-150403042259924-oozie-oozi-W] ACTION[0000013-150403042259924-oozie-oozi-W@java-node] callback for action [0000013-150403042259924-oozie-oozi-W@java-node]
2015-04-03 07:32:20,027 ERROR CompletedActionXCommand:536 - USER[-] GROUP[-] TOKEN[] APP[-] JOB[0000013-150403042259924-oozie-oozi-W] ACTION[0000013-150403042259924-oozie-oozi-W@java-node] XException, 
org.apache.oozie.command.CommandException: E0800: Action it is not running its in [KILLED] state, action [0000013-150403042259924-oozie-oozi-W@java-node]
&nbsp;at org.apache.oozie.command.wf.CompletedActionXCommand.eagerVerifyPrecondition(CompletedActionXCommand.java:85)
&nbsp;at org.apache.oozie.command.XCommand.call(XCommand.java:251)
&nbsp;at org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:175)
&nbsp;at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
&nbsp;at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
&nbsp;at java.lang.Thread.run(Thread.java:744)
2015-04-03 07:32:20,027&nbsp; WARN CallableQueueService$CallableWrapper:542 - USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] exception callable [callback], E0800: Action it is not running its in [KILLED] state, action [0000013-150403042259924-oozie-oozi-W@java-node]
org.apache.oozie.command.CommandException: E0800: Action it is not running its in [KILLED] state, action [0000013-150403042259924-oozie-oozi-W@java-node]
&nbsp;at org.apache.oozie.command.wf.CompletedActionXCommand.eagerVerifyPrecondition(CompletedActionXCommand.java:85)
&nbsp;at org.apache.oozie.command.XCommand.call(XCommand.java:251)
&nbsp;at org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:175)
&nbsp;at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
&nbsp;at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
&nbsp;at java.lang.Thread.run(Thread.java:744)
2015-04-03 07:32:20,324&nbsp; INFO StatusTransitService$StatusTransitRunnable:539 - USER[-] GROUP[-] Acquired lock for [org.apache.oozie.service.StatusTransitService]
2015-04-03 07:32:20,325&nbsp; INFO StatusTransitService$StatusTransitRunnable:539 - USER[-] GROUP[-] Running coordinator status service from last instance time =&nbsp; 2015-04-03T14:31Z
2015-04-03 07:32:20,326&nbsp; INFO StatusTransitService$StatusTransitRunnable:539 - USER[-] GROUP[-] Running bundle status service from last instance time =&nbsp; 2015-04-03T14:31Z
2015-04-03 07:32:20,327&nbsp; INFO StatusTransitService$StatusTransitRunnable:539 - USER[-] GROUP[-] Released lock for [org.apache.oozie.service.StatusTransitService]
2015-04-03 07:32:21,767&nbsp; INFO PauseTransitService:539 - USER[-] GROUP[-] Acquired lock for [org.apache.oozie.service.PauseTransitService]
2015-04-03 07:32:21,772&nbsp; INFO PauseTransitService:539 - USER[-] GROUP[-] Released lock for [org.apache.oozie.service.PauseTransitService]
2015-04-03 07:32:24,491&nbsp; INFO RecoveryService$RecoveryRunnable:539 - USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] last updated heart beat is 1428071484482
2015-04-03 07:33:19,378&nbsp; INFO CoordMaterializeTriggerService$CoordMaterializeTriggerRunnable:539 - USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] last updated heart beat is 1428071544492
2015-04-03 07:33:19,379&nbsp; INFO CoordMaterializeTriggerService$CoordMaterializeTriggerRunnable:539 - USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] CoordMaterializeTriggerService - Curr Date= Fri Apr 03 07:38:19 PDT 2015, Num jobs to materialize = 0
2015-04-03 07:33:20,327&nbsp; INFO StatusTransitService$StatusTransitRunnable:539 - USER[-] GROUP[-] Acquired lock for [org.apache.oozie.service.StatusTransitService]
2015-04-03 07:33:20,328&nbsp; INFO StatusTransitService$StatusTransitRunnable:539 - USER[-] GROUP[-] Running coordinator status service from last instance time =&nbsp; 2015-04-03T14:32Z
2015-04-03 07:33:20,329&nbsp; INFO StatusTransitService$StatusTransitRunnable:539 - USER[-] GROUP[-] Running bundle status service from last instance time =&nbsp; 2015-04-03T14:32Z
2015-04-03 07:33:20,330&nbsp; INFO StatusTransitService$StatusTransitRunnable:539 - USER[-] GROUP[-] Released lock for [org.apache.oozie.service.StatusTransitService]
2015-04-03 07:33:21,773&nbsp; INFO PauseTransitService:539 - USER[-] GROUP[-] Acquired lock for [org.apache.oozie.service.PauseTransitService]
2015-04-03 07:33:21,778&nbsp; INFO PauseTransitService:539 - USER[-] GROUP[-] Released lock for [org.apache.oozie.service.PauseTransitService]
2015-04-03 07:33:24,500&nbsp; INFO RecoveryService$RecoveryRunnable:539 - USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] last updated heart beat is 1428071544492
2015-04-03 07:34:20,331&nbsp; INFO StatusTransitService$StatusTransitRunnable:539 - USER[-] GROUP[-] Acquired lock for [org.apache.oozie.service.StatusTransitService]
2015-04-03 07:34:20,331&nbsp; INFO StatusTransitService$StatusTransitRunnable:539 - USER[-] GROUP[-] Running coordinator status service from last instance time =&nbsp; 2015-04-03T14:33Z
2015-04-03 07:34:20,332&nbsp; INFO StatusTransitService$StatusTransitRunnable:539 - USER[-] GROUP[-] Running bundle status service from last instance time =&nbsp; 2015-04-03T14:33Z
2015-04-03 07:34:20,333&nbsp; INFO StatusTransitService$StatusTransitRunnable:539 - USER[-] GROUP[-] Released lock for [org.apache.oozie.service.StatusTransitService]
2015-04-03 07:34:21,779&nbsp; INFO PauseTransitService:539 - USER[-] GROUP[-] Acquired lock for [org.apache.oozie.service.PauseTransitService]
2015-04-03 07:34:21,784&nbsp; INFO PauseTransitService:539 - USER[-] GROUP[-] Released lock for [org.apache.oozie.service.PauseTransitService]
2015-04-03 07:34:24,509&nbsp; INFO RecoveryService$RecoveryRunnable:539 - USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] last updated heart beat is 1428071604501
2015-04-03 07:35:20,334&nbsp; INFO StatusTransitService$StatusTransitRunnable:539 - USER[-] GROUP[-] Acquired lock for [org.apache.oozie.service.StatusTransitService]
2015-04-03 07:35:20,335&nbsp; INFO StatusTransitService$StatusTransitRunnable:539 - USER[-] GROUP[-] Running coordinator status service from last instance time =&nbsp; 2015-04-03T14:34Z
2015-04-03 07:35:20,336&nbsp; INFO StatusTransitService$StatusTransitRunnable:539 - USER[-] GROUP[-] Running bundle status service from last instance time =&nbsp; 2015-04-03T14:34Z
2015-04-03 07:35:20,337&nbsp; INFO StatusTransitService$StatusTransitRunnable:539 - USER[-] GROUP[-] Released lock for [org.apache.oozie.service.StatusTransitService]
2015-04-03 07:35:21,785&nbsp; INFO PauseTransitService:539 - USER[-] GROUP[-] Acquired lock for [org.apache.oozie.service.PauseTransitService]
2015-04-03 07:35:21,790&nbsp; INFO PauseTransitService:539 - USER[-] GROUP[-] Released lock for [org.apache.oozie.service.PauseTransitService]
2015-04-03 07:35:24,519&nbsp; INFO RecoveryService$RecoveryRunnable:539 - USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] last updated heart beat is 1428071664509
2015-04-03 07:36:20,338&nbsp; INFO StatusTransitService$StatusTransitRunnable:539 - USER[-] GROUP[-] Acquired lock for [org.apache.oozie.service.StatusTransitService]
2015-04-03 07:36:20,338&nbsp; INFO StatusTransitService$StatusTransitRunnable:539 - USER[-] GROUP[-] Running coordinator status service from last instance time =&nbsp; 2015-04-03T14:35Z
2015-04-03 07:36:20,340&nbsp; INFO StatusTransitService$StatusTransitRunnable:539 - USER[-] GROUP[-] Running bundle status service from last instance time =&nbsp; 2015-04-03T14:35Z
2015-04-03 07:36:20,341&nbsp; INFO StatusTransitService$StatusTransitRunnable:539 - USER[-] GROUP[-] Released lock for [org.apache.oozie.service.StatusTransitService]
2015-04-03 07:36:21,791&nbsp; INFO PauseTransitService:539 - USER[-] GROUP[-] Acquired lock for [org.apache.oozie.service.PauseTransitService]
2015-04-03 07:36:21,795&nbsp; INFO PauseTransitService:539 - USER[-] GROUP[-] Released lock for [org.apache.oozie.service.PauseTransitService]
2015-04-03 07:36:24,543&nbsp; INFO RecoveryService$RecoveryRunnable:539 - USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] last updated heart beat is 1428071724527
2015-04-03 07:37:20,342&nbsp; INFO StatusTransitService$StatusTransitRunnable:539 - USER[-] GROUP[-] Acquired lock for [org.apache.oozie.service.StatusTransitService]
2015-04-03 07:37:20,342&nbsp; INFO StatusTransitService$StatusTransitRunnable:539 - USER[-] GROUP[-] Running coordinator status service from last instance time =&nbsp; 2015-04-03T14:36Z
2015-04-03 07:37:20,343&nbsp; INFO StatusTransitService$StatusTransitRunnable:539 - USER[-] GROUP[-] Running bundle status service from last instance time =&nbsp; 2015-04-03T14:36Z
2015-04-03 07:37:20,344&nbsp; INFO StatusTransitService$StatusTransitRunnable:539 - USER[-] GROUP[-] Released lock for [org.apache.oozie.service.StatusTransitService]
2015-04-03 07:37:21,796&nbsp; INFO PauseTransitService:539 - USER[-] GROUP[-] Acquired lock for [org.apache.oozie.service.PauseTransitService]
2015-04-03 07:37:21,801&nbsp; INFO PauseTransitService:539 - USER[-] GROUP[-] Released lock for [org.apache.oozie.service.PauseTransitService]
2015-04-03 07:37:24,551&nbsp; INFO RecoveryService$RecoveryRunnable:539 - USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] last updated heart beat is 1428071784544
2015-04-03 07:38:19,382&nbsp; INFO CoordMaterializeTriggerService$CoordMaterializeTriggerRunnable:539 - USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] last updated heart beat is 1428071844552
2015-04-03 07:38:19,382&nbsp; INFO CoordMaterializeTriggerService$CoordMaterializeTriggerRunnable:539 - USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] CoordMaterializeTriggerService - Curr Date= Fri Apr 03 07:43:19 PDT 2015, Num jobs to materialize = 0
2015-04-03 07:38:20,345&nbsp; INFO StatusTransitService$StatusTransitRunnable:539 - USER[-] GROUP[-] Acquired lock for [org.apache.oozie.service.StatusTransitService]
2015-04-03 07:38:20,345&nbsp; INFO StatusTransitService$StatusTransitRunnable:539 - USER[-] GROUP[-] Running coordinator status service from last instance time =&nbsp; 2015-04-03T14:37Z
2015-04-03 07:38:20,347&nbsp; INFO StatusTransitService$StatusTransitRunnable:539 - USER[-] GROUP[-] Running bundle status service from last instance time =&nbsp; 2015-04-03T14:37Z
2015-04-03 07:38:20,348&nbsp; INFO StatusTransitService$StatusTransitRunnable:539 - USER[-] GROUP[-] Released lock for [org.apache.oozie.service.StatusTransitService]
2015-04-03 07:38:21,802&nbsp; INFO PauseTransitService:539 - USER[-] GROUP[-] Acquired lock for [org.apache.oozie.service.PauseTransitService]
2015-04-03 07:38:21,806&nbsp; INFO PauseTransitService:539 - USER[-] GROUP[-] Released lock for [org.apache.oozie.service.PauseTransitService]
2015-04-03 07:38:24,560&nbsp; INFO RecoveryService$RecoveryRunnable:539 - USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] last updated heart beat is 1428071844552

 
 
 
 

gschasie 9 comments Joined 01/10
09 Apr 2015

Heureka i finally found the solution
 
http://stackoverflow.com/questions/27653937/error-on-running-multiple-worlflow-in-oozie-4-1-0/27855432#27855432
 
Cheers
Guido

fgrimmer 660 comments Joined 04/09
09 Apr 2015

Guido, Glad to hear. Thanks for passing on the link for the solution!

Tuen 24 comments Joined 07/05
14 May 2015

Francine,
 
We have a new production Hortonworks Hadoop cluster running 2.2.4 before i start testing on it, i was doing some testing with TD Studio 15.10 against the Hadoop 2.2 sandbox and trying to connect just using the Hortonworks connection profile using JDBC, i can get connected but it will not display any of the tables under the default database.
I'm using the hue userid against the sandbox.hortonworks.com for the sandbox.  I wanted to make sure it was working on the sandbox before i tried against the new cluster.   I figure that it should work since TDCH 1.3.4 is out for 2.2, i have installed that into my 2.2 sandbox, and i can connect but i don't see any objects.   Is there something else I need to configure or setup?   I was  hoping to at least be able to write some simple sql against 2.2 using Studio so we could provide some of our developers and users a sql interface to Hadoop till querygrid is approved for 2.2/2.3.   I'm not seeing any sort of errors, just nothing getting displayed back in the explorer window when connected to the Sandbox.

fgrimmer 660 comments Joined 04/09
14 May 2015

Bill, When you say "but it will not display any of the tables under the default database", are you talking about displaying in the Data Source Explorer or are you trying to run a 'SHOW TABLES' HiveQL command?  The Data Source Explorer uses HCatalog to get the list and isn't executing JDBC commands. Maybe it the user, hue, you are logged in as that does not have permission to see the tables in default. Is there a 'guest' user login you can try for the sandbox, guest, guest-password?

Tuen 24 comments Joined 07/05
16 May 2015

In this instance, this would be the data source explorer, if i connect to a Hadoop 2.1 sandobx, i can see the tables under the default database in the data source explorer, if i log into hadoop 2.2 i see the default database but when i expand it out, nothng shows up.  I granted the user hue select on the default database since i figured that it might be a permissions issue, but that didn't seem to address the issue.   You mention that it is using HiveQL, does that mean i need to have something else installed on the hadoop 2.2 machine?  such as TDCH or something else for it to be able to query?   I know querygrid isn't supported on 2.2 yet, i did install TDCH 1.3.4 and ran the oozie configuration script which appears to completed with no issues.   Is TDCH required if I just want to use the JDBC connection method?  Just trying to figure out how some of this is supposed to work before i start trying to mess with our production system.

fgrimmer 660 comments Joined 04/09
17 May 2015

Bill, Studio is using WebHCat to gather information from Hadoop for the displaying the objects in the DSE. The SQL Editor window uses HiverServer2 JDBC driver to execute the HiveQL commands. You do not need TDCH installed if you are just running HiveQL commands. The WebHCat user can be different than the JDBC user. Are you using hue for both? Just curious if you can execute the SHOW TABLES command in the SQL Editor to your 2.2 sandbox?

kallu2020 10 comments Joined 03/14
11 Jun 2015

Hi, Thanks for putting the info.I am facing issue while connecting to HDP2.2 VM sandbox thru TDS. My operating system is Os X and the HDP2.2 VM is running and configure Oozie.sh ran successfully on hadoop VM. I am also using VM for TD15.
TD15 VM has a studio client which is at version 15.00 and doesn’t have hadoop data source option. Hence I have installed TDS 15.10 separately on my local machine. Now my HDP 2.2 is running on VM but I am not able to make connection to Hadoop thru TDS. I have been receiving ‘ping failed’ error with exception
java.lang.Exception: Cannot connect to WebHDFS, check host name and port.Could you help.Could you help.

fgrimmer 660 comments Joined 04/09
11 Jun 2015

@kalu2020, Here is a response for a customer regarding the HDP sandbox. It may be the same issue:
Here is a response for a customer with the same issue:
Are you connecting to the sandbox or a multi-node cluster?  One step that is usually required is adding the internal hostnames of the cluster (sandbox.hortonworks.com for the sandbox) to your Window’s hosts file.  Here is some example entries (your internal names could be different):
10.25.12.151         hdp005-2
10.25.12.158         hdp005-4
10.25.12.159         hdp005-5
10.25.12.154         hdp005-6
10.25.12.155         hdp005-7
192.168.67.131    sandbox.hortonworks.com
The reason we need to add the entries to the Window’s hosts file is because when we try to read a file in WebHDFS, it redirects us to a datanode, however that redirection has the internal datanode name.  By adding them to the hosts file, we are able to resolve them.
Another possibility:
Check if your client can access Oozie, you can visit http://{ooziehost}:11000/oozie in any web browser. One thing to note here is that I had a multiple node cluster before, with 2 masters.  On master 1, it declared the Oozie URL as pitH1.td.teradata.com:11000 and on master 2, I declared the Oozie URL as pitH2.td.teradata.com:11000.  For some reason Studio could only connect when Oozie was specified as pitH2.td.teradata.com.  Not sure if you have a multiple node cluster, but this could be the issue.
Regards.

kallu2020 10 comments Joined 03/14
14 Jun 2015

Thanks. Well, there is an entry in host file for IP addr with corresponding server name(sandbox.hortonworks.com).But still I was not able to connect to hadoop using TDS. However when I use IP address directly in webHDFS host name, I am able to make connection.
Now in my TDS, I have connectivity for TD15 VM and hadoop cluster.However I am not seeing anything under databases which is under hue(hue is the webHDFS username I am using). I have few DBs/tables created in Hive also there are test files available in HDFS.
Do you know why I am not seeing any lineage in TDS under hadoop databases(the screenshot that you provide in your article is very informative but the I am not seeing the same here)

Name node and data node are up which I can see by typing jps in hadoop
Fyi, I am not able to browse into access http://sandbox.hortonworks.com:8000
~Kal

harristd 5 comments Joined 07/12
17 Jun 2015

@kallu2020  If you are able to connect to the VM using the IP address but not the Hostname, then the entry must be incorrect in the hosts file.
As for the separate issue of not being able to see the DSE tree under "Hadoop Databases" in Studio, that problem almost always comes from VMs that don't have enough memory.  Try increasing the memory of the hadoop VM to at least 8 GB.  Our VMs with only 4 GB memory have the same problem, no response from webHDFS (which is why you see an empty tree).  When we increase them to 8 GB, the tree will start working properly because the Hadoop system has enough resources to respond.

kallu2020 10 comments Joined 03/14
01 Jul 2015

Thanks harristd. Right now I moved on to TDCH command line option to perform export to Teratata form hadoop since my VM memory is causing a bottleneck.
Followed all the step that Francine suggested earlier. However I keep getting error 'Export tool parameters is invalid' like below.
 
Command:
hadoop com.teradata.hadoop.tool.TeradataExportTool
-url jdbc:teradata://192.x.x<ipaddress>/database=userdb
-username dbc -password dbc
-classname com.teradata.jdbc.TeraDriver
-fileformat textfile -jobtype hdfs –method internal.fastload
-sourcepaths /home/user/sample1.txt -nummapper 20 -separator ','
-targettable  employee
 
**Note: I am able to connect to database thru workbench using the above url and username/pwd credentials
 
Error details:
15/07/01 00:20:36 INFO tool.ConnectorExportTool: com.teradata.connector.common.exception.ConnectorException: Export tool parameters is invalid
at com.teradata.connector.common.tool.ConnectorExportTool.processArgs(ConnectorExportTool.java:250)
at com.teradata.connector.common.tool.ConnectorExportTool.run(ConnectorExportTool.java:52)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
at com.teradata.hadoop.tool.TeradataExportTool.main(TeradataExportTool.java:24)
 
Steps followed: (also referred doc 'Teradata Connector for Hadoop Tutorial v1 0 final')
1. After unzipping Teradata Connector for Hadoop tar.gz package, copy the following files into Hadoop's lib directory:
/usr/lib/hadoop/lib
a) teradata-connector-1.0.6.jar   b) tdgssconfig.jar   c) terajdbc4.jar
2. Copy the following files into Hadoop's conf directory:
/etc/hadoop/conf
   a) teradata-export-properties.xml b) teradata-import-properties.xml
3. Modify mapred.child.java.opts property value in mapred-site.xml and add /etc/hadoop/conf
   -Djava.security.egd=file:/dev/./urandom
 4. Modify the following OS networking values on the name node (recommended - Optional)
   /proc/sys/net/core/somaxconn to 1024
   /proc/sys/net/ipv4/tcp_max_syn_backlog to 2048
Could you please suggest how to overcome this? Appreciate your help. Thanks.
 

fgrimmer 660 comments Joined 04/09
01 Jul 2015

@kalu2020, Please post you TDCH command line question to the Hadoop Forum (http://forums.teradata.com/forum/hadoop). Thanks

TDThrottle 11 comments Joined 11/11
04 Jul 2015

Hi,
Want to confirm if Smart loader JDBC connectivity support Load utility checkpoint while moving data from Hadoop to Teradata.
Thanks!!
 
 
 

fgrimmer 660 comments Joined 04/09
04 Jul 2015

@TDThrottle, Smart Loader for Hadoop supports moving data from Hadoop to Teradata. It uses the Teradata Connector for Hadoop (TDCH), which opens a JDBC FastLoad connection between Hadoop and Teradata.

kallu2020 10 comments Joined 03/14
08 Jul 2015

Sure Francine will do that going forward. Just so everyone aware, I am able to over come Export tool error with TDCH by removing the file in hdfs and properly issuing CopyToLocal again to copy it in hdfs. Thanks.

You must sign in to leave a comment.