Hadoop systems [1], sometimes called Map Reduce, can coexist with the Teradata Data Warehouse allowing each subsystem to be used for its core strength when solving business problems. Integrating the Teradata Database with Hadoop turns out to be straight forward using existing Teradata utilities and SQL capabilities. There are a few options for directly integrating data from a Hadoop Distributed File System (HDFS) with a Teradata Enterprise Data Warehouse (EDW), including using SQL and Fastload. This document focuses on using a Table Function UDF to both access and load HDFS data into the Teradata EDW. In our examples, there is historical data already in Teradata EDW, presumably derived from HDFS for trend analysis. We will show examples where the Table Function UDF approach is used to perform inserts or joins from HDFS with the data warehouse.

Map-reduce, or its open source version Hadoop, is a parallel programming framework for running scripts, Java, C, and other external programming languages on hundreds of nodes. It is popular with Dot.Com companies who have large server farms and need to produce reports on website activity or produce search indexes. In general, Map-reduce applications overlap BI applications and data warehouses. However, Map-reduce applications can coexist with a data warehouse: one parallel processing, the other parallel database. Coexistence allows each subsystem’s best capabilities to be used to complement the other. With Teradata’s in-database processing technology, Map-reduce can become MPP ETL subsystem, or we can run Map-Reduce functions inside the EDW, or using table functions we can directly integrate with the Map-reduce nodes. This article illustrates a commonly used Map-reduce function running inside the Teradata EDW.

In my previous article we explored Teradata's spatial features using geometric shapes on a 10x10 grid.  I hope this was a useful introduction into the new ST_Geometry data type and its powerful set of functions.  And now that we've covered the basics, let's take a step into the world of geospatial.  By geospatial, we're now talking about locations based on geographic reference points on Earth; for example longitude and latitude.

Now that you’ve installed Teradata’s geospatial features with Teradata Express let’s roll up our sleeves and see what we can do with this. (If you haven’t setup your TD Express with Geospatial yet, see my article. I think the best way to begin exploring these features is to start is with some basic geometric shapes.

We’re in the midst of an explosion in the world of location data. GPS tools have greatly expanded our ability to capture location data and mapping tools have broadly expanded its use. We have Web navigation sites like Google Earth and Yahoo Maps. Our cars have navigation devices to keep us from getting lost. Companies use GPS devices to track their shipments and mobile assets. Our digital cameras add location tags to photos. And we even have dog collars that can track Fido’s adventures around the neighborhood!

From time to time I’m asked how one might go about writing an UDF or an INMod or some other procedural extension to the database. The question isn’t a “where do I learn C or C++” question; rather it is a how do I go about debugging and testing my extension without the overheads and constraints of running within the database (or utility).

My response is you can use “my Teradata” for which I have the source code (attached). This will allow you to run, test and debug your function outside of Teradata. In short with “my Teradata” all of the features of your IDE will be available to you, including your debugger.