Teradata’s Unified Data Architecture is a powerful combination of Teradata, Aster, and Hadoop in a single platform.  Viewpoint has always provided monitoring and management of Teradata systems and launched support for monitoring of Aster in Viewpoint 14.01.  In order to complete Viewpoint’s monitoring of the different systems in Teradata’s Unified Data Architecture, Viewpoint 14.10 includes support for monitoring of Hadoop running in this architecture.

The biggest technical challenge Viewpoint faced when monitoring a Hadoop system was how to reliably and easily collect the necessary data from Hadoop.  The different components of Hadoop expose their data in a variety of different ways, including using Ganglia, Nagios, JMX, and some really ugly web interfaces.  There are two primary issues with using these existing technologies for Hadoop monitoring: parsing the data from each different interface and being able to locate and connect to these interfaces on each Hadoop node.  Each of these technologies exposes their data in a different format, and it would take significant development time to properly parse the data from each source.  There’s also a challenge in locating and communicating with the nodes to obtain this data.  Just to collect data from the namenode and jobtracker, the location of these services would have to be configured or discovered, and then failover would have to be accounted for as well.  Expanding the monitoring solution beyond that to collect data from every node poses both connectivity and security issues as well.  Surely there must be a better way!

Luckily Apache Ambari addresses all of these technical challenges by providing a collection of RESTful APIs from which a plethora of Hadoop monitoring data can be obtained.  Ambari handles the work of collecting the monitoring data from a variety of the monitoring technologies mentioned above.  It then aggregates this data and provides a series of RESTful APIs.  These APIs can all be accessed by making web service calls against a central node in the Hadoop cluster.  All data is provided in JSON format so it can easily be parsed by just about any programming language.

Since Viewpoint is written in Java and uses the Spring Framework quite extensively, Spring’s RestTemplate class was a natural choice for calling the RESTful APIs and parsing the results into Java model objects.  Here is some sample code to demonstrate the collection of the number of running MapReduce jobs, map tasks, and reduce tasks from Ambari.

 

package com.teradata.viewpoint.ambari;

import java.io.IOException;
import java.net.HttpURLConnection;
import java.util.ArrayList;
import java.util.List;

import org.apache.commons.codec.binary.Base64;
import org.codehaus.jackson.annotate.JsonProperty;
import org.codehaus.jackson.map.DeserializationConfig;
import org.springframework.http.MediaType;
import org.springframework.http.client.SimpleClientHttpRequestFactory;
import org.springframework.http.converter.HttpMessageConverter;
import org.springframework.http.converter.json.MappingJacksonHttpMessageConverter;
import org.springframework.web.client.RestTemplate;

public class AmbariClient
{
    private String host;

    private String clusterName;

    private String user;

    private String password;

    private RestTemplate restTemplate;

    public AmbariClient(String host, String clusterName, String user, String password)
    {
        this.host = host;
        this.clusterName = clusterName;
        this.user = user;
        this.password = password;

        List<MediaType> supportedMediaTypes = new ArrayList<MediaType>();
        MediaType plainTextType = new MediaType("text", "plain");
        MediaType jsonType = new MediaType("application", "json");

        supportedMediaTypes.add(plainTextType);
        supportedMediaTypes.add(jsonType);

        MappingJacksonHttpMessageConverter mappingJacksonHttpMessageConverter = new MappingJacksonHttpMessageConverter();
        mappingJacksonHttpMessageConverter.setSupportedMediaTypes(supportedMediaTypes);
        mappingJacksonHttpMessageConverter.getObjectMapper().configure(
                DeserializationConfig.Feature.FAIL_ON_UNKNOWN_PROPERTIES, false);

        List<HttpMessageConverter<?>> messageConverters = new ArrayList<HttpMessageConverter<?>>();
        messageConverters.add(mappingJacksonHttpMessageConverter);

        restTemplate = new RestTemplate();
        restTemplate.setMessageConverters(messageConverters);
    }

    public <T> T getAmbariHadoopObject(String url, Class<?> clazz)
    {
        SimpleClientHttpRequestFactory requestFactory = new SimpleClientHttpRequestFactory()
        {
            @Override
            protected void prepareConnection(HttpURLConnection connection, String httpMethod)
                    throws IOException
            {
                super.prepareConnection(connection, httpMethod);

                String authorisation = user + ":" + password;
                String encodedAuthorisation = Base64.encodeBase64String(authorisation.getBytes());
                connection.setRequestProperty("Authorization", "Basic " + encodedAuthorisation);
                connection.setConnectTimeout(30000);
                connection.setReadTimeout(120000);
            }
        };

        restTemplate.setRequestFactory(requestFactory);

        String fullUrl = "http://" + host + "/api/v1/clusters/" + clusterName + url;
        return (T) restTemplate.getForObject(fullUrl, clazz);
    }

    /**
     * Model class to hold the data from the JSON response.
     */
    private static final class JobTrackerData
    {
        public class Metrics
        {
            public class MapReduce
            {
                public class JobTracker
                {
                    @JsonProperty("jobs_running")
                    private Integer jobsRunning;

                    @JsonProperty("running_maps")
                    private Integer runningMaps;

                    @JsonProperty("running_reduces")
                    private Integer runningReduces;
                }

                @JsonProperty("jobtracker")
                private JobTracker jobTracker;
            }

            @JsonProperty("mapred")
            private MapReduce mapReduce;
        }

        @JsonProperty("metrics")
        private Metrics metrics;
    }

    public static void main(String[] args)
    {
        AmbariClient client = new AmbariClient("ambari.teradata.com",
                "clustername", "admin", "admin");
        JobTrackerData data = client.getAmbariHadoopObject(
                "/services/MAPREDUCE/components/JOBTRACKER", JobTrackerData.class);
        System.out.println("Jobs running: " + data.metrics.mapReduce.jobTracker.jobsRunning);
        System.out.println("Map tasks running: " + data.metrics.mapReduce.jobTracker.runningMaps);
        System.out.println("Reduce tasks running: "
                + data.metrics.mapReduce.jobTracker.runningReduces);
    }
}

Following Viewpoint’s standard data collection practices, all of the data collected from Ambari is stored in the Viewpoint database.  The data is collected from Ambari every minute by default, and therefore the database has a view of the state of the Hadoop system over the course of an hour, day, or week.  This historical data is used to generate a variety of different charts in the Viewpoint web portal, and also to enable the use of Rewind to enable users to go back and see exactly what was occurring on the Hadoop cluster at a specific point in time.

By using Ambari for monitoring of a Hadoop cluster, Viewpoint was able to deliver a comprehensive Hadoop monitoring solution in a relatively short amount of time.  Viewpoint’s Java and web developers were able to focus on the tasks at which they excel: getting the data from the source system (Ambari) and displaying it in Viewpoint’s portlets.  No time was wasted trying to get up to speed on Ganglia, JMX, or many of the details of Hadoop’s inner workings.  Ambari was a critical piece of technology to help Viewpoint roll out this solution and enhance Viewpoint’s support of Teradata’s Unified Data Architecture.

Discussion
shendong 3 comments Joined 09/13
17 Sep 2013

Hi Stever,
    I wonder if we can deploy HDP by using Ambari RestTemplate API instead of the curl commands。
    Thank you!

stever 21 comments Joined 08/09
17 Sep 2013

Teradata Viewpoint is presently only using the monitoring APIs that are part of HDP and Ambari, so I'm not sure I can provide you a definitive answer to your question about deploying HDP.  However, I'm guessing that Ambari doesn't really care how you call any of the REST APIs.  Certainly you can see that in Viewpoint we are using RestTemplate to successfully call the monitoring APIs.  So while CURL might be good for command line or PERL integration, if you want to develop something in Java to deploy HDP I'm guessing the Spring RestTemplate programming model would work as well.

shendong 3 comments Joined 09/13
23 Sep 2013

Hi Stever,
I've tried to run your demo. However I got a wrong message, 400 Bad Request. Following is the ambari-server log.
 

DEBUG Server:358 - REQUEST 0 on AsyncHttpConnection@d996e77,g=HttpGenerator{s=0,h=-1,b=-1,c=-1},p=HttpParser{s=7,l=0,c=-3},r=2

DEBUG ContextHandler:894 - scope null||0 @ o.e.j.s.ServletContextHandler{/,file:/usr/lib/ambari-server/web/}

DEBUG ContextHandler:966 - context=||/ @ o.e.j.s.ServletContextHandler{/,file:/usr/lib/ambari-server/web/}

DEBUG session:183 - sessionManager=org.eclipse.jetty.server.session.HashSessionManager@7cfae78f

DEBUG session:184 - session=null

DEBUG ServletHandler:360 - servlet |/|null -> org.eclipse.jetty.servlet.DefaultServlet-1

DEBUG ServletHandler:422 - chain=null

DEBUG AbstractHttpConnection:497 - 

org.eclipse.jetty.io.EofException

at org.eclipse.jetty.http.HttpGenerator.flushBuffer(HttpGenerator.java:914)

at org.eclipse.jetty.http.AbstractGenerator.flush(AbstractGenerator.java:443)

at org.eclipse.jetty.server.HttpOutput.flush(HttpOutput.java:100)

at org.eclipse.jetty.server.AbstractHttpConnection$Output.flush(AbstractHttpConnection.java:1072)

at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:161)

at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:107)

at org.eclipse.jetty.util.ByteArrayISO8859Writer.writeTo(ByteArrayISO8859Writer.java:107)

at org.eclipse.jetty.server.Response.sendError(Response.java:390)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:752)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)

at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:652)

at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:447)

at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)

at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:559)

at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:227)

at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1038)

at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:374)

at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:189)

at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:972)

at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)

at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)

at org.eclipse.jetty.server.Server.handle(Server.java:359)

at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:483)

at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:920)

at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:982)

at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:444)

at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)

at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)

at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:627)

at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:51)

at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)

at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)

at java.lang.Thread.run(Thread.java:662)

Caused by: java.nio.channels.ClosedChannelException

at sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:135)

at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:324)

at org.eclipse.jetty.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:293)

at org.eclipse.jetty.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:361)

at org.eclipse.jetty.http.HttpGenerator.flushBuffer(HttpGenerator.java:850)

... 32 more

 

OS is RedHat 6.2.

Version of Ambari Server is 1.2.1.

And I have installed HDFS, MAPREDUCE, GANGLIA and NAGIOS. The whole system works well.

I hope you can give me some suggestions. Thank you.

stever 21 comments Joined 08/09
24 Sep 2013

What is the exact error message?  It should be returned in the HTTP response body.

shendong 3 comments Joined 09/13
25 Sep 2013

I have found the problem.
BASE64 will end the transformed code with "\r\n", however the server can not analyze it correctly. 
As a result, user authorization can not go through.
It works well after I delete the "\r\n".

Teradata_Mark 10 comments Joined 09/10
2 months ago

Can you tell me what are the Hadoop login credentials you used in Viewpoint under Teradata Systems pane?
I am trying to get the portlet to display but only get blank screen.  I'm running Horton 1.3 sandbox and getting login error when I click the TEST button. I tried admin/admin since this is the Ambari login, but keep getting Orange triangle.
 
 

stever 21 comments Joined 08/09
2 months ago

You need to figure out what port the Ambari web server is listening on.  By default in Teradata Hadoop distributions it's 8081.  You might also try 8080.  You specify the port number in the HOST ID field just like you would for any other URL.  For example, 192.168.100.130:8081.
As for the password, there isn't one on the Teradata Hadoop distribution, so you can enter whatever you like in the login name and password fields.

cranies 2 comments Joined 10/09
1 month ago

Hi,
Can you use Viewpoint to monitor non-TD Appliance Hadoop systems?

stever 21 comments Joined 08/09
1 month ago

No, this is not currently supported.

You must sign in to leave a comment.