Supervision et administrationdes plateformes HadoopOù en est-on ? Comment relier cela à la supervision du SI ?

Cédric Carbone - @carboneBertrand Dechoux - @BertrandDechoux

Hadoop

“perdre une machine est un non-événement”

mais

avez vous besoins de vos applications ?

Hadoop 2 : stockage haute dispo

NameNode NameNode

DataNode DataNode DataNode DataNode

DataNode DataNode DataNode

Hadoop 2 : calcul haute dispo

RessourceManager

RessourceManager

NodeManager

NodeManager

NodeManager

NodeManager

NodeManager

NodeManager

NodeManager

Mesurer, pourquoi ?

● Notifier les erreurs, matérielles ou applicatives

● Diagnostiquer les erreurs, matérielles ou applicatives

● Planifier les ressources nécessaires

Apache Ambari, pour Hadoop

● Provision a Hadoop Clustera. Ambari provides an easy-to-use, step-by-step wizard for installing Hadoop services across any number of hosts.b. Ambari handles configuration of Hadoop services for the cluster.

● Manage a Hadoop Clustera. Ambari provides central management for starting, stopping, and reconfiguring Hadoop services across the entire cluster.

● Monitor a Hadoop Clustera. Ambari provides a dashboard for monitoring health and status of the Hadoop cluster.b. Ambari will send emails when your attention is needed (e.g., a node goes down, remaining disk space is low, etc).

Ambari consists of a set of RESTful APIs and a browser-based management interface.

Cloudera Manager, pour Hadoop

Apache Ambari / Cloudera Manager

● Both Apache Ambari and Cloudera Manager do the job of managing and monitoring your cluster

● Difference on the Security layer (Sentry vs Apache Ranger) and on the Business Model (OSS vs Proprietary)

● Cloudera Manager started 3 years before Ambari but the feature-gap is quickly closing.

● However, with both Ambari / Cloudera Manager you cannot dig into your application => you need to add a dedicated monitoring/log management tool

Log centralisation

Logmatic

Monitoring

● série temporelle

● RRD○ “Round Robin Database”

Monitoring : build or buy

hardware

OS

applications

build

buy

Cédric Carbone - @carboneBertrand Dechoux - @BertrandDechoux