lug2015 - frédérick lefebvre - monitoring an heterogenous lustre environment

Download LUG2015 - Frédérick Lefebvre - Monitoring an heterogenous Lustre environment

Post on 09-Feb-2017

124 views

Category:

Documents

0 download

Embed Size (px)

TRANSCRIPT

  • Calcul Qubec - Universit LavalCalcul Qubec - Universit Laval

    Monitoring an heterogenous Lustre environment

    LUG 2015

    frederick.lefebvre@calculquebec.ca

  • Calcul Qubec - Universit LavalCalcul Qubec - Universit LavalCalcul Qubec - Universit Laval

    PlanSite descriptionA bit of historyReal time utilization monitoringMoving forward

  • Calcul Qubec - Universit LavalCalcul Qubec - Universit LavalCalcul Qubec - Universit Laval

    Who we areAdvanced computing center @Universit LavalPart of Calcul Qubec & Compute CanadaOperate 2 Clusters & 4 Lustre fs

  • Calcul Qubec - Universit LavalCalcul Qubec - Universit LavalCalcul Qubec - Universit Laval

    Our infrastructure1000+ lustre clients (2.5.3 & 2.1.6)

    On 2 IB fabrics - 6 LNET routers

    4 Lustre file systems (~2.5 PB)500 TB - Lustre 2.5.3 + ZFS - 16 OSSs500 TB - Lustre 2.1.6 - 6 OSSs1.4 PB - Lustre 2.1.0 (Seagate) - 12 OSS120 TB - Lustre 2.1.0 (Seagate) - 2 OSS

  • Calcul Qubec - Universit LavalCalcul Qubec - Universit LavalCalcul Qubec - Universit Laval

    IB S

    witc

    hIB

    Sw

    itch

    IB Switch

    IB Switch

    IB S

    witc

    hIB

    Sw

    itch

    IB S

    witc

    h

    IB S

    witc

    h

    Lustre1 - 2.5.3 + ZFS

    Lustre2 - 2.1.6

    Lustre3 - 2.1.0 Seagate

    Lustre4 - 2.1.0 Seagate

    Topology (2 IB fabrics)

    IB S

    witc

    h

    6x LNET routers

    Colosse : 960 nodes

    Helios : 21 nodes - 168 GPUs

  • Calcul Qubec - Universit LavalCalcul Qubec - Universit LavalCalcul Qubec - Universit Laval

    An history of monitoringWe initially monitored because

    We knew we shouldWe wanted to identify failures

    Tried multiple options over the yearscollectlcerebro +lmt

  • Calcul Qubec - Universit LavalCalcul Qubec - Universit LavalCalcul Qubec - Universit Laval

    An history of monitoring (cont.)Converged towards custom gmetrics scripts for Ganglia

    Monitored IBLustre IOPS and throughput

  • Calcul Qubec - Universit LavalCalcul Qubec - Universit LavalCalcul Qubec - Universit Laval

    An history of monitoring (cont.)Converged towards custom gmetrics scripts for Ganglia

    Monitored IBLustre IOPS and throughput

    Worked reasonably wellBut quality of our scripts was inconsistent Some used a surprising amount of resources when running

  • Calcul Qubec - Universit LavalCalcul Qubec - Universit LavalCalcul Qubec - Universit Laval

    IssuesUsers keep asking the same questions

    Why is it so long navigating my files ?Why is my app so slow at writing results ?Which filesystem should I use ?

    We had most of the data somewhere but searching for it was always painful

  • Calcul Qubec - Universit LavalCalcul Qubec - Universit LavalCalcul Qubec - Universit Laval

    Issues (cont.)Data is everywhere

    Performance data from compute node is in mongoDB Lustre client and server data in Ganglia (RRD) Scheduler data in text log filesEasy to end up with a spiderweb of complex script to integrate all data

  • Calcul Qubec - Universit LavalCalcul Qubec - Universit LavalCalcul Qubec - Universit Laval

    New reqs1. Give staff a near-realtime view of the filesystems

    utilisation and load

    2. Provide end-users with data to identify their utilisation of the Lustre filesystemsa. Bonus point for cross-referencing with the schedulers stats

  • Calcul Qubec - Universit LavalCalcul Qubec - Universit LavalCalcul Qubec - Universit Laval

    New reqs (cont.)We want to keep historical data for future analysis and comparisonWith the amount of storage now available, it didnt make sense to keep losing granularity over time with RRD.

  • Calcul Qubec - Universit LavalCalcul Qubec - Universit LavalCalcul Qubec - Universit Laval

    Enter SplunkWe have been using Splunk to store and visualize system logs and energy efficiency data.Its well suited to indexing a lot of unstructured data

  • Calcul Qubec - Universit LavalCalcul Qubec - Universit LavalCalcul Qubec - Universit Laval

    PlanCollect data from /proc on all nodes and send it to the systems log with loggerThe logs are indexed and visualized with Splunk

    We use a separate index for performance data

  • Calcul Qubec - Universit LavalCalcul Qubec - Universit LavalCalcul Qubec - Universit Laval

    Data collectionExisting options to monitor Lustre tend to be tied to a specific backend (RRD or DB)

    More flexibility is needed

    We decided to write another oneWe could have modified an existing application

  • Calcul Qubec - Universit LavalCalcul Qubec - Universit LavalCalcul Qubec - Universit Laval

    On Lustre clients and serverscq_mon

    Overview

    Run...Log...Sleep...

    python-lustre

    python-psutil

    plug

    ins

    ...

    rsyslogd

    syslog

    On Splunk server

    rsyslogd /var/log/...

    Splunk

    IndexSearch engineHTTPS

    syslog write to file

  • Calcul Qubec - Universit LavalCalcul Qubec - Universit LavalCalcul Qubec - Universit Laval

    cq_mon

    cq_mon

    Run...Log...Sleep...

    plug

    ins

    Writen in PythonConfiguration specifies which plugins to runRuns each plugin at interval and log the results as key=values to syslog

  • Calcul Qubec - Universit LavalCalcul Qubec - Universit LavalCalcul Qubec - Universit Laval

    cq_mon

    cq_mon

    Run...Log...Sleep...

    plug

    ins

    Writen in PythonConfiguration specifies which plugins to runRuns each plugin at interval and log the results as key=values to syslogAdd syslog filter so as not to polute system logs with performance data

    syslog

    rsyslogd

  • Calcul Qubec - Universit LavalCalcul Qubec - Universit LavalCalcul Qubec - Universit Laval

    cq_mon

    python-lustre_util

    Run...Log...Sleep...

    plug

    ins

    Library to interact with lustre through /procLoads stats for different Lustre components make them available through an appropriate object (ie: lustre_mdt, lustre_ost, etc)

    python-lustre

  • Calcul Qubec - Universit LavalCalcul Qubec - Universit LavalCalcul Qubec - Universit Laval

    python-lustre_util (cont.)Work in progress

    Tested with Lustre 2.1.6, 2.4.x, 2.5.x+ZFS and 2.1.0-xyratexAvailable through Github: https://github.com/calculquebec

    Offer stats for:MDS, OSS, MDT, OSTClient stats from the client and not the OSS/MDS

  • Calcul Qubec - Universit LavalCalcul Qubec - Universit LavalCalcul Qubec - Universit Laval

    python-lustre_util (cont.)cq_mon

    cq_mon_lustre_oss

    loads plugins

    cq_mon.cfg-defines tests to run and intervals-Managed by puppet

    reads

    lustre_oss lustre_oss_stats

    instantiate

    reads /proc lustre_oss_stats and compare with previous instance

    lustre-utils

    Other plugins use python-psutil to provide OS level stats

    write to syslog

  • Calcul Qubec - Universit LavalCalcul Qubec - Universit LavalCalcul Qubec - Universit Laval

    Splunk dashboardsPut together custom dashboards for our needs

    Lustre filesystems overview1 detailed dashboard for each Lustre filesystemJob report

    Some reports are slower than we would like

  • Calcul Qubec - Universit LavalCalcul Qubec - Universit LavalCalcul Qubec - Universit Laval

    Lustre - overview

  • Calcul Qubec - Universit LavalCalcul Qubec - Universit LavalCalcul Qubec - Universit Laval

    Lustre overview (cont.)

  • Calcul Qubec - Universit LavalCalcul Qubec - Universit LavalCalcul Qubec - Universit Laval

    Lustre overview (cont.)

  • Calcul Qubec - Universit LavalCalcul Qubec - Universit LavalCalcul Qubec - Universit Laval

    Lustre overview (Client IOPS)

  • Calcul Qubec - Universit LavalCalcul Qubec - Universit LavalCalcul Qubec - Universit Laval

    Per FS view (Utilization)

  • Calcul Qubec - Universit LavalCalcul Qubec - Universit LavalCalcul Qubec - Universit Laval

    Per FS view (top users)

  • Calcul Qubec - Universit LavalCalcul Qubec - Universit LavalCalcul Qubec - Universit Laval

    Job report

  • Calcul Qubec - Universit LavalCalcul Qubec - Universit LavalCalcul Qubec - Universit Laval

    LimitationsThe quantity of data collected is scary

    Especially true for clients (~1MB/client per day)

    Impact of running extra codes on computes nodesStill uncertainty on how to monitor or visualize some components

    ie: LNET routers, failovers, DNE setups, drives

  • Calcul Qubec - Universit LavalCalcul Qubec - Universit LavalCalcul Qubec - Universit Laval

    Looking aheadAdd support for multiple MDS/MDTCollect clients data on the serversCross-reference statistics with data from RobinHood DBMake the code more resistant to failovers

  • Calcul Qubec - Universit LavalCalcul Qubec - Universit LavalCalcul Qubec - Universit Laval

    Thank you

    frederick.lefebvre@calculquebec.ca