Grid 5000: exposing distributed systems to their limits

Transcription

1 Grid 5000: exposing distributed systems to their limits Franck Cappello INRIA Director of Grid 5000 Scientific Director of ALADDIN fr lri.fr/ fr/~fci 1

2 Agenda The Grid 5000 project Examples of Research Platform results (as an instrument) International extensions ALADDIN 2

3 Grids raise many research challenges Applications gather many components: user interface, middleware, OS, protocols on computing, storage and visualization devices. Computational Steering of a Multi-physics application (code coupling, Multi-heterogeneous sites, Heterogeneous network, Main Large Scale Distributed Systems volatility, Interactive etc.) researchuse, issues: Security, Performance, Fault tolerance, Scalability, Load Balancing, Coordination, deployment, accounting, Data storage, Programming, Communication protocols, Accounting, etc. 3

4 Existing Grid Research Tools SimGrid3 Discrete event simulation with trace injection Originally dedicated to scheduling studies Single user, multiple servers France USA Australia Japan GridSim Dedicated to scheduling (with deadline, budget), DES (Java) Multi-clients, Multi-brokers, Multi-servers Support for SLAs, DataGrids, Service Model Titech Bricks Discrete event simulation for scheduling and replication studies GangSim Scheduling inside and between VOs MicroGrid, Emulator, Dedicated to Globus, Virtualizes resources and time, Network (MaSSf) --> Simulators and Emulators are quite slow and do not scale well 4

5 What about production platforms used for experimental purpose? No reproducible experimental conditions: Scientific studies require reproducible experimental conditions Not designed for experiments: Many researchers run short length, highly parallel & distributed algos Preparation and execution of experiments are highly interactive Not optimized for experiments: Experimental platforms should exhibit a low utilization rate to allow researchers executing large collocated experiments Not reconfigurable: Many projects require experiments on OS and networks, Some projects require the installation of specific hardware Nowhere to test networking/os/middleware ideas, to measure real application performance, 5

6 Grid 5000 VS. EGEE Bronze Standard method addressing the issue of medical image algo. evaluation. Application on estimation of the spatial rigid transformation between two images (convenient to align two different images of a same patient acquired separately). Complex workflow of computations on large number of data sets. Typically require 10s to 100s of 3D images pairs. 15 minutes per image pair. The application is executed with MOTEUR (workflow engine) Several degrees of parallelism are tested: Execution time (seconds) Execution time (seconds) Naive execution only the workflow intrinsic parallelism Data Parallelism data sets are processed concurrently Data Parallelism data sets are processed concurrently Data Parallelism + Pipelining Data Parallelism + Pipelining services in sequential branches are pipelined number of images number of images 6

7 Exploration tools In 2002 (DAS2) and 2003 (Grid 5000), the design and development of an experimental platform for Grid researchers was initiated: log(cost & coordination) Major challenge Challenging Reasonable SimGrid MicroGrid GridSim Bricks Model NS, etc. Protocol proof Data Grid explorer Emulab This talk RAMP Dave Patterson s Project on Muticore Multi-processor emulator Grid 5000 DAS3 PlanetLab GENI OneLab, PANLab SensLab, etc. log(realism) math simulation emulation live systems 7

8 The Grid 5000 Project 1) Building a nationwide experimental platform for Large scale Grid & P2P experiments 9 geographically distributed sites Every site hosts a cluster (from 256 CPUs to 1K CPUs) All sites are connected by RENATER (French Res. and Edu. Net.) RENATER hosts probes to trace network load conditions Design and develop a system/middleware environment for safely testing and repeating experiments 2) Use the platform for Grid experiments in real life conditions Port and test applications, develop new algorithms Address critical issues of Grid system/middleware: Programming, Scalability, Fault Tolerance, Scheduling Address critical issues of Grid Networking High performance transport protocols, Qos Investigate novel mechanisms P2P resources discovery, Desktop Grids 8

9 Grid 5000 map Lille: 500 (198) Nancy: 500 (334) Rennes 522 (522) 1000 (684) Bordeaux 500 (198) Toulouse 500 (116) Lyon 500 (252) Grenoble 500 (270) Sophia Antipolis 500 (434) 9

10 Grid 5000 network Renater connections 10 Gbps Dark fiber Dedicated Lambda Fully isolated traffic! 10

11 FAIL: FAult Injection Language High-level language for fault scenario Fault-injection middleware Also suitable for stress testing and user simulation Quantitative fault-injection: Finding rupture point Qualitative fault-injection: Finding implementation bugs 11

12 Geophysics: Seismic Ray Tracing in 3D mesh of the Earth Building a seismic tomography model of the Earth geology using seismic wave propagation characteristics in the Earth. Seismic waves are modeled from events detected by sensors. Ray tracing algorithm: waves are reconstructed from rays traced between the epicenter and one sensor. A MPI parallel program composed of 3 steps 1) Master-worker: ray tracing and mesh update by each process with blocks of rays successively fetched from the master process, 2) all-to all communications to exchange submesh in-formation between the processes, 3) merging of cell information of the submesh associated with each process. Reference: 32 CPUs 12

13 BitDew How to manage data transparently and efficiently on distributed resources? Scalable, Simple, Transparent and Efficient environment offering Fault Tolerance, Replication, Distribution, High performance and Reliability (transport) For Data sets on heterogeneous, Volatile resources. Programmer uses MetaData to specify attributes on data sets. (#replicas, FT, Affinity, Protocols) Tests on Grid 5000, 3 clusters 13

14 TCP limits at 10Gb/s What transport protocol for large data sets on Grids? In Grids, TCP is used by most applications & libraries (MPI, GridFTP.) Long distance High speed networks are challenging for TCP & transport protocols Designing new schemes is not straightforward (fairness, stability, friendliness) Without reverse traffic With reverse traffic New variants are proposed/implemented in Linux (Bic) & Windows (Compound) Experimental investigations in real high speed networks are highly required iperf iperf iperf 1 to N (=40) nodes with 1 or 10GbE interfaces Grid5000 backbone 10 GbE iperf iperf iperfd 1 to N ( =40) nodes with 1 or 10GbE interfaces Without reverse traffic: Efficiency, Fairness & equilibrium Around 470Mb/s per flow 9200 Mb/s global With reverse traffic: less efficiency more instability Around 400Mb/s per flow 8200 Mb/s global 19 flows forward /19 flows reverse Affects flow performance & global throughput by 20%, global instability 14

15 Fault Tolerant Open-MPI performance measured on G5K Linpack (mx10g, np=90) Ping pong on various Networks mx10g mx2g mx10g Giga-Ethernet 15

16 Grid 5000 users ~400 active Users Coming from >70 Laboratories. 2DRMP.uk AIST.jp BRGM CEDRAT CERFACS CDS CICT CIRIMAT CS-VU.nl EADS CCR EC-Lyon edam ENS-Lyon ENSEEIHT ENSIACET EPFL.ch FEW-VU.nl France-telecom HPCS.jp I3S IBCP ICPS-Strasbourg IDRIS IECN IMAG IME/USP.br IN2P3 INF/UFRGS.br INP-Toulouse INRIA INSA-Lyon INT IRISA IRIT LAAS LABRI LAGA LBP LEN7 LGCT LICA LIFC LIFL LIFO LINA LIP6 LIPN-Paris XIII LIRIS LMA LORIA LPC LSR LRI Prism-Versailles RENATER SUPELEC UCD.ie UFRJ.br UHP-Nancy UTK.edu 16

17 > 400 Experiments 17

18 >100 Applications Main purpose: Evaluate the performance of applications ported to the Grid, Test alternatives, Design new algorithms and new methods Application domains: Life science (mammogram comparison, protein sequencing, Gene prediction, virtual screening, conformation sampling, etc.) Physics (seismic imaging, parallel solvers, hydrogeology, Self-propelled solids, seismic tomography, etc.) Applied Mathematics (sparse matrix computation, combinatorial Optimization, parallel model checkers, PDE problem solvers, etc.) Chemistry (molecular simulation, estimation of thickness on Thin films), Industrial processes, Financial computing 18

19 >400 Publications Including: -5 HDR -20 Ph.D. >300 international publications 19

20 Resource activity April 06: just before SC 06 March 07 and Grid 06 deadlines Activity > 70% 20

21 Extension to Europe Renater-Geant-Surfnet (a dedicated lambda at 10G) Grid CPUs 2007 (July 27) DAS CPUs 21

22 Recent long range connection Connection of Grid 5000 to the NAREGI R&D Tokyo OS Reconfiguration in both side (first tests on transport Protocols for large data sets) Japan The Netherlands France July 7th, Gbps 22

23 ALADDIN A LArge-scale Distributed and Deployable INfrastructure The ACI GRID stopped in July 2007 Evaluation of Grid 5000 in mars 23rd 2007 by Jean-Claude ANDRÉ (CERFACS) and Domenico LAFORENZA (CNR-ISTI, Italie) --> globally very positive. The evaluation and the community confirms: The significance of the research works and results obtained on the platform, The need to keep the scientific platform up and running. INRIA wishes to pursue research in Computer Science with Grid 5000 by extending its usage to other disciplines in addition to Grid&P2P Service Infrastructures, Networking, Future Internet, etc. INRIA is starting a novel program called ALADDIN (expected duration: 4 years) Dir.: T. Priol, Scientific Dir. F. Cappello, Technical Dir. D. Margery ALADDIN provides human resources (Engineers), coordination and equipment to sustain and strengthen the research activities using the Grid 5000 platform. 23

24 New Challenges 1) Transparent, safe and efficient large scale system utilization and programming. 2) Providing Service Agreement to Users in Large Scale Parallel and Distributed Systems 3) Improve user confidence on the infrastructure 4) Efficient exploitation of highly heterogeneous and hierarchical large-scale systems. 5) Efficient and scalable composition and orchestration of services 6) Model large scale systems and validation of their simulators. 7) Scalable application for large scale systems (algorithm, programming, execution model) 8) Dynamic interconnection of autonomous and heterogeneous resources. 9) Efficiently manage very large volumes of information (search, mining, classification, secure storage and access, etc.) for a wide spectrum of applications areas (web applications, image processing, health, environment, etc.) 24

25 Not yet started but already Popular Thème Site Equipe-projets Thématique NumB Nancy ALGORILLE Algorithmes pour la Grille NumB Lille DOLPHIN Calcul parallèle et distribué à grande échelle NumB Grenoble MESCAL Intergiciel, passage à l échelle NumB Rennes PARIS Programmation des systèmes parallèles et distribués pour la simulation numérique à grande échelle ComA Rocquencourt REGAL Répartition et gestion d'applications à large échelle NumB Lyon RESO Protocoles et logiciels optimisés pour réseaux très haut-débit NumB Bordeaux RUNTIME Supports exécutifs performants pour architectures parallèles NumD? NumB Lyon GRAAL NumB Saclay GRAND- LARGE Sophia- Antipolis NACHOS? Modélisation numérique et calcul intensif pour des problèmes d'évolution en domaines complexes et milieux hétérogènes? ComA Nantes OBASCO Objets, aspects et composants NumB Rennes SAGE Simulations et algorithmes sur des grilles de calcul appliqués à l'environnement notamment Monte- Carlo multiparamétriques ComA Grenoble SARDES Expérimentation des technologies de reconfiguration développées par l équipe-projet SymB Saclay TANC Factorisation des entiers pour casser RSA NumB Bordeaux SCALAPPLIX Schémas et algorithmes hautes performances pour les applications scientifiques complexes ComC Grenoble VASY Vérification ("model checking" et "equivalence checking") distribuées. Bio Rennes VISAGE Expérimentations dans le cadre du projet ANR NeuroLOG Thème Site Equipe-projets Activités en lien avec l instrument ComA Lille Adaptation des applications et des intergiciels dans des environnements répartis multi-échelles. ComD Saclay ALCHEMY Emulation de machines avec un grand nombre de cœurs (1000+) ; simulation d architectures et mise au point de compilateurs SymC Rocquencourt ALPAGE Traitement linguistique de grands corpus textuels. ComB Lyon ARES Architectures de réseaux de services SymC Nantes ATLAS Gestion des données complexes dans les systèmes distribués CogD Rennes BUNRAKU Simulation de foule sur systèmes massivement parallèles SymB Nancy CACAO Factorisation et étude fine de la distribution a grande échelle de l algèbre linéaire (factorisation RSA- 768) NumD Nancy CALVI Simulation pour ITER ComC Saclay COMETE Sécurité SymB Sophia COPRIN Resolution de systèmes, de l'optimisation globale ou du traitement d'inégalités ComB Rennes DISTRIBCOM Expérimentations en services Web et QoS ComB Rocquencourt GANG Expérimentation de systèmes P2P Bio Grenoble HELIX Gridification du logiciel MkDom2; travaux sur la phylogénie CogB Rocquencourt IMEDIA Calcul parallèle de signatures d'images sur de grandes bases, Génération de structure d'index à grande échelle, Classification en temps réel de vecteurs de signatures, Recherche (comparaison de signatures) de copies d'images/vidéo en temps réel ComB Nancy MADYNES Supervision des réseaux et services dynamiques ComB Sophia MASCOTTE Simulation distribuée de grand réseaux P2P NumB Grenoble MOAIS Multi-programmation et Ordonnancement pour les Applications Interactives de Simulation NumD Sophia NACHOS Modélisation numérique et calcul intensif pour des problèmes d'évolution en domaines complexes et 25

26 Questions? Acknowledgements Michel Cosnard,Thierry Priol and Brigitte Plateau Steering committee: Michel Dayde, Fre de ric Desprez, Emmanuel Jeannot, Yvon Jegou, Stéphane Lanteri, Nouredine Melab, Raymond Namyst, Olivier Richard, Pascale Vicat-Blanc Primet and Dany Vandromme, Pierre Neyron 26