MÉTHODES DE KRYLOV POUR LES PROBLÈMES DE RECHERCHE DE VALEURS PROPRES ADAPTÉES AU PETASCALE.

Transcription

1 MÉTHODES DE KRYLOV POUR LES PROBLÈMES DE RECHERCHE DE VALEURS PROPRES ADAPTÉES AU PETASCALE. C. CALVIN 1, F. BOILLOD-CERNEUX 1,2, T. DRUMMOND 4, J. DUBOIS 1, N. EMAD 3,5,S. PETITON 2,3, F. YE 1,3 1 CEA Saclay France 2 CNRS/LIFL Lille France 3 Maison de la Simulation Saclay France 4 LBNL Berkeley US 5 PRISM UVSQ - France SEMINAIRE MANON Christophe CALVIN christophe.calvin@cea.fr SÉMINAIRE MANON 29 MARS 2013 INSTN - SACLAY CEA 10 AVRIL 2012 PAGE 1

2 INTRODUCTION CEA 10 AVRIL 2012 PAGE 2 26 MARS 2013

3 SUPERCOMPUTING TRENDS Petascale and post-petascale The degree of parallelism is increasing The number of levels of coarse and fine parallelism is increasing The number of levels of memory is increasing Heterogeneous processing units (with CPUs + accelerators) New algorithms and programming paradigms TOP # of cores #1 # Moyenne MARS 2013 CEA FEB PAGE 3

4 SUPERCOMPUTING TRENDS Main Challenges for Exascale: Power (Energy Efficiency Computing) Low power processor (ARM) Maximize Flops/watt (GPU, MIC) Minimize the memory per core Communication Minimized global communication Optimize communication GBytes/core #1TOP 500 Fault tolerance MTBF <1h 5 4,5 4 3,5 3 2,5 2 1,5 1 0, MARS CEA FEB PAGE 4

5 IMPACTS SUR LES APPLICATIONS SCIENTIFIQUES ET LES ALGORITHMES NUMÉRIQUES POUR L EXTREME SCALE Nécessité de repenser les algorithmes et les architectures logicielles Minimisation : De la consommation énergétique Des communications globales dont les synchronisations Maximiser le degré de parallélisme des algorithmes et leur vectorisation (SIMD) Multi / hyper threading - 1 à 2 threads par cœur de calcul sur SandyBridge, IvyBridge, Hazwell quelques 10 aines de threads par sockets - 4 threads par cœur de MIC 244 threads - 1 thread par cœur de GPU > 1500 threads vectorisation (SIMD) - AVX1, AVX2, AVX3 26 MARS 2013 CEA MARCH 2013 PAGE 5

6 IMPACTS SUR LES APPLICATIONS SCIENTIFIQUES ET LES ALGORITHMES NUMÉRIQUES POUR L EXTREME SCALE Nécessité de concevoir des nouvelles architectures de codes scientifiques et d algorithmes numériques pour prendre en compte ces modifications matérielles : Algorithmes polymorphiques (s adaptant au degré de parallélisme), Utilisation de composants optimisés (comme les bib. numériques) Auto tuning / smart tuning Prise en compte de la tolérance aux pannes au niveau de l application et au niveau des algorithmes Nécessité d avoir des runtime performants pour la gestion dynamique du multithreading StarPU 26 MARS 2013 CEA MARCH 2013 PAGE 6

7 ALGORITHMIQUE NUMÉRIQUE POUR L EXTREME SCALE Nos axes de R&D Co-méthodes Auto-tuning et smart-tuning Programmation efficace des multi-cœurs et accélérateurs Evaluation des modèles et langages de programmation Notre terrain d application Méthodes de Krylov pour la résolution de problèmes aux valeurs propres 26 MARS 2013 CEA MARCH 2013 PAGE 7

8 PROBLÈMES AUX VALEURS PROPRES Problème récurrent dans de nombreuses applications Page ranking Equation de Boltzmann Forage pétrolier Av Bv Au u 26 MARS 2013 Modeling of Epidemic Spread CEA MARCH 2013 PAGE 8

9 PROBLÈMES AUX VALEURS PROPRES De nombreuses méthodes Méthodes directes : tri-diagonalisation suivie de QR. Méthodes semi-directes : Lanczos Méthodes itératives : Méthode de la puissance Méthodes de Krylov (basée sur Arnoldi) - IRAM (Implicit Restarted Arnoldi Method) - ERAM (Explicit Restarted Arnoldi Method) 26 MARS 2013 CEA MARCH 2013 PAGE 9

10 KRYLOV SOLVER FOR THE NEUTRON TRANSPORT EQUATION Example of different solvers for the neutron transport equation with DENOVO/CASL MARCH 26, 2013 CEA MARCH 2013 PAGE 10

11 EXPLICITELY RESTARTED ARNOLDI METHOD Krylov Method for non symmetric eigenvalue problems MARCH 26, 2013 CEA FEB PAGE 11

12 ERAM EN PARALLÈLE SUR DES ARCHITECTURES MASSIVEMENT PARALLÈLES ET HÉTÉROGÈNES CEA 10 AVRIL 2012 PAGE MARS 2013

13 PARALLEL ERAM Two levels parallelization Intra-node: multithreaded implementation based on BLAS kernels On CPU: multi-threaded BLAS through MKL On GPU: CUBLAS Inter-node: using MPI Matrix decomposed using a chess pattern in order to minimize communication times #MPI process #rows x columns 1x1 2x1 3x1 2x2 5x1 3x2 7x1 4x2 3x3 5x2 11x1 4x3 13x1 7x2 5x3 4x4 Perfomance study Weak and strong scaling On CPUs and GPUs MARCH 26, 2013 CEA FEB 2013 PAGE 13

14 TEST MACHINE CURIE MACHINE One of the European PRACE supercomputer located at TGCC CURIE thin nodes: 5040 nodes B510 bullx, - Each node: 2 8-cores Intel Sandy Bridge EP (E5-2680) 2.7 GHz 64 GB 80,640 cores 1.6 Pflops CURIE fat nodes: 360 nodes S6010 bullx, - Each node: 4 8-cores Intel Nehalem-EX(X7560) 2.26 GHz 128 GB 11,520 cores 105 TFlops CURIE hybrid nodes: 16 nodes B505 bullx - Each node: 9 hybrid lames B505 with 2 Intel Westmere 2.66 GHz/ 2 Nvidia M processors Intel processors Nvidia. 192 TFlops. 26 MARS 2013 CEA FEB 2013 PAGE 14

15 WEAK SCALING Dense square matrix N=26,000 6, elements 8 eigenpairs Krylov subspace size=16, e= MARS 2013 CEA 10 AVRIL 2012 PAGE 15

16 WEAK SCALING ANALYSIS T T exec Communication time model: N _ mvp ( p, N) 2 b PCIe 2 log t PCIe Where: comm p: number of GPUs N: vol. of data to transfer (Matrix order/ p) b PCIe = PCIe latency (around 100ns) b MPI = QDR/MPI latency around1 ms t PCIe = PCI-express bandwidth= 1 GB/s t MPI = QDR/MPI bandwidth = 6 GB/s ( p, N) T T _ ( p, N) comp comm mvp N p b MPI t MPI 0,001 0,0008 0,0006 0,0004 0, total transfer time (6GB/s) gpu transfer time Communication for the Matrix-Vector product MARCH 26, 2013 CEA FEB 2013 PAGE 16

17 STRONG SCALING Dense square matrix N=22, elements 135 eigenpairs Krylov subspace size=225, e=10-8 MARCH 26, 2013 CEA FEB 2013 PAGE 17

18 STRONG SCALING ANALYSIS For strong scaling, 2 parameters are involved for performances Computing performances which decreases due to decreasing amount of data Communication ratio which increases as the computing time is decreasing Performance evolution on a single GPU depending on the matrix size Three main phases: 1. Exec. Time decreases : Comp. Time is greater than Comm. Time 2. Optimum point. 3. Exec Time increases: Comm. Time greater than Comp. Time MARCH 26, 2013 CEA FEB 2013 PAGE 18

19 STRONG SCALING ON CPU Fat Nodes Thin Nodes 26 MARS 2013 CEA FEB 2013 PAGE 19

20 PERFORMANCE ON CPU AND GPUS Partition Execution time(s) Perf. reached / Perf. max = % # of nodes # of cores or GPUs Fat 4, / 302 = 56% Thin 3, / 437 = 49% Hybrid 2, / 900 = 34% MARS 2013 CEA FEB 2013 PAGE 20

21 LES CO-MÉTHODES : MERAM CEA 10 AVRIL 2012 PAGE MARS 2013

22 MERAM Principle Co-Method based on ERAM Launch several ERAM with different parameters Exchange information in order to improve each ERAM convergence Parameter Set: Krylov subspace size Restarting strategy # desired EigenPairs MERAM ERAM (Parameter Set 1) ERAM (Parameter Set 2) ERAM (Parameter Set 3) CEA JUNE 2012 PAGE 22

23 MERAM Well adapted for Exascale Smaller scalar product MERAM ERAM (Parameter Set 1) ERAM (Parameter Set 2) ERAM (Parameter Set 3) Fault Tolerance CEA JUNE 201 PAGE 23

24 RESTARTING STRATEGIES : ERAM General Formula : Σα i φi Y. Saad Default αi = 1 Linear αi = i Proposed Restarting Strategy Idea : Using Ritz values Information in restart coefficients Residual αi = i. 1 - Ri Lambda αi = λi λi-1 Linear Residual αi = i. 1 - Ri Lambda Residual αi = λi. 1 - Ri CEA MARCH 2013 PAGE 24

25 MERAM ALGORITHM ERAM 1 Controler ERAM n Initialization Arnoldi Reduction Recept & order Restart Parameters Initialization Arnoldi Reduction QR Test Reception Send Restart Parameters Received results are better than mine QR Compute restarting vectors Sending restarting vectors Recept Restart Vectors Select & Send best Restart Vectors CEA MARCH 2013 PAGE 25 Test Reception Complete restarting vector with my data

26 Temps d exécution en secondes SCALABILITÉ DE MERAM SUR TITANE ,8 MERAM avec trois solveurs ERAM ,33 111, ,9 5 8,75 5,5 5,95 4, Nombre de nœuds à 8 cœurs utilisés (1184 pour 148 nœuds) 26 CEA MARCH 2013

27 Execution time (seconds) MERAM VS ERAM x x 7x x x 31x x x ERAM MERAM xquadcore nodes used 27 CEA MARCH 2013 PAGE 27

28 AUTO-TUNING ET SMART-TUNING CEA 10 AVRIL 2012 PAGE MARS 2013

29 AUTO-TUNED MATRIX-VECTOR PRODUCT Hardware Nehalem Tesla Max perf GFlops 37.5 GFlops Why to auto-tune the matrix-vector product? Performance for GPU sparse computations : - Depends on the matrix data pattern - Paper from Bell&Graham shows improvement in performance of 10-15x! - Hardware used: memory controller or caches - How to exploit efficiently the GPU computing power? Restarted Arnoldi Iterative method: matrix is fixed for all iterations CSR, CSC, COO, ELLPACK, Hybrid (ELL+COO). Our strategy: - Before starting the solver, we micro-benchmark the matrixvector product using different matrix formats - We run several times the matrix vector product, following the well-known scheme A ( A.x + x ) + x - The benchmarking is limited for each format either by the number of runs, or a time limit - We control the authorized overhead of the micro-benchmarking MARCH 26, 2013 CEA MARCH 2013 PAGE 29 Speed-up reference ~3x

30 AUTO-TUNED MATRIX-VECTOR PRODUCT Experiments on the MVP Matrix: nlpkkt80 from university of Florida sparse matrices collection 1,062,400 order matrix with 28,192,672 non zeros Format Perf. (Gflops) CSR scalar CSR vector COO ELL HYB Speed-up Ref. 3.25x 1.75x 8.5x 8.5x Improvement for the whole computation Restarted Arnoldi on multiple processors or GPUs with Nlpkkt80 (1 M order, 28 Mnnz) MARCH 26, 2013 CEA MARCH 2013 PAGE 30

31 SMART-TUNING APPLIED TO ERAM/MERAM Krylov Method for non symmetric eigenvalue problems MARCH 26, 2013 CEA MARCH 2013 PAGE 31

32 SMART-TUNING APPLIED TO ERAM/MERAM Limitations Restarting vector computation: Critical step The restarting vector may disrupt ERAM convergence There is no universal method to choose an efficient restarting scheme Subspace size: influences convergence Subspace size too small: no convergence Optimal subspace size depends on matrix, desired eigenpairs, How to find: Good restarting coefficients? Optimal subspace? MARCH 26, 2013 CEA MARCH 2013 PAGE 32

33 SMART-TUNING APPLIED TO ERAM/MERAM Basically, we use the convergence rate CR as a measure. There are several parameters for the auto tuner: max_idle_time: iterations during which auto tuner did nothing CRmin, CRmax: min and maximum CR Experiments conducted on: a laptop: 2 cores (4 threads total) Two testing phases: Phase I: Testing ERAM subspace autotuning autotuning for same RS Same autotuning for RS Phase II: Testing MERAM subspace autotuning Same autotuning for each ERAM, RS. autotuning for each ERAM, RS. CEA MARCH 2013 PAGE 33

34 COMPARING ERAM W/ AND W/O AUTOTUNING Matrix CRY10000 Desired eigenpairs: 20 Tolerance: 10 e-09 Autotune: Step = 5, Idle = 15, Avg = 5 Subspace size With AT (MVs) Without AT (MVs) ERAM(20) (min) - Def Lin Res Def. No conv - Lin. No conv - Res. No conv ERAM(75) Res Res ERAM(85) Res. 675 Res. 680 AT ERAM(20) vs ERAM(75) 3,75x AT ERAM(75) vs AT ERAM(20) 2x AT ERAM(75) vs ERAM(75) 7,5x AT ERAM(85) vs AT ERAM(20) 7,5x AT ERAM(85) vs ERAM(85) similar CEA MARCH 2013 PAGE 34

35 SMART-TUNING APPLIED TO ERAM/MERAM Smart-tuning on restarting strategies MARCH 26, 2013 CEA MARCH 2013 PAGE 35

36 #Iterations SMART-TUNING ON RS WITH MERAM Matrix CRY10000 MERAM / ERAM # Iterations DIV DIV ERAM which cv first 0 CEA MARCH 2013 PAGE 36

37 PAGE 37 CEA 10 AVRIL 2012 Commissariat à l énergie atomique et aux énergies alternatives Centre de Saclay Gif-sur-Yvette Cedex T. +33 (0) F. +33 (0) DEN/DANS DM2S 26 MARS 2013 Etablissement public à caractère industriel et commercial RCS Paris B