XtremWeb-HEP 8.0.0 Interconnecting jobs over DG Virtualization over DG Oleg Lodygensky
Objectives 1.Deploy Virtual Machines in XtremWeb-HEP desktop grid to: protect volunteer resources generalize «pilot job» to interconnect grids deploy complex applications 2.Internode communications to: deploy client/server jobs over DG propose an interactive DG This does not make DG a cloud http://www.xtremweb-hep.org 2
State of the Art Desktop Grids using VM Submit/ retrieve Download/ update Distributed User monitor the platform upload applications jobs, datas.. download job results Distributed ressource (volunteer PC) download jobs, applications, data compute jobs upload results run end user jobs in a VM use a preinstalled VM http://www.xtremweb-hep.org run jobs in CernVM use a preinstalled CernVM 5
VM over StratusLab proposes distributions with the IaaS paradigms. Centralized and securized services store applications and data scheduler repository etc. Distributed ressource may declare the virtualbox application. Distributed User submit a virtualbox job with a virtual disk Distributed ressource (volunteer PC) download the virtual disk create and run a new VM inside its local VirtualBox http://www.xtremweb-hep.org 4
VM : Contextualization It is not simple to create a virtual disk (VD) containing a virtualized OS. we need a repository of «template» VD to ease the user life any user can still create and use its own VD any VD can be contextualized contextualization VD if exists, mounted on /context if exists, /context/context.sh executed at the end of boot process /dev/hdd /dev/hda /dev/hdb bootable VD typically a live CD (350Mb) created on the fly by the (=> not downloaded) mounted on /scratch typically 8Gb this is a variable size file => upload size then depends on result size (max: 8Gb - hopefully less) http://www.xtremweb-hep.org 5
VM : Use Case HEP applications are linked to ROOT (http://root.cern.ch) DG resources don t have ROOT submitted jobs will not run Deploy a VM with ROOT and Submit such VM on the fly the native shares VirtualBox the native the VM the virtualized shares ROOT the virtualized run HEP jobs http://www.xtremweb-hep.org 6
BitDew and The BitDew P2P network places the virtual disk to the expected s. http://www.xtremweb-hep.org 7
Internode Communications Interconnect jobs and applications over DG Centralized and securized services Submit/ monitor Download job/ upload results SmartSockets Hub client A communicating application untouched code reconfigured only User PC (3) connect A communicating job untouched code reconfigured only (3) A communicating job untouched code reconfigured only Bypassing http://www.xtremweb-hep.org FireWalls and NAT (3) 8
Internode Communications Example 1 : connect client to a job running on volunteer resource. E.G. : deploying jackrabbit over DG (http://jackrabbit.apache.org) Centralized and securized services SmartSockets Hub Submit/ monitor Download job/ upload results client (3) connect User PC http://www.xtremweb-hep.org 9
Internode Communications Example 2 : connect a job running on volunteer resource to a service E.G. : deploy wget and connect to a web server running on client Centralized and securized services SmartSockets Hub Submit/ monitor Download job/ upload results client (3) connect User PC http://www.xtremweb-hep.org 10
Internode Communications Example 3 : interconnect jobs running on volunteer resources E.G. : interconnect wget and jackrabbit running on volunteer resources Centralized and securized services SmartSockets Hub Submit/ monitor Download job/ upload results client User PC (3) connect http://www.xtremweb-hep.org 11
Internode Communications Example 4 : connect client to a job running on volunteer resource. E.G. : ssh to a running VM Centralized and securized services SmartSockets Hub Submit/ monitor Download job/ upload results client (3) connect ssh User PC http://www.xtremweb-hep.org 12