#MesosUniversity

Transcription

1

2 Agenda Mesos Frameworks Mesosphere Marathon

3

4 Ressources partagées Master cpu: 2 cpu: 2 cpu: 32 cpu: 4 mem: 2048 mem: 4096 mem: mem: Slave Slave Slave Slave

5 Resources 2 méthodes permettent de décrire les slaves du cluster Resources Tags (ou «Attributes»)

6 Resources Certaines resources types sont pré définies par Mesos cpu mem disk port

7 Resources Certaines resources types sont pré définies par Mesos cpu OBLIGATOIRE! mem disk port

8 Resources Trois différentes types de resources scalars ranges sets

9 Resources cpu:24;mem:24576;disk:409600;ports:[ ];disks:{1,2,3,4,5,6,7,8,9}

10 Resources cpu:24;mem:24576;disk:409600;ports:[ ];disks:{1,2,3,4,5,6,7,8,9} Scalar

11 Resources cpu:24;mem:24576;disk:409600;ports:[ ];disks:{1,2,3,4,5,6,7,8,9} Range

12 Resources cpu:24;mem:24576;disk:409600;ports:[ ];disks:{1,2,3,4,5,6,7,8,9} Set par exemple /mnt/disk/{disknum}

13 Mesos: Web UI

14 Mesos: Web UI

15 Mesos: Web UI

16 Mesos: Web UI

17 Mesos: Web UI

18

19 Framework Framework = Application

20 Frameworks

21 Framework : API

22 API

23 API Scheduler Est notifié par le master de la mise à disposition de ressource Demande l exécution de tâches au master Executor Réalise les taches soumises par le master

24 API Scheduler Mesos (master) Executor register time

25 API Scheduler Mesos (master) Executor register registered time

26 API Scheduler Mesos (master) Executor register registered offer time

27 API Scheduler Mesos (master) Executor register registered offer launch launch time

28 API Scheduler Mesos (master) Executor register registered offer launch launch update update time

29 Framework : API API du Scheduler

30 Scheduler : Lifecycle management Scheduler registered( ) Master Master Standby Slave Executor void registered(schedulerdriver driver, FrameworkID fwkid, MasterInfo masterinfo);

31 Scheduler : Lifecycle management Scheduler disconnected( ) Master Master Standby Slave Executor void disconnected(schedulerdriver driver);

32 Scheduler : Lifecycle management Scheduler reregistered( ) Master Master Slave Executor void reregistered(schedulerdriver driver, MasterInfo masterinfo);

33 Scheduler : Resource Allocation Hadoop Scheduler Cassandra Scheduler resourceoffers( ) Master Master Offre de ressources Standby Slave Executor void resourceoffers(schedulerdriver driver, List<Offer> offers);

34 Scheduler : Resource Allocation Hadoop Scheduler Cassandra Scheduler Le scheduler Cassandra offerrescinded( ) exécute le premier la tâche Master Master Standby Slave Task Cassandra Executor void offerrescinded(schedulerdriver driver, OfferID offerid);

35 Scheduler : Communication Hadoop Scheduler Cassandra Scheduler statusupdate( ) Master Master Standby Slave Task Cassandra Executor void statusupdate(schedulerdriver driver, TaskStatus taskstatus);

36 Scheduler : Communication Hadoop Scheduler Cassandra Scheduler frameworkmessage( ) Master Master Standby Slave Task Cassandra Executor void frameworkmessage(schedulerdriver driver, ExecutorID execid, SlaveID slaveid, byte[] message);

37 Scheduler : Error events Hadoop Scheduler Cassandra Scheduler slavelost( ) Master Master Standby Slave Task Cassandra Executor void slavelost(schedulerdriver driver, SlaveID slaveid);

38 Scheduler : Error events Hadoop Scheduler Cassandra Scheduler executorlost( ) Master Master Standby Slave Task Cassandra Executor void executorlost(schedulerdriver driver,executorid execid, SlaveID slaveid, int status);

39 Scheduler : Error events Scheduler error( ) Master Master Standby Slave Executor void error(schedulerdriver driver, String message);

40 Framework : API API de l Executor

41 Executor : Lifecycle management Scheduler Master Master Standby registered( ) Slave Executor void registered (ExecutorDriver driver, ExecutorInfo execinfo, FrameworkInfo fwkinfo, SlaveInfo slaveinfo);

42 Executor : Lifecycle management Scheduler Master Master Standby disconnected( ) Slave Executor void disconnected(executordriver driver);

43 Executor : Lifecycle management Scheduler Master Master Standby Slave reregistered( ) Executor void reregistered(executordriver driver, SlaveInfo slaveinfo);

44 Scheduler : Task management Cassandra Scheduler Master Master launchtask( ) Standby Slave Task Cassandra Executor void launchtask(executordriver driver, TaskInfo taskinfo);

45 Scheduler : Task management schedulerdriver.killtask( ) Cassandra Scheduler Master Master killtask( ) Standby Slave Task Cassandra Executor void killtask(executordriver driver, TaskID taskid);

46 Scheduler : Communication Cassandra Scheduler Master Master frameworkmessage( ) Standby Slave Task Cassandra Executor void frameworkmessage(executordriver driver, byte[] msg);

47 Mesosphere : Chronos @YesWeScale

48 Mesosphere Florian Leibert Benjamin Hindman Tobi Knaup

49 Mesosphere Our mission is to make building and running distributed systems as easy as building or running an app on your smartphone.

50 Chronos

51 Marathon

52 Quoi? Init Daemon pour Data Center Private PaaS

53 Pourquoi? Simplifier API pour les développeurs Automatiser Améliorer l efficacité

54 Fonctionnalities Déploiement élastique Haute disponibilité Service Discovery Sécurité API REST Système d évènements

55 UI : liste des applications

56 UI : créer une application

57 Cycle de vie Marathon Tâches Évènements / Offres Master Slave Slave Slave Slave

58 Marathon API REST

59 Démarrer une Self Contained App POST /v2/apps { "id": "myrestapi", "instances": 2, "cpus": 0.3, "mem": 128, "ports": [0], "uris": [" RestApi.jar"], "cmd": "java -jar RestApi.jar --server.port=$port0" }

60 Démarrer une Self Contained App POST /v2/apps { "id": "myrestapi", "instances": 2, "cpus": 0.3, "mem": 128, "ports": [0], Slave ports:[0] $PORT0 myrestapi "uris": [" RestApi.jar"], "cmd": "java -jar RestApi.jar --server.port=$port0" }

61 Démarrer une application Dockerisée POST /v2/apps { } "id": "ubuntu", "container": { "type": "DOCKER", "docker": { "image": "libmesos/ubuntu" } }, "instances": 1, "cpus": 0.5, "mem": 512, "uris": [], "cmd": "while sleep 10; do date -u +%T; done"

62 Health check Par défaut, information provenant de Mesos TASK_RUNNING => Healthy Configurable via l API REST de Marathon

63 Health check { } "id": "my-restapi-example-app",... "healthchecks": [ { "protocol": "COMMAND", "command": { "value": "curl -f -X GET }, "graceperiodseconds": 300, "intervalseconds": 5, "timeoutseconds": 3, "maxconsecutivefailures": 3 } ]

64 Health check { } "id": "my-restapi-example-app",... "healthchecks": [ { "protocol": "HTTP", "path": "/simpleapi/info", "graceperiodseconds": 300, "intervalseconds": 5, "timeoutseconds": 3, "maxconsecutivefailures": 3 } ]

65 Les contraintes Contrôler le où lors du déploiement

66 Les contraintes : tags Identifier les ressources par des tags : type de hardware emplacement physique (datacenter, rack) Positionnées au démarrage du slave $ mesos-slave --attributes="rack_id:2"

67 Contrainte : UNIQUE POST /v2/apps { "id": "my-restapi-example-app", "instances": 2, "cpus": 0.3, "mem": 128, "ports": [0], "uris": [" RestApi.jar"], "cmd": "java -jar RestApi.jar --server.port=$port", "constraints": [["hostname","unique"] }

68 Contrainte : UNIQUE POST /v2/apps { Slave 1 Slave 2 UserMgt UserMgt "id": "my-restapi-example-app", "instances": 2, UserMgt CreditApp "cpus": 0.3, "mem": 128, Slave 3 Slave 4 "ports": [0], "uris": [" UserMgt CreditApp RestApi.jar"], "cmd": "java -jar RestApi.jar --server.port=$port", "constraints": [["hostname","unique"] }

69 Contrainte : CLUSTER POST /v2/apps { Slave 1 Slave 2 ssd UserMgt UserMgt "id": "my-restapi-example-app", "instances": 2, UserMgt CreditApp "cpus": 0.3, "mem": 128, "ports": [0], Slave 3 Slave 4 hdd UserMgt UserMgt "uris": [" RestApi.jar"], "cmd": "java -jar RestApi.jar --server.port=$port", "constraints": [["harddrive","cluster", "ssd"] }

70 Contrainte : CLUSTER POST /v2/apps { Slave 1 Slave 2 UserMgt "id": "my-restapi-example-app", "instances": 2, UserMgt CreditApp "cpus": 0.3, "mem": 128, Slave 3 Slave 4 "ports": [0], "uris": [" UserMgt UserMgt CreditApp RestApi.jar"], "cmd": "java -jar RestApi.jar --server.port=$port", "constraints":[["hostname","cluster", slave-3 ] }

71 Contrainte : GROUP_BY POST /v2/apps { Slave 1 Slave 2 Rack 1 UserMgt "id": "my-restapi-example-app", "instances": 2, CreditApp UserMgt "cpus": 0.3, "mem": 128, Slave 3 Slave 4 Rack 2 "ports": [0], "uris": [" UserMgt CreditApp UserMgt RestApi.jar"], "cmd": "java -jar RestApi.jar --server.port=$port", "constraints": [["rack_id","group_by"] }

72 Contrainte : UNLIKE POST /v2/apps { Slave 1 Slave 2 UserMgt UserMgt "id": "my-restapi-example-app", "instances": 2, UserMgt CreditApp "cpus": 0.3, "mem": 128, Slave 3 Slave 4 "ports": [0], "uris": [" UserMgt UserMgt RestApi.jar"], "cmd": "java -jar RestApi.jar --server.port=$port", "constraints": [["hostname","unlike", "slave-[2-3]"] }

73 Contrainte : LIKE POST /v2/apps { Slave 1 Slave 2 UserMgt UserMgt "id": "my-restapi-example-app", "instances": 2, UserMgt CreditApp "cpus": 0.3, "mem": 128, Slave 3 Slave 4 "ports": [0], "uris": [" UserMgt UserMgt RestApi.jar"], "cmd": "java -jar RestApi.jar --server.port=$port", "constraints": [["hostname","like", "slave-[1-3]"] }

74 Application groups Définir des liens de dépendance entre les groupes Scaler tout un groupe à la fois

75 Marathon : Application groups myapp /myapp/bdd /myapp/service postgresql instance=3 apache instance=5 Dépendance redis instance=4 tomcat instance=8

76 Application groups { } "id": "/myapp", "groups": [ { "id": "/myapp/bdd", "apps": [ { "id": "/myapp/postgresql",... }, { "id": "/myapp/redis",... } ] },{ "id": "/myapp/service", "dependencies": ["/myapp/bdd"], "apps": [ { "id": "/myapp/apache",... }, { "id": "/myapp/tomcat",... } ] } ]

77 Service Discovery Où est mon application?

78 HaProxy : reload configuration HaProxy - Marathon bridge (script) Bamboo

79 Service Discovery Marathon Slave 1 Slave 2 HA Proxy Marathon Bridge HA Proxy Marathon Bridge HA Proxy HA Proxy

80 Service Discovery Marathon UserMgt Port : CreditApp Port : Slave 1 Slave 2 UserMgt Port HA Proxy Marathon Bridge HA Proxy Marathon Bridge UserMgt Port UserMgt Port HA Proxy HA Proxy CreditApp Port 31200

81 Service Discovery Marathon UserMgt Port : CreditApp Port : Slave 1 Slave 2 UserMgt Port HA Proxy Marathon Bridge HA Proxy Marathon Bridge UserMgt Port UserMgt Port HA Proxy HA Proxy CreditApp Port 31200

82 Service Discovery Marathon UserMgt Port : CreditApp Port : Slave 1 Slave 2 UserMgt Port HA Proxy Marathon Bridge localhost:10000 { slave1:31100 slave1:31200 slave2:31100 } localhost:20000 { slave2:31200 } HA Proxy Marathon Bridge UserMgt Port UserMgt Port HA Proxy HA Proxy CreditApp Port 31200

83 Service Discovery Marathon UserMgt Port : CreditApp Port : Slave 1 Slave 2 UserMgt Port HA Proxy Marathon Bridge localhost:10000 { slave1:31100 slave1:31200 slave2:31100 } localhost:20000 { slave2:31200 } HA Proxy Marathon Bridge UserMgt Port UserMgt Port HA Proxy HA Proxy CreditApp Port 31200

84 DCOS

85

86 Q @YesWeScale