DevOpsDays Paris 18-19 avril 2013 G. Bloquel S. Révéreault www.groupe-sii.com 02/07/2013 Page 1
Introduction DevOpsDays : Série d événements dans des grandes villes internationales Modèle unique : conférences, «ignite talks», Open Spaces (+ demos) Co-organisation avec Patrick Debois Paris : 1 er événement français Sponsors : Page 2
Conf #1 : CustomerOps: a culture of visibility and metrics applied to customer support Alexis Le-Quoc - @alq - Co-founder and CTO of Datadog Monitoring Infrastructure As A Service What we do : write software What we sell : service No marketing, just word of mouth Quality for us = Quality of product + quality of support DevOps engineers supported by DevOps engineers Culture engineering culture == build && understand answer every single question, give as much insight as possible "is the problem solved?", "Does the answer make sense?" Every engineer does customer support 1 week at a time. => we learn Split roles in support : interrupt driven / problem solve Page 3
Conf #1 : CustomerOps: a culture of visibility and metrics applied to customer support Automation : Identify channels for answering questions : FAQ, IRC, in-product chat, email,... Desk, #freenode, olark reaching out individuals : auto-email (emails based on what you're doing) Intercom.io, customer.io broadcasting : status page, twitter Stashboard + gae, twitter sharing: code, API Docs Github, jekyll + S3 observing : analytics kissmetrics Reactive + Proactive approach Page 4
Conf #1 : CustomerOps: a culture of visibility and metrics applied to customer support Measurement : Monthly metrics Sharing : New cases: Quality Reopened case ratio: Quality Time 1 st response : Engagement Time to resolution : Engagement 1 st contact resolution %: Engagement Volume per channel : Productivity Impact of support on sales : Sales Contribution aux DevOpsDays. Questions ouvertes : What can we improve? How can I talk to my customers? Page 5
Conf #1 : CustomerOps: a culture of visibility and metrics applied to customer support Q/R : How do you prevent your support team from optimizing metrics? We don't really have the problem. Would this approach work for "normal" end users (not technical ones)? I think that still applies, filters might be different. Links : Description : http://www.devopsdays.org/events/2013- paris/proposals/customerops/ Slides : https://speakerdeck.com/alq/customerops Vidéo : http://vimeo.com/66153903 Twitter : @alq Page 6
Conf #2 : How ops improved my dev Florian Gilcher - @agorak - Ruby Dev, consultant When I started : developping on a LAMP stack, with a strict Dev/Ops separation. => It wasn't even that bad until projects got special One of the most efficient teams was a dev, an admin and a cup of coffee. «git push developer mindset/devops» Explications du fonctionnement autour d'un outil de récupération et d'encodage de vidéos (architecture, problèmes, changements). Infrastructure as code, mais aussi code for infrastructure. Importance des métriques et des logs dans les systèmes complexes la plupart des équipes de dev l'implémentent trop tard, alors que ça doit être une des premières choses faites. Page 7
Conf #2 : How ops improved my dev Importance pas des moindres, l utilisation d outils interne peut nous faire économiser beaucoup de temps Platform refactorings Code reviews Pas assez de gens sont qualifiés pour en faire Moins de permission, plus de choses serait faite Elargisser la connaissance dans vos équipes Conclusion Les equipes avec une culture «devops» Peuvent s occuper de problème plus complexe Peuvent trouver des solutions alternatives à des problèmes Page 8
Conf #2 : How ops improved my dev Quelques citations : «Devops-minded teams can cope with missing team members easier» «My ugliest piece of code ran 1,5 years in production without a change» «The Devops mindset takes away a lot of friction» Links : Description : http://www.devopsdays.org/events/2013- paris/proposals/developmentgainsthroughops/ Slides : https://speakerdeck.com/skade/how-devops-improved-my-dev Vidéo : http://vimeo.com/66153905 Twitter : @agorak Page 9
Conf #3 : Product or Project? Rémy-Christophe Schermesser, @el_picador, Octo Technology What is a project A project is a set of activities and actions with a specific need in specified time and a budget What is a product A product is a result a creative human activity Spot the differences between product and project, and the IT involvement. Project is composed action, activity, specific need, time, budget Product is composed creative activity, satify need, client Project fail, product succeed, how to make projects succeed like products? Don't forget product team are users too, and they have good ideas. Page 10
Conf #3 : Product or Project? Team project Business analyst (MOA) Project manager Team product Developper Marketing Designer Product Manager Give responsability Page 11
Conf #3 : Product or Project? What makes a product succeed? Measure Learn Build : Measure Make the Minimum Viable Product (MVP), and go to production ASAP Help end users to change (for instance, quick tutorials). Voir chardin.js => librairie javascript pour faire du micro-tutoriel. (Tools base sur Jquery permettant d ajouter une légende par superposition) Users lie to survey Build things and then ask people for their feedback (Ford : Si j'avais demandé aux gens ce qu'ils voulaient, ils m'auraient dit "des chevaux plus rapides"). Get out of the building Build Page 12
Conf #3 : Product or Project? Learn : Fail to learn Think product and do projects. Links : Description : http://devopsdays.org/events/2013- paris/proposals/produitouprojet/ Slides : https://speakerdeck.com/elpicador/how-products-can-improve-projects Vidéo : http://vimeo.com/66153906 Twitter : @el_picador Page 13
Conf #4 : Transforming devs into devops Fabrice Bernhard, Fondateur et Directeur Technique Theodo Why do you need DevOps? Business needs velocity. V-cycle : time consuming, tunnel effect. Slower and Riskier than you can estimate (voir étude 2 profs Oxford sur la réussite des projets de développement) 1 projet / 6 dépasse le cout attendu par 3 Our brain sees the world through Bell Curves (livre : the black swan), but complex systems do not act like bell curves Small iterations are important (Divide and Conquer). Lean startup Devops => DevOps is the result of lean thinking Page 14
Conf #4 : Transforming devs into devops Etapes : 1) Dev to DevOps friendly 2) DevOps friendly to DevOps Dev to Devops friendly Typical junior dev knows nothing about Ops Learnt java at university Uses windows because of the games Has already installed linux because he's curious 1 st step: improove developpers Developping under Linux Git versionning Correct git branching Scrum method Unit and functionnal tests Bring Devs to DevOps : Shell scripting? forget it Capistrano? too magic Fabric? good compromise Page 15
Conf #4 : Transforming devs into devops => Every project has a deploy.py script in a "devops" folder. Give the chance for Devs to play with a server Acquire experience, need sandbox dev, simple to reset IaaS for dev sandbox OK for dev, less for testing : clients complained that the testing platform was unstable Scripted provisioning must stay simple : Dev need to install packages and modify config files on their server Chef-server / puppet-server : designed for clusters Chef-solo : a "who wirites the most magical ruby" contest Puppet/Chef modules : too much abstraction, not always work "out of the box" Fabtools : easy and developer-friendly Puppet apply without modules (currently testing) Use Vagrant to create the dev environment and use chef / puppet Page 16
Conf #4 : Transforming devs into devops Devops friendly to DevOps Devs + ops tools = devops? no Deep cultural differences between dev and ops Pair devopsing as an efficient teaching solution sysadmin should always pair with a dev when working on a project scrum compatible hire devs that are eager to learn stuff Issues and constraints for developing fast TDD Performance Provisioning Scaling Backups How do you make devs focus on performance? Make it part of the DONE definition Include performance solutions in the standard provisioning (for instance use Varnish in dev if you use it in prod) AND do visible performance graphs Page 17
Conf #4 : Transforming devs into devops Make scaling cool and intellectually challenging The power of NoSQL : devs have to think the model in a scalable way IaaS is and infrastructure with an API... cool! How do you bring devs to feel responsible for production (http://xkcd.com/705/ ) Why do ops care, and not dev : not our job What about backup and server monitoring? SaaS (Newrelic, Pingdom, Idera ServerBackup) for bigger needs, you need a sysadmin Links : Description : http://www.devopsdays.org/events/2013- paris/proposals/transformerdesdevsendevops/ Slides : 404 Not Found Vidéo : http://vimeo.com/66153907 Twitter : @theodo Page 18
Ignite #1 : DevOps Productivity Survey 2013 (RebelLabs) http://vimeo.com/66153908 http://zeroturnaround.com/labs/rebel-labs-release-it-ops-devops-productivity-report- 2013/ DevOps est mieux que l IT traditionnelle (métriques à l appui) Performance perpétuelle Octo http://vimeo.com/66155658 http://fr.slideshare.net/henrit/perf-devops-day Pourquoi mesurer la performance en continu? Deployer en continu des VMs Test de charge (Gatling) Casser le build si performance KO Page 19
Ignite #1 : Lost paradise of DevOps (Serena) http://vimeo.com/66154532 DevOps is not new! Comparaisons «bibliques» : la tour de Babel, le miracle de la mer rouge, Project Raindrops http://vimeo.com/66154530 http://projectraindrops.net/login Service de création automatique d'images de VMs pour différentes infra (Aws, vagrant) What if DevOps was invented by Coca Cola? P. Debois http://vimeo.com/66154531 http://fr.slideshare.net/devopsdays/what-if-devops-was-invented-by-coca-cola «Don t take devops too serious, use with caution» Page 20
Open Spaces Continuous Integration for infrastructure Quels outils? Jenkins, mais est-il bien adapté Puppet, Chef, CFEngine, Packaging : RPM, Deb, MSI, Jar/War, Tar.gz, Que doit-on emprunter aux méthodos de développement? Tests unitaires sur les recettes de déploiement Smoke test Factorisation du code (exemple : Specs RPM) Conclusion : L outillage n est pas mûr Méthodos encore obscures pour la partie infra Page 21
Open Spaces Introducing DevOps in a traditionnal organisation / Tips and tricks to spread culture Par où commencer : Trouver les leviers (douleurs et comment DevOps peut les résoudre) Trouver les personnes de bonne volonté Fixer les objectifs Projets pilotes Quelques trucs : Faire travailler Dev et Ops sur des douleurs communes Logs On duty call Attention à ne pas établir une relation client / fournisseur entre Dev et Ops Faire contribuer les Ops à la création de la liste de métriques Page 22
Démos jrebel / LiveRebel jrebel : voir les changements en temps réel sur du code développé en Java LiveRebel : gestion de configuration et de déploiement CFEngine Un agent prend la main sur le port en écoute de Tomcat En cas de déploiement progressif, l agent va se charger des redirections éventuelles de trafic Démo de suppression d un élément et reconfiguration automatique de la plateforme. Page 23
Conf #5 : How we release software for GOV.UK Kushal Pisavadia - GDS Présentation de GDS (sous-organisation de "Cabinet Office", service public Anglais). But : "Digital services so good that people prefer to use them". Focus on user needs Ship Fast Measure everything Manifeste: "Revolution, not Evolution" Build a centre of excellence Fix publishing Fix transactions Go wholesale Gov.uk : site d'information sur l'activité du gouvernement britanique "Nobody ever got fired for choosing open source Le site est publié sur Github : github.com/alphagov Page 24
Conf #5 : How we release software for GOV.UK Outils utilisés pour le déploiement : Capistrano au départ Ensuite Jenkins Commit to master, kick off a build, test, deploy Build screen Un bel exemple de User Story pour décrire une exigence de performance Configuration management testés : Opscode PalletOps Puppet => choisi (un blog post décrit la comparaison) We gave all the developers sudo access (we were working on a beta). Page 25
Conf #5 : How we release software for GOV.UK Toutes les pages d'info peuvent être récupérées en json. 1-click deploy : utilisation d'un "gros bouton" pour symboliser la MEP. Simplify with abstractions (exemple : vous connaissez le script, mais vous ne connaissez pas la physique quantique. Pourtant, vos scripts sont exécutés par des ordinateurs qui utilisent des transistors qui s'appuient sur de la mécanique quantique). Getting ready for the October release => being the home page of a nation. Enforced rotation of developers Mise en place d'une application pour gérer les modifications Page 26
Conf #5 : How we release software for GOV.UK Retour sur la faille Ruby on Rails de janvier => application patchée en 1h30 (nobody was "on call"... they just cared). https://www.gov.uk/service-manual/operations/devops.html 4 hours between code change and production (including manual validation by product team) 15 to 20 deploys a day Links : Description : http://www.devopsdays.org/events/2013- paris/proposals/howwereleasesoftwareforgov.uk/ Slides : 404 Not Found Vidéo : Bientôt sur http://vimeo.com/user9086015 Twitter : @KushalP Page 27
Conf #6 : Le territoire et la carte - une histoire de visibilité Pierre-Yves Ritschard, @Pyr, developeur OpenBSD Visibility. Design, Building, but mostly corrections => How to improve? Break Silo Approach: 30% dev and 70% ops Agile Approach 70% build et 30% correctino We want lower defect rates. We want to make informed decisions. => Visibility : extracting meaningful state data from heterogenous event sources, over time - Meaningful (toward business) - State data - Heterogeneous (everyone is involved in creating data) - Over time Page 28
Conf #6 : Le territoire et la carte - une histoire de visibilité How does it help my system's life cycle. "The map is not the territory." Break out of our mental model. Less maps, more territory... or at least better maps. Systems are increasingly complex. En 2000 : LAMP (2 serveurs). Visibility : MRTG graphs En 2012 : HA Proxy, NGinx, Cassandra, Redis, MySQL,... (27 serveurs). Visibility : we're still looking at the same metrics... WRONG! Figure out key metrics : We need appropriate tooling Metrics & Events : correlation accross : system, components, software Page 29
Conf #6 : Le territoire et la carte - une histoire de visibilité The event stream approach : Plenty of small producers, and few big consumers Anything that happen or moves : normalize & stream Aggregate : compute compound metrics Correlate : find trends Decide : track, alert, ignore, scale Implementing : on premise, saas, or in between? SaaS : Loggly, papertrail, librato, datadog,... On Premise : collectd, logstash, graphite, statsd, riemann Page 30
Conf #6 : Le territoire et la carte - une histoire de visibilité The path to visibility : Find key metrics Find the right tools Rely on an event stream Involve everyone Challenge your mental model Hopefully improves quality and lower perfect rates in the process Links : Description : http://www.devopsdays.org/events/2013- paris/proposals/territoirecarte/ Slides : https://speakerdeck.com/pyr/map-and-territory-a-story-of-visibility Vidéo : Bientôt sur http://vimeo.com/user9086015 Twitter : @Pyr Page 31
Conf #7 : The 10 major traps to avoid in order to succeed your DevOps transition Alain Delafosse, Consultant Objet Direct et Directeur Technique KelKoo Eric Mattern,Manager Open Source Objet Direct Trap #1 : Think it's only tooling Tools, Processes, culture Trap #2 : Start with the wrong tooling strategy Build your own solution? Pick up a deployment tool? Appliance images? Adopt PaaS approach? Cloner Configurer Developper Packager Configurer Deployer Infra / OS Applications Page 32
Conf #7 : The 10 major traps to avoid in order to succeed your DevOps transition Trap #3 : Think only server deployment Configuration management (OS & software configuration items) Software deployment (+ Software components & middleware) Platform deployment (+ Infra config & provisioning, Deployment strategy) Trap #4 : Trying to fully automate data & database upgrades Tricky to automate Try to uncouple database changes from software upgrades Defer database structure change New database schema NOSQL database schemaless Page 33
Conf #7 : The 10 major traps to avoid in order to succeed your DevOps transition Trap #5 : Rebranding teams/persons as DevOps Don't Do rename teams setting DevOps as title or function create a new team Forget the name empower your teams to foster the transition Trap #6 : Underevaluate the cultural shift Collaboration Long and difficult journey DevOps cultural aspects : sense of continuous improvement, collaboration level, transparency Re-use Agile transitions practices Page 34
Conf #7 : The 10 major traps to avoid in order to succeed your DevOps transition Trap #7 : If you don't measure it, you can't value it Define goals -> Find metrics -> Expose KPIs Contextualize metrics to your goal and context Evaluate progression & share the success Deployement rate and delay Release quality, number of rollback Effort and delay for conf update Trap #8 : Forget continuous improvement loops DevOps is a endless journey Setup a continuous improvement loop : tools and processes Improve implementation and deploy Capture feedbacks Backlog Milestone and priority Formalize and categories Page 35
Conf #7 : The 10 major traps to avoid in order to succeed your DevOps transition Trap #9 : Focus only on deployment process There are other processes / areas where collaboration between dev and ops make sense. Monitoring Common dashboard using dev, ops and BIZ Log management Performance management Integration continous Incident / problem management BCP (Business Continuity) / DRP (Disaster Recovery) design Security management Trap #10 : Conflict with ITIL Some production environments do require strong service management DevOps doesn't enforce a specific organization of processes DevOps will seed the ITIL processes by bringing new tools and new practices Change management Page 36
Conf #7 : The 10 major traps to avoid in order to succeed your DevOps transition Links : Description : http://www.devopsdays.org/events/2013- paris/proposals/les%2010%20pie%cc%80ges%20principaux%20a%20e%cc%81v iter%20pour%20re%cc%81ussir%20votre%20transition%20devops/ Slides : 404 Not Found Vidéo : Bientôt sur http://vimeo.com/user9086015 Twitter : @AlainDelafosse Page 37
Ignite #2 Testing. When working with puppet, a lot of manual testing => unit tests on Puppet A voir : Fizzgig (librairie de test de recettes Puppet). Managing Complexity (Diego Zamboni - CFEngine) Abstraction, Perspective. Our tools are not ready, but we are working on it. Aeolus (R. Di Cosmo - Inria) Deployment plan on a Cloud. Zephyrus : an automatic tool for creating a full system of configurations. Page 38
Open Space DevOps coaching : Tips : Scrum, Scrum de scrum Le Scrum Master doit être responsable de l implication des ops Trouver les leviers, trouver les points de motivation Charte d utilisation des métriques Attention à la limite coaching / audit Equipes pilote, évangélisation par les pilotes Constat : On manque de méthodologie S appuyer sur agile, scrum Page 39
DEMO #2 Ruddler GUI Tool using CFEngine No need to write own files use generic template through GUI Rollback possible thanks to historic configuration Page 40
Conclusion Beaucoup d échanges autour de la culture Une vraie effervescence Intérêt des grands acteurs L outillage est incomplet Adapter les concepts Page 41