Software Testing - General Introduction

Transcription

1 Software Testing General Introduction Benoit Combemale 1

2 WHAT ARE WE LOOKING FOR? 2

3 We look for bugs source: 3

4 Local bugs Some bugs are very local redundant code wrong condition omission lack of checks divide by zero approximations 4

5 Many bugs in conditions source: 5

6 Zune bug while (days > 365)! {! if (IsLeapYear(year))! {! if (days > 366)! {! days = 366;! year += 1;! }! }! else! {! days = 365;! year += 1;! }! }! (days >= 366) source: 6

7 Zune bug Zune 30 was the first portable media player released by Microsoft release date: november 2006 On Dec 31, 2008 all Zune stop working Software bug in the firmware: infinite loop when dealing with leap years Huge loss of business 7

8 Heartbleed bug! if ((err = ReadyHash(&SSLHashSHA1, &hashctx))!= 0)! goto fail;! if ((err = SSLHashSHA1.update(&hashCtx, &clientrandom))!= 0)! goto fail;! if ((err = SSLHashSHA1.update(&hashCtx, &serverrandom))!= 0)! goto fail;! if ((err = SSLHashSHA1.update(&hashCtx, &signedparams))!= 0)! goto fail;! goto fail;! if ((err = SSLHashSHA1.final(&hashCtx, &hashout))!= 0)! goto fail;!!err = sslrawverify(ctx,! ctx>peerpubkey,! datatosign,!!!!/* plaintext */! datatosignlen,!!!/* plaintext length */! signature,! signaturelen);!!if(err) {!!!sslerrorlog("ssldecodesignedserverkeyexchange: sslrawverify "! "returned %d\n", (int)err);!!!goto fail;!!}!! fail:! SSLFreeBuffer(&signedHashes);! SSLFreeBuffer(&hashCtx);! return err;! } source: 8

9 Heartbleed bug source: 9

10 Heartbleed bug Bug introduced March 2012 Bug revealed in April 2014 Without using any privileged information, it is possible to retrieve secret keys used for X.509 certificates user names and passwords instant messages s and business critical documents 10

11 More local bugs examples USS Yorktown (1998) Division by zero stops the engine of the submarine Guiding system (2002) Wrong initialization The Patriot and the Scud (1991) approximation in substraction 1 1

12 Global bugs Some bugs emerge from interactions wrong assumptions about third parties error in reuse concurrency bugs hardware/software/user improbable interactions 12

13 Northeast blackout of 2003 Root cause of the outage was linked to a variety of factors, including FirstEnergy's failure to trim back trees encroaching on highvoltage power line Software bug in the alarm system at a control room of the FirstEnergy corp. When triggered, race condition caused alarm system to stall for over an hour backup server kicked in, it could not keep up with unprocessed data warnings and alarms were not sounded because the systems were struggling to process old data. employees did not take action blackout spread to a huge region 13

14 Northeast blackout of

15 Northeast blackout of 2003 Widespread power outage on Aug 14, 2003 Affected an estimated 10 million people in Ontario and 45 million people in eight U.S. states. source:

16 Race condition public class SimpleApplet extends java.applet.applet{! java.awt.image art;!! public void init() {! art = getimage(getdocumentbase(),!!!!!!!!!!getparameter("img")) }!! public void paint(java.awt.graphics g) {! g.drawimage(art, 0, 0, this);! }!! an Applet's paint() method can be }! called before its init() method. 16

17 Check input public class SimpleApplet extends java.applet.applet{! java.awt.image art;!! public void init() {! art = getimage(getdocumentbase(),!!!!!!!!!!getparameter("img")) }!! public void paint(java.awt.graphics g) {! }! if (art!=null){!!!!g.drawimage(art, 0, 0, this);!!!}! }! 17

18 Ariane 501 H0 > H0+37s : nominal Dans SRI 2 (Inertial Reference System) : BH (Bias Horizontal) > 2^15 convert_double_to_int(bh) fails! exception SRI > crash SRI2 & 1 OBC (OnBoard Computer ) disoriented Angle attaque > 20, charges aérodynamiques élevées Séparation des boosters 1 8

19 Ariane 501 H0 + 39s: autodestruction (cost: 500M ) 1 9

20 Why? (cf. Jézéquel et al., IEEE Comp. 01/97) Ariane 5 reused a component from Ariane 4, which had an implicit assumption! Assumes a constraint on input domain Précondition : abs(bh) < OK for Ariane 4 but not Ariane 5 Need to specify exact contracts 2 0

21 More global bug examples London Ambulance System (1992) delays in medical emergencies bad data checks, memory leaks, GUI issues, bad HW reuse, etc. Mars orbiter (1999) Comparing inches with meters makes the probe crash on landing Orange (2012) bug in the replicated, brand new HLR, no alarm triggered Facebook IPO glitch (2012) race condition 21

22 Even more global bugs Therac25 (official report) The software code was not independently reviewed. The software design was not documented with enough detail to support reliability modelling. The system documentation did not adequately explain error codes. AECL personnel were at first dismissive of complaints. The design did not have any hardware interlocks to prevent the electronbeam from operating in its highenergy mode without the target in place. Software from older models had been reused without properly considering the hardware differences. The software assumed that sensors always worked correctly, since there was no way to verify them. (see open loop) Arithmetic overflows could cause the software to bypass safety checks. The software was written in assembly language. While this was more common at the time than it is today, assembly language is harder to debug than highlevel languages.. 2 2

23 Even more global bugs Système d information du FBI abandonné en avril 2005 : coût 170 M $ mauvaise spécification, exigences mal exprimées réutilisation dans un contexte inadapté trop d acteurs concurrents (hommes politiques, agents secrets, informaticiens) 2 3

24 Software fails Multiple causes various sources various levels various reasons True for every domain Has all sorts of consequences 24

25 Amazon s $23,698, book about flies Algorithmic pricing: Once a day profnath set their price to be times bordeebook s price, then bordeebook noticed profnath s change and elevated their price to times profnath s higher price. 25

26 WHY IS IT SO HARD TO BUILD CORRECT SOFTWARE? 26

27 Programminginthesmall Acquérir une valeur positive n Tant que n > 1 faire si n est pair alors n := n / 2 sinon n := 3n+1 Sonner alarme; Prouver que l alarme est sonnée pour tout n? Indécidabilité de certaines propriétés problème de l arrêt de la machine de Turing... Ø Recours au test ici, si machine 32 bits, 2^31 = 10^10 cas de tests 5 lignes de code => 10 milliards de valeurs! 2 7

28 Programminginthelarge See 2 8

29 Programminginthelarge Windows XP is compiled from 45 million lines of code. See Example*: Linux Kernel ,142,481 Firefox ,172,520 MySQL ,768 PHP ,892 Apache Http 2.0.x 89,967 * eloc (effective line of code) is the measurement of all lines that are not comments, blanks or standalone braces or parenthesis (see rsm_software_project_metrics.htm) Greg KroahHartman, Jonathan Corbet, Amanda McPherson. "Linux Kernel Development: How Fast it is Going, Who is Doing It, What They are Doing, and Who is Sponsoring It" (March 2012). 2 The Linux Foundation. Retrieved

30 Programminginthelarge See Retrieved

31 Programminginthelarge But also 3 1

32 Programminginthelarge See Retrieved Interoperability 3 2

33 Programminginthelarge Collaborative projects 137 contributors, 5000 commits, 1300 forks 97 contributors, commits, 173 forks 33

34 Programminginthelarge Reusability Durability Variability 34

35 Programminginthelarge Critical Realtime Embedded système embarqué : avion, voiture

36 Programminginthelarge Ø "The avionics system in the F22 Raptor [ ] consists of about 1.7 million lines of software code. Ø F35 Joint Strike Fighter [ ] will require about 5.7 million lines of code to operate its onboard systems. Ø Boeing s new 787 Dreamliner [ ] requires about 6.5 million lines of software code to operate its avionics and onboard support systems. Ø if you bought a premiumclass automobile recently, it probably contains close to 100 million lines of software code. [ ] All that software executes on 70 to 100 microprocessorbased electronic control units (ECUs) networked throughout the body of your car. Ø Alfred Katzenbach, the director of information technology management at Daimler, has reportedly said that the radio and navigation system in the current Sclass MercedesBenz requires over 20 million lines of code alone and that the car contains nearly as many ECUs as the new Airbus A380 (excluding the plane s inflight entertainment system). Ø IBM claims that approximately 50 percent of car warranty costs are now related to electronics and their embedded software "This Car Runs on Code", By Robert N. Charrette, IEEE Spectrum, Feb. 2009, see

37

38 Programminginthelarge Autonomic Computing Cloud Computing PaaS, SaaS, IoS, IoT... 38

39 ProgrammingintheDuration (maintenance) Etalement sur 10 ans ou plus d une ligne de produits Près de 80 ans dans l avionique! Age moyen d un système : 7 ans 26% des systèmes ont plus de 10 ans (Cf. Application banquaire et Cobol) 3 9

40 Long term availability AIRBUS A300 Life Cycle Program began in 1972, production stopped in = 35 years... Support will last until = 78 years!! On board software development for very long lifecycle products

41 Vcycle Innovation factory processes CMMI SCRUM spiral distributed embedded security critical constraints scalability enduser environments cloud lean innovation economics SOA outsourcing agility CP CBSE OOP AOP languages enduser Procedural DSL DSML UML 41

42 HOW TO BUILD RELIABLE SOFTWARE? 42

43 Engineering reliable software Constructive approach Formal modeling Garantees by construction Analytical approach Program analysis Detect and fix errors Faulttolerance Admit the presence of errors Enhance software with faulttolerance mechanisms 43

44 Constructive approach Garantee the absence of bugs Topdown approach Modeldriven development + formal analysis Formal proof Automatic or manual Offers exhaustive garantees based on logical modeling and reasoning Examples: Isabelle/HOL, B, KeY, Coq Used on specific parts of critical software (e.g., certified C compiler) 44

45 Constructive approach Model checking Formal behavioral model (transition system) Exhaustive verification of properties on model executions (e.g., absence of deadlock, safety and liveness properties) Examples: SCADE, Java PathFinder Used in hardware and software verification at the system level for systems engineering (defense, nuclear plant, transportation, etc.) 45

46 Analytical approach Look for the presence of bugs Heuristicbased Analyze all sorts of software artefacts (code, models, requirements, etc.) Software testing 46

47 Faulttolerance Assume that it is impossible to prevent the occurrence of bugs in production software Enhance the system with the ability to deal with it Design diversity at the systems level Exception handling at the source code level Randomization at the machine code level 47

48 Faulttolerance Nversion programming version 1 version 1 version 2 input version 2 vote output version 3 version 3 48

49 Google s innovation factory Google (2012 Update from Larry Page, CEO): Over 850,000 Android devices are activated daily through a network of 55 manufacturers and more than 300 carriers. Google Chrome browser has over 200 million users. Google launched Gmail in 2004 and now is used by more than 350 million people. YouTube has over 800 million monthly users who upload an hour of video per second. See

50 Google s innovation factory

51 Netflix s simian army Streaming TV network service approx. 40 million subscribers very high dependence on software and cloud (runs on Amazon EC2) major player in open source Induce failure regularly break production code to check the system s ability to react Chaos monkey: randomly terminates an instance in production Chaos kong: take an entire region offline Latency monkey: artificial delay in RESTful clients 51

52 Loop perforation: when good enough is better It used to be that people used computers for computations where there was a single, hard, logical right answer Tradeoff between accuracy and performance for (i=0; i<b; i++){ } transform for (i=0; i<b; i+=n){ } source: 52

53 Loop perforation source code Instrumentation instrumented binary Monitoring and perforation running program Compile In memory Execution unsound transformation still useful Sasa Misailovic, Stelios Sidiroglou, Henry Hoffmann, Martin C. Rinard: Quality of service profiling. ICSE (1) 2010:

54 Loop perforation 54

55 A question of perspective Verification: $ "Are we building the product right" The software should conform to its specification Validation:$ "Are we building the right product" The software should do what the user really requires 55

56 A question of perspective Stakeholder customer, developer, sales Qualitative functionality, usability, safetycritical, etc. Application kind embedded, adaptive, reactive, etc. 56

57 In this class Software testing most probably the technique you ll have to use for verification validation is essential but requires the involvement of users => usually done by specific teams, who don t develop from the developer s perspective you should test your software, and you ll be assigned testing tasks or supervision in your future software projects 57

58 Testing technique (Examples) Findbugs: PIT: static analysis on Java bytecode Evosuite: automatic generation of test cases for Java programs JavaPathFinder: automated analysis of Java programs Gzoltar: fault localization in Java programs mutation testing tool for Java programs Robolectric: test Android apps Sikuli: automates anything you see on the screen SDK android: test and debug Android apps 58

59 WHAT DO YOU MEAN BY $ SOFTWARE TESTING? 59

60 Tester pour prévenir une erreur du développeur introduit Une erreur est une décision inappropriée ou erronée, faite par un développeur, qui conduit à l'introduction d'un défaut. un défaut dans le système qui provoquera Un défaut est une imperfection dans un des aspects du système qui contribue, ou peut potentiellement contribuer, à la survenance d'une ou de plusieurs défaillances Parfois, il faut plusieurs défauts pour causer une défaillance. sa défaillance à l exécution. Une défaillance est un comportement inacceptable présenté par un système. La fréquence des défaillances reflète la fiabilité. 60

61 Le test : une première définition «Le test est un processus manuel ou automatique, qui vise à établir qu un système vérifie les propriétés exigées par sa spécification, ou à détecter des différences entre les résultats engendrés par le système et ceux qui sont attendus par la spécification» Extrait de la norme IEEESTD729,

62 Le test Essayer pour trouver des bugs. 62

63 Le test Essayer pour voir si ça marche. 63

64 Le test trouver des bugs Essayer pour voir si ça marche. Apprendre pourquoi c est fait ce que ça doit faire comment c est fait comment ça marche Modéliser S en faire une idée Exécuter Analyser Qu y atil à voir? Que fautil regarder? Qu estce qui est visible? Qu est ce qu on cherche? Comment le regarder? Qu est ce qui devrait marcher? Identifier une erreur Diagnostiquer une erreur Catégoriser ces erreurs 64

65 Qu estce qu on teste? quelles propriétés? Fonctionnalité Sécurité / intégrité Utilisabilité Cohérence Maintenabilité Efficacité Robustesse Sûreté de fonctionnement Etc. 65

66 Comment on teste? Test statique relecture / revue de code analyse automatique (vérification de propriétés, règles de codage...) Test dynamique on exécute le programme avec des valeurs en entrée et on observe le comportement 66

67 Comment on teste? Test fonctionnel (test boîte noire) Utilise la description des fonctionnalités du programme 1 O 1 I 2 O 3 2 Test structurel (test boîte blanche) Utilise la structure interne du programme I 1 I 2 I 3 O 1 O 2 6 7

68 Avec quoi on teste? Une spécification: exprime ce qu on attend du système des règles de codage un cahier des charges (en langue naturelle) commentaires dans le code contrats sur les opérations (à la Eiffel) un modèle UML une spécification formelle (automate, modèle B...) 6 8

69 Test de logiciel Plusieurs techniques Dynamique / statique Génération de test Fonctionnel / structurel Plusieurs niveaux/étapes: Unitaire, intégration, système, de nonrégression 6 9

70 Hiérarchisation des tests Problème analyse des besoins Définition des besoins Cahier des charges Conception globale Conception détaillée implémentation Plan de test système (fonctionnel) Plan de test d intégration? programme Programme livrable Intégration Composants unitaires Tests unitaires Système Tests d intégration Maintenance Test de recette (avec client) Tests systèmes

71 Test unitaire Validation d un module indépendamment des autres Valider intensivement les fonctions unitaires Les unités sontelles suffisamment spécifiées? le code estil lisible, maintenable...? 7 1

72 Test unitaire Pour un langage procédural unité de test = procédure void Ouvrir (char *nom, Compte *C, float S, float D ) { C>titulaire = AlloueEtCopieNomTitulaire(nom); (*C).montant = S ; (*C).seuil = D ; (*C).etat = DEJA_OUVERT ; (*C).histoire.nbop = 0; EnregistrerOperation(C); EcrireTexte("Ouverture du compte numero "); EcrireEntier(NumeroCourant+1); EcrireTexte(", titulaire : \""); EcrireTexte(C>titulaire); EcrireCar('"'); ALaLigne(); } Node Dans un contexte orienté objet 0..1 item: String 0..1 unité de test = classe previous + Node() + isfirst() + islast() next 72

73 Test d intégration Choisir un ordre pour intégrer et tester les différents modules du système 7 3

74 Test d intégration Cas simple: il n y a pas de cycle dans les dépendances entre modules Les dépendances forment un arbre et on peut intégrer simplement de bas en haut 7 4

75 Test d intégration E_RETRY Cas plus complexe: il y a des cycles dans les dépendances entre modules Cas très fréquent dans les systèmes à objets Il faut des heuristiques pour trouver un ordre d intégration INSTRUCTION CALL E_CHECK IFTHENELSE IFTHEN E_DEBUG E_LOOP ONCE_PROCEDURE ONCE_FUNCTION +then_compound ONCE_ROUTINE PROCEDURE FUNCTION SMALLEIFFEL FEATURE_CLAUSE +list LOCAL_NAME ARGUMENT_NAME + magic_count : Integer TYPE + is_ready : Boolean +list +result_type load_class() + get_started() +clients +name_list +to_runnable + falling_down() TYPE +result_type + afd_check() +clients CLIENT_LIST +name NAME BASE_CLASS + is_manifest_string + : path Boolean : String + is_result : Boolean + is_deferred : Boolean + is_void : Boolean + is_expanded : Boolean /+ is_generic : Boolean + is_any : Boolean = initval + is_general : Boolean DEFERRED_FUNCTION DEFERRED_PROCEDURE NATIVE_C NATIVE_SMALL_EIFFEL NATIVE_JVM EFFECTIVE_ROUTINE DEFERRED_ROUTINE EXTERNAL_ROUTINE +native NATIVE WRITABLE_ATTRIBUTE CST_ATT E_INSPECT WHEN_LIST E_WHEN WHEN_ITEM LOCAL_VAR_LIST ROUTINE ATTRIBUTE REVERSE_ASSIGNMENT EXPRESSION WHEN_ITEM_1 CREATION_CALL ASSIGNMENT PROC_CALL +call +type E_FEATURE DECLARATION TYPE DECLARATION_LIST FORMAL_ARG_LIST + is_deferred : Boolean = initval + count : Integer +result_type EXPRESSION CALL_PROC_CALL 1 GLOBALS +writable 1 DECLARATION_GROUP DECLARATION_1 +target 0..* +arguments FEATURE_NAME LOCAL_ARGUMENT CLASS_NAME + is_frozen : Boolean +origin_base_class COMPOUND FEATURE_NAME_LIST EXPORT_ITEM RENAME_PAIR (from InheritanceClause) (from InheritanceClause) +base_class_dictionary +base_class +base_class +export_list +rename_list +undefine_list +parent_list PARENT_LIST +redefine_list (from InheritanceClause) PARENT +select_list (from InheritanceClause) +base_class FEATURE_CLAUSE_LIST +feature_clause_list +creation_clause_list +base_class +base_class EXTERNAL_FUNCTION EXTERNAL_PROCEDURE +list CLASS_NAME +names CREATION_CLAUSE FEATURE_NAME_LIST +procedure_list +list CREATION_CLAUSE_LIST FEATURE_NAME EXPRESSION FROZEN_FEATURE_NAME INFIX_NAME PREFIX_NAME SIMPLE_FEATURE_NAME TYPE / path : String +generic_list +current_type FORMAL_GENERIC_ARG + +constraint constrained : boolean + rank : Integer +list FORMAL_GENERIC_LIST TYPE_CHARACTER +formal_generic_list TYPE_GENERIC TYPE_FORMAL_GENERIC +formal_generic_list (from TYPE) TYPE_CLASS (from TYPE) (from TYPE) TYPE_POINTER TYPE_ANY TYPE_NONE 7 5

76 Test système Valider la globalité du système Les fonctions offertes La qualité du système charge, ergonomie, sécurité, etc. A partir de l interface 7 6

77 Test de nonrégression Vérifier que des modifications apportées au logiciel n ont pas introduit de nouvelles erreurs vérifier que ce qui marchait marche encore Dans la phase de maintenance du logiciel Après refactoring, ajout/suppression de fonctionnalités Après la correction d une faute 7 7

78 Développer du logiciel pour tester du logiciel Test unitaire drivers (lanceur des tests), oracle (succès/échec), intrumentation (mesure couverture) Test d intégration idem + bouchons de tests (stubs), pour simuler les modules non disponibles Test système test des fonctions + environnement matériel + performances 78

79 The History of Software Testing