GALATEAS: donner du sens aux journaux de logs pour une meilleure connaissance des clients et utilisateurs de sites web

Transcription

1 GALATEAS: donner du sens aux journaux de logs pour une meilleure connaissance des clients et utilisateurs de sites web Frédérique Segond Manager, Research & Development Unit Viseo Research & Innovation Domoina RABARIJAONA Engineer, Objet Direct Viseo, With the contribution of the GALATEAS partners

2 GALATEAS partners Project coordinator: Xerox Research Centre Europe (France) Objet Direct (France) CELI (Italy) University of Trento (Italy) Gonetwork (Italy) Bridgeman Art (England) Humboldt University (Germany) University of Amsterdam (Netherlands)

3 GALATEAS - EU Project Today content providers cannot customize content and indexing as they don t know their users. The GALATEAS project offers digital content providers an innovative approach to understanding users' behaviour by analysing languagebased information from transaction logs technologies facilitating improved navigation and search for multilingual content access

4 GALATEAS - web services GALATEAS develops two web services: Understanding with LangLog: It analyzes transaction log containing queries to search engines for a given content provider. By applying statistical technologies coupled with language oriented services, it produces reports concerning the informational needs of the users accessing that particular aggregation. LangLog provides generalizations of the actions that information seekers perform in order to find contents inside a searchable collection of digital objects. Gobalizing with QueryTrans: It translates queries coming from an external search engine into several target languages: the external search engine returns to the user results into languages different from the one in which the query was formulated.

5 GALATEAS Search Semantic Analytics Understanding web visitors intention is key to convert them into customers or users OBJECTIVES Understanding web users behavior, what users are really looking for, how do they look for information, whether they find it or not, and if I have it in my content Improving conversion rate Optimize sales on e-commerce sites Optimize communication on PA Decide over product innovation Improve search techniques APPROACH Applying semantics and semantic technologies to an on-line system for interpreting the search and web log data Applying reporting tools to cross and visualize data obtained in the above phase Development of an on-line service independent or integrable with the client s web analytics that provides actionable reports on users behavior. The customer access the visitors needs immediately 5 RESULTS

6 LangLog : Objectives To provide an intelligent platform to analyze user queries to resident search engine. Problem driven research: main-stream query analysis system are completely language unaware. 6

7 GALATEAS - Search Semantic Analytics Standard web analytics results Category Keyword pensione casalinga pensioni casalinghe Retiremen t pensione casalinghe pensione sociale casalinghe obbligo gomme termiche piemonte 2011 obbligo pneumatici invernali piemonte Search Galateas results Transport albo autotrasportatori obbligo catene a bordo 2011 piemonte gtt clorofluorocarburi Environme nt combustibili fossili assistenza domiciliare Elderlies integrata and accompagnamento handicape assegno mensile di d other; 13% flora and fauna; 23% 49 assistenza cesedi School bosso monti cascina falchera wastes and pollution wastes and pollution waterworks; 8% energy flora and fauna other energy; 25% waterworks

8 Ex. Google 8

9 Identify queries that are diferent on surface but refer to the same concept Discover natural grouping and unexpected events Normaliz e Cluster Classify To understand mapping of queries on corporate categories

10 Technological approach LogFile parsing/structuring Linguistic processing Semantic Enrichment NER Tagging Morphology Classification 10 Clustering Reporting

11 LogFile parsing/structuring Automatic dowload of logfiles from web server Parsing different logfile formats (es. Apache transaction log, solr logfiles, etc.) Automatic normalization/correction (format inconsistencies, encoding, etc.) Filling the database with structured information about query, user sessions, click-through, etc. 11

12 Linguistic Processing Linguistic processing Language Identification: As a prerequisite of any linguistic processing Morphological analysis and lemmatization: For better aggregation and further processing Detection and normalization of special tokens (e.g. abbreviations, spelling variations,etc.) Named-Entities recognition (eg. persons, products, places etc.) : To understand mapping of query to object multi-words expression recognition (es. pro loco, mezzi pesanti», etc.) 12

13 LangLog Linguistic Processing Challenge Recognise named entities and deal with multilingual terms in very short texts Query 1 Tableau Mona Lisa (F) Query 2 Oil painting la Gioconda (EN) Query 3 La Gioconda pitturi da Vinci (IT) Index term 1 La Gioconda Index term 2 Oil Index term 3 GALATEAS Identify appropriate index terms according to what the user is looking for Painting

14 Semantic Enrichment Linguistic processing Adding semantic information based on: Generic ontologies Domain specific semantic networks Semantic enrichement enables to analyse words used in users query, from a «meaning» point of view at diverse abstraction levels 14

15 Classification Query classification according to a given/predefined taxonomy, generally corresponding to the organisation of web site content Two classification systems: Unsupervised, for customer without a learning set (or click through information). Supervised for customers with a learning set (or click-through information) 15

16 Unsupervised Classification Documents1 Vector1 Documents2 Vector2 Documentsn Vector3 Similarity Vector Documents Query Category1 Category2 Categoryn 16 Learner Wikipedia

17 Supervised Classification Meta-Data Topic Models Feature Computation Module CT-Info Features Naïve Bayes Decision Tree Learning Algorithm Category1 Category2 Categoryn 17 Model SVM Query

18 Clustering Linguistic processing Automatic grouping of semantically omogenous query in order to Cluster dynamically set of queries selected by the user. Provide static clusters of large sets of queries (e.g. last week, last month, last year) Topic Models, approach using the LDA algorithme - Latent Dirichlet Allocation and Gibbs Sampling A "topic" consists of a cluster of words that frequently occur together. Using words and information added at the semantic enrichment step, topic models can connect words with similar meanings and distinguish between uses of words with multiple meanings. Queries are clustered on the basis of topical words 18

19 Clustering The service implements two kinds of clustering: An Exclusive Clustering (or hard clustering). The queries are partitioned in k distinct groups A Hierarchical Clustering Solution. This solution is built on top of the hard clustering solution by successively merging the most similar clusters in a tree. The service returns a GEPHI graph representation of the Hierarchical Solution 19

20 Clustering 20

21 LangLog Classification and clustering Query ID Query Class Query ID Query Class Query 1 Leonardo da Vinci, La Gioconda Art Query X Science Query 2 Leonardo da Vinci, Vitruvian Man Science Leonardo da Vinci, hydraulics, hydrometer Query 2 Oil painting, la Gioconda Art Query 3 La Gioconda, pitturi, da Vinci Art Query 5 Leonardo da Vinci, meteorology Science Challenge Perform classification and clustering with short query texts GALATEAS Assign to previously unseen queries a class from your indexing hierarchy

22 Reporting Data and graphical representation easy to read: Grafics for classification and clustering Temporal view of most relevant topics Classification of frequent queries, normalised click-through statistics on results Overlapping analysis of classification/clustering Analysis of next-page and next-page-click 22

23 General framework All technologies are incorporated in a web services framework that allows easy integration of third-party technologies and great extensibility Web services Translated query Original query Customer Query logs Semantic similarity Customised reports Web services Semantic services GALATEAS core Natural Language Processing services Named Entity Recognition/ Part of Speech Tagging

24 Business Intelligence et son application à GALATEAS Buts du BI Différentes sources de données Tirer connaissance de ces données Aide à la décision

25 Architecture standard d un système de BI Data source Integration Datawarehouse Reporting ODS DWH ETL ETL DSA CDW 25 DataMining

26 Outils BI 26

27 Méthodologie Etude des besoins De quoi le décideur a-t-il besoin de savoir pour améliorer les rendements? Recueil des données Inventaire des sources de données Mise en place d'un référentiel commun (MDM) Fixer les données à avoir en sortie Conception Modèles de données Traitements à faire sur les données Réalisation 27

28 Architecture Galateas Data source Integration Datawarehouse ODS ETL DSA DB 28 ETL DWH Reporting

29 ETL 1. Préparation identification des données à extraire des sources extraction de ces données ODS 2. Intégration nettoyage des données extraites archivage éventuel définition d un format commun transformation des données vers ce format 29 DSA

30 ETL 30

31 ETL Data source Integration ODS ETL DSA DB 31

32 ETL De l'ods au DSA Filtrer les lignes "Parser" chaque ligne selon le format Id_Session Requête envoyée Ressource accédée Formater les valeurs Distinguer les recherches des clics Format CommonLogFormat (W3C) [03/Feb/2012:14:38: ] "GET /extension/ezaepi/design/ezaepi/images/favicon.ico HTTP/1.1" "-" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0;.NET CLR ;.NET CLR ;.NET CLR ;.NET CLR )" Format ExtendedLogFileFormat (Apache) [john]&thumb=x150&num=15&page=2&img=24557f6ff ba7b253ff Mozilla/5.0+(compatible;+DotBot/1.1;

33 Enrichissement des logs Language Identification Lemmatization NER DB Classification Clustering 33

34 Datawarehouse 34

35 Datawarehouse Méthodes de construction Top-down Réalisation de toutes les dimensions et faits Bottom Up Création par étoile puis regroupement Middle-out Conception totale, création par partie 35

36 Importance de la modélisation Problème de modélisation différents modèles de données sont utilisés Problèmes de terminologie un objet est désigné par 2 noms différents un même nom désigne 2 objets différents Incompatibilités de contraintes 2 concepts équivalents ont des contraintes incompatibles Conflit sémantique, de représentation 36

37 Mise à jour du datawarehouse Reconstruction périodique Mise à jour périodique Mise à jour instantanée 37

38 Datawarehouse et Database 38 Datawarehouse Database OLAP OLTP Système décisionnel Système opérationnel Bulk insert et Select Insert, update, delete, select Historisé Volatile

39 Approche OLAP 39 MOLAP (Multidimensional) - Base de données multidimensionnelles - Données pré-agrégées - Limitations sur les quantités de données - MDX ROLAP (Relational) - HOLAP (Hybrid) - Approche hybride entre MOLAP et ROLAP SOLAP (Spatial) - Modèle multidimensionnel optimisé pour l'analyse spatio-temporelle DOLAP (Desktop) - Petite quantité de données directement stockée sur le poste du client Base de données relationnelles Simulation des cubes Tables d'agrégation Grande capacité de données SQL

40 Approche OLAP 40 SQL MDX SELECT * FROM table WHERE x=y GROUP BY z SELECT {[Measures].[Search Count]} ON COLUMNS, {[DIM1].[Value1]} ON ROWS, FROM Fact WHERE ([DIM2].[Value2])

41 Approche OLAP Opérations sur un hypercube Rotate Drill down (forage avant) Roll Up (forage arrière) Slice & Dice Drill through 41

42 Reporting Web Portal Rapports statiques ireport Cube OLAP (hypercube) Mondrian (serveur) Jpivot (client) 42

43 Démo 43

44 Tendances du BI BI Mobile Fonctions In-memory Fonctionnalités de collaboration Big Data 44

45 Thanks to Natural Language processing Technologies we now have a clear view of customers needs! Thank you! Page 45 45