FAST Enterprise Search Platform

Transcription

1 FAST Enterprise Search Platform June, 18, 2007 HR-XML Conference Yves SIMON Search engine department Director Hemisphere / ITS Group

2 Tentative plan Why should we locate experts? What are the main required functions? Demo Expert location Fast Search statistic & semantic approach 2

3 ITS Group ITS Group is a french IT services company publically traded on the EURONEXT PARIS Compartiment C (Code 7384) ITS Group is a 50 millions euros total sales and 550 employees company ITS Group is Fast Search&Transfer reseller for the french market & developed a specific approach regarding Expert Location & HR-XML in collaboration with Fast Global Services Team 3

4 Fast Search & Transfer European & International Créée en employés R&D = 275 pers Norway Quartiers Généraux Siège : Oslo San Francisco Salt Lake City Chicago London Boston Paris Washington Rome Munich Tokyo Amérique du Nord : Boston Dubai Dept. Linguist.: Munich croissance la + rapide du secteur C.A 2005 = $103M, C.A 2006= $162M => +62% sur un an Profitable, trésorerie positive, + de $200 million de liquidité Quotée en Bourse - (Oslo Stock Exchange 'FAST', Juin 2001) Selon le Gartner la meilleure technologie FAST, des références concrètes Rio Sydney 4

5 FAST solutions E-Commerce FAST Impulse ebusiness Knowledge Discovery Moteurs embarqués (OEM) Intellig. Econom FAST MarkeTrac FAST ESP Annuaires en ligne FAST Advisor Site Web FAST SiteSearch Publishing FAST ProPublish Wap/Mobile FAST MSearch 5

6 Critical points Identity management My HR information validity over time My HR information splet over Internet applications My HR information in context of my communities, organizations needs Agregation of CV data s to reflect organization s competencies Multilingual environment Fuzzy matching, Proximity, Interoperability Import / Export XML Federated search within CV Security Experience Incrementation issue 6

7 Why should we detect experts? To increase information velocity To build a project team To answer training needs To forecast future job & competencies needs To validate professional experiences To set up qualification thesaurus Workforce retirement issue faced by any organization may direct HR to capitalize knowledge & expertize Internal / external mobility Are as many cases where a coherent information organization enabling qualification and expert location is required 7

8 Target : augment communication & collaboration By putting in place some advanced information access tools in the Organization Information System : Everybody needs to be able to show his expertises Everybody within the organization needs to be recognize as an expert by the group / community Links between persons, fields of interest, projects, clients, competencies are to be shown Increase data drill down ability to: Simplify acces to dynamic communities Increase network work & self apprenticeship capacities through multidimentional and interactive tools 8

9 Information Eco system (Re)trouver les experts STOCK Directories & Social networks Identifier les compétences et expériences disponibles (Re)trouver les experts Dynamic mapping FLUX Experts Weblogs Intranets Retrouver l auteur S T O C K S Retrouver les contributions WIKIS + Collaborative work Search Engine (Re)trouver les discussions Document repositories CMS Capitaliser Consulter les documents et connaissances 9

10 4 goals (FUSE Concept) «Find» car nous devons être à même de permettre aux internautes d'identifier la bonne information au regard de leurs besoins au bon moment (alertes ). «Use»car nous devons proposer des outils pour traiter l'information dans le but d'accomplir une tâche ou de réaliser un but. «Share» car nous sommes persuadés que c'est dans l'échange que l'on avance le plus vite. «Expand» car il est important de fournir le plus large éventail de réponses et de solutions possible. 10

11 Different functions needed in an expert location tool Moderate a profile Rating Personal comments / shared comments Manual Tags, Folksonomies, Automatic tags (vectorization based) Backlinking Send a profile , sms, im, fax, print Save a profile, export it Filtering and alerting tools to personalize and automate the flow of information specifically needed in modification / update cases mail, sms, flux rss 11

12 Information visualization (1) Information visualization Drawing sometime better than a long speech! Create visibility around information by graphically describing data and data relations using a map analogy Information synthesis To have the information meaning emerge out of a massive amount of data Show Collective intelligence Go beyond hard to maintain traditional classification ways 12

13 Information visualization (2) Dynamic visualization To describe data and their relations using a graph/map analogy Used graph/map representations are user friendly and ergonomic to enable a quick and easy usage access Correlations and proximity analysis Accessible data amount endless growth create a need for a global and instant visualization Without previous automatic classification, enabling a great amount of data analysis to quickly and properly represent and understand information meaning 13

14 Information visualization (3) 14

15 Expert location demo 15

16 Case Study: CareerBuilder.com About CareerBuilder Number One Online Job eclassifieds Site Online Recruiting Founded by 3 US Newspapers publishers Fastest-growing recruitment and career-advancement source for employers, recruiters and job seekers Presence in over 200 local markets, nationwide in USA Challenges Increase market position, through traffic growth Provide job and résumé search on site while addressing index latency and speed of search Leverage search to streamline location of jobs and resumes Scale to support more listings and more end-users Enhance the user experience Drive revenues through new distribution channels FAST Differentiators Performance and scalability Geographic search capability Navigation and user experience With FAST, we can go out and sign up huge customers... and not worry for a minute as to whether we can handle the huge influx in volume. For the first time, we can scale our software capabilities as fast as we can sign a deal. Eric Presley, CTO, CareerBuilder.com Average over 200 QPS; Superbowl 2005: handled 2,000 QPS after TV ads Significantly lowered index time of new job postings Query search time reduced by order of magnitude (10 sec to sub-second) July 2004, become #1 trafficked recruiting site (from being #4); entered syndication partnership with AOL and MSN 300% increase in job search capacity Results with FAST ESP 16

17 Fast Search technology highlights

18 FAST Enterprise Search Platform FAST PSP FAST MSP Market Management Solutions Risk Management Solutions Surveillance & Enforcement Solutions Information Management Solutions ecommerce Solutions Online Media Solutions Mobile Solutions Expert Location Embedded Application Solutions Options FAST ESP Legacy Data Internet Applications RDBMS DMS, CMS, Files Rich Media XML Elements, Portals RSS Feeds, XPath Data Streams STRUCTURED UNSTRUCTURED 18

19 FAST ESP : Architecture STRUCTURED DATA SEARCH RICH MEDIA UNSTRUCTURED DATA CONTENT REFINEMENT Content API ALERT QUERY PROCESSING RESULT PROCESSING Search API SEARCH COLLECTIONS CONNECTORS REFINEMENT SEARCH & ALERT ENGINES PROCESSING SEARCH PROFILE USER ADMINISTRATION 19

20 FAST Security Models Document Security ACLs indexed with documents Document Security Security server content indexed with documents Products Content API Site News Collection Security System Admin Collection Admin Collection Manager Real-Time Content Refinement Generic Express SiteSearch NewsSearch Encryption Scalable Search SSL on the Engine wire Document Security XML Inverted File Optional check against security server for latest information Document Security Result set strips out documents user not permitted to see Query Processing Result Processing XML YP Connectors Push & Pull Document Security Security server content indexed with documents User Authentication Portal handles initial user authentication and passes UID with query Search Client 20

21 FAST ESP Core Capabilities Flexible Relevancy Model Monetization Dynamic Navigation Search Business Center Advanced Linguistics Diverse Content Geo-Search Sentiment Analysis Real Time Alerting Personalization Tune, boost, block search results to meet your business needs Drive advertising, subscription and transaction revenues through search Automatically categorize content and create browsing options Manage the entire search, navigation, monetization through GUI studio Mine text and data, discover entities, concepts, and relationships Aggregate, index, process, deliver structured and unstructured content Deliver location-aware results, local search results and news content Aggregate content with an understanding of its emotional tone, message Serve content-driven and event-based alerts to subscribers Incorporate user preferences/experiences to drive revenue and customer loyalty across multiple platforms 21

22 Why using FAST? 1 Exploitation simultanée des contenus structurés et non-structurés (Pertinence optimale) 2 Plateforme unique riche 77 langues Traitements sémantiques ET statistiques Contextual Insight Tarification en fonction de l intensité d utilisation 3 Personnalisation des traitements & du service de recherche (ranking ) 4 Performance & Sécurité (Temps d indexation + temps de réponse Vs Qté de données) 22

23 Document Processing Pipelines Real-time Content Refinement Conversion Language Detection Synonyms Spellcheckin g Lemmatization Files, Documents Entity Extraction Categorizatio n Vectorization PLUG-INs Scopifier Alerting Indexing Fast Search & Transfer vient de lancer Fast Enterprise Search Platform 5.0. Cette plate-forme de recherche offre un accès à toutes les données structurées, non structurées et multimédia (Rich Media). Elle inclut une nouvelle recherche contextuelle, pour la fusion entre l'extraction et l'analyse des informations et le centre de recherche Fast Search Business Center pour la gestion des requêtes, afin que tout utilisateur puisse configurer et adapter diverses applications de recherche à partir d'une seule et même installation. 23

24 Entity extraction Un framework prêt à l emploi et ouvert Extraction d entités sur la base de listes et/ou de modèles (patterns) Combinaison d entités élémentaires (prénom => personne, code postal => adresse) Outils de création et de test d extracteurs Les entités extraites en standard : Personnes, lieux, organisations, dates, universités, adresses Création faciles de nouvelles entités : compétences, métiers Largement utilisées dans FAST ESP5 : Correction orthographique, normalisation, navigation contextuelle Peuvent être cryptés pour préserver la propriété intellectuelle 24

25 Contextual Insight Digg & define pertinence at the sentence level instead of the document level 25

26 Similarity search (identity matching ) Index Indexation Requêtes CVx CV1 CV2 CV2 CV3 Entity extraction (diplômes, écoles, employeurs, fonctions ) Vectorization Vectorization Entity extraction (diplômes, écoles, employeurs, fonctions ) 26

27 EFICIENCY MORE LES Classification/Categorization High Efficiency Low Rule-based Keyword Boolean logic Dictionaries, thesauri, linguistic analysis Manual Subject tagging Forms High CATEGORIZATION METHODS Supervised Example training Bayesian, SVM, k- NN, Neural Networks Cost and Control Unsupervised Clustering Auto taxonomy generation Low COMPLETE SOLUTION Unsupervised Supervised Rule-based Manual 27

28 Pipeline = Request treatment Fuzzy matching enabled Requête : les feuiles de congé? Analyse Syntaxique feuiles feuilles de congé les feuilles congés Correction orthograph. Expressions Antiphrase Normalisation feuilles de congé Vacances Repos Formulaire pour Demande de vacances N 1 - Procédure de demande de congé Paramètre pertinence Lemmas Synonymes Thesaurus PLUG-IN Règles de priorité Pour recherche congé Lancer requête sur collection «Ressources Humaines» Droits D accès Recherche sélective Requête modifiée 28

29 Relevance Ranking: The CASQF Framework Completeness How well does the query match superior contexts like the title or the url? Example: query= Mexico, Is Mexico or University of New Mexico best? Authority Is the document considered an authority for this query? Examples: Web link cardinality, article references, product revenue, page impressions,... Statistics How well does the contents of this document on overall match the query? Examples: Proximity, context weights, tf-idf, degree of linguistic normalization,++ Quality What is the quality of the document? Examples: Homepage?, Entry point to product group?, Press release?,... Freshness How fresh is the document compared to the time of the query? 29

30 Relevance Ranking: Machine Learning Optimization Relevance Primitives Auto-language detection Approximate matching Lemmatization (grammar) Phrase detection Anti-phrasing (stop words) Multiple dictionaries thesauri Natural language (who, what, where, etc.) Proximity search Spatial relevance Contextual tagging CASQF Attributes Completeness Authority Statistics Quality Freshness Correlations Complete & Fresh? Authority & Frequent query. Transfer functions Attribute weights Correlations 30

31 Drill down (1) Information discovery 31

32 Drill down (2) Information discovery 32

33 Drill down (3) Information discovery 33

34 Merci