slides - Fangbo Tao

Aviation Safety Data Analysis System
Fangbo Tao, Xiao Yu, Jiawei Han
The data we focus:
Following a normal approach and landing
to runway 4 in roc; aircraft was taxied clear
of the end of runway to the gate .several
ground snow removal vehicles were
operating to left of aircraft so we moved
to the right side of ramp.….
Huge Collection of Logs
Each Document
Power of Text-Rich Data Cubes
Hierarchical Data Cube
Text Analysis
Power of Text-Rich Data Cubes
Data Cube
Efficient Summarization
Rich Text
Powerful Text Mining
Power of Text-Rich Data Cube
Other features
Contextual Search
: :support
based on Contextual Search
Contextual Search
 Motivation:
 Every word/concept may have equivalent word/concept
 “SVM” = “Support Vector Machine”, “Alt” = “Altitude”
 Connections between words
 “Kernel Method” - “SVM”, “altitude” – “flight level”
Contextual Search
 We develop a contextual search framework to build the
 Contains 4 different relationships:
 A “Use” B: Equivalent terms, B is more common
 A “RT” B: Related terms, not hierarchical
 A “BT” B: B is the broader word
 A “NT” B: B is the narrower word
Contextual Search
 Step 1: Generate word-net when uploading dataset.
 Step 2: Return the related terms when inputing.
 Step 3: Automatically include the equivalent terms when
 Step 4: Operator Support “AND”/”OR”/”NOT”
Hierarchical Dimension Support
 Multiple Choice Support
 Each Dimension can support several
 Powerful examples:
 “B-737” VS. “B-747”
 “Boeing” VS. “Airbus”
Document List Result
 Using the default Mysql “natural
language full text search”
 Extract the title based on the
most relevant part.
 Show tags of dimension values
for target dimensions
 Highlight the keywords
Similar Document
 Also contextual search
 Step 1: Extract meaningful terms from the original report
 Step 2: Using these terms as input, conduct contextual
Top Cells
 Search all the cells in the
targeted dimensions, find the
most relevant cells
 A multi-dimensional cell ranking
Single Dimension Distribution Based
on Keywords
Single Dimension Distribution Based
on Keywords
 Using a offline + online framework to calculate the
 If Offline:
 Combination of keywords are exponential
 If Online:
 Retrieve the whole corpus every time.
 Strategy:
 Store the single keyword distribution in the database. [Offline]
 Combine the single ones to a new distribution online. [Online]
Single Dimension Distribution Based
on Keywords
 Offline process:
 Step1: Map equivalent terms into one.
 Step2: Build both keyword reverse index and cell reverse
index based on report
 Step3: Compare these two reverse indexes and calculate
the single term distribution.
 Online process [with a list of terms and dimensions]
 Step1: match each term into it’s equivalent term.
 Step2: Calculate the combined distribution based on the
independent assumption, for each dimension
 Val( = 1 –π(1-val(ti));
Topic Distribution
 Based on Topic Cube
 Applying topic model.
 Support comparison between different cells
Unigram/Multigram description
 Based on Qiaozhu’s paper,
“Automatic Labeling of
Multinomial Topic Models”
 Find multi-gram candidate from
the whole text
 Scoring it based on unigram
 Adjust it based on it’s length
 Data Cube:
 Efficient Summary
 Highly Structured Data.
 Rich Text:
 Topic Analysis, keyword search
 Common: ASRS, IMDB, Publication-Net, News…
 Network (HIN)
 Good at mining, contains structural information.
 No information loss
Motivation of EventCube
 Combine Data Cube with Rich Text.
 Combine Summary with Keyword Search
 Build a general search/analysis system for rich text cube data.
 1. Aviation Safety Reporting Data
 Time, Weather, Location, Model…Flight logs
 2. Publication Data
 Author, Conf, Time, Field, Affliation…Abstract
 3. IMDB
 Time, Country, Style, Director…Description

similar documents