Table of Contents

• Introduction:
  • Task Setting
  • Previous Work
• Experiments:
  • Linguistically Motivated Metrics
  • Resource Selection
  • Threshold Tuning
• Results:
  • Discussion
  • Conclusions

Semantic Category Disambiguation (SCD)

• Task definition:
  • Given: a textual span in it’s context and a set of semantic categories
  • Assign: a single semantic category to the span

Example: An annotated noun-phrase

• Related to Named Entity Recognition (NER):
  • Conceptually NER consists of two sub-problems
    • Find textual spans containing entities
    • Disambiguate between multiple semantic categories for each span

Motivation

• Event Extraction (EE):
  • Recent move towards complex EE (Kim et. al. 2009,2011)
  • Complex event annotations contain a multitude of semantic categories (80 for Genia (Kim et al. 2008))
  • State-of-the-art NER targets a single-class
  • Annotated resources for multi-class NER exist
  • Joint multi-class NER is beneficial (Dang and Aizawa, 2008)
• Why Biomedical?:
  • Extensive annotated/lexical resources
  • Real-world applications, substantial resources to be saved for curation projects
• Applications:
  • EE
  • Coordination-analysis
  • Co-reference resolution

Previous Work (Cohen et al. 2011)

• Goal:
  • Annotation assistance for ontology annotations
• Procedure:
  • Assume semantic categories are defined by ontological resources
  • Map a given textual span to one or several ontologies using a set of rules
• Performance:
  • Evaluated on single-corpus annotated data
  • 77.1% to 95.5% depending on the category

Previous Work (Stenetorp et al. 2011)

• Goal:
  • Show benefits of using approximate string matching to lexical resources
• Procedure:
  • Standard NER-inspired feature set
  • Novel approximate string matching features using SimString (Okazaki and Tsujii 2010)
  • 170 lexical databases containing > 20,000,000 entries (large)
  • Semantic categories learned from training data
• Performance:
  • Evaluated on six annotated corpora
  • Accuracy of 85.9% to 95.3% (not directly comparable to Cohen et al. (2011))

Methods

• Issues not investigated by Stenetorp et al. (2011)
• For this work:
  • Linguistically Motivated Measures
    • Distance Measure
    • Start/End Guards
  • Resource Selection
  • Threshold Tuning

Linguistically Motivated Measures: Distance Measure

• SimString uses a cosine measure over n-grams:
  • Any change in character is equally “harmful”
  • This does not match reality: “EGR–1”, “EGR 1” and “FGR–1”
• Use Lehvenstein with a protein cost-matrix from the literature (Tsuruoka 2003)
• Normalised and non-normalised version

Table: String distance costs

• Problem: O(n^2) time complexity:
  • Compare the top 10 results from SimString
  • Pick the best result using the cost-matrix

Linguistically Motivated Measures: Start/End Markers

• SimString uses a cosine measure over n-grams:
  • n-grams don’t naturally preserve gram position
• Use Start/End guards:
  • Example: “Foobar” -> “|Foobar|”
  • Standard NER feature, but not used for look-ups

Resource Selection

• SimSem used a total of 170 lexical databases:
  • No detailed analysis of individual impact
  • Possible negative impacts of excessive resource use
• Greedy descent:
  • Iteratively knock out non-beneficial resources over several rounds
• Greedy ascent:
  • Iteratively add beneficial resources over several rounds
• Overfitting protection:
  • Cross-validate on the test set for each iteration

Threshold Tuning

• SimString uses a threshold to determine matches between strings
• Ranges from 1.0 to 0.0:
  • 1.0 is an exact match
  • 0.0 matches anything
• Stenetorp et al. (2011) used a threshold of 0.7
• We tried to find a better threshold:
  • 0.4 appeared to give better results for our development set

Experiments - Corpora

• Same set-up as Stenetorp et al. (2011)

Table: Corpora used for experiments

Experiments - Measures/Baselines

• Performance measures:
  • Accuracy
  • Primary: Accuracy Area Under the Learning Curve (AUC)
• Baselines (Stenetorp et al. 2011):
  • Internal Classifier (Int.)
  • Internal Classifier + SimString (Int.Sim.)

Results - Baselines

Results - Distance Measure

Results - Start/End Markers

Results - Threshold Tuning

Results - Resource Selection

Conclusions

• Take-home points:
  • Optimal approximate string matching strategy depends on the dataset
  • Approximate string matching helps SCD
• Future work:
  • Detailed error analysis
  • Incorporating large-scale unsupervised/statistic aspects
  • Automatic look-up strategy selection per dataset (category too?)

Thank You for Your Attention

• In the spirit of internationallity:
  • 日本語：ご清聴ありがとうございました
  • Svenska: Tack för er uppmärksamhet
• About SimSem:
  • Open-source and freely available
  • Resources, extended experiments and code: http://github.com/ninjin/simsem
• About the author:
  • Website (for paper and slides): http://pontus.stenetorp.se/
  • E-mail: <pontus stenetorp se>
  • If you have any questions or are pursuing similar research I’d love to talk to you