arsenal.nlp package

Subpackages

• arsenal.nlp.data package
  • Submodules
  • arsenal.nlp.data.ptb module
  • Module contents
• arsenal.nlp.tests package
  • Submodules
  • arsenal.nlp.tests.annotation module
  • arsenal.nlp.tests.bio2spantest module
  • Module contents

Submodules

arsenal.nlp.annotation module

exception arsenal.nlp.annotation.ParseError

Bases: Exception

Custom exception class used by this module.

class arsenal.nlp.annotation.Span(label, begins, ends)

Bases: object

begins

ends

label

arsenal.nlp.annotation.bio2span(seq, tagger=None, include_O=True)

arsenal.nlp.annotation.bracket2bio(x)

generate BIO-token pairs from bracket-style annotation. Note: splits text of spaces, so wordsplitting should already be done.

>>> x = bracket2bio("[TITLE Cat in the Hat][AUTHOR Dr. Seuss]")
>>> list(x)                                  #doctest:+NORMALIZE_WHITESPACE
[('B-TITLE', 'Cat'), ('I-TITLE', 'in'), ('I-TITLE', 'the'),
 ('I-TITLE', 'Hat'), ('B-AUTHOR', 'Dr.'), ('I-AUTHOR', 'Seuss')]

arsenal.nlp.annotation.extract_contiguous(s, labeler=None)

>>> list(extract_contiguous(""))
[]

>>> list(extract_contiguous("AAAA"))
[Span(label='A', begins=0, ends=4)]

>>> list(extract_contiguous("AABBC"))
[Span(label='A', begins=0, ends=2), Span(label='B', begins=2, ends=4), Span(label='C', begins=4, ends=5)]

>>> list(extract_contiguous("AABBB"))
[Span(label='A', begins=0, ends=2), Span(label='B', begins=2, ends=5)]

arsenal.nlp.annotation.fromSGML(f, linegrouper='\n', bioencoding=False)

arsenal.nlp.annotation.line_groups(text, pattern)

Very simple function for breaking up text into groups based on a single pattern.

>>> list(line_groups("a BB c d BB", "BB"))
['a', 'c d']

arsenal.nlp.annotation.sgml2bio(x)

>>> sgml2bio('<title>Cat in the Hat</title><author>Dr. Seuss</author>')
[('B-title', 'Cat'), ('I-title', 'in'), ('I-title', 'the'), ('I-title', 'Hat'), ('B-author', 'Dr.'), ('I-author', 'Seuss')]

arsenal.nlp.annotation.sgml2segmentation(x, lexer=re.compile('\\S+'))

>>> sgml2segmentation('<title>Cat in the Hat</title><author>Dr. Seuss</author>')
[('title', ['Cat', 'in', 'the', 'Hat']), ('author', ['Dr.', 'Seuss'])]

arsenal.nlp.annotation.sgml2seq(x)

>>> sgml2seq('<title>Cat in the Hat</title><author>Dr. Seuss</author>')
[('title', 'Cat'), ('title', 'in'), ('title', 'the'), ('title', 'Hat'), ('author', 'Dr.'), ('author', 'Seuss')]

arsenal.nlp.evaluation module

Evaluations methods common in NLP and information extraction.

TODO: Have a look at

https://github.com/nschneid/pyutil/blob/master/chunkeval.py, there appear to be richer evaluation methods.

class arsenal.nlp.evaluation.F1(confusion_matrix=False)

Bases: object

add_relevant(label, instance)

add_retrieved(label, instance)

confusion()

latex()

report(instance, prediction, target)

scores(verbose=True)

arsenal.nlp.evaluation.plot_confusion(y_true, y_pred, alphabet, normalized=False)

Draw confusion matrix

Options:

• normalized: Normalize the confusion matrix by row (i.e by the number of samples in each class)

Module contents