Part-of-Speech Tagging and the Penn Treebank Tagset
On this page
Part-of-Speech Tagging and the Penn Treebank Tagset
Recall first
Tag Book the flight and The book is heavy. Which word changes tag, and what evidence causes the change?
First principles
Part-of-speech (POS) tagging assigns a syntactic/lexical category to each token in context. It is sequence labeling: given tokens w₁…wₙ, predict tags t₁…tₙ. A tagset is an annotation convention, not a universal inventory of natural categories.
Common Penn Treebank (UPenn/WSJ) tags include NN common noun singular, NNS plural noun, NNP proper noun singular, VB base verb, VBD past verb, VBG gerund/present participle, VBN past participle, VBP non-3rd-person present, VBZ 3rd-person singular present, JJ adjective, RB adverb, DT determiner, PRP personal pronoun, IN preposition/subordinating conjunction, CC coordinating conjunction, CD cardinal number, and . punctuation. Always consult the exact tagset: conventions differ.
Worked example. Book the flight → Book/VB the/DT flight/NN; The book is heavy → The/DT book/NN is/VBZ heavy/JJ. The surface book is ambiguous between noun and verb. Local lexical identity is insufficient; neighboring function words, position, morphology, and sentence structure provide evidence.
Challenges include unknown words and names, novel usage (Google/VB), tokenization and contractions, punctuation, domain shift, long-range agreement, morphology, code-switching, and annotation disagreement. POS tags simplify a continuum: -ing can be gerund or participle depending on the scheme and analysis. For regional languages, tagsets must reflect their morphology and syntax; translating Penn labels blindly can hide distinctions or invent ones.
Exercise — reveal after answering
Tag Can you can the can? using a plausible Penn-style analysis.
Answer: Can/MD you/PRP can/VB the/DT can/NN ?/. Context makes the first modal, the second verb, and the third noun.
Exam lens
Define POS tagging as contextual sequence labeling. Give 4–6 Penn tags with meanings and a context-sensitive example. Mention tagset dependence and ambiguity.
Rapid revision checklist
- Define token-to-tag sequence labeling.
- Recall
NN,NNS,VB,VBD,JJ,DT,PRP. - Explain context-sensitive
book/can. - List unknown-word and domain challenges.
Key takeaways
- POS is a contextual label, not a fixed property of a spelling.
- Penn Treebank tags encode useful morphology but are corpus conventions.
- Tagger errors often begin with tokenization, unknown words, or domain mismatch.