Assignment (POS Tagging and Named Entity Recognition)¶
Name: Qylle Christian Quino
Section: DS4A
Get a paragraph from your favorite literature. Analayze your chosen paragraph and do Textual Cleaning if needed. Conduct POS Tagging then Analayze and interpret the following:
- POS Frequency (How many nouns, verbs, adj, etc.)
- Identify dominant POS types - Is the paragraph Noun-heavy or is the paragraph Verb-dominated or etc.? Explain your answer.
- Observe Patterns - *******ex. frequent Adjectives might indicate rich imagery or emotional tone*******
What are your other observations and interpretations?
In [2]:
import nltk
import re
import pandas as pd
from nltk.tokenize import word_tokenize
from nltk.corpus import stopwords
from nltk.tag import pos_tag
from nltk.chunk import ne_chunk
from collections import Counter
In [18]:
nltk.download('punkt')
nltk.download('averaged_perceptron_tagger')
nltk.download('averaged_perceptron_tagger_eng')
nltk.download('stopwords')
nltk.download('maxent_ne_chunker')
nltk.download('words')
nltk.download('maxent_ne_chunker_tab')
Out[18]:
Selected Paragraph¶
In [4]:
corpus = ''.join(["The hobbits stood before the King and all the great assembly. ",
"Aragorn rose from his throne and looked down at them with eyes full of love and pride. ",
"The four hobbits had saved Middle-earth through their courage and friendship. ",
"They had walked through fire and darkness, faced terrible dangers, and never wavered. ",
"Now they stood in the great hall, small among the tall warriors and noble lords. ",
"Aragorn spoke softly but his voice carried to every corner. ",
"My friends, you bow to no one. ",
"Then the King of Gondor knelt before the hobbits, and all the court followed, ",
"bowing low to honor the brave halflings who had carried hope when all seemed lost."])
corpus
Out[4]:
Textual Cleaning¶
In [5]:
def rem_punct(corpus):
if not isinstance(corpus, str):
return corpus
corpus = re.sub(r'[^a-zA-Z\s]', '', corpus)
return corpus
stop_words = set(stopwords.words('english'))
tokens = rem_punct(corpus)
tokens = word_tokenize(tokens.lower())
tokens = [word for word in tokens if word not in stop_words]
tokens
Out[5]:
POS Tagging¶
In [6]:
statistical_tags = pos_tag(tokens)
statistical_tags
Out[6]:
In [7]:
df = pd.DataFrame(statistical_tags, columns=['word', 'pos'])
df
Out[7]:
Analysis and Interpretation¶
1. POS Frequency¶
In [8]:
# Count POS tag frequencies
pos_counts = df['pos'].value_counts()
pos_counts
Out[8]:
In [9]:
# Group by major categories
def categorize_pos(tag):
if tag.startswith('NN'):
return 'Noun'
elif tag.startswith('VB'):
return 'Verb'
elif tag.startswith('JJ'):
return 'Adjective'
elif tag.startswith('RB'):
return 'Adverb'
elif tag.startswith('PRP'):
return 'Pronoun'
elif tag == 'DT':
return 'Determiner'
elif tag == 'IN':
return 'Preposition'
elif tag == 'CC':
return 'Conjunction'
else:
return 'Other'
df['category'] = df['pos'].apply(categorize_pos)
category_counts = df['category'].value_counts()
category_counts
Out[9]:
In [10]:
# Calculate percentages
total_words = len(df)
category_percentages = (category_counts / total_words * 100).round(2)
frequency_df = pd.DataFrame({
'Count': category_counts,
'Percentage': category_percentages
})
frequency_df
Out[10]:
2. Identify Dominant POS Types¶
In [11]:
dominant_category = category_counts.index[0]
dominant_count = category_counts.iloc[0]
dominant_pct = category_percentages.iloc[0]
print(f"""
DOMINANT POS TYPE ANALYSIS
The paragraph is {dominant_category.upper()}-HEAVY
- {dominant_category}s appear {dominant_count} times ({dominant_pct}% of all words)
EXPLANATION:
This passage is {dominant_category.lower()}-dominated, which is typical of narrative prose.
The high {dominant_category.lower()} frequency indicates:
- Focus on characters (hobbits, King, Aragorn, warriors, lords)
- Emphasis on places and settings (throne, hall, court, Middle-earth, Gondor)
- Concrete imagery establishing the scene
- Descriptive storytelling rather than action-focused writing
""")
3. Observe Patterns¶
In [12]:
# Extract specific word types
nouns = df[df['pos'].str.startswith('NN')]['word'].tolist()
verbs = df[df['pos'].str.startswith('VB')]['word'].tolist()
adjectives = df[df['pos'].str.startswith('JJ')]['word'].tolist()
adverbs = df[df['pos'].str.startswith('RB')]['word'].tolist()
print(f"""
OBSERVED PATTERNS:
NOUNS ({len(nouns)} total):
{', '.join(nouns)}
VERBS ({len(verbs)} total):
{', '.join(verbs)}
ADJECTIVES ({len(adjectives)} total):
{', '.join(adjectives) if adjectives else 'None found'}
{"INTERPRETATION:\n - Adjectives create emotional depth and vivid imagery\n - Words like 'great', 'terrible', 'brave', 'noble' convey grandeur and reverence\n - Contrasting adjectives: 'small' vs 'tall' emphasize the reversal of status\n - Emotional tone: love, pride, courage, honor" if adjectives else ''}
ADVERBS ({len(adverbs)} total):
{', '.join(adverbs) if adverbs else 'None found'}
""")
In [13]:
# Verb tense analysis
print("\nVERB TENSE DISTRIBUTION:")
past_verbs = df[df['pos'] == 'VBD']['word'].tolist()
past_participle = df[df['pos'] == 'VBN']['word'].tolist()
present_verbs = df[df['pos'].isin(['VBP', 'VBZ'])]['word'].tolist()
print(f"Past tense (VBD): {len(past_verbs)} - {', '.join(set(past_verbs))}")
print(f"Past participle (VBN): {len(past_participle)} - {', '.join(set(past_participle))}")
print(f"Present tense (VBP/VBZ): {len(present_verbs)}")
print(f"\nINTERPRETATION:")
print(f" - Dominance of past tense indicates narrative storytelling")
print(f" - Creates a classic storytelling voice")
print(f" - Action sequence: stood → rose → looked → spoke → knelt")
In [14]:
# Proper nouns
proper_nouns = df[df['pos'].isin(['NNP', 'NNPS'])]['word'].tolist()
print("\nPROPER NOUNS:")
print(f"Found: {', '.join(set(proper_nouns))}")
print(f"\nINTERPRETATION:")
print(f" - Establishes specific characters and places in Tolkien's world")
print(f" - Grounds the narrative in the fantasy setting")
4. Additional Observations and Interpretations¶
NARRATIVE STRUCTURE:
- The passage uses descriptive narrative prose
- Past tense dominance creates a storytelling atmosphere
- Builds from description to action (stood → rose → spoke → knelt)
EMOTIONAL TONE:
- Adjectives like 'great', 'brave', 'noble' create reverence
- Contrasting words ('small' vs 'tall', 'darkness' vs 'hope')
- Emotional words: love, pride, courage, honor
IMAGERY AND DESCRIPTION:
- Rich in concrete nouns (throne, hall, warriors, lords)
- Creates visual scene through noun-heavy descriptions
- Physical positioning emphasized (stood, knelt, bowing)
THEMATIC ELEMENTS:
- Reversal of status (King bowing to hobbits)
- Recognition and honor as central themes
- Contrast between physical size and moral greatness
WRITING STYLE:
- Balanced use of nouns and verbs creates dynamic description
- Epic, formal narrative style befitting fantasy literature
- Selective but powerful adjectives carry emotional weight
Summary¶
This POS tagging analysis reveals that the passage from The Return of the King is noun-heavy, which is characteristic of descriptive narrative prose. The high frequency of nouns establishes characters, settings, and concrete imagery that builds the epic scene.
Key Findings:
- Dominant Type: Nouns dominate the passage, creating vivid scenes and establishing characters
- Verb Usage: Past tense verbs create a storytelling atmosphere with sequential action
- Adjective Pattern: Selective but powerful adjectives (great, terrible, brave, noble) create emotional depth and contrast
- Literary Style: Epic, formal narrative with balanced description and action
Interpretation:
The POS analysis demonstrates how Tolkien's grammatical choices support the passage's themes:
- Noun-heavy construction builds vivid scenes
- Contrasting imagery (small/tall, darkness/hope) emphasizes themes
- Past tense narrative creates classic storytelling voice
- Formal, elevated language appropriate to the climactic, ceremonial scene
This iconic moment uses grammatical structure to convey honor, recognition, and the reversal of expectations, where the King bows to the hobbits, acknowledging that true greatness comes not from physical stature but from courage and sacrifice.
Named Entity Recognition¶
In [15]:
original_tokens = word_tokenize(corpus)
original_pos_tags = pos_tag(original_tokens)
In [19]:
# Perform NER using NLTK's ne_chunk
ne_tree = ne_chunk(original_pos_tags)
print("\nNER TREE STRUCTURE:")
print(ne_tree)
In [20]:
entities = []
for subtree in ne_tree:
if hasattr(subtree, 'label'):
entity_name = ' '.join([word for word, tag in subtree.leaves()])
entity_type = subtree.label()
entities.append((entity_name, entity_type))
print("EXTRACTED NAMED ENTITIES:")
if entities:
for entity, entity_type in entities:
print(f" {entity:<20} → {entity_type}")
else:
print(" No named entities detected by NLTK's default NER")
In [21]:
if entities:
entity_types = Counter([ent_type for _, ent_type in entities])
print("ENTITY TYPE DISTRIBUTION:")
for ent_type, count in entity_types.items():
print(f" {ent_type}: {count}")
In [22]:
if entities:
entity_df = pd.DataFrame(entities, columns=['Entity', 'Type'])
print("ENTITY DATAFRAME:")
print(entity_df)
For fantasy/literary texts, NLTK's default NER may not recognize fictional entities. Manual analysis of this corpus reveals:
PERSONS:
- Aragorn (protagonist, King)
- hobbits (main characters - collective reference)
- halflings (alternate term for hobbits)
LOCATIONS:
- Middle-earth (fictional continent)
- Gondor (kingdom)
- throne (location reference)
- hall (location reference)
GROUPS/ORGANIZATIONS:
- assembly (gathered group)
- court (royal court)
- warriors (group reference)
- lords (group reference)
INTERPRETATION:
- The text focuses on 2 main character types: hobbits and Aragorn
- 2 major locations are referenced: Middle-earth (macro) and Gondor (specific)
- The scene depicts a formal ceremony with multiple groups present
- This is a climactic recognition scene from Tolkien's "The Return of the King"