1. What is Named Entity Recognition?

Named Entity Recognition (NER) is an NLP task that automatically identifies and classifies named real-world entities mentioned in text into predefined categories.

Example: Given the sentence — “Apple Inc. CEO Tim Cook announced the new iPhone in Cupertino on September 9th, valued at $999.”

NER would extract:

• “Apple Inc.” → ORG (Organisation)
• “Tim Cook” → PERSON
• “iPhone” → PRODUCT
• “Cupertino” → LOC (Location)
• “September 9th” → DATE
• “$999” → MONEY

NER is a foundational NLP task — it is a prerequisite for many downstream applications including information extraction, question answering, knowledge graph construction, and document summarisation.

2. Common Entity Types

Different NER systems use different entity type sets. spaCy’s English model uses 18 types; CoNLL-2003 (the standard benchmark) uses only 4: PER, ORG, LOC, MISC.

3. BIO Tagging Scheme

NER is typically framed as a sequence labelling problem — assign one label to each token in the input. The BIO (Beginning-Inside-Outside) scheme handles multi-word entities:

• B-TYPE: Beginning of an entity of TYPE
• I-TYPE: Inside (continuation) of an entity of TYPE
• O: Outside any named entity

Example: “Jawaharlal Nehru University hosted the event”

Extended schemes include BIOES (adding E for End and S for Single-token entity) which can improve model performance by providing more boundary information.

4. Approaches to NER

Rule-Based

Handcrafted rules using regular expressions, dictionaries (gazetteers), and grammatical patterns. Example: “any word following ‘Mr.’ or ‘Dr.’ is likely a PERSON.” Fast, interpretable, no training data needed. But brittle — fails on unseen patterns and requires constant manual updates.

Classical ML — CRF (Conditional Random Field)

CRF is the traditional ML approach for NER. It models the conditional probability of the entire label sequence given the input, considering dependencies between adjacent labels (e.g., I-ORG cannot follow B-PER). Features include: current word, POS tag, word shape, prefix/suffix, surrounding words. Requires feature engineering but achieves good results on small datasets.

Deep Learning — BiLSTM-CRF

The dominant approach before transformers: a bidirectional LSTM processes the sequence and produces contextual representations, which are then fed into a CRF layer for sequence-level decoding. Learns features automatically from data — no manual feature engineering.

Transformer-Based — BERT Fine-Tuning

Current state-of-the-art: fine-tune BERT (or domain-specific models like BioBERT for medicine, LegalBERT for law) on NER-labelled data. BERT’s contextual embeddings capture nuanced entity semantics — “Apple” as a company vs “apple” as a fruit. Achieves F1 scores above 90% on standard benchmarks.

5. Evaluation Metrics

NER is evaluated at the entity level, not at the token level. A prediction counts as correct only if both the entity span (all tokens) and the entity type are exactly correct.

State-of-the-art models achieve F1 ~93% on CoNLL-2003 English NER. Domain-specific NER (medical, legal) typically achieves 80–88% F1 due to specialised terminology and ambiguity.

6. Applications

7. Python Code

# Install: pip install spacy transformers
# Download model: python -m spacy download en_core_web_sm

import spacy
from transformers import pipeline

# ============================================================
# Approach 1: spaCy (fast, production-ready)
# ============================================================
nlp = spacy.load("en_core_web_sm")

text = """
ISRO launched the Chandrayaan-3 spacecraft on July 14, 2023, from
Sriharikota, Andhra Pradesh. The mission cost approximately 615 crore rupees.
Dr. S. Somanath led the team at the Indian Space Research Organisation.
"""

doc = nlp(text)

print("Named Entities found:")
print("-" * 50)
for ent in doc.ents:
    print(f"  Text: {ent.text:<35} Label: {ent.label_:<10} ({spacy.explain(ent.label_)})")

# Visualise in Jupyter Notebook:
# from spacy import displacy
# displacy.render(doc, style="ent", jupyter=True)

# ============================================================
# Approach 2: HuggingFace Transformers (BERT-based, highest accuracy)
# ============================================================
ner_pipeline = pipeline(
    "ner",
    model="dbmdz/bert-large-cased-finetuned-conll03-english",
    aggregation_strategy="simple"  # Merge B/I tokens into full entities
)

text2 = "Sundar Pichai, CEO of Google, announced new AI features in San Francisco."
entities = ner_pipeline(text2)

print("\nBERT NER Results:")
print("-" * 50)
for entity in entities:
    print(f"  Entity: {entity['word']:<25} Type: {entity['entity_group']:<10} Score: {entity['score']:.3f}")

# ============================================================
# Approach 3: spaCy with custom entity training (outline)
# ============================================================
# For custom entity types not in the pre-trained model:
# 1. Annotate training data with your custom entities
# 2. Load blank or existing spaCy model
# 3. Add 'ner' component with your custom labels
# 4. Train with nlp.update() on annotated examples
# 5. Save model with nlp.to_disk()

# Example annotation format:
TRAIN_DATA = [
    ("The RTX 4090 GPU is powerful", {"entities": [(4, 13, "PRODUCT")]}),
    ("Python 3.12 was released last year", {"entities": [(0, 10, "PRODUCT")]}),
]
# Use spaCy's DocBin format and train with: python -m spacy train config.cfg
    

8. Common Mistakes Students Make

• Evaluating at token level instead of entity level: A model that correctly tags "New" and "York" individually but misses that they form a single "New York" entity should not get full credit. Always evaluate at the entity span level.
• Ignoring ambiguity: "Apple" can be a company (ORG) or a fruit (no entity). Context is essential — rule-based systems fail badly on ambiguous entities. Use contextual models (BERT) for production.
• Using a general model for domain-specific text: A model trained on news text will perform poorly on medical reports or legal documents. Use domain-specific pre-trained models (BioBERT, ClinicalBERT, LegalBERT) and fine-tune on domain data.

9. Frequently Asked Questions