Unit 3 — Information Extraction and Representation

IN DEPTH

What it is

Find the spans of text that name a thing, and say what kind of thing.

GPE vs LOC is examined. A GPE has a government; a LOC does not. Karnataka is a GPE, the Western Ghats is a LOC.

IN DEPTH

The tagging scheme

NER is a sequence labelling problem, and the standard encoding is BIO:

B begins an entity, I continues it, O is outside. Without B/I you cannot tell two adjacent entities from one long one — which is the reason the scheme exists.

NOTE

🎯 This is domain shift, and it is the single most important thing to

know before using a pre-trained NLP model on Indian text

The model is not broken. It is being used outside its training distribution, and the errors are systematic rather than random — which means you can predict and fix them.

KEY INSIGHT

And the fix is not a bigger model

The lab adds a gazetteer through spaCy's EntityRuler — nine lines of patterns naming Indian cities and states — and re-scores:

ruler = nlp.add_pipe("entity_ruler", before="ner")
ruler.add_patterns([{"label": "GPE", "pattern": p} for p in places])

When your domain has a closed list of entities, matching it beats any amount of inference. A gazetteer is unglamorous and very often correct. The alternative — annotating a few hundred sentences of your own domain and fine-tuning — is the right answer when the list is open.

NOTE

That is the most common defect in a hand-built evaluation set: the annotator marks what they were thinking about and misses the rest, and every model is then punished for finding it. Annotate exhaustively, or state explicitly which categories you covered.

IN DEPTH

Bag of words

Count each word; discard order. A document becomes a vector as long as the vocabulary.

NOTE

This is exactly the failure word embeddings were invented for.

IN DEPTH

Word2Vec — the two architectures

Remember them by direction: CBOW collapses context into a word; skip-gram fans a word out into its context.

IN DEPTH

How it is trained, and why negative sampling exists

The naive objective requires a softmax over the whole vocabulary at every step — hundreds of thousands of exponentials per training example.

Negative sampling replaces it: for each true (word, context) pair, draw k random "negative" pairs and train a binary classifier to tell them apart. The cost drops from |V| to k + 1, typically 5–20 instead of 500,000.

"Why negative sampling?" — because the full softmax over the vocabulary is computationally impossible. That is the answer.

IN DEPTH

GloVe, and how it differs

Word2Vec is predictive and works locally, one window at a time. GloVe is count-based: build the global word–word co-occurrence matrix, then factorise it so that the dot product of two vectors approximates the log of their co-occurrence count.

In practice they perform similarly. The distinction to state is local prediction against global co-occurrence factorisation.

IN DEPTH

The steps, in order

raw text
  → tokenize
  → normalise (lowercase, optionally stem/lemmatise)
  → vectorise (BoW / TF-IDF / embeddings)
  → classify (Naive Bayes / logistic regression / neural)
  → evaluate AGAINST A BASELINE

IN DEPTH

Sentiment analysis as an application of the pipeline

Sentiment analysis is the pipeline above with a polarity label, and it is the one application where the preprocessing advice from Unit 2 inverts: keep the stopwords, because not carries the label.

KEY INSIGHT

The cheapest sanity check in all of NLP

A linear model hands you its coefficients. Print the ten most positive and ten most negative features.

• If the strongest features are sentiment words, the model learned the task.
• If they are punctuation, a film title, or a reviewer's name, it learned an artefact of the corpus and will not transfer.

Neural Networks and Deep Learning's deep models do not offer this, which is why that course had to build a dataset with a known answer to get the same check.

NOTE

The general principle: every preprocessing step discards information, and you must be able to say what it discards and for whom that matters. "It is standard practice" is not an answer.