Topics Covered

On this page

1. 1.1 What NLP is, and what makes it hard
2. 1.2 Applications, and what each one actually requires
3. 1.3 Ambiguity — the core of the unit
4. 1.4 The toolchain
5. 1.5 Regular expressions
6. What to be able to do after this unit

Syllabus topics: Definition, goals and scope of NLP. Real-world applications (assistants, chatbots, translation, summarization, QA, spam detection). Fundamentals of language processing. Ambiguities in NLP (lexical, structural, contextual). Installations: Python setup, NLTK, spaCy basics. Regular expressions (essential patterns, findall, split, sub, matching tokens).

---

1.1 What NLP is, and what makes it hard

---

1.2 Applications, and what each one actually requires

Application	The hard part
Virtual assistants	intent classification plus slot filling, under speech-recognition errors
Chatbots	retrieval (find an existing answer) or generation (write one) — very different systems
Machine translation	word order differs between languages; the encoder-decoder was invented for this
Summarization	extractive is safe, abstractive can hallucinate (Unit 5)
Question answering	extractive QA finds a span; generative QA can invent one
Spam detection	an adversarial problem — the spammer adapts to your classifier

---

1.3 Ambiguity — the core of the unit

The garden-path sentences worth knowing

Sentence	Why it breaks you
"The old man the boats."	"man" is the verb; "the old" is the subject
"Time flies like an arrow."	"time" noun or imperative verb; "flies" verb or noun; "like" preposition or verb
"The horse raced past the barn fell."	"raced" is a reduced relative — "that was raced"

These are grammatical. Your parser fails on them for the same reason you do: it commits to the likely reading early and cannot cheaply revise.

---

1.4 The toolchain

The pipeline, which is the thing to understand about spaCy

nlp = spacy.load("en_core_web_sm")
print(nlp.pipe_names)
# ['tok2vec', 'tagger', 'parser', 'attribute_ruler', 'lemmatizer', 'ner']

Each name is a component that runs in order on every document: the tagger before the parser, the parser before the entity recogniser.

When spaCy is slow, the fix is nlp.pipe(texts, disable=[...]) — turn off the components you are not using. Doing NER only? Disable the parser.

---

1.5 Regular expressions

⚠️ Greedy against lazy — the single most common regex bug

01_setup_regex.py measures it on <b>bold</b> and <i>italic</i>:

Pattern	Result
<.*>	['<b>bold</b> and <i>italic</i>'] — one match, the whole line
<.*?>	['<b>', '</b>', '<i>', '</i>']

* is greedy: it takes as much as it can and gives back only what it must. *? is lazy and stops at the first >. One character, entirely different behaviour.

⚠️ And the email pattern everyone writes first

Pattern	On asha.reddy@nrigroup.ac.in
\w+@\w+	reddy@nrigroup
[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}	the whole address

\w does not match . or -, so the naive pattern truncates silently. The output still looks like a list of emails, which is exactly why this bug survives into production.

⚠️ Where regex stops being the right tool

The lab splits two sentences on ".":

True sentences	Naive split gives
2	5
1	4

"Dr.", "p.m.", "B.Sc." and "Rs." all contain a full stop that does not end a sentence.

---

What to be able to do after this unit

• [ ] Define NLP and give the three properties that make language hard
• [ ] List the levels of analysis and place a technique at the right one
• [ ] Distinguish lexical, structural and contextual ambiguity, with an example of each
• [ ] Explain why "The old man the boats" is grammatical
• [ ] Say when to use NLTK and when to use spaCy
• [ ] Explain what spaCy's pipeline is and how to speed it up
• [ ] Write patterns for emails, dates, hashtags and phone numbers
• [ ] Explain greedy vs lazy quantifiers and give the <.*> example
• [ ] Say why splitting on "." is wrong, and what to use instead

Cross-check yourself: run 01_setup_regex.py and 03_ambiguity_tokenize.py. Every figure in this unit is printed by one of them.