The NLP Workflow: What "Correct" Looks Like

A practitioner on your team ships a sentiment analysis pipeline. It was AI-generated, runs without errors, and returns scores that look plausible on the examples they tested. Three weeks later, production reports arrive: the model is systematically wrong on short reviews and neutral on anything with negation. The pipeline was never broken — it just used a different tokeniser at inference time than the one used to build the vocabulary at training time. Every prediction was computed against a vocabulary the model had never seen.

This is the most common class of NLP failure: not a crash, not an obvious error, but a silent mismatch that produces confidently wrong outputs. Your job as a practitioner is to know where these mismatches happen and what to look for.

This lesson maps the five stages every NLP workflow passes through — and identifies the checkpoint at each stage where AI tools most reliably introduce silent errors.

The five-stage NLP workflow

Every NLP task, from spam detection to semantic search to machine translation, passes through the same five stages:

Raw text → Tokens → Representation → Model → Evaluation

Understanding this pipeline is not about memorising steps. It is about having a mental model of where the state of your data changes — and therefore where a mismatch between training and inference can silently corrupt your results.

The same stages in an LLM pipeline

If you are using an LLM API (GPT, Claude, LLaMA), the pipeline is the same — but some stages are hidden inside the model:

Stage	Classical NLP (this course)	LLM API
1. Raw text	Encoding, HTML strip, normalisation	Same preprocessing still needed before sending
2. Tokens	Tokeniser (spaCy, NLTK, regex)	Subword BPE/WordPiece (inside the LLM)
3. Representation	Vectors (TF-IDF, embeddings)	Contextual embeddings from attention (inside the LLM)
4. Model	Classifier, similarity scorer, etc.	LLM API call or fine-tuned model
5. Evaluation	BLEU, ROUGE, BERTScore, F1, etc.	Same metrics—LLM outputs must be evaluated the same way

This lesson teaches the classical pipeline so you understand what is happening inside. When you use an LLM, the representation and tokenisation are black-box — but the bugs this course covers still happen at the boundaries (preprocessing before sending, evaluation of outputs).

Stage 1: Raw text

Your input. This stage includes everything before any processing: encoding (UTF-8, Latin-1), whitespace normalisation, HTML stripping, lowercasing decisions, and handling of special characters.

This stage is where most practitioners underinvest. AI-generated pipelines frequently skip it entirely, assuming the input is already clean. In production, text is never clean. Customer reviews contain emoji, URLs, and mixed encodings. Medical notes contain abbreviations and non-standard punctuation. Legal documents contain structured headers that should not be treated as sentence content.

The checkpoint: Does your preprocessing at inference time match exactly what you did at training time? If your training data was lowercased and HTML-stripped and your inference pipeline is not — or vice versa — your vocabulary will not match.

Stage 2: Tokens

Tokenisation converts raw text into discrete units — words, subwords, or characters — that the rest of the pipeline can process. This stage includes decisions about:

• Whether to split on whitespace, punctuation, or subword boundaries
• Whether to remove stopwords (and which stopword list to use)
• Whether to stem or lemmatise
• Whether to handle contractions ("don't" → "do not" or "dont"?)

These decisions are not cosmetic. Removing the word "not" before sentiment classification inverts the polarity of negated phrases. Using whitespace tokenisation on German text loses compound-word semantics. Using a general-purpose tokeniser on medical abbreviations produces a broken vocabulary.

The checkpoint: The tokeniser used at inference time must be identical to the one used at training time, including all its configuration options. If they differ, your vocabulary is wrong.

Stage 3: Representation

Tokens must be converted to numbers before a model can use them. The representation choice determines what information the model can see:

• Bag-of-words / TF-IDF: Counts or weighted counts. Sparse. No word order. No semantics.
• Word embeddings (Word2Vec, GloVe, fastText): Dense vectors. Captures some semantic relationships. No context — "bank" has one vector regardless of whether it means a riverbank or a financial institution.
• Sentence/document embeddings (sentence-transformers): Dense vectors for the whole input. Context-aware. Computationally heavier.

The representation stage is where the vocabulary is defined. Fitting a vectorizer on your full dataset — rather than only on training data — leaks information about the test set into the vocabulary. This is covered in depth in Lesson 3 and Lesson 5.

The checkpoint: Whatever fits your representation (vectorizer, embedding model) must be fit only on training data, then applied consistently at inference.

Stage 4: Model

The model receives numerical representations and produces outputs. At this stage, NLP is largely the same as any supervised learning task. The concerns from ML201 apply: correct evaluation, no leakage, calibrated probabilities, appropriate metrics.

The additional NLP concern here is that text data often has natural structure — documents from the same source, temporal ordering in reviews, class imbalance in categories — that standard random splits do not respect. A model that has seen some sentences from a document in training has an unfair advantage on other sentences from the same document in the test set.

The checkpoint: Are your train/test splits respecting the natural structure of your text data?

Stage 5: Evaluation

NLP introduces evaluation metrics that do not exist in standard ML: BLEU and ROUGE for generation tasks, BERTScore for semantic evaluation, entity-level F1 for named entity recognition. Each has failure modes where it reports high scores on outputs that are qualitatively wrong.

This stage is the primary way you audit AI-generated NLP pipelines. You are not evaluating whether the code runs — you are evaluating whether the outputs are correct on your data, using metrics appropriate to your task.

The checkpoint: Are you using the right metric for your task? Are you evaluating on a held-out test set that your model (and preprocessing) has never touched?

What this course is for

You will not learn to write NLP pipelines from scratch. You will learn to read them, direct AI tools to generate them, and audit them against the five checkpoints above.

By the end of this course, when an AI tool generates an NLP pipeline, you will know:

• Which stage to look at first
• What the specific failure modes are at each stage
• What correct looks like, so you can catch what is wrong

The next lesson opens at Stage 2: tokenisation — where most silent failures begin.

Summary

• Every NLP pipeline passes through five stages: raw text, tokens, representation, model, evaluation
• The most common failure is a silent mismatch between training-time and inference-time processing — not a crash, but systematically wrong outputs
• Each stage has a specific checkpoint; knowing these checkpoints is the core skill this course develops
• Next: Tokenisation and Normalisation — what AI tools change without telling you