Browse Courses

What is NLP

July 10, 2025 4 min read Docs AI-Developer Speech Computer-Vision

Explains natural language processing (NLP), how it translates unstructured text into structured data, and the key steps and tools in the NLP pipeline with real-world use cases and examples.

On this page

This document explains natural language processing (NLP), how it translates unstructured human language into structured data, and the essential steps in the NLP pipeline. It covers real-world use cases, the difference between NLU and NLG, and the tools used to process language for AI applications.

Introduction to NLP

Natural language processing (NLP) is the field of artificial intelligence that enables computers to understand, interpret, and generate human language. While humans naturally comprehend spoken and written language, computers require specialized methods to process unstructured text and convert it into structured data.

Unstructured vs Structured Data

Unstructured text is everyday language as spoken or written by humans, such as “add eggs and milk to my shopping list.” Computers need this information in a structured format, for example:

1shopping_list:
2  - item: eggs
3  - item: milk

NLP acts as a bridge, translating between unstructured and structured data. Translating unstructured to structured is called natural language understanding (NLU), while the reverse is natural language generation (NLG).

Key Use Cases for NLP

Use CaseDescription
Machine TranslationConverts text or speech from one language to another, considering context.
Virtual AssistantsInterprets spoken or written commands to perform actions (e.g., Siri, Alexa)
ChatbotsProcesses written language to traverse decision trees and respond to users.
Sentiment AnalysisDetermines sentiment (positive, negative, sarcastic) in text.
Spam DetectionIdentifies unwanted or suspicious messages using content analysis.

The NLP Pipeline: From Text to Meaning

NLP uses a variety of tools and steps to process language:

1. Tokenization

Breaks text into smaller units called tokens (words or phrases).

2. Stemming

Reduces words to their root form by removing prefixes and suffixes (e.g., “running”, “runs”, “ran” → “run”).

3. Lemmatization

Finds the dictionary root (lemma) of a word, considering context and meaning (e.g., “better” → “good”).

4. Part of Speech (POS) Tagging

Identifies the grammatical role of each token (e.g., “make” as a verb or noun depending on context).

5. Named Entity Recognition (NER)

Detects entities such as names, places, or organizations (e.g., “Arizona” as a US state).

Example: NLP in Action

Given the unstructured text:

1add eggs and milk to my shopping list

The NLP pipeline processes it as follows:

  1. Tokenization: [add, eggs, and, milk, to, my, shopping, list]
  2. Stemming/Lemmatization: “eggs” → “egg”, “better” → “good”
  3. POS Tagging: “add” (verb), “milk” (noun)
  4. NER: “milk” (item), “shopping list” (object)
  5. Structured Output:
1shopping_list:
2  - item: eggs
3  - item: milk

Conclusion

NLP is a powerful set of tools and techniques that enables computers to process and understand human language. By converting unstructured text into structured data, NLP powers applications like translation, chatbots, sentiment analysis, and more.

FAQ

  1. Translating unstructured human language into structured data computers can process
  2. Increasing computer hardware speed
  3. Designing new programming languages
  4. Building physical robots
(1) NLP enables computers to process and understand human language by converting unstructured text into structured data.

Skipping tokenization would prevent the system from breaking text into manageable units, making it difficult to analyze or process language accurately.

ToolPurpose
A. Lemmatization1. Assigns grammatical roles to tokens
B. POS Tagging2. Finds the dictionary root of a word
C. NER3. Detects entities like names or places
D. Stemming4. Removes prefixes and suffixes
A-2, B-1, C-3, D-4.

  1. NLU converts unstructured text to structured data
  2. NLG converts structured data to unstructured text
  3. NLU and NLG are the same process
  4. Both are essential in NLP applications
(3) NLU and NLG are distinct processes; NLU interprets language, NLG generates it.

Context is crucial for accurate language understanding, as it helps distinguish between different meanings of the same word or phrase.

Stemming and lemmatization always produce the same result for every word.

False. Stemming and lemmatization can yield different results, especially for irregular words.

The quality and representativeness of the training data should be checked first to ensure the system can accurately identify spam.

  1. Tokenization
  2. Lemmatization
  3. Object detection
  4. Named entity recognition
(3) Object detection is a computer vision task, not part of the NLP pipeline.

The system tokenizes the sentence, applies stemming or lemmatization, tags parts of speech, recognizes entities, and outputs a structured shopping list.
NLP, Speech, Vision
Self-Driving Cars