http://ghafoorsblog.com/tags/tokenization/Recent content in Tokenization on Ghafoor's Personal BlogHugoennoreply@example.com (AG Sayyed)noreply@example.com (AG Sayyed)Copyright © 2024-2026 AG Sayyed. All Rights Reserved.Sat, 16 May 2026 17:37:05 +0100http://ghafoorsblog.com/courses/ibm/rag-agentic-ai-content/rag-agentic-ai-pcert/01-develop-genai-apps/01-module/003-nlp/Wed, 19 Nov 2025 14:42:43 +0000noreply@example.com (AG Sayyed)http://ghafoorsblog.com/courses/ibm/rag-agentic-ai-content/rag-agentic-ai-pcert/01-develop-genai-apps/01-module/003-nlp/<p class="lead text-primary"> This document explores natural language processing as the bridge between human communication and computer comprehension. Through a comprehensive examination of NLP techniques including tokenization, stemming, lemmatization, part of speech tagging, and named entity recognition, the discussion reveals how computers transform unstructured text into structured data for AI applications. </p> <hr> <h2 id="understanding-natural-language-processing">Understanding Natural Language Processing</h2> <p>Natural language processing occurs whenever humans communicate, and computers attempt to comprehend that communication. When listening to words and sentences, humans naturally form comprehension from the language structure. When computers perform this same task, it constitutes NLP or natural language processing.</p>http://ghafoorsblog.com/courses/ibm/fullstack-content/fullstack-pcert/14-generative-ai/01-module/005-tokens-in-genai/Wed, 20 Nov 2024 06:21:53 +0000noreply@example.com (AG Sayyed)http://ghafoorsblog.com/courses/ibm/fullstack-content/fullstack-pcert/14-generative-ai/01-module/005-tokens-in-genai/<p class="lead text-primary"> This document explains the concept of tokens in generative AI, detailing tokenization, processing, and strategies for optimizing AI model performance and cost. </p> <hr> <h2 id="tokens-in-generative-ai">Tokens in Generative AI</h2> <p>Tokens play a vital role in generative AI models, influencing how text is processed, generated, and priced. Understanding tokens and related concepts provides insights into their function and significance in AI systems.</p> <h3 id="definition-of-tokens">Definition of Tokens</h3> <ul> <li> <p>Tokens: Tokens are the fundamental units of text that AI models process. They can represent characters, words, subwords, or even punctuation. For instance, in the sentence &ldquo;AI is evolving,&rdquo; the tokens might be:</p>