Dividing content into short passages before it is embedded and indexed. Oversized chunks mix unrelated sections and weaken meaning, so Azure AI Search advises around 512 tokens each, overlapping by about 10-25%.
Read more: Microsoft Learn
In the Ultra Transcenders books
Each book explains Chunking in context, with comparison tables and the common traps.
Terms in this definition
- Azure AI Search
Managed Azure service that builds indexes over your content and answers keyword, vector, hybrid and semantically ranked queries, optionally with AI enrichment. Retrieval-augmented generation, Foundry agents and knowledge mining all use it to fetch relevant content.
Related terms
- Advanced data parsing
Structure-aware document processing: it applies OCR to scanned pages, joins tables spanning pages, recovers headers and produces chunks tagged with heading, page and table metadata, in contrast to fixed-size or page-by-page chunking.
- Markdown output (Document Intelligence)
Option on the layout model that returns the extracted content as Markdown, rendering tables as HTML; handy when chunking documents for RAG.
- Text Merge skill
A utility skill whose output, merged_content, combines document content with OCR text or image captions. Chunking is not something it does.