Chunking Strategies in RAG: A Deep Dive

RAG systems live or die on one decision that happens before any embedding is computed: how the source document gets cut into pieces. This post walks through three chunking mechanisms available in LangChain, traces exactly how each one decides where to cut, and reports what happened when all three were run against a real 2,209-word blog post rather than a toy paragraph.

Introduction

Three strategies are covered:

1. CharacterTextSplitter (fixed-size),

2. RecursiveCharacterTextSplitter (recursive), and

3. SemanticChunker (semantic).

4. They differ in exactly one thing: what signal they use to decide where a cut belongs. Fixed-size uses a raw character count. Recursive uses a priority list of structural separators. Semantic uses embedding similarity between sentences.

⏭️ If you already know what an embedding and cosine similarity are, skip to Big Picture.

Background / Prerequisites

📝 Terminology — Embedding: a numeric vector representation of text, produced by a model, positioned in vector space such that semantically similar text produces nearby vectors.

📝 Terminology — Cosine similarity / distance: similarity = cos(θ) between two vectors, ranging from -1 to 1 (in practice 0 to 1 for text embeddings). distance = 1 - similarity. Used throughout SemanticChunker to measure how related two pieces of text are.

📝 Terminology — chunk_overlap: the number of characters from the end of one chunk repeated at the start of the next, intended to preserve context that would otherwise be severed at a chunk boundary.

Big Picture: How the Three Strategies Decide Where to Cut

Each row down this diagram trades one thing for another: fixed-size gives up quality for speed and zero cost; recursive gives up a guarantee for a strong heuristic; semantic gives up a size guarantee entirely in exchange for meaning-awareness, at the cost of needing an embedding model at chunking time.

Fixed-Size Chunking: CharacterTextSplitter

from langchain_text_splitters import CharacterTextSplitter

splitter = CharacterTextSplitter(separator="", chunk_size=500, chunk_overlap=80)
chunks = splitter.split_text(document)

With separator="", the splitter performs a hard cut every chunk_size characters, sliding forward by chunk_size - chunk_overlap each time. It has no concept of a word, sentence, or heading — the cut point is purely a character index.

Measured result (real blog post, "How to Pick the Perfect Database," chunk_size=500, chunk_overlap=80): 22 chunks, mean length 791 chars, only 5% ended on a sentence-terminating character (., !, ?, :). One chunk boundary landed inside a heading marker, producing "...ETL into an analytics store. ## Graph Databases — The Relationship Specialists A gr..." split directly across two chunks.

⚠️ Gotcha: chunk_overlap in CharacterTextSplitter is a raw character count. It does not try to land on a word boundary — the repeated text at a chunk's start can itself begin mid-word.

Recursive Chunking: RecursiveCharacterTextSplitter

from langchain_text_splitters import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(
    chunk_size=800,
    chunk_overlap=100,
    separators=["\n## ", "\n### ", "\n\n", "\n", ". ", " ", ""],
)
chunks = splitter.split_text(document)

Decision Flow

The default separator list is ["\n\n", "\n", ". ", " ", ""], ordered from largest structural unit to smallest, with "" (raw character split) as an unconditional fallback so the algorithm can never fail to produce a chunk under chunk_size.

💡 Key Point: reordering this list to match your document's actual structure is the single highest-leverage tuning knob for this splitter. Markdown content with real ##/### headers benefits from putting \n## and \n### at the front, ahead of the generic paragraph/sentence defaults.

Measured result (same document, headers-first separator list, chunk_size=800, chunk_overlap=100): 30 chunks, mean length 511 chars, 100% ended on a sentence-terminating character — up from 5% for fixed-size on comparable settings.

⚠️ Gotcha — size ceiling persists: recursive chunking respects structure but is still size-bound. The document's "Graph Databases" section (heading through the closing sentence of its "Common pitfalls" subsection) measures 1,277 characters. At chunk_size=700, the splitter is forced to cut through this section regardless of separator ordering. At chunk_size=1300, the section survives as one chunk. This was verified directly, not assumed:

Semantic Chunking: SemanticChunker

Concept: What "Semantic" Means Here

SemanticChunker has no chunk_size parameter. It instead measures the cosine distance between the embeddings of consecutive sentence groups and cuts wherever that distance exceeds a threshold — i.e., wherever the topic changes.

Pipeline (Data Flow)

📝 Terminology — buffer_size: the number of neighboring sentences folded into each embedded unit. At buffer_size=1, combined_sentence[i] = sentence[i-1] + sentence[i] + sentence[i+1]. This means the distance between position i and i+1 compares two overlapping windows that share most of their content — only the sentence falling off one edge and the new one entering the other edge actually differ. This is a deliberate smoothing mechanism: a short, low-information sentence is never embedded in isolation.

⚠️ Gotcha: combined_sentence[i] is what gets embedded, not the raw sentence[i]. Confusing the two leads to incorrect assumptions about what the similarity score is actually measuring.

The Four breakpoint_threshold_type Options

Verified failure mode — standard_deviation vs interquartile on distance sequence [0.05, 0.06, 0.04, 0.90, 0.05, 0.06, 0.35, 0.05, 0.06, 0.04] (two real topic breaks: one dramatic at index 3, one moderate at index 6):

• standard_deviation(1): threshold = 0.426 → breakpoints at [3] only. The dramatic outlier (0.90) inflates the mean and std enough that the moderate break (0.35) falls below threshold and is missed.

• interquartile(1.5): threshold = 0.181 (Q1=0.050, Q3=0.060, computed from the tight middle cluster, unaffected by either outlier) → breakpoints at [3, 6]. Both breaks are caught, because IQR's own spread measurement isn't corrupted by the outliers it's trying to detect.

The Embedding Model Determines Correctness, Not Just Cost

SemanticChunker accepts any object implementing LangChain's Embeddings interface (embed_documents, embed_query). Two were tested:

TF-IDF (word-overlap statistics, used as a free/offline stand-in):

"The cat sat on the mat." vs "A feline was resting on the rug."
  → same meaning, zero shared content words → similarity: 0.000

"The bank raised interest rates today." vs "I sat by the river bank yesterday."
  → different meaning, one shared word ("bank") → similarity: 0.181

TF-IDF ranks the second, unrelated pair as more similar than the first, meaning-identical pair, purely on shared vocabulary. Run through SemanticChunker, this produces measurable structural noise: on the test document, chunk sizes ranged from 16 to 1,257 characters (mean 411, std 342) — a single bulleted list item was isolated as its own 16-character chunk, while an entire section was merged into one 1,257-character chunk, because TF-IDF's similarity signal doesn't track actual topic boundaries reliably.

HuggingFaceEmbeddings(model_name="BAAI/bge-small-en-v1.5") — a 384-dimension neural embedding model trained to place semantically similar text near each other regardless of shared vocabulary:

from langchain_huggingface import HuggingFaceEmbeddings

embeddings = HuggingFaceEmbeddings(
    model_name="BAAI/bge-small-en-v1.5",
    model_kwargs={"device": "cpu"},
    encode_kwargs={"normalize_embeddings": True},
)

⚠️ Gotcha — environment dependency: this requires network access to huggingface.co to download model weights (~130MB, one-time). In a network-restricted environment, this call fails; a working fallback pattern is shown in the referenced script (§8) that catches the download failure and substitutes TF-IDF so the pipeline still runs end-to-end for demonstration purposes.

Comparison / Trade-offs

💡 Key Point: chunk size uniformity and chunk quality are different axes. Fixed-size had the lowest coefficient of variation (0.05) and the worst quality (5% clean endings) of the three. A tight size distribution is not evidence of good chunking.

References

• LangChain text splitters documentation (langchain_text_splitters)

• LangChain experimental SemanticChunker source (langchain_experimental)

• BAAI/bge-small-en-v1.5 model card (HuggingFace) for dimension count and intended use

• All measured percentages, chunk counts, and character counts in this post were produced by running the scripts in against my own blog post, "How to Pick the Perfect Database Without Losing Your Mind" (gauravbytes.dev) — no numbers were estimated or fabricated.