# Chunking Strategies in RAG: A Deep Dive

RAG systems live or die on one decision that happens before any embedding is computed: how the source document gets cut into pieces. This post walks through three chunking mechanisms available in LangChain, traces exactly how each one decides where to cut, and reports what happened when all three were run against a real 2,209-word blog post rather than a toy paragraph.

## Introduction

Three strategies are covered:

1.  `CharacterTextSplitter` (fixed-size),
    
2.  `RecursiveCharacterTextSplitter` (recursive), and
    
3.  `SemanticChunker` (semantic).
    
4.  They differ in exactly one thing: **what signal they use to decide where a cut belongs.** Fixed-size uses a raw character count. Recursive uses a priority list of structural separators. Semantic uses embedding similarity between sentences.
    

> ⏭️ If you already know what an embedding and cosine similarity are, skip to Big Picture.

## Background / Prerequisites

📝 **Terminology — Embedding**: a numeric vector representation of text, produced by a model, positioned in vector space such that semantically similar text produces nearby vectors.

📝 **Terminology — Cosine similarity / distance**: `similarity = cos(θ)` between two vectors, ranging from -1 to 1 (in practice 0 to 1 for text embeddings). `distance = 1 - similarity`. Used throughout `SemanticChunker` to measure how related two pieces of text are.

📝 **Terminology — chunk\_overlap**: the number of characters from the end of one chunk repeated at the start of the next, intended to preserve context that would otherwise be severed at a chunk boundary.

## Big Picture: How the Three Strategies Decide Where to Cut

![](https://cdn.hashnode.com/uploads/covers/616e87a8f1f4c944cc6b49a1/914afab4-7e27-413c-afa2-9eaf88fa1dc5.png align="center")

Each row down this diagram trades one thing for another: fixed-size gives up quality for speed and zero cost; recursive gives up a guarantee for a strong heuristic; semantic gives up a size guarantee entirely in exchange for meaning-awareness, at the cost of needing an embedding model at chunking time.

## Fixed-Size Chunking: `CharacterTextSplitter`

```python
from langchain_text_splitters import CharacterTextSplitter

splitter = CharacterTextSplitter(separator="", chunk_size=500, chunk_overlap=80)
chunks = splitter.split_text(document)
```

With `separator=""`, the splitter performs a hard cut every `chunk_size` characters, sliding forward by `chunk_size - chunk_overlap` each time. It has no concept of a word, sentence, or heading — the cut point is purely a character index.

**Measured result** *(real blog post, "How to Pick the Perfect Database,"* `chunk_size=500`*,* `chunk_overlap=80`*)*: 22 chunks, mean length 791 chars, only **5% ended on a sentence-terminating character** (`.`, `!`, `?`, `:`). One chunk boundary landed inside a heading marker, producing `"...ETL into an analytics store. ## Graph Databases — The Relationship Specialists A gr..."` split directly across two chunks.

⚠️ **Gotcha**: `chunk_overlap` in `CharacterTextSplitter` is a raw character count. It does not try to land on a word boundary — the repeated text at a chunk's start can itself begin mid-word.

## Recursive Chunking: `RecursiveCharacterTextSplitter`

```python
from langchain_text_splitters import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(
    chunk_size=800,
    chunk_overlap=100,
    separators=["\n## ", "\n### ", "\n\n", "\n", ". ", " ", ""],
)
chunks = splitter.split_text(document)
```

### Decision Flow

![](https://cdn.hashnode.com/uploads/covers/616e87a8f1f4c944cc6b49a1/54777c5a-55f6-4d5d-bde4-ce8a509c113a.png align="center")

The default separator list is `["\n\n", "\n", ". ", " ", ""]`, ordered from largest structural unit to smallest, with `""` (raw character split) as an unconditional fallback so the algorithm can never fail to produce a chunk under `chunk_size`.

💡 **Key Point**: reordering this list to match your document's actual structure is the single highest-leverage tuning knob for this splitter. Markdown content with real `##`/`###` headers benefits from putting `\n##` and `\n###` at the front, ahead of the generic paragraph/sentence defaults.

**Measured result** (same document, headers-first separator list, `chunk_size=800`, `chunk_overlap=100`): 30 chunks, mean length 511 chars, **100%** ended on a sentence-terminating character — up from 5% for fixed-size on comparable settings.

⚠️ **Gotcha — size ceiling persists**: recursive chunking respects structure but is still size-bound. The document's "Graph Databases" section (heading through the closing sentence of its "Common pitfalls" subsection) measures **1,277 characters**. At `chunk_size=700`, the splitter is forced to cut through this section regardless of separator ordering. At `chunk_size=1300`, the section survives as one chunk. This was verified directly, not assumed:

| chunk\_size | Section stays intact? |
| --- | --- |
| 700 | No |
| 1300 | Yes |
| 1500 | Yes |

## Semantic Chunking: `SemanticChunker`

### Concept: What "Semantic" Means Here

`SemanticChunker` has no `chunk_size` parameter. It instead measures the cosine distance between the embeddings of consecutive sentence groups and cuts wherever that distance exceeds a threshold — i.e., wherever the topic changes.

### Pipeline (Data Flow)

![](https://cdn.hashnode.com/uploads/covers/616e87a8f1f4c944cc6b49a1/ce083b0e-cd9e-4809-b8e4-ec3b090ea6f0.png align="center")

📝 **Terminology —** `buffer_size`: the number of neighboring sentences folded into each embedded unit. At `buffer_size=1`, `combined_sentence[i]` = `sentence[i-1] + sentence[i] + sentence[i+1]`. This means the distance between position `i` and `i+1` compares two overlapping windows that share most of their content — only the sentence falling off one edge and the new one entering the other edge actually differ. This is a deliberate smoothing mechanism: a short, low-information sentence is never embedded in isolation.

⚠️ **Gotcha**: `combined_sentence[i]` is what gets embedded, not the raw `sentence[i]`. Confusing the two leads to incorrect assumptions about what the similarity score is actually measuring.

### The Four `breakpoint_threshold_type` Options

| Type | Threshold formula | Sensitive to | Best for |
| --- | --- | --- | --- |
| `percentile` (default) | `np.percentile(distances, amount)` | rank, not magnitude | general default, unknown distribution |
| `standard_deviation` | `mean(distances) + amount × std(distances)` | absolute deviation from mean | uniform baseline + one dramatic shift |
| `interquartile` | `mean(distances) + amount × (Q3 − Q1)` | absolute deviation from a *robust* center | multiple shifts of varying magnitude |
| `gradient` | `np.percentile(np.gradient(distances), amount)` | rate of change, not raw distance | gradual topic drift (legal/scientific text) |

**Verified failure mode —** `standard_deviation` **vs** `interquartile` on distance sequence `[0.05, 0.06, 0.04, 0.90, 0.05, 0.06, 0.35, 0.05, 0.06, 0.04]` (two real topic breaks: one dramatic at index 3, one moderate at index 6):

*   `standard_deviation(1)`: threshold = 0.426 → breakpoints at `[3]` only. The dramatic outlier (0.90) inflates the mean and std enough that the moderate break (0.35) falls below threshold and is missed.
    
*   `interquartile(1.5)`: threshold = 0.181 (Q1=0.050, Q3=0.060, computed from the tight middle cluster, unaffected by either outlier) → breakpoints at `[3, 6]`. Both breaks are caught, because IQR's own spread measurement isn't corrupted by the outliers it's trying to detect.
    

### The Embedding Model Determines Correctness, Not Just Cost

`SemanticChunker` accepts any object implementing LangChain's `Embeddings` interface (`embed_documents`, `embed_query`). Two were tested:

**TF-IDF** (word-overlap statistics, used as a free/offline stand-in):

```plaintext
"The cat sat on the mat." vs "A feline was resting on the rug."
  → same meaning, zero shared content words → similarity: 0.000

"The bank raised interest rates today." vs "I sat by the river bank yesterday."
  → different meaning, one shared word ("bank") → similarity: 0.181
```

TF-IDF ranks the second, unrelated pair as more similar than the first, meaning-identical pair, purely on shared vocabulary. Run through `SemanticChunker`, this produces measurable structural noise: on the test document, chunk sizes ranged from **16 to 1,257 characters** (mean 411, std 342) — a single bulleted list item was isolated as its own 16-character chunk, while an entire section was merged into one 1,257-character chunk, because TF-IDF's similarity signal doesn't track actual topic boundaries reliably.

`HuggingFaceEmbeddings(model_name="BAAI/bge-small-en-v1.5")` — a 384-dimension neural embedding model trained to place semantically similar text near each other regardless of shared vocabulary:

```python
from langchain_huggingface import HuggingFaceEmbeddings

embeddings = HuggingFaceEmbeddings(
    model_name="BAAI/bge-small-en-v1.5",
    model_kwargs={"device": "cpu"},
    encode_kwargs={"normalize_embeddings": True},
)
```

⚠️ **Gotcha — environment dependency**: this requires network access to `huggingface.co` to download model weights (~130MB, one-time). In a network-restricted environment, this call fails; a working fallback pattern is shown in the referenced script (§8) that catches the download failure and substitutes TF-IDF so the pipeline still runs end-to-end for demonstration purposes.

## Comparison / Trade-offs

|  | Fixed-size | Recursive | Semantic |
| --- | --- | --- | --- |
| Size guarantee | Yes, exact | Yes, upper-bound | None |
| Structural awareness | None | Yes, via separator priority | Yes, via meaning |
| Cost at chunking time | None | None | Embedding calls (API cost or local inference) |
| Measured clean sentence-ending rate | 5% | 100% | 100% |
| Chunk size uniformity (CV) | 0.05 (most uniform) | 0.34 | 0.83 (least uniform, by design) |
| Dependent on embedding model quality | No | No | Yes — critically |

💡 **Key Point**: chunk size uniformity and chunk quality are different axes. Fixed-size had the lowest coefficient of variation (0.05) and the worst quality (5% clean endings) of the three. A tight size distribution is not evidence of good chunking.

## References

*   LangChain text splitters documentation (`langchain_text_splitters`)
    
*   LangChain experimental `SemanticChunker` source (`langchain_experimental`)
    
*   `BAAI/bge-small-en-v1.5` model card (HuggingFace) for dimension count and intended use
    
*   All measured percentages, chunk counts, and character counts in this post were produced by running the scripts in against my own blog post, "[***How to Pick the Perfect Database Without Losing Your Mind***](https://gauravbytes.dev/how-to-pick-the-perfect-database-without-losing-your-mind)" ([gauravbytes.dev](https://gauravbytes.dev)) — no numbers were estimated or fabricated.