Many retrieval-augmented generation (RAG) projects begin with a round number such as 500 tokens per chunk. The value looks reasonable, but it does not tell you whether a retriever will return the passage that contains the answer.

To choose a RAG chunk size, prepare representative questions with labeled answer passages and build a separate index for each chunk-size and overlap setting while keeping the retriever fixed. On a development set, measure whether the complete answer evidence appears in the first result (Hit@1) or the first three results (Hit@3); select the lowest-cost setting that meets your target, verify it once on a separate holdout set, and inspect every miss.

This tutorial runs that process on a small fictional shared-studio handbook. It tests four word-window settings with a transparent TF-IDF retriever, so you can study chunking without downloading a language model or using an API.

Before You Begin: Install and Inspect the Data

You need Python 3.10 or later and basic knowledge of lists, loops, and tables. A pandas DataFrame is a table with named columns. A retriever is the part of a RAG system that ranks passages for a question; generation happens after this step and is outside this experiment.

Create a virtual environment, activate it, and install the three packages used in the lesson. On macOS or Linux, run:

python -m venv .venv
source .venv/bin/activate
python -m pip install pandas scikit-learn matplotlib

On Windows PowerShell, use .venv\Scripts\Activate.ps1 for activation. The executed build used Python 3.13.2, pandas 3.0.3, scikit-learn 1.9.0, and Matplotlib 3.11.0.

Download these two input files and place them beside your Python program:

• fictional shared-studio handbook
• labeled retrieval questions

You can also inspect the complete outputs produced by the executed build:

• executed chunk-setting metrics
• executed per-question predictions

The handbook and questions are original synthetic teaching data released under CC0 1.0. Synthetic means they were written for this lesson. The studio, policies, equipment, section IDs, questions, and values are fictional. The other two files are derived outputs from the executed Python build.

Load the two inputs first. Printing identifiers rather than full paragraphs makes the structure easy to check:

import pandas as pd

sections = pd.read_csv("studio_handbook_sections.csv")
questions = pd.read_csv("retrieval_questions.csv")

print(
    sections[["document_id", "section_id", "heading"]]
    .head(4)
    .to_string(index=False)
)
print()
print(
    questions[["question_id", "expected_section", "split", "question"]]
    .head(4)
    .to_string(index=False)
)

The executed output begins as follows:

  document_id section_id                 heading
visitor_guide        V01 Quiet room reservations
visitor_guide        V02             Guest entry
visitor_guide        V03         Step-free route
visitor_guide        V04           Lost property

question_id expected_section split                                                        question
        Q01              V01   dev How late can I arrive before my quiet room booking is released?
        Q02              V02   dev                  When does a visitor's paper entry pass expire?
        Q03              V03  test                      Where is the wheelchair-friendly entrance?
        Q04              V04   dev             How long does the desk keep an ordinary found item?

There are 12 handbook sections and 12 questions. Eight questions form the development set, called dev. We use them to choose settings. Four questions form the holdout set, called test, which remains untouched until the choice is complete.

How Chunk Size Works in Retrieval

A chunk is a passage that the retriever can return. Fixed-size chunking moves a window across each document and saves the words inside that window. Overlap repeats words from the end of one window at the start of the next.

The experiment follows this evidence path:

handbook -> word windows -> TF-IDF index -> ranked chunks
                                              |
question -> labeled answer span --------------+-> Hit@1 / Hit@3

Each question has an evidence_span: the shortest handbook text that contains its answer. Q01, for example, labels within 12 minutes of its start. A result counts as Hit@1 only when the first-ranked chunk contains that complete span. It counts as Hit@3 when any of the first three chunks contains the span.

This rule is stricter than checking a section ID. A 30-word chunk may come from the correct quiet-room section but end after within 12, leaving minutes of its start in the next chunk. The topic is right, yet the retrieved passage cannot support the answer by itself.

Small and large windows create different risks:

• A small chunk contains less unrelated text, but it can cut an answer in two.
• A large chunk keeps more context, but extra topics may weaken ranking and consume more model context later.
• Overlap protects text near a boundary, but repeated words enlarge the index.

There is no universal best value. Document structure, question wording, embedding model, ranking method, and the amount of context sent to the generator all affect the result.

Build Traceable Word Windows

Keep the source section ID attached to every word before creating windows. This metadata does not decide the score, but it makes a failed retrieval easier to investigate.

The next function joins sections by document and stores each word with its section ID:

import re
from collections import defaultdict


def token_records_by_document(sections):
    documents = defaultdict(list)

    for row in sections.itertuples(index=False):
        words = re.findall(r"\S+", f"{row.heading}. {row.text}")
        documents[row.document_id].extend(
            (word, row.section_id) for word in words
        )

    return documents

Now move a fixed window through each document. The step is chunk_words - overlap_words, so a 50-word window with 10 words of overlap advances 40 words at a time:

def make_chunks(sections, chunk_words, overlap_words):
    if overlap_words >= chunk_words:
        raise ValueError("overlap_words must be smaller than chunk_words")

    chunks = []
    step = chunk_words - overlap_words

    for document_id, records in token_records_by_document(sections).items():
        for start in range(0, len(records), step):
            window = records[start : start + chunk_words]
            if not window:
                continue

            chunks.append({
                "chunk_id": f"{document_id}-{start:04d}",
                "document_id": document_id,
                "text": " ".join(word for word, _ in window),
                "section_ids": tuple(dict.fromkeys(
                    section_id for _, section_id in window
                )),
                "token_count": len(window),
            })

            if start + chunk_words >= len(records):
                break

    return chunks

The final short window is kept rather than discarded. Removing it would lose the end of every document. dict.fromkeys() preserves the order of section IDs while removing duplicates.

This lesson counts whitespace-separated words, not model tokens. A model tokenizer may represent text as whole words, parts of words, punctuation, or other model-specific units. Word windows keep the example readable, but a production test should use the tokenizer and limits of the embedding and generation models you plan to run.

Rank Chunks with One Fixed Retriever

Change only the chunk settings during this comparison. If the vectorizer or ranking method also changes, you will not know which change caused a different result.

We use TfidfVectorizer to convert passages and questions into numeric features. TF-IDF gives more weight to terms that are useful in a smaller number of passages. The current scikit-learn TfidfVectorizer reference describes the API as converting raw documents to a TF-IDF feature matrix.

The following function fits the vectorizer on chunks, transforms questions with the same vocabulary, and compares them with cosine similarity:

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity


def rank_chunks(chunks, questions):
    vectorizer = TfidfVectorizer(
        ngram_range=(1, 2),
        stop_words="english",
    )
    chunk_vectors = vectorizer.fit_transform(
        [chunk["text"] for chunk in chunks]
    )
    question_vectors = vectorizer.transform(questions["question"])
    return cosine_similarity(question_vectors, chunk_vectors)

Cosine similarity compares the angle between two numeric vectors. A larger value means their weighted terms point in a more similar direction. Scikit-learn’s current pairwise metrics guide notes that it is a common choice for TF-IDF document vectors.

TF-IDF is a lexical baseline: it relies on shared words and phrases. It will not reliably connect every synonym. That limitation is useful here because the experiment stays fast and repeatable. You can later replace only the retriever with your embedding system and retain the same labels and metrics.

Score Complete Answer Evidence

For each question, sort similarity scores from largest to smallest. Then check whether the normalized evidence span appears in the retrieved text.

This helper expresses that rule directly:

def contains_evidence(chunk, evidence_span):
    chunk_text = " ".join(chunk["text"].casefold().split())
    expected = " ".join(evidence_span.casefold().split())
    return expected in chunk_text


def score_question(chunks, score_row, evidence_span):
    ranked_indexes = score_row.argsort(kind="stable")[::-1]
    top_chunks = [chunks[index] for index in ranked_indexes[:3]]

    return {
        "hit_at_1": contains_evidence(top_chunks[0], evidence_span),
        "hit_at_3": any(
            contains_evidence(chunk, evidence_span)
            for chunk in top_chunks
        ),
        "top_chunk_id": top_chunks[0]["chunk_id"],
        "top_chunk_sections": "|".join(top_chunks[0]["section_ids"]),
    }

Exact span matching is appropriate because a person wrote the minimum span from this fixed handbook. It is not a general meaning evaluator. If several separate passages can answer a question, store several accepted spans and pass when a retrieved chunk contains any one of them.

The experiment compares these settings:

configs = [
    {"chunk_words": 30, "overlap_words": 0},
    {"chunk_words": 50, "overlap_words": 10},
    {"chunk_words": 80, "overlap_words": 15},
    {"chunk_words": 120, "overlap_words": 30},
]

For each setting, the complete build records Hit@1 and Hit@3 separately for dev and test. It also calculates indexed_token_ratio: all words stored across the chunks divided by the source word count. A ratio of 1.20 means overlap caused the index to store about 20% more word positions than the source contains.

Select on Development Data, Then Open the Holdout

Choose the configuration without reading the test scores. The executed selection ranks configurations by development Hit@1, then development Hit@3, and then prefers a lower indexed-token ratio when the retrieval scores tie:

selected = max(
    metric_rows,
    key=lambda row: (
        row["dev_hit_at_1"],
        row["dev_hit_at_3"],
        -row["indexed_token_ratio"],
    ),
)

This rule selected 80 words with 15 words of overlap. The complete executed table is:

                config  chunks  index ratio  dev Hit@1  dev Hit@3  test Hit@1  test Hit@3
  30 words / 0 overlap      29        1.000       0.125       0.625         0.00         0.75
 50 words / 10 overlap      22        1.228       0.750       1.000         0.75         1.00
 80 words / 15 overlap      14        1.198       1.000       1.000         1.00         1.00
120 words / 30 overlap       9        1.216       1.000       1.000         1.00         1.00

The 30-word setting placed complete evidence first for only one of eight development questions. Its top chunk often came from the expected section, but the labeled answer crossed a chunk boundary. This is why section-level relevance alone would have hidden the problem.

Both larger settings achieved perfect development scores. The tie-breaker selected 80/15 because its indexed-token ratio was 1.198, slightly below 1.216 for 120/30. Only then did we read the holdout results. All four holdout questions passed Hit@1 and Hit@3.

Read the left chart as retrieval evidence, not answer accuracy. It says whether the ranked chunk contained the labeled passage. It does not test how a language model would use that passage. The right chart shows one cost of overlap, but it does not measure latency, memory, or model context directly.

The 100% holdout score is encouraging only for these four synthetic questions. It is not a reliable estimate for a large knowledge base. Add more questions from the real search patterns, document formats, languages, and hard cases that your system must handle.

For a wider check after retrieval, see how to evaluate RAG answers without an LLM judge. That lesson keeps retrieval, citations, claim support, completeness, and abstention as separate measures.

Troubleshooting Chunk Experiments

Every setting scores 100%. Your questions may be too easy, or your labels may cover whole sections instead of the answer text. Add boundary cases, similar sections, short questions, and alternative wording. Keep a minimum evidence span that must be present.

Every setting scores poorly. Inspect the top chunks and their scores before changing chunk size. A lexical retriever cannot connect many synonyms. The issue may be retrieval features, missing documents, or incorrect labels rather than window length.

A top chunk has the right section but fails. Check where the evidence span was cut. More overlap or a larger window may help. Structure-aware splitting at headings or paragraphs may be better than forcing a larger fixed window.

Overlap is equal to or larger than the window. The step becomes zero or negative. Reject this configuration before the loop, as make_chunks() does.

The final part of a document disappears. Keep the shorter final window. Also test empty documents and documents shorter than one window.

The test set influences the choice. Looking at holdout results and then changing settings turns the holdout into development data. Record that decision and create a fresh holdout set before reporting a final check.

One configuration wins by a tiny amount. Review individual questions. A difference of one success in a small set may be unstable. Add cases, repeat the comparison on document groups, and include practical costs before choosing.

Exact evidence matching rejects a valid alternative. Store several reviewed answer spans or use claim-level human labels. Do not silently expand a span until a preferred configuration passes.

TF-IDF vocabulary is fitted separately for every setting. That is intentional because each setting creates a different chunk collection. Keep vectorizer options fixed, and never fit on the questions themselves.

The chart fails on a server. Select Matplotlib’s Agg backend before importing matplotlib.pyplot, then save the plot to a file. The current Matplotlib backend guide lists Agg as a non-interactive backend for file output.

Recap: Choose with Labels, Then Recheck

Chunk size is a retrieval setting, so choose it with retrieval evidence. Create representative questions, label the smallest passages that answer them, compare several sizes and overlaps with one fixed retriever, and keep accuracy separate from storage and context costs.

In this executed lesson, 30-word windows frequently found the right topic but split the answer passage. An 80-word window with 15-word overlap achieved 100% evidence Hit@1 on eight development questions and four holdout questions while indexing about 1.20 times the source word count. Those numbers describe this synthetic handbook, not a general recommendation.

Apply the method to your own documents next. Start with a small development set that you can inspect by eye, save every ranked chunk, and treat each miss as information about boundaries, labels, or retrieval—not as a reason to copy a chunk size from another system.