Chunk a PDF for Ingestion into a Knowledge Base

LlamaIndex nodes from the chunk JSON

Build TextNodes with page metadata and a content-stable id so re-ingestion is safe.

from llama_index.core.schema import TextNode
import json, hashlib
chunks = json.load(open('handbook.chunks.json'))
nodes = [TextNode(
    text=c['text'],
    id_='hb-'+hashlib.sha1(c['text'].encode()).hexdigest()[:12],
    metadata={'page': c['page'], 'page_range': c['pageRange']}
) for c in chunks]
index = VectorStoreIndex(nodes)

LangChain documents with page metadata

Each chunk becomes a Document; page fields ride along for citation.

from langchain_core.documents import Document
docs = [Document(
    page_content=c['text'],
    metadata={'page': c['page'],
              'page_range': c['pageRange'],
              'source': 'handbook.pdf'}
) for c in chunks]
vectorstore.add_documents(docs)

Citing the source page in a KB answer

Because every chunk carries provenance, the assistant can ground its answer.

Retrieved node metadata: { page: 12, page_range: [12, 13] }

System prompt: "Answer from the context and cite
  the handbook page as [p.N]."
Assistant: "PTO accrues at 1.5 days/month [p.12]."
# a reviewer can open p.12 and verify.

Tables go to a structured index, not the prose chunks

Keep tabular knowledge as structured records so numeric lookups stay accurate.

# prose -> pdf-to-chunks -> text nodes
# tables -> pdf-table-to-json -> structured rows
# index both; route table-style questions to the
# structured store, narrative questions to the chunks.

Re-ingesting an updated handbook safely

Use content-stable ids and a version tag so an updated PDF doesn't orphan old nodes.

version = 'handbook-2026Q2'
for c in chunks:
    node_id = version + '-' + sha1(c['text'])[:12]
# delete old version's nodes, then upsert new ones,
# rather than relying on the chunker's positional id.

Managed assistants like OpenAI's file search chunk on upload. Running this tool first is unnecessary unless you specifically want your own boundaries and page metadata. Use it for custom pipelines (LlamaIndex, LangChain, bespoke search) where you own chunking.

id renumbers whenever you re-chunk, so a handbook update can shuffle ids and orphan or overwrite the wrong nodes. Derive node ids from document id + content hash, scoped to a version tag.

There's no OCR. A scanned policy or runbook yields empty chunks, leaving gaps in your KB. OCR with PDF OCR before chunking image-only documents.

Benefit schedules, rate cards, and spec tables lack sentence punctuation and chunk poorly, so numeric lookups miss. Extract them with PDF Table to JSON and index the structured rows separately.

Running headers ('Confidential — Internal Use') and footers get pulled into the text on every page and dilute embeddings. Strip boilerplate before ingestion, or post-process the chunk text, since the chunker keeps everything pdf.js extracts.

Two-column manuals can interleave columns in a chunk because extraction follows content order, not reading order. Verify a sample and reflow the source if needed before building the KB.

The tokens field is a word count for budgeting, not the BPE count your embedder or LLM uses. Multiply by ~1.3 when checking chunk sizes against a model limit.

For KBs that benefit from denser overlap between neighbouring nodes, the browser's fixed 50-word overlap is the only option; use the API where overlap is configurable (capped at half the chunk size).

Free allows 50 pages / 2 MB; Pro 500 pages / 50 MB. A thick handbook hits the page ceiling first. Split with PDF Split (Fixed) and ingest each part, or upgrade the plan.