Split a PDF into Sentence-Aware Chunks for RAG

Retrieval-Augmented Generation — a technique where an LLM retrieves relevant document chunks and uses them as context before generating an answer. These chunks are what you'd load into a vector database for that retrieval step.

Chunks target a configurable token count (default ~500) and split on sentence boundaries rather than mid-sentence, with a configurable overlap between consecutive chunks so context isn't lost at the seams. Each chunk records its page range and an estimated token count. Note: this is sentence-aware, not embedding-based topic segmentation.

It's sentence-aware with overlap — it respects sentence boundaries and keeps a sliding context window, which is what most RAG pipelines actually need. It does not compute embeddings to detect topic shifts, so it's not 'semantic' in the embedding sense. For most retrieval use cases the difference doesn't matter; if you need topic-boundary detection, post-process the chunks with your own embedding model.