Skip to main content

Ingestion

Ingestion is the process of taking a raw document, converting it to searchable text, splitting it into chunks, generating embeddings, and writing everything to the vector store. Understanding each sub-stage helps you choose the right parser, configure metadata correctly, and track progress in production.

The four stages​

IngestAsync progresses through four sequential stages. If you pass an IProgress<IngestionProgress>, a callback fires at the end of each stage:

StageIngestionProgressStage valueWhat happened
1ParsingDocument parsed into DocumentSection objects
2ChunkingSections split into TextChunk objects; metadata applied
3EmbeddingBatch embedding call completed
4StoringChunks written to the vector store and BM25 index

DocumentMetadata​

Every ingestion call requires a DocumentMetadata record:

public sealed record DocumentMetadata
{
public required string DocumentId { get; init; }
public required string FileName { get; init; }
public string? ContentType { get; init; }
public IDictionary<string, MetadataValue> Tags { get; init; }
= new Dictionary<string, MetadataValue>(StringComparer.Ordinal);
}
PropertyPurpose
DocumentIdStable identifier for the document. Used to delete or overwrite all its chunks. Must be unique per document across your corpus.
FileNameHuman-readable file name. Written into every chunk's Metadata["file_name"].
ContentTypeMIME type used for parser selection (e.g., "application/pdf"). Defaults to "text/plain" when null.
TagsArbitrary key-value pairs propagated into every chunk's Metadata. Use these for metadata filtering at query time.
var metadata = new DocumentMetadata
{
DocumentId = "policy-hr-001",
FileName = "hr-policy.docx",
ContentType = "application/vnd.openxmlformats-officedocument.wordprocessingml.document",
Tags = new Dictionary<string, MetadataValue>
{
["category"] = "hr",
["version"] = "2024-01",
},
};

After ingestion, every TextChunk.Metadata for this document will contain:

  • "document_id" → "policy-hr-001"
  • "file_name" → "hr-policy.docx"
  • "category" → "hr"
  • "version" → "2024-01"
  • Plus any heading metadata injected by the parser (see below)

Typed metadata values​

Metadata values are MetadataValue, a small discriminated value carrying one of four kinds — string, number (a double; int/long convert exactly up to 2^53), boolean, or date (DateTimeOffset, compared and stored by UTC instant). Implicit conversions exist from each carried type, so plain-string write sites keep their shape, and a number written as a number stays a number all the way into the vector store — filterable numerically, not as the text "3":

Tags = new Dictionary<string, MetadataValue>
{
["category"] = "hr", // string
["page_count"] = 42, // number
["confidential"] = true, // boolean
["approved_at"] = DateTimeOffset.UtcNow, // date
},

There is deliberately no implicit conversion back to string — that would silently stringify numbers at read sites, which is exactly the bug the typed value removes. Read through Kind and the kind-checked accessors (StringValue, NumberValue, BooleanValue, DateTimeOffsetValue), or ToString() for a lossless textual form of any kind.

The typing runs the whole chain: a data-provider connector's FileEntry.Metadata, DocumentMetadata.Tags, TextChunk.Metadata, SearchOptions.MetadataFilter, and each store's persisted representation (see Vector stores). Values stored before metadata carried types read back losslessly as string-kind values.

Chunk-level values win: MetadataBehavior applies document tags to chunks with TryAdd, so a key a chunking strategy already set is never clobbered by a document tag.

IngestionOptions​

public sealed class IngestionOptions
{
public bool Overwrite { get; set; }

/// <summary>
/// Maximum number of documents to ingest concurrently
/// when using IngestFromProviderAsync. Default is 1 (sequential).
/// </summary>
public int MaxDegreeOfParallelism { get; init; } = 1;

/// <summary>Chunks per embedding batch within a single document. Default 100.</summary>
public int EmbedBatchSize { get; init; } = 100;

/// <summary>Maximum embedding batches in flight concurrently per document. Default 2.</summary>
public int MaxConcurrentEmbeddingBatches { get; init; } = 2;
}

When Overwrite = true, the pipeline calls IVectorStore.DeleteByDocumentIdAsync before the document is parsed, purging the previous version's vectors outright. That vector-store delete is now the only thing Overwrite exclusively buys:

await pipeline.IngestAsync(stream, metadata,
options: new IngestionOptions { Overwrite = true });

Removal from the BM25 index and the data manager is unconditional — it happens on every ingest, whether or not Overwrite is set, immediately before the new postings are written. (Overwrite still removes from those two as well, but up front, before parsing, so its extra guarantee is narrow: a re-ingest whose new content fails to parse or yields no chunks leaves nothing of the old one behind.)

So the old advice — "without Overwrite, re-ingesting the same DocumentId accumulates duplicate chunks" — needs correcting in both directions:

StoreRe-ingest without OverwriteRe-ingest with Overwrite
BM25 indexClean replaceClean replace (purged earlier, before parsing)
IRagDataManagerClean replaceClean replace (purged earlier, before parsing)
Vector storePartial — upserted on (documentId, chunkIndex), so a shorter replacement strands the tailClean replace
Parent chunk storePartial — upserted on (documentId, parentChunkIndex)Partial — Overwrite does not purge parent chunks either; only IIngestor.DeleteAsync does

A 9-chunk document re-ingested as 5 chunks therefore leaves chunks 5–8 in the vector store and retrievable unless Overwrite is set. Set Overwrite on refresh operations where the document may have shrunk. You no longer need it to avoid duplicate BM25 postings.

BM25 scores changed with this correction. A corpus that had been re-ingested carried inflated term statistics from the duplicated postings; they are gone, so keyword and hybrid scores move. Re-baseline any score thresholds tuned against the old behaviour. See Re-ingest semantics.

MaxDegreeOfParallelism controls how many documents IngestFromProviderAsync processes concurrently. The default 1 preserves the previous sequential behaviour. Increase it when your vector store and embedding service can handle concurrent requests:

var result = await pipeline.IngestFromProviderAsync(provider, "my-corpus",
options: new IngestionOptions { MaxDegreeOfParallelism = 4 },
hashStore: hashStore);

A value of 4 is a reasonable starting point for most cloud embedding APIs. The optimal value depends on your embedding service's rate limits and your vector store's connection pool size.

Chunk-batch embedding​

Within a single document, chunks that need embedding are sliced into batches of EmbedBatchSize (default 100) and the batches are embedded concurrently, bounded by MaxConcurrentEmbeddingBatches (default 2). A document with at most EmbedBatchSize pending chunks is embedded in one generator call, exactly as before — batching only kicks in for larger documents. Chunk order and precomputed embeddings are always preserved; results are reassembled by original chunk index. Tune EmbedBatchSize to your embedding API's maximum inputs per request, and raise MaxConcurrentEmbeddingBatches when the service tolerates more parallel requests. Both values must be greater than zero. Note that MaxDegreeOfParallelism (documents) and MaxConcurrentEmbeddingBatches (batches per document) multiply: with 4 × 2 you can have up to eight embedding requests in flight. This changed the default behaviour for documents with more than 100 pending chunks: previously they were embedded in a single request, now in up to 2 concurrent requests of at most 100 chunks each — operators with strict embedding-API rate limits can set MaxConcurrentEmbeddingBatches = 1 to keep requests sequential.

Concurrent ingestion of the same DocumentId is not supported. The BM25 index update and vector store write are not transactional, and there is no per-DocumentId lock anywhere in the pipeline — the unconditional remove-then-re-add above takes the index's write lock once per call, but nothing holds it across the pair, so two concurrent ingests of one document can interleave as A.Remove → B.Remove → A.Add → B.Add and reproduce the duplicate postings the replace exists to prevent. Serialise ingestion per document at the application layer. The Service Bus trigger can do this for you: sessions give per-document FIFO.

Parsers​

Parsers implement IDocumentParser. The pipeline selects the first registered parser whose CanParse(contentType) returns true.

Built-in parsers (always available in Rag.NET core)​

Content typeParserNotes
text/plainTextDocumentParserProduces a single DocumentSection
text/markdownMarkdownDocumentParserHeading-aware: extracts HeadingLevel and Heading per section

Optional parsers (separate packages)​

Content typePackageNotes
application/pdfRag.NET.Parsers.PdfTable extraction (default on) + OCR for scanned pages via Tesseract or Azure Document Intelligence — see below
text/htmlRag.NET.Parsers.HtmlHeading-aware (AngleSharp)
application/vnd.openxmlformats-officedocument.wordprocessingml.documentRag.NET.Parsers.OfficeWord .docx (OpenXml)
application/vnd.openxmlformats-officedocument.spreadsheetml.sheetRag.NET.Parsers.OfficeExcel .xlsx (OpenXml)
application/vnd.openxmlformats-officedocument.presentationml.presentationRag.NET.Parsers.OfficePowerPoint .pptx (OpenXml)
message/rfc822, application/vnd.ms-outlookRag.NET.Parsers.Email.eml and .msg (MimeKit); attachments are traversed and flattened
application/epub+zipRag.NET.Parsers.EpubChapter-aware, emits a DocumentSection per chapter
application/zip, application/x-zip-compressedRag.NET.Parsers.ArchiveContainer: each entry is parsed by whichever parser claims its content type, under read and entry-count budgets
audio/wav, audio/mpeg, audio/flac, audio/mp4, audio/oggRag.NET.Parsers.AudioTranscription via Whisper.net (local, no API key)
image/tiff, image/heif, image/heicRag.NET.Parsers.AzureDocumentIntelligenceOCR via Azure AI Document Intelligence; PDF, image, Office and HTML formats are opt-in — see below
image/png, image/jpeg, image/webp, image/gif, image/bmpRag.NET.Parsers.VisionDescription via a vision LLM — UseImageDescription()
video/mp4, video/webm, video/quicktime, video/x-matroska, video/x-msvideoRag.NET.Parsers.VisionKeyframe description via FFMpeg + a vision LLM — UseVideoDescription()
text/csv(core) CsvDocumentParser
application/json(core) JsonDocumentParser

Three packages register under a different name from the parser class, because one package claims several types: Rag.NET.Parsers.Office is Word, Excel and PowerPoint in one (it was three packages before the decomposition — Rag.NET.Parsers.Word, .Excel and .PowerPoint no longer exist), and Rag.NET.Parsers.Vision carries both the image and the video parser.

Register additional parsers via AddParser<T>() or by calling the package-specific extension method:

services.AddRagNet(rag => rag
.AddPdfParser()
.AddHtmlParser()
// All three live in Rag.NET.Parsers.Office — one package, three registrations.
.AddWordParser()
.AddExcelParser()
.AddPowerPointParser()
.AddEmailParser()
.AddEpubParser()
.AddArchiveParser()
.AddAudioParser());

The two vision parsers are registered by the feature call rather than an Add…Parser() method, because each needs a chat client to describe with:

services.AddRagNet(rag => rag
.UseImageDescription(o => o.ChatClient = visionChatClient)
.UseVideoDescription(o => o.ChatClient = visionChatClient));

Both sanitise the model's output before it reaches a chunk (SanitiseOutput, on by default): a described image is untrusted text arriving from outside the corpus, and the security guide covers why that matters.

Some parsers take options. AddHtmlParser accepts a callback for how links are handled — by default a link's URL is appended to its text, which for site-internal paths is noise in the embedding:

services.AddRagNet(rag => rag
.AddHtmlParser(o => o.HrefHandling = HtmlHrefHandling.MakeAbsolute));

Remove drops the URL and keeps the link text; MakeAbsolute resolves it against the page's <base href>, failing that the document's url tag — which every web data provider here sets — and failing that HtmlParserOptions.BaseUri. With no base available the URL is left as it is rather than resolved against a guess.

To register your own parser implementation directly:

services.AddRagNet(rag => rag
.AddParser<MyXmlParser>());

Content-type ownership and the claim model​

Two parsers can end up claiming the same content type — a package you added and a built-in, or two packages you added deliberately. AddRagNet detects that before anything is resolved: a registration that declares a ParserClaim for a content type another declared claimant already holds is an InvalidOperationException at startup, naming both parsers, both registration calls, and (when one exists) a way out — never silent content loss at ingestion time.

Not every parser declares a claim, and that is by design, not a gap. CanParse is a predicate, not an enumeration — nothing can discover what an arbitrary parser accepts without probing it against a guessed list of content types, which is worse than an undetected collision. A parser can opt in to being seen by implementing IDeclaresContentTypes alongside IDocumentParser:

public sealed class MyXmlParser : IDocumentParser, IDeclaresContentTypes
{
public static IReadOnlyCollection<string> ContentTypes { get; } =
["application/xml", "text/xml"];

public bool CanParse(string contentType) =>
ContentTypes.Contains(contentType, StringComparer.OrdinalIgnoreCase);

// ParseAsync(...) as usual
}

When TParser implements IDeclaresContentTypes, AddParser<TParser>() declares one ParserClaim per reported type automatically. A parser that implements only IDocumentParser — most of the parsers this library ships, and any custom parser that does not opt in — declares nothing and stays invisible to the guard, exactly as before this interface existed; nothing about it needs to change to keep working.

A deliberate override is expressed with AddParser<TParser>(replaces:, replacesTypeNames:), which is different from silencing the conflict — it removes the replaced parser's IDocumentParser registration and its claim together:

services.AddRagNet(rag => rag
.AddParser<MyCsvParser>(replaces: typeof(CsvDocumentParser)));

Removal, not just silencing, is load-bearing: parser selection takes the first registered parser whose CanParse matches, and built-in parsers register before your configure delegate runs. An override that only suppressed the conflict check would still lose selection to the parser it was supposed to replace. replaces matches by full type name, so replacing a parser you cannot reference at compile time — one from an optional package that may not even be installed — use replacesTypeNames instead:

services.AddRagNet(rag => rag
.AddParser<MyExcelParser>(replacesTypeNames: ["Rag.NET.Parsers.Excel.ExcelDocumentParser"]));

A name (or type) that matches neither a registration nor a claim removes nothing and is not an error — replacing a parser from a package you never installed is a no-op, which is exactly what an optional dependency needs. Rag.NET.Chunking.Templates's UseQAPairsChunking() uses this to declare QAPairsDocumentParser as a deliberate override of core's CsvDocumentParser and, when Rag.NET.Parsers.Office is installed, its ExcelDocumentParser — see domain-specific chunking templates for the resulting behaviour change.

A parser registered by factory lambda is reachable only if it was registered to be. A plain services.AddSingleton<IDocumentParser>(sp => new MyParser(...)) leaves ServiceDescriptor.ImplementationType null, so replaces has nothing to match. If the parser declares claims, naming it in replaces or replacesTypeNames throws InvalidOperationException rather than doing nothing silently; a name that matches nothing at all is still a no-op. All replaced names are validated before any is removed. Every parser Rag.NET itself registers is reachable: the built-in TextDocumentParser and MarkdownDocumentParser are registered by type, and the packages that build their parser from a factory (Rag.NET.Parsers.Vision, .Email, .Archive, .AzureDocumentIntelligence, and .Pdf with its configuring overload) register through AddParserFactory, so replacing any of them works.

For package authors: AddParserFactory<TParser>. A parser whose construction needs the container belongs behind AddParserFactory<TParser>(factory, replaces:, replacesTypeNames:), an extension on IRagBuilder. It registers the singleton IDocumentParser and a ParserRegistration marker recording TParser's full name, which is what lets a later replaces: find and remove it. TParser must therefore be the concrete type the factory returns, the same type your claims name; an interface or abstract TParser is rejected with ArgumentException. The marker covers only AddParserFactory registrations: a parser registered by type or as an instance is matched directly, with no marker. It declares no ParserClaims. A generic helper cannot know the label users see, such as AddPdfParser(configure), and declaring claims would duplicate the ones your package already declares. Keep declaring them yourself:

rag.AddParserFactory(sp => new MyXmlParser(sp.GetRequiredService<TimeProvider>()));
rag.Services.AddSingleton(ParserClaim.For<MyXmlParser>("application/xml", "AddMyXmlParser()"));

PDF: table extraction and OCR​

The PDF parser accepts PdfParserOptions via a configuring overload:

services.AddRagNet(rag => rag
.AddPdfParser(options =>
{
options.ExtractTables = true; // default
options.MinTableRows = 3; // default
options.MinTableColumns = 2; // default
options.OcrMinCharacters = 50; // default; the OCR trigger, shared by both engines
options.MaxOcrPages = 200; // default; document-level (paid) OCR engines only
options.UseOcrFallback = false; // default; the Tesseract switch, requires <EnableOcr>true</EnableOcr>
options.TessDataPath = "./tessdata"; // default; Tesseract only
options.OcrLanguage = "eng"; // default; Tesseract only
}));

Table extraction​

Table extraction is on by default. A pure-geometry heuristic clusters each page's words into rows by baseline Y-bands and detects column gutters — word-free X-intervals that persist across at least MinTableRows vertically adjacent rows. Each detected table is emitted as a pipe-delimited Markdown table in its own DocumentSection with Heading = "table" and PageNumber set; the page's remaining prose is emitted as separate sections interleaved in document order (above → table → below).

Two behavioral notes versus the pre-table parser:

  • Reading order: on pages with a detected table, prose text is reassembled from word geometry (sorted top-down, then left-to-right) rather than taken verbatim from page.Text, so whitespace can differ slightly. Pages without tables keep the exact legacy page.Text output.
  • Header assumption: the first detected row is rendered as the Markdown header row.

Known limitations (the guards deliberately prefer a conservative false negative — the page parses as prose, exactly the old behavior — over a false-positive table):

  • Extraction is per page: a table spanning a page break is emitted as two tables.
  • Column gutters narrower than 1.5x the median word height are not detected, so very tight tables degrade to prose.
  • Tables whose cells average more than 4 words degrade to prose (e.g. long description columns).
  • A 2-3-column run of 8 or more rows spanning more than half the page's rows is treated as a multi-column page layout (academic two-column, newsletter three-column) and stays prose, unless its cells average 2 words or fewer — dense Key/Value content is extracted even when it fills the page. Whole-page 2/3-column tables whose cells run longer than that are still missed by design.
  • Any extractor failure logs a warning and the page parses as plain text (degraded, never broken).

OCR for scanned PDFs​

Scanned pages are full-page images with no text layer, so PdfPig extracts little or nothing from them. The parser can route those pages through an OCR engine, and there are two, with different shapes, costs and limitations:

TesseractAzure Document Intelligence
PackageRag.NET.Parsers.PdfRag.NET.Parsers.Pdf.AzureDocumentIntelligence
Unit of workOne embedded image at a timeThe whole PDF — one call per document
RunsIn process, local native libraryAzure cloud service
Compile gate-p:EnableOcr=true, source builds of this repository only — the published package compiles the engine outNone
Opt-inUseOcrFallback = trueRegistering the engine
CostFreePaid, per page of the submitted document
ConcurrencySerialized (Tesseract is not thread-safe)Unserialized — it is a network client

Configuring both is a registration-time error rather than a silent precedence rule: UseOcrFallback = true combined with a document-level engine throws an InvalidOperationException from whichever registration call comes second.

The trigger and the output shape are the same either way. When a page's extracted text is shorter than OcrMinCharacters (default 50), recognized text replaces it and is emitted as a DocumentSection with Heading = "ocr" and PageNumber set; pages PdfPig read successfully keep PdfPig's text, which is exact. Every degraded case is lossless: no recognized text, an engine failure, or a skipped call logs a warning and leaves the page exactly as it would be with no engine configured — short-but-real extracted text is never lost by enabling OCR, and genuinely empty pages still emit nothing.

Tesseract: the per-image fallback​

When UseOcrFallback is enabled and a page falls below OcrMinCharacters, the parser extracts that page's embedded images (largest display area first) and runs Tesseract over each until one yields text.

Tesseract is off by default, compile-gated (the same pattern as Rag.NET.Parsers.Vision) — and the gate is a property of this repository's build, not yours. The published Rag.NET.Parsers.Pdf package is compiled without the Tesseract engine, deliberately, so consumers do not carry its native payload; setting EnableOcr in a consuming project has no effect on the already-compiled package assembly. To get the per-image Tesseract path you must build Rag.NET.Parsers.Pdf from source:

  1. Build this repository (or your vendored copy of it) with dotnet build -p:EnableOcr=true — this defines ENABLE_OCR and pulls in the Tesseract package for Rag.NET.Parsers.Pdf, and reference that build instead of the NuGet package.
  2. Provide a tessdata directory (e.g. download eng.traineddata from tessdata) and point TessDataPath at it.
  3. Set OcrLanguage to the language code matching your traineddata (default eng).
  4. Set UseOcrFallback = true.

Enabling UseOcrFallback against a build without the gate — which includes the published package, always — throws an instructive InvalidOperationException at parser construction that points at Azure Document Intelligence below: misconfiguration fails fast, not at the first scanned page. If you consume Rag.NET as packages and need OCR, Azure Document Intelligence is the supported engine.

Azure Document Intelligence: the whole-document engine​

dotnet add package Rag.NET.Parsers.Pdf.AzureDocumentIntelligence
using Azure;
using Rag.NET.Parsers.Pdf.AzureDocumentIntelligence;

services.AddRagNet(rag => rag
.AddPdfParser()
.UseAzureDocumentIntelligenceOcr(
new Uri("https://my-resource.cognitiveservices.azure.com/"),
new AzureKeyCredential(key),
o =>
{
o.ModelId = "prebuilt-read"; // default
o.PricePerPage = 0.0015m; // default — indicative only, see below
o.PollingInterval = TimeSpan.FromSeconds(1); // default
o.Locale = null; // default: let the service detect
}));

The endpoint is a Uri; the credential is either an AzureKeyCredential or a TokenCredential (managed identity / OAuth) — there is an overload for each.

No <EnableOcr> compile gate applies, and UseOcrFallback stays false. That gate exists for Tesseract's native binaries and out-of-band traineddata; a managed REST client has neither, and reusing it would force an Azure-only consumer to pull Tesseract's native payload. Registering the engine is the opt-in — the parser routes sub-threshold pages through it without any options change.

The service is called at most once per document, the moment the first sub-threshold page appears — never once per page, and never at all for a PDF PdfPig reads in full. It receives the PDF itself, rasterizes server-side, and returns every page from a single long-running operation. PollingInterval governs how often that operation is polled when the service sends no Retry-After. When it does send one, the longer of the two applies — so raising PollingInterval throttles polling, while lowering it below what the service asks for does not speed anything up.

Configuration is validated at registration (ModelId non-empty, PricePerPage and PollingInterval non-negative), so a bad value throws from the UseAzureDocumentIntelligenceOcr call rather than out of a DI factory during the first parse.

Service failures are not fatal: the parser logs a warning and falls back to PdfPig's own extraction. Cancellation is not a failure and propagates.

What Azure OCR costs​

  • Every page of the submitted document is billed, not just the pages that needed OCR. A 500-page PDF containing one scanned page costs 500 pages — that document is above the default cap, so out of the box it would be skipped and billed nothing; raise MaxOcrPages past 500 and this is what you buy. Extracting only the pages that need it would mean writing PDFs, a dependency this repo does not have.
  • MaxOcrPages (default 200) is what bounds that exposure. A document with more pages than the cap skips OCR entirely: the parser logs a warning naming both numbers and emits PdfPig's text exactly as it would with no engine configured — lossless, not silent. The default is a tenth of Azure Document Intelligence's verified 2,000-page per-document service limit, and generous enough that the documents people actually ingest (reports, papers, contracts, slide exports) are never quietly downgraded. Raise it deliberately, with the per-page price in view. It has no effect on the Tesseract path, which runs locally and free.
  • PricePerPage defaults to 0.0015 — the widely published pay-as-you-go rate for the prebuilt-read model (USD 1.50 per 1,000 pages) at the time of writing. That default is indicative, not authoritative. Azure pricing varies by tier, region, model and commitment, and changes without this library changing; set it from your own price sheet. It is used only to compute what is written to the cost ledger — the service bills whatever it bills.
  • prebuilt-read is the default model because the parser wants page text; the richer prebuilt models cost more per page for structure this package deliberately discards.

Memory: budget for roughly twice the document​

Two costs that compound, both specific to the document-level path:

  • Registering a document engine turns the parser from streaming into whole-file-resident. PdfPig consumes its stream lazily throughout GetPages(), so the PDF is buffered in full up front to give PdfPig and the engine each their own view of it. This happens for every PDF on that path, not only the ones that turn out to need OCR — you cannot know which those are before parsing. Large files land on the large object heap, multiplied by however many documents ingest in parallel.
  • During an OCR call the PDF is buffered a second time, because the SDK's AnalyzeDocumentOptions accepts BinaryData rather than a Stream. Peak resident bytes are therefore roughly 2× the document size per concurrent OCR call. That is forced by the SDK's surface, not a defect here.

Size the host accordingly, or keep the engine off the parser that handles your largest inputs.

OCR spend, the cost ledger and your budget​

When an ICostLedger is registered, the Azure engine records each OCR call to it as a CostKind.Ocr entry carrying Pages and zero tokens — no token count is fabricated for an API that never reports one. Cost is billed pages × PricePerPage, computed by the engine because the ledger prices nothing itself. With no ledger registered, recording is a silent no-op rather than an error, and a ledger write failure is logged and swallowed: the OCR result was already paid for.

Two consequences to know before enabling it:

  • OCR spend counts toward the same budget window UseCostBudgeting enforces for chat and embedding calls, so enabling OCR can cause those gates to trip. See cost budgeting.
  • OCR emits no ragnet.llm.cost / ragnet.llm.tokens telemetry. The type that publishes those meters (CostAccounting) is internal to Rag.NET and unreachable from the Azure package. Dashboards built on those meters therefore under-report total spend by exactly the OCR portion — query the ledger for the complete picture. This is a known limitation, not an oversight.

OCR limitations, by engine​

Tesseract only — none of these applies to the Azure path:

  • Only embedded images are OCR-ed. Vector-only scanned pages (no embedded images) cannot be OCR-ed without a PDF rasterizer and degrade to the plain-text path with a warning. Azure rasterizes server-side, so vector-only pages are recognized normally.
  • CCITT G4 / JBIG2-compressed scans (common in real scanned PDFs) may not decode via PdfPig's PNG re-encoding, and their raw streams are not loadable by Leptonica — such pages also degrade to the plain-text path. Azure receives the PDF itself, so its own decoders apply.
  • Tesseract engines are not thread-safe: the parser serializes OCR calls, so scanned-page throughput does not scale with parallel document ingestion. The document path takes no such lock.

Azure Document Intelligence only:

  • It is a paid API reached from an automatic fallback, billed per submitted page — see what Azure OCR costs and MaxOcrPages.
  • Whole-file buffering, twice over during a call — see memory.
  • Recorded spend reaches the budget window but not the ragnet.llm.* meters — see OCR spend.
  • Only text is used. Tables, key/value pairs and selection marks the service returns are discarded; the PDF parser has its own table extractor, and merging two table sources is a separate question.

Both engines:

  • OCR replaces only pages below OcrMinCharacters; a page whose extracted text clears the threshold keeps PdfPig's text, which is exact. On the Azure path that page is still submitted and still billed — the whole document goes in one call — but its recognized text is discarded.

Azure Document Intelligence as a parser​

Rag.NET.Parsers.AzureDocumentIntelligence is a parser in its own right: it sends a whole document to an Azure AI Document Intelligence model and turns the result into sections. It is not the PDF OCR engine above, and installing one does not register the other.

dotnet add package Rag.NET.Parsers.AzureDocumentIntelligence
using Azure;
using Rag.NET.Parsers.AzureDocumentIntelligence;

Uri endpoint = new("https://example.cognitiveservices.azure.com/");
var credential = new AzureKeyCredential("<key>");

services.AddRagNet(rag => rag
.AddAzureDocumentIntelligenceParser(endpoint, credential));

The credential is either an AzureKeyCredential or a TokenCredential (managed identity / OAuth); there is an overload for each. The package does not depend on Rag.NET.Parsers.Pdf, so it brings in no PDF library.

What it claims by default​

TIFF and HEIF only: image/tiff, image/heif and image/heic. Those are the formats no other Rag.NET parser reads. .tif, .tiff, .heif and .heic files resolve to those content types, so a scan from any data provider reaches this parser. Without it, they are warned about and skipped like any other unclaimed binary type.

Every other format is an opt-in on Formats, because a local parser already reads it for free:

services.AddRagNet(rag => rag
.AddAzureDocumentIntelligenceParser(endpoint, credential, o =>
{
o.Formats |= AzureDocumentIntelligenceFormats.Office; // Word, Excel and PowerPoint
o.Analyzer.ModelId = "prebuilt-read"; // default
o.Analyzer.PricePerPage = 0.0015m; // default; indicative only
o.MaxBytes = 500_000_000; // default
}));

What Azure returns, and what it costs, per format​

From Microsoft's Read model page for v4.0, API version 2024-11-30:

FormatFormats flagAzure returnsBilled perRag.NET without Azure
TIFFTiff (default)OCR text, one page per frameframenothing
HEIF / HEICHeif (default)OCR textimagenothing
PDFPdfOCR text, lines, paragraphspageAddPdfParser, plus the PDF OCR engine for scanned pages
PNG, JPEG, BMPImagesOCR textimageUseImageDescription: a vision LLM's description, optionally Tesseract first
DOCXWordthe embedded text as is; images ignored; no lines3,000 charactersAddWordParser, locally, free
XLSXExcelthe sameworksheetAddExcelParser, locally, free
PPTXPowerPointthe sameslideAddPowerPointParser, locally, free
HTMLHtmlthe same3,000 charactersAddHtmlParser, locally, free

For Office and HTML, Azure adds cost, not text. It extracts the same embedded text the local parsers do and ignores embedded images. The only reason to route them here is one pipeline for every format. Word and HTML are billed per 3,000 characters, Excel per worksheet and PowerPoint per slide, so their cost depends on the amount of text, sheets or slides and cannot be known, or capped by a page count, before the call.

Opting in to a format another parser claims​

Every claimed content type is declared to AddRagNet's claim check. Enabling a format that another registered parser also claims is an InvalidOperationException at startup naming both parsers: Word alongside AddWordParser(), Pdf alongside AddPdfParser(), Html alongside AddHtmlParser() or Images alongside UseImageDescription(). Nothing is resolved first and no parser silently wins. Choose one parser per format; see the claim model. The default TIFF and HEIF claims conflict with nothing this library ships.

An empty Formats is a registration error, not a parser that claims nothing.

Replacing the default formats​

Formats starts as TIFF and HEIF. |= adds to that; = replaces it, which is how you drop TIFF and HEIF when you want Azure for something else only:

services.AddRagNet(rag => rag
.AddAzureDocumentIntelligenceParser(endpoint, credential, o =>
{
o.Formats = AzureDocumentIntelligenceFormats.Word; // Word only: TIFF and HEIF are dropped
}));

Taking a format over from another parser​

There are two routes. The first is to remove the other parser's Add...Parser() call, so Azure is the only claimant. The second is to keep that call and set ReplaceExistingParsers, which removes the parsers that already claim the formats Azure takes:

services.AddRagNet(rag => rag
.AddWordParser() // registered first
.AddAzureDocumentIntelligenceParser(endpoint, credential, o =>
{
o.Formats = AzureDocumentIntelligenceFormats.Word;
o.ReplaceExistingParsers = true; // Azure now reads .docx
}));
  • Register Azure last. Only parsers registered before the Azure call are replaced. A parser registered after it still conflicts with Azure's claims at AddRagNet, as it does without the option.
  • Only declared claims are found. The option looks at the ParserClaims already declared for the content types Formats enables, compared case-insensitively. A parser registered without claims is invisible to it, as it is to the conflict check.
  • Partial overlap throws. Replacing removes a whole parser. If a parser Azure would replace also claims content types outside Formats, registration throws InvalidOperationException, naming every such parser with its leftover types sorted, and adds nothing. Removing the parser anyway would leave those types with no reader.
  • Images with UseImageDescription() always throws. The vision parser also claims image/gif and image/webp, and Azure never emits those: Images covers PNG, JPEG and BMP only. No setting of Formats changes that, so there is no way to take PNG, JPEG or BMP over from the vision parser. Keep the vision parser for images, or drop UseImageDescription().
  • Bundled services stay registered. Only the parser is removed. For example the chunking strategy that UseImageDescription() registers remains in the container.
  • The override is recorded. When exactly one parser is replaced, the claims Azure declares carry its type name in ReplacesParserTypeName.

Sections​

  • One section per page, with PageNumber set, built from the result's paragraphs joined by a blank line. A paragraph that crosses a page break appears once, on the page it starts on.
  • Office and HTML pages come from Azure's page units. Their paragraphs carry no position on a page, so each is placed on the page whose content range holds the paragraph's start, and groups with that page like any other. For Word and HTML a page is a run of about 3,000 characters, for Excel a worksheet and for PowerPoint a slide. A paragraph no page holds, or any paragraph in a result without page ranges, comes out as its own section without a PageNumber, in document order, after the paged sections.
  • Headings come only from a layout model. prebuilt-read, the default, assigns no paragraph roles, so it yields no headings. With o.Analyzer.ModelId = "prebuilt-layout", title and section-heading paragraphs set Heading and HeadingLevel (1 and 2) from the section that starts with them until the next heading. A heading partway down a page applies from the next section, the next page, onwards. Page headers, footers and page numbers are dropped. Tables, key-value pairs and selection marks are not mapped.

Limits and spend​

  • MaxBytes defaults to 500,000,000 bytes, Azure's documented 500 MB limit on the S0 tier. It is checked before anything is sent: a larger document throws AzureDocumentIntelligenceException naming the file and the cap, and no request is made. An empty document is refused the same way. Values above Array.MaxLength are rejected at registration, because the document is buffered in one array.
  • Budget roughly 2.5 to 3.5 times the document in memory per concurrent parse. The SDK takes the document as one in-memory buffer, not a stream. A stream without a length is buffered in an array that grows by doubling, so it can hold up to twice the document, and the SDK then posts the document base64-encoded inside a JSON body, about 1.33 times the document again. MaxBytes caps the document, not this multiple of it.
  • Spend is recorded, not capped. When an ICostLedger is registered, each call is recorded as a CostKind.Ocr entry with the analyzed page count and that count × PricePerPage, as the PDF OCR engine does. That spend counts toward the cost budgeting window, but the parser checks no budget before submitting a document. MaxBytes is the only limit applied in advance.
  • PricePerPage defaults to 0.0015 (USD 1.50 per 1,000 pages for prebuilt-read), which is indicative only; set it from your own price sheet. It only prices the ledger entry.
  • Failures throw. There is no local text to fall back to, so a service failure (throttling, authentication, unsupported content) throws AzureDocumentIntelligenceException, naming the file and wrapping the SDK's exception. Cancellation propagates unwrapped.

The PDF OCR package is unchanged​

Rag.NET.Parsers.Pdf.AzureDocumentIntelligence keeps its id, API and behaviour. It serves a different purpose: it is an OCR engine for AddPdfParser(), called only for PDFs with pages below OcrMinCharacters, and the PDF parser keeps PdfPig's text for every other page. It now depends on this package, whose analyzer makes its service call, but it does not register the parser. To read scanned PDFs, use the OCR engine; to read TIFF and HEIF scans, or to send whole documents of other formats to Azure, use this parser.

DocumentSection​

Parsers produce a stream of DocumentSection records:

public sealed record DocumentSection
{
public required string Text { get; init; }
public required string DocumentId { get; init; }
public int? HeadingLevel { get; init; } // 1–6 (H1–H6), null if no heading
public string? Heading { get; init; } // heading text, null if no heading
public int? PageNumber { get; init; } // null for non-paginated formats
public int SectionIndex { get; init; }
}

Heading-aware metadata​

When HeadingLevel and Heading are set (by the Markdown or HTML parser), the pipeline automatically builds a breadcrumb trail and writes three entries into every TextChunk.Metadata produced from that section:

KeyExample value
heading"Section 2"
heading_level"2"
heading_breadcrumb"Chapter 1 > Section 2"

The breadcrumb is built by concatenating all ancestor headings in order, separated by >. A new heading at level N resets all headings at levels N+1 through 6.

These metadata keys can be used for metadata filtering.

Page attribution​

When PageNumber is set (the PDF parser reads it from PdfPig's page number on every path — plain text, per-page OCR fallback, and document-level OCR — and the PowerPoint parser sets it to the slide number), every chunking strategy copies it onto the chunks it produces as the reserved page / page_end metadata pair, written as numbers, so a retrieved chunk can be cited back to its source page and filtered numerically:

  • A chunk that sits entirely on page 3 has page: 3, page_end: 3 — the keys are always written together, so consumers can render a range without probing for a missing half.
  • A chunk merged from sections spanning pages 3–4 (hierarchical merging, semantic document-level grouping, proposition passages) has page: 3, page_end: 4 — the min/max of the contributing sections' page numbers.
  • A merged run that mixes paginated and unpaginated sections keeps the pages that are present: a chunk touching page 3 and an unpaginated section is still findable on page 3.
  • Both keys are absent (not null-valued) for non-paginated formats (Markdown, HTML, plain text, …) and for chunks whose origin page is unknowable (LLM-rewritten resume fields; video "pages" are scene timestamps and stay in timestamp_seconds).

page and page_end are reserved metadata keys: the framework writes them, and a connector emitting either from entry metadata is rejected with ReservedMetadataKeyException. Chunks stored before the keys existed read back without them until re-ingested.

Progress reporting​

Pass any IProgress<IngestionProgress> to receive stage-completion callbacks:

public sealed record IngestionProgress
{
public required IngestionProgressStage Stage { get; init; }
public required string DocumentId { get; init; }
public int? Current { get; init; }
public int? Total { get; init; }
public required string Message { get; init; }
}
var progress = new Progress<IngestionProgress>(p =>
Console.WriteLine($"[{p.Stage}] {p.Message} ({p.Current}/{p.Total})"));

using var stream = File.OpenRead("report.pdf");
var result = await pipeline.IngestAsync(stream, metadata, progress: progress);

Example output:

[Parsing] Parsing complete (/)
[Chunking] Chunked into 42 chunks (42/42)
[Embedding] Generated 42 embeddings (42/42)
[Storing] Stored 42 chunks (42/42)

Current and Total are null for the Parsing stage because the total section count is not known until parsing completes.

Ingestion return value​

public sealed record IngestionResult
{
public required string DocumentId { get; init; }
public required int ChunksStored { get; init; }
}

ChunksStored can be 0 if the document parsed to no content (empty file or all-whitespace input). The pipeline short-circuits before the embedding call in that case.

Performance notes​

See benchmarks for detailed measurements. Key takeaways:

  • Parsing and chunking of a 50 KB document completes in under 400 µs (mocked embedder).
  • Real ingestion time is dominated by the embedding API call, typically 50–500 ms per batch.
  • Use Overwrite = true and a stable DocumentId for incremental refreshes to avoid accumulating stale chunks.

Data providers​

For batch ingestion from a directory, website, or GitHub repository, use IngestFromProviderAsync instead of calling IngestAsync in a loop. It handles ETag/hash deduplication, optional cleanup, and — via IngestionOptions.MaxDegreeOfParallelism — parallel processing of multiple documents.

LocalFilesDataProvider​

var provider = new LocalFilesDataProvider("/data/docs", new LocalFilesOptions
{
Extensions = [".pdf", ".docx", ".md"],
SearchOption = SearchOption.AllDirectories,
Filter = path => !path.Contains(".git"),
});

var result = await pipeline.IngestFromProviderAsync(provider, "my-corpus",
hashStore: sp.GetRequiredService<IContentHashStore>(),
cleanupMode: CleanupMode.Full);

Console.WriteLine($"Ingested: {result.IngestedCount}, Skipped: {result.SkippedCount}, Failed: {result.FailedCount}, Deleted: {result.DeletedCount}");

// The counts are derived from lists, so you can name the files rather than just tally them:
foreach (var entry in result.Failed)
Console.WriteLine($" failed: {entry.FileName} ({entry.Id.Value})");

SitemapDataProvider​

var provider = new SitemapDataProvider("https://docs.example.com/sitemap.xml", httpClient);
await pipeline.IngestFromProviderAsync(provider, "docs-site", hashStore: hashStore);

RssDataProvider​

var provider = new RssDataProvider("https://example.com/feed.rss", httpClient);
await pipeline.IngestFromProviderAsync(provider, "blog-feed", hashStore: hashStore);

Supports RSS 2.0 and Atom feeds. Id is the <guid> or <link> element; ETag is <pubDate> / <updated> — so unchanged posts are automatically skipped on subsequent runs.

WebCrawlerDataProvider​

var provider = new WebCrawlerDataProvider("https://docs.example.com", httpClient, new WebCrawlerOptions
{
MaxDepth = 3,
MaxPages = 500,
SameDomain = true,
RespectRobotsTxt = true,
});
await pipeline.IngestFromProviderAsync(provider, "docs-site", hashStore: hashStore);

GitHubDataProvider​

var provider = new GitHubDataProvider("my-org", "my-repo", githubClient, new GitHubDataProviderOptions
{
Branch = "main",
Extensions = [".md", ".cs"],
Filter = path => !path.StartsWith("docs/plans/"),
LastIngestedCommitSha = settings.LastIngestedCommitSha, // null on first run
});
await pipeline.IngestFromProviderAsync(provider, "github-repo", hashStore: hashStore);
// Save result to settings for next run: settings.LastIngestedCommitSha = latestCommitSha;

Registration​

services.AddRagNet(b => b
.UsePgVector(connectionString, vectorDimensions: 1536)
.UseContentHashRecordManager("ragnet-hashes.db"));

Event-driven ingestion​

Ingestion can also be push-based. Three triggers ship:

  • an HMAC-verified webhook endpoint (Rag.NET.Api), which enqueues onto the bounded in-memory job queue drained by the UseEventDrivenIngestion BackgroundService processor;
  • a background polling trigger (UsePollingIngestion), which re-runs IngestFromProviderAsync on an interval;
  • an Azure Service Bus trigger (UseServiceBusIngestion, Rag.NET.Ingestion.AzureServiceBus), which ingests each message end to end and settles it on the outcome — complete, abandon for redelivery, or dead-letter with a reason. It does not use the job queue; that is what makes it durable, and it is the only ingestion path with a dead-letter queue.

See Event-driven ingestion in the data providers guide for setup, the shared payload contract, signature examples, and the settlement table.

Embedding versioning & re-indexing​

Switching embedding models invalidates every stored vector — dense similarity scores are only meaningful within one model's embedding space. UseEmbeddingVersioning tracks which model produced each document's vectors so you can re-embed only what is stale instead of wiping and re-ingesting the corpus.

Model migration walkthrough​

1. Register versioning before (or at) your first ingest:

services.AddRagNet(b => b
.UseEmbeddingVersioning(o => o.DatabasePath = "ragnet-versions.db"));

// Stores chunk text — enables real re-indexing (otherwise ReindexStaleAsync is report-only):
services.AddSingleton<IRagDataManager>(new SqliteDocumentStore("ragnet-data.db"));

After every successful store, the pipeline stamps the document with the resolved model identity and the vector dimension. The identity comes from the generator's EmbeddingGeneratorMetadata ("{ProviderName}/{DefaultModelId}"); for adapters that expose no metadata, set o.ModelId explicitly — without either source, stamping is disabled with a one-time warning (the identity is never guessed). DeleteAsync removes the stamp along with the document.

2. Switch the embedding model (new registration, new deployment — nothing else changes). Newly ingested documents are stamped with the new identity; existing documents keep their old stamp.

3. Re-index the stale documents:

var result = await pipeline.ReindexStaleAsync(serviceProvider, cancellationToken: cancellationToken);
// Optionally pass IngestionOptions to tune the re-embedding batch size:
// await pipeline.ReindexStaleAsync(serviceProvider, new IngestionOptions { EmbedBatchSize = 50 }, cancellationToken);
// result.Reindexed — re-embedded, re-stored, re-stamped
// result.ReportedStale — stale but not re-indexable (no data manager registered)
// result.Failed — (documentId, error) pairs; the loop continued past them

A document is stale when its stamped model identity differs from the current one, or when its stamped dimension differs from the current model's output dimension (learned by embedding one constant probe text — a single extra embedding call per run, only made when at least one stamp matches the current model id). Stale documents are re-embedded from the chunk text stored by the IRagDataManager, their old vectors are deleted (so surplus stale chunks under higher chunk indices cannot survive), the new vectors are stored, and the stamp is updated. Re-embedding honours IngestionOptions.EmbedBatchSize. When a sparse encoder (ISparseEmbeddingGenerator) and a sparse-capable store are both registered, sparse vectors are regenerated from the same text; a sparse failure is logged and the dense re-index still succeeds. BM25 needs no re-index (the text is unchanged).

An overload taking explicit dependencies (versionStore, embedder, vectorStore, dataManager, options) is available for non-DI composition, mirroring IngestFromProviderAsync.

Limitations​

  • Chunks are reused verbatim. Re-indexing re-embeds the stored chunk text; it does not re-parse or re-chunk the source document. If you also changed chunking settings, re-ingest instead.
  • Report-only without a data manager. Without an IRagDataManager (which stores the chunk text), stale documents land in ReportedStale for caller-driven re-ingest from the original source.
  • Fixed-dimension backends need the collection recreated first. On Qdrant and pgvector the collection/column dimension is fixed at creation. For a dimension-changing migration, recreate the collection (or column) for the new dimension before calling ReindexStaleAsync — otherwise every stale document lands in Failed with a backend error. Only the in-memory store tolerates mixed dimensions.
  • Quiesce ingestion while re-indexing. Concurrent ingestion converges (both paths replace by (DocumentId, ChunkIndex) and re-stamp), but search results can transiently mix old- and new-model vectors — prefer running ReindexStaleAsync while ingestion is paused.
  • Only stamped documents are seen. Documents ingested before UseEmbeddingVersioning was registered have no stamp and are invisible to ReindexStaleAsync — re-ingest them once to get them stamped. The same applies when stamping itself failed: a stamp failure is logged as a warning (ingestion still succeeds), but until the document is successfully re-ingested it will be missed — or, after a model switch, mis-reported — by re-indexing.
  • The ragnet reindex --stale CLI command ships with the CLI tool (Milestone 3).