Small Upload, Large Job: Bound AI Document Parsing

A four-megabyte upload is not necessarily a four-megabyte job. An archive can unpack into much more data; an image can require a large decoded buffer; a document can contain other documents that trigger further parsing. The expensive work may happen before the AI model receives its first token.

The decision is not simply which file size to allow. It is which processing profile the service can afford to complete, and what the service will do when that profile is exceeded. An upload limit protects one boundary. It does not define the cost or completeness of extraction.

This guide separates facts in the linked documentation from AIEngine's proposed engineering controls. The documentation was checked on 21 September 2026. The numerical example is a synthetic budget exercise, not a customer incident, product benchmark or recommended universal limit.

Start with an accepted processing profile

Describe the job before choosing thresholds. “Accept documents” leaves too much undefined. A text-only PDF intake, a scanned-document OCR service and an archive-inspection service need different allowances. Decide whether embedded files, attachments, animation frames and recursive archives belong in the supported input at all.

The OWASP File Upload Cheat Sheet recommends business-required extension allowlists, warns against trusting the supplied Content-Type, and says limits must account for size after decompression when files will be extracted. These are complementary controls: recognising an allowed format does not establish that processing it will be cheap.

An intake profile should answer:

• Which formats and parser versions are supported?
• Are embedded objects processed, ignored by declared policy, or grounds for rejection?
• What resource limits apply across the whole job?
• Which extraction outcomes are eligible for indexing or downstream decisions?

Keep these choices attached to a profile version. A later parser upgrade that starts visiting embedded objects can change workload and coverage without changing the uploaded bytes. The worker should report the profile actually applied, not merely the profile the caller requested.

Avoid promising “complete extraction” without defining the scope. A service may legitimately extract only top-level text. It must say so, especially if a user expects attachments or scanned pages to be included.

Use several budgets, not one compression ratio

There is no useful universal conversion from upload bytes to processing cost. Expansion ratio helps identify suspicious cases, but small files with low ratios can still demand expensive operations. Conversely, a highly compressible legitimate document may be inexpensive under absolute resource limits.

Python's ZIP documentation identifies decompression-related disk exhaustion as a pitfall and notes that interruption can leave extraction incomplete. That makes both resource consumption and partial output part of the interface, not just exceptional implementation details.

Define separate counters for different kinds of consumption. The following is a proposed design checklist, not a set of vendor defaults:

Peak and cumulative measurements solve different problems. Deleting temporary files may lower current disk use, but it does not undo CPU work or storage writes. Releasing a child's memory does not justify resetting the parent's elapsed time.

Do not rely exclusively on checks after a library returns. A decoder might allocate its buffer before your application can count the resulting pixels. Admission checks on dimensions or declared sizes are useful, but the worker still needs external containment for inaccurate declarations and uninterruptible operations.

Make child parsers spend the parent's allowance

A recursive pipeline can accidentally create a fresh budget at every layer. Each embedded document receives its own timeout, each archive its own byte allowance, and each image its own OCR quota. Every individual operation looks compliant while the root request has no meaningful bound.

Give the root job one budget owner. Children receive the remaining allowance, not a reset copy of the initial limit. Track actual emitted bytes at the boundary you are protecting; archive directory metadata is not evidence that those bytes were safely processed.

Parallel children require coordinated accounting. If two workers both observe ten megabytes remaining and each writes eight, a read-then-update check has already failed. Reserve capacity atomically before a bounded write, or place the output behind a supervisor that enforces the shared limit. Unused reservations can be returned; consumed work must not be erased from cumulative counters.

Apache Tika's versioned 4.0.x limits documentation describes a total task timeout covering a document and its embedded children, with individual parser timeouts clipped to the remaining total. This is a useful distinction between a root deadline and independent child timeouts. Check the equivalent controls in the version you actually deploy.

For an AI pipeline, include preprocessing in the overall request contract. A model timeout that begins only after OCR leaves the earlier work unbounded. Our deadline and cancellation guide explains the related problem of carrying cancellation across asynchronous work.

Test the arithmetic with a harmless fixture

Suppose a test profile accepts a four-MiB upload, permits at most 64 MiB per embedded member, and allows 128 MiB of cumulative member output. Its fixture describes four members of 40 MiB each. No large archive needs to be generated: a counter-level test can represent the writes as bounded chunks.

All four members individually satisfy the 64-MiB rule. Their total does not satisfy the 128-MiB rule. If output is streamed rather than reserved per member, only eight MiB remain when the fourth begins. A chunk that would exceed that remainder must not be committed to the bounded destination.

The root result is incomplete even if the first three members parsed successfully. Keep any partial fourth member out of the admitted document set. Whether complete earlier members are retained for diagnostics is a separate storage policy; retaining them does not turn the overall extraction into a complete result.

Add time to the same fixture. A 25-second root allowance with 18 seconds already spent leaves at most seven seconds, not another 25, for the next child. A stalled child and a busy child consuming CPU are different failure modes, so a progress detector alone is insufficient.

These numbers illustrate accounting, not safe production settings. Choose real limits using representative legitimate files, measured resource use, service capacity and the consequence of rejecting a document. Test counters and supervisor behaviour separately: passing arithmetic tests does not prove that the parser can be stopped.

Treat warnings and filesystem safety as separate controls

Some library protections signal a condition without automatically rejecting it. Pillow's Image documentation describes a decompression-bomb warning above its configured pixel threshold, an error above twice that threshold, and the option to turn the warning into an error. Decide explicitly how your service handles the warning; do not assume that a logged warning means decoding was blocked.

A per-image safeguard also does not establish the total allowance for a document containing many images. Count the pages or frames your profile actually processes and budget their aggregate work. If the integration cannot reliably enumerate that work, narrow the supported profile or isolate it in a more constrained service.

Archive path safety is another independent issue. Python 3.14's tarfile documentation makes the data extraction filter the default, but explicitly warns that filters do not prevent every danger, including denial of service. It recommends additional resource limits and notes that a failed extraction can leave partially extracted files behind.

Use a private working directory per job and a deliberate policy for links, names, collisions and permitted output types. Do not unpack directly into a shared ingest directory that another process watches. A consumer that sees files as they appear can admit a half-finished extraction before the parent job reports failure.

Safe paths, bounded resources and successful parsing answer different questions. None proves that the content is authorised, accurate or safe to follow as instructions.

Put the stopping mechanism outside the parser

The component doing risky work should not be the only component responsible for stopping it. Run parsing with constrained privileges, no ambient application credentials and only the network access the profile genuinely requires. Separate its writable scratch space from published content and from other tenants' jobs.

An external supervisor should own the wall-clock deadline, resource restrictions and cleanup lifecycle. When it stops a job, it must account for subprocesses such as OCR workers. Killing the immediate parent while descendants continue can leave both resource consumption and temporary files behind.

This is a workload-isolation problem as well as a file-validation problem. Our AI sandbox guide discusses choosing an isolation boundary appropriate to the work. A container label alone is not evidence that memory, storage, process and network policies are actually enforced.

Bound admission too. A perfectly limited worker can still be overwhelmed by an unlimited queue or too many concurrent jobs. Apply tenant-aware concurrency and queue limits, and distinguish “try again later” capacity failures from “this document exceeds the supported profile” rejections.

Retries deserve particular care. Automatically restarting a deterministic over-budget file multiplies cost without increasing its chance of success. Retry transient infrastructure failures under a bounded policy; require an explicit alternative profile or a different input for a reproducible resource-limit result.

Make completeness a first-class result

Do not reduce the result to a process exit code. Tika's same limits documentation gives a particularly useful example: in Pipes UNPACK mode, reaching the total embedded-byte limit stops further extraction while the parse still reports SUCCESS; the truncation is logged as a warning rather than returned in result metadata. This is documented behaviour for that mode, not a claim that we tested every Tika deployment.

Wrap the parser with an outcome contract that your application can enforce. A useful result distinguishes complete-within-profile, unsupported input, resource limit reached, parser error and cancellation. Include the profile version, parser version, applicable limits, observed consumption and the reason processing stopped.

“Complete-within-profile” still does not mean every visible detail was extracted correctly. Reading order, OCR accuracy and table interpretation need separate quality checks. It means the declared processing scope finished without a known limit or execution failure.

Promote output to the trusted collection only after evaluating that contract. Our RAG ingestion security guide covers the separate admission decision around provenance and content authority. A resource-complete result can still fail that admission check.

If the product offers partial previews, label them as partial and keep that status attached when text moves into chunks, summaries or downloads. An assistant must not infer “no relevant clause exists” from a document whose final pages were never processed. Silence in a truncated extraction is not evidence of absence.

Measure legitimate rejection as well as containment

A limit that rejects nearly every real customer file is operationally safe only in a narrow sense: the service no longer performs its intended job. Measure the cost of protection alongside its effectiveness.

Maintain a small, harmless regression corpus and explicit expected outcomes:

• A valid file just below each configured threshold completes.
• A file just above the threshold returns the named limit outcome.
• Several individually permitted children exceed the cumulative budget.
• A nested child inherits the remaining deadline and byte allowance.
• Concurrent children cannot overspend shared capacity.
• Warnings follow the service's declared reject-or-continue policy.
• Cancellation stops descendant work and leaves no published partial result.
• Interrupted extraction is cleaned up or quarantined without crossing tenant boundaries.

Use mocked counters for very large logical sizes and controlled fixtures for integration tests; there is no need to deploy a destructive archive to a shared environment.

Track rejection rate by profile and legitimate file type, consumption near each cap, parser terminations, orphaned processes, cleanup failures and partial-output admission attempts. Investigate a rising limit rate before simply increasing the threshold: it might reflect a changed parser, a newly common document layout or abusive input.

Revisit the profile whenever parsers, OCR engines or supported formats change. The durable contract is simple: the service knows how much work it permits, can stop that work from outside, and never mistakes stopping early for having read the whole document.