What Does an AI PDF Benchmark Actually Give the Model?

By Yuki Yu

Two delivery proofs contain the same three version names. On one page a filled circle approves Version A. On the other it approves Version C. Send their extracted text to a model and it receives exactly the same words twice. Even a perfect reader of those words cannot tell which proof approved which version.

This is an easy detail to miss when comparing an AI PDF benchmark. A score reflects a model working with a particular representation of the document. The file a customer uploads, the content the application extracts and the evidence the model receives can differ. For a team building a document assistant, the first procurement question is concrete: will the route preserve the detail that decides the answer?

Artificial Analysis announced its Intelligence Index v5 on October 27, 2026. GDP.pdf is part of the Knowledge Work category. Its evaluation documentation explains how professional documents are prepared for the model, making it a timely example for teams comparing document-reading systems. Intelligence Index v5 announcement and index methodology.

Read first. Decide how much source material your task actually needs and how you will check retrieval across it. Token360’s long-context model analysis.

How does Artificial Analysis run GDP.pdf?

Artificial Analysis describes GDP.pdf as its independent evaluation of a benchmark developed by Surge AI. The dataset contains 100 tasks, 4,592 source pages and 1,275 individual grading criteria across ten domains. Its implementation prepares document text with LiteParse and adds rendered page images for models that accept images. A task is assessed against criteria specific to that piece of work. GDP.pdf evaluation and input methodology.

When using the results to shortlist models, carry the input description into your comparison sheet. A team that sends only extracted text has changed the experiment. A provider that accepts a PDF file directly may perform its own parsing and rendering. The file extension alone tells you little about the material available to the model for a particular question.

The grading unit also matters. A report can follow the requested structure and cite the right page while choosing the wrong approved version. Before borrowing a published score, identify the individual errors that would make a report unusable in your workflow. That is a better basis for a pilot than treating every completed paragraph as equivalent.

Can two PDFs have identical extracted text and different answers?

Yes. We created two single-page PDFs with the same title, legend and three row labels. Each row has an outlined circle drawn as a vector shape. The only meaningful content difference is which circle is filled. We rendered each page and independently extracted its text with Python’s pypdf. The side-by-side page images below show the different approval marks.

Delivery proof
Filled circle = approved version
Version A
Version B
Version C

That is the text returned for both documents. Including the final newline, the two UTF-8 outputs compared equal byte for byte. The visible answer changed from A to C. This is a local input test completed on November 2, 2026. It uses synthetic documents that we designed to isolate one failure mode. We made no model calls and measured no model accuracy.

Imagine the text-only assistant answers A twice. It would earn one correct result out of two by repeating the same guess, despite having received no case-specific evidence. Conversely, a response asking to see the marked page would correctly identify the missing information. Record those outcomes separately when testing a live route.

What should the application pass through?

Start with the cue that makes the answer recoverable. For a policy sentence, selectable text and a page reference may suffice. For our proof, the route needs the mark and its relationship to a row. That could be a legible page image or a structured representation that explicitly associates the selected state with Version A or C.

Question about these proofs	Evidence needed	What the extracted text contains
What versions are listed?	Three row labels	All three labels
What indicates approval?	The legend	The complete legend
Which version is approved?	The filled circle and its row	The labels and legend, without the mark

Three questions about one page require different evidence. If your acceptance set contains only the first two, a text-only integration can look complete. Add the third and the missing relationship becomes visible immediately. For an actual candidate model, also test whether it can use the relationship: a page image makes the answer available, but reading the correct row remains a separate capability.

Where else does this affect professional documents?

Look for decisions encoded through position, marking or page boundaries. An engineering drawing may use a leader line to associate a note with one component. A revision sheet may strike through an obsolete value. In a comparison table, the answer may depend on which column header spans two subcolumns. Collect examples from the actual family of documents you expect customers to submit.

Consider a procurement table where the first page establishes that amounts are monthly and the next page continues the rows. A pipeline might preserve every amount while dropping the header from the retrieved passage. A model can repeat the number accurately and give the wrong unit. Test the passage that reaches the answer step, including the header, instead of assuming that an accurate whole-document text dump resolves the question.

Accessibility is relevant here too. A tagged document or an actual form field may expose information that our drawn circles do not. A template under your control could encode the approved version as text alongside the mark. That gives both screen-reader users and downstream tools another way to recover it. Confirm this on the exported file because the authoring template and the delivered PDF can expose different information.

Separate a parsing failure from a reasoning error

Save what went into each stage. A reviewer should be able to open the original page, inspect the extracted passage or rendered image, and compare that evidence with the model’s answer. Label the earliest stage at which the decisive detail disappeared or was misread.

For the two proofs, first ask a human to identify the approved version on each rendered page. Next show the reviewer only the text actually sent to the candidate. Ask where the selection is recorded. If the reviewer cannot locate it, adding a longer prompt is unlikely to restore the missing bit. If the page was supplied and the mark is readable, inspect row alignment, image size and the model response.

Keep one control that should succeed under either route. Asking for the three row labels confirms that the assistant is looking at the intended page. Then change just the selected row and repeat the decision question. A system that answers the control correctly and changes its decision with the mark is more informative than a single convincing response to a single file.

Which details belong in a vendor comparison?

Record the model identifier, input route, parser and rendering settings, page selection rule and answer rubric. Keep one row per configuration. A model tested with page images and the same model tested with text alone answer two different implementation questions.

For long documents, include input and output usage, processing time and the amount of human review required. Rendering every page may preserve a cue that retrieval would miss, while adding work to requests whose answers live in plain text. A selective route can begin with text and fetch a page image when a question concerns a mark or a diagram. Measure how often it requests the correct page and how often the image actually resolves the issue.

For Token360 integrations, verify the chosen endpoint and model route before promising a particular document path. An image input, a file upload and application-managed extraction are different contracts. Use the capability documentation as a starting point, then send a bounded pair of known documents through the same path your product will use. Token360 model capabilities.

Frequently asked questions

Does GDP.pdf test native PDF upload on every provider?

Artificial Analysis documents its own preparation pipeline, including extracted text and page images for image-capable models. Check the operator’s methodology before comparing it with a native file-upload trial.

Would more context recover the marked version in your text example?

Both extractions contain the entire page’s text and are identical. The route needs an additional source of evidence about the mark to distinguish the files.

Would OCR solve this particular case?

It depends on the output representation. Recognizing the printed words still leaves the association unresolved. A system that detects the marked state and attaches it to the correct row could preserve it. We did not run an OCR comparison.

Should a document assistant cite page numbers?

A page reference helps a reviewer locate the source. For a selection or table lookup, ask for the row or field as well, then verify that the evidence supports the answer.

Do these two files establish which model is better?

They establish a difference between the information retained by two representations. Ranking models would require actual runs, representative tasks and a consistent scoring rubric.

What is a useful first pilot question?

Choose one production decision that depends on a visible mark or a table relationship. Prepare two files in which only that decision changes, then observe what each configured route sends and returns.

Read next. Put representation checks and answer-level checks into the same acceptance set before comparing candidates. AI model evaluation framework. More editorial resources from Token360.

Editorial revision: November 4, 2026. Test dates and dated source snapshots remain as stated in the article.