How the screening works
An analysis is not a button press with a number at the end. It is a chain of five steps, each of which can be followed: get the text out, split it up, judge it item by item, check every quote against the original text, add it up. This page describes each step, including the places where the method reaches its limits.
1. Extraction
What is accepted: PDF files with a text layer and DOCX files, up to 50 MB. A DOCX file is converted to a PDF version on the server, and the text is read from that. Page numbers in the report then mean the same page as in the document in front of you.
From the PDF, the full text is extracted together with a mapping from every position in it to its page. A scanned PDF with no text layer (a picture of a page, not text) is refused at upload, with a message saying what the problem is and what to do about it. The threshold is mechanical: if the document yields fewer than 200 extractable characters, it counts as an image document.
That is a deliberate decision against the more convenient alternative. An analysis of a document from which nothing can be read would not fail. It would produce a report in which everything is missing. That report looks like a finding about the manuscript when it is a finding about the file.
2. Chunking
A manuscript rarely fits into a single model call. The extracted text is therefore split into overlapping chunks, along the usual section headings, falling back to paragraphs and sentences. Every chunk is a contiguous excerpt of the original text. Nothing is reassembled, nothing inserted, nothing rewritten. That is what later makes it possible to find a quote again at its place in the document.
Every chunk is screened against every checklist item. One item can be judged missing in the introduction and present in the methods. The partial results are then merged deterministically: in a fixed order, with no unordered sets, so the same input gives the same result. The merge follows a fixed precedence. Present beats partial, which beats not applicable, which beats missing. An item that any chunk finds evidence for cannot be lost because nine other chunks do not mention it.
Throughout, the analysis stays pinned to one published, immutable checklist version. If the checklist is revised later, that does not change a report that already exists. It stays bound to the version it ran against.
3. Quote verification
This is the point at which it is decided whether a report holds up. A language model can invent a quote: a sentence that sounds like the manuscript, plausibly phrased, but not present in the text. Anyone who believes such a quote is not checking anything. They are reading a confirmation.
VetCaseAudit therefore never takes a text position reported by the model. The model supplies only the quoted text. Its response format has no field for a position at all. The server looks the character positions up itself in the extracted manuscript text: first as an exact string within the chunk being screened, then insensitively to case and whitespace, then with whitespace removed entirely (some PDFs lose word spacing during extraction), and finally the same three rungs across the whole document, because a quote can cross a chunk boundary.
If none of those rungs finds the place, the quote is not discarded and not treated as found. It is flagged as unverified in the report. You see what the model claims, and whether that claim can be shown in the text.
4. When something fails
If screening a chunk breaks off (a timeout at the provider, a refused request, an unreadable response), then the whole analysis fails visibly. You get an error message rather than a report, and you decide yourself whether to start the analysis again.
That is inconvenient and still the only option we will stand behind. The alternative would be to score the failed chunk's items as missing. Such a report would not look faulty. It would look like a manuscript with reporting gaps when the gap was in the procedure. The score would be biased downwards without anyone noticing.
For the same reason there is no silent failover to another provider or another model. If the chosen provider is unavailable, the analysis fails. It does not continue on something else that the report would then document as something it was not.
5. Scoring
Every checklist item carries four things in the report: the verdict, a short rationale, the supporting quote with its page reference, and a concrete improvement suggestion. The complete report exports as PDF and as JSON. The JSON export carries the same information in machine-readable form.
The verdict itself is exactly one of four. Three of them enter the score. One takes the item out of the calculation:
- Presentcounts 1
- Partialcounts 0.5
- Missingcounts 0
- Not applicabledrops out of the denominator
Present counts 1, partial counts 0.5, missing counts 0. Not applicable drops out of the denominator: an item that does not apply to this study does not lower the score. The compliance score is the sum achieved divided by the number of applicable items.
The score is additionally assigned to one of four bands:
| Band | Range |
|---|---|
| Excellent | 90 % and above |
| Good | 75 % up to but not including 90 % |
| Moderate | 50 % up to but not including 75 % |
| Needs improvement | below 50 % |
The band is assigned on the unrounded value. The display uses the rounded one. Keeping those apart is not a detail. It prevents a statement that is not true: a score of 89.96875 % is displayed as 90.0 % and is still assigned to the good band, because arithmetically it does not reach 90 %. Rounded, it would slip into a band it does not belong in.
The report also works out what individual corrections would be worth. That projection is a calculation on the verdicts already in hand. It does not predict how a revised manuscript would be judged.
6. Provenance
For every analysis, how it came about is recorded: provider, model identifier, checklist key and version, the generation parameters, which key was used, the number of chunks, the tokens consumed and produced, and the start and completion times. These details appear in the report and travel with the PDF and JSON exports.
That makes an analysis auditable, but not identically repeatable. The procedure is documented and can be run again, yet a language model's verdicts are not deterministic. A second run can differ on individual items. That is why every report states which model and which checklist version produced it, and why every finding carries its supporting quote.
7. Limits
Three constraints belong to this method. They are named here because they cannot be written out of the procedure:
- Quote fidelity depends on the model. In our own test runs, smaller models produced quotes that came close to the manuscript but were slightly altered. Those quotes failed verification, which is the correct outcome. The verification mechanism works in those cases. The number of quotes flagged as unverified is higher all the same. Which model was used is stated in the report.
- The report is AI-generated and can be wrong. The verdicts are a working basis for authors to check. This is stated in the application and in the PDF export.
- Page references for unverified quotes are approximate. If a quote could not be located, the application falls back to determining a page from the longest substring that does occur verbatim, so that jumping into the document is possible at all. That reference is a navigation aid and is labelled as one. It does not make the quote verified.