MLPerf separates inference workloads by testing conditions and quality requirements.[1] A buying decision for private AI needs the same discipline: define what counts as useful work before comparing costs. Our position is that cost per accepted task is a better pilot measure than a token price on its own.
The measure is straightforward. Add the costs assigned to the pilot, including review and correction, then divide by the number of outputs that pass the agreed acceptance check. This is a proposed evaluation method, not a claim about savings already achieved by Software Tailor customers.
Name the finished job
A document summary is not finished simply because a model produced paragraphs. The pilot team might require the summary to identify the decision, preserve every deadline, and provide a route back to the relevant source passages. An output that misses the deadline needs correction before it counts as accepted.
Write those conditions before testing. Use a small collection of permitted documents with different lengths and layouts. Include an awkward example: a scanned page, a conflicting date, or a table whose meaning depends on a footnote. Keep the same input set for every candidate configuration so changes in the work do not masquerade as improvements in the model.
Our AI PDF Reader is one possible starting point for a document workflow. The evaluation should still assess the actual source and answer together. The existence of a citation is useful for review; its presence is not the acceptance decision.
Record rejected outputs as well as accepted ones. Otherwise the report describes the best cases while the team pays for the whole queue.
Put operating effort in the numerator
For a local deployment, allocate a share of the equipment cost over a stated evaluation period. Add measured electricity where it is material, software charges that apply to the chosen configuration, and the time spent setting up or maintaining the service. State the assumptions beside the result. An existing workstation with spare capacity and a newly purchased dedicated server are different purchasing situations.
For a hosted model, record the charges for the pilot's actual requests, including retries. Also count the same review and integration work included in the local case. Applying a complete cost model to one route and a narrow model to the other produces a comparison that cannot support a decision.
Review time needs an explicit valuation. It is reasonable to report both elapsed minutes and an estimated labour cost, provided the estimate is labelled. Do not present a staff member's recovered minutes as cash savings unless the organisation has a credible way to realise them. A shorter task can be valuable even when payroll does not change.
Keep exceptional setup work separate from recurring operations. The team can then see whether a disappointing first week reflects installation effort or a continuing problem with the workflow.
Measure the deployed configuration
Hugging Face's model-card format supports information about intended use, limitations and evaluation.[2] Treat that documentation as the beginning of the candidate record. Add the exact model revision used, its local file or variant, the runtime version, and the machine that ran the test. A model name alone is insufficient to reproduce a purchase evaluation.
The same principle applies to performance evidence. MLCommons describes benchmark results in relation to the submitted system and software, and distinguishes comparison divisions.[1] A published benchmark can help frame a question. It does not supply a measured result for an untested office workflow.
Test a cold start and an already running service separately. Record time to a useful response as well as the completion time. If two people will use the system together, test that condition rather than extrapolating from a single request. These are pilot design choices, not claims that every deployment has the same bottleneck.
The AI Server product page describes the server route for shared inference. Use that route when a shared service fits the proposed job; do not add a server merely to make the pilot resemble a future organisation-wide deployment.
Report a decision that can be checked
The final record should show the input set, acceptance rules, accepted count, review time and cost assumptions. Include the most consequential failure with its source material suitably redacted. A reviewer should be able to understand why the team preferred one configuration without relying on the author's enthusiasm.
If quality is unacceptable, a cheap result does not rescue it. If quality is acceptable but operating effort is excessive, narrow the workflow or change the configuration and rerun the same test. Our companion article on evidence for private AI in critical infrastructure applies this habit to a different decision boundary.
Start with one task available in the Software Tailor product catalogue, define its acceptance check, and keep the first costing sheet small enough to audit. The useful number is the cost of work that can be used.
References
- MLCommons. MLPerf Inference: Datacenter. Accessed 2026-09-12.
- Hugging Face. Model Cards. Accessed 2026-09-12.
Related articles
- Private AI and critical infrastructure: keep the evidence specific
- Keeping an honest record of AI-assisted articles