Budget Human Evaluation Before Widening an AWS AI Pilot

Reconcile comparison bundles, calibration, second review and adjudication against separate staffed lanes. Work favorable and adverse budgets without hiding unresolved...

Choose an AI comparison's breadth only after budgeting the human work needed to make its results interpretable. Count case preparation, reviewer calibration, first judgment, independent second judgment and disagreement resolution in separate staffed lanes. More generated responses do not create more qualified evaluation capacity. If one lane cannot complete its required work, narrow the study or obtain authorized capacity rather than calling unreviewed outputs successful.

This article helps an evaluation lead price the human evidence work for a finite Bedrock-related comparison. It does not estimate a production queue's service level or supply staffing benchmarks. All counts, minutes and prices below are hypothetical planning inputs. No AWS jobs, customer review sessions or timed human observations were executed. A favorable budget is a feasible arithmetic proposal, not evidence that reviewers can deliver those times or permission to begin processing data.

Define whether a review unit is an output or a comparison bundle

One case may produce two candidate answers. A reviewer might score each answer separately or compare them together with the same source evidence. These are different work units. If six minutes covers reading the source and judging both answers as a bundle, multiplying by two outputs double-counts that visit. If six minutes covers each output independently, multiplying only by the case count omits half the initial work.

Write the unit and included visits before supplying a rate. Record whether it covers source retrieval, evidence verification, labels, reasons and corrections. Add repeat visits only when the initial rate excludes them. A retained case identifier joins the two methods, but their outputs and versions remain separately attributable. Do not replace distinct disagreements with one combined “case accepted” flag.

Amazon Bedrock's human evaluation dataset documentation supports supplied inference responses with one or two responses per line and consistent model identifiers. That format is not a formula for human effort. Record which proposed workflow the budget represents, and check the actual job's model and Region support before planning it. A four-candidate application study may require a different job arrangement; it is not automatically one supported four-response dataset.

A fair comparison also needs a credible incumbent or non-AI alternative. The baseline selection article owns that choice. Here the budget assumes the selected alternatives have already been defined. If they require different source access or reviewer qualifications, make that difference explicit rather than imposing one average time on all of them.

Separate calibration from independent evidence

Calibration lets reviewers learn the task and discuss examples. Its time belongs in the budget, but the discussed cases cannot also supply untouched independent agreement evidence. Record rubric revisions and the examples that informed them. Once the rubric is stable enough for the question, collect independent judgments before reviewers see each other's reasoning or a preferred candidate label.

NIST's randomized block design guidance describes controlling nuisance factors such as operators or conditions. For this application, reviewer expertise, case difficulty and order can change observed effort or preference. A balanced assignment and masked candidate labels may help, but do not guarantee that reviewers cannot recognize a model's style. Budget the administration and explain the residual limitation instead of claiming complete blinding.

Preserve adjudication as a distinct task. A second reviewer independently judges a specified subset. An adjudicator then resolves defined conflicts using evidence and the frozen rubric. If adjudication requires new information, the case remains open until that information arrives. Recording agreement after a group discussion is not the same measurement as independent first judgments. Neither high agreement nor a negotiated label alone proves validity.

The AWS work-team guidance covers chosen workers and instructions. Creating an account or assigning a job does not establish available minutes, qualifications or delivery. Confirm the reviewer allocation with the relevant manager and distinguish ordinary operational review from this study's independent evidence work.

Worked budget: a two-candidate comparison with three lanes

Stipulate 120 source-evidence cases and two candidate outputs per case. The primary reviewer judges each two-output bundle in six active minutes. A second reviewer independently judges 30 of those bundles, also at six minutes. Twelve cases require additional adjudication at ten minutes each. These are supplied assumptions, not a recommendation that only 25 percent of cases should receive second review.

Before those visits, both reviewers spend 45 minutes each on separate calibration work. The primary reviewer also has 60 preparation minutes to reconcile the supplied case register and instructions. Those preparation minutes exclude source-rights assessment and actual generation. Adjudication minutes include the stated resolution visit, not the first or second review already counted. Keep that boundary if replacing the assumptions with observed work.

First comparison visits

Supplied calculation: 120 × 6

Active minutes and lane: 720 primary

Independent second visits

Supplied calculation: 30 × 6

Active minutes and lane: 180 secondary

Additional adjudication

Supplied calculation: 12 × 10

Active minutes and lane: 120 adjudicator

Reviewer calibration

Supplied calculation: 2 × 45

Active minutes and lane: 45 primary + 45 secondary

Case and instruction preparation

Supplied calculation: 1 × 60

Active minutes and lane: 60 primary

Total included human work

Supplied calculation: 720 + 180 + 120 + 90 + 60

Active minutes and lane: 1,170 minutes

Allocate 900 primary, 300 secondary and 150 adjudicator minutes within the intended observation window. Primary need is 720 + 45 + 60 = 825, secondary need is 180 + 45 = 225 and adjudication need is 120. The separate margins are 75, 75 and 30 minutes. Combined allocated capacity is 1,350 and total margin is 180, but that combined number cannot transfer a qualified adjudicator's work to a primary reviewer.

The budget leaves model generation, platform charges, rights/security assessment, engineering changes, waiting and post-study operating review outside its human-work subtotal. That exclusion is explicit, not a claim those costs are zero. Preparing 120 cases can itself exceed the supplied hour when source records are difficult; challenge that assumption before treating the proposal as staffed.

Adverse review work can exhaust a lane before the total looks alarming

Keep 120 cases and the same allocations, but change both review visits to eight minutes and assume 18 adjudications at twelve minutes. Primary need becomes 120 × 8 + 45 + 60 = 1,065. Secondary need is 30 × 8 + 45 = 285. Adjudication needs 18 × 12 = 216. Total included work is 1,566 minutes. Primary exceeds its lane by 165 minutes and adjudication by 66, while the secondary lane still has 15 minutes spare.

Do not erase the conflict queue or ask an unqualified reviewer to supply the missing 66 minutes. Retain unresolved cases and the consequence for the study decision. Additional candidate generation while adjudication is blocked can increase unusable output inventory. Pause the affected comparison, preserve its records and get the study owner to choose additional capacity, a narrower question or a revised deadline.

Reducing to 80 cases could make the stated adverse visit times feasible: first visits use 640 minutes, second visits on 20 cases use 160 and 12 adjudications use 144. With unchanged calibration and preparation, total need is 1,094. The lanes require 745, 205 and 144 minutes, leaving 155, 95 and six. This is a different study with narrower evidence, not proof that the original 120-case design finished.

The six-minute adjudication margin is fragile. If 14 cases instead need twelve-minute adjudication, that lane requires 168 minutes and exceeds its allocation by 18 even though the combined total is only 1,118, below 1,350. The study owner must evaluate the loss of case coverage and this adverse scenario before accepting the reduced plan. Never select a smaller sample solely to obtain a convenient pass fraction.

Widening the candidate set changes the bundle

Suppose a separate fictional proposal keeps 120 cases but adds two candidates, making four outputs per case. Stipulate ten minutes per first bundle and ten for each of 30 second bundles. Keep the original 12 ten-minute adjudications, calibration and preparation only as explicit lower-complexity assumptions. The subtotal becomes 1,200 + 300 + 120 + 90 + 60 = 1,770 minutes. Primary need is 1,305 and secondary need is 345, exceeding their allocations by 405 and 45.

More candidates can also increase disagreement and rubric complexity, so unchanged adjudication and calibration are not established. Re-estimate them before commissioning. If the goal is to eliminate clearly unsuitable candidates, an approved development screen can reduce the later comparison. It cannot reuse the final acceptance set for repeated winner selection while keeping an untouched-test claim.

Reducing independent second review is another design change. It may lower effort while weakening the evidence for grader reliability or consequential judgments. Record which endpoints would no longer be supported. A model-based grader can help prioritize review after task-specific validation, but its token cost is not a substitute for the absent qualified judgment. The study is not complete because an automated score exists for every output.

Translate effort into commitments without inventing savings

Separate allocated time from paid cost. A salaried employee's study allocation is a capacity commitment even when it creates no new invoice. External evaluator fees, overtime and reassigned internal work follow different commercial rules. Use an actual rate and basis if a cost comparison is needed. Do not call all allocated minutes incremental cash expense or report a salary-based estimate as an observed saving.

For the favorable 120-case budget only, stipulate USD 60 per primary hour, 90 per secondary hour and 120 per adjudicator hour. The included human amount is 825/60 × 60 = 825, plus 225/60 × 90 = 337.50, plus 120/60 × 120 = 240, totaling USD 1,402.50. Adding a separately supplied USD 80 platform allowance gives USD 1,482.50 on that declared boundary. These are teaching values, not AWS prices, Ampity rates or a full project quote.

If preparation already appears in an external fixed study fee, do not add it again as a separate invoice. If a rate covers both calibration and live judgments, keep the ledger of minutes but reconcile the charge basis rather than billing each activity twice. Accepted-task cost requires outcomes from the same study and complete relevant costs. A prospective budget cannot borrow another cohort's correct-answer count to claim observed unit economics.

Next action: allocate a finite comparison record

Decision and unit

Name the study question, case count, candidate count and whether a visit covers an output or a complete bundle.

Included effort

Separate preparation, calibration, first judgment, independent second judgment, additional adjudication and repeat visits. Identify overlapping rates.

Qualified lanes

Record named role allocations within the same window, qualification/authority constraints and known absence. Combined spare minutes cannot substitute for a missing role.

Uncertainty and breadth

Retain adverse visit times, disagreement counts, unresolved labels and the evidence lost when reducing cases or candidates.

Cost and exclusions

State currency, rate provenance, paid versus allocated basis, platform allowance and missing project costs. Do not infer saving from a capacity margin.

Hold and recovery

Pause the unsupported comparison, preserve completed and disputed records, and choose additional capacity or a revised study. Bind any restart to its new case/rubric/version record.

The review-queue capacity article owns ongoing arrivals, waiting and backlog recovery. The pilot decision paper owns whether another experiment can change the investment choice. For this finite study, ask the evaluation lead and the three lane owners to replace the supplied minutes with a permitted calibration observation, reconcile the included visits and challenge the smallest margin before widening the candidate set.