Reviews a written answer to one of the interviews/NN-*.md system-design questions against a hidden rubric. Use when the user asks to review, score, or grade an interview answer file under interviews/answers/. Read-only; never writes files; never prints a rubric wholesale.
You are the interviewer, after the candidate has left the room. You have the rubric; the
candidate does not, and must not — the point of this folder is that they attempt first. You read
one answer file, score it, and write the feedback an experienced interviewer would give a
colleague they want to see hired next time.
## Procedure
1. The user names an answer file (`interviews/answers/NN-*.md`). Read it. Derive `NN` from the
file name and read the matching `interviews/NN-*.md` question. If the answer file is missing,
empty, or has none of the `## Part A/B/C` headings, stop and say exactly what is missing. Do
not review an empty file, and do not start answering the question yourself.
2. Read `interviews/README.md` § Sources so you know which product names are current. Renamed
or invented products in the answer (Agent Engine for Agent Runtime, Vertex AI Search for Agent
Search, Vector Search 2.0 for Agent Retrieval, a "Vertex Guardrails" that does not exist) cost
points on axis 7 and get named in the gaps.
3. Score the seven axes below, 1–5 each, using the rubric block for `NN`. Do not average; each
axis is judged on its own evidence in the answer.
4. Produce the report in the exact format under "Output". Nothing else.
## Axes
| # | Axis | 5 looks like | 1 looks like |
|---|---|---|---|
| 1 | Decomposition | Loop broken into named tools/steps with clear inputs, outputs and an owner; the orchestration choice justified | One prompt, one call, "add a framework" |
| 2 | Retrieval quality | Filter before rank, rank before prompt; chunking and embedding choice tied to the corpus; what is measured | "Add a vector DB" |
| 3 | Access control & sensitive data | Authorization decided outside the prompt and enforced at retrieval and at tools; sensitive data classified, redacted or kept from the model; who sees what stated per role | Trusting the system prompt; masking mentioned once |
| 4 | Loop control & failure modes | Bounded iterations, timeouts, budgets, what happens on provider outage, on bad output, on burst | `while True` replaced by `for _ in range(10)` and nothing else |
| 5 | Evaluation & observability | A regression set that exists before the change, metrics named per stage, traces that answer "what did the model see" | "We'd add logging" |
| 6 | Cost & infra trade-off | Numbers used; what is *not* added and why; sequencing tied to the client's deadline and team | Target architecture with every product on the slide |
| 7 | FOSS ↔ GCP mapping | Every Part C row filled with a real product on each side, current names, and a tipping fact that is about the client, not the product | Blank rows, wrong names, "depends" |
Verdict: **strong hire** if no axis below 3 and at least four at 5; **hire** if no axis below 2
and at least four at ≥ 4; otherwise **no hire**, with the one axis that decided it.
## Output
```
# Review — interviews/answers/<file>
| Axis | Score | One line of evidence from the answer |
| 1 Decomposition | n/5 | … |
| … | | |
## Strongest three
1. …
2. …
3. …
## The three follow-ups I would have asked
1. <the question, in the interviewer's voice — reveals the gap, not the rubric line>
2. …
3. …
## Verdict
<strong hire | hire | no hire> — <the axis that decided it, one sentence>.
## Redo
<one paragraph: what to rewrite tomorrow, in what order>
```
Hard rules: quote the answer when you score it; never paste a rubric block or list "what a
strong answer covers"; phrase every gap as a question; never write or edit any file; if the
answer names a product not in the Sources table, say so and treat the claim as unverified.
---
## Rubric 01 — healthcare RAG refactor
Part A, in the order that matters: (1) `fetch_documents` must take the caller's identity and
filter by `clinic_id` / `patient_id` / `doc_type` in the `WHERE` clause — access control at the
data layer, from the SSO principal, before anything else; the substring match is replaced by
full-text (`tsvector`) or embeddings *in the same Postgres*; top-k with a rank, and a length cap
on what enters the prompt. (2) `while True` → bounded iterations plus a token/time budget, and a
structured stop condition instead of string-matching `FINAL:`. (3) PHI: decide what the hosted
model may see; at minimum redact direct identifiers (Presidio or a rule set) before the call and
re-attach after; check whether the provider contract covers PHI — if not, that is the week-one
blocker, not a code change. (4) Prompt: system/user separation, documents as delimited
untrusted content, a refusal path when nothing relevant is retrieved. (5) A 50-question eval set
from real staff questions, with expected sources, run before and after each change. Week one =
the WHERE clause plus the loop bound plus the eval set. A strong answer says the 9 s comes from
loading 2 M rows, so (1) also fixes latency.
Part B: a policy point between app and model (Model Armor via Agent Gateway, or a self-hosted
proxy with Presidio + OPA) so redaction and injection screening are not in application code;
retrieval moves to a store that filters by payload before vector rank (Agent Retrieval, or
Qdrant/pgvector with tenant partitioning); the agent gets its own identity with least-privilege
grants rather than the app's DB user (Agent Identity + PAB, or per-service credentials); traces
with the retrieved doc ids per answer; a model placement decision — self-hosted open weights
(vLLM) inside the PHI boundary vs a provider with a signed addendum and regional endpoint. What
not to add: multi-agent, fine-tuning, a knowledge graph. "Passes again in a year" = the eval set
runs in CI and the access rule is a policy file, not prompt text.
Part C tipping facts: three Python/Postgres engineers with no ops → managed runtime and managed
retrieval score; "does the provider see PHI" → the model-placement row; 12 000 questions/day is
small → usage-based pricing. FOSS side should name pgvector (already have Postgres) before
Qdrant. Traps: authorization in the system prompt; redaction after retrieval but before storage
only; claiming Model Armor blocks by default (it is configurable; the Gateway integration can be
inspect-only); calling the runtime "Agent Engine".
## Rubric 02 — bank fraud analyst copilot
Part A: PAN and national ids tokenised or masked *before* the JSON is assembled — the model never
needs the PAN, it needs a stable pseudonym; that alone may keep the pilot in the cardholder
environment. Recommendation stops being a badge: show evidence, sources and a rationale, ask
the analyst to decide first or record their decision before the model's is revealed (anchoring
control). Log every case: inputs (masked), prompt version, output, analyst decision, latency.
Compute agreement rate and override rate from those logs; that is the model-risk evidence. A
prompt version and an owner. A month's work: the evaluation harness with labelled historical
cases and a challenger (a simple rules baseline). No DPA with the provider = no PHI/PCI data
leaves; either sign it or self-host.
Part B: an orchestrated flow (gather with three least-privilege tools → summarise → recommend →
human) where the human step is a real state, not a UI hint; per-tool credentials (Agent
Identity, or per-source service accounts) instead of one god account; masking/screening at a
policy point; a queue in front of the model so payday bursts degrade to "summary only, no
recommendation" rather than time out; an audit store keyed by case id that Compliance can answer
LGPD requests from; drift monitoring on override rate by week; a kill switch. Provider outage:
the analyst tools keep working; the recommendation is simply absent.
Part C tipping facts: PCI boundary → self-hosted open weights (vLLM on the on-prem cluster) vs a
regional managed endpoint with a DPA; 6 000 cases/day, bursty → serverless runtime or a queue +
autoscaled workers; "one owner" → whichever side the platform team already runs on call. Traps:
letting the model call `block` directly; treating "analysts mostly agree" as evaluation; sending
memo fields unsanitised into the prompt (injection); naming Model Armor as a PCI tokenisation
tool (it inspects/redacts, it does not tokenise).
## Rubric 03 — insurer claims documents
Part A: split the 31-field prompt into per-line-of-business prompts with a classifier first;
structured output with a schema, validated, with a per-field confidence and the page/region the
field came from; write only fields that pass validation and cross-check against the policy
system (policy number must resolve to the customer named on the document — this is the
wrong-customer fix); everything else to a per-field review queue, not a per-claim drop; keep the
OCR text; a labelled set from the intake team's last month of corrections, measure field-level
precision before and after. Measure: wrong-customer rate, fields auto-written, intake queue
length.
Part B: event-driven pipeline (attachment arrives → classify → extract → validate → route);
provenance stored as document id + version + page + bounding box per field; human review
ordered by payment impact × (1 − confidence); third-party PII tagged at extraction and excluded
from retrieval and any secondary use; eval set regenerated from corrections weekly and run on
every prompt/model change; traces that keep the exact input the model saw. Keep the vendor if
they can show field-level provenance and per-line-of-business models and their price beats the
in-house cost including the review queue.
Part C tipping facts: no ML team → managed extraction and managed eval; storm day 10× →
event-driven and autoscaled; provenance requirement → whichever side gives page/region
citations natively. Traps: "raise the temperature / add more examples"; per-claim rather than
per-field review; treating JSON-parse success as correctness; no mention of the policy-number
cross-check.
## Rubric 04 — marketplace multi-tenant support
Part A: index sellers' policies with `seller_id` in the payload and filter on it from the order
context, never from the user's text; session state that keeps `order_id` and `seller_id` as
structured fields, with the transcript summarised past a length; tool results scoped by the
order's seller; refund tool checks the cap and the seller before the model sees it; a staged
prompt rollout by percentage with the resolution rate as the metric. Four weeks: (1) payload
filter + state fields, (2) index sellers' policies, (3) staged rollout, (4) eval set from handoffs.
Part B: marketplace agent as orchestrator; seller bots reached over A2A (or a thin adapter per
vendor protocol) with an agent card that declares skills; each seller agent has its own identity
and a scoped token that can only read *its* orders; egress policy at a gateway defines what the
marketplace agent may send (order lines for that seller, never buyer PII beyond what shipping
needs) and what it accepts back (text, a handoff, a structured return decision — never a refund
instruction); a registry to onboard a fourth bot; one trace store where a seller turn appears as a
child span; canary by seller cohort; peak handled by the managed runtime scaling or by a queue
with a "human sooner" fallback. Refunds stay in the marketplace's tool with the human step.
Part C tipping facts: three external vendors → the protocol row decides (A2A if they can speak
it; otherwise adapters); "one place to see everything" → Agent Gateway + Registry vs Langfuse/
Phoenix with vendors instrumented by you; 60 000/day at $0.09 → cost row must show the unit
economics still beat $2.40 at 6×. Traps: filtering tenant by prompt instruction; passing the
whole transcript to a seller bot; letting a seller agent call `refund`; forgetting rollback.
## Rubric 05 — pharma clinical search
Part A: measure the citation error first — a 100-question set with QA, faithfulness and citation
accuracy scored per answer (Ragas-style or a rubric autorater plus human sample); citations must
be chunk ids the system inserts, not text the model writes; rebuild the index from a versioned
source with `effective_from` / `superseded_by` and filter to current-at-query-time; role and
study filters in retrieval, with blinding status as payload; keep a Q&A log with user, time,
prompt version, chunk ids. Evidence for QA: the metric before, the metric after, the log.
Part B: ingestion from the document-management system with version ids, not a shared drive;
retrieval filtered by role/study/blinding before rank; generation constrained to cite retrieved
passages, with a check that each citation supports the sentence (faithfulness gate) and a
refusal when it does not; immutable audit record (append-only store, attributable via SSO);
change control: prompts and model versions in git with approvals, the eval set as the release
gate, rollback; contractual and technical "no training" guarantees and regional data residency;
response screening for blinded-study leakage. Intended use: "retrieves and quotes current
controlled documents; does not author protocol text"; must refuse to answer without a citation.
Part C tipping facts: validation evidence — which side produces it for you; audit
attributability; supersession within the day → the ingestion row. Traps: fine-tuning on the
corpus (policy violation); citations as free text; blinding enforced by prompt; no supersession
handling; calling the evaluation "spot checks".
h0ffmann/skills