Learning to check the answers

An AI answer can contain several kinds of statement in the same paragraph. It might describe a function accurately, infer a retry policy from a method name, and conclude that a proposed change prevents duplicate emails. The prose can make those statements feel equally established even though they depend on different evidence.

Checking the answer means separating those claims and finding out what would support each one. Source code can establish the order of operations in a handler. A controlled experiment can show what happens when persistence fails. A provider's contract determines what a repeated request means outside the application.

The previous design discussion proposed preserving a notification request and submitting retries with a stable idempotency key. That gives us a concrete answer to examine. The interesting question is how much of its promised behaviour we can establish ourselves, and where the evidence runs out.

Claims and assumptions #

Suppose the assistant describes the proposal like this:

The worker uses the notification ID as its idempotency key, so retries cannot send duplicate emails.

The first part can be checked in the implementation. The conclusion reaches across the provider boundary and all possible retries. It includes assumptions about key retention, concurrent requests, adapter behaviour, and what “send” means.

A narrower claim is easier to investigate: if the provider accepts a request and local persistence then fails, a retry submits the same request with the same key. Under the assumed provider contract, those submissions refer to one accepted operation.

Here is a self-contained version of the application boundary. As in the design article, sent means provider acceptance. The queue invokes this function again after a failed attempt; queue handling itself is outside this example.

interface EmailRequest {
  to: string;
  subject: string;
  body: string;
}

interface NotificationRecord {
  id: string;
  request: EmailRequest;
  status: "pending" | "sent";
  providerMessageId?: string;
}

interface ReceiptStore {
  load(id: string): Promise<NotificationRecord>;
  markAccepted(id: string, providerMessageId: string): Promise<void>;
}

interface EmailProvider {
  send(
    request: EmailRequest,
    options: { idempotencyKey: string },
  ): Promise<{ messageId: string }>;
}

async function submitNotification(
  id: string, store: ReceiptStore, provider: EmailProvider,
) {
  const notification = await store.load(id);
  if (notification.status === "sent") return;

  const receipt = await provider.send(notification.request, {
    idempotencyKey: notification.id,
  });
  await store.markAccepted(notification.id, receipt.messageId);
}

Reading the function establishes that the key comes from the stored notification ID. It also shows that the local success record is written after the provider call. It leaves open what happens when that write fails and what the provider does with a repeated key.

The engineer can ask the assistant to address the first gap with an experiment:

Exercise the case where provider acceptance succeeds, persistence fails, and the same notification is retried. Show the provider submissions, accepted operations, and final local state separately.

That question defines what needs observing. A test that only checks whether send was called would miss the distinction between submitting twice and creating two provider operations.

Failure simulation #

For a local experiment, we can model a provider that remembers each key and returns the same receipt when the same request is repeated. This is an explicit assumption encoded in a fake. Its memory lasts for the lifetime of the instance.

class FakeProvider implements EmailProvider {
  submissions: string[] = [];
  accepted = new Map<string, { payload: string; messageId: string }>();

  async send(request: EmailRequest, options: { idempotencyKey: string }) {
    const key = options.idempotencyKey;
    const payload = JSON.stringify(request);
    this.submissions.push(key);

    const previous = this.accepted.get(key);
    if (previous) {
      if (previous.payload !== payload) {
        throw new Error("Same key used with a different request");
      }
      return { messageId: previous.messageId };
    }

    const messageId = `email-${this.accepted.size + 1}`;
    this.accepted.set(key, { payload, messageId });
    return { messageId };
  }
}

The fake exposes both submissions and accepted operations. Checking the payload also makes a changed request visible. Its string comparison is sufficient for these fixed-shape example objects; a real adapter's comparison rules would come from its contract.

The store can then fail exactly once, after the provider has accepted the first submission. The following framework-free exercise runs the function twice and checks the observations:

function assert(condition: boolean, message: string): asserts condition {
  if (!condition) throw new Error(message);
}

async function verifyRetryBoundary() {
  const provider = new FakeProvider();
  const persistenceFailure = new Error("Injected persistence failure");
  let failNextWrite = true;
  const record: NotificationRecord = {
    id: "N42",
    request: {
      to: "reader@example.test", subject: "Updated", body: "Request changed",
    },
    status: "pending",
  };
  const store: ReceiptStore = {
    load: async (id) => {
      assert(id === record.id, "Unexpected notification");
      return structuredClone(record);
    },
    markAccepted: async (id, messageId) => {
      assert(id === record.id, "Unexpected notification");
      if (failNextWrite) {
        failNextWrite = false;
        throw persistenceFailure;
      }
      Object.assign(record, { status: "sent", providerMessageId: messageId });
    },
  };

  try {
    await submitNotification("N42", store, provider);
    throw new Error("Expected persistence to fail");
  } catch (error) {
    assert(error === persistenceFailure, "Unexpected failure");
  }
  assert(record.status === "pending", "First write must not succeed");

  await submitNotification("N42", store, provider);

  assert(provider.submissions.join(",") === "N42,N42", "Key changed on retry");
  assert(provider.accepted.size === 1, "Expected one accepted operation");
  const saved = await store.load("N42");
  assert(saved.status === "sent", "Retry must persist acceptance");
  assert(saved.providerMessageId === "email-1", "Receipt must match");
}

await verifyRetryBoundary();

This supplies evidence about a specific failure window. The first run reaches the provider and fails at persistence. The second makes another submission with the same key, receives the same receipt, and records acceptance. The experiment would fail if the implementation swallowed the persistence error and pretended the operation had completed locally.

It does not exercise worker termination, queue redelivery, or a real database. Injecting a rejected write models the local state left by this particular failure. A process stopping at a different point could require another experiment.

Test sensitivity #

A passing exercise becomes more informative when a plausible defect makes it fail. The engineer can temporarily change the key construction:

idempotencyKey: `${notification.id}:${crypto.randomUUID()}`,

Each invocation would now submit a new identity. In this experiment the fake would accept two operations, and the stable-key assertion would fail. Returning to the original implementation should restore the passing result.

That small intervention checks whether the experiment is sensitive to the defect it claims to detect. It also exposes a weakness in a simpler test that might assert only that the final record is marked sent. Such a test could pass even though the application created two provider operations.

An assistant can help propose these deliberate variations. The engineer still needs to inspect what they change. If the same conversation invents the implementation, fake, and assertions, all three can share an incorrect interpretation of the requirement. Expected behaviour needs an anchor outside that agreement: the product decision, interface contract, or observed failure the change is meant to address.

Verification scope #

The local exercise demonstrates how the application behaves with the provider semantics encoded in FakeProvider. It provides no evidence that a real provider retains keys indefinitely or treats concurrent requests in the same way.

The actual adapter must pass the stable key to the documented API field. The provider's applicable contract must cover repeated identical requests, conflicting payloads, concurrent submissions, and retention limits. If the application retries after that retention period, the experiment's indefinitely remembered key no longer represents the external system.

A sandbox integration exercise can check that the adapter sends the intended fields and can probe supported behaviours. Its result still needs a precise description of the environment, API version, and scenario exercised. A successful request is weak evidence for a guarantee about every failure and replay condition.

This distinction changes the answer the team can carry forward. “The local retry exercise passed under an idempotent-provider model” is established by this code. “The production integration satisfies our replay requirements” needs additional evidence about the adapter, provider contract, and retry policy.

Evidence and conclusions #

In a conversation, it is easy for a proposed check to become a reported result. An assistant may supply a test and then describe the behaviour as verified. The code has to run before its outcome can support that claim, and the tested revision matters when implementation continues changing.

A useful result in the issue or review would identify the failure injected, the revision exercised, and the observations: two submissions using N42, one accepted fake-provider operation, and a persisted receipt after retry. It would also record the provider assumptions that remain to be confirmed.

That account lets another engineer judge what the check means without trusting the confidence of the original answer. It gives the team something to revisit when the adapter changes or the allowed replay period grows.

Checking AI answers becomes part of ordinary engineering work when claims are connected to inspectable evidence. A conversation can help formulate the question and build the experiment. The team decides what the result establishes and whether that is enough to proceed. Those habits become more consequential when the assistant starts changing the code itself.

Continue exploring

• The first useful conversation AI conversation helping investigating the problem Same collection · Same subsection
• Understanding the codebase Trace behaviour through an unfamiliar repository Same collection · Same subsection
• Thinking through a design Use an AI conversation to examine design assumptions Same collection · Same subsection