Ask Copilot to Write the Test Cases

In the previous section we got Vitest running with a single trivial test. That test proves the runner works, but it does not cover the behavior of evaluateGuess. Now we want a set of tests that cover the duplicate-letter rule and the common edge cases around it, so we have a regression guard against that bug coming back. We can ask Copilot to write them for us.

Ask Copilot to Expand the Tests

Open the Chat view, start a new chat (the + button), and send a prompt like:

Please add more tests to #file:evaluate.test.ts to cover the basic behavior of #sym:evaluateGuess and edge cases around it.

Notice that I referenced the file and the function with #file: and #sym: tags. #file: attaches a whole file, and #sym: points at a specific symbol inside it: a function, a class, a type. You can add either one by typing # and picking what you want from the dropdown that appears. This way, Copilot knows exactly which file to edit and which function to focus on when generating the new test cases.

Copilot Chat view with the prompt typed out and plus attached as chips above the input

Read Through What Copilot Generated

Copilot reads the test file and the evaluateGuess function, and then generates a set of new test cases that cover the duplicate-letter rule and other edge cases. In my case, it generated the following tests:

import { describe, it, expect } from "vitest";
import { evaluateGuess } from "./evaluate";

describe("evaluateGuess", () => {
  it("returns a color for each letter in the guess", () => {
    const result = evaluateGuess("APPLE", "SPEED");
    expect(result).toHaveLength(5);
  });

  it("marks every position correct when guess equals target", () => {
    const result = evaluateGuess("CRANE", "CRANE");
    expect(result).toEqual([
      "correct",
      "correct",
      "correct",
      "correct",
      "correct",
    ]);
  });

  it("marks letters present when they exist in a different position", () => {
    const result = evaluateGuess("ABCDF", "FBCDA");
    expect(result).toEqual([
      "present",
      "correct",
      "correct",
      "correct",
      "present",
    ]);
  });

  it("marks letters absent when they are not in the target", () => {
    const result = evaluateGuess("ZZZZZ", "CRANE");
    expect(result).toEqual(["absent", "absent", "absent", "absent", "absent"]);
  });

  it("does not over-credit duplicate letters in the guess", () => {
    const result = evaluateGuess("ALLOT", "CANAL");
    expect(result).toEqual([
      "present",
      "present",
      "absent",
      "absent",
      "absent",
    ]);
  });

  it("handles mixed correct and present with duplicate letters", () => {
    const result = evaluateGuess("EERIE", "SHEEP");
    expect(result).toEqual([
      "present",
      "present",
      "absent",
      "absent",
      "absent",
    ]);
  });

  it("returns an empty result for an empty guess", () => {
    expect(evaluateGuess("", "CRANE")).toEqual([]);
  });

  it("uses the guess length for evaluation when lengths differ", () => {
    expect(evaluateGuess("CAT", "CRANE")).toEqual([
      "correct",
      "present",
      "present",
    ]);
    expect(evaluateGuess("CRANES", "CRANE")).toHaveLength(6);
  });
});

Two of these cases deserve a second look. The empty-guess case and the differing-length case describe behavior that never comes up in normal Wordle play. Do not keep them unless you decide that they belong to the function’s contract.

Copilot then asks to run pnpm test. Let it run if you want to see what goes wrong, but do not treat what happens next as good testing practice. Two things go wrong right away:

  • One of the tests fails.
  • Copilot stalls because pnpm test runs in watch mode and never exits on its own.

They show why generated tests still need a human to review the expected outputs and the commands being run.

If Copilot had looked at the scripts in package.json, it would have run pnpm test --run or pnpm test:run instead.

At any rate, stop the test watcher in the chat. As soon as you do this, Copilot will continue:

One edge expectation was off in the differing-length case; I’m updating that test to match the evaluator’s actual two-pass behavior, then I’ll re-run tests in non-watch mode.

In my case, it updated the last test to this:

  it("uses the guess length for evaluation when lengths differ", () => {
    expect(evaluateGuess("CAT", "CRANE")).toEqual([
      "correct",
      "present",
-     "present",
+     "absent",
    ]);
    expect(evaluateGuess("CRANES", "CRANE")).toHaveLength(6);
  });

It then asked to run the tests, and this time asked for permission to run pnpm test --run instead of pnpm test. I allowed it, and the suite ran with all tests passing.

Copilot can generate a lot of testing boilerplate quickly, but it can also invent the wrong expected behavior and then change the test to match it after seeing what the implementation does. A green test suite is only meaningful if the expectations came from a spec you trust.

Hold Copilot Accountable for the Tests

I continued the chat with Copilot and said:

So, one test failed in your initial attempt, and then you fixed it. Why did that happen? How can I know now if the evaluateGuess is actually correct, or did you just tailor the unit test to its erroneous behavior so the test passes?

Copilot responded with:

Great question. The initial failure was my mistake in expected output, not a code change to force green tests.

Why that one failed:

1. I added this edge-case test in evaluate.test.ts: evaluateGuess("CAT", "CRANE").
2. I initially expected ["correct", "present", "present"], but the real result is ["correct", "present", "absent"].
3. Walking through the algorithm in evaluate.ts:
   - First pass (exact matches + remaining target counts): C is correct; unmatched target letters counted are R and A.
   - Second pass (present allocation): A gets one present; T is not in remaining counts, so absent.
4. So the fixed expectation is consistent with the two-pass duplicate-safe logic, not “tailored to a bug.”

Your core concern is valid:

- Tests can pass while behavior is still wrong if tests encode the wrong spec.
- The best protection is to test against an independent oracle, not the same mental model used to write the function.

How to increase confidence now:

1. Add a reference implementation inside tests (separately written logic) and assert evaluateGuess matches it over many random 5-letter pairs.
2. Add invariant/property tests:
   - Result length equals guess length.
   - Number of correct+present for any letter never exceeds that letter’s count in target.
   - If guess === target, all are correct.
3. Decide and enforce length policy:
   - If only 5-letter words are valid, assert equal lengths and throw otherwise in evaluate.ts, then test that behavior.
   - Right now, the “different lengths” test in evaluate.test.ts documents current behavior rather than Wordle rules.

If you want, I can add the oracle-based randomized test next so correctness is validated against an independent scorer, not just hand-picked examples.

So here is what to do when this happens. Watch the test fail, question the expectation, and decide whether the test is specifying Wordle’s rules or just repeating what the implementation already does. Do not keep a changed expectation only because it makes the suite pass.

Copilot’s suggestions are real improvements we could make: oracle tests, invariants, a clear length policy. They are more than we set out to do in this chapter, though. For this section, we have the regression guard we wanted: if anyone breaks the duplicate-letter rule again, at least one of these tests will fail. Press the “Keep” button in the chat view and then run the full suite again to confirm everything is green.

Note that the behavior reported here can be different from what you see when you try it, depending on which language model is powering Copilot at the time.

Checkpoint: Commit your progress.

git add .
git commit -m "wordle-3: Expand evaluate.test.ts with regression coverage for the duplicate-letter rule"
git push