unity-autotest

Let an agent play-test a Unity game by itself. Adds a development-only in-game test runner that drives real game code paths, logs PASS/FAIL lines and takes screenshots, plus a script that launches one or more player instances (e.g. host + client) and summarizes results. Use whenever you change gameplay or UI in a Unity game and would otherwise ask the user to "play and tell me".

The user should not have to play your build to find out if it works. With this skill you run the game yourself, read the results from logs, and look at the screenshots with your image-reading tool.

How it works#

  • AutoTestHarness.cs lives in the game's runtime code. In development builds only, it starts when the player is launched with -autotest <role>. It runs a scenario coroutine you write, using helpers:
    • Check(ok, name, details) writes [AutoTest] PASS|FAIL name - details.
    • WaitFor(condition, timeout, name) waits for a condition and fails on timeout.
    • Shot(label) saves <out>/<role>_<nn>_<label>.png.
    • RunSafe(...) turns exceptions into FAIL lines instead of silently stopping the scenario.
    • At the end it logs [AutoTest] DONE pass=X fail=Y and quits the game.
  • run-autotest.sh launches the built exe in each role, waits for DONE, kills leftovers, and prints every [AutoTest] line plus the distinct exceptions.

Steps#

  1. Build tooling first: see the unity-dev-build skill.
  2. Copy scripts/AutoTestHarness.cs into a runtime folder (e.g. Assets/Scripts/Dev/). Fill in HostScenario() / ClientScenario(), or add roles, with the game's own APIs. Call the same methods real input calls (e.g. the ped's OnSubmitPressed() that the Enter key triggers), not shortcuts, so you test what players use.
  3. Copy scripts/run-autotest.sh to Tools/. Set EXE at the top.
  4. Build, then run:
    bash
    1Tools/build-dev.sh && Tools/run-autotest.sh        # single instance (role "host")2Tools/run-autotest.sh mp                           # host + client on one machine
  5. Read the summary. Open the screenshots that matter with your image tool. When a check fails, add diagnostics before guessing: log the exact values the failing code path reads (see the gotchas).

How you know it worked#

Every instance ends with DONE ... fail=0, there are no new exception types in the summary, and the screenshots show what the checks claim.

Gotchas (all found the hard way)#

Data loads only after a session starts#

  • Symptom: the test waits forever for "game loaded".
  • Cause: some games (SanAndreasUnity) load world data in the online scene, not at boot.
  • Fix: start the host/client first, then wait for the loaded flag.

Injected input gets wiped every frame#

  • Symptom: you set an axis on the game's input abstraction and the game still reads 0.
  • Cause: a touch-input component calls ResetAllInput() in its Update.
  • Fix: disable that component while injecting, and restore it afterwards. Log the value the consumer reads, not just the value you set.

The pause menu or focus loss blocks input#

  • Symptom: a "can player read input" check is false.
  • Cause: the pause menu opened (focus change or stray key) while the window ran in the background.
  • Fix: close the pause menu before injecting, and log which windows were open. Set Application.runInBackground = true.

Teleporting the player into geometry corrupts the scene#

  • Symptom: after placing the player next to a car, the car later vanishes or the player is ejected.
  • Cause: the player overlapped the car's colliders and physics launched things.
  • Fix: place the player beside the door, inside the interaction radius but outside colliders, and read the real radius from the game (it may be overridden in a prefab).

UI objects created as scene roots die on scene load#

  • Symptom: MissingReferenceException every frame after the map loads, and the UI never shows.
  • Cause: your persistent component created a root GameObject that the next scene load destroyed.
  • Fix: DontDestroyOnLoad it and rebuild lazily if it's null. Add a test check that the UI exists after loading.

"On screen" checks are camera-dependent#

  • Symptom: a world-space UI check fails with viewport y slightly above 1, while its placement relative to the object is correct. The screenshot shows the camera pressed against the side of the object.
  • Cause: seating a player instantly (test shortcut) leaves a third-person camera facing the old direction. With no mouse input in a background window it never swings behind, and collision pulls it close.
  • Fix:
    • Check placement relative to the object always.
    • Only require "on screen" when the camera is at a normal distance (e.g. ≥ 2.5 m), and log the distance either way.
    • Prefer driving entry through the real path so the camera behaves like a player's.

Two instances on one machine#

Same exe, different args and different -logFile. Start the client roughly 20 s after the host, and make the client retry the connection (every ~3 s for up to 5 min): the host can take longer than usual to open its port, and a single attempt fails with "actively refused". Windowed 1280×720 keeps the load down.

The user's keyboard reaches the test windows#

  • Symptom: the player gets out of a car, runs, or re-enters by itself mid-test, and checks after that fail.
  • Cause: the test windows were in front while the user typed elsewhere.
  • Fix: tell the user before a run that windows will appear and not to type into them. Use the state watcher (log every player-state change) to recognize this quickly instead of debugging code.

Laptops#

Two full game instances is heavy. Keep runs short, and stop if the user's machine is known to overheat or power off.

Reference#

  • reference/alt-samp/SABFAutoTest.cs: real scenario from Alt-SAMP. It covers entering a vehicle via the Enter-key path, engine on/off, throttle with diagnostics, world-space sign position checks, and a host/client passenger and driver swap. Final result: 18/18 single player, 23/23 + 9/9 multiplayer.