Opentrons · Flex harness

flex-testing-agent

Release-driven robot test suggestions with live Jira and monorepo PR links.

← All suggestions

validated · updated 2026-08-12

Flex 10.0.0-alpha.1 release-delta validation (chore_release-10.0.0 since alpha.0)

Install-ready checklist for external v10.0.0-alpha.1 on KansasFLEX. Covers the eight monorepo commits merged into chore_release-10.0.0 after v10.0.0-alpha.0 (PRs #22216–#22226), plus a short Pyro / protocol-subprocess regression from the alpha.0 suite. File new product bugs under RQA-5847. Narrative helpers: docs/pyro-testing.md and docs/robot-logs.md.

Jira RQA-5847 10.0.0-alpha.1 BugsJira RQA-5848 GET /health diskDetails.runStartLimitFreeSpaceMb reports wrong thresholdJira RQA-5849 CRS-on POST /protocols/{id}/analyses returns 500 AssertionErrorJira RQA-5484 10.0.0 BugsJira RQA-5843 Run commands do not appear in robot log JSON in downloaded log period zipJira RQA-5830 Modules do not save calibration offsetsJira RQA-5818 App: reanalyze protocol doesn't work unless you refresh/restart appJira EXEC-2877 Gate protocol analysis creation/deletion behind access controlJira EXEC-2663 Add filesystem storage usage management to the audit serverJira RQA-5846 POST /runs intermittently 500: Can't resolve pyro proxy ot-protocol_<uuid>Jira RQA-5835 [Pyro and APP]: Robot becomes unreachable through the app after running and/or device resetPR #22216 feat(audit-server): add disk usage monitoringPR #22217 feat(app): audit logs UI updatesPR #22221 refactor(app,robot-server): move disk status to serverPR #22213 fix(app,robot-server): Require authentication and documentation for robot-side protocol analysisPR #22224 fix(robot-server): clearing a run in pyro returns actual data for commands and other run result propertiesPR #22226 fix(server-utils): update period id typePR #22215 fix(api): Stop caching module offsets across pyro processesPR #22222 fix(app): fix stale analysis RTPs in ChooseRobotToRunProtocolSlideout
Monorepo branch chore_release-10.0.0
Compared tov10.0.0-alpha.0
Robot OS10.0.0-alpha.1
Hardware flex_1channel_50 (right) or existing KansasFLEX pipettes, temperatureModuleV2 or thermocyclerModuleV2 (for module-offset fix only), protocol with runtime parameters (for App RTP slideout only)

Harness commands

uv run flex-test inspect
uv run flex-test status
uv run flex-test releases
ALLOW_MUTATIONS=true uv run flex-test put 10.0.0-alpha.1 --channel external
uv run flex-test wait-health --timeout 1200 --interval 5
uv run flex-test probe --no-picture
ALLOW_MUTATIONS=true uv run flex-test api-suite
uv run flex-test audit list
uv run flex-test audit download <periodId>
uv run flex-test logs archive

Tests

IDPlan
I0
Confirm build published then install
pass

External CDN already lists 10.0.0-alpha.1 / v10.0.0-alpha.1; install before fix validation.

  • uv run flex-test releases (external alpha lane should show 10.0.0-alpha.1).
  • uv run flex-test inspect (record current host / version; DHCP may move KansasFLEX).
  • ALLOW_MUTATIONS=true uv run flex-test put 10.0.0-alpha.1 --channel external (CRS-on needs ROBOT_USERNAME/PASSWORD).
  • uv run flex-test wait-health --timeout 1200; expect GET /health 200 and API version 10.0.0-alpha.1.
  • If still 502 after several minutes: full robot reboot, then wait-health again (RQA-5787 class).

2026-08-12 KansasFLEX put succeeded; resulting v10.0.0-alpha.1; host 192.168.0.21; CRS off.
A1
Pyro stack healthy after install
pass

Regression gate; alpha.1 still ships enableHardwareSubprocess + enableProtocolSubprocess.

  • SSH or serial: systemctl is-active opentrons-pyro-nameserver opentrons-hardware-api opentrons-robot-server.
  • Confirm /data/feature_flags.json has both subprocess flags true.
  • GET /health, /instruments, /subsystems/status, /robot/door/status via flex-test inspect|status|probe.

All three units active; enableHardwareSubprocess/enableProtocolSubprocess true; NS OT3API + robot-server-resource.
A2
Create-run / analysis smoke (no motion)
pass

Catch RQA-5846-class ot-protocol resolve failures early before longer suites.

  • Upload docs/test-suggestions/protocols/pyro_smoke_no_motion.py; wait analysis completed.
  • POST /runs; expect 201; NS shows ot-protocol_<uuid> (or equivalent).
  • Stop the idle run; note any 500 Can't resolve pyro proxy (file under RQA-5847 if new).

Upload/analyze/create-run 201; play succeeded (home+comment). Later api-suite Tier C hit RQA-5846.
F1
Cleared run retains commands (RQA-5843 /
pass

Pyro clear_pyro used to return empty stub commands/parameters/annotations and skip clear_command_history; audit run-log JSON then looked empty.

  • CRS on preferred (matches discovery path). Create a current run from a protocol that produces commands.
  • Prefer a short non-motion protocol, or seed_simple_home_move only if live motion is explicitly approved.
  • Play or otherwise populate command history; confirm GET /runs/{id}/commands is non-empty while current.
  • Stop / uncurrent (CRS-on may need run signoff). Confirm GET /runs/{id}/commands still returns the real command list (not empty stubs).
  • Also check parameters / commandAnnotations / commandPreconditions are present when expected.
  • uv run flex-test audit list; download the period covering the run; unzip and confirm robot run-log JSON contains those commands (RQA-5843).

CRS-on disruptive retest 2026-08-12: play+signoff+uncurrent kept 2 commands (home, comment). audit download period 20 zip contains full command list (RQA-5843 fix confirmed on audit path). Evidence: artifacts/pyro-tests/alpha1-audit-20.zip, artifacts/pyro-tests/alpha1-crs-on-f3-f6-f1.json.

F2
Health diskDetails + CRS low-space run gate (#22221)
pass

Server now owns disk status in GET /health (runStartLimitFreeSpaceMb, isDiskSpaceBelowRunStartLimit). With CRS on, POST /runs returns 507 when free space is below the limit (~2048 MB default).

  • GET /health; confirm diskDetails includes systemAvailableMb, systemTotalMb, imagesDirectorySizeMb, runStartLimitFreeSpaceMb, isDiskSpaceBelowRunStartLimit.
  • Normal free space: isDiskSpaceBelowRunStartLimit false; POST /runs still works.
  • Optional stress (disruptive): create a large file under /data until free < runStartLimitFreeSpaceMb.
  • CRS on: POST /runs → 507 Insufficient Storage (NoSpaceForRun). CRS off: run create still allowed (per PR test plan).
  • Remove the filler file; confirm health flag clears and runs work again.

diskDetails present (RQA-5848: runStartLimitFreeSpaceMb=250 vs gate 2048). Disruptive fill 2026-08-12: ~1.5GB free, isDiskSpaceBelowRunStartLimit true. CRS off POST /runs → 201; CRS on → 507. Filler removed; /data recovered. Evidence: artifacts/pyro-tests/alpha1-f2-disk-stress-crs-off.json, alpha1-f2-disk-stress-crs-on.json.

F3
CRS analysis requires auth + reason (#22213 / EXEC-2877)
fail

POST /protocols/{id}/analyses is now access-control gated when CRS is on.

  • CRS on. Upload a protocol (labware-free comment-only protocol is fine for reanalysis UX).
  • Unauthenticated POST /protocols/{id}/analyses → reject (401/403 class).
  • Authenticated analysis without Opentrons-User-Notes when requireReasonForInteraction is true → expect documentation/reason enforcement per CRS settings.
  • Authenticated + reason → analysis created and completes.
  • App/ODD smoke: clone run / confirm RTP values / LPC setup should prompt login+documentation modal before reanalysis.

CRS enabled 2026-08-12. Unauthenticated POST /protocols/{id}/analyses → 401 (pass). Authenticated with/without Opentrons-User-Notes → 500 (AssertionError _coordinator is None). Bug: RQA-5849. App/ODD reanalysis modal not exercised.

F4
Module offsets visible after calibration (#22215 / RQA-5830)
skipped

Process-local LRU cache made GET /modules serve stale empty offsets under Pyro (HW process writes JSON; robot-server read cached miss).

  • Attach temperature or thermocycler module.
  • Calibrate / complete module setup on ODD or App.
  • Without reboot: GET /modules (or App setup UI) must show calibrated offsets, not empty/missing.
  • Optional: calibrate module A, start setup for module B; A must still appear calibrated.

Heater-shaker HSDVT22041138 attached; moduleOffset null; no calibration performed this session.
F5
Audit logs File Manager UI (#22217)
skipped

Rename Compliance Ready files → Audit logs; new columns; priority placement.

  • CRS on. Desktop App → robot settings → File Manager.
  • Expect section titled Audit logs, placed directly under robot storage; columns protocol (or N/A), start, end; no status column.
  • ODD File Manager: Audit Logs is first/preselected tab with the same columns.
  • After a protocol run in an audit period, protocol name should resolve when run record links logPeriodId.

Needs CRS on + App/ODD File Manager.
F6
Audit period id / download smoke (#22226 +
pass

Period id string typing + audit-server disk usage monitoring must not break list/download.

  • uv run flex-test audit list (CRS on).
  • Download a recent period via flex-test audit download <id>; zip opens; actions summarize.
  • No 500 / type-coercion errors around period ids.

CRS on 2026-08-12: audit list 20 periods; download period 20 (12549 bytes) opens; run-log JSON present. Evidence: artifacts/pyro-tests/alpha1-audit-20.zip.

F7
App RTP slideout after reanalyze (#22222 / RQA-5818)
skipped

ChooseRobotToRunProtocolSlideout must show updated RTPs after a new analysis without app restart.

  • Desktop App: open a protocol with runtime parameters; note RTP options.
  • Change protocol RTP defaults / remove an option; reanalyze on the robot or via App.
  • Open Choose robot / run slideout without restarting the App; expect updated RTP set (stale options gone).

App-only RTP slideout; not run this session.
R1
CRS-off api-suite regression
fail

Broad HTTP coverage; previously surfaced RQA-5845 (dup) and RQA-5846.

  • Prefer CRS off for this pass (or document CRS-on skips).
  • ALLOW_MUTATIONS=true uv run flex-test api-suite.
  • Watch Tier C create-run for intermittent ot-protocol resolve 500 (RQA-5846).
  • Missing commandAnnotation id should be 404 not 500 (RQA-5825 / RQA-5845 class).
  • uv run flex-test logs archive; write review.md; file new defects under RQA-5847.

A 52/0, B 34/0, C 6/1. Tier C POST /runs 500 ot-protocol resolve (RQA-5846). Missing annotation historical → 404.
R2
Open alpha.0 pyro bugs still present?
fail

Confirm whether alpha.1 fixed or still carries RQA-5846 / RQA-5835 / uncurrent orphans.

  • Retest intermittent POST /runs proxy resolve (RQA-5846) after create/uncurrent cycles.
  • If device reset / long run path is available: recheck App reachability (RQA-5835).
  • Uncurrent teardown / orphan run_process_entry_point (RQA-5791 class); CRS-on may need signoff.
  • Update tickets; do not close without evidence on alpha.1.

Post-reboot orphan cleanup done. RQA-5846 not retested this pass. CRS-on disruptive path completed (disk gate, audit, F1). RQA-5835 device reset + App reachability still needs ODD device reset + Desktop App. F4 module calibrate and F5/F7 App/ODD UI still manual.