Pyro / protocol-subprocess smoke (Flex internal 4.0.0-alpha.10)

validated · updated 2026-08-03

Pyro / protocol-subprocess smoke (Flex internal 4.0.0-alpha.10)

Validated suite for Pyro5 IPC on KansasFLEX after installing internal ot3@4.0.0-alpha.10 (hardware + protocol subprocess flags default on). Covers systemd/nameserver health, restart recovery failures, HTTP hardware paths, door events, protocol upload/analyze/create-run, process teardown leak, and a live P50 tip pickup/return smoke. Full narrative: docs/pyro-testing.md. Parent bug epic: RQA-5786.

Jira RQA-5786 4.0.0-alpha.10 Bugs Jira RQA-5787 Pyro: robot-server fails Application startup / nginx 502 after update while hardware-api firmware flash is in progress Jira RQA-5788 Pyro: TypeError unhashable type 'dict' when robot-server updates subsystems via OT3API proxy Jira RQA-5789 Pyro: after nameserver restart, OT3API / robot-server-resource never re-register Jira RQA-5790 Pyro: robot-server does not reattach after hardware-api restart (stale OT3API proxy → HTTP 500)Jira RQA-5791 Pyro: uncurrenting a run removes ot-protocol from nameserver but leaves run_process_entry_point processes running Jira EXEC-2187 Protocol Subprocesses (Current)Jira RQA-5796 Pyro: GET /runs/{id}/commands returns empty list for succeeded runs (totalLength=0) on ot3@4.0.0-alpha.10 Jira RQA-5797 Pyro/alpha.10: legacy runs fail StateSummary load (cameraSettings.errorRecoveryCameraEnabled required)PR #22022 feat(api): Update subprocess feature flags to default to On PR #22094 fix(api): Update hardware entrypoint API instance reference and update Pyro interface to support property fset and fdel

Monorepo branch edge

Compared toot3@4.0.0-alpha.9

Robot OS4.0.0-alpha.10

Hardware flex_1channel_50 (right), opentrons_flex_96_filtertiprack_50ul in A1 (for C3-live only)

Harness commands

uv run flex-test inspect
uv run flex-test probe --no-picture
ALLOW_MUTATIONS=true uv run flex-test put 4.0.0-alpha.10 --channel internal
ssh -i ~/.ssh/robot_key -o IdentitiesOnly=yes root@$ROBOT_HOST

Tests

ID	Plan
A1 Systemd pyro stack active pass	Nameserver must precede hardware-api and robot-server for IPC. SSH to KansasFLEX. Run: systemctl is-active opentrons-pyro-nameserver opentrons-hardware-api opentrons-robot-server Expect all active and enabled. Validated 2026-07-31 on ot3@4.0.0-alpha.10 after install + recovery
A2 OT3API registered on nameserver pass	Hardware entrypoint notifies ready only after OT3API is listed. With Pyro5, locate_ns() and confirm OT3API in ns.list() within 60s. Optional: Proxy(resolve('PYRONAME:OT3API')).get_fw_version(). Present in ~0.05s when healthy; fw version 72
A3 Health OK with HW proxy path pass	robot-server identify_hardware_process depends on NS + OT3API. Run uv run flex-test inspect and/or GET /health; expect 200. GET /instruments and /subsystems/status populate.
A4 Nameserver restart recovery fail	Producers should re-register or dependents should fail clearly. Baseline NS includes OT3API and robot-server-resource. Run: systemctl restart opentrons-pyro-nameserver Poll NS 90s; note /health and unit active states. RQA-5789: app names never re-register; /health can stay 200; services stay active
A5 Hardware-api restart reattach fail	robot-server must refresh OT3API proxy after HW daemon URI changes. Note OT3API URI; run: systemctl restart opentrons-hardware-api Confirm new OT3API URI; poll /health and /instruments for 2 minutes without restarting robot-server. RQA-5790: HTTP 500 CommunicationError to stale port until robot-server restart
B1 Instruments and subsystems via HTTP pass	Lower-line IPC must serve robot-server instrument inventory. GET /instruments and /subsystems/status. Expect pipettes ok and subsystem fw versions present. left p50_multi + right p50_single; subsystems fw 72
B2 Module nested proxies skipped	pyro_behavior attached_modules should expose module proxies. Attach a module; GET /modules; optional on-robot Proxy list. No modules attached on KansasFLEX during validation
B3 Door open/close via HTTP pass	Door watcher / HW door events must surface through robot-server. Poll GET /robot/door/status while opening and closing the door. Confirm closed to open to closed; optionally match hardware-api DoorSwitchStateInfo logs. Two full cycles 2026-07-31; currentState does not include door (expected)
C1 Create run registers ot-protocol pass	Run lifetime should own a protocol subprocess + NS name. Upload docs/test-suggestions/protocols/pyro_smoke_no_motion.py (or equivalent). Wait analysis completed; POST /runs with protocolId. Confirm NS has ot-protocol and ps shows run_process_entry_point.
C2 Protocol analysis completes pass	Analysis path uses simulating / subprocess machinery. After upload, poll protocol analysisSummaries until completed. GET analysis document; expect result ok (or record errors).
C3a Idle run actions contract pass	Actions must cross robot-server to ot-protocol Pyro boundary. On idle run: pause expects 409; stop accepted on this build; cancel is not a valid actionType.
C3b Live tip smoke play pass	Play must execute motion commands across the protocol subprocess boundary. Deck tiprack A1; use protocols/pyro_live_p50_tip_smoke.py; door closed; estop disengaged; play; wait succeeded. Store bare UUIDs only in helper files (not PROTO_ID= prefix). Live tip smoke run 1c4e5cce-… succeeded (summary pipette+labware). Post-hoc GET /commands empty: RQA-5796.
C6 Uncurrent tears down process fail	Spec: protocol process lifetime matches run record create to uncurrent. With current run and ot-protocol in NS, PATCH run current=false. Confirm NS drops ot-protocol and no orphan run_process_entry_point remains. RQA-5791: NS cleared but ot-protocol and ot-simulating-protocol processes remained under robot-server
D1 Door enum via HTTP pass	DoorState must serialize through robot-server product path. GET /robot/door/status; expect open or closed. closed
D2 DoorState via Proxy with registry pass	Specialty enums require Opentrons Serpent registry on the client. SSH with HOME writable (e.g. /tmp/ot-home); PYTHONPATH=/opt/opentrons-robot-server. register_hardware_types(); Proxy(OT3API).door_state is DoorState. DoorState.CLOSED
D3 attached_instruments with registry pass	NonBuiltinKeyDictWrapper / Mount keys must deserialize with registry. With register_hardware_types, read attached_instruments; expect Mount keys and pipette name dicts. LEFT p50_multi_flex, RIGHT p50_single_flex
D4 Run currentState payload pass	StateSummary-shaped HTTP payload must load for a current run. Create current run; GET /runs/{id}/currentState; expect tipStates/estopEngaged (or similar).
D4b Legacy StateSummary persistence fail	Older stored StateSummary JSON must still validate or migrate. GET /runs; note any ok=false RunDataError InvalidStoredData on StateSummary. RQA-5797: cameraSettings.errorRecoveryCameraEnabled required
D5a Analysis command slices pass	Pydantic command models must round-trip on analysis path. GET protocol analysis document; confirm commands list (e.g. home/comment).
D5b Succeeded run command history fail	Executed run commands must persist and list via HTTP. GET /runs/{succeededTipSmokeId}/commands?pageLength=100; expect totalLength > 0. RQA-5796: tip smoke 1c4e5cce-… and other same-day runs return totalLength=0
D6 Invalid request error surface pass	Error responses must serialize cleanly over HTTP. POST loadPipette with mount=not_a_mount; expect InvalidRequest / errorCode 4000.
D7 Subsystems status healthy pass	Regression check for RQA-5788 unhashable dict on subsystem path. GET /subsystems/status; expect all subsystems ok. 6 subsystems ok; fw 72
D8 Raw Proxy without registry (negative) pass	Specialty types must fail closed without registry (not silent wrong values). Proxy(OT3API).door_state and attached_instruments without register_hardware_types. Expect SerializeError (or similar) rather than wrong primitive values. SerializeError DoorState and NonBuiltinKeyDictWrapper