A1XY Teleoperation

One or two Galaxea A1X/A1Y arms controlled by matching hands through a phone or desktop camera. The application includes MuJoCo simulation, a browser panel, arm selection and a 10–100% speed control. Native CAN FD on macOS uses external XCAN USB FD adapters without ROS, Docker or a system CAN driver.

The two physical arms completed supported parking of all six axes to native zero and a coordinated base cycle ±30°. Both were later confirmed A1X; their gripper endpoints were measured separately, and physical right-arm Mac webcam control ran. See measured results and limits. The tracked configuration stays uncommissioned (hardware_confirmed: false); this installation uses an ignored measured profile. Full workspace collision clearance and grip force remain unverified. The current home setup is one A1X and a Star Arm 102-LD leader. The earlier two-arm measurements describe a different bench; they are not calibration for the home arm. Leader integration is planned in lerobot-a1xy, not implemented by the camera controls here.

Setup and simulation

Clone the SDK and application as siblings before installing. The SDK is not published to PyPI; uv resolves the development dependency from ../a1xy-sdk.

mkdir -p ~/projects/a1xy
cd ~/projects/a1xy
git clone git@github.com:dtecx/a1xy-sdk.git
git clone git@github.com:dtecx/a1xy-teleop.git
cd a1xy-teleop

Requirements: Python 3.11 through uv, Git, Node.js 20+. Setup fetches pinned official A1X/A1Y models, the MediaPipe hand model, Python dependencies and the web client. It can be rerun.

uv run python scripts/setup.py
uv run pytest -q
npm --prefix web test
npm --prefix web run build

export SANDBOX_ADMIN_TOKEN=local-a1xy-test
uv run sandbox --backend sim --arm galaxea_right --arm galaxea_left --no-render --no-cameras

Open http://localhost:8000/#admin=local-a1xy-test. Select Right only, Left only or Both arms, then pinch and move the matching human hand. Switching selection stops pending motion and requires a fresh gesture. The excluded arm holds its pose and gripper command. The speed slider scales the configured teleoperation and Home joint speeds; it is not a verified percentage of a motor's physical maximum. E-STOP applies to both arms and requests holding. Physical Park closes to contact and keeps holding at rest; Unpark avoids a motor-mode transition. Thumb + pinky held for 0.6 s locks only the matching arm and its gripper. Release and repeat, or use that arm's Unlock button, to resume.

Single-model simulation: uv run sandbox --backend sim --variant a1x (or a1y). The default two-arm configuration uses A1X as an explicit placeholder until the physical model identities are established.

Complete phone test commands cover Android USB/ADB, camera permissions, selection, speed, API checks and dry-run. Phone camera access uses localhost through adb reverse. Handedness and camera orientation must be verified in simulation before physical teleoperation.

Native SDK dependency

../a1xy-sdk/src/a1xy_sdk/    command types, wire codec, native USB transport
src/a1xy_teleop/            configuration, gripper policy, IK, simulation, cameras, server

The independent SDK imports no application configuration, MuJoCo, FastAPI or camera code. USB dependencies load only on enumeration/open. This application depends on a1xy-sdk==0.2.0; its wheel contains only a1xy_teleop. Installing with pip requires installing the sibling SDK first. Native transport has macOS bench evidence; cross-platform USB hardware validation remains pending. The SDK owns measured state, freshness, status, limit validation and hold/release checks. Teleop retains application configuration, gripper guards and motion supervision.

uv sync --all-extras
uv run python ../a1xy-sdk/examples/read_state.py --list
uv run python ../a1xy-sdk/examples/read_state.py --channel usb:1.2 --seconds 5
uv run a1xy-arms list
uv run a1xy-arms probe --json

Stop any live sandbox before probing. Each adapter has one owner. These receive-only checks acknowledge CAN traffic but send no motor commands. The historical two-arm mapping was right usb:1.2, left usb:1.4; enumerate and commission the home adapter independently. Classic CAN-only adapters cannot carry the 64-byte FD command. A direct USB control protocol for the ports on the arms has not been established.

from a1xy_sdk import Command
from a1xy_sdk.protocol import decode_feedback, encode_command
from a1xy_sdk.transport import XcanTransport, list_adapters

Command contains seven positions, velocities, gains and effort fields. Codec bounds describe the wire representation, not safe robot operating limits. See protocol and lifecycle. Do not send arbitrary encoded targets to physical arms; the app adds commissioning and motion checks.

See repository ownership and migration for the four-project layout.

Hardware application checks

uv run sandbox --backend real --dry-run --no-cameras
# Alternative runs, one at a time:
uv run sandbox --backend real --dry-run --arm galaxea_right --no-cameras
uv run sandbox --backend real --dry-run --arm galaxea_left --no-cameras

Dry-run opens no USB device. Arms start HOLD at 10%, with Home required before camera motion. Live startup requires verified model/coordinates, supported folded park pose, measured gripper endpoints and fresh healthy feedback. It holds the measured pose first and never switches motor mode or zeros encoders automatically. Released/frozen or faulted arms require operator inspection; Reset does not clear hardware faults with an unverified function command.

The arms are brakeless in this setup. The software E-STOP cannot guarantee holding after communication or power loss. Normal live exit requires explicit Park first; uncertain pose/velocity refuses release. No physical emergency-stop arrangement has been verified for this installation.

Other application modules

• control: joint/workspace/speed checks, Home/Park, tracking and communication watchdogs.
• teleop: validated hand features, gesture state, filters and camera-to-arm mapping.
• camera: USB/network capture, reconnect and MJPEG. Use a1xy-cameras scan / probe.
• sandbox: authenticated admin controls, WebSocket hand/state traffic and session recording in logs/.
• chess: rules, board calibration, vision and waypoint planning. Board placement must fit the A1XY workspace; the existing 25 mm demo board is refused where it extends outside that workspace.
• control.trick: generic scripted trajectories. No physical A1XY trick presets are supplied; tricks.allow_real is false. The simulation lab requires an explicitly configured preset.

Camera commands:

uv run a1xy-cameras scan
uv run a1xy-cameras probe
uv run a1xy-cameras android --open

The default HTTP binding is loopback. Keep admin tokens private. Read-only phone URLs omit the admin fragment; use the Mac panel for selection, speed and emergency controls. Assets and model sources are recorded in third_party/README.md.