A short instruction such as “remove the left battery module” hides several decisions: which module is meant, which fasteners block it, what must happen first, and which robot can perform each action. This paper tackles the intent-to-plan part of that problem, turning a perceived scene and a human request into an inspectable removal sequence.
Why a List of Objects Is Not Yet a Plan
A detector can identify a screw and a module without knowing whether that screw holds the module in place. Likewise, a language model can write a plausible disassembly procedure while mentioning a part that is absent from the workbench. The challenge is to connect semantic instructions with the particular objects actually observed, then respect their dependencies.
The study deliberately separates three layers. Perception produces an object-level description; language-based planning selects and orders objects; MoveIt handles lower-level motion and inter-robot collision checking. An ordered list is therefore an intermediate representation, not a trajectory or a safety certificate. This separation also makes errors easier to locate: a missing detection, an incorrect prerequisite and an infeasible grasp are different failures.
How the Ensemble Works
YOLOv8 detections from RGB-D cameras become structured text describing component labels, positions and group relationships. Coordinates are rewritten into the operator’s frame so phrases such as “left” refer to a consistent viewpoint. The generator receives that scene together with the request and an explicit output contract: resolve the target, include necessary blockers, avoid duplicates and return an ordered object list.
“Six LLMs” here means six stochastic samples from one Qwen3-32B checkpoint, not six independently trained model families. Different seeds expose alternative candidate sequences without additional fine-tuning. A verifier invocation receives the original context and candidate lists, checks their formatting and ordering, and selects a candidate satisfying its instructions. Because it uses the same checkpoint, it remains capable of sharing the generator’s mistakes.
The final deterministic filter performs a narrower but dependable check: proposed object labels and coordinates must match the detected scene. This removes ungrounded entries, but its guarantee is relative to perception; it cannot recover an undetected screw or establish physical graspability. The accepted object list is then expanded into approach, grasp, retreat and placement actions for two UR10e arms.
What the Results Actually Show
The benchmark contains 200 real scenes and 600 operator prompts across five component classes. Full-sequence correctness asks whether the entire requested removal order is correct; next-object correctness asks only whether the immediate next selection is correct. Those are different levels of success.
| Measure | Single sample | Six samples, verifier and filter |
|---|---|---|
| Full-sequence correctness | 0.761 | 0.824 |
| Next-object correctness | 0.866 | 0.894 |
| Mean end-to-end planning time | 5.57 s | 32.74 s |
The full-list gain is 6.3 percentage points, accompanied by substantially more computation. This is a reliability–latency trade-off, not a speed improvement. The deterministic filter adds a relatively small final increment in correctness and roughly 0.2 seconds of overhead compared with the much larger sampling and verification cost.
A separate pilot involved seven volunteers issuing two intents each. All 14 actions were executed as intended after participant review and approval; mean time to action was 197.86 seconds and mean NASA-TLX workload was 14.95 out of 100. The approximately 300-second manual reference came from an expert operator, rather than a matched interface-control condition, so it provides context rather than a general causal estimate of time saved.
Interpretation and Limits
The strongest contribution is the explicit bridge between a worker’s request and a grounded, reviewable procedure. The operator can supply a complete order, name a partial goal or request system-generated ordering, while retaining approval and override. This is useful flexibility when product variants make one fixed disassembly script inconvenient.
The tested domain still has five component classes, one backbone checkpoint and a small human pilot. Repeated samples can agree on the same wrong assumption, and text-only verification cannot establish reachability, contact safety or collision freedom. Reproducing the numerical benchmark requires the original scene descriptions, prompts, model configuration and scoring rules; transferring the system requires new perception and motion validation.
Testing and Real-World Use
The approach could be tested on held-out component scenes by scoring complete removal orders, unknown-object errors and planning latency under identical prompts for single-sample and ensemble configurations. It could help technicians plan variable maintenance or disassembly jobs while keeping the proposed sequence visible for approval before robot motion.
Paper and Demonstration
- Journal paper, methods, figures and results.
- Supplementary Video 1: the paper’s robot demonstration, linked by the journal; opens on demand rather than loading a player with the article.
- Earlier preprint.
- Follow-up: interactive and inspectable planning.