Natural Language Planning for Four-Arm Robots: Our Research Collaboration with NUS

What if you could walk up to a workcell of four robot arms and simply tell them, in plain English, what you want done — and they would work out among themselves which arm picks, which arm receives, and where the handoffs happen? That is the question behind a new research paper from our collaboration with the National University of Singapore: Natural Language Planning for Collaborative Four-Arm Manipulation Using Large Language Models, by Xiaohan Mao, Shrabasti Chakraborty, Chew Hoe Alfred Ang and Li Hong Idris Lim, submitted to RAAI 2026, the 6th International Conference on Robotics, Automation, and Artificial Intelligence, held here in Singapore.

The project pairs NUS College of Design and Engineering researchers with Tertiary Infotech Academy, and it sits squarely in the fastest-moving corner of robotics today: connecting large language models and vision-language-action (VLA) systems to real hardware. Here is what the work is about, why it matters, and how you can build the same skill set hands-on in a single day.

The problem: one arm can't reach everything

Industrial robot arms are precise but parochial. Each arm has a fixed workspace — a reachable envelope determined by its mounting point and joint limits. The moment a task spans a larger area than one envelope, you need multiple arms, and that is where the engineering cost explodes:

• Reachability gaps. An object placed outside arm A's envelope is simply unreachable for it, even if the task "move the object to the far tray" is trivial to state.
• Hard-coded choreography. The classical fix is to program each multi-arm sequence explicitly — pick points, transfer poses, synchronisation signals — per task. Change the task, and an engineer re-programs the cell.
• Coordination risk. Four arms sharing a workspace must not collide, and a mid-air handoff between two arms is one of the harder coordination problems in manipulation.

In short: multi-arm systems multiply capability, but under traditional programming they also multiply the engineering effort per task. That trade-off is exactly what the paper attacks.

The idea: let a language model do the planning

The team's approach uses a large language model as the task planner for a collaborative four-arm workcell. Instead of scripting each sequence, the operator issues a natural language command. The LLM decomposes it into a plan across the four arms — deciding which arm should act, in what order, and critically, when an object must be handed off from one arm to another because the goal position lies outside the first arm's reach.

Three properties make this design significant:

1. Intelligent handoffs solve reachability. When a target lies beyond the acting arm's envelope, the planner routes the object through an intermediate handoff to an arm that can reach the goal. Coverage becomes a property of the team, not of any single arm.
2. No task-specific programming. The arms are not programmed for particular tasks. The same workcell adapts to a generic range of manipulation tasks purely through what the operator asks for — the plan is generated, not authored.
3. Prompting becomes the operator skill. Instead of teach pendants and robot-vendor scripting languages, operators learn a prompt structure for commanding the arms. The interface to an advanced robotic cell becomes structured natural language — a skill transferable across hardware.

This reframes what "robot programming" means. The expensive artefact is no longer a task program; it is the planning system and the prompt conventions around it. And it points at the direction the whole field is moving: world-model and vision-language-action approaches where perception, language and control share one learned representation.

Where VLA models fit in

Language-driven planning is one half of the modern robotics stack. The other half is the action model — the component that turns "grasp the red block" plus a camera image into actual joint motions. That is the job of vision-language-action (VLA) models, and the best-known open example is OpenVLA, a 7-billion-parameter open-source model trained on 970,000 real robot demonstrations from the Open X-Embodiment dataset (Kim et al., 2024).

A VLA model maps pixels and language directly to robot actions. What makes OpenVLA practical rather than a lab curiosity is fine-tuning: with LoRA (low-rank adaptation), the model adapts to a new robot arm, a new camera setup and a new workspace on a single consumer GPU — no data-centre training run required. A planner like the one in the RAAI 2026 paper decides what each arm should do; a fine-tuned VLA policy is one natural candidate for how each arm does it.

Making it run in the real world

Research demos run on lab GPUs; deployed robot cells have latency budgets. A 7B-parameter model queried at control frequency is a real engineering constraint, which is why inference optimization is part of the practical VLA skill set. The standard levers apply:

• Quantization — running weights at lower precision (8-bit or 4-bit) to cut memory and latency, usually with minimal loss in task success rate.
• Caching — reusing intermediate computations across control steps instead of recomputing the full forward pass.
• Early exit and pruning — letting the model produce actions before processing all layers, and removing weights that contribute little.
• Deployment strategy — deciding what runs on the robot, what runs on a workstation beside it, and what can tolerate a round-trip to a server.

These are the same techniques used to serve chat models at scale, applied to closed-loop control — and they are the difference between a paper result and a workcell that meets cycle time.

Learn to fine-tune a VLA model yourself — in one day

The most striking thing about this research area is how accessible the tooling has become. The skills used in this line of work — collecting demonstrations, formatting episodes, LoRA fine-tuning, evaluating success rates and deploying safely — are exactly what we teach in our one-day Fine Tuning OpenVLA Model course (C1074) at Tertiary Infotech Academy.

In 7.5 hours you will work through the full loop:

1. Vision-Language-Action foundations — how OpenVLA maps camera pixels and language instructions to robot actions.
2. Data collection and preparation — teleoperation, episode formatting, dataset cleaning and augmentation.
3. LoRA fine-tuning on a single GPU — hyperparameters, workspace adaptation, and when fine-tuning beats prompting.
4. Evaluation and deployment — measuring success rates, analysing failures, the sim-to-real gap, and safety practices.

The course is beginner-level and runs in physical classroom, synchronised Zoom and corporate on-site formats. If you are aiming for funded upskilling instead, the closely related WSQ — Generative AI Model Development and Fine Tuning course covers model fine-tuning under SkillsFuture funding support.

Robotics is converging with generative AI faster than most engineering curricula can follow. Whether you are a robotics engineer who wants to add language models to your stack, or an AI practitioner curious about embodiment, register for the Fine Tuning OpenVLA Model course and build a working fine-tuned VLA pipeline before the field moves again.

Frequently asked questions