MIT 6.4210 — Lecture 2

§ Has AI Solved Robotics?

As spoken about on X, GPT-6 Astra scored 95% on a robot control task in which robots are told, e.g., “Pick up the red block from the table and place it inside this bowl.”

In essence, we have these Omni models that are currently capable of taking audio, text, image, and video inputs and producing audio, text, image, and video outputs. The current trend is to add robotics to these lists.

Question: If AI is solving robotics, then why get into robotics at all?

• There is limitless potential for impact, and the state of the art is moving fast in every field.

• Hardware can't scale (quite) as fast as software.

  • (But exponentials can still happen… consider robots building robots, for example.)

• In general, the gap between “vibe coder” and “expert” is still nontrivially large.

§ Reinforcement Learning

One way to train robots is reinforcement learning (RL), which consists of the following steps.

1. Make a simulator.

2. Define and write a reward function.

3. Take gradients for optimization.

4. Deploy in the real world.

One downside of RL is that it requires a simulator—which it needs because:

• It's much safer and more convenient to implement.

• It's much easier to define and compute a reward function (because the full state is perfectly known).

Another downside of RL is that it requires fine-tuning the reward function, because “misalignment”.

§ Behavior Cloning

Another way to train robots is behavior cloning—the point being that humans do things pretty well, so we should just demonstrate to robots what humans would do.

This can be done via teleoperation. Training becomes a supervised-learning objective; no simulation or reward function required.

Naturally, one might also consider imitation-guided RL, which may possibly yield “superhuman” performance.

§ Training in Practice

For single-task learning, one usually uses roughly 200 human demonstrations to produce a map from image readings into actions (visuomotor policy). For multitask behavior cloning, one usually uses a lot of robot and internet data to produce a language-conditioned visuomotor policy.