Fortunately, we know a lot about what deep neural networks represent. As a starting point, the universal approximation theorem (UAT) says that a feed-forward neural network with at least one hidden layer can approximate any continuous function over a compact domain to any desired degree of accuracy, provided it has enough neurons and a non-linear activation function.

This begs a number of follow-up questions:

• What kinds of tasks are (not) solved by approximating a continuous function?

• For this purpose, are transformers equivalent to feedforward neural networks, or do they do something different?

• How do these map to computational hardware, like CPUs, GPUs, or NPUs?

Answering these questions requires a review of what “computation” means, looking all the way back to the writings of Turing, Minsky, and Chomsky. In exchange we get some insights into the versatility as well as the energetic cost of current AI.

I’ll provide some answers to the first two questions in this post, and a detailed look at the last one in a follow-up.

Subscribe now

Universal Approximation

The prototypical “feedforward neural network” from the UAT is a multi-layer perceptron (MLP). This is typically composed of linear layers (which multiply its inputs by a weighting matrix) and a nonlinear activation function.

In the plots below1, we’re approximating a quasi-sinusoidal curve on the left and a square wave on the right using an MLP.

With larger model width and depth:

You’ll notice that the square wave is much more difficult to approximate than the sinusoidal one. Why is that? If you recall from above, the UAT promised that the MLP would be good at approximating continuous functions, and the square wave has periodic discontinuities.

Before you think that this is some pedantic example that would never occur in practice, let me offer two more practical ones that are equivalent.

Suppose you have a drone flying through a forest of tall trees:

The task is obstacle avoidance: the input is the front camera view, and the output we’d like is a path that won’t collide with a tree. In such a view, if the view changes continuously in such a way that a path becomes too narrow to pass through, the safe path must jump to a different one discontinuously.

An MLP would need to sample the input space very densely to quickly interpolate between the left path and the right one (as in the square wave example above). This has a high model size penalty, and additionally needs to interpolate through an unsafe part of the output space.

Another practical example is related to the title of the article. Assuming an LLM’s output is a token view of an internal reasoning state, “changing its mind” on a yes / no question requires a similar jump in its state. However, the internal computing machinery of a modern LLM, the transformer, is more complex than an MLP. We’ll look into the two categories separately below.

Subscribe now

Lookup Tables to Turing Machines

The computation power guaranteed by the UAT is equivalent to a lookup table. A lookup table effectively pairs inputs and outputs so that it can “look up” the appropriate output when queried with an input. In continuous spaces, this can include some interpolation or extrapolation. The curve approximation figure above is a good visualization of this: the table would contain {x, y} entries. The compact domain condition of the UAT effectively ensures that the number of entries in the lookup table is finite.

On the other end of complexity, we have a Turing machine: an automaton that has access to unbounded memory, and is able to make discrete decisions based on what is in its memory. While this may sound foreign, it is actually a very familiar concept. A CPU paired with almost any programming language is a Turing machine (putting aside the implementation detail of potentially running out of memory). You can control a program’s flow using if, while, etc. and call subroutines, and with these building blocks, you can build any software that has ever been written.

It should be clear that a Turing machine can do fundamentally more than a lookup table:

1. It can process an input that is arbitrarily large, which a lookup table cannot do. For example, you can very easily write a CPU program that factorizes an integer, but we could never fit such an algorithm on a lookup table, since you could always input a larger integer. A more current example is a pre-transformer language model, which could not handle sequences of arbitrary length, and thus could not exhibit the level of capability we got with a GPT.

2. It can exhibit irregular flow control, like branching and jumping. In the “flying through forest” example above, it can do something like

if left_path_too_narrow:
    take_right_path()
else:
    take_left_path()

While this looks benign, it is deeply connected to the continuity clause of the UAT. An MLP cannot represent an algorithm that needs this kind of branching to have a discontinuous or symbolic jump.

In the example above, a square wave was still able to be approximated by an MLP, but at the expense of a large number of parameters. As a contrast, here’s an almost trivial program that could accomplish the requisite classification with very few parameters:

if x mod 2 < 1: # if the remainder of x/2 is < 1
    return 1
else:
    return -1

This shows the expressive power of a Turing machine compared to a lookup table. Adding a little structural or organizational complexity drastically reduced the number of required parameters.

Subscribe now

The Transformer Attention Mechanism

We discussed earlier how the UAT only addresses a finite set of inputs. This is true in practice for MLPs as well: it will typically be used to process a fixed image size, or in an transformer feedforward network, a fixed layer width.2

The attention mechanism of transformers is different. In an LLM, when a sequence of tokens is fed in, each token can attend to each other token, enabling a computation paradigm that can handle sequences of arbitrary length. This makes it different from a lookup table, because the input dimension itself is unbounded. You don’t need to retrain for longer sequences since the attention mechanism adapts the algorithm.

In practical terms, a transformer’s sequence length has to be limited to a maximum context length to manage the mapping to computational hardware. By the same token, CPUs also needed unbounded memory to be true Turing machines.

So, are implementable transformers, like general purpose CPU programs, Turing machines in all but the most pedantic terms?

Not quite — there’s still a fundamental gap that cannot be closed. Transformers are still continuous function approximators and cannot efficiently exhibit irregular flow control. A 2026 paper from Oracle AI looks at discrete reasoning with transformers, and I’ll let it speak for itself:

Through this synthesis, we provide readers with a cohesive understanding of why transformers succeed in interpolation tasks (e.g. summarization) but fall short in reliably executing symbolic algorithms.

Symbolic algorithms are characterized by discontinuous outputs that present a challenge to transformers. Like in the square wave example above, you can try to circumvent the issue by increasing model width or dataset size, but this comes at the cost of greatly increased model size and inefficiency. Moreover, as the paper points out, as you compose symbolic tasks (task A → task B → …) the number of switching boundaries grows combinatorially.

For an LLM to change its mind on a yes / no answer, architecturally it needs to continuously interpolate through reasoning trajectories, traversed by generating (lots of) reasoning tokens.

Closing Thoughts

Deep neural networks can solve a huge variety of problems, founded on their universal function approximation ability. Transformers’ ability to process arbitrary sequences advances them into a new computational category beyond lookup tables.

However, they are still not well suited to problems with symbolic or discontinuous outputs. This is common in problems to do with safety or symbolic reasoning. In current successes of deep learning, solutions to these kinds of problems are attained in a similar fashion as the square wave approximation above — it works, but is extremely inefficient.

These problems could potentially be solved with much smaller models if they had Turing machine-style universal computation capabilities. ’s article linked below advocates for the same thing, approaching it from the computational hardware perspective for some classes of problems. In a follow up post, I’ll tie together the first-principles analysis in this post to current computational hardware, to discuss how different algorithm classes effectively map.

Thanks for reading!

If you enjoyed this post, please like (❤️) and restack — it helps others find my writing. Subscribe to receive new posts. All of this is greatly appreciated.

References and Further Reading

Are Transformers Turing-complete? — Hessam Akhlaghpour (2024)

Barriers to Discrete Reasoning with Transformers — Oracle AI (2026)

Artificial Intelligence Made Simple

The Great Compute Re-Architecture: Why Branching & Sparsity Will Define the Next Decade of Silicon [Breakdowns]

It takes time to create work that’s clear, independent, and genuinely useful. If you’ve found value in this newsletter, consider becoming a paid subscriber. It helps me dive deeper into research, reach more people, stay free from ads/hidden agendas, and supports my crippling chocolate milk addiction…

Read more

a year ago · 57 likes · 17 comments · Devansh

For robotics, we may be in the midst of the first time this has ever happened.1 The start of our research careers resembled the “wild west” of emerging techniques and technologies, but ideas have converged more now. On one hand, robotic hardware has gotten good enough to see thousands of robots of getting shipped and used, by consumers and researchers alike. On the algorithm side, the bitter lesson and its corollary — hypothesized “scaling laws” — have provided a scaffolding around which progress can be evaluated. End-to-end behavior cloning policies seem like they can generalize to all sorts of tasks, and performance predictably improves with more data. We’ll refer to these two trends as commoditization and architectural convergence, and discuss how they shape the current paradigm below.

The establishment of this current paradigm has also had side-effects on the nature of research that may in themselves be setting us up for paradigm shifts. While it is a bit of an overreach to use the term “revolution” for robotics (as Kuhn did for science), such a shift would be pivotal for researchers and is worth understanding.

This article is co-written by and , both robotics researchers with experience in academia as well as industry. Chris writes about AI and robotics, and Avik writes about robotics, computing, and AI.

min{power}

Explorations in computing and robotics focused on power-efficiency and safety -- personal posts by Avik De, robotics Ph.D. and founder

By Avik De

It Can Think!

Robotics and AI; the future we're building and how we'll get there

By Chris Paxton

Trends in Robotics and AI

1) Commoditization

Going back to 2013, Avik’s Ph.D. research included the development of an internal research robot, Minitaur:

It took a lot of (Ph.D. student) effort to build the infrastructure, but resulted in a unique development platform that was easy to program (as an Arduino), lightweight and relatively safe (5 kg), and capable of producing very agile and exciting-looking behaviors. There was nothing like it that you could buy. All in all, this endeavor to develop a new robot led to papers, cool movies to show in talks, and even a startup company.

In the decade after, four-legged robots started to get out of the research lab and into public consciousness. The show Silicon Valley had a Boston Dynamics Spot cameo in 2016, and robot videos designed to appeal to a broad audience, like dancing, started to appear. Four-legged robots were officially out of the lab and in the wild, and this led to increased expectations for what they should do. Stably walking around used to be cutting edge, but became table stakes. Expectations for specs such as reliability, battery life, compute capability, ruggedness drove designs to be more complex. It became much more difficult for a couple of researchers with minimal engineering experience to put together a new robot. Moreover, after Chinese company Unitree entered the market and dropped the asking price by almost 30x in 2021, it became not worth the time and dollars to even try.

The pre-paradigm period of lab-developed robotic hardware is being replaced by algorithm development for commoditized hardware.

We have seen this play out in several robotics research labs. DJI commoditized consumer drones aggressively from 2013 onward, making it hard to justify custom builds even for capability reasons. By the mid-2010s, labs doing serious flight research (e.g., Davide Scaramuzza’s group at University of Zurich) were exclusively using commercial platforms. ETH Zurich’s Robotic Systems Lab (which built ANYmal originally, and also STarLETH) now deploys their locomotion research on the ANYmal platform rather than building new hardware. Boston Dynamics has an article that talks about how commercial platforms let researchers hit the ground running.

Post-commoditization, researchers who want to demonstrate algorithms working on robots can reap the benefits. Humanoid research circa 2015 meant figuring out “how do we actually build these things and make them not fall over,” whereas post-commoditization, time can be spent on higher-level algorithms and methods — we refer to this phenomenon as “moving up the stack.”

A secondary effect of commoditization is that parts are now easier to get, and researchers can put together novel modular combinations of more mature components. The WidowX 250 Dynamixel-based arm from Trossen Robotics has become the default low-cost manipulation platform because it is cheap (~$3k) and can be used to create “leader-follower” setups for data collection. The ALOHA paper notes that the whole system with two arms costs ~$20k off-the-shelf. More recently, we have seen robots like the YOR assembled from off-the-shelf parts for research purposes. This effect enables new types and form-factors of robots to be built — we will return to this in the next section.

The same trend applies to non-hardware AI research. Frontier language models cannot really be trained by academic research labs any more — research in these areas moves to fine-tuning commercial models instead. The following plots were generated from arXiv data and confirm these trends toward pretrained model usage in research compared to building them from scratch.

The Qwen series of models by Alibaba have nearly taken over the research world by facilitating fine-tuning. In 2026, no academics would think of training their own language models or even vision-language models from scratch — why would you, when Qwen 3.5 can already beat anything that’s within reach of an ordinary academic lab?

Just like for robotics hardware, the pre-paradigm period of lab-developed models is being replaced by fine-tuning commercial models.

Here as well, there are research ideas which can be pursued by moving up the stack: agentic reasoning, reinforcement learning, world representations, novel model architectures, etc. Robotics models are not like language models; there are fewer real world benchmarks and it seems that even within the domain of end-to-end deep learning there are plenty of ideas left unexplored.

2) Architectural Convergence

Labs used to have a narrower focus where they could carve their niche, e.g. computer vision, legged locomotion, etc. However, for a robot to demonstrate complex sensorimotor tasks, you need all of the Sense-Plan-Act functions implemented in some way. If you subscribe to the bitter lesson, even the best computer vision algorithm, when connected using hand-crafted interfaces to a planner and other downstream systems, cannot compete with end-to-end systems. General-purpose manipulation / locomotion research is converging on behavior cloning and VLAs since it works well enough across many tasks, and performance improves with larger models and more data.

This trend has pushed many previously-diverse labs toward developing end-to-end models, which is a significant reduction in the diversity and richness in the research ecosystem. For better or worse, we appear to solidly be in a paradigm of behavior cloning with end-to-end models.

This has several benefits for researchers: they can build on existing work easily without re-inventing the wheel, and it creates a scaffolding for new contributions. However, it also has the side-effect of suppressing other schools of thought. In Kuhn’s somewhat ominous words,

But there are always some men who cling to one or another of the older views, and they are simply read out of the profession, which thereafter ignores their work. The new paradigm implies a new and more rigid definition of the field. Those unwilling or unable to accommodate their work to it must proceed in isolation or attach themselves to some other group.

How do research labs and out-of-paradigm ideas stand out in the face of homogenization and consolidation in this paradigm? We discuss what we can learn from computer engineering in the next section.

Subscribe now

What we can learn from Computer Engineering

By necessity, computer engineering has always been a bit ahead of the same technology curve as robotics. After all, we needed the chips to facilitate computations needed for robots to work.

We saw there a similar commoditization trend, with hardware complexity outgrowing what a research lab could build. The initial university fab era was anchored by DARPA’s VLSI Project, which produced BSD Unix, the RISC concept, and MOSIS (a shared fab for academia). Once that era ended, academic research pivoted to what could be done without a fab.

As a response, computer engineering therefore shows a good set of examples of moving up the stack (transistors → meta-design tools and ISAs). Circa 2010, rather than building chips, Krste Asanović’s group at Berkeley designed the open RISC-V ISA explicitly motivated by the problem of proprietary architectures impeding academic research. With Chisel (Berkeley), academics built better tools for designing chips, by expressing hardware designs in a high-level language, and it became the foundation for most RISC-V implementations.

In addition, CPU architectures converged to x86 for desktop and ARM for mobile because they worked well enough for most workloads, and design costs could be amortized across different applications — a general-purpose computing paradigm.

Hennessy and Patterson’s 2017 Turing Award lecture argued that the post-Dennard-scaling era opens up a new window for research in domain-specific accelerators, where the design space is exploratory again. Coincident with the success of deep neural networks, the last few years have seen a Cambrian explosion in AI accelerators, ushering in much more innovation in computer architecture and silicon than was possible in CPUs.

In other words, computer engineering’s paradigm shift resulted in domain-specific diversification.

How do these apply to robotics and AI?

Just as chip fabrication leaving academia didn’t end computer architecture research, robotics research will find a home in core algorithms, training methodologies, and novel architectures. While papers can continue to be written on new methods and algorithms, unfortunately, the flashy demonstrations (important for fundraising and PR) may go out of lab reach. Similar to how ChatGPT capitalized on published transformer research, companies will capitalize on published public-domain research. It may become crucial to have a credit mechanism for academics for commercial usage of their work (this is not covered by academic metrics such as h-index).

The largest robotics companies are converging on general-purpose humanoids, optimizing for the broadest possible applicability and commercial value. By analogy to computer engineering’s domain-specific diversification, the next productive frontier for academic labs may be task-specific robots: surgical, agricultural, soft robots, etc., which diverge enough from general-purpose designs to make bespoke solutions worthwhile. A positive side-effect of the commoditization of hardware components (like actuators, IMUs, perception systems like the Kinect) all come together to facilitate this kind of development.

The Future

While the external perception of robotics and AI research is that we are undergoing a revolution today, the internal view is more consistent with commoditization and convergence. This paradigm has had a lot of positive side-effects, like establishing a framework and shared infrastructure, but also some serious downsides, like stifling research that doesn’t fit the mold.

In response, we already see the reality of robotics research moving up the stack, and we will potentially begin to see examples of domain-specific diversification if the largest companies with the largest datasets corner the end-to-end behavior cloning approach.

Beyond that, it’s too early to predict if there is a paradigm shift coming. Kuhn says on this topic:

Sometimes a normal problem, one that ought to be solvable by known rules and procedures, resists the reiterated onslaught of the ablest members of the group within whose competence it falls. On other occasions a piece of equipment designed and constructed for the purpose of normal research fails to perform in the anticipated manner, revealing an anomaly that cannot, despite repeated effort, be aligned with professional expectation.

Will there be a “piece of equipment” or “normal problem” whose unexpected result paves the way for the next robotics revolution? Optimistically, it seems like the current paradigm still has legs for a little while longer, but there is already work at the fringes looking toward the next set of leaps, like world model research, neuromorphic computing, etc. We’ll be writing about these topics over the coming weeks and months; stay tuned!

Subscribe now

If you enjoyed this post, please like (❤️) and restack — it helps others find my writing. Subscribe to receive new posts. All of this is greatly appreciated.

What is a good comparison is this performance to last year’s, when the best robot time was over 160 minutes, or more than 3x this year’s time. That’s a remarkable improvement in one year. My doctoral thesis involved building and controlling hopping and running robots, and since then I’ve tried to design and build efficient commercial legged robots, giving me a decent idea of the constraints involved. So, in this article I wanted to try and examine — how did they do it? Is there some magical technology or technique that unlocked this performance? How did they beat the significantly better-known Unitree (who reportedly had to supply an ice pack backpack to try and complete the race without overheating)? Could a western robot have won?

This publication and this post contain the author’s personal thoughts and opinions only, and do not reflect the views of any companies or institutions.

The basic physics of hopping and running

Hopping, very simply, consists of alternating phases of a leg pushing against the ground (“stance phase”) and the body flying through the air (“aerial phase”).

In aerial phase, the body simply free-falls (constant acceleration due to gravity). You can think of this as losing vertical momentum. In stance phase, the job of the leg is to push against the ground to reverse this vertical momentum. The job of the “knee” actuator is primarily to generate this force in stance phase.

The other basic leg function is repositioning for the next foothold. In bipedal running, while one leg is pushing against the ground, the other leg is swinging to reposition for the next step. The job of the “hip” actuator is primarily to swing the leg forward.

Bipedal running is simply these two functions alternating in the two legs — while the left leg pushes against the ground, the right leg swings forward, and vice versa. Of course, this is an oversimplification in many ways, but it still captures the main effects that contribute to running energetics. Namely, it becomes clear that:

• the knee actuator must produce enough torque to reverse the entire robot momentum in the stance duration T_s

• the hip actuator must product enough power to accelerate the leg forward in the swing duration T_sw

The way a robot runs faster is that it increases its stride length and/or shortens the stance duration.

Shortening the stance duration requires a higher amount of knee torque to be needed to accomplish the same momentum reversal. Swinging the leg faster, and covering a longer stride length requires more torque and power from the hip actuator.

And just like that, with very basic physics, we’ve recovered the dependence of running speed on the torque and power produced by the actuators.

The basic physics of motors

Electric motors dissipate energy in an exact relation to the amount of torque they produce, and these quantities are related by an appropriately-named constant termed the motor constant, K_m. If τ is the torque produced by the motor and Q is the heat it produces,

In the “New Motor Models” section in my thesis (2017) I described how a K_m scaling relation can be approximated from rough first-principles geometry arguments. In particular, for a fixed length scale, K_m scales with the square root of motor mass √m. In a recent post, longtime blogger and roboticist Ben Katz generalizes and gives this coefficient a name , the “figure of merit (FoM),” which we can use here:

The r above is the motor diameter. To estimate motor mass m, I decided to relate it to the motor diameter and (unknown) length. With these, and assuming a high but reasonable FoM of 15, we can extrapolate the likely K_m.

To estimate the rotor inertia, we can relate it to the motor mass and inertia as j ~ mr² as Ben Katz also does.

Adding a geartrain (gear ratio G) after the motor amplifies its torque and reduces its speed by G. So, it helps with torque production, but it has a very deleterious effect in legged systems when accelerating. Since the rotor of the motor itself has to spin faster, the rotor inertia j in the output frame appears scaled to G²j, which can quickly become very large. Thus, a small motor with large gearing becomes very sluggish at accelerating its output, even if it can statically produce a large torque. This is obviously bad for the “swing phase” described above.

The Honor Lightning’s technology

There isn’t a technical report on this robot as far as I know, but some online articles list a few specifications. I referred to this substack article for this post. A couple of notes:

• This article and a few others say that the robot has 55 joints, but that is definitely a mistake. Potentially with hands (that were not equipped on these half-marathon versions) it could have 55 joints, but as deployed, they probably had closer to half as many joints.

• The page also lists “Leaderdrive” as a harmonic reducer technology partner implying that strain wave gearing was used. However, based on the analysis below, a lower reduction-ratio planetary or another type of gearing is more appropriate, especially for this kind of efficiency-critical application.

Actuation: motor, gearing, gait

These three factors are all interrelated and have an effect on how much energy is required and how much heat is produced. To see how, let’s start with the motor.

Typically, the motor K_m can be found in the datasheet, but in this case there’s no public reporting on the motor specs.

However, we can see the size of the fairly large hip/knee motors attached to the upper leg (my rough estimation is that the outer diameter is somewhere between 110-150mm from the image above). We can look at a couple of potential options: first, a reasonable 115mm diameter catalog motor, which I chose from TQ’s frameless motor catalog for similar reasons to Ben Katz’s blog post — they are well-documented and have a large selection. Second, we can use the scaling principles to make some reasonably good approximations of K_m for a hypothetical larger motor. I extrapolated to a 150x25 sized motor to obtain a K_m of 1.52 Nm/sqrt(W), and a mass of almost 2 kg.

Since we don’t know the gear ratio, we can use our simple physics model (script linked in references below) to estimate the power consumption for running for the “small” and “big” motors above as a function of G:

Note that:

• A high gear ratio is nice to minimize the power in the knee actuator (since its job of supporting the robot weight is made easier with mechanical advantage), but a high gear ratio also makes the leg swing energetically difficult.1 There’s usually a middle-ground optimum.

• The larger motor (150x25) prefers a smaller gear ratio (~23:1), and the smaller motor (115x25) prefers a higher gear ratio (~40:1).2 These are shown with the dashed gray lines.

• Both of these options seem to be able to accomplish the basic push-ground and leg-swing functions, with modest robot power consumption of 400-500 W.

So, in sum, the motor is not magical technology and in fact, a range of existing or projected options would work, when appropriately sized for this task. I’ll get back to this last bit and the green lines in the plot later.

The dissipated knee power (which is typically the main thermal limiting factor) is ~150W for both solutions. This is almost an unavoidable consequence — due to the predictable scaling of motor K_m, running at human speeds with a humanoid-sized robot will inevitably generate this amount of heat!

This, finally, is where we would see a potentially large difference between the two motors. Motor cooling is affected by the surface area over which heat removal can occur, and the larger motor has 70% more surface area. Even so, over a prolonged period, 150W is a large amount of power to dissipate from a single motor, and this is where one of the stated innovations in this robot design appear to be coming to bear (source):

According to Honor, the liquid - cooling pipes penetrate deep into the motors like capillaries. The high - power liquid pump has a heat - exchange flow rate of more than 4 liters per minute. Each of the four drive motors in the lower limbs is equipped with an independent liquid - cooling circuit.

Liquid cooling is not new, but it’s definitely not what I would call a commodity. It has shown up in research periodically, and on the commercial side Apptronik tried it for a few of their prototypes but (to my knowledge) does not use it on their main Apollo platform. While it definitely is not magical technology, it has been niche so far. As described above, it is absolutely essential (and so far quite challenging) to be able to dissipate ~150W from a motor for running at these speeds. From that respect, the liquid cooling tech is a key enabler of this type of performance.

Caveat: The script I used to generate the plots above makes a lot of simplifying approximations. It doesn’t capture the energy dissipated in other motors (arms, ankles, abduction, etc.). The basic physics principles don’t lie about the periodic center-of-mass behavior, but this doesn’t model other oscillations in the orientation as the body sways etc., or losses like friction or air resistance. The inertia of the leg is left out of the swing inertia calculation, since there is no way to approximate it properly with the information available. Published materials emphasize a lightweight leg construction, which indicates that the rotor reflected inertia will likely dominate it (and so the script’s approximation is likely good). There are more accurate ways to estimate the swing energetics incorporating the leg kinematics and swing trajectory, but I wanted to not increase the complexity of this analysis and chose to err on the side of simplicity. Still, I think the main estimates and talking points (motor / gearing selection for the knee motor, and power dissipated in it) can be trusted.

AI and autonomy

There’s nothing to write home about here. The gait controller could have used either a reinforcement learning (RL) controller, which is easy to train for flat ground, or a model-based controller. The autonomous navigation system used a provided GNSS system and just had to follow the route waypoints. This is all very well-understood technology.

Battery

Let’s assume that the battery was chosen to last 1.5 hrs (the robot finished in < 1 hr). For 600 W consumption (based on the figures above with some buffer), the battery would have had to have 900 Wh capacity, and at 300 Wh/kg energy density, the pack would have weighed 3 kg. This is well within reason for a 45 kg robot. Additionally, a 1.5 hour discharge time indicates a 1/1.5 or 0.67C discharge, which is well within the ratings of most existing batteries.

The Unitree H1 reportedly needed “pit stops” and battery cooling ice, indicating that it was consuming much higher power. We’ll talk about that more next.

Engineering always involves tradeoffs

Engineering is always characterized by tradeoffs — that’s what makes it challenging but also fun. Especially today with ever-stronger AI language models, the very human skill of judgment and knowing how to made tradeoffs is much more important than the rote work of completing a design to spec.

Even with the very simple model above, it was not that complex to roughly design a drivetrain that is theoretically capable of this feat. Then why did the competitors in the race, including more established and widely-shipped humanoids such as from Unitree or Agibot, not compete as well?

We can use the simple model to generate an equivalent energetics plot for walking at 1.5 m/s, a much more modest but potentially more common activity for a commercial humanoid robot:

The gray lines are as before — gear ratios optimized for half-marathon running. The green lines are where the power is minimized for walking, and they are significantly different!

Let’s say you design your robot to excel at the normal walking task and chose the green gear ratios. The knee motor power to run a half marathon with that green design consumes > 300 W, more than 2x what we had with the running-optimized gray designs. It wouldn’t be so surprising to need ice packs!

Conversely, the running-optimized gray design, when used for the walking task, wastes significantly more motor power than the green designs (as seen from where they intersect the blue curves). We couldn’t model this effect with the information available, but using larger motors sized for running also increases the weight of the robot and constantly wastes power when it isn’t running at full speed. You can visually see the difference in motor sizes between the Unitree H1 and Honor Lightning:

The larger motors will have all sorts of practical (if not fundamental) consequences like bumping into objects while operating in homes or factories.

Closing thoughts

What should we conclude from Honor’s accomplishment? First, the capillary motor cooling solution, if mass manufacturable, is a genuine advance, and I suspect this running pace would not have been sustainable without it. Second, even if there wasn’t any “magic” needed, this was a really impressive engineering effort and result. For better or worse, it deserves to be a landmark tantamount to Deep Blue v. Kasparov.

Having said that, I don’t believe this says anything at all about human half-marathon performances. It doesn’t even imply that a humanoid robot could join a race among a sea of people without GPS and resiliently finish the race. I wish those comparisons would be left out of the press coverage.

Another thing I found interesting is that the Lightning robot was reportedly developed in about a year, between MWC in March 2025 and the April 2026 race. That is incredibly fast. However, what is even more stunning is that the R&D team reportedly had 2,600 people. Comparing to a few US humanoid robot companies, to my knowledge, that eclipses the headcounts of Boston Dynamics, Figure, Agility, and Apptronik combined (I am not sure of Tesla’s Optimus-specific headcount). On top of that, you have to account for the partner and manufacturing ecosystem that was brought to bear, as reported by the same linked article.

Is all this worth it? It probably isn’t for most of these companies who need to spend their resources developing applications customers need and will pay for, but the cooling and weight-reduction advances may well be useful for more practical purposes like carrying heavy payloads down the line.

If you enjoyed this post, please like (❤️) and restack — it helps others find my writing. Subscribe to receive new posts. All of this is greatly appreciated.

Subscribe now

References

• Script used for power estimates (Github gist)

• Spreadsheet with motor parameters and estimates

To showcase and demonstrate some of the insights from the past articles, I’ve put together a demonstration of the insights from this article series that you can try out, modify, and learn from. This demo combines the flexible task programming and reasoning of the Gemini ER Vision-Language-Model (what is the scene, and what should I do?) and classical camera calibration, kinematics, motion controllers.

This post describes how it is put together, goes over of some of its interesting capabilities, and the aspects of its design that directly impact its strengths and weaknesses. To conclude, we will try to compare this approach against fully modular (model-based) as well as fully end-to-end methods. The code is open source, and I’m putting the ideas out there for discussion and feedback.

This article is the last part of a series on end-to-end robotics pipelines. Links to the other articles are below.

Trying out the demo

To make it as accessible as possible, the demo runs in the browser with no software installation required, and can be accessed from your computer or even a phone. Click this button or to open the page:

Link to demo

The environment is set up for tabletop manipulation with a robot arm. The colored blocks are objects that we can instruct the arm to move, the “plates” can serve as potential goal locations, and the grey cylinders can serve as obstacles to be avoided.

The demo uses a Gemini Robotics ER model for task reasoning and perception. To try it out, you need to grab your own Gemini API key (free tier), or use the pre-baked fallback plan, which will execute the “Put the blocks away where they belong” default task. Correspondingly, click “Run Task” (with API key) or “Use Cached Task” and watch! Use the mouse to orbit the camera, and check the console for debug logs.

What it does well

Flexible task programming and reasoning. Tasks can be prompted without needing task-specific programming, which is a major selling-point: the possible tasks are not limited by what is programmed at the factory. Gemini processes the prompt together with the scene and can break down multi-step tasks. We’ll go over how Gemini’s outputs are used by the rest of the system below.

Results from some tasks:

Place the red block on the blue target

This simple task shows the VLM’s visual and task understanding. Additionally, its language understanding can parse semantically similar words in the context of the scene (e.g. block vs. cube, or plate vs. coaster vs. target).

The video also shows the reactive obstacle avoidance allowing the arm to not collide with the cylindrical obstacles. This capability, with associated safety benefits, does not require any training or motion primitives to be built into the VLM. More on that below.

“Put the blocks on matching targets”

The VLM successfully reasons that blocks go on color-matched plates, and breaks down the task into a number of steps (move red block, move blue block).

“Swap the red and blue blocks”

This task requires a multi-step plan to move one of the blocks out of the way first, and the selection of a free location to store it.

The wireframes displayed in the animation show the spatial understanding ability built from a combination of a Gemini VLM with classical computer vision. Objects in the scene are semantically classified — into objects (blue wireframes), potential goal locations (green), and potential obstacles (black) — by the VLM guided by prompting, without hardcoding.

As a note of caution, I had a few runs where it chose the “free” location incorrectly on top of another block.

“Put away the blocks”

The success of this (underspecified) prompt showcases the language and intent understanding of the VLM. However, I will temper with the note that in some runs, it did try to move the green block and confuse itself — feel free to try it yourself!

“Wave”

This silly task shows that the VLM’s task understanding goes beyond tabletop manipulation, as it can produce waypoints just intended for arbitrary motion. However, this demo will only successfully perform horizontal motion due to the 2-dimensional understanding of the VLM — more on that below.

What is challenging

The principal weaknesses are also to do with the 2-dimensional understanding of the VLM.

“Stack the blocks”

It correctly moves multiple blocks to the same horizontal position, but does not properly reason about the vertical location of each drop-off. This results in the later blocks getting smashed into the ones already placed.

The architecture explains strengths and weaknesses

The architecture of the demo is shown below:

Gemini (VLM) blocks are blue, and blocks built using classical methods are green. Each layer is independently swappable, and the AI model doesn’t need to know anything about the robot’s embodiment. This recreates the modularity of a Sense-Plan-Act architecture while retaining the semantic reasoning of a foundation AI model.

Vision-Language Model (VLM)

The demo uses Gemini ER, whose inclusion I previously motivated with a coding agent analogy. Its inputs are the text prompt and a single image, and its outputs are grounded in pixels in the same image. This keeps its behavior well-defined and decoupled from the robot embodiment, solving many of the issues with X-VLA in a similar setup.

However, it builds in a few assumptions that should be acknowledged. Most importantly, its understanding of the world is decidedly planar (pixels in the image plane).1 The view must therefore be chosen to avoid occlusion, parallax-related issues as the camera moves, and tasks that require positioning along the camera axis (like the block-stacking task above).

Gemini ER can be prompted to output structured JSON, which is easy to work with in downstream layers. The system is first prompted for “perception”, which does object detection, semantic classification, and bounding box identification. All of these are very common functions, and easy for this model to complete in ~1 second. An example output for the perception block is below:

[
   {
      "label":"green block", // <- a name
      "point":[637, 232], // <- position in image coordinates
      "box_2d":[531, 157, 743, 305], // <- image coordinates of bounding box
      "type":"block" // <- semantic classification
   }, // ... other detections 
]

The next step asks the model to plan the motion for a task. We specify an output format that limits the output to “calling functions” that the arm and its lower-level controller is capable of executing. Example output from Gemini (took anywhere from 4-10 seconds):

[
   {
      "function":"move", // a function that moves the arm
      "args":[584, 753, false], // position (image coords) + 1 bit indicating height
   },
   {
      "function":"setGripperState", // a function that closes or opens the gripper
      "args":[false] // false to close, true to open
   }, // ... other steps
]

The full prompts to get these outputs are part of the open-source package.

Spatial understanding

First, we convert the image-plane understanding of the VLM into spatially accurate waypoints that the arm can act on. For this conversion, I also sampled depth values from the camera location (easily reproducible with stereoscopy or a model like DepthAnything). I chose to use the bounding boxes to isolate the depth values in a region around the object center, and use that to fit primitive shapes to the detections (rendered with wireframes in the videos above). This can be done by well-understood camera geometry transformations, and also allows for relocation of the camera, unlike in a VLA where the camera geometry is inextricably linked into the rest of the model. The output of this block is 3D waypoints and a representation of the obstacles.

The object shape affects how well a bounding box captures inlying depth pixels. Gemini also has a native ability to output segmentation masks, which could allow for further refinement in this computational block.

Model-based local planner

The next part is a model-based local planner that actually generates control signals. This decouples the control rate from the slow runtime of the VLM completely, and no retraining is needed to generate novel motions for new scene compositions. Adaptations for payload could be built into this layer without affecting VLM.

For obstacle avoidance, we use a “potential field” that pushes the end-effector away from obstacles (you can see orange arrows appearing briefly in the animations above), while moving toward the desired goal. This is a classic reactive motion planning technique, one of a family of well-understood algorithms along with sampling-based and grid-search planners.

The interface is crucial

The VLA approach had no choices to be made about the type of interface — when trained on the same embodiment end-to-end, input pixels get mapped straight through to actions. However, with this hierarchical controller, the choice of interface is quite important. While it resolved many of the drawbacks of the full end-to-end approach that held back a demonstration like this, one of the interpretations of the bitter lesson is that any hand-crafted interface design hampers system performance.

For example, for grasp generation in this demo, we have to assume that knowing the location of block is sufficient to produce an action to grasp it. However, different grasping actions may be needed for soft or unusually-shaped objects, like eggs, cloth, etc. One possible extension to resolve this is to incorporate a grasp generation module seeded by the object centers and bounding boxes. A VLA will just directly output actions, which is not limited by this kind of architectural judgment, but also may require a lot more training data and fail unpredictably when out of distribution.

Scoring the criteria from the first article

The architecture described above is neither an end-to-end VLA, nor a modular model-based one. For specificity in this section, I’ll assume the former camp as being represented by something like Physical Intelligence’s models (a small version of which we tried hands-on with X-VLA), and the latter as being represented by the MIT 2014 Atlas method. Both were discussed in the first part in this series about end-to-end robotics pipelines.

All that said, where does this “hybrid” hierarchical strategy fall? We identified a number of criteria in previous articles, and can try to roughly size up where each falls:

My summary would be that the end-to-end method can be the best if it is scaled potentially ad infinitum and has very fast computational hardware, which has practical (data requirements) and efficiency drawbacks. I think the hybrid architecture could be a good middle ground to greatly expand capabilities with less data and added safety and efficiency, but has some bottlenecks from interface choices that may impact some applications (but in a predictable way). I’m open to your thoughts — let me know below!

Leave a comment

Closing thoughts

This demo was put together with models released within the last year, but also with ideas that have existed for decades. We’ve been seeing transformational improvement in the capabilities of deep neural networks, leading in many cases to large strategic shifts to embrace fully end-to-end architectures. However, this shift brings with it new problems in safety, efficiency, and predictability. This post goes over a proposal for a hybrid architecture that attempts to draw on the strengths of both camps.

There is room for improvement in end-to-end VLA approaches with scaling, as well as in this kind of hybrid architecture (faster VLM inference, multi-view VLM). “World model” methods are rapidly gaining popularity as a component of larger modular pipelines (stay tuned for future posts on this topic). I also plan to look more into how to build an “embodied reasoning” open-weight VLM in future posts.

Please check out the demo, and the source code.

If you liked this post, please like ♡, share, restack, and subscribe — it helps others find my writing.

Subscribe now

Further reading

Other articles in this series:

1. The architecture behind “end-to-end” robotics pipelines

2. Online motor adaptation

3. VLA debugging insights

4. This article

5. Demo combining the best features of end-to-end and classical approaches

In this part, we finally close the loop to get our WidowX robot arm in the MuJoCo simulation to execute some manipulation tasks. I’ll go over how to build up (from scratch) something like the following behavior from a text prompt, and what we can learn about the architecture of robotics pipelines in the process.

Result of “Place the red block on the blue target”:

An end-to-end Vision-Language-Action (VLA) model is the obvious modern technology researchers and companies are moving toward for this kind of functionality, and part 3 of this series was dedicated to understanding them from the inside. The deployment exercise for this part made clear that a small VLA’s failure modes are difficult—to the point of impossible—to eliminate without retraining.

That observation ultimately forced a pivot to a different architecture, where the flexible programming and semantic reasoning layer delegates physical grounding to explicitly separate tools. This post explains how that architecture works, and what it says about robotics pipelines more broadly.

In addition to the story, all the code is open-source — feel free to learn from, star, and fork! Also, if you like this kind of post, please like, share, and subscribe:

Moravec’s paradox and VLAs: control bandwidth problem

In the previous post, we spent some time interpreting the perception and language understanding in VLAs — specifically, the Vision and Language parts. In many ways, the action head exhibits the most architectural diversity in VLAs.

There are broadly two types of action heads. (Auto)-regressive action heads generate actions sequentially, one at a time. This is similar to how most LLMs work today, generating tokens one after the other. Generative (or diffusion / flow-matching) action heads, in contrast, generate a whole action sequence at a time and incrementally refine it, similar to diffusion-based image generators.

Regressive action generators have a fundamental difficulty when used for behavior cloning in continuous action spaces. As Max Simchowitz presents in his recent CMU RI seminar1, the issue is that a small deviation takes the red robot trajectory off the training (expert) demonstration distribution, and it is unable to recover.

The same problem doesn’t occur in discrete spaces (like text generation) because they can be trained with a 0-1 or cross-entropy loss function, encouraging very aggressive contraction to the training distribution. Simchowitz identifies this challenge in continuous spaces with Moravec’s paradox (why learning hasn’t been as effective in physical tasks as in symbolic tasks like language).

Action chunking2 presents a way to get around this problem. By producing an action sequence, over which the natural dynamics of the system is assumed to prevent compounding error, the rate of divergence is kept under control:

A key assumption there was that the underlying system needs to have some strong stability properties. I won’t go into definitions here, but in practice, this means that VLA actions are almost exclusively desired positions (as opposed to velocity or torque). More generally, this means that the behavior is what is called “quasi-static”, i.e. the robot goes through a sequence of statically stable configurations. As an aside, this is also why VLA-implemented manipulation behaviors are slow and wouldn’t apply to dynamic behaviors like agile locomotion; quoting this Quanta magazine article, “Atlas moves like molasses while grasping auto parts but glides like a gymnast when it’s not touching anything except the floor”.

So, action chunking is a way to address the control bandwidth problem from part 1 for regressive policies. Generative action heads don’t have the same inherent divergence issue, and it does seem like in practice most VLAs use that strategy — this also applies to our demo setup with X-VLA from part 3. They learn a distribution over actions, and at inference time, start with a pure “noise” action and iteratively denoise it. One thing to note is that the trajectory horizon does not impact how long the inference takes (it is just the size of the action distribution learned during training). This means that shortening the action horizon size in order to get faster results isn’t an option like it typically is in model-predictive control.

Now that we understand VLA action heads a little better, let’s move on to closing the action loop.

Closing the loop with X-VLA: generalization and separation problems

The VLA outputs action chunks (a sequence of desired poses), and we now need to control the motors to reach them. The model for the WidowX arm in our simulation is set up for position control on the joints. This is in part due to how most people are using this arm (in some cases due to algorithmic constraints as mentioned above). For this article, I chose to keep that as is, and as a first pass, implement the most reasonable control method in this situation: inverse kinematics (IK). The implementation uses gradient descent to iteratively find the joint angles that reach a certain pose. This is a generalizable and quick method that will probably get replaced in the last part of the series by a non-IK solution.

After closing the control loop, prompting the VLA to “pick up the red block”, and running the simulation — well, it didn’t work. At this point, it was a little bit of the same challenge of “black box debugging” as in part 3, but now with more (literal) moving pieces.

It’s important to remember that X-VLA is a small VLA, and its generalization capabilities are limited by model size. As we saw in part 3, the model’s spatial reasoning (how far to reach, when to close) is tightly coupled to the training camera viewpoints. The camera intrinsic and extrinsic parameters are wrapped up in the full X-VLA policy and not separable3, and so I tried to modify the images received by the policy to try and match the training dataset.

I went into the BridgeData training dataset, and found the most similar task in the training data, grabbed the training video, and tried to make my scene resemble it as closely as possible. To do this, I manually tuned the camera position, robot gripper initial pose and framing (camera extrinsics), image field of view (intrinsics), aspect ratio “squishing” to match training data, lighting / shadows, table appearance:

Unfortunately, despite the manual tuning, and also completely decluttering the scene, the policy didn’t succeed with the prompt “Pick up the red block”:

It consistently overshot the block, which indicated to me that the visual processing had a consistent error, but fiddling with the camera settings didn’t yield a better result. The structural issue with VLAs (non-separability of camera and kinematics parameters) makes this debugging quite challenging, even beyond the techniques from part 3. If you know of anything that could have gotten this to work, let me know in the comments!

Leave a comment

I suspect that the generalization abilities of this size of VLA are just not sufficient to be able to use the policy zero-shot. There are two reasons why that is a roadblock: First (isolated to my usage here), I didn’t have a leader-follower arm or space mouse to collect more training data and go through a fine-tuning process. The second (and more fundamental) issue is that this limits how this kind of strategy can be used by robot end-users in ad-hoc unknown environments.

The flexible task programming and semantic task understanding of VLAs were some of the motivations for this project. Is there an alternative solution that can keep those strengths while adding some needed structure?

An “agentic” modular alternative

For scene and task understanding combined with flexible programming, we need some kind of VLM, but is there a way to get information out of the VLM in a more structured way?

In late 2025, Google released Gemini Robotics 1.5, which consists of two models designed to have a hierarchical interface:

Out of the two, I only used the ER (Embodied Reasoning) model, which has been trained to output structured text combining the spatial understanding and function calling capabilities of the impressive Gemini model family. As documented here, the “pointing” feature is effectively a customizable vision processing pipeline, and I found it to be incredibly robust:

The function calling capabilities can also be used to break down complex tasks into sub-steps, which is what I used for the working demo in the first section of this article. Here you can see that it is flexible to different prompts with no other changes:

Most shockingly, I spent only a couple of hours with the Gemini models to get to the successful end result above, after unsuccessful attempts over a significantly longer period with X-VLA.

So, why does this work so much more easily?

Just like Simchowitz did in his RI seminar, I think I’d have a pragmatic answer to do with scale, as well as an algorithmic answer independent of it. On model size, Gemini ER 1.5 is described as achieving “the low latency of a Gemini Flash model” for spatial tasks, which suggests it's Flash-scale (~8B range) but much larger than X-VLA (0.9B). On the algorithmic side, the difficulties we ran into with the VLA often had to do with inseparability of concerns (kinematics, calibration parameters not separable from the policy), and generalizability (difficult to tell when we were out of distribution).

I think an appropriate analogy here is between an LLM (even a coding-tuned one) to a coding agent like Claude Code (an LLM in a larger system that can interact with “tools”). A coding agent doesn’t ask the LLM to emit machine code directly; it keeps the model in the semantic reasoning layer and delegates execution to existing well-understood tools. In this analogy, I’m suggesting that camera calibration, kinematics, motion controllers are tools that the VLM can benefit from interfacing with. Gemini ER just works on images; a well-defined, separable concern without introducing variability due to the robot morphology. Our known camera transformations then lift its image-space outputs into 3D. If we move the camera (impossible with X-VLA without retraining), we can simply replace the camera calibration parameters.

However, this structural separation appears to contradict the pure end-to-end view that goes back to the “bitter lesson.” Overall, in my opinion, the bitter lesson essay has been interpreted more broadly than current evidence supports, and we will continue to see reinterpretations and corrections.

Closing thoughts

In this part of our series on robotics pipelines, we demonstrated a simple setup that exhibits flexible task programming. Despite our best efforts with an end-to-end VLA, this success came from coupling a strong VLM with model-based “tools” such as camera geometry and inverse / forward kinematics. This seems to me to reflect some of the strengths of agents that interacts with tools vs. an equivalent chatbot-style LLM. It certainly provided a clean way to integrate the strengths of a large learning-based model with structured model-based methods — something I’d set as a goal in part 1 of this series.

While this is a nice result, there are still a number of limitations: Gemini’s task planning is slow, even with cloud hardware. In the current implementation, the full plan is created at startup and there is no replanning for dynamic environments. The model is also not “open” and likely an order of magnitude larger than X-VLA. In the future, I may look into what it takes to develop an “embodied reasoning” model — it seems like the Gemini ER model appears to build on the ideas of the published SpatialVLM.

In the last part of this series, I will plan to improve the lower-level controller from its naive IK implementation to show more responsive and adaptive behavior. I will also aim to publish it in a browser-runnable format so you can very easily and rapidly see the effects of different prompts. As a reminder, the code is all open-source.

If you liked this post, please like (♡), share, restack, and subscribe — it helps others find my writing.

Recognizing performance deterioration

This Verge article from March 19 reports that there could be an impending recall of Tesla’s Full-Self Driving (FSD) service. I’m not interested in making any judgments about self-driving capability, but rather whether the root cause has anything we can learn from in broader robotics.

The issue appears to be that the system didn’t know when it wasn’t working well (causing the issues in the NHTSA filing), or that it did and didn’t notify the driver (which is unlikely, so we’ll assume the former).

This phenomenon isn’t isolated to AVs. The latest article in my Vision-Language-Action (VLA) robotics pipeline series went hands-on into debugging one, and while we found some techniques that can aid developers, they didn’t directly help at inference time. Item 1 in Rui Xu’s candid post-mortem of K-Scale Labs mentions the pitfalls of trusting a “large model” vs. dedicated safety features. Recent papers on VLAs mention the fragility when moving away from the training distribution (e.g. Fang et al Jun 2025, Hu et al Jan 2026).

Potential solutions: redundancy, confidence, architecture

NVIDIA recently announced their new Alpamayo model and accompanying AV stack as a reference open model and toolchain. During the CES 2026 keynote, Jensen Huang said something intriguing about safety:

This parallel or hybrid architecture with a classical stack and a policy arbitrator were also covered in this CounterPoint research article. Interestingly, I can’t find references from NVIDIA themselves about this parallel system other than Jensen’s keynote — it’s possible it is just early in development.

A related approach is to have the VLA output some kind of confidence (vs. a separate “policy arbitrator”). Zollo et al (Dec 2025) formalizes the problem of confidence calibration for VLA policies, describes how to extract confidence estimates from contemporary VLA architectures, and notes that current VLAs lack a reliable mechanism for quantifying the uncertainty of their chosen action sequences. It also introduces two potential remedies: prompt ensembles and action-wise Platt scaling.

Lastly, inserting some debuggable interfaces into end-to-end pipelines can facilitate inspection and safety — lower-level controllers can apply dedicated safety constraints based on the information passed down from a higher-level controller. This appears to still be possible in most successful humanoid robotics demonstrations of today due to a combination of factors. Keeping that architectural feature around may have long-standing benefits, based on current events in the AV industry!

Thanks for reading! I have been working on the next part of the end-to-end pipeline series, with a deep dive into the action head and closed-loop behavior. If you liked this post, please share and subscribe.

All these matrix multiplication units are based on the systolic array, an architectural concept invented by HT Kung at Carnegie Mellon University in the late 1970’s. And Kung’s group didn’t stop at matrix multiplication, they presented a concept of systolic networks of arbitrary processing nodes that could do way more. While some of those concepts appear in niche signal-processing ASICs today, the dominance of deep neural networks over the last decade has caused this history and potential to be significantly overlooked in my opinion.

My interest in this (and the goal of this article) is twofold: (1) Shine a spotlight on this fascinating research and preview the types of problems that can be solved with systolic architectures. (2) Dig into and potentially uncover jumps in performance and efficiency for AI and robotics. I believe that holistic full-stack understanding and optimization (bringing together algorithms and hardware) will be key in advancing these technologies.

Beyond this post, we won’t stop at a theoretical overview — leveraging the computer engineering experience and story-telling of we will actually build up accelerators to use in general-purpose robotics, AI, and scientific applications. We have an article coming soon with the first step, so make sure to subscribe!

Subscribe now

Why systolic architectures

A systolic architecture is characterized by a network of processing elements (PE) that feed data to each other instead of going to the memory hierarchy for operands.

The core benefits are:

• It alleviates memory bottlenecks by allowing multiple compute operations to occur without going to memory (as nicely depicted by the figure above). The design can allow computation time to be balanced with I/O if designed properly, avoiding one stalling due to the other.

• It can create simple, regular designs → a modular setup that can be extended for different functions. It is relatively easy to write the RTL!

• 2D arrays can very easily be deeply pipelined (as we will see below), naturally taking advantage of algorithm concurrency.

The PE network can look like a 1D array (pictured above), 2D array (the most common today), or even other connections for specialized computations.

Data flows between cells in a pipelined fashion, and communication with the outside world is at boundary cells.

The foundation of TPU — a MAC systolic network

A multiply-accumulate (MAC) PE has two input edges and two output edges. In the form drawn below (“weight-stationary”), the weight w is a parameter loaded into the PE. The data x flows in and is passed unchanged left to right, and the current “accumulation” b flows in from the top (usually from a PE connected to the north). The PE does the multiply-accumulate (x * w + b) and passes the accumulated sum down. We assume that the calculation happens in a single “tick” or clock cycle.

A PE in a systolic network is typically a simple compute primitive. Its power comes from connections to other PEs to express complex calculations.

The easiest way to understand how a weight-stationary systolic array works is to understand how a dot product is computed. This is shown in the following image for 3 cycles, and we will walk through the computation.

In each cycle, a new entry of x appears from the left, and one term is added to the dot product. The column of PEs contains a vector of weights. In each cycle, one term of the dot product is accumulated, and after 3 cycles, we have accumulated the full dot product b + w·x.

We now draw the exact same operation, but in an abridged form (not showing the intermediate calculations and instead just showing the inputs and outputs at the ticks they appear).

• A column of the array is drawn as vector weight w_i

• The inputs are drawn as a diagonal (and enters the array skewed in time)

• The output is shown at the bottom, appearing after R cycles from when the input hits row 1, where R is the number of PE’s in the column

In this form it is easy to see that we are pipelining different x_i by starting the second one the cycle after the first one. The initial accumulator value can be set to b_i, an affine bias.

So, with a single column systolic array, holding a column vector w, we are computing y = b + X·w, where the rows of X are x₁, x₂, …

It is also noting the latency between when the x_i starts getting input to when we receive the output: The first element of x₁ enters the array at time t=1, and we get the result out at t=R, so the latency is R-1 cycles.

Making this a 2D array (recall that the input x’s are bypassed to the right from each PE), we see that x_i will just arrive to interact with w₂ one cycle later. We can appropriately skew the columns of the B matrix:

The operation that is executes is Y = B + X·W, where b_ij above is in row i and column j of B, W = [w1, w2] is the fixed weight matrix loaded in first. If W is n×n, and X is m×n, the matrix product is O(mn²) operations (as is standard), but due to the structurally-enforced pipelining, it was completed in O(m+n) cycles!

And just like that, with a very simple MAC-computing PE, we can build up the matrix multiplication hardware unit that is the core of most AI hardware accelerators.

There is much more to be said about how it is implemented in RTL, how it performs, how the matrix shapes affect utilization, the total latency, throughput and efficiency benefits. We will go over that and intuitive insights in an upcoming post. In the remainder of this article, we will turn our attention to other, more overlooked, uses of systolic networks in applications to broader AI, robotics, and numerical methods.

Moving beyond MAC

Keeping the exact same 2D array structure and the skewed input feeding, observe that we had two underlying binary operations: the PE (node) computed multiply (·) and the arrow (edge) computed sum (+). In general, the array will compute the result with those operators replaced by any counterparts: (x₁⊙w₁) ⊕ (x₂⊙w₂) ⊕ ⋯

I’ll be brief with these and list further reading below, and try to draw special attention to ones that are interesting for applications in robotics and AI.

1) Pattern matching (Kung group)

Using logical and (∧) and logical or (∨) as the operations: y = (x₁∧w₁) ∨ (x₂∧w₂) ∨ ⋯

This will return 1 if the vector x matches the vector w, and 0 otherwise.

2) Sorting (Kung group)

This one is fascinating and intuitive. Each PE performs a simple compare and swap operation, and passes the max downward and the min rightward. With n rows and n columns, it will execute the odd-even sort algorithm and produce the sorted array.

A glance at that wikipedia page reveals both a weakness and a strength of systolic arrays. They can only execute algorithms that can work based on local connections (the odd-even sort takes O(n²) operations, vs. more optimal algorithms), but as in matrix multiplication above, the latency is O(n). While the best sort algorithm takes O(n log n) steps sequentially in scalar hardware, the systolic network lets suboptimal algorithms complete with lower latency.

3) 2D motion planning

Deterministic motion planning (identifying environmental obstacles and planning a path through free areas respecting the system dynamics) is a fundamental problem in robotics. About 10 years ago there was even attempt to build chips to solve this problem.

Dynamic programming solutions (including Dijkstra’s algorithm, A*) can be implemented by local and iterative propagation from the goal, and just as with odd-even sort, the nearest-neighbor connection pattern can be mapped well to a systolic array.

Unfortunately, the number of grid cells grows exponentially with the dimension of the ambient space, and this is problematic if we need to have one PE per cell. This makes systolic motion planning impractical unless we only have a 2D problem to solve, but I think it is an interesting application nonetheless.

4) Stereo vision semi-global matching

A PE that accumulates matching costs along a scanline can be used to form a systolic array that implements semi-global matching (SGM). This algorithm is used to calculate disparity in the very popular Intel RealSense camera ASICs.

SGM systolic arrays on FPGAs run this at pixel rate, processing one scanline per clock, and deterministic low-latency computation is obviously paramount here.

5) Matrix decompositions for numerical methods

To some extent, I’ve saved the most promising (at least in my view) for last. Matrix decompositions that aid in factorization are key to solving systems of equations, and this is ubiquitous in all sorts of robotics and general problems.

5.1) QR decomposition. This matrix factorization is the numerically stable way to solve least squares or pseudoinverses in overdetermined systems, and has applications to robot kinematics, SLAM, sensor fusion, online parameter estimation, etc. Additionally, it is a key component of quadratic program (QP) solvers: in active set solvers after the active set is identified, for Jacobian factorization in SQP, etc. These workloads are important in low-level control in robotics and typically need deterministic and low-latency solutions. The Givens rotations method (gentle explainer here) performs local operations on 2x2 submatrices, which lends itself very well to locally-connected CORDIC-implementing PEs in a systolic array.

5.2) Cholesky decomposition for symmetric positive-definite (SPD) matrices. This is a slightly easier factorization if the matrix is SPD, which comes up for example in state estimation, Kalman filtering, normal equations in interior point methods, etc. These workloads would come up in dedicated state estimation blocks in robotics pipelines. For decomposing A = LL^T with lower-triangular L, each PE computes one entry of L using only its left and upper neighbors, making the data dependencies purely local. This is repeated on the smaller matrix till completion.

Both of the systolic implementations referred to above use non-MAC PEs, and a triangular (not rectangular) network — this is very uncommon in current hardware, but was represented in the Kung references above.

For this post, I wanted to stick to high-level intuitive descriptions, but in the open-source TinyXPU project, we will aim to implement and analyze some of these non-traditional systolic networks for robotics and AI pipelines. Stay tuned for the upcoming post introducing this project!

Subscribe now

Conclusion

Systolic arrays were both invented before people think, and are more general than people think. They can have deterministic (no cache misses) high throughput and energy efficiency for algorithms which can work on local data. However, they are bad for working with sparse data (e.g. for sparse linear system solving), and bad for algorithms that need global data (e.g. Householder QR, which needs to operate on a full matrix column at a time).

In the deep neural network boom, the MAC array is so dominant in workload (>95% of operations in any DNN) that the non-MAC compute takes a tiny fraction of time. Dedicating a full systolic array with n² PEs to non-MAC operations would be area-inefficient for neural net workloads. This is why commercial vendors have not explored the co-design of systolic networks with algorithms, including PEs that can do MAC but also other functions like Givens rotations on one chip. For robotics workloads and other general scientific methods, the mix of primitives is different and (in my opinion) worth revisiting.

Further reading

• Kung 1982: Why Systolic Architectures - Great high-level overview of the motivation beyond systolic architectures

• Kung 1982: VLSA Array Processors - Further detail on applications such as QR decomposition

• Google TPU v1 paper

Related posts:

Chip Insights

Mapping algorithms to custom silicon - Part 1

Read more

3 months ago · 22 likes · Bharath Suresh and Avik De

1. The architecture behind “end-to-end” robotics pipelines

2. Online motor adaptation

3. This article

4. Closing the action loop with a VLM “agent”

5. Demo combining the best features of end-to-end and classical approaches

In this part, we get hands-on and build a VLA pipeline from scratch. I’ll be transparent about my starting point: while I have experience with model-based methods, RL controllers, and LLMs/VLMs, generalist end-to-end policies — almost exclusively being realized today as Vision-Language-Action (VLA) models — were new territory. That makes this post a useful vantage point to evaluate their strengths and weaknesses from first principles, and should be interesting to those who have never heard of VLAs as well as those who use them daily.

“Pick up the red block”

The demo is simple: take a specified prompt (like the one in the heading above), run it through the model, and visualize the actions that the model outputs. Obviously, when it is run in closed loop, you would get motion that hopefully results in the action described by the prompt, but there was so much to dig into with just this visualization that it made sense to spend an article on it. In the next part, we will close the action loop and explore some of the low-level controller facets mentioned in part 1.

In the following animation, the configuration of the arm is changed using the sliders (while being given the same prompt), showing that the output action is responsive to the robot and environment state.

The learning journey for this article is captured in a Jupyter notebook that can be accessed and run for free on colab — click here. All details on the software stack are in the open-source github repository for this project (which is where the notebook file also is). If it is a helpful learning tool or template, I’d welcome any feedback, fixes, contributions, stars, forks, etc.

First, let’s quickly go over what a VLA is.

Anatomy of a Vision-Language-Action (VLA) model

A Vision-Language-Action model has three functional components: a vision encoder, a language encoder, and an action head. In practice, the vision and language encoders are almost always a single pretrained VLM, i.e. the vision and language processing are already jointly trained before the action head is added. This means the “vision encoder” and “language encoder” aren’t independently tunable modules; they’re entangled by pretraining.

The architecturally interesting variation is in how the action head attaches to the VLM, and how much of the VLM is modified during robot training. This single design choice has large downstream consequences for what you can and cannot inspect at inference time.

VLA “action head” architectures

Two illuminating (but not exhaustive) designs:

Octo uses a dedicated readout token — a learned embedding (~384-dim) that aggregates action-relevant information from the transformer before a small decode network produces actions. This bottleneck is the closest thing to an inspectable interface in any current VLA: you can probe whether the readout encodes directional intent, object identity, or nothing interpretable.

Transformer → readout_action embedding (384-dim)
                        ↓
            Action Head (direct decode)
                        ↓
                    Actions

X-VLA processes images, language, proprioception, and noisy action candidates together in a single 24-layer transformer, conditioned by 32 learnable soft prompt tokens selected per embodiment. Flow matching then iteratively refines the action chunk over 10 steps. Action-relevant information is distributed across all layers and token types simultaneously.

Input: Images + Language + Proprio + Domain ID
               ↓
┌───────────────────────────────────────────┐
│  Soft Prompt Selection (per embodiment)   │
│  Domain 0 → Prompt_0 (32 learnable tokens)│
│  Domain 1 → Prompt_1 (32 learnable tokens)│
│  Domain N → Prompt_N (32 learnable tokens)│
└──────────────┬────────────────────────────┘
               ↓
┌──────────────────────────────────────────┐
│  Unified Transformer Stack (24 layers)   │
│  ┌────────────────────────────────────┐  │
│  │ [Soft Prompt | Vision | Lang |     │  │
│  │  Proprio | Noisy Actions]          │  │
│  │                                    │  │
│  │  All processed together with       │  │
│  │  standard self-attention           │  │
│  └────────────┬───────────────────────┘  │
└───────────────┼──────────────────────────┘
                ↓
       Flow Matching (10 steps)
                ↓
       Action Chunk (32 actions)

The soft prompts enable efficient cross-embodiment adaptation: only ~9M parameters (1% of the model) need updating for a new robot. But they also mean embodiment-specific behavior is encoded in vectors with no interpretable structure.

The deeper point applies to both architectures: even where a vector interface exists between components (Octo’s readout token, X-VLA’s soft prompts), end-to-end training means those vectors don’t have a physical interpretation that safety constraints can be applied to.

There is more to be said on action chunking and control bandwidth, which I’ll plan to do in the next part of the series.

Subscribe now

Model choice

I approached this as a user rather than a researcher: published weights only, no training data collection, and no fine-tuning iterations before deployment. The target application is tabletop pick-and-place on a WidowX, which is a common manipulation benchmark and exposes the control and perception properties I care about. Another soft constraint was that I’d be able to run it on my personal laptop (12GB VRAM).

These three criteria limit which VLAs can be tried. OpenVLA-7B requires task-specific fine-tuning and won’t fit in 12GB without quantization. π0 needs 24GB+. GR00T requires a Jetson Thor. Gemini Robotics On-Device is trained on dual-arm configurations and isn’t publicly accessible. Octo (93M params) fits the hardware but its pretraining doesn’t support zero-shot transfer without fine-tuning. SmolVLA likewise requires fine-tuning.

X-VLA seems to fit the bill. Its soft-prompt architecture was designed for cross-embodiment zero-shot transfer, and xvla-widowx provides a checkpoint fine-tuned on BridgeData for the WidowX embodiment specifically, meaning embodiment adaptation is handled, while task generalization remains zero-shot. It also has a ee6d (end-effector coordinates) action output mode, which appealed to me because it would allow elimination of kinematics-related variability.

What’s different about VLAs: task programming

VLAs have been heralded as revolutionary for robotics, and it’s true: the prospect of robot programming with natural language is a decided shift. Thinking about my own fielded robotics experience at Ghost Robotics, either customers would (a) directly command the robot, (b) pick between preprogrammed tasks (which can be considered a fixed small vocabulary of commands), or the robot would start its own tasks. Giving natural language commands increases the set of tasks without retraining or reprogramming. The natural language interface changes who can program a robot, not just what it can do. With a VLA, a non-technical operator can in principle specify novel tasks.

The flip side worth mentioning fairly: natural language as an interface trades a small precise vocabulary (preprogrammed tasks) for a large ambiguous one. “Pick up the red block” sounds more expressive than running the “pick_red” preprogrammed task, but as the next section will show, the boundary of what the model actually understands is opaque in a way that a fixed command vocabulary is not.

What’s different about VLAs: calibration and debugging

With classical methods, the process of setting up and debugging a task includes several well-delineated steps:

• calibrate cameras → check camera detection overlay → perception ✅

• calibrate joints → send arm “move up” command and ensure it moves as expected → actuators ✅

With VLAs, there are a few reasons why this kind of unit testing or debugging is simply not possible.

1. Camera extrinsics or joint torque constant parameters will not be isolated: datasets are typically trained with multiple camera angles without explicit calibration, and network learns spatial transforms end-to-end.1 Another example: swapping the camera lens for a fisheye for a wider FOV won’t generalize without retraining, unlike traditional vision where you just recalibrate intrinsics.

2. There aren’t obvious equivalents of non-end-to-end interfaces such as the camera detection overlay or a “move up” command, but we will try to come up with methods to work around this in the next section.

3. Randomness: the trajectory will vary with no environmental change. Flow matching stochasticity is in the action head specifically; the VLM backbone is deterministic given the same input. X-VLA uses 10-step flow matching. Even with same seed, numerical precision in GPU ops causes drift by step 5-6

Due to a combination of 1 and 2 (and slightly exacerbated by 3), it can be complex to reason about the root cause of a failure. Is failure due to (a) vision error, (b) action discretization error, (c) world model mismatch, or (d) all three? When can you dismiss a failure as being out of distribution vs. not?

For a developer, this ambiguity is an inconvenience; with enough time, you can run more experiments and form hypotheses (as we do in the next section). For a deployed system in customer hands, the same ambiguity becomes a safety property: the robot has no reliable mechanism to detect that it is out of distribution and should stop. Classical systems fail loudly (joint limit hit, object not detected, planner infeasible); VLAs fail silently, producing plausible-looking but wrong trajectories. This isn’t a criticism of VLAs specifically, but it is a structural consequence of end-to-end training, and it applies equally to any system where the failure boundary is defined implicitly by a training distribution rather than explicitly by an engineer.

VLA debugging ideas and techniques

Despite the structural challenges mentioned above, I had a fascinating experience coming up with ways to probe and understand what the VLA was doing.

Passive debugging: inspect what the model is already computing

1. Interpret VLM output (infeasible). My first instinct was to query the VLM backbone directly, e.g. by asking something like “Is there a red cube?” or “What objects are on the table?” to verify perception. This turns out not to be feasible for most architectures. In X-VLA and SmolVLA, the action head attaches to the VLM’s final hidden states and generates actions through flow matching in a continuous space, bypassing the text vocabulary entirely. You could query the underlying base VLM (e.g. SmolVLM2 for SmolVLA) separately, but that’s not a fair proxy: fine-tuning on robot manipulation data shifts the VLM’s internal representations, so its text generation behavior no longer reflects what the VLA backbone actually sees. This technique only works cleanly in text-token VLAs like VLA-0-Smol, where actions are generated as autoregressive text strings from the same output head as language. There, scene description quality and action quality share a representation and if the model produces a poor scene description, it will likely produce poor action tokens.

2. Visualize attention on tokens. The ubiquity of transformer-based architectures means that we can leverage the HuggingFace transformer’s output_attentions feature to try to visualize where the vision and text encoders are spending their attention, and if it is appropriate for the task specified. E.g. if we ask it to pick up a red block, is the vision encoder indeed looking at the red block?

Active debugging: intervene on inputs and observe behavioral change

1. Camera ablations (test whether vision is doing object detection or spatial template matching). Move the camera position, and introduce occlusions into one of the views if there are multiple. If attentions fail to track the desired object, it suggests the model learned spatial heuristics tied to camera geometry rather than object identity. In a classical pipeline, object detection is camera-pose-invariant by design (you’d re-project into robot frame), but here, camera pose is baked into the learned policy implicitly through the training distribution.

2. Counterfactual prompting to test semantic understanding. Use variations of the prompt (e.g. red block vs. red cube) that effectively mean the same thing and observe if the output stays consistent. Different outputs exposes that the action head is sensitive to tokenization differences that the VLM alone would smooth over. Also,

3. Primitive action prompts (tests action head’s semantic understanding of motion). E.g. if “don’t move” produces as much motion as “pick up block”, it shows that the action head is always generating motion from its training distribution, v.s. containing a deeper understanding of what motion is.

I suspect that some (if not all) of these will be familiar to seasoned VLA users, but please let me know in the comments if you’re aware of a better technique — chances are that it will many prospective and current VLA users!

Leave a comment

Debugging results

For each experiment, I’ll write what a reasonable expectation would be, the result we see, and the resulting insight or the deeper reason why.

Baseline: pick up the red block

In this baseline, the attention mask on the image looks like it is looking at the red block as well as the gripper. The reaching trajectory output looks like it moves to directly over the red block. Overall, this looks to be a great initial result.

Experiment 1: Picking a different block in view

Expectation: Symmetric action based on spatial understanding from multiple views

Result: The visualized attention shows that it is looking at approximately the correct part of the primary image, though it appears a little offset to the outside of the block. The reaching action appears to not reach as far toward the blue block.

Insight: Most likely, the “3D” spatial understanding from the images is not exactly what we would expect from an exactly calibrated perception and object identification setup.

Experiment 2: Swap blue / red positions

Expectation: Symmetric behavior from previous experiment.

Result: Blue trajectory overshoots more compared to initial red block trajectory, and red trajectory overshoots.

Insight: Spatial understanding and behavior is not symmetric when it is expected to be, indicating a bigger effect of things like training data distribution.

Experiment 3: Altered primary camera view

Expectation: Same behavior as the initial camera view.

Result: The red block trajectory now exhibits the under-reaching from the blue trajectory before, and vice versa.

Insight: The actions are inseparably tied to the camera view and not associated with absolute spatial understanding.

Experiment 4: Remove second camera view

Expectation: Slight degradation in performance.

Result: Removing the side view has minimal effect, but removing the over-the-shoulder view has a disastrous effect on performance.

Insight: It appears that BridgeData has a disproportionately high number of trials with the over-the-shoulder view and significantly altered view points may silently produce much worse results.

Experiment 5: Occluded primary view

Expectation: Second view provides redundancy.

Result: Trajectory moves away from the red block.

Insight: The side camera view seems to not be useful in X-VLA.

Experiment 6: Prompt variations

Expectation: Similar-meaning prompts will produce similar actions.

Result: All these similar prompts largely resulted in similar actions.

Insight: The language encoder is effective at collapsing equivalent prompts to the same actions.

Experiment 7: Don’t move

Expectation: No motion.

Result: Approximately as much motion as when asked to pick the red cube with the left shoulder view.

Insight: the model is still interpolating / extrapolating from training samples and does not have an explicit understanding of commands.

Experiment 8: Change picking position

Expectation: The output trajectory moves to the modified block position.

Result: The visual attention is strangely not on the block in the second example, but largely, the trajectory is responsive to the environment change.

Insight: As long as the prompt is visually grounded, the results generalize in the expected way. Soft prompt for WidowX likely encodes “approach visible object” as primitive (trained on Bridge dataset).

Experiment 9: Move forward / backward / up / down

Expectation: Move as asked.

Result: Approximately the same motion toward the tabletop, largely uncorrelated with the prompt.

Insight: No visual grounding for blind motions. The model has no spatial primitive vocabulary because VLMs are trained on image-caption pairs where “up” describes scene composition, not robot workspace direction.

Experiment 10: Move toward / away from base

Expectation: Move as instructed.

Result: Discernible difference in the two trials accordingly, suggesting some comprehension of the prompt.

Insight: The introduction of the robot base as a (visible) target makes things significantly easier for the model compared to the previous experiment.

Experiment 11: Move away from block

Expectation: Motion away from the block.

Result: Motion largely toward the tabletop.

Insight: The word “away” is probably not having the spatial effect that it should in this context, exposing the ambiguity inherent in using language for robot programming. Whether we like it or not, at least unless the language model is huge, it is safer to assume that the prompt effectively indexes or extrapolates among training data, and that positional prepositions (commonly used by humans to communicate spatial commands) are not reliable to use.

What the experiments reveal about current VLAs

1. Camera view is tied to the behavior, not a calibrated parameter.
Experiments 2, 3, 4, and 5 collectively show that the model’s spatial behavior is tied to the training distribution’s camera geometry rather than to a camera-pose-invariant object representation. Swapping shoulders changes reach distance; replacing the over-shoulder view with a side view breaks the policy entirely even though the scene is identical. This is a consequence of any VLA trained end-to-end without explicit camera calibration. The practical implication is that deployment requires camera placement matching the training distribution, and the model will fail silently when out of distribution.

2. The action manifold is object-centric, not spatially general.
Experiments 7, 9, 10, and 11 collectively show that the model has no spatial primitive vocabulary independent of objects. “Move up/forward/back” all produce similar grasping-like motions; “don’t move” produces motion; “move away from block” produces motion toward the block. “Move toward/away from base” works only because the base is a visually grounded object in the scene. This generalizes beyond X-VLA: any VLA at this scale trained predominantly on pick-and-place demonstrations will have an action manifold that approximates “move toward salient object and grasp.” Spatial relation commands only work when they can be reduced to object identity. This has a direct safety implication: you cannot issue a recovery command (”stop,” “move away,” “back off”) and expect it to override the trained behavioral prior.

3. VLAs at this scale appear to lack compositional generalization.
Experiments 7, 9, and 11 show that novel combinations of spatial primitives and objects (even using vocabulary the model demonstrably knows) produce behavior dominated by the training distribution rather than the instruction. This is distinct from the question of whether larger VLAs generalize better, which is likely true, but out of scope for this article. But it does suggest that for sub-1B parameter VLAs, natural language commands are most reliable when they closely match the task distribution the model was trained on, which significantly narrows the practical definition of "zero-shot generalization" for deployment.

Closing thoughts

For flow-matching VLAs like X-VLA, the classical debugging question “is this a vision problem or a control problem?” is not just difficult to answer but structurally unanswerable. End-to-end training eliminates the interfaces that would make the question meaningful.

The debugging ideas presented here offer partial remedies: passive inspection via attention visualization and active intervention via camera ablations and language variation. These experiments also surfaced three concrete findings: spatial understanding is tied to training-distribution camera geometry rather than calibrated object pose; the action manifold is object-centric and lacks spatial primitive vocabulary; and compositional generalization breaks down for novel combinations of known concepts. These are echoes of the reliability concerns of consistency, robustness, predictability, and safety that are crucially important to evaluate robotics progress.

None of this diminishes what VLAs actually deliver — flexible task programming and meaningful robustness to environmental variation, without any robot-specific programming. The path to reliable deployment is augmenting the strengths of VLAs with explicit interfaces for safety constraints, reducing complexity by utilizing known tools for camera and kinematics calibration, and out-of-distribution detection.

In the next part, we will close the loop with this demo’s action outputs to try and leverage the strengths of VLAs in conjunction with low-level control ideas from parts 1 and 2.

If you liked this kind of analysis, please subscribe for future posts, and thanks for reading!

Wiener was a prodigious child, receiving a PhD by age 18 from Harvard, and becoming MIT mathematics faculty. By the account of Dark Hero of the Information Age, the biography by Flo Conway and Jim Siegelman, he was simultaneously one of the most intellectually alive and emotionally turbulent figures in twentieth-century science: touched by manic-depressive episodes and collegial feuds, yet capable of a mathematical breadth that few of his contemporaries could match.

That breadth is visible in the book he published in 1948: Cybernetics, or Control and Communication in the Animal and the Machine. Its thesis was that information flow and message-passing are central to control and communication in both animals and machines. It appeared the same year as Shannon’s “A Mathematical Theory of Communication” and the year before Shockley’s transistor paper. Wiener was at the center of the founding of the information age, and yet he has been largely forgotten in the recent technological development. His legacy was overshadowed by Shannon, who had the more implementable theory, and by von Neumann, who had the more implementable architecture.

Reading Cybernetics now, almost 80 years later, is awe-inspiring and unsettling in equal measure. It is mathematically dense in places and dated in others, but the program it laid out is strikingly relevant to modern AI development. Here are the ideas from it that I found most resonant.

1. Feedback

“Cybernetics” originates from the Ancient Greek κυβερνήτης (kybernētēs), meaning “steersman” — the same root that, via Latin, gave us the word “governor.” It is perhaps not coincidence that Maxwell’s paper on governors was the first known exposition on feedback control. I don’t need to elaborate on the value of feedback in modern technology, but two nontrivial leaps Wiener makes are worth highlighting.

First, he draws a connection between communication and control in neurology. The feedback loop (sense the error, apply a correction, repeat) describes voluntary movement in biological systems. When this feedback is damaged, as in cerebellar injury, the result is a tremor or oscillation: too aggressive a correction followed by too aggressive a counter-correction. This convergence of engineering control theory and neurology was a founding observation of cybernetics: the same mathematics governs servomechanisms and nervous systems.

The second leap is the identification of a fundamental tradeoff: do you invest in modeling or in feedback? Wiener’s answer depends on how constant and knowable your system is. He called systems that leverage explicit models compensators, contrasting them with pure feedback mechanisms. In today’s terms, Wiener’s compensator needs a world model: an internal representation of how the system behaves that allows action without waiting for error to accumulate. The model vs. feedback tradeoff he identified has strong echoes of the one playing out now in the debate between scaling-based and structured AI architectures, not to mention in robotics. Model-free reinforcement learning is a direct descendant of the feedback side of this tradeoff: an agent interacts with an environment, receives a reward signal reflecting the gap between its behavior and a desired outcome, and adjusts its policy accordingly.

2. Neuron structure: digital vs. analog

Wiener asks in the book: in what ways are the computational substrates of brains and machines alike, and in what ways are they fundamentally different?

Wiener’s first observation is that neurons obey an “all-or-none” law (they fire fully or not at all) and in this sense are digital. This is in tension with von Neumann’s later analysis, covered in a prior post: von Neumann argued that individual neurons function more like small analog computers, with temporal dynamics and nonlinear integration beyond what a simple threshold element can do. The understanding of neuronal computation has deepened considerably since both accounts, and the honest answer is that neurons are neither purely one nor the other.

What followed from the digital view, at least in the engineering tradition, was eventually deep learning: stack enough simple threshold units to sufficient depth, and powerful computation emerges. But as the next sections argue, Wiener himself was skeptical that the generic stacking of simple units was sufficient.

3. Neuron organization: flexible vs. dedicated

What Wiener does not dispute is that even if neurons are digital in their firing, their organization is anything but generic. He writes:

The structure of our visual cortex is too highly organized, too specific, to lead us to suppose that it operates by what is after all a highly generalized mechanism.

As a mathematician, he frames this in terms of group theory: the visual system is built to be invariant under transformations of position, rotation, scale, and illumination. Image recognition is comparison at the level of structural properties that persist across transformations, and not comparison of photoreceptor signals. The retina has broadly distributed and low-resolution rod cells and foveally-concentrated cones, and layers beyond it extracting features at multiple spatial frequencies in parallel. Structure encoded by biology is doing work from the very first stage.

The unifying point is that the brain does not apply a general-purpose function to raw sensory data and let structure emerge. It applies a pipeline in which each stage is specifically organized to extract the right kind of information. Most modern vision models posit that this structure will emerge from scale and data; capsule networks, group-equivariant CNNs etc. attempt to encode it explicitly but remain outside the mainstream. This is the same tension at the heart of the world models debate: whether sufficient scale applied to a general architecture will recover the structure that biology built in deliberately, or whether that structure ought to be encoded.

4. The switchboard analogy

Wiener is next interested in how neurons are organized, and here his analysis diverges sharply from the digital computer model he was comparing against.

A digital computer of his era had specific circuits for specific operations: an adder, a multiplier, a comparator, each doing one thing reliably and repeatedly. The brain, he argues, does not work this way. Rather than dedicated permanent circuits, the brain reconfigures its functional connections dynamically, routing signals through different pathways depending on context. He uses the telephone switchboard as his analogy: the same physical wires serve different conversations depending on how the exchange is configured at any moment.

He makes this concrete with a visual processing example depicted above: recognizing a letter regardless of its size (a large “A” and a small “A”) with a fixed array of photocells. His proposed solution uses a switchable connection layer between the photocell array (bottom) and a fixed set of processing elements (top). By selecting different connection patterns (the diagonal lines), photocell activations at different scales get mapped onto the same processing elements, achieving scale invariance through reconfigurable routing rather than through a learned function. In deep learning perception, this is similar to ideas like spatial pyramid pooling or adaptive pooling.

In contrast, a vision transformer applies the same operation at every layer to every token, with flexibility coming entirely from learned weights at massive scale. There is no dynamic routing or reconfiguration based on the nature of the input. Wiener pointed out that this approach carries a cost: a large fixed architecture must run in its entirety even when most of it is irrelevant to the current input. A 175B parameter model processing a simple query still activates the full machinery, paying the energy and latency cost of elements that contribute nothing to that particular computation.

Some modern work moves toward Wiener’s direction. Mixture-of-experts architectures route inputs to specialized sub-networks rather than running everything; sparse transformers use dynamic attention patterns; early-exit networks use only as much compute as the input requires. These remain the exception rather than the rule, but they are each, in a real sense, implementations of the switchboard principle Wiener described in 1948.

Subscribe now

5. Spatial efficiency: foveation

A thread running through Wiener’s treatment of vision is that the brain achieves capable perception not by processing everything uniformly and in parallel, but by being strategically non-uniform in both space and time.

The spatial side is foveation. The fovea provides high-resolution detail while the periphery offers broad, low-resolution motion detection. The brain doesn’t passively receive a full image, it actively steers the fovea toward informative regions via saccades, driven by a continuous feedback loop. The implication is that high-resolution processing is a scarce resource allocated dynamically, not applied uniformly.

6. Temporal efficiency: the television analogy

The temporal side is more surprising. Wiener observes that the brain may serialize what would otherwise require parallel hardware, using alpha waves (the ~10 Hz electrical rhythms visible in EEGs) as a scanning clock. Just as a television converts a two-dimensional image into a sequential stream by sweeping line by line, the brain may sweep through its representational space cyclically, interrogating stored patterns at each clock cycle. The efficiency principle is time-multiplexing: reuse the same hardware over time rather than duplicate it in space.

Together these describe a coherent alternative to the architecture modern AI has converged on. Transformers process all positions in a spatially uniform and temporally instantaneous manner, which is expensive in both compute and energy. Biology does neither: it allocates spatial resolution selectively and serializes computation over time. Foveation-inspired architectures (glimpse networks, recurrent attention models) and ideas like conditional computation point in this direction but remain outside the mainstream, largely because uniform dense operations map cleanly onto GPU hardware. Wiener’s architectural intuitions may become increasingly relevant if the AI energy crisis makes the efficiency argument more economically compelling.

7. Avoiding blunders: redundancy and verification

The brain produces behavior of remarkable precision despite individual neurons being surprisingly unreliable: they fire spontaneously, transmit probabilistically, and have far worse signal-to-noise ratios than transistors. Wiener’s answer, developed in the psychopathology chapter, is that there are two complementary strategies for error correction. The first is the “what I tell you three times is true” strategy: running two or three computing mechanisms simultaneously on the same problem, so that errors can be recognized by agreement across parallel channels. The second is backtracking: sequential verification where the system checks its own output and revises when something goes wrong. One is spatial (parallel redundancy), the other is temporal (serial correction) — the same tradeoff from before, now applied to reliability rather than perception.

This maps directly onto one of the most discussed failure modes in LLMs: hallucinations. Wiener suggests that they are the expected behavior of a system optimized for speed without redundancy or verification, not simply a quirk to be patched. A single forward pass through a transformer produces an answer with no mechanism for catching its own errors. Reasoning models which iterate, self-check, and backtrack are exploiting exactly the reliability/overhead tradeoff Wiener described:

But verification has limits. As I argued in the world models post, a system that lacks a grounded semantic model of the world can cross-check its outputs without ever catching the deeper class of errors that stem from not understanding what it’s talking about.

The human use of human beings

Wiener was clearly one of the founders of the information age, but he was also deeply worried about what was being built. A passage from his follow-up book The Human Use of Human Beings reads like something written last week:

The first industrial revolution was the devaluation of the human arm by the competition of machinery. The modern industrial revolution is similarly bound to devalue the human brain, at least in its simpler and more routine decisions. The average human being of mediocre attainments or less has nothing to sell that it is worth anyone’s money to buy.

He was not predicting this as an inevitable law of nature. His proposed answer was equally striking: rather than trying to preserve the market value of human labor artificially, he argued that society would need to restructure itself around non-market values like dignity, community, creativity, meaning. He wrote letters to labor unions warning them of what was coming, but he was not listened to.

In 2026, AI systems are starting to now inexpensively perform many of the cognitive tasks (writing, coding, analysis, translation, legal research) that defined middle-class professional employment in the twentieth century. The policy infrastructure to manage this transition does not exist. The urgency Wiener felt in 1950, when he had no working computer to point to, is more justified now.

Brian Christian, in his introduction to the recent reissue, calls Wiener “the progenitor of contemporary AI safety discourse.” That may be the most accurate short description of the man. He was not a pessimist or a technophobe — he was a technologist who had thought seriously about what he was building and felt obligated to say what it implied. That combination of technical depth, ethical seriousness, and willingness to deliver uncomfortable conclusions publicly is just one more reason to read and remember him.

I’m a strong proponent of reading and non-echo-chamber thinking. If you know of any other writing of this ilk, please let me know in the comments. If you liked this post, please share it, and subscribe!

Leave a comment

Share

This post draws primarily on “Cybernetics: Or Control and Communication in the Animal and the Machine” and the biography “Dark Hero of the Information Age”. It continues themes from previous posts on von Neumann and world models in AI.

When battery life is crucial, and when microseconds matter to ensure control loops remain stable, the hardware doesn’t care about elegant class hierarchy or beautiful code. The end-product of programming is data transformation, and not the code itself.

This post, written with my friend (software engineer and CS lecturer), started as a simple teaching example about Array-of-Structures vs Structure-of-Arrays (AoS vs. SoA) layouts. We thought we’d show a clean and universal performance curve demonstrating cache effects tied to C++ code.

We built what seemed like a straightforward benchmark: measure access time for different memory strides. What we didn’t anticipate was encountering five distinct issues spanning multiple abstraction layers—from compiler behavior to microarchitecture to hardware characteristics:

1. The compiler deleted our measurement code and unpredictably stored variables in memory vs. registers

2. The CPU’s pipeline hazards dominated our memory access time

3. The CPU’s dynamic frequency scaling skewed our results

4. The hardware prefetcher made our predictions wrong

5. Different processors gave wildly different results

This illustrates the gap between abstraction and performance. Programming languages provide abstraction above the hardware, but achieving good performance requires understanding how code executes on the underlying architecture. While some of our issues may be familiar to experienced programmers, others might be surprising even to veterans.

Four examples showing why you should care

There are many real-world examples using an "array of structures" organization for good reasons: it's faster to prototype, easier to reason about when objects manage their own state, and typically more readable for developers.

Example 1: PCL (Point Cloud Library) PointXYZRGB structure.

When you have pcl::PointCloud<PointXYZRGB> with millions of points, the memory layout looks like

[x0, y0, z0, pad, rgb0, x1, y1, z1, pad, rgb1, ...]

For an example task of filtering by distance (operating on the xyz only), we get 40% extra cache misses. For a color segmentation task operating on rgb only, 4x extra cache misses.

Example 2: Unity GameObject-Component System. GameObjects directly contain Component instances by value, e.g. a GameObject with Transform, Rigidbody, and Collider components stored as member data. This is classic AoS: each GameObject owns its component data, providing flexible composition but poor cache locality when iterating over many objects.

Example 3: Box2D (version 2.x). Each b2Body contains position, velocity, and force data as members (e.g. b2Vec2 m_linearVelocity). Most traditional object-oriented game engines before the ECS trend used composition with value semantics—each enemy/player/NPC object contained all its data directly. However, Box2D v3.0 (2024) moved away from this, now using handle-based IDs and storing body data separately for better performance.

Example 4: Humanoid joints. Last but not least, here is a practical example of a humanoid robot joint that should be quite relatable:

A Limb would be composed of Joints, with Joint specialized into different joint types.

Suppose the humanoid robot has 50 joints. Computing Jacobians requires accessing each joint’s position: 5.5kB loaded into cache (87 cache misses), when we only need 200 bytes (4 cache misses if organized as an array of positions).

Now that we have shown that this organization occurs commonly, we will dig in and try to measure the effect it has.

An even simpler example to dig into

We created an even simpler example with a single data array and a parameterized “stride” for a strided access pattern. This would occur in the example above with stride = sizeof(Joint). Our goal was to time how long it takes to access a fixed number of elements with different strides, as in the code below.

The actual code for replicating all these measurements, and more, is on github: feel free to try it out.

What we expect to see: Effectively, as we access data, the processor can load a segment from main memory into cache, in blocks.

Data:    |x| | | | |x| |...
          ← stride →
Cache:   |y|y|y|y|y|y|y|y|z|z|...
          ← line size  →

As stride increases, visiting the same number of elements requires caching more blocks. If memory movement dominates, we expect a linear rise in time as stride increases and more cache lines are touched. (More on what happens after each access hits a separate cache line below.)

Understanding the results from this "simple" example felt like peeling endless layers of an onion, but was very gratifying at the same time!

Issue 1: Controlling compiler optimizations

With the code snipped above, the assembler output showed:

testStride(unsigned long):
        ret
data:
        .zero   256000000

Of course! sink was being optimized out, and my firmware programming background caused me to add a volatile to its declaration. However, something in the asm output for the loop looked amiss. Can you spot it?

While data should be loaded from memory to a register, sink should be able to remain in a register. However, volatile forces it to be loaded and stored because the compiler must assume that it can be modified externally. So we get rid of volatile, and uncomment the last line:

if (sink == -1.0f) std::cout << "";

The new loop looks like

Comparing to the assembly above, the extra load-store are gone - first mystery solved.

Issue source: compiler / programming language

Issue 2: Data dependency hazard

The relevant part of the loop looked like this:

Timing this loop as we varied stride showed that for the first few strides, increasing stride had no effect on the time (solid lines in the plot below). With an Apple M2:

With the size of our loop, increasing stride definitely means more cache lines are touched, but it is making no difference. What’s going on?

Let’s look back at the assembly (same as the previous snippet).

If we manually unroll a few iterations, we have the following pattern:

FP add 2 cannot issue until FP add 1 has been committed, a classic Read-After-Write hazard. While a chip designer understands this very well, a programmer rarely needs to understand data dependency hazards in CPU pipelining. In this example, the float add dominates the effects from the load/store due to the data dependency and the long latency of floating-point add.

We add an unrolled version:

The unrolled version time is significantly smaller, as visible in the dashed lines in the plot above, and more importantly, now we see the linear rise we had predicted.

Issue source: microarchitecture, not visible in assembly instructions

Issue 3: Warmup effects

After root-causing issue 2, to avoid dealing with the unrolled loop, we changed the accumulate to a Read-Modify-Write. The time for each iteration is now longer because a load and store are required for each iteration, which should make data movement costs the dominating factor.

A number of stateful microarchitectural effects unrelated to the data cache contribute to performance characteristics, yet produce data cache-like behaviors. Such factors may include page table caching, page walk caching, prefetcher training, the memory controller, and even frequency ramping.

We attempted to stabilize the effect of these factors before running trials by running a warmup function at the beginning of the program. The warmup simply iterates over every element of data once to have the cache in a predictable state.

The results:

The warmup appears to universally make the program faster, irrespective of the stride (more pronounced effect on a different system in plots below). Our best guess is that the warmup ramping up the CPU frequency is the dominant effect. We also considered a trial for one stride affecting another, but running a single stride per run of the program didn’t yield clearer results (and took much longer).

Again, if you have any better ideas, we would love to know - please leave a comment!

Leave a comment

Issue source: microarchitecture / hardware

Issue 4: Initial no-effect; second slope after cache line boundary

4a) In the previous plot, there is an initial part till about stride ~5 (20 bytes) where we predicted a linear rise, but instead see no effect of stride on timing.

While we are not sure, this is likely due to hardware prefetching: Modern CPUs have hardware prefetchers that detect sequential / strided access patterns and automatically fetch data ahead of time. Once the stride grows large enough (~20-64 bytes), the prefetcher can no longer keep up—either because it can’t fetch far enough ahead, or because the access pattern becomes too sparse for it to predict. At this point, we finally see the expected linear increase as each access genuinely waits for data from main memory.

4b) We expected the access time to plateau after each access was already hitting a different cache line. However, there appears to be a slower rise after the cache line boundary at least on the Apple M2 processor

Some (unconfirmed) hypotheses for the slower rise after the boundary:

• L1 → L2 spilling if the working set exceeds L1 capacity, incorporating L2 access times

• TLB misses as large strides access many different memory pages

Issue sources: microarchitecture / hardware

Issue 5: Different behavior on different processors

Throughout uncovering the previous issues, we ran a few tests on other processors, and unfortunately that only served to increase the number of unknowns. In this section we will show you some of those results, but only be able to speculate about what causes them.

With an AMD Zen5 processor:

We see a plateau after an initial rise, which matches our naive prediction.

However, we observe a peak around 32 floats (128 bytes) followed by a drop. We don’t have an explanation for this behavior, which may be to do with advanced prefetcher behavior. In other words, the hardware may be making assumptions about our access pattern, and stride = 64-128 bytes hits the worst-case scenario where those assumptions fail. If you have any ideas about the cause, let us know in the comments!

Leave a comment

We also tested on an Intel processor on Windows, which confirmed that some of the strangest aspects of the two plots above are to do with AMD, and not the compiler.

This resembles our Apple M2 plots more closely, including the slower rise after the cache line boundary. It also adds an even slower rise after 2x the cache line boundary.

Issue sources: secret microarchitectural optimizations

Back to programming

Through this journey, it is safe to say we learned a lot about the complexity of modern processors. Fortunately, though, our central point about the (initial, then plateauing) rise of access time with stride still stands as universally true. Phew!

How do we utilize this knowledge as a programmer? The key is to ensure that commonly-accessed data is packed tightly in contiguous memory.

The naive OOP concept of owning data:

• The class directly contains/owns the data as member variables

• Example: Joint class with float sensed_position (and other things) embedded in it

• This creates the AoS memory layout problem

Instead store indices. In the literature on data-oriented design, this is sometimes called: Entity-Component-System (ECS) pattern, or data-oriented design with handles.

• The class contains references, pointers, or indices to data stored elsewhere

• This allows you to keep polymorphism while avoiding AoS layout issues

It isn’t object-oriented vs. polymorphism. Just to reiterate that data-oriented is not opposed to OOP conveniences, consider that Pinocchio uses polymorphism to specialize functions, but stores indices to the vectors, not the data itself. The actual positions and velocities live in contiguous arrays, giving cache-friendly SoA layout, while the polymorphic joint models provide the OOP interface. You can have the benefits of polymorphism (different joint types with specialized behavior) without the memory layout problems of AoS. This is the middle ground between pure OOP with composition and abandoning OOP entirely for data-oriented design.

Closing thoughts

In this post, we first showed how OOP-thinking can naturally lead to suboptimal cache usage, with several real examples. Then we looked at the effects this can have, uncovering many interesting “side-quest” root-causing exercises.

It isn’t coincidence that modern performance-critical systems say no to naive composed OOP:

• Machine learning libraries will often select the data layout (NCHW etc.) transparently, optimizing for cache locality.

• Pinocchio, a robotics kinematics / dynamics library, has its functions access array data.

• Drake, a larger robotics-oriented library, eventually has data in arrays below abstraction layers.

• Unity DOTS stores all Transform data in packed arrays, not in GameObjects.

• Box2D v3.0 switched from OOP bodies to ID-based handles with SoA storage.

• Unreal Mass Entity is an ECS system for high-object-count scenarios.

Even if in an isolated example the performance gain seems small, these patterns occur so frequently that they can add up to large losses that are difficult to eliminate.

Thanks for reading! If you enjoyed this kind of full-stack analysis and root-causing, please share and subscribe for more posts on robotics, AI, and computing.

Share

References and further reading

• Code for demonstrations in this post, and more on github

• “Better memory representation” in Jeff Dean’s “Performance Hints”

• “Efficiency with Algorithms, Performance with Data Structures” - Chandler Carruth [CppCon 2014]. Note: I don’t fully agree with the statement (10:45) that “efficiency is only affected by algorithms” - a good example is the energetic cost of moving a byte from DRAM -> core being significantly higher than from L1, meaning the same algorithm with poor cache performance actually consumes more energy, in addition to completing slower.

• Data-Oriented Design and C++ - Mike Acton [CppCon 2014]

• Explicit cache control via software prefetching

1. Architecture of end-to-end: learning → control

2. This article

3. Dissecting a VLA

4. Closing the action loop with a VLM “agent”

5. Demo combining the best features of end-to-end and classical approaches

Last week, I wrote about modern end-to-end robotics pipelines; why this is the new north star, and the hidden architecture behind successful implementations. Part 1 reviewed some implementations showing signs of a cascade of a high-level (HL) → low-level (LL) controller in the actuation end of the pipeline:

I have first-hand experience demonstrating walking robots to customers in a sandy desert, and as the robots slipped they asked, “is it learning?” With the prior of people adapting their gait as they walk on ice (for example), a reasonable expectation is that an isolated robot can adjust its behavior after some trial & error or adaptation period.

However, this is not how naive foundation-model end-to-end pipelines (such as those covered in part 1) work today; a particular robot can only change its behavior once the “hive brain” is updated with new data in its training. Due to the size of these models, it is impractical that training happens on-device or frequently.

So, in part 2, we ask: how can a fielded robot adapt to unexpected conditions? Why do we even need adaptability? Given the HL → LL controller cascade structure in modern end-to-end pipelines from part 1, where does this adaptability live, and how does it affect the mapping to computing hardware? Lastly, we will also look at some published implementations and see how they approach or ignore this issue.

Updates to part 1, hot off the presses

Before we dig into part 2, I need to add a couple of updates from relevant news releases that I wasn’t able to review before part 1 was published (Jan 26):

1. Microsoft’s Rho-Alpha model announcement with commentary on Tech talks (Jan 24) reveals “split architecture” including dedicated low-level controller, underscoring at least two points in the part 1 post. (a) Tactile and proprioceptive information is incorporated in the action expert, showing that the action head facilitates feedback loops; (b) higher control bandwidth via so-called bypass mechanism. Quoting the post, “The long-term goal, Kolobov said, is to have the action expert or a part of it operate on proprioception and physical sensing modalities at a significantly higher frequency than on visual and language data.”

2. Figure Helix 02 Jan 27 update reveals new “System 0” controller, underscoring at least four points in the part 1 post. The “system 0” implementation is described as a dedicated whole-body controller (WBC), which conventionally converts desired accelerations or velocities to joint torques based on a model of the robot. (a) S1 went from controlling the upper body to the whole body, and this reduced the overall system complexity by separating concerns; (b) S0 and S1 incorporate tactile data in tighter feedback loops, without adding complexity to the large VLM S2; (c) S0 runs at a KHz rate increasing the last-level control bandwidth; (d) it is trained for that specific robot (vs. cross-embodiment), localizing robot body-related parameters in one place (and presumably enabling generalization of S2/S1 to a different robot). The purpose of the WBC is similar to the model-based reference in part 1, but the difference here is that it is also a neural network trained from data.1

I expect that we will continue to see further evidence and refinement of hierarchical control structures in commercial robots, vs. unstructured end-to-end pipelines. Make sure to subscribe to get future updates:

This publication and this post contain the author’s personal thoughts and opinions only, and do not reflect the views of any companies or institutions.

Why do we need adaptability?

When a robot leaves the lab and is in customers’ hands, it will at some point inevitably be subjected to an unexpected operating condition, stemming from component failure, perturbation, environmental condition, or operating condition (e.g. payload). To address this, one recourse is to build a large-enough model that has enough experience to handle all these situations (i.e. domain randomization, multi-embodiment, etc.). This of course takes (much) more data and more training, as OpenAI showed from their dexterity result in 2019:

The other option is adaptation in a strategic part of the pipeline to address as many of these variations as possible. In this post, we are focused on the action end of the pipeline, and the classes of variation we are interested in include variability in joints / motors (friction, motor torque), terrain properties, payload.

Let’s clarify the timescale hierarchy, because the word “adaptation” can refer to changes at various timescales. Within-movement corrections can happen in milliseconds, and is typically part of reactive control within the low-level controller. Skill acquisition across many tasks using large datasets during training will typically happen offline. The intermediate adjustment occurring in the seconds-to-minutes timescale, which we refer to as motor adaptation, is the focus of this post.

Historical context from biology, control theory, and LLMs

Cerebellar timescales (seconds to minutes) match closely with the motor adaptation timescale referred to above, and several research efforts identify its role in adaptation of behavior in that time range.

Morton (2006) further associates the cerebellum with motor adaptation and the spinal column to reactive control:

Cerebellar damage does not impair the ability to make reactive feedback-driven motor adaptations, but significantly disrupts predictive feedforward motor adaptations during splitbelt treadmill locomotion … The cerebellum seems to play an essential role in predictive but not reactive locomotor adjustments. We postulate that reactive adjustments may instead be predominantly controlled by lower neural centers, such as the spinal cord or brainstem.

In control theory, there is a long tradition of adaptive control and model-reference adaptive control (MRAC) which utilize a (model-based) adjustment mechanism to modify the parameters of the controller.

The arrows in the figure above reveal a degree of interconnectedness beyond the cascade connections we primarily reviewed in part 1. The adjustment mechanism can also act in discrete steps instead of continuously, or without a model, in which cases it is called “gain scheduling”.

Self-improving learning systems are beginning to appear in the news more frequently in the LLM world: Ilya Sutskever said in Nov 2025, “There has been one big idea that everyone has been locked into, which is the self-improving AI”. The aforementioned Rho-Alpha model has an ability to update weights while running using teleoperation feedback. However, this can lead to a common side-effect called “catastrophic forgetting” due to all weights being in one huge monolithic structure, and so updates needed to be made either in judicious layers or in careful batches.

Motor adaptation in practice

One advantage in robotics pipelines is that they may (as we saw in part 1) have a hierarchical HL → LL structure. In such a situation, there are motor adaptations that can be integrated the LL controller without impacting the behavior of the HL controller, sidestepping the catastrophic forgetting issue.

I’ll go over a few illustrative examples, and especially discuss their ability to handle unexpected conditions. If I missed an idea that is pertinent and relevant, let me know in the comments:

Leave a comment

Adaptation in model-based LL control: robot arms, drones, humanoids

The pre-”end-to-end” era had many examples of adaptation in practice. Old ideas such as MRAC show up in industrial and commercial manipulators, such as in the payload estimation feature in Universal Robots arms. Commercial drones estimate wind to remain stable, sometimes using neural networks. In a 2019 HyQ paper, an explicit terrain compliance estimation module estimates parameters used by the LL controller. In a 2023 demonstration of the Atlas robot using model-based controllers while picking up heavy objects, Atlas “has access to the mass properties” of the object it is picking up, which I would lump into a gain-scheduling type of approach.

In all these examples, because the LL controller is model-based, it is easier to adapt for quantities like payload mass because it is clear where those terms appear in the controller. This is an advantage of having physically interpretable parameters, compared to black-box latent-space interconnections.

Mapping to computation:

HL → WBC inverse dynamics/QP (CPU) → Joint/servo controllers (microcontroller/CPU) → Torques
                ↑
    *Adjustment mechanism (CPU/GPU)*

Meta-learning for adapting among training environments

The concept of meta-learning (2019) is targeted at the motor adaptation problem, but needs samples over environments during training. This leads to the aforementioned prolonged training and large models, as well as susceptibility to truly unexpected (out-of-distribution) conditions. The authors of the paper are among the founders of Physical Intelligence, so it is possible that they could institute meta-learning-type methods for online adaptation in their action expert (not the case today as far as I can tell).

Mapping to computation of this hypothetical scenario:

VLM (GPU) → Action expert *with internal model and meta-learning* (GPU/CPU) → Trajectory tracking (CPU) → Torques

Learning-based latent parameter estimation for locomotion

As my sand locomotion example above might hint at, unexpected payload and terrain conditions are particularly prevalent in locomotion.

RMA: Rapid Motor Adaptation (2021) introduces a dedicated adaptation module that predicts a set of “latent parameters” that can adjust the action policy to better suit different conditions. These varied conditions are trained by randomizing in simulation, potentially suffering from a few of the same issues with out-of-distribution encounters and training difficulty.

One of the authors founded Skild.AI, and this quote from their Aug 2025 blog post

A striking aspect of our model is that it is not just robust, but it is also adaptive and graceful

(emphasis theirs) suggests incorporation of something like RMA. Absent too many details, here is my best guess of the composed pipeline mapped to computational hardware:

HL action policy (GPU) → *Adaptation module (GPU)* → LL action policy (GPU) → Torques

Where RMA had a large-ish latent vector, there are similar approaches toward predicting parameters with more physical meaning, from a reduced or a full state estimate. These state-estimation networks concurrently learn base state and contact probabilities alongside policy, enabling better perception of ground interactions.

The end-result pipeline is quite similar, just potentially decomposing the adaptation module a bit:

HL command → *History encoder (GPU) → Estimator (GPU)* → Actor (GPU) → Impedance control (CPU) → Torques

In-context learning to fix recent mistakes

A different method called in-context learning (appearing in Covariant.AI’s Mar 2024 blog post, and in the RICL method from Aug 2025) attends to recent action history as opposed to encoded observation history. These relevant demonstrations are added to the VLA context before its forward pass.

The end-result pipeline adds retrieval buffer of demonstrations before the VLA, and an interpolation unit after the action module:

*Retrieval buffer* → VLM (GPU) → Action expert (GPU/CPU) → *Action interpolation (CPU/GPU)* → Trajectory tracking (CPU) → Torques

This method is in a slightly different category, where relevant demonstrations need to occur and be reflected upon to adapt, compared to the potentially faster adaptation enabled by the previous methods. This strategy would not be sensible for time-sensitive or safety-critical tasks, but is categorically different and seemed worth reviewing.

Closing thoughts

In part 2 of this article series reviewing modern end-to-end robotics pipelines, we discussed why it may be useful to have some adaptation capability for fielded robots to handle unexpected conditions, and some examples of how it can be implemented. We also discussed some historical context from biology and control theory.

In part 3, we will try to get more hands-on and utilize what we learned from the first two parts to build up an effective pipeline from scratch. I’m still debating whether to use existing tools such as Isaac sim or build even more from first principles for clarity, so it may take some time before we get there. If you have any suggestions or feedback, let me know in the comments. If you found this article interesting, please share and subscribe for future posts. Thanks for reading!

Share

Leave a comment

1. This article

2. Online motor adaptation

3. Dissecting a VLA

4. Closing the action loop with a VLM “agent”

5. Demo combining the best features of end-to-end and classical approaches

Recent progress and excitement in humanoid robotics are largely driven by rapid gains in generalist capabilities. Historically, most robots were engineered for narrow, well-defined tasks. The current wave of companies, in contrast, is pursuing systems intended to operate across a broad range of activities, shifting both public and economic expectations toward robots that can serve as general-purpose physical agents.

A central part of this shift is the widespread claim of end-to-end pipelines, often described as going from “pixels to actions,” in contrast to earlier approaches built from hand-designed perception, planning, and control modules. This post examines what “end-to-end” means in practice: where the pipeline actually begins and ends, the tradeoffs between different architectural choices, and how the algorithms map to computing hardware.

Part 1 focuses on the “actions” side of “pixels to actions”: how learned systems interface with the physical control of the robot body. Part 2 will examine how these architectures adapt to environmental uncertainty and contact-rich interaction. Later parts will include hands-on comparisons using small standalone examples to make these differences concrete.

This publication and this post contain the author’s personal thoughts and opinions only, and do not reflect the views of any companies or institutions.

Why “end-to-end”

Classical AI was built up from a strict idea of separation of sensing, planning, and action. To my knowledge, the first robot to embody Sense-Plan-Act was Shakey the robot (~1970), which also employed one of the first symbolic AI systems. This tiered structure was so formative to robotics research that most research labs today are dedicated to different portions of this hierarchy, such as “perception”, “planning”, or “locomotion”.

The sense-plan-act view today is dying a very rapid death. The modern narrative of general-purpose robotics holds that modular pipelines often fail because of limitations imposed by this decoupling; for example, perception errors break planners, planners produce infeasible motions, and most importantly, interfaces encode wrong assumptions.

As influencial AI researcher Sergey Levine puts it,

for any learning-enabled system, any component that is not learned but instead designed by hand will eventually become the bottleneck to its performance

End-to-end training avoids hand-designed intermediate representations, manually tuned cost functions, and any bottlenecks imposed by module interfaces.

Additionally, “end-to-end” sends a sociological signal to do with modern AI foundation-model alignment, scalability with data, and positions the company as an AI lab instead of a controls shop.

The action end in practice

The practical reality of “end-to-end” is more subtle than it might seem. In this section we’ll review what some published academic and commercial implementations actually appear to be doing today, and also try to outline how the implementation is mapped to computational hardware.

The old way: model-based stacks (~2014)

It is very common to have a whole-body controller at the low-level, as exemplified by the 2014 MIT Atlas team’s report. After a high-level plan is created, a tracking controller is implemented as a quadratic program, and that generates the signals sent to the actuators:

Mapping to computational hardware:

Trajectory optimizer (CPU) → WBC inverse dynamics/QP (CPU) → Joint/servo controllers (microcontroller/CPU) → Torques

Learning followed by impedance controller (~2017-2020)

To my knowledge, the first fielded robots using learning-based locomotion controllers appeared ~2018 from Google (using Minitaur) and in Marco Hutter’s group. As documented in the 2018 paper from Google and the highly-cited Hwangbo et al (2019) paper, the most effective choice of action space was an impedance controller in turn influenced by Peng et al (2017):

Our experiments suggest that action parameterizations that include basic local feedback, such as PD target angles, MTU activations, or target velocities, can improve policy performance and learning speed across different motions and character morphologies

The policy outputs desired joint positions and sometimes velocity offsets or gain modulation, and the torque applied is a simple algebraic equation:

The virtue of this architecture is that it is very generic, and succeeds in decoupling the fast time-scales and discontinuities of making and breaking contact from the learning algorithm.

Mapping to computational hardware:

Policy eval (CPU/embedded GPU) → Impedance controller (CPU) → Actuators

Figure AI’s “System 1” policy (2025)

Figure AI’s Feb 2025 blog post describes a “System 2 / System 1” design where a high-level vision-language model (S2) reasons about goals and semantics at low frequency, and a fast visuomotor network (S1) executes continuous control at high frequency. While this reflects a separation of timescales and roles, both modules are trained end-to-end with an abstract latent interface, meaning there is not a principled, physically interpretable handoff between high-level strategy and low-level control. As a result, Helix achieves generalization in perception and task reasoning but does not isolate physical control concerns (such as dynamics stabilization, contact interaction, or actuation abstraction) into structured model-based or classical control modules.

In a Mar 2025 blog post, they describe what sounds more like the impedance controller above than the system 1 design, so it’s possible some combination of both architectures is utilized:

We additionally run the policy output through kHz-rate closed-loop torque control to compensate for errors in actuator modeling

Mapping to computational hardware:

System 2 (Transformer, GPU) → System 1 (Network, GPU) → [Impedance control (CPU)] → Torques

Physical Intelligence’s action expert (2025)

The architecture described is similar to the system 1 above, but specifically suggests that the end-to-end training causes problems:

When adapting a VLM to a VLA in this action expert design, the VLM backbone representations are exposed to the gradients from the action expert. Our experiments show that those gradients from the action expert lead to unfavorable learning dynamics, which not only results in much slower learning, but also causes the VLM backbone to lose some of the knowledge acquired during web-scale pre-training.

This is conceptually analogous to known problems like vanishing/exploding gradients in deep nets, where lower layers dominate or drown out meaningful gradients for higher layers.

Another blog post describes issues to do with the mismatched control bandwidth of foundation model output to robot dynamics, solved by outputting short horizon trajectories that are played out by a low-level controller.

Mapping to computation:

VLM (GPU) → Action expert (GPU/CPU) → Trajectory tracking (CPU) → Torques

Boston Dynamics + TRI’s pose tracking (2025)

Their blog post describes an architecture with the higher-level cognitive layer outputs joint positions and end-effector poses. While there isn’t an explicit decription of how these position setpoints are tracked, the post mentions Atlas’s MPC, and it is reasonable to assume that that is the lower-level controller.

Mapping to computational hardware:

LBM inference (GPU) → MPC (CPU) → Actuator torques

1X’s inverse dynamics model IDM (2026)

1X also describes a hierarchy in their blog post:

World Model Backbone (WM): A text-conditioned video prediction model trained on internet-scale video data and fine-tuned on robot sensorimotor data. It predicts future visual states based on current observations and candidate actions.

Inverse Dynamics Model (IDM): Converts predicted future states into feasible robot action sequences that will produce those outcomes in the real world. The use of the term “inverse dynamics” suggests that the output actions are torques, though that isn’t specified.

Mapping to computational hardware:

World model (GPU) → IDM (GPU) → Actuator torques

Why not end-to-end

From the previous section, it is apparent that “end-to-end” doesn’t usually mean that a single algorithm or network is going from pixels to torques. In this section, we’ll try to list some potential intuitive reasons for this.

Separation of concerns

We saw above on Physical Intelligence’s blog post that there are difficulties in training an end-to-end policy that does so many different things. Another quote:

One hypothesis of why this is happening is the following. A pre-trained VLM, by its nature, pays attention to language inputs well. The gradients from the action expert now severly interfere with the model’s ability to process language, which leads the model to pick up on other correlations first.

These problems are a side-effect of one network trying to solve a lot of different problems. The old Sense-Plan-Act schema enforced a separation of concerns very strictly, but even with a more relaxed architecture, low-level control priors drastically reduce the policy search space.

A human nervous exhibits similar separation with a cortex (goal-directed commands), cerebellum (fast adaptation, prediction), spinal reflexes (fast control loops), and even mechanical impedance control in muscles / tendons.

Training complexity

Related to the separation of concerns above, an end-to-end network must learn contact mechanics, actuator dynamics, delays, friction, impact stabilization, as well as task-level planning, all in one gradient signal.

This creates extremely long credit chains and high sample complexity. Hierarchical control factorizes the learning problem.

Feedback control loops; tactile and force feedback

With a fully end-to-end system, any feedback on how the executing is going can only come in at the top. In contrast, a dedicated low-level control unit can run its own feedback controller that performs stabilization functions. This is in effect what we saw above with the selection of the impedance controller in the Peng and Hutter papers above.

Secondly, a low-level controller also provides a great opportunity to incorporate a rich set of sensory signals such as tactile and force feedback information. Rodney Brooks underlines the importance of non-visual feedback in his Sep 2025 essay, going as far as to flag it as a roadblock. The problem is, if you must have force feedback in an end-to-end model, you first have to contend with the lack of large-scale force data to train it from, as well as the much larger end-to-end model you now have to train and evaluate at inference-time. As I responded to a Substack comment here, a low-level control unit is a potential way that that data could be incorporated, without increasing the dimensionality of the higher-level brain.

Control bandwidth

Real-world physics and dynamics don’t wait for end-to-end inference to complete, and most implementations (Physical Intelligence’s action chunking, Figure’s rate-decoupled system 1, etc.) need to decouple the control bandwidth of the cognitive layer from the low-level controller.

talks about this aspect as an action inference limitation in his excellent post about VLA’s which you should read if you haven’t.

Sim2real transfer

As discussed in my recent world models post, almost all these implementations that utilize large-scale demonstration data need to follow it up with reinforcement learning post-training in simulation. This surfaces an issue that has been named “sim2real transfer,” where the simulator’s accuracy can limit the deployed behavior. This has a number of solutions including domain randomization and actuator networks, but alternatively, having a low-level controller can in many cases absorb modeling error with their inverse dynamics functionality. Physics errors affect torque-level policies massively, but impedance control, whole-body control, or model-predictive control absorb modeling error by actively driving mismatch errors to zero.

Safety constraints

We can explicitly add torque constraints, joint kinematic limits, self-collision avoidance, to a low-level controller. This is intuitively true, but I’ll leave an example of a recent research paper which found out exactly this. Quoting the author:

Introducing UMI-FT: the UMI gripper equipped with force/torque sensors (CoinFT) on each finger. Multimodal data from UMI-FT, combined with diffusion policy and compliance control, enables robots to apply sufficient yet safe force for task completion.

Generalization across hardware embodiment (*maybe)

In principle, if the low-level controller completely abstracts the hardware, the higher-level brain’s functionality can be kept the same with different embodiments. Intuitively, you can reuse high-level policies if low-level layers abstract hardware, and you can improve low-level stability without retraining ML.

However, this intuitive point is difficult to verify due to the methodology of how the cognitive models are developed today. The end-to-end pixel → action policies always incorporate some amount of information about the embodiment, so it isn’t possible to train an abstract cognitive model. In practice, the foundation models of today train on cross-embodiment data to obtain generalizable knowledge. To get to the bottom of this facet, we would need to understand what constitutes a cognitive model separate from embodiment, and that is not known yet as discussed in my previous world models post:

Closing thoughts

With the end of Sense-Plan-Act, the new robotics north star is an end-to-end pipeline that does away with the need for any task-specific pipeline architecture or programming. However, today’s successful implementations tell a different story, and there are a number of intuitive reasons for this.

Foundation models excel at semantic, perceptual, and strategic reasoning, but they are mismatched to high-bandwidth, stability-critical motor control. A robust robotic architecture separates concerns into layers aligned with physical timescales and modeling regimes.

In this (part 1) article, we focused on standard visuomotor task execution. In part 2 of this series, we’ll look at how unexpected events and motor adaptation are handled in these architectures. After that, to continue this series, I’d also like to explore a standalone demonstration that can be published as an open-source repo that examines a few of these architectures and compares them fairly.

If you found this post interesting, please let me know in the comments, and share, and subscribe. Thanks for reading!

Leave a comment

Share

I’ve known of von Neumann’s huge legacy on modern computing from a college computer organization course, but I was stunned at how much he was able to extrapolate into ideas about computation in general. His writings, from before the first transistor-based computer was built, are ever-relevant after 70 years of exponential growth in computing technology. He wasn’t correct about everything—that would be impossible—but the ways in which he was wrong are even more revealing and thought-provoking. They anticipate the reason deep learning has been so capable, and also predict the architectural limits we are now running into—memory bottlenecks, brute-force scale, and energy-hungry intelligence. They also anticipate the future directions we can go in to overcome these deficiencies.

The book is very short and absolutely worth a read if you can pick it up from a library or used bookstore, but I had four broad and powerful takeaways that contextualized decades of development for me.

1. Scale & memory: Basic operations force massive memory movement

2. Precise vs. statistical: Deep learning (DL) escapes numerical fragility by becoming brain-like

3. Depth vs. architecture: DL substitutes scale for structural sophistication in the brain

4. Representation & substrate: DL is rigid where the brain is fluid

I’ll explain these four aspects below, but together they point to the same overall thesis:

Modern AI succeeded by replicating the statistical aspect of natural computation, but suffers from brute-force scaling inside an architecture that von Neumann already suspected was fundamentally mismatched to cognition.

1. Scale & memory

As the book says, the principle of “one organ for each basic operation” necessitates memory for intermediate values, on top of instruction and data memory. Von Neumann predicted that computation systems built from simple primitives can only scale by also scaling memory.

Scalar CPU architecture is still very close to von Neumann’s artificial automaton. Post-von-Neumann architectures include systolic arrays (TPUs) and near-memory compute; GPUs are a bit of a hybrid with shared memory (scratchpads), and tiled matrix multiply (data reuse). Even heavily optimized post-von-Neumann machines are still dominated by data movement, because the algorithmic structure forces it.

Modern deep learning vindicates this: intelligence is achieved not through complex operations but through scale, which makes memory movement, not computation, the central bottleneck of contemporary hardware. We have been talking about the AI memory wall for a few years, but it was inevitable from these predictions 70 years ago.

A related aspect which von Neumann couldn’t have anticipated was the energetic impact of memory access. He did write about the energetic cost of logic operations, but today, moving 1 bit from DRAM costs more energy than 1 FLOP, to the tune of 100× a multiply. This pressure is driving technology development in near-memory compute, in-memory analog MACs, optical interconnects. The same architectural tension von Neumann identified today drives the economics1 of AI hardware.

2. Precise vs. statistical

One of the central topics in the book is how digital computation needs very high precision because of the high arithmetic depth of repeated basic operations. If each operation has error ε, after N steps you expect error ≈ O(Nε). Deep networks have extreme arithmetic depth with thousands of layers and trillions of operations. However, empirically, 4-bit quantization in deep learning works with nearly no drop in accuracy.

Why doesn’t error compound the way von Neumann predicted?

The key: von Neumann analyzed precise numerical methods (like solving equations, integrating trajectories), but neural networks are different in a couple of important ways:

1. Noise is inherent in the training process, resulting in a function approximator with inherent robustness to input noise.

2. In the accumulation function, the errors are mixed across thousands of dimensions, clipped by nonlinear saturating functions, and averaged out statistically.

The overall error is clamped and damped, and does not propagate in the same way that von Neumann assumed.

Relatedly, von Neumann argued that the brain works with low precision (1-10 bits), and performs a different type of computation than digital computers (32-64 bits). He referred to the brain as performing “statistical computing”. So deep learning is not violating von Neumann, but it is occupying the biological side of his dichotomy.

3. Depth vs. architecture

Von Neumann emphasizes three biological facts about neurons:

1. Low precision

2. Low speed (~10 ms per spike, though they can respond slightly faster under extreme stimulation)

3. Shallow circuits

We discussed the precision above; let’s dig into the others next. The nervous system is very slow, with each “layer” taking on the order of 10 ms to fire and reset (compared to digital lines changing state in < 1 ns). This means that while it is feasible to have a “deep” digital computation, that would be infeasible in a natural system.

The shallowness is also important: a crucial example in the book is that the retina does significant computation using three synapse layers, which is orders of magnitude smaller than is needed for modern Vision Transformer (ViT) encoders (hundreds of layers, billions of parameters).

How is this possible? The answer is that a biological neuron is not a basic linear unit + nonlinearity; it is more like a small analog computer. Each neuron has temporal dynamics, neuromodulators, and plasticity rules. Its connections are even more complex: each has hundreds of synapses, nonlinear integration of activations with potential spatial and geometric relations.

So the contrast is stark:

• The brain has shallow compositions of slow and low-precision units

• Deep nets have very deep compositions of very fast and medium-low-precision units

Von Neumann predicted that these fundamental differences in the basic blocks would result in different natural vs. artificial computing paradigms:

Hence the logical approach and structure in natural automata may be expected to differ widely from those in artificial automata.

Modern deep learning compensates for architectural simplicity with scale. Biology compensates for slow, noisy hardware with architectural sophistication and better primitives. This distinction strongly influences why our systems are large, power-hungry, data-hungry, and memory bound.

I hadn’t anticipated this connection when I started reading the book, but my article from last week also visits this architectural distinction from a world-model-representation perspective:

Why did we choose scaling of simple units for computing? Among other reasons (as discussed in the previous article), deep learning was built around universality + scalability, not biological realism. Simple units have advantages: easy to parallelize, easy to implement on GPUs, and easy to map to silicon.

What does this mean for general artificial intelligence? Von Neumann suspected that digital logic gates were too primitive to model cognition efficiently, and with today’s technology it is certainly true that the brain’s performance at 10W cannot be matched even at much higher power.

4. Representation & substrate

Von Neumann observes that representations of quantities which go through the nervous system may change from digital to analog and vice versa repeatedly. They can also have adaptive precision representations2. In contrast, digital machines commit very early to fixed-width numbers everywhere (FP32, FP16, INT8, etc.), and even “mixed precision” is coarse and static.

These points seem to suggest another architectural dichotomy (not just the connections between units, but also in how numerical quantities are represented). The brain has adaptive primitives, precisions, and numerical representations, whereas they are all fixed in the digital computing paradigm.

Is the answer analog computing? Von Neumann himself rejected naive analog computing due to its problems of scalability and reliability. The brain may be powerful while being efficient because it is representationally flexible, not because it is analog per se.

Neuromorphic computing is exactly about this axis, with conceptual departures such as event-driven computation, mixed analog/digital circuits, co-located computation and memory. My knowledge of the field is limited and I am not sure that any of the existing research in that area truly captures what von Neumann was hinting at, but I suspect that in the long-term future of this publication, neuromorphic computing will come up again.

Thanks for reading! Let me know if you’d suggest any related historical or modern writing on this topic, and please share and subscribe if you liked the essay.

Share

In parallel with these debates, especially on the heels of numerous announcements at CES 2026, the cultural focus is increasingly driving toward robotics or “physical AI”; is there a physical equivalent to this intellectual debate between scaling and structured models?

Here, we’ll try to go over some of the key aspects of this intellectual and conceptual spectrum starting with the informational world, and examine the implications of the equivalent schools of thought in the physical world.

This publication and this post contain the author’s personal thoughts and opinions only, and do not reflect the views of any companies or institutions.

Today’s AI is a product of scaling a simple architecture (mostly)

Breaking down this heading, by “today’s AI,” I’m referring to the most pervasive products, such as chatbots, search, coding and writing assistants. These systems are typically based on large transformer architectures composed of many repeated layers and trained on vast datasets, with models today having hunders of billions of parameters.1 In simplified form, these systems operate by mapping input tokens into embeddings, processing them through a stack of transformer blocks, and producing probability distributions over possible next tokens via a final linear projection and softmax layer.

Since the initial release of ChatGPT, the dominant trend in the development of these models has been to increase their size and the amount of data used for training, rather than to introduce fundamentally new architectural principles.

Given this architectural simplicity, the range of capability expressed by LLM-based tools is frankly impressive. Much of this capability therefore arises from the interaction between large model size and extensive training data, rather than from task-specific design and bespoke computational structures.

Sam Altman’s early 2025 blog post and the empirical observations of companies building LLMs added on evidence and expectation of continued scaling of intelligence this way. These observations led to a “scale is all you need” movement that has had enormous impact on our society and economy, with $1.7 trillion of projected investment by 2030.

The larger debate we’re looking at in this post is about the prediction that scale is sufficient (more below), but it is also important to ask if it is necessary. I.e. is scale required to exhibit the same progress? The answer to this is likely yes; as stated in a Dec 2025 preprint by Quattrociocchi et al, when the models are restricted to the transformer architecture described above, it appears to be true that “their apparent intelligence emerges only under conditions of massive scale”.

Another natural question is why there has been so much investment into exploiting scaling, vs. exploration of other architectures. The first is that progress is consistent and predictable (even suggesting scaling “laws” as in the Altman blog post) which enable predictable engineering and financial projections. Innovation and development of new architectures is a relatively unpredictable and risky process. Another very prominent virtue is that simple architectures are much easier for collaboration with other parts of the engineering stack, and has been key for the adoption of hardware acceleration for deep learning.

Many leading researchers such as Demis Hassabis, Geoffrey Hinton, and teams at OpenAI and Anthropic maintain that scaling remains a primary driver of progress.

The other side of the AI debate

Over the recent past, there have been an increasing number of arguments disagreeing with the claim that scaling is sufficient to get to arbitrary “intelligence.”

Per the March 2025 findings of the annual meeting of the AAAI, including responses from more than 475 members (67% of them academics),

More than three-quarters of respondents said that enlarging current AI systems ― an approach that has been hugely successful in enhancing their performance over the past few years ― is unlikely to lead to what is known as artificial general intelligence (AGI).

Well-respected AI researchers are starting to form the next wave of AI companies that try to encode some kind of “world model” or semantic understanding of the world: Dr. Fei-Fei Li’s World Labs generates images and videos but only via an intermediating representation of a 3D world. Yann LeCun’s new startup AMI labs is likely also building world models via some form of his published JEPA work. Ilya Sutskever (one of OpenAI’s founders, who had a large contribution to Sam Altman’s perspective above) went on Dwarkesh’s podcast and said that scaling alone would not carry us to AGI and that “something crucial is missing.” Cognitive scientist Gary Marcus has frequently writes about the need for symbolic reasoning for AI and is often in the thick of the debate on how to get there.

What is a world model?

There is at present no clearly-victorious architecture for how to encode added structure in large AI models. Consider a few examples from the AI world:

• World Labs, whose product generates consistent images and video, would define it as metric information about a 3D scene

• Many AI researchers using a working definition for a world model as a (potentially latent-space) dynamical model that predicts how the state of the world evolves under actions.

  • Schmidhuber wrote a paper about world models in 1991, with the working definition as “predicting future sensory data given our current motor actions”

  • Yann LeCun proposes learning and predicting latent-space dynamics in his JEPA research (papers 2022-2025) — crucially, the projection to latent space is also learned from data, making it more general but less grounded in physical laws

  • The 1x world model is described in Jan 2026 as having latent space prediction capability and used to generated predicted future video states

  • DeepMind’s 2025 paper also seeks a ”predictive model of its environment” — In the paper it is a markov process, but for a continuous system such as a robot, it would be continuous or discretized dynamics governed by physics. It does not, however, specify how one would design architectures to take advantage of world models: “Future work should explore developing scalable algorithms for eliciting these world models and using them to improve agent safety.”

Zooming out to broader science, models have been developed and used in almost all fields; biologists have been discovering models for navigation in animal brains, physicists have been developing models for the behavior of the universe from quantum to astronomical scales for centuries, civil engineers have been using models of mechanics to build our houses and bridges, etc. Gary Marcus defines a cognitive world model as “a computational framework that a system (a machine, or a person or other animal) uses to track what is happening in the world … persistent, stable, updatable (and ideally up-to-date) internal representations of some set of entities within some slice of the world.” Each of these parties would likely have different opinions on models of the world / universe that AI should be imbued with.

In this post, we’ll stay focused on whether the added structure is important, but not discuss the relative merits of these varied proposals. (That is a potential topic for future posts; make sure to subscribe to get notified when they get published)

Subscribe now

Why do we need world models?

The critical view is that while LLMs are designed to predict what to do next, but are not designed to build an underlying semantic understanding, and that there are many examples of errors (or “hallucinations”) that can ultimately be root-caused to this:

LLMs can parrot rules of chess but will make illegal moves at the same time, they do not generalize well to out-of-training scenarios or under uncertainty and can produce unpredictable responses to uncommon inputs such as SolidGoldMagikarp, they exhibit “semantic leakage” of concepts and semantics in their input streams, with real-world impacts on usage of AI for policing. While capabilities of LLMs do keep increasing, there is concern that errors such as these cannot be universally eradicated without an architectural shift.

Is there an equivalent debate in robotics?

Humanoid robotics in particular has been having a prominent rise into the cultural and economic consciousness in the last few years. Humanoids have been featured at NVIDIA keynotes for about two years now, clearly signaling that the time is here for robotics companies to show their products and get mass-market adoption. While the field of robotics has existed for a long time, it is undeniable that the capabilities demonstrated have been seeing large improvements along with this increased exposure to the public eye.

Does the same architectural divide we just discussed for LLMs also exist in robotics? Less is known (much less agreed upon) about the best way to develop advanced capabilities in these robots, but we can use public information from some companies that have made product announcements to guess some patterns:

• The Boston Dynamics CEO says that they “need to be able to bring a new task to bear in a day or two … because, I think in a factory, there’s literally hundreds of tasks and the tasks evolve,” and their 60 minutes feature shows the ability to rapidly deploy motion capture or VR demonstration data to their Atlas robot

• Figure describes its “Project Go-Big” as an effort to collect human demonstration data in the form of first-person video for pre-training2 a navigation model

• 1x described its plan to collect teleoperated demonstration data with its robot in people’s homes for continued training of its AI model in Oct 2025, and released an update in Jan 2026 suggesting learning from internet-scale videos as demonstration followed by RL in simulation

I want to note that all these companies have very intelligent researchers and engineers on their staff, and it is very possible (and likely) that there is more going on in these particular demos; I only include these specific reference points as context to pick out broad themes. Some surfacing patterns are that (a) the rate at which different tasks are demonstrated is a high priority for these companies, (b) many of them are looking to pre-training with motion data collected from humans, and (c) this will be followed by post-training using reinforcement learning (most likely in simulation) where the system’s reward will include matching the demonstration.

My rough summary here is largely echoed by Rodney Brooks in his 2025 post on humanoid robot dexterity:

How the humanoid companies and academic researchers have chosen to do this is largely through having a learning system watch movies of people doing manipulation tasks, and try to learn what the motions are for a robot to do the same tasks. In a few cases humans teleoperate a robot, that they can see, along with the objects being manipulated ...

A robotics parallel of LLM development

Very roughly, the training process for both status-quo approaches have similar-looking steps:

• pre-training - reading internet-scale text (LLMs), vs. watching internet-scale human demonstration video or motion data (robots);

• post-training - RLHF and its modern equivalents vs. RL in simulation followed by sim-to-real porting and deployment

With this grounding, we can ask whether robotics applications will run into the same problems and debates as we discussed for LLMs above.

One unknown is whether motion data is the best analogue of text data. Rodney Brooks articulates some concerns about this in his dexterous manipulation essay, suggesting that tactile sensing data is needed (but internet-scale tactile sensing data, or any other kind of robot data, doesn’t exist). It is likely that all the robots will include tactile sensors in some form, but it isn’t clear yet how they will fit into this human demonstration large-data paradigm.

The larger question is whether a navigation capability trained with motion data will generalize to unseen and unexpected situations, since it is not designed to encode an explicit understanding of “objects” or “inertias” or “positions”. This concern exactly mirrors the ones about semantic understanding in LLMs. It is likely that the rate of this class of error will go down with larger models trained with more data (effectively, the scaling argument). To accomplish that goal, the “robot data gap” will need to be closed, which will take a lot of compute for data generation and training larger models due to the large dimensionality of the sensory and action spaces in robotics.

It is also relatively more difficult to “scale-up” in robotics for several reasons. First, latency and real-time reaction is much more important than in a chatbot setting, and so increasing model size at the cost of latency is not viable. In Figure’s Feb 2025 blog post, we can see that a 7B parameter VLM is used, at a time when when much larger (and presumably more accurate) models were available, and 1x states that 11 seconds of thinking are required for 5 second tasks. Second, as Chris Paxton has written about many times, getting diverse and useful data to feed a larger model has a lot of challenges. Third, robots need to carry their own battery packs, and so adding a larger GPU to run larger models introduces runtime and thermal management concerns.

On the other hand, the architecture (albeit with many details glossed over) seems to be consistent across many tasks and does not require too many architectural decisions to be made or parameters to be tuned (except for training metaparameters). It also allows for a myriad of types of demonstrations to be stood up quickly for garnering buy-in and support from customers or investors, which is a significant benefit. This is understandably a parallel of some of the observations that led to the scaling-based improvements of LLMs.

World models in robotics

First, we must observe that the post-training process described above will typically use simulation environments (with simulated physics) for the training process. Despite having the appearance of being model free, the properties of the simulator (which itself uses physics models) are implicitly embedded into the learned policy.3 The 2025 DeepMind paper referenced above suggests that it may be possible to prove that this implicitly captured information can be used to extract an explicit physics model after training.

So, does that mean we should put world models out of mind and learn an implicit one as needed (or not)? Well, this is a very inefficient way to learn physics equations and parameters: Euler-Lagrange equations, and classical system ID or adaptive control methods may be able to capture the same model much more easily and in a way that is more easily generalizable. RL in general can require a large amount of training for the results they produce because rewards are typically sparse. In other words, an RL-trained policy possesses “a lot of knowledge, and in some ways far more than most, if not all, humans,” to have human-level performance on specific task. Of course, humans rely on enormous evolutionary and developmental pretraining, encoding which into specialized structures is exactly the pro-world-models argument in the debate.

In terms of the models themselves, for tasks like locomotion, Newtonian physics is very well understood, and roboticists have been building on it to develop and use models like ZMP, LIP for decades. For more abstract control systems, the concept of a “plant model” in control theory is not dissimilar to the abstract state-prediction models referred to in the section above.

Some classical methods to utilize these models are to use trajectory optimization subject to the model, model-predictive control, etc. These can impose constraints on future states, so that within the bounds of the model’s accuracy, some aspects of safety can be encoded in way that isn’t possible otherwise.

How to compare the two approaches

Now that we can recognize the “world model” debate in applications for informational and physical AI, it’s helpful to (in rough, broad strokes) know how to compare the two strategies from a number of perspectives:

1. Performance: Can the method produce results that are compelling? There are umpteen benchmarks to compare language models. The next generation of world-model-equipped LLMs aren’t here yet, so we’ll wait to wait a little while to see how they stack up. There aren’t robotics benchmarks of the sort yet, though some informal efforts are underway.

2. Scalability and time-to-market: This is a huge advantage of scaling a simple architectures. Deep neural networks with consistently-repeating matrix multiplication and reduction primitives have been able to be mapped to SIMT processors like GPUs and systolic array processors (NPUs, TPUs) with incredible performance gains. At the moment there is not even enough information about non-trivial architectures to consider mapping them to computational hardware. It is also possible that world models can be mapped into the existing computational frameworks (and we can assume that the first generation of them will have to do so to compete). Eventually, if the computations are quite different, modified paradigms and accelerators may be needed, and scaling those may require more care and thought than the straightforward process we have followed for scaling LLMs. Based on the current state of language models and humanoid robotics as recapped above, it is clearly easier to get initial proofs-of-concept working with model-free approaches scaling a simple architecture.

3. Computational efficiency: Newton’s equations descibe motion of bodies with very few parameters in great generality, and it is impractical to capture them with a “transformer-like” structure without significantly higher number of parameters. This is especially true where equations are discontinuous, which happens in robotics problems like locomotion and manipulation. AI is currently up against the so-called “memory wall” due to the fact that these models need to be so large, and the most recent innovations and movements in ML accelerators have been to do with addressing it. Utilizing appropriate models with differently-architected communications may completely sidestep this memory wall, as well as drastically improve the efficiency4 of equivalent computations.

4. Generalization: It should be clear that some of the models that need to be learned for robot motion have very applicable and general models that have been known for centuries, and the same holds for biologists, cognitive scientists, and psychologists in their fields. Ilya Sutskever, one of the architects of the current LLM era, says that their structure is weak at generalization and that generalizing in the way that humans can needs new architectures. The aforementioned DeepMind paper also cites domain adaptation and generalization to unseen tasks as something that could be improved by using world models.

5. Safety: We’ve discussed hallucinations in this post already, and the aforementioned Quattrociocchi paper makes an argument about the reliability of results from LLMs. The point of concern is how the system will react to unseen circumstances and whether it can extrapolate in reasonable ways. It may be especially important to have mechanisms for guaranteeing the possible range of actions the robot can take and explaining its decisions.

I didn’t feel like there is sufficient information to score the approaches yet, but it is clear that model-based approaches may offer advantages in generalization and interpretability, while model-free scaling currently dominates in deployment speed and tooling maturity.

Closing thoughts

Before closing out this article, I must point out that this “divide” is really a spectrum—there is likely a rich space of hybrids of the two approaches, which may consist of hierarchical structures combining the strengths of each. Deep learning excels at parsing and summarization of text and images, automatically finding the most appropriate dimensional reduction techniques. World models, when coupled with methods that know how to use them, are strong at generalization, abstraction, and can produce very computationally-efficient algorithms.

In future posts, I plan to write about any new developments on the informational or physical sides that are demonstrating usage and adoption of world models, or of new hybrid architectures. I will also be plan to write some posts where I construct simple scenarios to fairly evaluate competing architectures along the different metrics above. Last but not least, I will plan to go into more details on computational hardware acceleration of non-trivial architectures.

I believe this is going to be an ongoing recurring topic in this publication, so make sure to subscribe and share if you found this interesting.

Share

Just to provide brief context,

• the RoboBee hardware was at this point fairly mature, and on the “Split Dual-Actuator Bee” generation;

• the state-of-the-art flight controller was a capable, but task-specific hovering controller with limited generalizability.

The goal for this project was to develop a controller that could be easily generalized to more complex tasks using modern control methods. The resulting paper1 with Dr Rebecca McGill made some demonstrable advances in terms of better operation away from an upright configuration, the ability to stabilize tasks like following a desired path or executing more dynamic behaviors like perching and flipping, as well as robustness to suboptimal gain tuning and manufacturing variability.

Here are some hovering clips (short 32s video; no audio):

The remainder of this post explains how this result was achieved, and potential future extensions of the idea.

Background

RoboBee and flapping flight

The RoboBee is a 100mg flapping robot developed by Dr. Rob Wood, capable of hovering and controlled flight. To put it in perspective, a US nickel weights 5g or the equivalent of 50x RoboBees. Having spent a lot of time in the Harvard microrobotics lab fabricating them, it is no exaggeration to say that a sneeze can literally destroy weeks of work.

Along with a family of similarly-fabricated robotic systems developed at the Harvard microrobotics lab, they are actuated by piezoelectric bending actuators. The piezoelectric effect is commonly seen in the working of microphones, which convert vibrations created by acoustic pressure waves into electric signals. They also do that in reverse, converting electric pulses into vibratory motion. The RoboBee uses piezoelectric bending actuators, constructed similarly to a bimetallic strip, converting slight expansion and contraction of the piezoelectric material into a bending motion.

Generally, the piezoelectric actuators produce very small motions that need to be amplified to produce the requisite aerodynamic work. After the conversion to the bending motion, they also go through another transmission that converts the small bending translational motion into a rotational motion. In a previous post, I went into the details of how this transmission works, and a project I worked on to optimize it.

OK, now we are at the stage of converting electrical signals into rotational motion of the wing. The wing itself is attached to the end of the transmission via a passive hinge, so that when the base of the wing is flapped, it not only flaps, but also pivots about its hinge, thereby actively changing its pitch, or angle-of-attack. This motion is common among flapping animals:

RoboBee’s clever design allows the wing pitch to change passively as the wing flaps, i.e. only one actuator is needed per wing to obtain something resembling the complex wing motion of the hummingbird above:

Modeling RoboBee’s flight

A model of the motion produced is very important to understand how to use the available wing input signals to get to a desired goal. There is a large debate between model-based vs. model-free methods (which eschew models in gathering a lot of data with the black box system and approximating its behavior). Increasing computational power recently has resulted in increased temptation to abandon models, though in many sim2real reinforcement learning approaches, models are used in developing the simulation.

In the case of RoboBee, the difficulty with pursuing a fully model-based method is that aerodynamics is quite difficult to model. Nonetheless, some work in the early 2010’s on blade-element modeling has proved quite useful for understanding the relation of RoboBee wing motion to the produced lift and drag forces. Using that model, we developed a RoboBee simulator, which is open-sourced2. We will discuss the software supporting this work further below, but here is an animation of some fixed control inputs (similar to the 2013 flight control work) producing simulated flapping flight, complete with passive wing pitching (short 17s video; no audio):

The disadvantage of the model above that it is very complex and not possible to use to directly develop a controller. However, the other components of RoboBee dynamics (excluding how the wing produces lift and drag) are well-explained by Newtonian physics. In this latter area, there is a great degree of similarity to the control of legged robots.

Typically the world of flapping flight and legged control do not overlap, but there are a number of similarities that motivate the use of similar methods. They are both

• cyclic (though in the RoboBee case, the wings are assumed massless and flap so fast that their dynamics are considered decoupled from the body);

• mechanics-dominated (it is very important to consider the physics of ground interactions and aerodynamics); and

• underactuated (we don’t have enough actuators to fully stabilize the motion, and typically in these scenarios some amount of “lookahead planning” is required).

In the legged robotics field, there is a long tradition of using simplified models to aid in control development (so-called “spring-mass” models), as I have discussed before. In this paper, we introduce for the first time an equivalent for RoboBee-like flapping flight.

Model-predictive control (MPC)

As discussed above, in underactuated scenarios, it is typically the case that some knowledge about the future behavior of the system can be predicted in order to decide which inputs to supply. As a simple example, how should the cart be moved in order to get the attached pole to swing up?

A global understanding of the future behavior of the system can be summarized in a so-called value function, and knowing this function can tell us exactly which way we should move to get to our goal from all states.

The problem is, the value function is not “known.” It can be estimated by exhaustively poking and prodding the system (which is an approach that resembles reinforcement learning). However, when we know of a dynamical model for the system, it is sensible to use it, because it greatly reduces the dimensionality of the control system to treat the dynamics as fixed.

Model-predictive control (MPC) tries to create a small local approximation of the value function online by using the future state of the system over a short prediction horizon (subject to a model) as a proxy for the value of the current state. MPC is now an old technique, but widely used in industrial process automation, aerospace, etc.

Approach: model-based MPC and model-free inverse dynamics

Here is the overall plan:

1. Develop a simplified model capturing the desired behavior: for this step, I noted that we do not care about the heading, but instead simply that the robot stays upright.

2. “Anchor” the behavior on to the RoboBee: convert to signals that get sent to the acuators.

The system architecture figure below makes this explicit. The purple “flying brick” is the model, whose future states we can predict for known inputs. The MPC can then effectively back out the best inputs for that model to get to a desired state.

However, as the blue and green arrows in the figure show, that only partially solves our problem because the RoboBee is not a flying brick. To address the gap, we need to define the operations undertaken by the arrows:

• State projection (blue arrow): This process is relatively simple for this instance, because the state of the flying brick is effectively a subset of the state of the actual RoboBee. It has an elevation and body tilt angles just like the RoboBee, and we simply project the coordinates to those of the flying brick.

• Inverse dynamics (green arrow): The other direction is more complex—in essence, we want to go from the abstracted thrust/roll/pitch torque inputs for the brick → RoboBee wing actuator signals. This process is complex because of a couple of reasons:

  • The mapping is much more complex than any kind of projection; for the various components:

    • Wing voltage → actuator motion (depends on piezoelectric actuator electrical and mechanical properties)

    • Actuator motion → wing base motion (depends on transmission and its stiffness)

    • Wing base motion → wing motion (depends on hinge and wing mechanical properties)

    • Wing motion → reaction forces and torques (depends on wing aerodynamic interactions, ground effect, etc.)

  • Manufacturing variability makes this mapping inexact (if you manufactured two RoboBees, they may require different wing signals to produce the same wing motion)

For these reasons, models have limited utility for the green arrows, and so, the paper proposed a model-free method for that part.

Model-based MPC

Template: upright rigid body

First, we need to pick the model. As the saying goes, all models are wrong, but the goal here is to capture the most important parts of the dynamics, and the objective.

The RoboBee’s wings are very light, and so most of its mass is truly contained in its body (more on this below). Dynamically, this is well-approximated by the flying brick, with no other moving parts.

To capture the objective, we note that we do not particularly care about the heading of the RoboBee when we just want it to hover, or fly controllably. This allows us to effectively remove one degree of freedom from our specification of the objective, and capture the state of the flying brick with:

• To capture the position, we use the (x, y, z) Cartesian coordinates of the center of mass as expected.

• To capture the orientation, we only look at the components of the “upright vector” (a vector pointing up in the body frame). Note that an objective of hovering can be simply stated as the desire to have the upright vector point vertically up.

Waypoint tracking MPC

We write the dynamical equations for the flying brick using the Newton-Euler equations for the motion of a rigid body. After a small approximation as described in the paper, we get

where p represents the Cartesian position, s represents the upright vector, T represents the (scalar) upward thrust, and τ represents the (2-dimensional) roll, pitch torque vector.

These equations are quite simple, owing mainly to the fact that the wings are quite light, and so their flapping does not significantly impact the motion of the much more massive body. This concept is also utilized in many legged running robots, referred to there as “massless legs.” It’s worth taking a minute to appreciate the significance of this: in practice, human limbs are not massless, which allows (for example) a gymnast to adjust their body orientation while flying through the air by controllably moving their limbs and landing a flip. However, mastering that kind of control is much more difficult than the massless legs (or wings) paradigm, where we can safely make the assumptions that the appendages simply produce a force or torque that acts on the body. A helpful picture to have in mind is that in the massless appendage paradigm, we can substitute the appendages for thrusters attached at appendage base, and pretend we are controlling the thrust vector instead.

Upon further inspection, the equations are second-order (as expected for any mechanical system). The orientation equation is also unfortunately nonlinear, as can be seen from the product of s, T, and τ appearing on the right side. This is also normal for such systems, but adds a challenge to our MPC transcription.

To resolve this difficulty, we linearize these dynamics at the current orientation and thrust (s₀, T₀) before incorporating them into the model-predictive controller. The controller will reason about the best inputs based on how they act on the current state, which intuitively is fine for a short enough planning horizon.

As an analogy, a car driver on the highway will turn their steering wheel slightly to change lanes (an action that is appropriate for a planning horizon for a few seconds), even though that action would not be appropriate on a long enough horizon that they drive off the highway. Similar to the car driver, the RoboBee in this scenario will re-evaluate its inputs with a new state soon enough. MPC always works this way, with a finite planning horizon a short duration from the current time.

The objective for the MPC is to track a trajectory of future states, including a position and velocity. For example, to hover, the desired position is the hovering goal position, and the desired velocity is zero. To follow a particular path in space, that path can be discretized and substituted into the desired positions.

Simulation evaluation

To evaluate if the MPC with the linearized dynamics above works appropriately, we can compare the performance of the controller in a number of simple tasks in an apples-to-apples comparison with the prior state-of-the-art reactive controller.

Hovering, trajectory following

The tasks, as shown above, were:

• Hover task starting off withan initial orientation with roll​ and pitch angles set to 0.5 rad, -0.5 rad, and initial velocity 0.1 m/s in the x-direction​

• Waypoint tracking on an​ “S”-shaped trajectory in the xz-plane.​

• Tracking a commanded velocity of 2m/s for 0.5 seconds before stopping.

In each of these scenarios, the MPC performs better than the reactive controller (notes on tuning below), which is promising.

Perching, flipping

Specifying a task in terms of a reference trajectory can be onerous, for example, if we want the bee to do a backflip, it isn’t clear what sequences of positions and velocities are appropriate for the horizon.3 To test the robustness of the MPC, here we feed it “made up” infeasible trajectories and see how well it can track them.

The tasks we choose to test include the aforementioned flip, and a wall-perching behavior inspired by this past research:

The reference trajectories are selected intentionally naively:

• For the perch task, the desired position translates smoothly to the right, and the desired orientation steadily rotates to 90 degrees at the end of the motion

• For the flip task, the desired position is fixed, and the desired orientation smoothly rotates 360 degrees.

The results show that the MPC is able to compensate for the naiveté of the reference trajectories to accomplish the task to satisfaction. The reactive hover controller cannot solve these tasks.

A note on tuning the controllers

Something that most research papers will sweep under the rug is the process of how the controllers were tuned. The previous state-of-the-art reactive controller has hand-tuned PD gains, and the MPC has weights on the objective. To make a fair comparison, we have to tune both as best as possible.

In general, there is a tradeoff between tracking error and tracking effort. As an analogy, cruise control in cars often have an eco mode, where they may deviate from the speed setpoint a bit more, but waste less fuel. Similarly, you can spend less actuator effort in exchange for tracking the goal a little less precisely. This is usually one of the ways in which controllers are tuned in practive.

The plot on the right shows the MPC and the reactive controllers fairly compared with a variety of tuning gains, showing that the MPC is significantly easier to tune, and can track better with lower actuator effort than is possible with the reactive controller.

Data-driven inverse dynamics

As we discussed above, the mapping from actuator signal → produced force/torque is unknown/uncertain due to the system complexity and manufacturing variability.

An example of a common type of manufacturing variability is that some RoboBee transmissions just exhibit higher stiffness than others. If the left wing has a stiffer transmission than the right wing, the left wing may flap with a smaller wing amplitude than the right one when driven equivalently, and produce much less lift force.

In this project we took the approach of breaking down the components of this mapping, and just using data to characterize the variable parts. This meant collecting data of wing kinematics as a function of actuator signals and then fitting a function to approximate some “kinematics features” that could be expected for each actuator signal:​

We then used the blade-element model to predict the reaction force/torque from the wing kinematics.

To show the effect of this kind of mapping, we performed the same operation in the RoboBee simulator, and simulated the effect of adding a force bias of 3 mN to one of the actuators. With no force bias, the data-driven mapping and the manually-tuned mapping both work, but with the force bias, the data-driven mapping can still work while the manually-tuned mapping fails.

Hardware integration

Setup

Encouraged by the simulation results, we pushed ahead to integrate the MPC into the physical RoboBee control system, which looks as follows:

The actuators were connected to a Simulink real-time control PC, which was new to me. The setup encourages code to mostly be compiled from graphical blocks such as filters, delays, etc., but does allow for custom blocks written as MATLAB functions. While the state estimator and some other components were in fact MATLAB functions, we implemented the MPC in C using OSQP, as part of a more forward-looking architecture that could also run onboard the RoboBee on a microcontroller.

When run from the Simulink target PC, the iteration frequency was 5KHz for everything, locked together due to the Simulink architecture. The MPC itself also ran at 100-200Hz on small STM32G4 MCU​ that fell within the 25mg payload constraints of the RoboBee. We tested that the controller could successfully stabilize the simulator when run at rates of 100Hz.

Experimental results for hovering

A video clip of some of the hovering results were linked to in the introduction of this post. Some overlaid trajectories from those trials are shown in the figure below.

Each trial ended due to the motion capture system losing track of the RoboBee, or by a command we sent. We were able to keep the orientation stabilized in each trial, though the horizontal position drifted more than desired.

The hovering task was overall a good demonstration of the feasibility of integrating this much more advanced controller paradigm into the RoboBee.

In the future, it would be very exciting to see either or both:

• some of the tasks we tested (and compared to the reactive controller) in the simulation section running on the RoboBee

• the controller running on a microcontroller, along with onboard sensing and power, for fully untethered complex flight

Implementation details and replicating results

In the interests of open science, the code for various parts of this project are all online. While I don’t have continued access to the Simulink software and experimental setup, if you need support, please comment below—continued progress and replicability are well worth the support and debugging.

MPC

This is implemented as a quadratic program with OSQP.

• The quadratic program is defined in genqp.py. When that file is run as a script, it instantiates the controller and runs a test, or in the commented-out section at the bottom, run’s OSQP’s codegen feature to generate a standalone set of C files that can solve the QP. The codegen output is stored in the uprightmpc2 directory (though it can be regenerated as well).

• The codegen outputs define the structure of the problem, but the variables need to be updated as the current state of the RoboBee or the reference trajectory changes. To do this, the uprightmpc2.h file provides some simple interfaces with named parameters that can be called. The C file of the same name contains its implementation.

• The C code in the uprightmpc2 file can be built using CMake; something like

cd uprightmpc2
mkdir -p build && cd build
cmake ..

Simulations

• The simulations testing the MPC with the upright template model can be run from the template directory.

• The uprightmpc2.py file should recreate the test scenarios covered in plots above and in the paper when run as a script. The bottom of the file contains code describing the test scenarios that can be uncommented.

• The 3D pybullet simulation can be run by executing the robobee.py script.

Simulink setup

• The C code is integrated into the Simulink real-time setup as an S-function; the legacy_code_gen.m file configures the inputs and outputs of the block that will appear in Simulink. See this page for more guidance on this process, which was quite tricky.

• The simulink model files are slx files, and can be found here.

Modularity

In this talk, I started a bottom-up exploration of composition in robotics.

Dynamic legged locomotion

As with I’m sure many others, as a young graduate student, I was inspired by the dynamic legged locomotion work at the MIT Leg Lab in the 1980’s:

In his thought-provoking book, Raibert articulated an intriguing idea called “Control of Running Decomposed into Three Parts.” Researchers have been trying to understand when and how this may be possible, and how it generalizes, since then.

My Ph.D. advisor, Koditschek, has been doing that for decades. In the 1990’s, his research group built and impressive array of juggling robots (as a less-power-hungry proxy for cyclic dynamical behavior):

In the course of the juggling research, they introduced a formal idea of sequential composition with an intuitive but mathematically rigorous and useful idea:

Analogously, we can retroactively label Raibert’s “control in three parts” idea as an example of parallel composition. While the term is not extremely common in the robotics literature, similar concepts appear with names such as “decoupled control”. The idea has clearly been empirically useful, but formalizing it has been quite tricky with any degree of generality.

Sequential and parallel composition are a very intuitive idea with equivalents in programming and spoken language. Consider the example of generating spoken language – instead of outputting the sounds corresponding to an entire sentence at once, we may want to start by assembling words from phonemes, and assembling those into sentences. On the other hand, modern deep learning speech synthesis may not have any such compositional properties, which is an intentional counterpoint that we will return to.

Modularity elsewhere

Deep learning did evolve from neural networks, which evoke biology right in the name. Biology has inspired many of the working principles of quadrupedal robots, including behavioral modularity.

Animals have an abundance of sensory inputs and muscle, but the number of task-level variables important to any particular task is a lot smaller (Ting (2007)). Going further, Ting et. al. (2015) argues that motor modules arise from neural plasticity in spinal structures that selective coordinate and co-activate multiple muscles. The result is that animals can control tasks like balancing in a hierarchical fashion, keeping the dimension of the task-space control low.

While robots typically have fewer actuators than an animal has muscles, each individual task will typically be overactuated for a general-purpose robot. For example, a humanoid robot will not need its arms to maintain a standing posture.

If we accept the presence of these motor modules, these patterns of activation could be re-used for different behaviors. Quoting Ting et. al. (2015):

Multifunctionality: muscles can contribute to many actions; a few muscles can be combined in many ways to produce a wide range of different actions.

Making equivalences to the synthetic disciplines, there is a clear connection to the idea of re-using behavioral modules, as we showed with Minitaur vertical hopper compositions.

Putting it all together, I’d argue that there are equivalences between biology and robotics in three distinct aspects of modularity:

Modularity benefits

Some of the benefits of modularity that are enjoyed by biological systems can also apply to the synthetic disciplines as well.

Motor modules can help navigate a “difficult-to-search and nonlinear set of neuromechanical solutions for movement” (Ting et. al. (2015)) as well as the “curse of dimensionality” in various engineering disciplines. This has clear implications on the computational requirements for algorithms.

A slightly less obvious use case for modularity is for optimizing robot design for flapping, jumping, etc., using coordinated movement patterns (or, template trajectories).

Real-world robotics

As robotics tools proliferate, their side-effects will start to also have a larger and larger impact on society.

Safety and predictability

The autonomous vehicle industry is possibly the first (but certainly not the last) subfield that has been thrust into the limelight of the question of safety of autonomous systems. The responsible peer-reviewed efforts of the first-party companies (e.g. Waymo) are huge steps in the right direction, but that is certainly not the end of the story.

Robustness and multiple solutions inherent to a modular structure (as we saw above) is in stark contrast to the weakness of monolithic AI structures when subject to uncertainty (Cummings).

Intuitively, a modular architecture can be “debugged” and intermediate outputs can be logged and inspected. Just like a black box recording of an aircraft allows review of inputs made from the pilot to the machine, a modular structure allows insight into, and thresholding of, the function of individual modules:

Energy

While mechanical work done by robots necessarily needs energetic input (and the conversion efficiency can be quite high), the cost of computational work is nowhere close to the only known fundamental energetic limit based on Landauer’s principle.

Even as chips get more and more efficient, our appetite for computation outstrips those benefits, raising continual concern.

As already recognized by biology, a growing community of researchers are exploiting the fact that modular neural networks reduce power consumption.

The case for compositionality

Modularity comes with a price. The motor modules in humans have appeared over the (long) course of animal evolution, and the modular control structures developed for robots need to be hand-crafted. These processes are much less automatic, and need more work than scaling a simple structure with more data. In fact, the importance of pushing for architectural progress may not be limited to robotics (LeCun).

Additionally, modularity necessarily imposes limits on the space of usable methods or algorithms. For example, a modular controller reasoning with the equivalent of “motor modules” for a triple pendulum would never be able to accomplish this:

Nevertheless, the question of system abstraction with modularity has come up before in other fields such as digital VLSI and programming languages, and has clearly won out, in part due to the reasons discussed above.

We don’t yet have a generally accepted methodology or architecture in robotics that could be a foundation for symbolic behavior programming.

End-to-end deep neural networks have become a useful and generally-accepted architecture without compositional properties, but neural networks are not necessarily incompatible with compositionality (Hinton, Marcus). For more on this topic, I highly recommend the proceedings of this workshop on The Challenge of Compositionality for AI.

What is the path forward?

If we value the benefits of modularity discussed above, it will take more work to develop the correct architectures, but this work is essential to get to the point of robotics becoming a true scientific discipline with predictable outcomes.

Both these projects were extensions of the Raibert’s intriguing concept of “control in three parts” for the MIT Leg Lab planar hopper. However, it has been very difficult to formalize when this type of control may work. In some related work, it is called “decoupled control,” but it is clear that any robotic system of practical value will not have a decoupling property. The hallmark of articulated mechanical systems is that energy can be transferred among different components, which makes them expressive and capable, but also difficult to analyze.

In this project and paper1, we proposed hybrid dynamical averaging as a way to make progress toward making formal arguments about these complex systems. This project only scratched the surface, but we applied this idea to Minitaur vertical hopping in a sequel. I think the idea still has a lot of potential in helping us make formal guarantees about the behavior of complex systems with symmetries (ubiquitous in locomotion), maybe playing an important role in formally guaranteeing their behavior in safety-critical scenarios.

Dynamical averaging

Since the exact system dynamics are difficult to directly analyze, it’s helpful to consider approximating the behavior somehow. To this end, we looked to the well-established theory of dynamical averaging (Guckenheimer and Holmes), which applies to cyclic dynamical systems.

The idea is that a dynamical system with a limit cycle can be viewed in terms of a single “fast” coordinate along the limit cycle, and several orthogonal “slow” coordinates. As an example, an oscillating spring-mass system conserves energy, so the total mechanical energy is clearly a “slow” coordinate (it doesn’t change at all). A very slight generalization is a regulated oscillatory system, where the energy might vary as it stabilizes to its limit cycle.

These “fast/slow” dynamical systems can be approximated by averaging the dynamics along the limit cycle:

By doing this, we have reduced the dimensionality of our system by one, and crucially, eliminated any variability in f that “averages out” over a cycle. This has huge intuitive (and mathematical) implications that we will come back to below.

Choosing coordinates

The spring-mass example above conveniently was two-dimensional, resulting in a singular slow coordinate that we easily identified with total energy. What happens when there are (for example) several coupled spring-mass oscillators, with total system dimension n?

This example is instantiated in a very real manner in the Minitaur vertical hopping demonstrations, and can appear frequently in coupled mechanical systems.

Our idea here was to think about the different coordinates as being:

• A single fast coordinate identified as the system phase: the cyclic coordinate that increments at a near-constant rate

• A single slow coordinate identified as the system energy: an overall measure of the “amplitude” of the system

• n−2 slow coordinates identified as phase differences: the relative phase of different degrees of freedom of the system

These coordinates are related to (but not being formally connected to) Hamiltonian phase space coordinates. The phase differences intuitively arise when several degrees of freedom are coordinated together, as in a set of coupled oscillators.

A visual depiction of these coordinates, with (n=3), is shown below, where the blue manifold corresponds to the energy zero set, and the red manifold corresponds to the phase difference zero set. We intuitively refer to these as the regulated and neutral sets:

Note that for mechanical second-order systems, (n) must be even (twice the number of degrees-of-freedom); the (n=3) is for illustration.

Hybrid averaging

The major contribution of this paper was to prove that it is possible to extend dynamical averaging to hybrid systems. To do this, we used the saltation matrix to capture the effect of the hybrid reset near the limit cycle.

A visual depiction of this is below:

A much more technical treatment and formal proof of the intuitive idea is, of course, in the paper.

Symmetries and averaging

Applying (hybrid) averaging to study locomotion behaviors like hopping and running let us incorporate time-reversal symmetry to get amazing and intuitive reductions in system complexity.

First, the following slide, which some figures from Legged Robots That Balance, convey the ubiquity of time-reversal symmetry in locomotion. This is straightforward in systems like a one-legged hopper, but also appears in much more complex scenarios like a cat galloping.

As the slide suggests, the referenced symmetry is not merely a property of the resulting trajectory, but a property of the dynamics itself (i.e. the dynamics (xdot = f(x, u)) exhibits symmetry with respect to various components). This has exciting connections to Hamiltonian mechanics and Noether’s theorem that beg further exploration.

As annotated on the slide, the “symmetric” hopping trajectory in bold in the bottom right figure can be distinguished from all the asymmetric trajectories. These simply correspond to being on the “neutral” set in our nomenclature above, or not, respectively. Putting all this together, when the dynamics are averaged at a neutral limit cycle, the dynamics are greatly simplified (intuitively, odd functions integrate out), giving us a great degree of analytical simplification.

System architecture

The starting points for this method are a template model, and a task. In this case, the task is flapping, and the model is the blade element model applied to flapping wings. Those two, in conjuction give us a dynamical trajectory, with kinematics and interaction forces with the environment.

These kinematics are obviously parameterized by the parameters of the model that generated them. For example, if a flapping wing has more inertia, we can expect a smaller angular amplitude when subject to the same flapping torque.

For our design optimization, we non-dimensionalize the times and lengths in the trajectory. With this non-dimensional trajectory y(t), and a parameter vector p, we utilize the fact that in mechanical systems, the dynamics are affine in p, i.e. the dynamical equations of motion can be written in the form

This fact underpins classical adaptive control, and in our case here, allows us to establish a bilinear constraint between state and parameter variables.

So, the design optimization problems attempts to find paramters that can minimize energy consumption, or another objective specified in ϕ(p), while minimizing the deviation from the desired trajectory ‖Δy‖, and constrained by the system dynamics.

Nonlinear transmission design

The main empirical contribution of this paper was a new type of transmission I designed for the RoboBee (A in the figure below), where the transmission ratio varied as a function of the actuator angle.

The green line in B shows an ideal nonlinear transmission (with ratio τ=τ1+τ2qact), and the yellow/blue lines show the physically realizable transmission developed using a non-parallel linkage system for this paper.

Intuitively, the goal was to have a lower transmission ratio (higher mechanical advantage) at midstroke, where the drag force is the highest, and higher transmission ratio (lower mechanical advantage) at the end-stroke positions where the drag force is lower.

C/D in the figure above show the simulated results of using such a nonlinear transmission – the specific lift force (normalized by actuator mass) can be increased using this kind of nonlinearity.

Co-design pitfalls

The bilinearity in the constraint above exactly conveys some of the difficult aspects of “co-design” (i.e. simultaneous design and control). The plots below show some slices of contours of the objective function for the RoboBee design problem with highly nonlinear level sets.

Some observations from the plot above:

• The left plot shows a feasible set of wing mass (vertical axis) with respect to wing area (horizontal axis) between the two blue lines. The purple line is a minimum lift constraint needed to fly, for which a minimum wing area is needed. In the top right feasible region, there is a unique optimum that was found.

• The right plot shows two parameters controlling the transmission ratio between the actuator and the wings. The feasible set of transmission designs is under the blue line in this case. The red line is a constraint on the maximum actuator displacement allowed (so that the piezoelectric actuator does not break), and a conservative linearization of that constraint is shown by the purple line. The objective function is again highly nonlinear, but the design optimization is able to find an optimum that would intuitively likely have been challenging.

Using simple bottom-up, decentralized, decoupled controllers, we were able to show a wide range of gaits working stably on Minitaur, with pretty good performance:

Virtual bipeds

The way we used the idea of virtual bipeds here was to “project” the coordinates roughly along the gray arrows, and “pair” the legs projected together.

After this projection, we are left with a kind of planar bipedal system to analyze with three degrees of freedom, as shown in the right column above.

Vertical hopper compositions

Focusing on the bound/pronk projection, the types of limit cycles for these two gaits look like the flows roughly resembling the following picture:

Looking more closely the picture above, the axes in the plot are the “phases” of the two legs, a concept we talked about in the accompanying hybrid averaging paper. That identification now encourages us to think about our original quadruped as a pair of (vertical) hoppers “coupled” by a body.

The “coupling” is physically instantiated by the body itself, and its inertia properties have a significant effect on the type of coupling. The center of percussion is a well-studied property of baseball bats, juggling clubs, etc. that relate impulses on one end of the object to the wrench on a different point along the object. We used this definition:

Intuitively, the location of the center of percussion, which is in turn related to the mass distribution of the object, affects the type of coupling between the two hoppers.

Pronking and bounding

We showed using simulation, and by physically altering the inertia characteristics of sagittal-plane Minitaur by adding an “inertia bar” to its back, that we could indeed get both bounding and pronking limit cycles by programming its front and rear ends as completely independent vertical hoppers:

This is a bit of an extreme interpretation of the style of decoupled control originally pioneered by Raibert and also demonstrated on Jerboa, but it also has interesting implications on the possibility of distributed control of legged robots. Another way to think about it is that we are formalizing preflexes, which have historically played a pivotal role in the mechanical stabilization of animal and robot locomotion.

Robots tend to imitate naturally occurring, biological structures. What is it about a dog’s form that makes it ideal to roboticize?

The field of bio-inspired robotics is the result of close collaboration between biomechanists and roboticists. It is a synergistic discipline, where biologists study animals and distill their functioning principles into lessons that engineers can use to design robots. In the other direction, machines built with simple mechanisms and programmable behaviors provide testable hypotheses for the biologists to deepen their understanding of animals.

It’s important to note that useful robots are not usually created by mimicking animal form (biomimetics), but rather by taking inspiration from their working principles (bioinspiration). Engineers have access to different materials and power sources than those available to biological creatures, and animal bodies have evolved to serve many functions that are not relevant to a robot. For this reason, a four-legged robot may roughly resemble a dog’s form, and in particular move in a manner resembling a dog, but will not copy the intricate details of its biological structure.

Four-legged animals like cats or dogs usually walk on their toes (digitigrade), while humans and other two-legged animals walk on flat feet (plantigrade). This is related to the fact that the larger feet allow for a lot more control over balance than the toe-tips (imagine a human standing on one leg, compared to a cat or dog). In practical terms for engineers, this means that the legs of two-legged robots are a lot more complex, with many actuators needed for the hip, knee, and ankle in order to balance. In comparison, a four-legged robot can have much simpler leg designs, with no ankle or foot needed at all. In fact, typically, a four-legged robot has fewer motors than a two-legged robot, reducing cost and complexity, while having good agility and balance over difficult terrain.

Can you walk me through how a robot dog works/functions?

The body of a four-legged robot has four simple legs that can position the toe at any point within reach. For the robot to stand in place, four of these toes are in contact with the ground, and the forces generated using the motors in the leg create a reaction force on the body, thereby supporting it. When the robot needs to move, some of these legs leave the ground and swing through the air to reposition the toes in the direction of movement, or simply to find a better foothold. While the algorithms used to calculate these forces or where to reposition the legs can get quite complex, the basic working principle is simple: If you recall the last time you had to scramble up some terrain on a difficult hike, the idea of using limbs to either push against the ground, or reposition them, will seem quite familiar. These same principles are applied by algorithms to select when and where to reposition which legs, so that the robot stays balanced as it moves over different types of terrain.

Unlike wheeled vehicles, legged robots can move over broken or discontinuous terrain by picking isolated footholds, allowing them to go into locations without roads or trails. Other than the legs, the body of the robot contains a power source (typically a battery), computing devices to run the aforementioned algorithms, and sensors needed to sense the environment around the robot.

The Vision series includes some of the most successful quadrupeds to date. Can you elaborate on certain design features that make it superior to other models?

Despite the recent rise in popularity of legged robotics, we are still discovering newer and more impactful use-cases for this technology, and these applications inform which features robot manufacturers prioritize.

One important feature is energetic efficiency, which needs careful attention in all building blocks of a four-legged robot, ranging from its design to the algorithms used to control its motion. In the longer-term hopeful future of a proliferation of these robots in various use cases, this focus will allow a positive weighting of their utility against their energy consumption. Emerging technologies like artificial intelligence (AI) are currently having to contend with this cost-benefit analysis, and legged robots will be no different.

It is also important–because the technology is so new–to embrace developers and empower them to customize the robot to solve new problems. A wide range of peripheral connectivity options and a software developer kit can enable (for example), the development of a hose-pointing behavior for firefighter support. Much like an app store amplifies the utility of a smartphone, customizability will increase the utility of legged robots in niche but impactful use cases.

What are some of the trickier aspects when it comes to developing a robot dog? What challenges are developers in this space currently facing?

Locomotion in challenging terrain requires cutting-edge algorithms, spanning from traditional methods that leverage classical physics principles, to ones that use machine learning to learn from experience and data. In either case, more sophisticated strategies may result in increased functionality, but at the expense of more computational power and fragility to changes in the robot hardware, operating environment, or payload. Balancing the utility of an algorithm against its computational footprint and sensitivity is a difficult task for roboticists.

It is also challenging to increase the battery life of a legged robot. It needs to carry its power source, and so while installing a larger battery would increase battery life, doing so may introduce other issues such as reduced payload capacity. Improvements in battery energy density and microprocessor computational efficiency are innovations that are currently outside of the scope of our in-house research team, and the efficiency of electric motors for legged robots is limited by the fundamentals of electromagnetics. This leaves robot designers with difficult optimization problems in mechanical design and algorithm selection as some of the only tools available to increase their running time.

Lastly, legged robots are complex electromechanical machines, currently being designed and built by research-oriented organizations that only have limited manufacturing experience. Eventually, legged robots will be able to be mass-produced with the same reliability and cost-effectiveness that we have come to expect from automobiles or household appliances, but getting to that point requires scale that will only come with time and expanding applications for these robots.

Future of robot dogs: As robotics and AI become more integrated into industry and even daily life, how do you see robot dogs being applied in the near future? Especially considering the wide variety of attachment possibilities.

Four-legged robots are very versatile platforms that exhibit good mobility, payload capacity, and range for a given size and weight. This means that they are able to carry sensors and payloads (attachments) for long missions, in built environments including stairs, or in rugged outdoor terrain. This makes them very suited for automating tasks such as security and equipment inspection, surveying, and mapping. These jobs currently involve workers going into remote locations, hazardous environments, or performing repetitive tasks, and are ripe for automation. Automated inspection and security call for payloads such as high-resolution, thermal, electro-optical, or low-light cameras, acoustic imagers, and hazardous gas sensors.

These robots also provide a great deal of utility as the eyes and ears of first responders, going first into dangerous situations such as disaster response, and response to chemical, biological, radiological, nuclear, or explosive ordnance (CBRNE). For these applications, payloads such as multi-gas detectors, raman spectrometers, radiation, and explosive trace detectors are useful payloads.

Why is this tech important?

As the focus grows on the hazardous and unsafe conditions faced by workers across the world, it will be important for us to come up with machines and tools to perform these dull, dirty, and dangerous tasks in their stead. Parallel to the rapidly increasing utility and footprint of informational automation with AI, there is a need for automation of physical work, such as carrying sensors and other payloads, accessing dangerous locations, and automating mundane physical tasks. Four-legged robots are a flexible and versatile platform that have the potential to be adapted to all of these use cases in a cost- and energy-efficient manner.