Are brain waves the next unlock for physical AI?
The frontier of physical AI is a Jenga game in a warehouse in San Leandro, California. That warehouse is occupied by Encord, a company that
The frontier of physical AI is a Jenga game in a warehouse in San Leandro, California. That warehouse is occupied by Encord, a company that builds data tooling used to train AI models. Andrew Ceja is a pilotâthe companyâs term for its robotic trainersâand heâs carefully pulling wooden blocks from a tottering tower while wearing a headset with a camera that tracks what he sees. That alone is fairly common for collecting robot training data, but this headset includes sensors that measure his brain waves as he carefully disassembles the block tower. Encord is one of a small but growing number of startups betting that the next real constraint on humanoid and warehouse robotics wonât be model architecture but instead the sheer scarcity of real-world physical training data. Rather than just helping robotics companies manage the data they have, Encord is building a business around manufacturing the data they donât. The brain wave headset Ceja is wearing was built by Zander Labs, a German neuroscience startup thatâs betting measuring brain activity â to deduce mental states like error, intent and surprise â can create a more useful data set to train models. Encordâs work with Zander is currently a trial run; Encord says the goal is to build an initial brain wave-tagged data set, run it through customer robotics models, and evaluate whether it actually improves performance before deciding whether to scale it up. Lucas Gehrke, a Zander neuroscientist supervising the work, says that the amount of brain activity used at any point during a given task offers clues for model builders trying to figure out when they need to deploy their highest-effort models.
This is the âbleeding edgeâ of the effort to solve the robotics data bottleneck, according to Vineeth Velmurugan, Encordâs head of robot learning. A veteran of OpenAIâs robot lab and Berkshire Grey, the warehouse automation firm, Velmurugan joined Encord to build the companyâs internal data-creation team. Encord was founded to help companies building machine-vision applications annotate data and evaluate models. As their customersâVelmurugan says they work with many leading robotics firms but that heâs not authorized to name themâbegan to apply end-to-end learning to robotic manipulation tasks, executives realized they would have to produce training data themselves, rather than simply manage it. âThe data simply does not exist,â Velmurugan said. The bet that generative AI can do for robots what itâs done for chatbots keeps running into this same wall. LLMs were built on the text of the entire internet, and more. Finding the same raw materials to teach neural networks about physical manipulation is challenging: self-driving car companies collect it themselves, but thatâs hard to scale. Training from video can work, but it lacks the fidelity of real world data. Velmurugan says it will take a data set something like five times the size of YouTubeâs video corpus to break throughâa scale that helps explain why data-generation itself has become a business and not just a research problem. Feed your egocentric data needs Companies building robot brains are now turning to two main sources: âEgocentricâ video collected by workers wearing cameras, often augmented with additional camera angles and other metrics, and collecting data from robots operated remotely. Encord does both, drawing egocentric data from several factories around the globe, and using its San Leandro facility to experiment with new modalities, like brain waves, or collect data sets around specific skills for fine-tuning.
