What Happens When AI Runs the Robotics Experiments? - Jia Qi Yip

Vídeo em Inglês, publicado pelo canal SG Embodied AI Colloquium.

About this content: The summary, key points and topics below are generated automatically from the video transcript. They reflect the content, opinions, claims and predictions presented in the video and do not represent a position, validation or fact-check by IA Dots.

Video summary

The video features a presentation by Jati, lead research engineer at Melo Research, discussing the use of agentic auto-research for reinforcement learning in humanoid robot locomotion. The talk covers reinforcement learning for controlling motors, simulation challenges, the implementation of the Cyclron tool to automate experiments with AI agents, and future directions for fully autonomous robotics development cycles.

Key points

• Melo Research focuses on developing human-friendly humanoid robots and developer ecosystems to simplify robot deployment.
• Reinforcement learning is used to train neural networks (such as multi-layer perceptrons) to control 23 motors for robot locomotion without requiring hard-coded mathematical equations.
• Training a locomotion policy relies on simulation environments like Isaac Sim and Mujoko, bridging the sim-to-real gap through rigorous hardware modeling and system identification.
• Cyclron serves as an auto-research convenience layer on top of Isaac Lab, enabling AI agents to autonomously write code, adjust reward functions, and run optimization experiments.
• Experiments demonstrated that AI agents could automatically test neural network variations like LSTMs across generated terrains to improve performance metrics like path drift.
• The speaker envisions a future where autonomous agents power complete iteration loops, encompassing robot training, design, and manufacturing.

Topics

Reinforcement LearningHumanoid RobotsLocomotion PoliciesAuto ResearchSimulationRobotics EngineeringAI Agents

Video transcript

Transcript from available captions; it may contain errors.

Um, okay. So, hi everybody. My name is Jati. Uh, so yeah, I'm the lead research engineer at Melo Research. Um, and part of what we do is is train locomotion policies for humanoid robot. So, um, I'm just going to start with essentially a live experiment. Um, okay. So, this is this is the prompt. essentially you have so this this presentation should be 30 minutes and we'll basically check back uh with the agent in 30 minutes and we'll see what happens. Um so I I've done some prep uh but it should be it it's kind of run a baseline. Anyway, I'll explain all of this later. I just going to I'm just going to like start running this and then I've never done this before. Uh and this is the first time I'm doing it. So I have no idea if this is going to work at all. Um so I don't know if we can do it in 30 minutes. It's it's quite a tall order. Okay. So let me get on with my presentation for real. Um okay uh let's see. Okay. So yeah today the title of the talk is agentic auto research for reinforcement learning locom motion. Uh and and maybe a little bit of background about me. I I'm not a roboticist. I I don't actually don't know anything about reinforcement learning locomotion. Uh I just whatever I know I have learned in the last one and a half years working on the robot at Melo research. uh and so I I'm by no means an expert on this uh but uh I have done uh an experiment uh that I think is really interesting and points us towards kind of like the future of how this kind of research and robotics research is is headed. Okay. So so today I'll just briefly introduce Melo research um uh and then I'll talk about reinforcement learning locomotion and then uh this product we have called cyclron and how it can help with basically auto research. So you can help someone like me figure out how to train local motion policy for for a humanoid robot. Uh and where I think this is all headed. So so first of all research we are 35 human beings uh across three physical offices uh Vietnam, Singapore and USA. Uh and of course this is not everybody. It's just that a lot of people on our team don't even have a LinkedIn profile. So this is all all and all all the all the all the photos that like are save for work and we can put online on the slide. So there are many more people besides this that that work on the team and honestly like all the research and all the things that I'm doing is built on like all of their all of their work right and I'm I'm just doing the easiest part. Okay so like um what what does Melo research do? We we work with R&D teams to deploy robots. Um and here are some examples of the things that we've explored. semiconductor logistics, medical logistics, airport logistics, and hospitality uh across like different countries. Uh and you know um our our goal is to kind of like make all of these deployments like faster, more efficient, cheaper, and like economically viable. So um we we want to basically what we're trying to build is a developer ecosystem because we think that we can't do these deployments alone. I think with the with with deploying robots especially human robots and agents at scale what we need is like developers and we've spent a lot of effort basically making as one of the most developer friendly robots uh out there uh and we've grown to like a 22,000 discord community um tons of Twitter X followers uh and we have two events that that um there's one um that's not hosted that regularly in Silicon Valley um at Stanford the last time we were there, we had like a thousand people. Um, and then every month we have a a showcase at the Singapore National Library. So, this one has been regular for like almost two years. Um, where like everybody in the community, they they can come and give a presentation uh or talk about something that they really care about in AI and robotics. Uh, and it's like a open floor. Uh, so if any of you want to share about the stuff that you're doing in AI and robotics, feel free to come to some of our monthly meetups. Yeah. Uh and then also every Friday um actually just this morning we had a live stream of like all the stuff that like our robot team has been working on. Okay. So um we we kind of like want to do all the difficult things um so that developers can focus on the apps right like building a robot is like very hard like we I experienced the pain like firsthand um so so it and and we kind of want to target like um from like people from all backgrounds right whether you are hobbyist with just you're an Allen key in your garage or you know you're university lab or whether you're a startup that wants to build basically like a business on top of like deploying robots or whether are like a full kind of like enterprise company that wants to like re like wants to reinvent your business into something that's like robotics based, right? Um so we have a uh we we are willing to work with everybody, right? And we have kind of different ways and different tools for you at different levels of maturity. Okay. So, and also uh recently like we were all over the news. Uh we're Chinese Asia, we went next gen tech30. Um and we got this like cute plug that's in our office right now. It's made from recycled plastic. Okay. Um so let me get to the interesting bit which is reinforcement learning locom motion. Um so uh for for those of you who don't know like um in a humanoid robot like aimoff which is kind of like has 25 motors you kind of need to have some way of controlling uh all of those motors. So, so for all of those motors, it is possible for you to write down a series of equations like model predictive control for like you know 23 motors uh because there's two on the neck that you don't need to control to get it to walk. But let's say it's 23 motors. Uh it it is a lot of backbreaking work to write down how to control it and it then also becomes not very robust to like various random things that could happen to the robot. So kind of the trend that has been recently like I think most people right now the solution is not model predictive control but to use a neuronet network to to kind of solve this problem of controlling 23 motors such that your robot can walk right um so so so to train a neuronet network uh the the architecture that we are using today is actually fairly simple it's just a three layer multi-layer perceptron um with like some width uh that is variable Right. And and then uh you kind of put Yeah. Yeah. you put the neuronet network in control of all of the motors, right? Uh and you hope that the robot walks essentially is locom motion. Um and yeah, so um it the the the controller itself uh only receives a velocity command. So like which direction should the robot go? as well as the feedback from all of the motors plus the the gyroscope in the robot so that it can kind of balance itself. Uh and the reason why the neuronet network is like really small for locomotion uh is that you need to run it very very fast on only CPU. So you you you can't at least like to be reasonable. So you need to run like 50 Hz or 50 times a second inference. Uh and it needs to run on the CPU typically. So uh that's that's kind of reason why it's small. uh if you increase the compute, if you increase all of this like capacity for the model to be run and you know if you put a GPU on the robot in principle you can have a local motion policy that's like really large. Um but at the same time like walking is not a difficult enough problem to to require a giant model. So uh right now it's more or less just a three-layer MLP uh and it's blind. So it's currently blind. So if we would bring in the GPU once you need the robot to look at like to have vision and the reason why you need vision is to climb stairs and that's kind of like the next bound of research right uh but for for now today what we're talking about is blind locomotion policy just like for walking right um yeah uh and as the robot is walking you know you might imagine that someone might push it uh we will push the robot uh and we want we hope that this locomotion policy will be robust enough that the robot doesn't fall over Right. Okay. Um yeah. Okay. So reinforcement learning. So so how do we train such a uh such a locomotion policy? So um on the left you see we have basically the policy which is like a which is a neuronet network. And the way you train is that you can't really collect training data. Uh because in the first place the robot doesn't stand up right your starting your starting case right is a pile of motors on the floor connected together by pieces of plastic and metal. Right? Right? There's there's no training data to collect cuz like you the physical robot doesn't work, right? And there's it's not a supervised learning problem uh for the neuronet network, right? What it is is that you need a simulation. You need to simulate uh the robot walking in a virtual environment with like simulated physics. Well, yeah, with simulated physics, right? Um and then get the policy to try to make the robot walk in the simulation, right? Then you hope that after the policy has basically gone through this loop, right? Basically train itself in simulation, right? By basically attempting to control the motors to walk, right? Um after a while, uh it it kind of starts walking in simulation and it stops learning, right? The the policy basically converges um and and the robot is walking beautifully in your computer, right? Then then then is the leap of faith uh which is to take the policy and then load it onto the robot and then the robot either freaks out or it starts walking, right? Um and honestly there's there's no way to know one way or the other whether you've actually solved the problem. The only way to do is to do the empirical test which is take the policy, put it onto the robot and then debug from there. Right? And so that that's like a whole presentation by by itself, right? the sim to real gap. Um how do you how do you close that gap? So that means how do you make sure that your simulation is like kind of close to the robot? Um yeah. Yeah. But that that's like another half an hour 1 hour uh presentation. But basically um the high level is that as long as your robot is deterministic and you understand every aspect of the physical hardware and you model every kind of like parameter that you can identify in the simulation then you have closed the simil gap. Most of the time the synind is not close because you don't understand your system under test well enough right um and that's kind of like what we spend like a good like portion of a year trying to figure out right um and now that we've done it now that we've done it the similar gate is closed which means that you can basically do this loop and then uh expect that the policy would run on the robot right um and and and okay once once this loop is done right basically uh uh how what how do you how do you tell the policy okay when it runs you know makes the robot walk in the simulation uh how whether it's doing a good job or not because to train every neuronet network you more or less need a uh kind of loss function right uh so so in in in reinforcement we call these reward terms uh and and basically what you need to do is describe using math or using some calculation or some reference or uh some parameter right what walking is right so you You can't you can't explicitly write down okay like walk right you need to say okay you need to lift your feet off the ground you need to like bend your knees you need to follow the velocity command I give you in a certain to a certain kind of degree of like closeness right um so so you can measure all these things but you can't kind of explicitly describe what walking is um you can give it reference motion so so that's something that we do do so you pre-record some walking motion either from a human being or by generating it or by animating it by hand and then you give that also as a reward like how close are you to that reference motion that is good for like making the policy have the like have a good start like a warm start but over but after a while right um using reference motions kind of like has diminishing returns because walk walking straight is very straightforward right but then now I need to walk straight and turn left and walk straight and turn right or walk up a slope or like walk over rough terrain suddenly like you there's an infinite number of pre-programmed motions and combinations of environments, it's uh for you to go and pre-record which basically makes the problem impossible. So, uh you you you can use pre-recorded motions for bootstrapping kind of like a good starting point for the model. Uh but but after a while, you need to rely on the real like actual reinforcement learning with the environment involved, right? Yeah. So, um yeah. So, that's that's called a style prior, right? And then for scale, this is actually just a money problem. Uh so if you have a lot of GPUs, you can just uh have like a huge number of environments, right? And what that buys you is a training curve that's a bit more smooth, right? But if you only have like a desktop gaming GPU, you just reduce the number of environments. Your training curve is just like more noisy, but you will still get there in the end. So it's just a function of like it's a trade-off between your your your money and your patience. So it's is always doable because this is at the end of the day very tiny model, right? Uh so so yeah this this is within within the realm of possibility for as long as you have a gaming GPU. >> Yeah. Okay. >> What have you all found to be the best simulator? Is it is that same? >> Okay. So we do both. We do Muoko. We started in Moko. Moko was very good for the system identification stage because it's very easy for us to iterate. But after a while we moved back to Isaac Sim. Okay. Okay, we started Isaac Sim then we felt we found it very difficult to kind of like iterate quickly. So we moved to Moko and found it easy to iterate quickly but after we figured out the the SIM derail moving back to Isaac sim was better because they actually have more stable physics engine. Yeah. So so so the policies that came out Isak sim once everything has been sorted out sim just like just like goes. Yeah. So, so for early experimentation, if you have a brand new robot, then use Mudoko to try to figure stuff out. Um, then once you figure stuff out, go back to uh go back to uh Isaac Sim Isaac Lab and and scale. Yeah, that that that would be that that's our approach. So, so we found Isaac lab to be more more deterministic. Yeah. talking about the sim to real you have other stuff >> uh like how much of a additional >> training and so so sim to real is not a model training problem sim to real is an engineering problem um so I I can just say like high level we have a blog post on this on our website so sim to real is an engineering problem because basically your simulator can simulate anything but your simulator needs your simulator needs to know what to simulate right and to know what to simulate you need to understand your system which is an engineering problem, right? Uh you need to ensure that you understand your system and that your system is deterministic like which is actually a tall order in in in in modern software. Um deterministic to the point where like I know exactly the timing of all my signals coming from my motors, right? Because the moment there is some variation in the in the the timing of the signals coming from the motors, the whole thing gets thrown off, right? So you must know like when the signal is coming and and what what what is the range right and the range must be well known and well modeled. Then the other thing is you cannot have like random drops. So so so so now I'm just like listing off all the various problems we try to fix right. So first of all it's just modeling the real actuator latency. So the latency between when your policy receives it through the software stack through the firmware then to the canvas communication that entire latency must be well known and must be deterministic. So you must like know and it must come on a regular schedule and things like that right which is like which that alone took us like a months to go and figure out right while figuring this out we realized that like you know even like what communication protocol you use to talk to the motor matters right so so we had a communication protocol that was faulty like the silicon we were using the RK3588 didn't support that communication protocol in the latest version right because the silicon itself was like had had some issue and so they stopped supporting it. But we didn't know cuz we read the old documentation. So then basically at the middle of a demo, right, like the robot just freaked out for no reason at all, right? And it was because there was this bug in the silicon that that basically was unfixable. So we had to redesign the motherboard to use the other CAN communication protocol so that we can have like the good deterministic like communication with the CPU. So like down to that level of detail you need to make sure that like you know all your signals are kosher right once that is done once you understand that all of these problems once you iron out all these kings yeah then then this is this is becomes very simple right um and and what we've done is basically yeah we provide everybody this hardware um and we provide everybody with the the accurator configs that we use internally uh so that actually like everybody can reproduce our results without having go through having to go through the pain that we went through right um Even even the the the the the carrier board that we designed uh for this uh is is open source. Yeah. Yeah. So, so yeah. Um it's it's just it's just it's not very interesting research. It's it's it's really just like war stories of unexpected crap happening. Uh I I was actually not like deeply involved in like a lot of this like it was the engineers at MLO Research who worked on it. So, so this is kind of like the high level story I can tell. Yeah. But a lot of this is captured like on our blog post and if you come into our discord, you can ask us questions about it and we're happy to answer. Yeah. Okay. So, um let me go back to kind of like the training run, right? So, so we we we have like the locomotion policy kind of the sim is done. Now, we want to train, right? So this is kind of like the end to end use like user story of a training run for locomotion, right? First you have an idea like oh I think this this this reward or this this new metric I can add to constrain the reinforcement learning uh is going to be useful to achieve some behavior on my on my policy. Right? So that's like the design phase. You sit down and you think, "Oh, I have this idea." Right? Then you you write it out. You you craft the reward. Then you send it to Isaac Lab. uh you you you you create the task then you train it right um and then this this one is like depending on what GPU you have right you just um you just just train it for a bunch of hours um then after you've done training it you export the model right so when when stuff is being trained on GPU right uh uh you need to kind of convert it to onyx so that it can be run on CPU on the robot so you're going to export it then you need to benchmark it so benchmark is like done sim to sim so we use Isaac SIM for training or Isaac lab for training and then we use Mujoko for simtosim testing right and the purpose of that is basically to check whether our model overfeit itself on the particular physics right which is is is almost like superstition at this point right like in the early days of physics simulation this was kind of necessary what we found is actually not extremely necessary but it's kind of still good to have right because these days the rewards and all these things people have figured out the model generally does not overfeit itself on a particular physics simulation But it's always good to check. So, so the benchmarking basically is in Mudoko um just to check right by right actually if you done your system identification correctly right uh if you do everything in the same physics simulation you just do everything Isaac lab actually can also deploy just close your eyes and deploy uh but but I mean it's still good practice to go and check so you benchmark right but also benchmarking is good for you to know whether the policy you train actually made an improvement right uh and you kind of want to be able to do this quickly at scale without having to go and test on the physical robot right uh because then if you have a hundred different policies or a thousand different policies because uh you can train this very fast, right? Then nobody has time to go and train or or validate and test like a thousand policies on the physical robot, right? So you kind of need kind of this simulation benchmarking to to to see that you've at least like made some improvement, right? Um and then you decide whether you want to run this policy and decide whether this policy is good or not, right? So this is kind of like the the whole workflow. Each policy that we train through this thing usually takes us a day. So we have a cadence where like you know basically we the evening the day before we just train a model then the next morning we go and check right and then we test and then we come a new idea then we at the start of the workday then we send another training run then in the evening we test again so you can like do two rounds like it's a 12-hour clock kind of thing right um so so that's kind of how we did it right but there's like this issue which is that reinforcement learning is very prone to collapse it's very hard sometimes it's just not within your control sometimes you maybe set a parameter wrongly. Sometimes you set the parameter correctly. Uh but like reinforcement learning is just like black magic. It sometimes just does just fails, right? Then when the reinforcement learning fails and it's in the middle of the night, nobody's going to see it. So then you have this model that is like training and wasting GPU hours. Um because you only discover the next day, right? Um so so that happens. That happens very often. Um uh and one of the things we try to do is try to prevent that from happening. Um uh so so yeah. So, so how do we fix this? Right? Um, that's where auto research comes in. That's where an agent comes in. Um, so cyclron is basically a thin wrapper or convenience layer we put on top of Isaac lab uh to help you essentially run all your experiments uh and actually help let the agent do all the experiment. Right? And this is basically how it starts, right? So, so this was my experiment of experiments, right? Uh I I was too lazy to go and design my own reward. So I just went to clock code, right? So So I didn't even design a harness. This was just raw clock and I just recently learned about this feature of clock code which is /go. So so if you go to clock code, you put /go, right? And just tell the agent, look, help me explore, right? Uh and then and then just and then just I just kind of verbal diarrhea onto the onto the terminal, right? And then and then and then and then I say, "Okay, you got four GPUs. just go and run this thing, right? Uh and then over the course of a few days, so I ran a few experiments. I ran a few rounds like this. Over the course of few days, I burnt like $4,000 worth of tokens. Uh and then and then and then I have a better policy. And to be just reference, $4,000 seems like a lot. But when we originally went to China and asked someone who was an expert in this locomotion thing to help us train a locomotion policy, the guy quoted us $250,000, right? So actually $4,000 is pretty cheap, but also this not the real cost. This is like the estimated cost of the tokens, right? Uh but yeah, so so basically the this when I send this goal, this is this spike over here. Then then the basically the the this is all the instructions I gave it, right? I just like have this high level. The basically the thing I wanted to do was like try a new um try a new neuronet network. So, so we've always used kind of multi-layer perceptron and an experiment I always wanted to run was use LSTM, something with memory and something with history so that you can kind of like potentially be better at reacting to things, right? Um, including stairs. So, to climb a stairs blind, uh, if you have no eyes, basically you trip over the stairs and pray that you can catch yourself before you fall to the ground, right? So, so with with a history with a policy that captures history, you have a chance of doing that, right? And so, that was just the hypothesis I had. So, I just gave it to the agent and then now now the rest of this slide is more or less just the agent telling his own story. So, I'm just I'm just being the the Oh, oh, okay. No, no, that's this this. Okay. So, why did I think this will work, right? This very unhinged uh unreasonable way to do research, right? Um but why is it why is it why is it why why why why would this work, right? So first first of all in this particular use case manual research is like done like a ton of work with the simulation uh with the system identification um and the the framework and the benchmarking everything is like in place already right so a lot of human effort has already gone into like making constraining the experiment right and it's quite hard to cheat and if we've already created like the measurable metrics right on the benchmarking side to check whether uh the model is performing well or not. Right? So, so we've already we've already we already have a good we have a we have a well- definfined benchmark and we have a difficult like it's very difficult for the model to cheat. Right? In order for a model to cheat in a way that disrupts the deployment of the model, you the model basically would have to rewrite the physics engine. But even clot doesn't want to rewrite a physics engine because it's really just such a pain. But right so so it's not going to rewrite the physics simulator. Uh but but of course when I do slash go there's no guarantees of that. Um, so if you had a auto research harness, it would try to check for that, right? So, so, so that's number one. Number two is there's no theory. The reward shaping has no theory one. It's it's really just an empirical science. You just, it's trial and error until you get the thing that kind of looks nice, right? So, so it's a numbers game is like how how fast and how many policies you can train and how fast you can filter all the thing that comes out, right? So, so it's it's it's just it's just it's just an empirical numbers game. There's no there's there's very little theory or at least I I don't know of a very convincing theory of of of of reward shaping. Um >> you still can write papers. >> Yeah. Yes. Yes. Yeah. Yes. Yes. Papers. >> Yeah. Yeah. Yeah. Yeah. Yes. Yes. Not not Yes. Of course. Write all the papers you want on on on the on the thing on the thing. Yeah. Uh, and you can also just get the agent to write the paper and get the agent to read the papers that the other agents wrote and then the whole thing is just an auto auto farm and you can have coffee at Laurang AI while all the journals are being ddosed. Okay. Um, and then also the frontier models are really very good now. So, so if you just This was not possible like 6 months ago, right? like the this lack of this lack of um kind of like precise instructions was not possible before like the fable five era of like frontier models right if they've basically just reach a new level of capability that wasn't there before right when I started this I had no expectation that it will work I just wanted to see what the baseline would be right and then it just worked so then I gave this talk because then I didn't have to do any more work um so um so so so basically okay now now these all the agent slides right So, so this this next few slides done by the agent. Um, yeah. So, so, so basically I set the goal and the metric and then the agent just writes all the training code, write the reward function, all these things and then it ran like a few studies, right? So, first of all, I one of the biggest problems of our robot was that it didn't walk straight. So, if I let it like if I gave it a velocity command of only in the only walk straight over like as short as like 5 m, the robot will drift to the right. And I had no idea why and like nobody can tell me why. So I just like agent can you just make it walk straight right and actually it just did it. So it's very simple it just walk straight right and then and and I can see in the benchmarking and that that it does it does walk more straight than it did it did before like the than the baseline. Then I asked you to study LSTMs, right? Because I I had this idea that like LSTM would work, right? Uh it did not really work, right? So LSTM didn't really help in the scenarios that I was giving it, which is just walk straight on a flat ground, right? So I was like, ah because like when you walk on a flat ground, you don't really need to know history because the the ground before was the same as the ground ahead, right? So then I was like, okay, how about LSTM plus terrain? That means like make it walk stairs, make it climb the hill and all kind of thing, right? So, but then like am I going to personally design the stairs in the simulation and the slope in the simulation? No, I'm not. So, the agent just did it by itself, right? So, it just go and I mean stairs are very easy to define. So, it just wrote this as a mathematical function and the simulation just spawns it, right? And then and then the robot just walks itself over all of this terrain that it generated for itself and just learns how to overcome all of this stuff, right? Uh and and and with this I think the LSTM finally kind of like proved itself, right? So LSTM actually does walk better over all all of this terrain. Um and and it actually can climb stairs. So uh and actually it's not just me by the way. So um there's there's a guy at Melo Research which is basically in charge of like our growth and marketing, right? Um he's in charge of all our social media accounts, right? And he wanted to understand what our developers would be doing. So he ran this entire thing himself as well. So separate from my experiments he also did it and he achieved the same result as me by using a coding agent as well. So uh is reached the point where our marketing guy can also train locomotion policies. So um yeah and but once you have this measurable metric right once you have something for the for the agent to optimize against it can basically draw this graph right and you can basically observe and and kind of track its own progress right so you can see MLP at the top MSTM at the bottom right the agent just optimizes towards the goal that you give it right uh and it just keeps doing training runs and it doesn't do like really long training runs that the way a human does because human need to sleep right it does like 200 like iterations at a time observes and then makes a small adjustment. it can do it can do these things very rapidly and it's always awake and it's constantly adjusting its method to kind of do it and even as during the course of a typical training right so over here 8,000 iterations right in the past we will get one experiment out of these 8,000 iterations cuz I would just be sleeping right but now it's like all of these dots are like different interventions that the agent made along the way right with different iterations because it can just keep starting and stopping experiments whenever it notices that something is going ary right so it it is that kind an outer loop of an inner loop training, right? Uh so so that's kind of like um how how how it kind of optimizes itself, right? Um and so yeah, it based on the metrics that I gave it, which is path drift in in the sim to sim benchmark, right? It kind of um achieved the goal. Uh and it also achieved the the the terrain walking. So it doesn't it doesn't fall anymore. Although actually if you look the MLP also did the job, but I I I think the MLP might have uh overfitted. Yeah. Okay. So, uh this is just this is just some uh yeah this is the same thing as the first slide. This AI slot. Okay. Um so so like uh this these are all some of the things that I mentioned. Um yeah the the the the agent is always awake always tuning always tweaking the the training run that I would never be able to do right and the judge is already like defined by me right I I designed the benchmark right and I tell the agent like optimize this thing right and the agent doesn't basically stray from that doesn't try to cheat um as far as as I know right but we can always check right we can always hash the code and things like that right and then as a human all you need to do is like take the policy run it on the physical robot and figure out what's the next thing you to improve, right? Or or basically just just just demand what you want from the locomotion policy, which is actually the most fun part, right? The training of this of the thing, designing rewards, that's not the fun part. The fun part is having a policy to deploy on the physical and watching it watching watching it run and then like coming up with new ideas for what you want the local motion policy to do, right? So, we keep the fun part and then let the agent do all the all the backbreaking work of tuning random things. Okay, so here's the result. Um okay so uh on on the top left you see the original baseline and then yeah so after 35 iterations it still falls the the ground and then the last two bottom two it basically has solved the problem right and all of this is done autonomously even even the videos so so even the videos was done for me uh I didn't have to do I didn't all I had to do was write a paragraph in the terminal right okay So next, okay, where where where is this all headed, right? Um, so I think like everybody's looking for a robotics chat GBD moment and I think it will come not from like a very good model, right? But from a better way to deploy robots, right? Um, and the reason why I think this is that like CG GBT was not a model. ChatGB was an application. It was an app built on top of GPD 3.5 which wasn't even that good, right? right? It wasn't state-of-the-art, right? Um and and I think like on top of a very good model, right, you also need like all these things, right? You need basically the application and the robot is the application, right? Um and what we've built at Menllo is basically a legged humanoid, a nonlegged humanoid uh for all possible deployment use cases. Um we've we've built in right to repair with our open source hardware. Um it's customizable. So the version two of Azimov is like customizable and we'll share more about that in the future right and we have a global supply chain and it's AI native right and I think the AI native part of the robot is kind of like the most important thing right like the new foundation models have given us kind of like a new way to do robots right that's very different from the way you traditionally do robots it's it it is it's become kind of like a lot faster a lot easier Right. Um, and this is an example, right? So, this is currently how we do the robot deployment, right? We have like this, this is our own kind of internal, this is the various people around the company all doing various things to handle a deployment, right? So, we have some agents, right, that are figuring out, oh, how do I read the SOP? How do I encode this? How do I program a robot at a high level for long horizon? Then you got a bunch of people like me trying to make the robot walk, trying to make a robot manipulate stuff like individual skills, which are two calls for the agent. And then you have like the hardware team that's doing doing all the system identification, doing all the sim to real work, right? To do hardware prototyping, like making the robot exist in the physical world. And these are all iteration loops, right? That hum that we as a company are doing, right? Um and then you have a bunch of guys who basically handle the simulation. The simulation itself is like a really like a huge problem, right? Like uh how do you have a digital replica of the deployment scenario, right? That everybody can iterate against, right? Because iterating in the real world is just very slow, right? Iterating in simulation is much faster, right? And then you have a bunch of people who will take the robot and bring it to the client, deploy it there, and then find that there's a bunch of problems and then go and fix all those problems, right? So then there's this outer loop, right? And then this goes back to the original thing which is finding the use case, right? Finding your use case then feeds itself back into me developing all the solutions to to do handle the use case, right? So these are all just a giant loop and like as a company this is how we improve right we iterate and iterate and improve every single day right um and I think we had a slogan of like making the robot like 1% better every day right so this brings me to how I can basically put all of us out of a job now so every every iteration loop right that Melo research is doing can be something that an agent does right so today I talk about control policy auto research which is basically putting myself out of a job cuz like I don't need to actually go and manually go and train policies, right? Um the agent can go and do that, right? So then I can go and do other fun stuff, right? Um there's actually three more talks I can give about like three other loops that Melo research does, right? And and our agents can take over, right? Um the so for the skill level, right, uh is is kind of RL auto research. There's an automatic kind of deployment auto research loop for the agent level, right? So that that agent basically improves itself at doing the the task that we give it. So uh the the deployment task so the high level long horizon task and then there's recently there's been a lot of developments in agentic cat right because the foundation mod are really good now they can actually do CAD right which means that mechanical designer doesn't have to go and handcraft every everything it just tells the agent like look go and make this thing right and Azimov 2 is designed exactly for that. So Azimov 2 is designed to be parametric which means that you just the whole robot is just a function right. So you give it a set of you just give it a YAML file. I mean give our system give the Azimov tool kind of configurator a YAML file of the constraints you want for this robot and then like you have a deterministic function that generates the robot body for you and that can instantly be printed. Right? Which brings us to like the the third part of the loop which is robots building robots. Right? Once you build a robot that can do anything, the first thing that robot should be doing is building itself, right? Once that robot can build itself, then you can have unlimited robots, right? Um, and having unlimited robots is the best thing for reinforcement locomotion because if the robot breaks, you can just have another one, right? Um, and so kind of having that loop uh circle back and close um will give us basically recursive self-improvement for robots, right? And that's kind of like what we're trying to build towards, right? a robot that can build itself, a robot that can train itself, a robot that can deploy itself, a robot that can design itself, right? Once you once we've been able to close this loop, basically um everybody at Melo Research would be able to have will be able to do the fun things and not do the boring stuff, right? And that's what we're trying to achieve, right? Uh and there will be like an agent that's powering like the entire thing or maybe humans will give an opinion once in a while. So this brings us to accelerationist robotics, which is where I want to end this talk. Um, I think we we have reached the inflection point for robotics. Um, with the latest models, uh, we're going to be able to kind of build robots, design robots, train robots a lot faster than before, right? And yeah, we're at the we're at the bottom of the J curve for for robotics right now. And I think that what we really want to do is like accelerate into this uh, and just basically make make robots exist everywhere and do lots of useful economically valuable tasks. Yeah. So, um, thank you everybody. move on to Q&A. Uh, who would like to get the ball rolling first? >> Any questions? Okay. >> Um so you said before that uh the you know research agent is able to do this because uh the human already do some kind of I guess for streaming. So I want to ask Si is that the actual psycho trauma research that you freer is that the stream >> that you were talking about or something else. >> Well, primarily it's Isaac it's Isaac lab. So Isaac lab is already a set of a like it's already a set of pre-made scripts. So all the training code has already been written in Isaac lab like what we've done is provide the actuator configurations for like the sim to real kind of configurations for Isaac lab. Uh so so but then most of the training is Isaac lab. Um, and it's kind of like it's it's it's already it's already there. Uh, and and and all of that work belongs to Nvidia who who's who's done it. Yeah. >> Right. So, I guess it's also the fact that the task itself is kind of you know just a t and >> ah yes >> boring task. >> Yeah. Yeah. >> Yeah. Okay. So, the training is really written right and then you got a list of rewards. So we have written out some of the rewards of ourselves but because the reinforcement only has a bunch of rewards right like let's say you got like seven maybe got eight rewards and then you need to adjust like the weights of each of these rewards and maybe there's some math that can help you auto adjust but like what the engineers seem to be doing today is just manually adjusting weights but like I think like if you do multitask learning in the rest of you know AI research you would know that these weights can be automatically adjusted uh but right now I just get an agent to do it. Yeah. >> Yeah. >> Okay. Next question. >> Hi, thank you so much for sharing. Um just like a question on like what would be next. If you can basically automate the entire loop from training to manufacturer, then do you think like the the next stage or like I guess the most exciting part would then be like optimizing the most versatile like locomotion policy policies or like >> what what do you think would be like the most valuable effects of micro models? >> I I think there's two schools of thought. One is that okay you can use this to accelerate research into something that's like a very big general model that can do everything or because it's so easy and cheap to train small models you just like have bunch of specialized models that are harnessed and chained together by a by a very powerful agent right uh I don't right now I'm leaning towards the composible skills with a very powerful agent in control of everything um be and I I think that's that's that's my my vision at least. Yeah. But I don't know if it's going to be correct. It could be that like a all powerful model will take over everything. But but a model that can that can improve its own skills is going to be more powerful than a model that like knows everything from the start because because it can't that you can't retrain it, right, with the new use cases and things that come out, right? So the model needs to be able to self-improve, right? Yeah. On like the specific thing. It's like kind of like YOLO. YOLO where it was trained was like general use case, right? >> But then like anybody who deploys YOLO in a production setting always fine-tunes YOLO for like the specific thing and I think something similar will happen here. >> Yeah. >> Yeah. >> So is local motion solved here? >> Yeah. I I mean it's basically like I mean there's probably like stuff to do and interesting stuff to do but at this point yeah the agents will do I mean there's still stuff to do stuff for the agent to do but I wouldn't want to be a >> general walking of for your humanoid is kind of one time thing you're pretty much done as agency and everything. >> Yeah. Yeah. Yeah. >> Then yeah then the extension question is uh what's the experience doing the same thing for manipulation? Uh so so so for manipulation is a lot harder because then we have to start solving SIM to real for cameras which we are we are which we are doing but so the main blocker is the sim for cameras and camera calibration and things like that but once you solve that the same pipeline will work >> right yeah >> okay so actually for going back to local motion you have that problem for if you actually want to have the >> vision >> the vision part of it created that's your open question is that open question or is that just you haven't done it yet >> we We haven't done it yet. I mean, people have done it. >> Uh we are we are not at the frontier. We're like maybe 6 months behind the frontier uh uh in when it comes to this because we're not full-time researchers yet. Yeah. >> Why is that conceptually it shouldn't be difficult, right? You can just get it to do all the research. >> So, yeah, in sim in sim is trivial. I can I can train a vision based like the first thing we did was train a vision based locom motion policy but it's not deployable because there's there's the engineering problem of the camera and sim is not the same as a camera in real right there there like a bunch of like camera intrinsics things to get right yeah >> but that that does not strike me as the difficult challenge >> but it's I mean maybe we're just not good enough engineers but for us it has been pretty difficult >> as as camera is fitting something you can just calibrate. There's there's no models for calibration. >> Yeah. Yeah. Yeah, there are. But we we've only recently hired the talent that we need to go do that. Um but then also on his first day, we needed to fight a different fire. So So then he's just never gotten around to doing it. >> Okay. Then in that So then going back to manipulation, >> um you suggest that the biggest challenge is just the vision side. >> Yeah. I mean >> what about the interactions with the objects and uh daily safety and resets of the >> test those are um >> like touch sensors >> no just interacting with the scenario right >> so that you can test and then you need to reset the environment to try again >> over is that a challenge or you don't >> or you think your framework will just work in >> so we have very initial experiments using momo bot uh momo board which basically train a whole VA model in simulation and then run it on the real it works. The problem is that in the approach uh for VA's training in simulation um the the the the approach fails a very basic uh benchmark which I call the clumsy intern benchmark which is that if my camera I put on the table and then the today the VA works if someone bumps the camera and the camera is not mounted properly and it shifts like by slightly by 1 cm then the VA just breaks right it's very brittle to this particular translation, right? And this is built into the vision encoder itself, right? The vision encoder has patches, right? And the moment your camera moves, right? Every pixel shifts. And when every single pixel shifts, the embedding space like shifts to a completely new domain. And basically none of the pixels in this domain, right, is part of your training data anymore, right? So then you you need to have a ton of training data. Um, so it's also a scale problem. I think over there will agree will say it's just a data problem. >> It's a well >> that's not >> yes I I think that a lot of people in industry will say this is a data problem and yes man research basically too poor to go and solve this problem. Um uh but I I personally feel this is also a model problem like a model architecture problem. I I I I think we're not being data efficient enough with the model architecture. Um, and I think there has to be a different way besides this way, right? Besides throwing data at it and and and given our constraints, that's what I've been thinking about, but I I I don't really know the answer. Yeah. >> Yeah. >> Anyone else like to continue? >> Hi, thanks for sharing. So um I also share other tech solution for lot AI and as you can see development also accelerate u robotics training. So also one question if for current speed in near future from your side in which field do you think the commercial application may apply in the near future or in the meter? >> Oo I would say I mean let's just bring it to Singapore right because that's where I think I have a lot of experience. Um I I I think it's any place where there's business to be done but no enough humans to go and do it right and and basically in any industry where that is the case um that there's opportunity right and so far we've seen a lot in logistics uh simply just moving boxes around is like fundamental to the economy and there's not enough human beings to do it. Um so that's that's one I think that that has has has is pretty good. I mean Singapore is also like one of the most sophisticated robotics. I think credit to maybe what NRP has been doing the last decades, right? Like Singapore is one of the most has the most automated warehouses and the most number of robots per capita, right? Like but everything all the lowhanging fruit has been has been eaten really, right? What's remaining is really like the most difficult task, right? But I think that's what is exciting about what we're doing. Yeah, I question also got some friends in the industry and there's a huge actually huge manufacturing industry in China and most my friends told me it's still most of their commercial application is still bought by uh institutions, universities, researchers and some even more performance only though it has huge manufacturing industry still a lot of things can be automated. So that's why also this triggers a lot of interesting uh among my friends are focusing in near future if there's a breakthrough in the technical training. So may also mix in the commercialized application what we can have in the future. I think I think it's not a technical problem, right? Like like maybe this is a bit controversial. Maybe it's hubris because I I don't really know much about robotics. But the I I don't think there's anything that we want to do that is technically impossible, right? Um the problem is that it's not economically possible because it's too expensive, right? So the breakthrough that's going to come is just making it cheaper, right? Um, and it's not just making the robot cheaper, but it's making the deployment cheaper. Like the total cost of that deployment cheaper and making the iteration cycle faster. And because time is money, right? Like I can I can take a I can take one year to do if you give me one year and unlimited money, I can do any robotics deployment, right? But the thing is that I have like two weeks and like no money, right? To do a P, right? So so so, so then actually bridging that gap, right? going from like one year deployments to like six months deployments to one month deployments to two weeks deployments that that that is that is what is going to get us there. It's not like technically speaking almost anything can be done with a combination of traditional robotics and like a little bit of AI, right? It's not AI that's going to solve everything. AI is just that last piece that accelerates everything, right? All the all the work that has been done before is not going to be thrown away. Like traditional robotics is still very important for enterprise deployments, right? Like all the traditional robot control, traditional precise robots in addition to like these like indeterministic AI powered robots. Uh it's the combination of the two that can get us there fast enough for any given deployments, right? Use the AI to make up for the limitations of traditional robotics. Yeah. >> Yeah. Um hi, you mentioned that one of the challenges with the sim to real for using or or rather for using vision for manipulation was sim to real being challenging. Yeah. >> Has Medllo considered or will you consider using other sensory modalities such as propception or even tactile feedback to perhaps establish correspondences with visual information? >> Yeah, I I think that's one of the biggest gaps. So So I I did run a small experiment on this. Basically, one of the problems is with the vision encoder is that the moment the hand of the robot is outside the frame of view of the robot, the robot doesn't know where its hands are, which seems like really ridiculous to me, right? uh and then the way that some of the earlier VAS do I think today the the newer VA already try to solve this problem but the way they do is that all the vision encoder uh the vision encoder uh tokens are entangled with the propriception of the robot so the robot does in a sense know um but the the the propension was not entangled with the vision of the robot right and what I tried to do in an early experiment is try to entangle these two things um it it kind of works so so so basically if I close my eyes I should still know where arms are that's fundamentally the the main principle and I think there's a lot of untapped like research potential in this idea. Uh so yeah if is that something you're interested in working on happy to try to collaborate on that. Yeah. >> Thank you. >> I I I I seen a lot of intelligence. I think the question right now is like like you say on department I think right now the economic sense seems to work whether it's the data whether it's the actual department the challenge are you putting on the spot uh how far we from the check GBP year for is it three years from now is it five years from now I mean I was our view was 3 years 4 years away but we seeing a lot of the V that is spend a lot on tokens the power law seems to work putting your spot uh is it 2 years one year 3 years away >> I just I I if you ask me like at the start of this year I would say five years but like you ask me today I think it's like six months like within six months you'll start to see AI agents uh start start solving all of these problems like the the frontier models have moved very fast in this direction and over the next six months you'll start seeing PC's not from not just from us but like from everybody else that that all of this is going to start to work. Yeah, that's that's my my prediction. Yeah. So by maybe June next year you will see kind of deployments run by agents. Yeah, >> thank you so much. Uh really really fun uh presentation uh and insightful. I I'm just really curious more so like uh on two questions, right? Uh >> yeah, >> I think Rich has you enough on that and >> um you mentioned a little bit about ELAS as well, but I think there's a lot of talk around like uh perception and action models >> and of course world models as well. U just curious if if that's something that Menlo is kind of experimenting with. >> I mean, yeah, once we are done shipping robots, we will get around to doing all the research. uh uh on on on all of these um on manipulation on on on kind of perception on proprioception on touch sensors I mean there's a lot that we will build ourselves but there's a lot that we're looking for partners to work with on right like it's it's too big of a of a of a task for us to all take on by ourselves and we are definitely looking to partner with people to work on a lot of these problems >> um second part to that question would be uh like there's also an acceleration in market towards like robot brains, right? So like robotics uh or robot intelligence also, right? So like curious to know if like within your whole open source value chain uh is that something that you guys would be thinking about when when kind of like owning that part of the value chain? >> I mean a lot of our customers are precisely trying to build the brain of robots and they just want a robot from us to go build the brain on top of, right? And we started wanting to build the brain of robots to to put in a robot, right? And and that's how we ended up building our own robot, right? We bought a Uni3 G1 and then we put a brain in it and then the Uni3 broke after two weeks. Um and then it took two months to repair. Uh and then in that time we're going to design our own robot. So so that so that we can own this part of the thing and we made it open source so that everybody else can own this part. So that if you are someone building the brain of a robot, you're not going to be beholden to like the repair the repairability of your manufacturer, right? You can you can basically if you if you are operating the robot you should be able to repair the robot like can imagine you're a taxi company but then like every time you need to change the tire you need to go back to the manufacturer it doesn't make any sense right so that's that's kind of like our value prop and what we're trying to do yeah >> thanks anybody else any questions for >> what's next for what should we faster and faster speed. All of this will come and move a lot faster and it's going to be Asian power all the way. Yeah. Yeah. >> Uh this is hardware wise. >> Oh, hardware wise, uh our AMR is coming, our robot arm is coming, and we have basically three new products in the pipeline for 2027. Yeah. Just to follow up as well, um I'm just curious like in terms of chaining different locomotion policies together um to see like a variety of different movements like let's say running to climbing up stairs to walking. >> Do you see that that in itself would be a whole another level of complexity to to take on or do you think like again within the next six months these agents will be able to to coordinate multi actions together? >> I think they already can. um and the locomotion polic so there's a balance right the agents can chain these things together very quickly because basically they the agents can be in charge of the vision and you can basically hot swap the policy underneath that's one solution to the problem the other solution problem is you have a locomotion policy that has vision itself and it generalizes across multiple terrains right um what we want to do at Melo Research is bas basically build the firmware and software to support both of these solutions right we're not going to bet that one is going to work over the other we want to make sure that no matter what happens. We we have the hardware to support people. >> Okay. Any final questions for Jesse? >> Are we going to circle back to what you did? >> I was going to say I can try. >> Oh, yeah. >> Let's see. Is it going to work? Where is my video? Uh, okay. Yeah, I think it did. It did. Train. Let's see if I can find a video. Where Where is it? Uh, wow. This is a bit hard for me to see. Where did you save it? Okay. I mean, do you guys have time for me to go? Ah, tada. >> Don't worry. >> Okay. Uh, that's a lot, man. Um, is there a video here? Oh, I can open this actually. Uh, I need my Okay, let me just download this and I'll just So, let's see. Did it download? Okay. Normally, I will open live server. I don't think live server is uh installed here. So, this is the report of the benchmark. So, this is Yeah, it it basically worked, but Oh, no, it didn't work. it fell down. Uh so this is the benchmark I fell. So I mean 30 minutes of training is not sufficient. Uh but it did uh do all of the all of the training. It just didn't succeed. Wait. Um yeah. So yeah. Okay. Okay. Let me try to find I don't know where it started. Uh where did it say the video is stored? Um both videos are rendered scratch 3026. Oh, I found it. Okay, good. Okay. Okay. It didn't do anything. Wait, let's try forward. No, I don't think it 30 minutes is too little. Yeah, it just does that. At least it kept itself straight as it fell down. Uh, but yeah, so you can see from the benchmark, it just falls down. Yeah, usually you need like a couple of hours to to do a full training run. Yeah, but he tries his best. He tried his best. Yeah. So, there you go. That's state of the art logo motion training right here. 30 minutes. All right. Let's give a round of applause. All right. Thank you.

Assistir ao vídeo original no YouTube

Assuntos: Agentes de IA, Robótica e humanoides. Publicado no YouTube em 2026-10-31.

Seleção automática por assunto e idioma. O vídeo pertence ao canal de origem.