Scaling AI Inference with Arm AGI CPU, Rebellions RebelCard, and OCP ORv3 Rack Design
Publicado no IA Dots ·
Agentes de IAAGIAI Signaldot-agidot-ai_agentsdot-artificial_intelligenceidioma-enInglêsyt-D1y1TEBqIb0
Vídeo em Inglês, publicado pelo canal Open Compute Project.
About this content: The summary, key points and topics below are generated automatically from the video transcript. They reflect the content, opinions, claims and predictions presented in the video and do not represent a position, validation or fact-check by IA Dots.
Video summary
The video presents a presentation by representatives from ARM and Rebellions discussing market trends in token generation, agentic AI workloads, and the collaboration between ARM's AGI CPU and Rebellions' AI accelerators to achieve high performance and power efficiency in data centers.
Key points
- Token production is projected to grow significantly by 2030, driven by applications such as agentic AI.
- Agentic AI workloads combine multiple tasks including tool calling, orchestration, reasoning, and planning, each with distinct CPU and accelerator demands.
- ARM launched its AGI CPU featuring 136 Neoverse V3 cores, optimized for high performance, low memory latency, and power efficiency.
- Rebellions builds inference accelerators like the Rebel 100, which integrate with ARM AGI CPUs in dedicated server systems.
- The combined solution of ARM CPUs and Rebellions accelerators delivers improved transactions per second per watt and rack-level performance compared to alternative offerings.
Topics
ARM AGI CPUAgentic AI WorkloadsToken Generation TrendsAI Inference AcceleratorsPower EfficiencyRebellions Rebel 100Data Center Architecture
Video transcript
Transcript from available captions; it may contain errors.
save the day. All right. Um uh just a brief introduction. Um my name is Dyla Prahachandran uh senior director of marketing for uh ARM cloud AI business unit. Um focus on data center verticals for the AGI CPU. So really driving adoption of our AGI CPU. Um I'll start out with a little bit of uh kind of market trends. what's going on with token usage, token production u as you know with uh start really started out with open claw uh but now you have many other applications that are driving really token generation agentic AI really driving dramatic amount of token generation and u you know essentially we expect the the trend is that by 2030 u you know the token production will go up quite a bit to you know 120 quadrillion I don't know how many zeros that is but It's it's a huge number and um you know it's it's essentially going up 24x and u and what it's driving is really u not only you know need for power efficient accelerators and that's where our partners rebellions come in uh but also on the CPU side you need a very power efficient CPU that can handle all these tokens in a very efficient way and essentially process that um at scale and uh if you look at different types of workloads You know, obviously you've seen the, you know, system configuration where you have the CPU next to the accelerator. Typically, you know, one CPU is talking to a to two accelerators. Um, essentially being the [snorts] host head node for that system. And the CPU uh you know, if you look at the workload distribution across different workloads, for example, enterprise automation, [gasps] software development and financial analysis, it can vary, right? uh some of them are more accelerator heavy some of them are u you know probably slightly tilted towards the CPU but netnet you know they work hand in hand and that's where a tight collaboration between the CPU and the accelerator is critical to ensure system performance are meeting requirements so these are some of the examples and you can see here um they're about roughly 50 to 40% uh depending on the use case uh let's talk a little bit about the agentic workloads uh which again there is no agentic workload is not really one single workload but it's combination of multiple tasks inside it. Um it starts with the tool calling to orchestration to reasoning and and and and uh planning and planning and reasoning. So essentially you have kind of a set of multiple workloads that are running and when you look at it uh for example on the tool tooling you have you know things like you know databases etc orchestration you have the sandbox containers and then on the uh planning and reasoning you have things like vector databases etc. So um you know these are not the same workloads they have different profiles and essentially have different needs for um you know the CPU use cases as well as the accelerated use cases. And when you look at for example some of the you know accelerator heavy workloads you know influencing etc they're entirely handled by the uh accelerator um when you have something that's uh more on the CPU side you know which is more parallel I mean you know kind of serial processing it's it's done on the CPU side. So uh that's how we kind of view the world in terms of the agentic workload. It's not just one workload but a combination of uh four workloads. And when you look at uh what it what are the demands from a CPU standpoint uh again they vary right so when you look at for example tool calling uh they are definitely it's more u the CPU u IPC which is instruction per clock the the performance of CPU really matters um how can how it can execute these tasks um and then IO's input output so you know for example databases there's a lot of input output that happens concurrency of uh you know different applications running etc that that's important important how the CPU can handle that and then when you look at orchestration um again you have CPU utilization that kind of drives up to 80 90% and being able to perform um at that level is very critical and then when you get to uh you know planning and uh reasoning u you have essentially some of the more fundamental CPU core architecture things like branch prediction being able to predict uh how an operation can actually u you know be executed needs to predict ahead of time. So branch prediction is a very critical and critical thing in the operation and the CPU design. So that's that's something we focus on. Uh cache, memory and latency uh memory latency as well as memory bandwidth are huge factors for this part of the so vector databases etc. They use quite a bit of cache that's you know in the in the CPU as well as external access to memory etc are very critical as well. So you can see here uh when when ARM looked at the kind of the CPU and designing this new new age CPU the AGI CPU we took this all of this into consideration to be able to build a a balanced system that can meet the performance goals for the agentic AI data center but at the same time have the ability to scale within within the server you know within the rack as well as across rack. Um and then the last area is around power efficiency which is really something that ARM has really perfected over the years being able to deliver um you know quite a bit of compute within the same power envelope and we'll we'll talk about some of the uh examples there. Um so we launched the ARM AGI CPU um in March of this year. Um and uh this is essentially ARM's first SOC product and uh you know we have quite a bit of uh interest and momentum with customers and deployment partners. they're listed here a lot of uh you know AI uh companies as well uh you know and some some of the cloud providers u and you know primarily meta I mean this is an OCP conference so u you know meta has been kind of a core development partner for ARM on this on this journey um and you know we have you know volcano engine which is really by dance in China um that that you know also is deploying with AGICPU so the moment is momentum is there in terms of adoption and uh you So if you have you know some time just stop by our booth on this show floor night right right next to the Meta uh booth. So we have some demos and uh hardware for you to look through. Um very quickly you know the AGI CPU um has 136 Neover V3 cores. Neover is the brand branding that we use for our data center CPU cores. And uh v3 is essentially if you want to look at it it's the same exact core that's used by Amazon AWS in graviton. So Graviton 5 uses Neo V3 cores um as well as Nvidia Vera um is also based on uh Neo V3. It's a modified version of that but essentially the underlying architecture is the same and um you know you can see it can go up to 3.7 or 3.5 GHz clock frequency and um you know it has very strong L2 cache again we talked about some of the reasoning workloads that require a lot of local cache access quick access is critical um then when we talk about some of the other other aspects you know PCI gen 6 96 lanes on a single socket system 128 lanes on a dual socket system uh CXL etc. Uh memory latency is optimized sub 100 nconds. Uh in terms of the memory subsystem it's DDR5 uh 12 channels uh going up to 8800. So today we're qualifying with 8,000 speeds but uh can go up to 8,800. And then last but not the least power efficiency. Uh it's a 300 watt uh typical TDP chip. Um, and the performance of this chip is comparable to that of x86 offerings that are in the 500 watt range. So, you essentially save 200 watts per chip. Uh, that translates to lower power consumption, lower you operating uh, expenses and much better TCO for our customers. Um, we've been partnering with Rebellions uh, for for quite a while now. Very close partnership. And essentially the again the AGI CPU is really doing a lot of the tools and orchestration uh handling uh whereas we have the rebel uh um you know solutions rebellion solutions actually doing a lot of the AI and Jennifer will go into more details over there but uh you can see that this is a very tightly integrated system a lot of joint optimization has been done it's continues to uh also uh be in progress so uh very excited uh with with what we're bringing to the market together. Um and you know in terms of um you know we looked at some uh examples of agentic AI u we talked about agentic AI and one of the common use cases is customer support and we looked at the profiling of uh in this case you know the CPU is kind of doing 60 60% of the work and the accelerator is doing 40% of the work and um you know you can see here uh how that has to be tightly managed together and then on the right hand side what you see is um this is just a comparison against AMD's offerings uh in terms of the performance. We've actually published a white paper as well. Actually, a third party has published a white paper that it's on our website. You can go and take a look at it where the AGI CPU delivers more than 2x performance on a rack basis. Uh this is comparing 36 kilowatt racks side by side and looking at how do we get u the best performance out of it. how how can we get the most compute in it given our power energy savings that we have and uh you know Jennifer will talk about how combined solution how that looks as well and that is in the same kind of range as well. Uh with that I'll pass it over to you. Thank you. Thank you. Um and thank you everyone for spending your lunchtime with the leap and I. My name is Jennifer Glor. I run product at Rebellions. For those of you that may not have heard of us, we focus on AI acceleration for inference. And we build an optimized hardware platform and software stack to serve inference models focused on small language models, vision language models, and large language models. It's been a pleasure to collaborate with ARM. It's something that we've been working on for some time. Um, and these are actual real things that are done. um already work is is is well uh along where we've actually integrated the ARM AGI CPU into our system. Um and so from a rebellion's perspective, I mentioned that we build our own hardware. So we build our own accelerator. It's the Rebel 100. This is the piece that does the inf inference uh acceleration. So think prefill and decode. Um and then the ARM AGI CPU is packaged in our system as well um for that component to help with orchestration tooling runtime etc. So, one of our servers that we've collaborated to build together, we call it a Rebel server, has ARM AGI CPUs in it along with eight of our Rebel 100 accelerators. Um the accelerators have um 512 megabytes of SRAMM, 144 GB of HBME. Rebel 100 is chiplet technology with UCIe just to give you some specs. Um we package those into PCIe Gen 5 cards. Eight of them in a 5RU server where we have um ARM AGI CPU running with us. Um what we're seeing today is you know up to 2x uh transactions per second per watt. Um so a lot of that performance that the leap talked about we're actually um seeing run in reality. Um and the power efficiency that they bring is extremely useful to agentic workloads that we're already demonstrating um on this platform. Um a little bit more about rebellions. Um I mentioned that we do both hardware and software. So while our technology is an accelerator, we package that into products that range from a server to a rack to a pod. Today inference is really at rack scale. We give our customers choices in how they would like to deploy us so we can meet them where they are. Most of our customers do choose to deploy Rebel racks, which is just combinations of our servers along with a management rack that includes networking or Rebel pods, which are preset larger configurations. Those start at 32 compute nodes. Um, all of these products that I talk about have ARM AGI CPU embedded in them and you get the advantages of the combined collaboration when you deploy Rebel with ARM. We also have a software stack. When you build your own um hardware, you do need to have software that understands that. Our strategy at Rebellions, however, is that you don't have to learn any new tools or systems to adopt our technology. And so what that means is we've adopted an open-source strategy, basically no forks. We integrate with the ecosystem and the tooling that everyone already knows how to use. So that means we support PyTorch. We haven't forked it. We support VLM. That's also native support. things like Kubernetes are also supported as part of our cloud orchestration. Um, an example of some work that we've done is work with with Red Hat where we're open shift certified. That doesn't mean that you need to run Red Hat Open Shift with our product. We can run other things as is highlighted on the right hand side, but it's just an example of our open-source strategy and how easy it is for us to integrate with whatever tool chain, whatever workflow that you use. Um, from a model perspective, um, as I said, our focus is really small language models, vision language models, and large language models, we think that's where the most workhorse models for enterprises and sovereign deployments will land. Um, and that's really where our team is focused. And we see price, performance, and energy efficiency being the key drivers and indicators in that space. And that's where our product shines. So, with that, I want to make sure we do leave a few minutes for questions. Um, so I'll just give a little bit of a call to action. Um, both ARM and Rebellions are here at the show. Um, we both have booths, so feel free to come and chat with us. You can certainly ask Deepipe and I questions here uh today as well. Um, you can learn more online about the collaboration that we've done if you don't have time to connect with us at the show. Um, and please get involved in the work that we're doing and um, uh, spend some time asking us more questions if you have them. And with that, I guess we'll open it up for questions if anyone has any questions. Happy to answer questions. U, any any questions from the audience? If you have questions, please proceed to the microphones in the aisles and um, ask away. Oh, no questions. All right. Yeah. Thank you very much. Appreciate it.
Assistir ao vídeo original no YouTube
Assuntos: Agentes de IA, AGI. Publicado no YouTube em 2026-10-15.
Seleção automática por assunto e idioma. O vídeo pertence ao canal de origem.
About the Arm AGI CPU
The Arm AGI CPU is a data center processor for agentic AI that Arm announced on March 24, 2026. Built on the Arm Neoverse platform, it marks the first time in Arm's more than 35-year history that the company delivers its own silicon products. According to Arm, it has up to 136 Arm Neoverse V3 cores per CPU and 6 GB/s of memory bandwidth per core, with a 300-watt TDP.
Arm's reference server is a 1OU, 2-node design with 272 cores per blade; 30 blades fill a standard air-cooled 36 kW rack with 8,160 cores, and a liquid-cooled 200 kW design made with Supermicro holds 336 CPUs, for more than 45,000 cores. Meta is the lead partner and co-developer, and the launch partners include Cerebras, Cloudflare, F5, OpenAI, Positron, Rebellions, SAP and SK Telecom. Arm also introduced a reference server in the Open Compute Project (OCP) DC-MHS form factor and plans to contribute its design to the OCP. The video's description adds that the session covers Arm AGI CPU systems with Rebellions RebelCard accelerators in OCP ORv3-aligned racks.
Fact sheet
- Product: Arm AGI CPU
- Company: Arm
- Announced: March 24, 2026
- Purpose: Data center CPU for agentic AI
- Cores: Up to 136 Arm Neoverse V3 cores
- Memory bandwidth: 6 GB/s per core
- TDP: 300 W
- Rack density: 8,160 cores (air-cooled, 36 kW); 45,000+ cores (liquid-cooled, 200 kW, with Supermicro)
- Lead partner: Meta
- Official page: arm.com/products/cloud-datacenter/arm-agi-cpu
Read also
- Stop Building Apps. Build Tools for AI Agents
- Mac Mini 16GB vs 32GB for AI Agents: OpenClaw Uses 1GB
- Harness² : Google Open Sources for Self Improving AI Agents
Sources and references: Arm Newsroom – Announcing Arm AGI CPU · Arm AGI CPU product page · YouTube description of the video by the Open Compute Project channel.