I Built a JARVIS AI Assistant That Can Talk Back and Control My PC
Publicado no IA Dots ·
Agentes de IAAI SignalAssistente pessoaldot-ai_agentsdot-ai_assistantsdot-artificial_intelligenceidioma-enInglêsyt-hNjBamiUPD0
Vídeo em Inglês, publicado pelo canal Microlearn.
About this content: The summary, key points and topics below are generated automatically from the video transcript. They reflect the content, opinions, claims and predictions presented in the video and do not represent a position, validation or fact-check by IA Dots.
Video summary
This video explains how to build a local, voice-controlled AI assistant named Jarvis that runs on older hardware without cost, using a combination of local audio processing, Google's Gemini in the cloud, Microsoft text-to-speech, SQLite memory, and a Python-based tool execution system.
Key points
- The AI assistant setup splits tasks by using local processing for audio input via faster Whisper and local speech output, while utilizing Google's Gemini in the cloud for reasoning.
- Jarvis executes computer actions through a defined menu of Python functions, ensuring the model only performs permitted tasks like opening apps or reading documents.
- Memory is handled locally using a single SQLite file that stores recent conversation history and persistent facts about the user.
- A lightweight web-based interface provides visual feedback, such as a sphere that changes state depending on whether the assistant is listening, thinking, or speaking.
- The system integrates with Telegram, allowing remote control of the PC assistant from a phone while restricting access to a specific user ID for security.
Topics
Voice AssistantLocal AIPython DevelopmentGoogle GeminiSpeech RecognitionSQLite MemoryAutomation ToolsTelegram Bot
Video transcript
Transcript from available captions; it may contain errors.
This is Jarvis and I built him this week. He hears me, he talks back to me, he remembers who I am and he actually does things on my computer. And all of that is running on a PC with a graphics card from 2016 without paying a cent. By the end of this video, you'll have your own version and you'll understand every part of it well enough to change it, fix it and make it yours. My first try was the obvious one, run the whole AI on my own machine. On this old card, one answer took more than a minute, which is useless when you're talking to something. So, I split the job in two. The thinking happens in the cloud on Google's Gemini and most of the rest lives on my PC. Your voice recording never leaves the machine because only the text of what you said goes up to Gemini and only its reply comes back. Gemini has a free tier that doesn't ask for a card and everything else I use here is free, too. Most of it open source. The free tier does have daily limits, though, and they depend on the model with some giving you surprisingly few requests a day. You can see yours on the rate limit page in AI Studio and I'll show you how Jarvis handles hitting one without crashing. Here's the map we'll follow. Ears turn your voice into text, the brain decides what to do and the mouth speaks the answer. The brain can also lean on a memory and on a set of hands that act on your computer. At the end, we'll plug in your phone through Telegram so you can talk to Jarvis from anywhere. Let's start with the ears because nothing works if Jarvis can't hear you properly. The microphone records in tiny slices, a tenth of a second at a time. For each slice, we check how loud it is. Once you start talking, we keep everything and as soon as you've been quiet for about 7/10 of a second, we decide you're done and stop recording. That pause length matters more than you'd think. Make it too long and Jarvis feels sluggish, too short and he cuts you off mid-sentence. 7/10 felt natural for me and it's one number you can change. If your room is noisy or Jarvis never seems to notice you've started talking, there's a second number right next to it. How loud a slice has to be to count as speech. Lower it if he misses you, raise it if the fan on your PC keeps waking him up. The recording then goes to Whisper, OpenAI's speech recognition model. We use a version called faster Whisper that runs right on your processor offline, so no audio ever leaves your PC. On my old machine, it usually turns a sentence into text in under a second. Whisper also gets a little help with silence. Before it starts, a voice detector cuts out the stretches where nobody's speaking, which helps stop it from inventing words out of background hiss, something these models are known to do. Early on, I'd say, "Hello Jarvis." And it would write down, "How's your risk?" Whisper had simply never heard the name. The fix is one line. You give Whisper a short hint before it listens, something like, "Talking to Jarvis, my AI assistant." And from then on, it gets the name right. You also get to choose the model size. I use the base English model, which is the quick one. And if it mishears you a lot, you change one setting to the small model and trade a little speed for accuracy. Now, the brain, which is the most important part to understand because it's what turns a chatbot into an assistant that can actually do things. A language model on its own can only read text and write text. It can't open notepad and it can't search the web. So, how does Jarvis open apps? We give it a menu. Every action Jarvis can take is a normal Python function, and each function has a short description written right under its name saying what it does and what it needs. Gemini reads that menu with every message. When you say open notepad, the model doesn't run anything itself. It writes back a request. Basically, please call open app with notepad. Our code runs that function, gets the result, and hands it back to the model, which then writes the reply you hear. If it needs more than one step, it just goes around again. Here's one real turn from my logs slowed down. I said, "Open notepad." The first trip to Gemini took about 6/10 of a second and came back with a request for the open app tool. Our code opened notepad and sent back opened notepad, and a second trip, another 6/10, turned that into the sentence he actually said. So, anytime Jarvis does something, it's two quick calls to the brain with your code working in between. Keep that split in mind because it's where all your control lives. The model can only ask for things you put on the menu, and your code decides what actually happens. If you watched my AI agent video, this is exactly that loop now with a voice. His personality comes from a few sentences of instructions that go in front of every conversation. Be a calm, witty British assistant. Call me, sir. Keep answers to a sentence or two because they're spoken out loud. That last part is easy to forget, and without it, he reads you whole paragraphs. Picking the model turned out to matter a lot. I started with the regular flash model, and on the free tier, it kept timing out, sometimes after 19 seconds. The lighter flashlight model answered in about 6/10 of a second almost every time. And for short-spoken replies, it's more than smart enough. Free servers do get busy, though. So, the brain never trusts a single attempt. If Google says it's overloaded, Jarvis waits a moment and tries again, then switches to a backup model. And if everything's busy, he tells you so out loud instead of crashing. Once the brain has an answer, the mouth has to say it. And this is where most homemade assistants feel slow. For the voice, I'm using the same text to speech that powers the read aloud feature in Microsoft Edge through a small library called Edge TTS. It runs on Microsoft servers, so it needs the internet, but it sounds natural. It has a properly British voice, and it doesn't need an account or a key. The trick that makes it feel quick is speaking sentence by sentence. Instead of waiting for the whole reply to turn into audio, we make the first sentence, start playing it, and prepare the next one while he's talking. In my logs, the gap between the end of the thinking and the first word he says is usually under a second, which is about how long a person takes to reply. You're not stuck with my choice of voice, either. Edge has hundreds of voices in all sorts of languages and accents, which you can list with one command, and switching is a single line in the mouth file. If you'd rather have one that works with no internet at all, you can swap in an offline engine there instead. Though, most of those don't sound quite as natural. A Jarvis that forgets you the moment you close him isn't much of a Jarvis. So, let's give him a memory. Everything is stored in SQLite, which is just a single file on your disk with no database server to install. Inside that file, he keeps the recent conversation, so he can follow what you said a minute ago and separately, a list of lasting facts about you. The clever part is that saving a fact is just another tool on the menu. When I said, "Remember that I make YouTube videos about AI." the model decided that was worth keeping and called the remember tool. And our code wrote it to the file. Then, every time you talk to him, those facts get added to his instructions. So, when you ask what he knows about you, he isn't searching anything. The answer is already sitting in front of him. You might wonder why we don't just send the whole history every time. Every word you send costs time. And on the free tier, it eats into your limits. So, Jarvis only sends the last 10 messages, which is about five back and forth. While the facts, which are short, always go along. And because it's a file, it survives restarts. Close Jarvis tonight, open him tomorrow, and he still knows you. Along with a bit of cheek I didn't ask for. It also means you're in charge of it. It's a plain file sitting in the Jarvis folder on your own machine. So, you can open it and see exactly what he's saved about you. And if you ever want him to forget everything, you just delete it. Now, for the fun part, the hands, which is everything Jarvis can actually do on your computer. He can open and close apps, play a song on YouTube, turn the volume down, or pause the music, open websites, search the web, and read files you point him to. Each of these is just a small Python function. Because these touch your real machine, every tool has rails. He can only open apps from a list you choose. He can only read files inside your documents folder, and he refuses to close Windows Explorer because that would take your whole desktop down with it. Adding your own skill is the part I'd love you to try. Say you want Jarvis to tell you the time. You write a tiny function that returns the current time, put one clear sentence under its name saying what it does, then add it to the list of tools. Next time you ask what time it is, the brain sees the new item on its menu and calls it, and you never touch the loop. That sentence is worth getting right because along with the function's name, it's pretty much all the model has to go on when it picks a tool. This next bit taught me the most important lesson of the whole build. Before I'd written a close tool, I asked Jarvis to close notepad. He searched the web for how to close notepad, and then he said, "Consider it done, sir." Notepad was still sitting there, wide open. Models often do this when you ask for something they can't do. They'd rather sound helpful than admit it. I fixed it in two places. I gave him a real close tool that reports back whether it worked, and I added one line to his instructions telling him to only say something's done when a tool result says so. Now, when something fails, he tells you. That's worth copying into anything you build with AI. Make every tool report what really happened, and make the model repeat that, not what it hoped would happen. He does have limits, and it's better to know them. He can start a YouTube video, but he can't click around inside a web page, and he can't see your screen. Those are bigger projects for another day. You've probably noticed the screen. Jarvis works fine in a plain terminal, but a face makes him feel alive, and it's simpler than it looks. Whenever something happens, say he hears you or calls a tool, the Python code sends a small message to a web page running on your own machine. That page is listening for them, and it updates the conversation on the left and the list of actions on the right the moment they arrive. Those messages use something called server-sent events, which is a fancy name for a connection that only goes one way, from Python to the page. That's all a screen like this needs. The one thing that goes the other way is the mute button, which just sends a tiny request back to Python to stop listening. In the middle, the sphere grows with your voice while you talk. While he's thinking, it turns amber. And when he speaks, it pulses along with him. The whole thing opens as its own window, so it can sit next to whatever you're working on. Last part, and it's the one I use the most, talking to Jarvis from my phone. I made a Telegram bot with BotFather and plugged it into the same brain that runs on my PC, memory and hands included. Now, I can be on the couch, type open calculator on my PC, and across the room it opens. There's no server to set up for this. While the bridge is running, it keeps asking Telegram whether you've sent anything new. And the moment you do, it hands your message to exactly the same think function the voice version uses. The catch is that your PC has to be on and running it. So, if you want Jarvis reachable around the clock, that's where hosting comes in, which is a whole video of its own. Because this can control your computer, the bot checks who's talking. It only obeys your own Telegram user ID, and anyone else gets a polite no. And since it shares the memory, everything you told him by voice, he knows on your phone, too. To run it yourself, first get a free Gemini key from Google AI Studio. It takes a minute. Just don't turn on billing for that project, because without billing it can't charge you. And if you hit the daily limit, he'll just tell you his brain is busy until the limit resets the next day. Then download the code from the link in the description. Make a virtual environment and install the requirements. The virtual environment is what saves you from the classic error where two versions of Python fight over your packages. Paste your key into the ENV file, run Jarvis.py, and say hello. If he can't hear you, check that Windows lets desktop apps use your microphone. A quota error means you've hit one of those free limits, not that anything's broken. Both fixes are in the readme, along with how to set up the Telegram part. Now, you've seen how every piece connects. Once you know the brain only ever picks from a menu of functions, adding a new skill to Jarvis is just writing one more function and describing it well. If you want to go further with the Telegram side and give your bot a real brain that reads your own notes, that video is on screen right now. I'll see you there.
Assistir ao vídeo original no YouTube
Assuntos: Agentes de IA, Assistente pessoal. Publicado no YouTube em 2026-10-24.
Seleção automática por assunto e idioma. O vídeo pertence ao canal de origem.
About faster-whisper
faster-whisper is a reimplementation of OpenAI's Whisper speech recognition model built on CTranslate2, a fast inference engine for Transformer models. Its repository is maintained under the SYSTRAN organization on GitHub. According to the official README, it is up to 4 times faster than the original openai/whisper implementation for the same accuracy while using less memory, and efficiency can be improved further with 8-bit quantization on both CPU and GPU.
The library requires Python 3.9 or newer and is installed from PyPI with pip install faster-whisper. Unlike openai-whisper, it does not need FFmpeg installed on the system, because audio is decoded with the PyAV library, which bundles the FFmpeg libraries. GPU execution needs NVIDIA's cuBLAS and cuDNN 9 libraries for CUDA 12. The library also integrates the Silero VAD model, which can filter out parts of the audio without speech when the vad_filter option is enabled.
Fact sheet
- Library: faster-whisper
- Based on: OpenAI Whisper, reimplemented with CTranslate2
- Repository: SYSTRAN/faster-whisper on GitHub
- Speed: Up to 4 times faster than openai/whisper at the same accuracy
- Quantization: 8-bit on CPU and GPU
- Requirement: Python 3.9 or newer
- Installation: pip install faster-whisper
- GPU requirements: cuBLAS and cuDNN 9 for CUDA 12
Read also
- How to build a JARVIS AI Voice Agent for FREE | Full Livekit Tutorial (2026)
- No more Claude or ChatGPT - Local AI on your GPU + DaVinci Resolve Studio 21.1 - MCP Server
- AI Agents Finally Share One Memory (Claude Mem Explained)
Sources and references: GitHub – SYSTRAN/faster-whisper (README) · video by the Microlearn channel on YouTube.