Aug 11, 2026 10:08 PM
(This post was last modified: Aug 20, 2026 02:36 PM by oRequiem.)
For anyone that doesn't know who Neuro-Sama is, she's basically an AI VTuber that has the ability to have memory, as far as I know = learns on the spot, can sing (I have an idea of how vedal987 does that), can talk with a human-like cadence (not too much by choice), can read and reply to chat via voice/text, can train to play games, etc.
I built my own pretty close to how Neuro-Sama is, but Neuro-Sama is quite advanced since vedal987 was probably the first to do something like this, and has the gpu power to do more. Mine has memory, personality (develops over time), thinking about mood changes depending on the conversation, gave her the ability to read my twitch chat, I also created a discord bot so she can join a vc and/or reply to people that mention her, and a filter system so she doesn't randomly say anything bad in my bannedwords.txt file. I would like to eventually fine-tune an LLM (her brain) for vision (ability to train her to play games and see what's on my screen).
I haven't really bothered working on the frontend of the project because of localhost issues (basically can't connect to a temporary web UI I made when my AI is "running"). It would be nice to make a clean/minimalistic web UI for ease instead of just focusing on the terminals being opened.
Curious to see if anyone has any more ideas I can work on with the project.
UPDATE: I did end up giving the AI an emotion and mood system. I am currently using Ollama for the LLM, but I might end up switching to llama.cpp since I've noticed that it seems to be better overall compared to Ollama.
UPDATE 2: I officially went to running her from Ollama to llama.cpp; Apparently, the performance and speed difference is noticeable, but I think I will stick with llama.cpp
UPDATE 3: Just gave the AI a "Hybrid Brain." I'm only using free API's for now since I'm trying to keep this 100% free; my local LLM is purely the fallback right now. I sadly can't implement a vision model using an API key since it gets rate-limited way too fast, and I simply don't want to invest any money in this project right now. 3 brains right now, 1 for deep thinking, 1 for local fallback, and 1 as her "main" brain.
UPDATE 4: I finally routed her TTS Python Audio perfectly in OBS Studio. It was a little tricky since I use my main stream mic as "Voicemeeter Banana" so using CABLE-Input was a bit tricky to split Python Audio in my OBS Audio Sources, but found a way to make sure my AI didn't talk through my mic's streamline.
UPDATE 5: Finally did a 2nd test running my AI Vtuber on stream today. Its responses were perfectly fast and accurate, had A LOT lower fps due to Call of Duty taking up way more VRAM than it should, especially when I'm running whispers locally, the way I'm routing my AI's Python TTS audio made it so it speaks in-game so people in the lobby heard replies when it replied to my twitch chat. Still no vision yet, but that's in the works. Still trying to figure out what single player game to test first for my AI to learn to play.
UPDATE 6: For giving her vision and to play games, I'm currently working on an organized structure and blueprint to start implementing. It looks like I'm going to have to invest some money so I don't get rate-limited almost instantly. Now for the Discord bot version of her, I'm currently working on implementing her hybrid brain so she can talk in voice channels freely, just like how I can talk to her when running her with my mic unmuted. Before, I had to make it so temporary .mp3/.wav files would go to my temp folder; she reads, responds (Python audio and local LLM would crash), then responds (never responds in voice due to LLM crash), then the audio files auto-delete. With her new hybrid brain, I can make it so that part isn't local unless it's needed as a fallback (which wouldn't work anyway, so it's 100% relying on the API's, hence the money investment). Vision will take longer to implement, so I wanna at least get this Discord bot version of her somewhat closer to being finished, and this will be a pretty good upgrade.
UPDATE 7: So 2 of my cloud-based models were deprecated at the same time. I spent 7 hours straight last night rewriting a large portion of my codebase, which is triggering me. For some reason, after my cloud-based models were deprecated, my AI couldn't detect any of my API keys even though they were 100% active. I finally got 2 of them to work (another new API), and it's somewhat back to normal again. For some reason, 1 of the API's stopped working, but the other started working again randomly. It's a pretty odd issue I ran into. Currently spending hours figuring out how to get her Discord bot to receive voice channel audio and respond via voice. Seems like nothing is really working; I've tried about everything that I can think of, including audio routing to a virtual audio cable. Right now, she only responds in the voice channel when someone types to her, not when they speak.
UPDATE 8: I've been attempting to invest some money in the project. One of the services I use for my API keys has really bad payment methods (all require a bank), which is annoying. There is crypto, but it requires a wallet extension that is overly confusing to me. I will provide more updates whenever I get some money invested in the API. I did test out Google's API, but their UI is a convoluted mess, and I'd prefer staying away from it.
I built my own pretty close to how Neuro-Sama is, but Neuro-Sama is quite advanced since vedal987 was probably the first to do something like this, and has the gpu power to do more. Mine has memory, personality (develops over time), thinking about mood changes depending on the conversation, gave her the ability to read my twitch chat, I also created a discord bot so she can join a vc and/or reply to people that mention her, and a filter system so she doesn't randomly say anything bad in my bannedwords.txt file. I would like to eventually fine-tune an LLM (her brain) for vision (ability to train her to play games and see what's on my screen).
I haven't really bothered working on the frontend of the project because of localhost issues (basically can't connect to a temporary web UI I made when my AI is "running"). It would be nice to make a clean/minimalistic web UI for ease instead of just focusing on the terminals being opened.
Curious to see if anyone has any more ideas I can work on with the project.
UPDATE: I did end up giving the AI an emotion and mood system. I am currently using Ollama for the LLM, but I might end up switching to llama.cpp since I've noticed that it seems to be better overall compared to Ollama.
UPDATE 2: I officially went to running her from Ollama to llama.cpp; Apparently, the performance and speed difference is noticeable, but I think I will stick with llama.cpp
UPDATE 3: Just gave the AI a "Hybrid Brain." I'm only using free API's for now since I'm trying to keep this 100% free; my local LLM is purely the fallback right now. I sadly can't implement a vision model using an API key since it gets rate-limited way too fast, and I simply don't want to invest any money in this project right now. 3 brains right now, 1 for deep thinking, 1 for local fallback, and 1 as her "main" brain.
UPDATE 4: I finally routed her TTS Python Audio perfectly in OBS Studio. It was a little tricky since I use my main stream mic as "Voicemeeter Banana" so using CABLE-Input was a bit tricky to split Python Audio in my OBS Audio Sources, but found a way to make sure my AI didn't talk through my mic's streamline.
UPDATE 5: Finally did a 2nd test running my AI Vtuber on stream today. Its responses were perfectly fast and accurate, had A LOT lower fps due to Call of Duty taking up way more VRAM than it should, especially when I'm running whispers locally, the way I'm routing my AI's Python TTS audio made it so it speaks in-game so people in the lobby heard replies when it replied to my twitch chat. Still no vision yet, but that's in the works. Still trying to figure out what single player game to test first for my AI to learn to play.
UPDATE 6: For giving her vision and to play games, I'm currently working on an organized structure and blueprint to start implementing. It looks like I'm going to have to invest some money so I don't get rate-limited almost instantly. Now for the Discord bot version of her, I'm currently working on implementing her hybrid brain so she can talk in voice channels freely, just like how I can talk to her when running her with my mic unmuted. Before, I had to make it so temporary .mp3/.wav files would go to my temp folder; she reads, responds (Python audio and local LLM would crash), then responds (never responds in voice due to LLM crash), then the audio files auto-delete. With her new hybrid brain, I can make it so that part isn't local unless it's needed as a fallback (which wouldn't work anyway, so it's 100% relying on the API's, hence the money investment). Vision will take longer to implement, so I wanna at least get this Discord bot version of her somewhat closer to being finished, and this will be a pretty good upgrade.
UPDATE 7: So 2 of my cloud-based models were deprecated at the same time. I spent 7 hours straight last night rewriting a large portion of my codebase, which is triggering me. For some reason, after my cloud-based models were deprecated, my AI couldn't detect any of my API keys even though they were 100% active. I finally got 2 of them to work (another new API), and it's somewhat back to normal again. For some reason, 1 of the API's stopped working, but the other started working again randomly. It's a pretty odd issue I ran into. Currently spending hours figuring out how to get her Discord bot to receive voice channel audio and respond via voice. Seems like nothing is really working; I've tried about everything that I can think of, including audio routing to a virtual audio cable. Right now, she only responds in the voice channel when someone types to her, not when they speak.
UPDATE 8: I've been attempting to invest some money in the project. One of the services I use for my API keys has really bad payment methods (all require a bank), which is annoying. There is crypto, but it requires a wallet extension that is overly confusing to me. I will provide more updates whenever I get some money invested in the API. I did test out Google's API, but their UI is a convoluted mess, and I'd prefer staying away from it.
![[Image: New-Signature-v2.png]](https://i.ibb.co/FLQmYqVX/New-Signature-v2.png)
![[Image: uRrBM3f.png]](https://i.imgur.com/uRrBM3f.png)