I Built an AI Companion That Lives Entirely on My Home Network — Chat, Photos, and Video, Zero Cloud
Researcher: {alanh0}
A local-first, mobile-first AI companion that chats in Cantonese or English, sends AI-generated selfies mid-conversation, shoots short video clips, and reads photos you send it — without a single byte leaving your LAN.
Most “AI companion” apps are thin wrappers around someone else’s cloud. Every message and selfie ships to a server you don’t control. I wanted the opposite: a companion that runs entirely on hardware I own, yet still feels alive — natural texting, photos it “takes” of itself, and short videos.
It’s called My Companion. Single origin (:18888), mobile-first so it feels like WhatsApp, and the only things it talks to are an Ollama server (the language brain) and a ComfyUI server (the image/video studio) on the same network.
The cast
Open it and you get a general-purpose Assistant (a local ChatGPT that can also draw and animate), plus any companions you build yourself — each a persona with its own personality and AI-generated face. A companion can be any gender or relationship: a girlfriend, a boyfriend, a bro, a buddy, a mentor — whoever you want.
Here I’ve built Mary — a 25-year-old Hong Kong PE teacher who texts in Cantonese and lives for football. Building one is quick: fill a couple of fields, tap ✨ Auto-describe and the LLM writes the visual description, then Generate profile picture turns it into an avatar (or upload your own).
1. It talks like a person, not a chatbot
Mary replies in Hong Kong Cantonese — real colloquial Traditional Chinese (係/喺/嘅/咗/啦), not stiff textbook Mandarin.
She also has agency: she teases, pushes back, disagrees, and won’t validate everything just to please you. And she won’t bluff — ask about a result she hasn’t seen and she says so rather than inventing a scoreline.
She remembers, too. A background memory quietly builds across the relationship — genuinely specific lines like “Mary uses Now TV… for exclusive sports coverage” and “Mary plays football after school as a stress-relief activity.” Nothing hard-coded; it’s distilled from the chat and fed back as quiet context.
And for anything genuinely current — a fixture, last night’s score — she doesn’t guess. The app fires a single lookup at a self-hosted SearXNG box and folds the real results in. It’s the one optional path that touches the open web; turn it off and everything stays on the LAN.
You can watch it work. I ask Mary who Germany plays tonight; she guesses Ecuador, 9pm. I push back — “check properly” (你check真啲?) — she runs the lookup and corrects herself: Germany vs Curaçao, 6pm. Asked where Curaçao is, she nails it too — a Caribbean island north of Venezuela. Real facts, woven in. No links, no “I searched.”
2. It sends photos — that look like the character
Mid-conversation, a companion can take and send a photo. Ask Mary, in Cantonese, to show where she trains and she sends a candid shot from her pitch — then, asked for a food pic, one of her noodles:
The text and the photo always agree, because one hidden instruction drives both. And the face stays consistent shot to shot via a fixed identity anchor plus an optional PuLID face-reference; without it, a long “beach in a suit” prompt would drift to a different look entirely. Photos default to a casual phone-snapshot look — candid, grainy, unposed — not a studio shoot, unless you ask.
3. …and it shoots video
Ask for a video and you get a short clip from Wan 2.2. If the companion is in it, the app first renders a still of them (so they look like themselves), then animates it; a person-free scene goes straight text-to-video.
The clip below is shown as a GIF so it embeds in this post — in the app it’s a real ~7-second MP4 that plays inline, rendered locally on an ASUS GX10, no cloud. Ask Mary for a clip of her eating ramen and she sends one, the same face from her photos, animated:
4. The Assistant: a local creative tool
The Assistant is the same engine minus the persona. It chats, generates images and videos from a plain request, reads images you upload, and — my favourite — edits an image you send it.
I pasted in a cartoon Goku and asked it to “make this anime character a real person”:
A vision model reads the upload, the app runs an image-to-image transform, and out comes a photoreal version — then I asked for a video of him eating ramen and it animated that:
You type plain words; the app does the prompt-writing. No literal image prompts, no parameters.
Under the hood: it’s all orchestration
The honest lesson of this project: no single local model does all of this. Even the hosted giants route video to a separate model. So the app is an orchestrator — a FastAPI backend conducting one specialist per lane:
React SPA (mobile)
│ served by FastAPI, one origin :18888
▼
FastAPI — the orchestrator
│
├──► Ollama chat LLM (Cantonese / English)
│ + a vision model that reads images
│
├──► ComfyUI Flux.2 — text→image, image→image,
│ + PuLID same-face identity
│ Wan 2.2 — image→video, text→video
│
└──► SearXNG optional web grounding (1 outbound hop)A chat turn is a small server-side pipeline, streamed to the browser over Server-Sent Events:
- Assemble the context — persona, language rules, memory, and a “right now” sense of time and place, so she stays consistent turn to turn.
- Stream the reply, and quietly decide if this turn also wants a photo or video.
- Hand the visual to ComfyUI while the text reply is saved immediately — so a phone that sleeps mid-render still shows the reply and a placeholder, and the media lands on its own later.
The real work is the glue between those steps — turning a casual message into the right generation and keeping a face recognisable across scenes.
Why per-lane specialists win
I tried to collapse the stack to one model. The unified “omni” models perceive everything and even stream speech, but their image quality trails dedicated Flux and video is barely usable. The rule held everywhere: a sharp specialist beats a do-everything generalist. The orchestrator is the product.
The local-first payoff
Everything runs on my LAN, so the trade-offs invert: latency is GPU-bound, not network-bound; there’s no per-message bill; and the conversations, faces, and memory are nobody’s training data but mine.
A note on responsibility: this can render a real person’s likeness on demand. The obvious line applies — likeness and voice belong to real people, and impersonating someone without consent isn’t on. Local means control, not a free pass.
Limitations
It’s not magic. One GPU is a serial bottleneck — a photo takes seconds-to-minutes, a clip longer, and you can’t render two heavy jobs at once. Face consistency is strong but not perfect; tricky poses or group shots can still drift. Video is the roughest lane: a clip is animated from a single rendered still, so the motion stays inside that photo’s scene — a short, gentle movement, not a new shot or location — and fine details (spiky hair, hands) can wobble or morph. The local chat model occasionally needs a nudge to send the right photo. The Cantonese still isn’t perfectly smooth — it’s good, but the odd phrase reads a touch off-tone or slips toward written Chinese. There’s no voice yet. And without SearXNG enabled, anything “current” is a guess — the lookup is what keeps it honest.
Future plan
Next is the sense it’s missing: a voice. Self-hosted, cloned-voice text-to-speech (Cantonese + English) and real-time voice conversion — same pattern, a new specialist behind the same conductor.
Because by now the lesson is clear: you don’t wait for one model to do everything. You build the orchestra.
Built on FastAPI + SQLite + a React/Vite/Tailwind SPA, with Ollama and ComfyUI as LAN services — all of it running on a single ASUS GX10 at home. Nothing leaves the house.
