PAPER DEEP DIVE
nanoMuse: An Open-Source Personal Agent for Every Device You Own
nanoMuse places a full personal agent on each device, routes every tool call through a scoped Sentinel policy gate, and keeps synchronization optional. The open-source stack supports readable memory, provider choice, phone and desktop control, and self-hosted relay deployment.
Paper: nanoMuse: An Open-Source Personal Agent for Every Device You Own
Authors: Guangyi Liu, Yong Liu, Jiangning Zhang
Version and date: arXiv 2610.08699v1, 6 October 2026, cs.AI
Paper: https://arxiv.org/abs/2610.08699
Project and demo: https://nanomuse.cn, web demo
Code status: open source under GPL-3.0; this analysis uses commit b3bf3e9.
One-sentence summary
nanoMuse does not train another foundation model. It turns full device-side agents, phone and desktop screen control, a uniform action gate, optional conversation synchronization, and human-readable memory into a self-hostable reference implementation of a personal agent.
Why this report matters
Assistants answer and wait; task agents complete one request and stop. A personal agent is supposed to persist across weeks, remember one person, act on that person's accounts and devices, speak first when appropriate, and answer for what it did. The paper treats this as a software form rather than a property of a stronger model. Its central problem is not text generation, but how a long-running system obtains context, reaches interfaces, receives authority, and leaves evidence after action.
The report uses Meta's September 2026 Muse as its closed reference point. According to the authors' reading of public material and a copy of a production prompt, Muse places each user's accounts, files, memory, and conversation in a dedicated cloud VM. A separate Sentinel is the sole permission authority for connector actions and network egress. The browser sees an accessibility snapshot rather than the whole desktop, and credentials can remain outside the agent container through surrogate tokens. This creates a meaningful privilege boundary, but also a virtual machine per user, a vendor-controlled model path, and limited ability to inspect or replace the stack.
nanoMuse takes a deliberately different route. Every Android, iOS, desktop, and web client carries a full agent and its own Sentinel. Android and desktop devices can operate their own screens, while iOS delegates tasks it cannot perform locally. A relay joins devices when the user wants shared conversation, but can be omitted for a single-device setup or self-hosted for a private deployment. The model provider is selected by the user instead of being fixed by a platform owner.
The work is therefore a system report, not a benchmark paper. It claims a complete, minimal construction and openly states the parts it has not measured. That distinction matters: a working end-to-end agent demonstrates architectural plausibility, but not reliable task completion, safety under adversarial interfaces, or economic viability at scale.
Background concepts
An agent is not one inference request. It is a loop that plans, calls tools, observes results, and decides what to do next. A harness supplies the runtime around that loop: model adapters, tools, memory, scheduling, sandboxing, and device communication. Hands are the last-mile executors that turn decisions into graphical actions such as taps, mouse clicks, keystrokes, and scrolling.
The most important part of the design is the separation between proposal and authority. A language model can propose that an email should be sent or a file should be deleted. It cannot be allowed to decide by itself that the action is authorized. nanoMuse routes each tool call through a Sentinel, making the model a planner rather than the root of trust.
flowchart LR
G[User goal] --> A[Agent on initiating device]
A --> R{Choose lowest viable rung}
R -->|Skill or tool| S[Sentinel gate]
R -->|Authenticated page| S
R -->|Browser| S
R -->|Hands| H[One screenshot to one action]
H --> S
S -->|Allow| X[Execute locally or on peer]
S -->|Deny| D[Return refusal]
S -->|Ask| P[Scoped user grant]
P --> X
X --> T[Trace and audit record]
T --> A
A <-->|Optional text sync| Y[nanoMuse relay]
Y <-->|Push and pull| B[Agent on another device]
The architecture also separates conversation synchronization from permission sharing. A relay can make the same main thread visible on several devices, but a remote device still executes an action under its own Sentinel and its own user's grant. The requesting device cannot lend a permission it does not have.
How nanoMuse is built
A peer-to-peer topology with an optional hub
The Android client is a modified OpenMinis application. It contains an agent, a sandboxed Alpine Linux environment, a shell, a browser, MCP servers, skills, scheduled tasks, and adapters for OpenAI-compatible models. The desktop app wraps the DeepSeek harness in Electron and carries a Python runtime; the same runtime serves the web client from a computer controlled by the user. The relay is the only shared service, and it is not required for single-device operation.
There are three deployment modes. A user can run one device with a private model key, with no account and no server. Several devices can share the community relay. Or the user can deploy the relay in Docker on a small server, control the provider key, and let invited accounts use that hub. These modes share the same client protocol, so the self-hosted path is not a separate experimental fork.
The action ladder
The agent tries the lowest-cost and most explicit interface first: an existing skill, command-line tool, or MCP server; then a page fetched with the user's authenticated session; then the browser; and only then the device's own screen. The authors connect this hybrid strategy to MAS-Bench's observation that capable phone agents use both shortcuts and direct interface control. Screen operation expands what the agent can reach, but every visual step adds latency and uncertainty. Hands are therefore off by default and treated as the final fallback.
This ladder is more than a product preference. A structured tool call declares arguments, risk, and egress targets that policy code can inspect. A tap has to be inferred from pixels, accessibility labels, and the model's description. The architecture keeps visual control because many personal workflows only exist in native applications, but it does not pretend that pixels are as governable as a typed API.
Hands: one screenshot, one action
nanomuse/phone/operator.py implements the Hands loop. For each step, the operator model receives the goal, one screenshot, a one-sentence history entry per prior action, and any accessibility nodes the device exposes. It returns a short thought, a sentence describing the intended action, and one function call. Coordinates live in a normalized 999 by 999 space before being mapped to the actual screen. The loop then takes a fresh screenshot. This prevents a long, unverified sequence from running blindly after the interface has changed.
If $s_t$ is the observable screen state, $g$ is the goal, $h_{<t}$ is compressed history, and $u_t$ carries the user's language context, the interaction can be summarized as $a_t=H(s_t,g,h_{<t},u_t)$. This is an explanatory notation for the report's text, not a training equation introduced by the authors.
Mobile and desktop use different function dialects but preserve the same thought-plus-one-action contract. The phone exposes clicks, long presses, swipes, typing, application launch, system buttons, waits, questions, hand-over, and termination. The desktop adds mouse buttons, drags, key combinations, and scrolling. The human-readable action sentence is not cosmetic: the Sentinel can inspect it for committing language such as payment, transfer, order, send, or delete, turning part of a visual decision into policy-checkable text.
Before a click is executed, the system also inspects accessibility text at the target and applies the stricter of the two judgments. A focused password field hands control back to the user. On the phone, a floating capsule shows the current step and a red stop control, while a ring marks the next tap. On the desktop, the screen edge glows and the cursor target is visible. The full trajectory remains in the conversation, and the user can take over. These affordances make interruption and inspection part of the execution path.
Sentinel and the order of decisions
Sentinel is the entry point for every tool call. Sentinel.guard() asks the tool to assess its arguments, target, and possible egress; evaluates policy; checks standing grants; asks the user when necessary; resolves credentials; executes; records taint; redacts secrets from the returned observation; and writes an audit event. The model can choose a tool, but the call does not reach the outside world until the gate permits it.
The paper gives the decision order as denied tools, person-defined rules, always-allow and always-ask lists, conversation taint, tool risk against the selected mode, and finally warnings. For explanation, write the categories as $D,R,L,T,M,W$. The precedence can then be represented as $S(c)=\operatorname{first}(D,R,L,T,M,W)$. The notation compresses the prose order; it is not a formal policy language claimed by the paper.
| Priority | Layer | Purpose | Can a standing grant bypass it? |
|---|---|---|---|
| 1 | Denied tools | Remove capabilities that must never run | No |
| 2 | Person rules | Express explicit limits on tools, targets, or paths | No |
| 3 | Allow and ask lists | Route familiar or sensitive calls through fixed policy | Partly |
| 4 | Taint | Require at least an ask for egress after private reads | Not reversed |
| 5 | Risk and mode | Compare action risk with the selected autonomy level | Yes |
| 6 | Warnings | Flag dangerous shell use, immediate execution of downloads, and committing labels | No, once only |
Approvals are scoped capabilities
An approval is not a permanent global Boolean. It is bound to a tool and a target such as a host, recipient, folder, application, or device. The user can grant it once, for the current conversation, for a limited period, or, when a target can be named, on a standing basis. Standing grants appear on the permissions page and can be revoked. Actions carrying warnings receive only a once option, so a payment or password-related action cannot become permanently automatic merely because the user said "always" at another time.
Taint forms a second dimension. After a conversation reads private data, later calls that send data outside the allow-list are elevated to an ask. The state is per conversation and is not casually cleared. This does not equal kernel isolation, but it directly addresses a common failure composition: read a private file, then quietly upload part of it.
Cross-device conversation without cross-device authority
Any device's agent can call another device as a tool. A task might find an order number on a phone and place it into a spreadsheet on a laptop. The destination device agrees first, using a grant supplied by the person at that device, and the action runs under the destination Sentinel. The initiating device cannot transfer its own permission.
The synchronization engine reveals several concrete design choices. A user message is pushed immediately, while the final assistant message is debounced by two seconds. Pulls run at startup, after sign-in, when the relay sends a signal, and every sixty seconds. History is paged in batches of 200. A fresh device first requests the newest tail of 300 messages so that the account's main conversation appears quickly. The main conversation syncs by default, side chats are opt-in, and attachments contribute only names and sizes, not bytes. Messages are deduplicated by IDs.
| Parameter | Implementation | Why it exists |
|---|---|---|
| User-message push | Immediate | Show activity on other devices without waiting for the turn to finish |
| Assistant-message debounce | 2 seconds | Coalesce updates near the end of a streamed turn |
| Routine pull interval | 60 seconds | Maintain eventual consistency with low request volume |
| History page | 200 messages | Bound request size |
| Fresh-device tail | 300 messages | Restore recent context before older history is needed |
| Hands coordinate grid | 999 by 999 | Represent targets independently of screen resolution |
What the relay stores and what it does not
The relay holds an account identifier in hashed form, issued keys and their device assignments, a ledger of model, token count, and cost, the agent profile, presence information, and the conversation text that the user chooses to sync. It does not store files, images, or the raw output of tools. Turning synchronization off deletes stored conversation text. Account deletion removes the account data; a hash of revoked keys remains for ninety days so that an offline device can learn that the account no longer exists.
That list is not the whole privacy story. The main conversation is synchronized by default, which means its text reaches a relay. The community relay also enables retention of chats with community models for training by default for new accounts, although the switch is visible and can be turned off. A self-hosted relay chooses its own default. The authors identify this as one of the choices they are least certain about. Open code and self-hosting reduce lock-in, but they do not make every default local-only or anonymous.
Memory as inspectable state
On the phone, memory is a set of Markdown files. SOUL.md describes the agent, USER.md describes the person, GLOBAL.md carries stable cross-conversation facts, a dated diary records events, and HEARTBEAT.md holds routines. A user can open and edit these files, and the agent can read them through its shell. This makes the source of remembered behavior visible and portable.
The desktop store uses SQLite. Lexical retrieval scores rare-token overlap and can fuse an embedding ranking when an endpoint is configured. Chinese text is covered with character bigrams rather than relying on whitespace tokenization. Every change is written to a log with the prior and new values, so a merge, rewrite, or deletion can be undone. The forget operation is logged too. The implementation is substantial, but the paper is explicit about the missing layer: entries do not reliably carry the authoring model, confidence, or a rule for rechecking old claims when stronger evidence arrives.
Model choice and the cost of a personal runtime
The clients share a catalogue of eighteen Chinese and international providers. A user can supply a direct API key, use a ChatGPT plan through OpenAI's Codex authorization flow for chat and Hands, or point the runtime to a local model server. With a direct key and one device, the relay is not in the inference path; only model requests leave the machine.
The reported size figures come from release files, one installation, and the project documentation rather than a large deployment study. Android is 38 MB to download for arm64. Desktop downloads range from 256 MB on Linux and 266 MB on Windows to 498 MB on macOS. A Linux installation occupies about 928 MB, and all desktop processes together use roughly 0.5 GB of idle proportional set size. The relay is one Python process plus SQLite, uses about 85 MB at idle with a fresh database, and targets a 1 vCPU, 1 GB server.
| Artifact or resource | Reported value | Measurement basis |
|---|---|---|
| Android client | 38 MB | Arm64 release download |
| Linux desktop client | 256 MB | Release download |
| Windows desktop client | 266 MB | Release download |
| macOS desktop client | 498 MB | Release download |
| Linux installed footprint | About 928 MB | One installation |
| Desktop idle memory | About 0.5 GB | Proportional set size of all processes |
| Relay idle memory | About 85 MB | Fresh database, single process |
A self-hosted relay is estimated at CNY 30 to 60 per month in mainland China or USD 4 to 6 per month elsewhere for the smallest server tier, excluding model usage. These ranges can be written as $C_{\mathrm{relay}}^{\mathrm{CN}}\in[30,60]$ and $C_{\mathrm{relay}}^{\mathrm{US}}\in[4,6]$. They illustrate the economics of a small synchronization node rather than a cloud VM for every user.
Using Bailian's October 2026 busy-hour list prices, the chat model costs CNY 2 per million input tokens and CNY 8 per million output tokens, giving the explanatory expression $C_{\mathrm{chat}}=2t_{\mathrm{in}}+8t_{\mathrm{out}}$. The Hands model costs CNY 3 per million input tokens and CNY 12 per million output tokens, or $C_{\mathrm{hands}}=3t_{\mathrm{in}}+12t_{\mathrm{out}}$. These are representations of published prices, not equations contributed by the paper. The practical cost of screen control depends on screenshot encoding, number of model calls, retries, and trajectory length.
| Cost item | Reported estimate | Includes model usage? |
|---|---|---|
| Self-hosted relay in mainland China | CNY 30-60 per month | No |
| Self-hosted relay outside China | USD 4-6 per month | No |
| Chat input | CNY 2 per million tokens | Busy-hour example |
| Chat output | CNY 8 per million tokens | Busy-hour example |
| Hands input | CNY 3 per million tokens | Busy-hour example |
| Hands output | CNY 12 per million tokens | Busy-hour example |
Paper-to-source correspondence
The repository supports the report's main implementation claims. Sentinel.guard() contains the assessment, policy, grant, execution, taint, redaction, and audit path. PhoneOperator._steps() builds a loop around a screenshot, a compact history, and a normalized coordinate contract. PhoneOperator._act() constructs the gesture call and routes it through Sentinel. The memory store shows SQLite persistence, lexical and optional semantic retrieval, and a reversible change log. The synchronization engine supplies the two-second push debounce, sixty-second pull interval, 200-message pages, and 300-message fresh-device tail.
This source inspection also reveals how much product behavior sits in the harness rather than the prompt. A single screen action involves dialect parsing, coordinate mapping, state checks, sensitive-field detection, approval, device communication, and trace recording. Cross-device continuity involves account ownership, main-thread identity, side-chat policy, message deduplication, and presence expiry. A personal agent is a runtime engineering problem before it is a prompting problem.
What the report establishes
The strongest result is end-to-end constructibility. Android, iOS, desktop, and web clients connect to an open relay; Android and desktop can operate their own screens; memory, model selection, permissions, and revocation are integrated; and the user can choose a single-device or multi-device deployment. Against the closed product that inspired the work, this is a meaningful open baseline.
The paper evaluates the design through five questions and three horizons: what the agent knows, where it acts, whom it answers to, how it learns, and how it is paid for and governed. nanoMuse offers a concrete start on readable memory, many-device action, revocable approval, and replaceable infrastructure. It falls short on memory provenance, measured Hands reliability, and open models for the action loop.
Compared with Muse, nanoMuse has native Hands, an open full stack, provider choice, and a serverless single-device mode. It lacks a dedicated VM, kernel-level taint tracking, surrogate credentials, financial wallet, artifact workspace, and the polish of a large product team. Compared with OpenMinis, it adds a personal-agent layer; compared with CopilotKit's OpenMuse, it gives each device the agent and its screen instead of building another server with remote windows.
The timeline is itself a warning about volatility. nanoMuse first shipped on 25 September 2026 and the paper describes version 0.1.40. Muse's public features and safety architecture were still changing within weeks. The commit hash and version matter more than a timeless claim that one architecture is final.
What is missing is equally clear. There is no measured task success rate for Hands, no human-takeover count, no held-out interface study, and no comparative benchmark against Muse or other personal agents. The roadmap names AndroidWorld, OSWorld, MemGUI-Bench, and OS-Harm as future evaluation foundations. Until those results exist, the defensible conclusion is that nanoMuse establishes a testable architecture, not that it establishes dependable personal automation.
Limitations
A policy boundary is not a privilege boundary. The authors state this directly. On one device, Sentinel runs in the same trust domain as the agent. Rules can stop a persuaded model, but the real credentials and policy engine are not held by an independent host outside the agent's reach. A compromise of the device or process remains a compromise of the agent.
Hands are judged by language and remain unmeasured. An icon-only control in an application that exposes no useful accessibility text gives policy little evidence about its consequence. Dense professional interfaces make target identification difficult, and the user watching the click ring is still a practical last line of defense. No success rate or takeover frequency is reported.
Platform coverage is uneven. Linux Hands run only in X11 sessions. iOS cannot operate other applications, so tasks must be delegated or handed to the user. Mobile and desktop share a conceptual loop but differ in inputs, permissions, and recovery, which limits claims of transferability.
Memory lacks provenance and revalidation. The store records changes and allows undo, but not consistently the model, date, confidence, and evidence behind each memory. A weak earlier guess may be read as fact by a stronger later model. The report calls for a reconciliation rule, but that rule has not yet been implemented.
Shared relay and retention defaults need scrutiny. The community relay stores synchronized conversation text on a server run by the project. The option to retain chats for improving community models is enabled by default on new accounts. The choice can be disabled and the relay can be self-hosted, but the default path is not the most conservative privacy setting.
The evidence about Muse has a thin layer. The account relies on four public documents, shipped clients, and a production prompt copy whose origin the authors cannot verify independently. Prompt text can describe policy without proving enforcement, and deployments can change quickly. The paper labels prompt, inference, observation, and documentation separately, which is honest, but those categories should be preserved when citing its claims.
Conclusion
nanoMuse is valuable because it exposes the machinery of a personal agent instead of presenting a chat surface over an opaque runtime. It places agents on the devices that hold the interfaces, makes synchronization optional, binds approvals to targets and lifetimes, keeps memory editable, and lets users choose the model and relay. It also shows why these choices are hard: local execution weakens cloud isolation, screen control lacks strong semantic signals, and persistent memory without provenance can become a durable source of error.
The next milestone is not another provider integration. It is evidence: a reproducible suite built from failed Hands trajectories, memory that records and rechecks its sources, and an open action model that can be evaluated independently. Those changes would turn the present open alternative into a component that a person could reasonably trust for long-running work.
"A personal agent is a software form that carries responsibility over time, not merely a more conversational model."
