I've been experimenting with AI coding agents for a while now, and there's something that always felt off. When I'm working on architecture or planning features, I sketch things out on a whiteboard or Excalidraw. But my agent just sees text. I describe diagrams, paste screenshots, try to explain spatial relationshipsâit's awkward and error-prone.
Most solutions I tried either required the agent to output some diagram format that I'd then render, or involved clunky screenshot pipelines. Neither felt natural. The agent can't look at what I'm pointing at, reference it in conversation, or edit it incrementally based on feedback.
What I wanted was the agent to actually sit with me at the canvas. Write AGENT: next to something, have it see what's around that area, edit it, and mark the note done. That's closer to how I think.
I ended up building something that does exactly thisâconnects Claude Code, Codex, or opencode to a live Excalidraw canvas. It handles the vision, the scene state, and bidirectional communication between the chat and the canvas.
Curious if others have run into this same friction, or if most people just don't try to use visual tools with their agents at all.
Top comments (1)
Sounds pretty cool. how do I check it out? I usually use Mermaid most of the time now or take a photo of anything Ive drawn up on paper or the whiteboard. Still occasionally use LucidChart with the team too which was our go to for years.