Why I Built a Library for AI Agents

I wanted to play a game in my terminal.

Specifically, I wanted Claude to run a Dungeons & Dragons campaign for me. For the uninitiated: D&D is a tabletop role-playing game where one person — the Dungeon Master — narrates a story, manages rules, and controls every character the players encounter. There are thick rulebooks that govern everything from combat mechanics to spell effects to monster statistics.

I figured a large language model would be perfect for this. Claude is creative, conversational, great at narrative — exactly what a Dungeon Master needs to be.

There was just one problem: Claude kind of knew the rules.

It was like playing with a friend who says "yeah, I've played before" and then makes up spells that don't exist, gives monsters 50 hit points when the book says 178, and invents abilities that break the game's core mechanics. Every time precision mattered, it guessed. Confidently, convincingly — and wrong.

So I did something simple. I gave it the actual books.

Player's Handbook. Dungeon Master's Guide. Monster Manual. Not summaries. Not "search the internet." The actual source material — hundreds of pages of rules, tables, and reference content.

Everything changed. Spells were accurate. Monsters behaved according to the rules. Every answer came with a reference to a specific page. Same model. Same Claude. The only difference: books.

That moment stuck with me — because I'd been hitting the exact same wall in my real work.

A few weeks earlier, I'd tried using Claude as an investment advisor. I'm deep into value investing and options hedging, and Claude came back with gems like "it's important to diversify your portfolio" and "maintain a balanced risk-reward ratio." Thanks, dad.

I gave it books. Security Analysis by Graham & Dodd. Options as a Strategic Investment by McMillan. Suddenly it was discussing specific margin-of-safety calculations, recommending covered calls against existing positions, analyzing sector-specific multiples. A completely different conversation.

Then the same thing happened with UI/UX — I loaded 21 design books, 6,114 pages, and Claude started giving feedback sharper than most designers I've worked with. Same thing with system architecture after loading Designing Data-Intensive Applications. And the same with Anthropic's own documentation on building agents and skills — material their model doesn't know because it was published after training.

Same model every time. The only variable was the books.

At this point, I couldn't ignore what I was seeing. There's a fundamental gap between what an AI model kind of knows from training and what it can do with actual reference material.

Training data is like a distant memory. The way you "remember" learning physics in high school — a general sense, but try solving a specific problem and you're stuck. A book open on the desk, on the other hand? Precise, specific, with page numbers.

Anthropic themselves wrote: "Intelligence is not the bottleneck — context is." Their research on multi-agent systems found that 80% of the variance in agent performance comes from how they consume and use information — not from the intelligence of the model itself.

Eighty percent. Not how smart the model is. How it reads.

Daniel Kahneman's framework of System 1 and System 2 thinking maps perfectly here. System 2 — deep analytical reasoning — operates on whatever material System 1 feeds it. If System 1 delivers noise, it doesn't matter how brilliant System 2 is. The output will be noise. An AI model without a structured library is a remarkable reasoning engine running on murky input.

There's a striking parallel in chess research. Studies found that grandmasters don't actually remember more individual positions than amateurs. The difference is organization — they store knowledge in schemata, structured patterns that enable instant retrieval. That's the difference between dumping everything into a context window and having a curated library with a table of contents.

I built a tool that gives any AI agent access to a structured knowledge library. Upload books and documents. Search across them. Read specific chapters. Follow cross-references ("see Appendix A"). Cite with page numbers.

I called it CandleKeep — named after the legendary library-fortress of the Forgotten Realms in D&D, the place where all the world's knowledge is stored and accessible to those who know how to search.

The name felt right because it came from the same moment of insight. I was thinking about what makes a good Dungeon Master: they don't try to memorize every rule. They know where their books are. When a question comes up mid-game, they open the right book to the right page. That's exactly how an AI agent should work — not "remembering" from training, but reading from the library.

CandleKeep is built around a simple idea: agents should read books the way developers read documentation.

• Upload any PDF or markdown document to your personal library
• Search across your entire library to find relevant content
• Read specific chapters, pages, or sections — not the entire book at once
• Follow references — when a book says "see Chapter 12" or "refer to Appendix B," the agent can actually go there
• Cite with page numbers — so you can verify anything the agent tells you

It works as a Claude Code plugin, a CLI tool, and a web app. Your library lives in the cloud, accessible from any session.

The key insight is progressive disclosure. Just like a Dungeon Master doesn't read every rulebook cover-to-cover before starting a game, an agent shouldn't dump entire books into its context window. It should search, find the relevant section, read it, and follow references as needed. Precisely the way Claude Code reads files in a repository — scanning some, jumping to related files, following imports.

In the AI agent ecosystem, there's an important distinction that often gets blurred. Skills tell an agent how to work — they're procedures, workflows, step-by-step instructions. Knowledge tells an agent what to know — it's reference material, domain expertise, the accumulated wisdom in books and documentation.

You need both. But most of the tooling today focuses on skills while ignoring knowledge entirely. That's like training a doctor on surgical procedures but never letting them read a medical textbook.

CandleKeep exists to fill that gap. It's the library that makes the skills work.

The difference between an agent that stammers and one that answers with confidence is never how smart it is.

The difference is whether it has a library.

Like the difference between a Dungeon Master who mumbles "uh... I think the monster has about 50 hit points?" and one who opens the Monster Manual and says: "Adult Red Dragon. 256 hit points. Legendary Resistance, 3/day. Let's play."

This is the first post in a series about CandleKeep. Next up: how I use it day-to-day, and how others in the community are using it.

Try CandleKeep at https://www.getcandlekeep.com