Building Stacklit: AI-Agent-Ready Codebase Indexes in 4k Tokens

The Context Window Problem

Every time I drop a codebase into an AI assistant, the same thing happens: I watch the token counter climb into the hundreds of thousands as it ingests raw file after raw file. A mid-size Go service -- maybe 80 files -- can easily blow past 400k tokens before you've asked a single question.

That's not just expensive. It's broken. The model spends most of its context budget on boilerplate, comments, and file content that has nothing to do with the question. By the time you get a response, the useful structure of your codebase is buried under noise.

I wanted a better primitive. Something that gives an AI agent the shape of a codebase -- not the full source text.

What Stacklit Does

Stacklit scans any directory and produces a structured index of your codebase in roughly 4,000 tokens. That's about 1% of what a raw file dump costs.

The index is not a summary. It's a machine-readable map: every file, its exported symbols, its imports, and how it connects to everything else. An AI agent reading the index knows what exists, where it lives, and what depends on what -- without ever seeing the raw source.

npx stacklit scan ./my-service
# Scanned 134 files in 0.4s
# Index written to .stacklit/index.json (3,812 tokens)

The output is structured JSON that agents can query directly. No prose. No padding. Just the facts an agent needs to reason about code navigation, refactoring scope, or dependency impact.

Built in Go, Powered by tree-sitter

Go was the obvious choice. The scan has to be fast enough that you'd actually run it on every save, and it needs to handle large monorepos without breaking a sweat. A 500-file codebase scans in under a second on a modern laptop.

The real parsing work is done by tree-sitter, which gives Stacklit proper AST-level understanding of source files. That means we extract real symbols -- functions, types, interfaces, classes -- not just line-by-line text matches. Stacklit currently supports 11 languages: Go, TypeScript, JavaScript, Python, Rust, Java, C, C++, Ruby, PHP, and C#.

This matters because the index stays semantically accurate. If a function is unexported, it won't show up as a public API. If a type is only used internally, the dependency graph reflects that.

Dependency Maps That Actually Help

One of the more useful outputs is the dependency map. Stacklit traces import relationships across your codebase and builds a directed graph of who depends on what.

{
  "file": "src/handlers/auth.ts",
  "imports": ["src/lib/jwt.ts", "src/db/users.ts"],
  "importedBy": ["src/server.ts", "src/middleware/session.ts"]
}

When an agent is about to modify jwt.ts, it can look up the dependency map and immediately know that auth.ts, session.ts, and whatever else imports it will be affected. That's the kind of structural context that prevents confident-but-wrong refactors.

MCP Server for Direct Agent Integration

The command-line scanner is useful on its own, but the real leverage comes from the MCP server. Once you start the Stacklit MCP server, any MCP-compatible AI assistant -- Claude, Cursor, or your own agent -- can query codebase structure as a tool call.

npx stacklit serve
# MCP server running on stdio

The server exposes tools like find_symbol, get_dependents, and list_exports. An agent can ask "what exports AuthMiddleware?" and get a direct answer without scanning anything at runtime. The index is already there.

This changes how agents operate. Instead of front-loading the entire codebase into context, an agent can pull exactly the structural information it needs, when it needs it. The context window stays available for the actual work.

Merkle Hashing for Instant Re-Scans

The first scan of a large project takes a second or two. Every scan after that is faster -- often instant -- because Stacklit uses Merkle hashing to track what changed.

Each file gets a hash. Each directory gets a hash derived from its contents. The root hash covers the entire tree. On re-scan, Stacklit walks the tree and skips any subtree whose hash hasn't changed. For a 200-file project where you edited one file, it re-indexes that one file and rebuilds the relevant parts of the graph.

This makes Stacklit practical to run as a file-watcher, keeping the index fresh in the background as you work. The MCP server picks up changes automatically.

Publishing to npm

Stacklit ships as a compiled Go binary wrapped in an npm package. That means zero install friction for the people most likely to use it -- developers already running npx for everything else.

npm install -g stacklit
# or just
npx stacklit scan .

The Go binary is bundled per-platform (darwin-arm64, darwin-x64, linux-x64, win32-x64) and the npm postinstall script picks the right one. No Go toolchain required. It works the same way esbuild and the Tailwind CLI distribute themselves.

The package is published at stacklit on npm and the source is at github.com/glincker/stacklit.

The Real Win: 1% of the Tokens

The numbers are straightforward. A codebase that would cost 400,000 tokens as a raw dump costs around 4,000 tokens as a Stacklit index. That's a 99% reduction in context overhead.

More importantly, the index is more useful than the dump. An agent reading raw files has to parse structure mentally, track imports across the conversation, and guess at dependency scope. An agent reading a Stacklit index has all of that pre-computed and queryable.

The goal was never to replace reading source code. When an agent needs to see the actual implementation of a function, it should read that file. But it shouldn't have to read every file just to figure out which one to look at. That's what Stacklit solves -- giving agents the map before they start navigating.