Gabriel Ayuso
@gabrielayuso.eth
Why do people think context window size is a big limitation for AIs to be able to fully replace humans as coders? I'd say most of us keep a very compressed version of the codebase in our head and rely on code search and tools to get the details we need. Coding agents are already starting to do this using grep and other tools. They just need that initial context to get them going and then they can do the rest. Creating that initial compressed representation of the codebase they can reference to get them started would do the trick without keeping the entire codebase in the context.
9 replies
2 recasts
50 reactions
polymutex
@polymutex.eth
Is there good tooling around automatically generating this compressed representation?
1 reply
0 recast
1 reaction
Gabriel Ayuso
@gabrielayuso.eth
Haven't spent much time on this other than the use of CLAUDE.md and Cursor rule files. You can ask Cursor and/or Claude to help generate these with the structure of the codebase, etc. Beyond that I mostly just guide the model by telling it relevant files, relevant things to look for, etc. Transferring my knowledge of the code base to the model. I should spend more time automating this though.
1 reply
0 recast
2 reactions
polymutex
@polymutex.eth
I ask because I'm working with local-LLMs only where the best they can reasonably do is in the 64k context window range. Still seems like they may be able to achieve something, if given tools to traverse some sort of tree of progressively-detailed nodes, with the root being a general short description of the codebase, and each node diving down into a specific aspect of it. That way they could select the nodes they need to care about, put that as background knowledge, and still get something done. The problem is generating such a tree to begin with, and having these local LLMs have enough reasoning ability to understand their own context limitation and their need to be selective about which nodes to keep. Feels like there's an opportunity here for a larger non-local model to do a one-off pass over the codebase, and generate this knowledge tree in a committable form that's a bit more structured than a flat CLAUDE.md... If you automate this let me know!
1 reply
1 recast
2 reactions
Lokp Ray
@lokpray
you can train an autoencoder for summarization and fine tune your LLM on the output (optional) while maintaining the 64k context window size. That's what ppl did a year ago when llama 3 was released
1 reply
0 recast
3 reactions
polymutex
@polymutex.eth
Is there ready-made tooling out there to create such finetunes on specific codebases, and have them be shareable (ideally in the repo itself)? (Also, would I need beefier GPUs to be able to create it?)
0 reply
0 recast
1 reaction
