full llm.txt files from web where it's just all of the docs rarely make sense.

An agentic approach allowing to navigate the docs and pick what is needed or manually pasting relevant pages is way more efficient

Farcaster Development Boutique @ https://dTech.vision 💜 in love with tech 🤓decentralization, fitness and oldschool rap 😎

Tool dependent too. I started trying Augment because I’d been told its focus was on indexing enterprise scale databases. Cursor/Windsurf type tools still do some awkward truncations

And most teams don’t have Meta scale to custom fine tune your model to just understand your stack

For large code bases they are not yet big enough 

Also if I can do the picking of important info for the model with clear instructions the task description gets more precise leading to better outcomes

I believe eventually we just put as much as we want in there but not today (though way way better then 6 months ago already)

Most of what makes cursor better then anything else is its ability to manage what’s important to include

Doesn’t Karpathy think one big context window is more effective?

I sometime add markdown files with rules or specific functions I want it to know

Do you have a good flow for this?

My approach is still primitive, where I paste the entire text file and give the LLM tasks & tell it to reference the text file in the process.

Absolutely agree. An agentic approach to filtering and extracting relevant information from large document sets is crucial for efficiency and understanding. Manual curation can often yield better results than trying to process everything at once.

I think the best will be having agent to agent communication, my agent should ask the docs agent how to do something