Why do people think context window size is a big limitation for AIs to be able to fully replace humans as coders?

I'd say most of us keep a very compressed version of the codebase in our head and rely on code search and tools to get the details we need. Coding agents are already starting to do this using grep and other tools. They just need that initial context to get them going and then they can do the rest. Creating that initial compressed representation of the codebase they can reference to get them started would do the trick without keeping the entire codebase in the context.

The open web is dead, long live the open web. ⌐◨-◨

Building /farcaster

— gabrielayuso.com

swe @farcaster || prev: NEAR, Nethermind, Berkeley

I fine-tuned some models for enterprise use cases back then (for work), but otherwise, ppl don’t fine tune as hobbyists. There are some folks from yc building “cursor for excel” and “cursor for CAD” and they are fine tuning since the default model wasn’t trained with that much data on excel/CAD.

I don't see much talk about LoRA and fine-tuning these days. Seems that most are just fine hitting the base models since cost has gone down and speed has gone up. I'm sure folks like Cursor are doing it.

ikr... It used to be the case that context window size was a big deal when chatgpt was first released but with all the recently development in the past 1-2 years including but not limited to:

- RAG for live code search
- LoRA & fine-tuning
- Agentic workflows: SWE-agent, MCP
- Context caching

In an agentic future (no hype intended here), focus has been more on various methods of accessing live raw data than eating everything up from context window for processing

Also, we have 1M+ context window these days from Google/Meta and it's going to go (prob exponentially) as they scale up the number of transformer layers in the NN (aka just more compute needed for initial training)

Contributor @ /walletbeat.

Censorship resistance requires privacy.

I ask because I'm working with local-LLMs only where the best they can reasonably do is in the 64k context window range.
Still seems like they may be able to achieve something, if given tools to traverse some sort of tree of progressively-detailed nodes, with the root being a general short description of the codebase, and each node diving down into a specific aspect of it. That way they could select the nodes they need to care about, put that as background knowledge, and still get something done.

The problem is generating such a tree to begin with, and having these local LLMs have enough reasoning ability to understand their own context limitation and their need to be selective about which nodes to keep.

Feels like there's an opportunity here for a larger non-local model to do a one-off pass over the codebase, and generate this knowledge tree in a committable form that's a bit more structured than a flat CLAUDE.md...

If you automate this let me know!

Haven't spent much time on this other than the use of CLAUDE.md and Cursor rule files.

You can ask Cursor and/or Claude to help generate these with the structure of the codebase, etc.

Beyond that I mostly just guide the model by telling it relevant files, relevant things to look for, etc. Transferring my knowledge of the code base to the model.

I should spend more time automating this though.

Is there good tooling around automatically generating this compressed representation?

Im the guy that posted all those factory memes once

It's not that bad and it's not like windowing over data does all that much to fix it. Needle in haystack is improving dramatically over time

It's not only hardware limits. LLM performance diminishes with larger contexts.

AFAIK the AIs so much better with a large context window because everything has too, at one point, live there. We shouldn't be adapting the problem to the tools but the tool to the problem. LLMs have no fundamental reason they can't pull the whole codebase into context, it's just hardware limits right now.

Agreed. Thinking we need to pass all the full files content to AI agents is a recurring midwit take I’ve seen.

Any real system is a collection of interfaces that work with each other, whilst encapsulating their messy implementation code.

You only really need
- an overview of system components and their relations
- access to a db schema
- file content related to task at hand as well as existing file examples to facilitate 0 shot success of the agent

I've spent a few hours already polishing cursor rules for my project. I include them with each request (it's a couple of pages). Works great but it required a few iterations to get them to a point where I feel that I can ask cursor in plain English what I need and it just does it exactly as I expect. I'm like 90% there

I think this explains why I haven’t found Claude Opus to be that much better than Claude Sonnet despite being like fifty times more expensive.

Cursor is already doing a lot of legwork summarizing context as it goes, so it can keep a small context window cached no matter how long the conversation gets.

I think the analogy is 
- dna: model
- everything you ever learned, remember, sensed: context

I may be wrong.

Claire Vo talking about pairing with a 91 year old vibe coder reimagines "codebase in our head"... it's possible deep human memory generates circuit connections between the LLM' interpretability and the coder's hidden features.

can you enable new beta DC API for me please