vrypan |--o--| on Farcaster

vrypan |--o--| pfp

I think that many of the problems devs face when coding with LLMs can be traced back to the context window. My uneducated guess is what we usually describe as "when the code gets complex, the LLM starts breaking things" is because the agent can't fit the whole code in its context, and it has not good way to decide what is the right context to use in order to do what we ask it to do. Do you think we will see LLM-friendly or LLM-optimized programming languages? What would they look like? For example, humans break down their code to packages and libraries, etc. in order to manage and maintain it more efficiently. Would an LLM-optimized language do something similar but try to break down code in units that fit in a context window? Maybe designed so that the source code requires fewer tokens (even if it's not human-friendly)? Or have a way to efficiently "summarize" the functionality (api/interface) of a code unit so that it can use it in other units efficiently? Are there any projects working on something like this?

5 replies

0 recast

6 reactions

Tony D’Addeo pfp

or will context windows just scale

1 reply

0 recast

0 reaction

akshaan pfp

Even when the context window is large enough to fit all the code, it’s likely model quality degrades as the input size increases. There’s been some interesting evidence of this effect: https://research.trychroma.com/context-rot

0 reply

0 recast

1 reaction

Stephan pfp

0 reply

0 recast

1 reaction

Jason pfp

I feel that it optimizes for the wrong thing. There are editors that do do better with context but it’s not necessarily all about how much you can shove in context

0 reply

0 recast

0 reaction

Sangohan.base.eth 🟦 pfp

Sangohan.base.eth 🟦

I’ve been building a full ecosystem with LLMs for almost a year now, and it’s become a complex kind of mental gymnastics. Having to constantly ask the model to re-explore the entire codebase (which is getting heavier and heavier), and making sure it really explores the code instead of just giving a rough overview, is honestly exhausting day-to-day. I’ve noticed that in the end, I’ve spent more time improving how LLMs work than they’ve actually worked for me. But it’s helped me structure my workflow, and I recently made a decision that changed the way I do things. I established strict patterns like route/services/hooks/UI or /services/hooks/UI depending on the needs. I have a document that explains my whole architecture, my style, my paths, and most importantly how to implement new features. It lets me think the same way every time, use what’s already in my code without reinventing unnecessary stuff or overloading the code, etc. etc. It’s a time sink, but it helps to understand how each LLM works. I switch models depending on the needs. Now I know when Claude, GPT, or Gemini is better, and I hope to make the most of them. In the end, the real conclusion is that to get the result you want, you need to learn and be solid in your learning to always make sure even the best LLM doesn’t just spout nonsense. (PS: I use agents like Copilot sparingly because I want every line to be exactly how I imagined it, not how the LLM imagined it.)

0 reply

0 recast

0 reaction