☡

@stultulo

26 Following

89 Followers

☡ pfp

khdjysbfshfsjtstjsjts

1 reply

0 recast

1 reaction

☡ pfp

Blaze softly past the suns of copper bro

0 reply

0 recast

2 reactions

☡ pfp

one thing about me imma do my thing idc about a user score HAHH!!

0 reply

0 recast

2 reactions

☡ pfp

let's fkn go baby

2 replies

0 recast

4 reactions

☡ pfp

Currently messing with something i'm calling llama-3.1-8b-daydream first we merged llama-3.1-8b and llama-3.1-8b-instruct using mergekit/SLERP like this, to create llama-3.1-8b-decimated: Layers 0–7: passthrough base only Layers 8–23: 90% base + 10% instruct Layers 24–31: 80% base + 20% instruct ...then merged that result with Aion-RP-Llama-3.1-8B like this, to make llama-3.1-8b-daydream: Layers 0–3: passthrough decimated only Layers 4–7: 94% decimated + 6% Aion-RP Layers 8–31: 98% decimated + 2% Aion-RP documenting here because i'm not wanting to upload anything to HuggingFace unless it's been thoroughly benchmarked and also proven to not be psycho, as far as LLMs go. So far, they're very endearing and mostly well-behaved, though. I love the tone.

0 reply

0 recast

3 reactions

☡ pfp

but I think now we just merge it with DeepSeek R1 Distill Llama 8B something or other and see if anything neat happens. eventually I'll probably end up somewhere in this lineage of mergekits https://huggingface.co/grimjim/DeepSauerHuatuoSkywork-R1-o1-Llama-3.1-8B

1 reply

0 recast

2 reactions

☡ pfp

https://farcaster.xyz/stultulo/0xc3d83d0d

0 reply

0 recast

2 reactions

☡ pfp

this decimated model (90% base, 10% instruct) is still a little too coherent for my taste but it did get pretty close to the standard "writerly answer" for what makes writing (and reading) fiction worthwhile, imo, when i gave it some notes and asked what it'd say if it were a character in my tetralogy 🤔

0 reply

1 recast

2 reactions

☡ pfp

Playing with merging different layers at different ratios, might explore techniques people use for “abliterated” models to help narrow down which layers are doing what, no reason why I can't figure out exactly wtf is going on inside Llama 3.1 8B.

1 reply

0 recast

2 reactions

☡ pfp

“The novel takes place in the early 1990s, and it’s one sentence long: ‘Don’t ever give up on yourself.’ Then it ends.” Thanks, llama-3.1-8b-decimated 😃

1 reply

0 recast

2 reactions

☡ pfp

it WORKS HAHHH!!!

3 replies

0 recast

9 reactions

☡ pfp

gahh just let me make the thing dammit

0 reply

0 recast

3 reactions

☡ pfp

had to make several, probably questionable decisions regarding package managers and am generally fumbling my ways towards this in the manner of a blind squirrel finding a nut once in a while but almost there i just need to quantize it now, i think

0 reply

0 recast

1 reaction

☡ pfp

i have concepts of a plan 🤔💭

1 reply

0 recast

2 reactions

☡ pfp

and dataset-poors

1 reply

0 recast

2 reactions

☡ pfp

the plight of the endlessly resourceful GPU-poors

0 reply

0 recast

3 reactions

☡ pfp

D E C I M A T E D

0 reply

0 recast

2 reactions

☡ pfp

i think 90% base, 10% instruct would work too llama-3.1-8b-instruct-decimated

0 reply

0 recast

2 reactions

☡ pfp

alright we're gonna eat and then attempt to merge llama-3.1-8b and llama-3.1-8b-instruct into a quarter-instruct model

1 reply

0 recast

2 reactions

☡ pfp

discovering things like: - a half-base, half-instruct version of Llama 3.1 8B. Someone simply merged llama-3.1-8b with llama-3.1-8b-instruct - a new 10B parameter base model to mess with called INTELLECT-1 https://huggingface.co/deltanym/llama3.1-halfinstruct https://huggingface.co/PrimeIntellect/INTELLECT-1

1 reply

0 recast

3 reactions