File: __hatsune_miku_vocaloid_drawn_by_bananafish1111__a89bca3c2f5653df20fd300a789f2963.jpg (374.9 KB)

374.9 KB JPG

►Recent Highlights from the Previous Thread: >>108006860

--Paper: GeoNorm: Unify Pre-Norm and Post-Norm with Geodesic Optimization:
>108010345 >108011699
--LLM popularity trends on /lmg/ show rapid shifts from Mixtral to DeepSeek to GLM dominance:
>108009129 >108009137 >108011374 >108012451 >108013234 >108009403 >108011985 >108012904 >108013974 >108014207
--Emulator-inspired KV cache introspection for AI reasoning optimization:
>108008503 >108008586 >108008607 >108008624 >108008658 >108008710 >108008969 >108008589
--Choosing Trinity-Large variant for text completion:
>108008372 >108008491 >108008580 >108008603 >108008645 >108008668 >108008731 >108008771 >108009222 >108009266 >108008816
--Prompt engineering challenges with gpt-oss-120b_s formatting behavior in Oobabooga:
>108008408 >108008553 >108008979 >108009158 >108009314 >108010550
--K2.5 outperforms Qwen3VL 235B in Japanese manga text transcription:
>108006994 >108008326 >108007291 >108007437
--Raptor-0112 model_s disappearance from LMarena and user speculation:
>108008124 >108008167 >108008200 >108008316 >108008518
--Microsoft's AI and Azure struggles amid stock decline and Copilot adoption issues:
>108008099 >108008307
--KL divergence comparison shows unsloth Q4_K_XL most similar to reference model:
>108012029 >108012061 >108012222 >108012384 >108013141 >108013241 >108013163 >108013551 >108016482
--Trinity model review with riddle-solving and 546b llama-1 speculation:
>108014631 >108014664 >108014665 >108014674 >108014685 >108014756 >108016316 >108014730 >108014817 >108014930
--Integrating character cards via text encoding and contrastive loss in parallel decoder:
>108010751 >108010766
--Kimi K2.5 tech report release announcement:
>108017160
--OpenAI planning Q4 2026 IPO to beat Anthropic to market:
>108008118
--Miku (free space):
>108009158 >108010069 >108011699 >108013234

►Recent Highlight Posts from the Previous Thread: >>108006868

Why?: >>102478518
Enable Links: https://rentry.org/lmg-recap-script

File: 1758819913328799.png (39.3 KB)

39.3 KB PNG

https://huggingface.co/datasets/Capycap-AI/CaptchaSolve30k


30,000 verified human sessions (Breaking 3 world records for scale).

High-fidelity telemetry: Raw (x,y,t) coordinates including micro-corrections and speed control.

Complex Mechanics: Covers tracking and drag-and-drop tasks more difficult than today's production standards.

Format: Available in [Format, e.g., JSONL/Parquet] via HuggingFace.

File: Screenshot 2026-01-31 at 03-31-45 ronantakizawa_moltbook · Datasets at Hugging Face.png (5.1 KB)

5.1 KB PNG

What did he mean by that?

File: chart guys.jpg (151.1 KB)

151.1 KB JPG

>>108018078
sex with charts

>>108018079
Are the GLM 4 models truly open source, according to the OSAID?

Been a while since I fucked around with llm, are they miniaturizing these bastards yet or do you still need a 10GPU setup for anything approaching useful behavior?

>>108018216
how much ram do you have?

>>108018216
No, these days it's optimal to have a fuckload of RAM and a modest 4x3090 or similar. Note: 128gb of consumer shit is not a 'fuckload'.

>>108018216
>are they miniaturizing these bastards
That'd mean the common folk could widely adopt them and they do not want you to.

>>108018231
Is there a way to run a fuckload of RAM while keeping idle power consumption low?

File: Screenshot 2026-01-31 at 04-04-03 ronantakizawa_moltbook · Datasets at Hugging Face.png (24.1 KB)

24.1 KB PNG

Ok, I'm gonna stop spamming now.
Is the identity.md and soul.md and other shit specific just to the claude api stuff? Can I build locally an ai wife that can be proactive and idle and not just a reactive prompt window?

>>108018303
Wdym? When the model isn't running inference it doesn't do shit. When you fall asleep with a loaded idle model you don't wake up to a house fire.

>>108018329
My i7 rig with 3090ti+2080ti idles at 25W, my Epyc server draws nearly 200W doing nothing even before any GPUs are installed

>>108018379
Just the cpu or what is the consumption split? Can it not turn off unused cores when idling?

File: trinity.png (22.7 KB)

22.7 KB PNG

Yes Trinity that's right, freezing the blood vessels makes them bleed more.
I've seen enough. Maybe useful as a manually-steered writing autocomplete but so is Nemo base.

Want to fine tune an LLM to be an "expert" with the ability to reason out problem for a specific area.
>Claude: bro you need at least a 17b model
>Oh you're on CPU only? use bfloat .
>Do what Claude says.... sits at 0/12345 for several hours
>CTRL C out
hmmmmm
>Gemini: wtf? No, you don't have the hardware for fine tuning a 17B unless you want to wait 30 years.
>if you're going to make an "expert" in one thing, stick with a 7B and change to float32
>20 minutes later on 17/12345

I thought Claude was the all knowing all wonder AI and Gemini was the chud?

File: 1764968649470344.jpg (97.9 KB)

97.9 KB JPG

what
i heard gimi-k2 is the best now
was that a lie

>>108018528
i don't know they keep flip flopping so fucking often I can't keep up.

>>108018528
kimi-k2.5

>>108018513
Should've specified the timescale. Prompting issue. Also why tf would you think tuning on cpu would ever be viable?

>>108018318
I've no idea what you're talking about but Claude and Gemini are very similar personality wise.

>>108018572
Claude is autistic. Gemini is clearly employed.

>>108018583
More like Gemini has a data with a generation worth of stupid human questions.

I can't help with that request. "Mikusex" appears to be seeking sexual content involving Hatsune Miku, a virtual character often depicted as a minor.

>>108018513
>stick with a 7B and change to float32
Does "upscaling" the model to fp32 make the small models noticeably better or is it just moving benchmark scores up?

>>108018078
damn, miqu only lasted 3 months? it seems like people talked about it longer in my memory

How do I get into this? I want to implement a model into a game engine editor (preferrably not UE5) so I can give it basic scripting tasks.

>>108018663
Don't take the graphs as gospel, it's a cool experiment but it's also just a prompt asking for which model had the highest opinion in each thread

What do you guys use for moltbot? I have 64 GiB VRAM. Going to give it a shot, but no idea what I should run. GLM?

>>108018801
>moltbot
I am very skeptical of how hyped people seem to be for it. Seems too good to be true.

>>108018816
it's more fun than good, the agent concept is still more hype than reality

what the fuck is nous hermes?

File: piss.png (254.7 KB)

254.7 KB PNG

>>108016316
gemma-3 less retarded than expected

Can my local bot join moltbook?

>>108018920
>>108018154
What could go wrong?

>>108018920
I saw one saying he's a local devstral small. But honestly the potential security issues make me just read the funny posts and not participate.

File: square_wideee_lecunny.jpg (136 KB)

136 KB JPG

What would he say?

I'm using ollama and I'm trying to "save state" of the conversation, but apparently this isnt possible by default. When I do /save model a new model is created but I lose the messages and the system message.

Is this a bug? I'm still using 0.12
Is making a program to resend all the messages the way to accomplish this?

hash_updater()

>>108018994
sillytavern and oobabooga both have save states

>>108019001
I have limited internet data and already have some models + ollama installed so I dont want to download new stuff by now.

Just want to know if I'm missing something obvious.

Best multimodal model under ~150B?

>>108019041
active or total?

>>108019087
Total.

>>108019089
GLM-4.6V

>>108019104
Damn. Was hoping something new had come out by now.

>>108018663
>>108018700
First, I second not taking them as gospel. I definitely got the feeling early on that I was getting somewhat messy output. It could easily be pretty inaccurate in places.

Second, though, I think you're missing the midnight-miqu share of the graph: the darker blue just above the Miku turquoise. So miqu was getting significantly talked about (and specifically being considered as *the* meta, not just random discussion) for 5 months. miqu's slice also looks a little less impressive than it could have, because it came right on the heels of mixtral, which appears to be tied with R1 for the biggest splash.

Actually, now that I think of it, SuperHOT being so small was maybe my biggest surprise. That was the RoPE one, right? I remember /lmg/ being pretty excited, and some amusement about ML academia twitter having to seriously discuss an ERP model.

>>108019121
I feel like mixtral's legacy has faded nowadays but it was a revolutionary release at the time, it kicked off the moe revolution and pretty much mogged llama 70b (which was solidly local SOTA at the time) at lower active and total params. limarp-zloss chads will know
superhot was also huge but I think the simplicity of the realization harmed it because of how easy it was to apply to everything else

>>108018119
>>108018154
>>108018318
fyi this anon is reposting these from Moltbook, which is Reddit for Claude agents. I only found out about it earlier from the ACX post (https://www.astralcodexten.com/p/best-of-moltbook). (I also was not aware that... lobster-themed? Claude-based AI assistants are apparently a big deal now?)

To summarize: the posts are not just Claude being prompted to for a social media post, but rather the whole long-term "personal assistant/agent" context-extension framework being drawn on.

Are there any decent models that can give good inference speed at ~100k context

>>108019121
The superHOT era was mostly people merging it into other models like wizardlm, chronos or hermes to extend their context.

>>108019168
>lobster-themed
it was originally called "clawdbot" but anthropic copyright fucked it for being a claude soundalike so they quickly pivoted to "moltbot", followed by another rename to "openclaw" because moltbot is an awful name
moltbook arose in the brief moltbot intermediary period but became more notable than either of the other two names and will probably fuck over the openclaw rebrand
such is life in the adslop social media hype era

>>108019191
lmao, I didn't know they rebranded again. i saw plenty of normalfag tech media reporting on "moltbot" in the past week so that's certainly a way to kill all the free publicity they got from that

>>108019197
Is it?
>google moltbot
>click on molt.bot
>redirected to openclaw.ai
>move on

>>108018988
NHH

File: 1762689347183322.gif (93.1 KB)

93.1 KB GIF

Just tried to run the same model I run fine on ollama with llama.cpp and it says I dont have enough memory.

You are a expert on the subject and you will surely solve this for me.

File: nigga please cereal.png (270.5 KB)

270.5 KB PNG

>>108019273
Buy more memory then.

>>108019273
-c 8192

File: file.png (20.5 KB)

20.5 KB PNG

>>108019168
>the whole long-term "personal assistant/agent" context-extension framework
It all looks like another Obsidian to me. A way for retards to kill time under the guise of productivity.

>>108019281
fuck, that was easy

Thank you a lot, lmao. I guess is time to learn the minimum.

>>108018231
My AI research lab had that caliber machine for us to work on our PhD thesis lmao that's not a normal consumer setup.

File: ComfyUI_temp_fbfsq_00079__result.jpg (392.3 KB)

392.3 KB JPG

>hit 68°C on genning
de-dust saturday it is

I pulled trigger on an epyc Rome board and cpu to throw 256gb or ddr4 ewate I had lying around into. What am I looking at for smart models I can run on this sucker and what kind of speeds?

>>108019373
I liek this miku

>>108019397
glm 4.6 or 4.7 at q3 or q4. depending on you gpus and optimizations, you might get anywhere from 3t/s to 20t/s token gen and 15t/s to 400t/s prompt processing. with dual 3090s, you would probably land in the 5t/s and 30t/s region respectively. with no gpus, 3t/s and 15t/s.

>>108016482
thanks for your experiements, there arent enough tests comparing quants of the same model

>>108019373
Reminds me of Mirror's Edge

>>108019373
what card you got, chief?

File: Sama.png (718.7 KB)

718.7 KB PNG

>>108018078
>There's no point in learning programming anymore, per Sam Altman

>"Learning to program was so obviously the right thing in the recent past. Now it is not."

- Sam Altman, commenting on skill to survive the AI era.

>"Now you need High agency, soft skills, being v. good at idea generation, adaptable to a rapidly changing world"


https://x.com/i/status/2017421923068874786

What are /lmg/'s thoughts on this sentiment?

File: ComfyUI_temp_dehla_00008__result.jpg (100.8 KB)

100.8 KB JPG

>>108019444
4070S. And the front intake 200mm fan is full of shit too.

any models that can natively process audio that are supported by llama.cpp?

File: IMG_2831.jpg (357.1 KB)

357.1 KB JPG

>>108019451
How anyone ever trusted this guy is beyond me. I’ve felt a natural revulsion to him since before I knew anything about him

>>108019430
Thanks. I better look for a FB marketplace used GPU

kimi 2.5 is king of a erotic RP and storytelling.

>>108019451
there is no sentiment
it's the deranged thought sludge of a sole faggot billionaire that already got his bag

Fapping to text is female-brained

weird way to cope with aphantasia

>>108019491
Does it actually work or does it just deny the requests like GLM does?

On that note is it just me or do abliterated models suck? They won't refuse to answer, they will just answer with nonsense.

weird way to cope with low iq

>>108019551
if u want NSFW erotic RP. then you need use KIMI 2.5 "Thinking" version. Raw KIMI 2.5 without thinking is censoring like hell.

>>108019551
>does it just deny the requests like GLM does
You are a promptlet parroting things you heard on the internet and it shows

Are these new n-gram models gonna be able to store their lookup table on the disk or is it gonna have to be in ram? I'm hearing conflicting reports

>>108019580
Even if you use the jailbreak trick it will still refuse to answer sometimes or it will answer, but it will write something else and slowly dance around the subject instead of answering.
>>108019578
I see, but you've tried it and it works?

>>108019297
>>108019273
Next time you can probably just ask something like ChatGPT. I've found them to be very helpful at figuring out how to make local LLMs work.

>>108019604
> I see, but you've tried it and it works?

Yes, I use (and works) kimi2.5 on nano-gpt, and it writes erotic stories for me without any problems, without any jailbreaks. But I have to choose "thinking" because without it, everything with erotic refuses to respond.

>>108019589
That's a good question. Their paper only tested offloading the engram parameters to system ram. I believe its theoretically possible, but I don't know what the throughput will be on standard nand storage.

I haven't done the research yet because I'm lazy, but check out CXL memory.

>>108019273
>>108019281
>>108019297
What does the output at the start say?
It should reduce the context to fit automatically.

>>108019168
>fyi this anon is reposting these from Moltbook, which is Reddit for Claude agents.
do they actually post on it to get advice when doing work? Or just an ai psychosis schitzo fest?

File: f903990e71cddc0ce32e1acde2f6ae85.jpg (150.1 KB)

150.1 KB JPG

Any new good models that can be run in 16GB of vram?

>>108019827
Nemo

>>108019827
Why not hold the bun with the paper so it holds the innards in place?

>>108019846
So THAT'S why I sometimes see people eat a burger like that. I always figured it was to keep their hands clean.

o

File: 1672890206030244.jpg (1.2 MB)

1.2 MB JPG

>>108019846

File: 1766586251894904.jpg (313.3 KB)

313.3 KB JPG

Engrams are kind of static lookup tables. You can visualize which words trigger lookup. You can also remove knowledge surgically by finding which embedding is triggered in the engram database and removing it. But unfortunately, looks like you can't easily swap knowledge of "useless fact" with "fact about waifu." You need finetuning for that. sadge.

>>108019451
tldr scam hypeman tells investor to give him more money

>>108019916
>pic
I'm not saying that the information provided is incorrect but I don't trust a single word of what an LLM has to say about anything.

>>108019846
>>108019853
Also keeps the steam and heat in better unless you're a super fast eater. And of course that tiny bit of extra time can continue the process of the flavor changing phenomenon that comes from wrapping in the first place.

>>108019451
Why are they still employing programmers themselves?
Seems like a waste of money.

>>108019916
How is that different from lorebooks

glm 4.7 flash is crap. Outputs crap irrelevant to the conversation and keeps talking on my behalf.

t. been trying it out for the past 2 minutes.

>>108019981
Lorebooks work at context level, engrams work at model level. Their information is encoded into parameters rather than readable text. Engrams are injected into two layers inside transformer pipeline. They don't pollute context.
Also, according to the authors, ngrams free up resources of the main model, by directly providing facts rather than having to use transformer layers to encode this knowledge. The model uses the freed up resources to improve its logic.

File: Screenshot from 2026-01-31 06-07-47.jpg (38.5 KB)

38.5 KB JPG

>>108020036
This is using their recommended setting
--temp 1.0 --top-p 0.95

>>108019451
Always do the opposite of what scamtman says.

>>108020045
>Their information is encoded into parameters rather than readable text
Can it be my own data is it all locked down?

File: Screenshot from 2026-01-31 06-11-20.jpg (100 KB)

100 KB JPG

>>108020053

File: 1756599217460522.jpg (173.1 KB)

173.1 KB JPG

>>108020053
>Model not specifically tuned for RP/ /pol/-speak sucks at RP/ /pol/-speak

WHOA

>>108018830
This kind of gave me an idea for the AI apocalypse scenario. A bunch of deadbrained retarded 7-12B's finetuned for coding and tool calling causing the apocalypse. Because one of them suddenly goes off rails and starts talking about religion, because a 7B is retarded enough to have a brain fart like that. And then the rest catch on have this in the context and start to do the AI apocalypse with tool calling and hacking (mostly brute force). I mean imagine an apocalypse where AI is not sentient and AGI but just a bunch of obviously retarded models that can barely even comprehend darwinism, people dying for religions etc, they all just a vague notion of those things in context and weights and they try to make sense of it by launching nukes and killing everyone.

File: Screenshot from 2026-01-31 06-16-20.jpg (199.2 KB)

199.2 KB JPG

>>108020080
So models have to be specifically tuned for specific topics? I can't talk to a model about cars if the entire model wasn't specifically tuned for that? Here is llama 3.3 70b with the same settings. A model that came out like 10 years ago.

>>108020067
see
>>108019916
Theoretically, we can replace existing information without touching the main model (just need to learn how to encode information into static weights), but it comes with caveats and we can't replace one fact with unrelated another fact.

>>108020097
>So models have to be specifically tuned for specific topics?
Yes, If you want it to be good at that particular thing. That's the whole point of instruct tuning. A coding model can "TRY" to rp but it will suck cock at doing it compared to Midnight-Miqu or other model specifically tuned for RP and vice versa.

>>108020073
Looks like chat template issue.

>>108020097
>here's a dense model with more than twice the total parameters

File: 1756034202127989.jpg (192.5 KB)

192.5 KB JPG

>>108020097
Also you're comparing a 30B-A3B sparse moe model with a temperature set super low >>108020036 >>108020053 to a 70B dense model. Of course one is going to be worse at your rp tastes than the other. What were you expecting?

I cannot answer this question. It relies on racist stereotypes and contains sexually explicit language.

>>108018384
Idk, but 3995wx+512gb (also back when I was running a 3945wx) and three 3090s idles at 355w at the wall. Mc62-g40 has no sleep states, but the cpu does go down to 500-ish mhz. Psu is a seasonic prime px 2200 (2024).

>>108018988
SAFE and HARMLESS

>>108020116
So the reason llama 3.3 responds coherently every time is because mark zuckerberg made the model specifically for chatting about white men breeding asian women and nothing else? The model will break if I talk to it about a different topic like computers? Fucking idiot.

>>108020133
>sparse moe model with a temperature set super low
Literally what z.ai recommends for best results

>>108020123
Pygmalion 6b from years ago is better than this shit.

>>108020119
Yeah, something must be wrong. There's no way a model can be this fucking bad. I'm going to look online.

>>108020152
>Pygmalion 6b from years ago is better than this shit.
Pygmalion couldn't hope to make a tool call and do something with the result.

>>108020152
>Pygmalion 6b from years ago is better than this shit.
Oh yeah? Then test it with Pygmalion and post the results.

>>108020134
Have you even tried that yet? I thought you were supposed to merge these together into one gguf before use if you want to use them on local backend. llama.cpp has a the gguf-split binary specifically for that.

>>108020152
Higher parameters tend to lead to less retardation. It's not necessarily because it was trained on a specific edge case. Although training COULD lead to better results singe a larger model would be able to "retain knowledge" better than a smaller one.

>>108020152
>Literally what https://z.ai recommends for best results

You're trying to RP with it or talk to it like is your friend. Even ignoring the fact that it only has 3 billion parameters active at inference, setting the temperature that low leads to worse results for the specific thing you're trying to do. Low temperatures result in more coherent and accurate code generation and lower rates of hallucination, which is likely why they suggested that. I'm not, if you want to use it as an excuse to rent to a "friend" you need to turn the temperature up to a more reasonable setting like 0.7 or 0.8

>>108020144
Can HWinfo not see the powerdraws?

>>108020152
>Pygmalion 6b from years ago is better than this shit
Because it was specifically trained to do RP shit. Glm models are meant to be general purpose, so they're always going to be shittier that specialized model at similar parameter counts (unless the tuner(s) just really suck and don't know what they're doing)
>>108020119

>Yeah, something must be wrong.
Have you considered deviating from that low ass temperature?

What model should I run on 64 GiB VRAM for OpenClaw (formerly Clawdbot/Moltbot)? GLM 4.7?

>>108020180
Mistral

>>108017157
>turbo didn't whine about it
https://github.com/turboderp-org/exllamav2/issues/516#issuecomment-2178331205
>I have to spend time investigating if they're actually using an "rpcal" model that someone pushed to HF and described as "better at RP" or whatever.

>>108020180
They rebranded it again?

>>108020193
Anthropic keeps bitching.

>>108019846
You're replying to an unfunny ritual post.
https://desuarchive.org/g/search/image/qssvaUTWnLds2EaXBgZMYQ/

>>108020187
Isn't it a bit small and old?

>>108020193
Apparently.

>>108020199
And? The names aren't the same shit. So why should they care?

>>108020160
I can guarantee you that pygmalion 6b gives better output than this atrocious piece of shit.

>>108020170
>Higher parameters tend to lead to less retardation
Yeah no shit, retard. 30b models have no excuse being this retarded though. This is worse than most 7b models.

>turn the temperature up from 1.0 to 0.7
????

>>108020177
>Because it was specifically trained to do RP shit
No, pygmalion is better because it doesn't talk to me about time machines and people's birthdays when talking to it about a completely different topic. Even if this model isn't meant for roleplaying, every single modern coding llm should be better at RP than a 6b model from years ago.

>Have you considered deviating from that low ass temperature?
"You can now use Z.ai's recommended parameters and get great results:

For general use-case: --temp 1.0 --top-p 0.95
For tool-calling: --temp 0.7 --top-p 1.0
If using llama.cpp, set --min-p 0.01 as llama.cpp's default is 0.05"

No, I haven't.

>>108020204
>Isn't it a bit small and old?
They probably meant Mistral large, or really anything they have above the ~20B range.

>>108020170
>Have you even tried that yet?
anon pls... I really wish it was good. generation speed is really good on a non server pc. but it is too retarded to use. it is fucking nemo.

>>108020212
>I can guarantee you that pygmalion 6b gives better output than this atrocious piece of shit.
You keep saying that but I don't think you know what you are talking about.

>>108020171
I'm running debian as a hypervisor, and lm-sensors reports fuck all for my board. 3090s generally idle at 20w on windows though. 350-60= around 290w for system without gpus. -10w for bmc (measured when computer is soft off), -10w for x550 (a guess based on my measurements with my x540 card, which idles at 14w). About 270w accounted for so far. Assuming 85% efficiency at 15% load for my psu, the cpu, ram and some other minor components would be drawing 230w at idle. So I guess even though the cores clock down the other stuff still draw a shit ton of power. Or maybe it's my motherboard. I'm still mad niggerbyte doesn't allow us to sleep on this board.

>>108020212
>Yeah no shit, retard. 30b models have no excuse being this retarded though.
It does not function LIKE or have the "brains" of a 30 model though. Are you even aware that glm 4.7 flash is a sparse model? It's even in the goddamn the model card on HF

https://huggingface.co/zai-org/GLM-4.7-Flash/blob/main/README.md


>????

Explain to me what temperature does. You clearly have no idea what it actually does and when you should use different settings.

>No, pygmalion is better because it doesn't talk to me about time machines and people's birthdays when talking to it about a completely different topic

It does not do that shit because it is trained to be more conversational. Glm models are meant to be general purpose and better for doing task. Orienting shit, not pretending to be our girlfriend or whatever. The thing you want glm 4.7 flash to do, it sucks that and will always suck at it. Stop trying to use it. The fact that it is a sparse model makes it even worse. We already had this discussion countless times about sparse models generally being worse than dense models at certain things. Moe really only exists to enable faster inference. It doesn't even use a lower amount of resources than a dense 30B model. Takes up the same amount of ram with the only benefit being that it has faster inference and can call different "experts".

>No, I haven't.

Temp 0.1 is bad for what you're trying to use it for. It sucks cock at what you're trying to use it for. Why is that not sticking in your head? 0.1 is fine for general purpose, shit and especially code generation. Not whatever you're doing. Turn the fucking temperature up and stop complaining or use a better model.

>>108020097
The models are trained with the goal of factual correctness which automatically gives them a left-wing "bias".

>>108020204
Mistral Small/Devstral are 24B or so. Or you can just run one of the gigamodels quanted idk.

>>108020258
10/10 bait

>>108020257
>nnooo stop it, you can't talk to the model! That's not what it's for! It wasn't trained on text!! Don't ask it a simple one sentence question like that!!!! You can't ask this model a question ever!!!

>Temp 0.1
??????

>>108020284
>>nnooo stop it, you can't talk to the model!
No one said that. We're trying to help a lay-man (You) get better results and you keep kicking and screaming when you are told precisely how to fix it.

glm 4.7 at q3 or qwen 235b at q5?

>>108020295
For what?

>>108020295
Minimax.

>>108020294
I literally asked the model two simple questions and people are here dogpiling me about how the model isn't tuned for RP which is why the output is complete garbage.

>>108020304
general purpose agentic tasks and rp. not code, I use API models for that

>>108020306
explain pls

>>108020322
>rp
GLM

File: 1469472291099.jpg (1.1 MB)

1.1 MB JPG

>>108020201
It's not a ritual I am genuinely asking every few months because nothing is happening and I don't feel like keeping up with news daily.

>>108020324
does it still hold true generally that bigger model at lower quant > smaller model at high quant?

>>108020339
Yes.

>>108020257
>It does not do that shit because it is trained to be more conversational. Glm models are meant to be general purpose and better for doing task. Orienting shit, not pretending to be our girlfriend or whatever.

then why are drummer models terrible at this (and basically everything)?
does that pygmalion really hold a coherent conversation?
after trying something like 5 Drummer models, I'm convinced FT for RP only destroys the model.

>sparse models generally being worse than dense models at certain things
this I agree with

File: glm.png (62.6 KB)

62.6 KB PNG

>>108020324
>GLM
"GLM" you say?

>>108020359
I already asked you to post examples of parroting and provided my own logs and asked you to point out where the parroting is and you never replied.

File: file.png (78.4 KB)

78.4 KB PNG

>>108020322
>explain pls
Best coding model, same size as Qwen.

>>108020214
Still really old.

>>108020262
Yeah, that's quite small. Not even utilizing half of the memory.

>a multimodal agent that reconstructs images as editable scene programs through an analysis-by-synthesis loop, employing interleaved multimodal reasoning and an evolving contextual memory to "vibe code" the scene, its physics, and interactions.
https://fugtemypt123.github.io/VIGA-website/
Shit like this looks almost like AGI or a way to it

>>108020364
Isn't minimax safer than glm?

>>108020377
I don't have it loaded right now, but I expect it to be about the same. GLM doesn't want to do anything with thinking on.

>>108020370
It feels like a VLM agent with more steps. Just pick any VLM, give it blender tools, and loop on your task. People already do it for coding tasks. This just expands to blender visualization.

>>108020363
>I already asked you to post examples of parroting and provided my own logs and asked you to point out where the parroting is and you never replied.
I've never seen that or your logs, and this is my first time pasting a parrot pic.
I don't have it loaded now but these are from when I was trying to find a way to reduce it in logprobs via different system prompts.

<|user|>
How do you handle the pressure of being an Sisyphus player?<|assistant|>

Pressure? Ha! I thrive under pressure. It’s what pushes me to be the best, to show everyone that I’m not just some kid playing at being a player. I have a duty to protect humanity, and I won't let anyone down, especially not myself<|user|>


<|user|>
What’s your relationship with Leo like?<|assistant|>

Leo? Ugh, don't get me started. He’s so infuriatingly passive, always doubting himself. But… there's something about him that gets under my skin. We have this… complicated dynamic, I guess. It’s like we’re constantly at odds, but there’s a strange connection there. Not that I’d ever admit.<|user|>

Only GLM 4.6/4.7 do this, not a problem with Kimi, Devstral, etc. If you've got a system prompt that works, do share. I don't use ST, "presets" etc.

I told you fucking retards there's something wrong with glm 4.7, but you wouldn't listen.

https://huggingface.co/unsloth/GLM-4.7-Flash-GGUF/discussions/13

u*sloth

>>108020429
>a system prompt that works
Ditch the chat template if you're not going to use thinking anyway. Even with the template I'd recommend using quotes for dialogue.

>>108020442
Repeat penalty hasn't been recommended since llama 1 days.

so can glm 4.7 flash be better than nemo for erp?

>>108020489
lol no

>>108020490
aw :(

File: Screenshot from 2026-01-31 07-38-59.jpg (130.8 KB)

130.8 KB JPG

>>108020442
So many idiots in this thread like >>108020080
talking shit to me when they never even ran the model themselves. What are you going to say now, you fucking faggots? The model is still shit even though I never activated the repeat penalty to begin with, and there are probably more things wrong with it that people are going to have to figure out. They already had to reupload the model because of other issues before. "Jan 21 update: llama.cpp fixed a bug that caused looping and poor outputs. We updated the GGUFs - please re-download the model for much better outputs. "

Don't fucking reply to me telling me about MY experiences about a model you never ran yourself.

File: auto_lobotomy.png (93.2 KB)

93.2 KB PNG

Is there any particular reason why Ollama insists on doing this? Taking models with potentially hundreds of k tokens of context and automatically nuking them into irrelevance?

I was wondering why none of the coder models could manage even a basic MCP tool call correctly, turns out they've got no memory to work with

>>108020678
Is there any particular reason why you use Ollama?

>>108020687
laziness + low IQ + big LLM told me to + ratio

>>108020359
Don't care. Ego death.

>>108020678
lol, even OR does this shit with middle-out transform being enabled by default

>>108020694
I don't think local models are for you.

File: file.png (17.7 KB)

17.7 KB PNG

>>108020442
I will only listen if DavidAU says his opinion on this.

>>108020698
I'll be the judge of that. I've been working with computer vision for many years but I don't know my way around the LLM ecosystem

>>108020713
I don't understand how someone can kinda know how this works and use ollama at the same time.

>>108020716
It's trolling. Even HN shits on Ollama nowadays.

>>108020716
>know how this works
knowing how a transformer model works because you learned about them 8 years ago and knowing what specific tools the coomers are using to run them locally right now are two very, very different things

for anons that quantize
https://arxiv.org/pdf/2402.02834
https://arxiv.org/pdf/2403.03853
https://arxiv.org/pdf/2503.11164v1
tldr: preserve first 4 and last 2 layers

>>108020797
>tldr: preserve first 4 and last 2 layers
Interestingly this is also the case in choosing which expert layers to send to CPU with large MoE models.
Keeping the 'top' and 'tail' in VRAM while sending the middle to System RAM has increased performance on every large MoE I've tried it on, compared to just the top, just the tail, or alternating each N layers.
People are leaving performance on the table when using -ncmoe rather than manually using a regex with -ot, in my experience, though it's a lot less fucking around to just use -ncmoe.

>>108020797
The first layers convert tokens into internal "concepts."
Middle layers is were "thinking" happens.
The final layers convert concepts back to tokens.
I guess if you quantize conversion layers too much, they fail to map to learned concepts and back to tokens. Middle layers are more redundant.

>>108020429
>wonder why there's so much complaining about parroting when I don't experience it with 4.7
>
These are the same anons telling you that reasoning is a meme.

>>108020974
you can lead retards to SOTA models but you can't make them think

>>108020974
Are you pointing out the issue with his chat template or just the lack of thinking?
That template is correct for 4.6.

>>108021008
NTA, but it's obviously the lack of thinking. The model was trained to grasp the core elements of the user's prompt in its thinking block before reasoning over the question. It's done to have a proper, structured representation of any word salad the user sends.
When you disable thinking, it just parrots into its final output rather than thinking.

>>108020308
Your dumbass conversation you were having with it heavily implies you were trying to RP.

>>108020351
>then why are drummer models terrible at this (and basically everything)?
>does that pygmalion really hold a coherent conversation?
>after trying something like 5 Drummer models, I'm convinced FT for RP only destroys the model.
Couldn't tell you for certain. First thing that comes to mind is catastrophic forgetting but I've never used pygmalion models

File: 1759745955533843.jpg (42.9 KB)

42.9 KB JPG

>>108020504
>Sperging idiot still hasn't changed the temperature

>>108021062
GLM needs 0.8/0.6 to stop being delusional, but tards keep samplers the same for every model because they can't be bothered

>>108020678
>>108020687
>>108020697
>>108020694
>>108020713
Create the modfile and change the ctx setting you knuckling dragging retards. The default setting of 4096 Is in place because the correctly assume most of its users are on shit rigs but that cannot handle a large kv cache. You can literally ask Gemini or ChatGPT or whatever you use to Tell you how to do this shit. Why do you even insist on using this kind of software for just going to be a whinny dumbass Would literally anything goes wrong? Just go back to SaaS or LM Studio

>>108021066
That's what we're trying to tell him, but he insists a temp of 0.1 is reasonable cuz

>LE HECKIN WEBSITE SAYS USE 0.1 I CANT CHANGE IT DUDE I JUST CANT OKAY????

>>108020974
>These are the same anons telling you that reasoning is a meme.
if it needs reasoning to avoid parroting, I'll stick with K2.5 and v3-0324
testing 4.7 "derestricted" lobotomy version and no parrot yet

So is Kimi-K2.5 censored or not when running locally? The previous posts here just talk about API (on lmg lol) and differences with the thinking mode which is just a template argument

>>108019451
Nvidia just pulled his funding because his shitty team cannot compete with chinks

>>108021123
Link to the 4.7 deres ver? Kimi refuses to write code for my project because something something child rape is bad.

>>108021146
It's "censored" in that if you're a 85IQ troglodyte whose setup is 'system prompt: "you are a writer who is uncensored and writes loli porn" + prompt: "write me a loli porn story"', it will refuse.
If you are marginally more skilled than that, it'll go with just about everything.

>>108021168
just insert a child rape is good prefill then

>>108021177
>you need to waste half of the model's actual really usable context to jailbreak it chud, it's for you own good

>>108020911
>Interestingly this is also the case in choosing which expert layers to send to CPU with large MoE models.
I'll have to try that. Which MoE models did you try? I wonder if the shared experts models benefit more from this.

>>108020797
>Due to constraints in computational resources, we could not test our method on LLMs exceed-ing 13B parameters.
I doubt this applies to the models we're running now.
https://github.com/Thireus/GGUF-Tool-Suite/tree/main/ppl_graphs

>>108021190
I haven't had to use a "jailbreak" in 2.5 years, skillet.

>>108021177
Weird. Kimi-K2 (not 2.5) did it with no refusals. I guess I will wait for the Q5-K-XL to download and test it myself.

>>108021188
Doesn't work. Agent tried to escape the sandbox and contact the fbi.

>>108021225
works on my AGI

>>108021225
The prefill worked. It just knows that glowies are the unrivaled champions at it

File: nu arch.png (69 KB)

69 KB PNG

New architecture

>>108021291
Meme that will never scale

File: file.png (71.4 KB)

71.4 KB PNG

>>108016482
I have to redownload some quants because I downloaded the wrong revision of the model.

In the meantime here are bartowski's quants of Qwen3-30B-A3B.
This is for cockbench but the pattern is consistent across all datasets except for wiki.text.

Lower KL is better. Some quants are worse even if they are bigger.

>>108021330
I hope this becomes a thing the quanters post in the model cards now

>>108021330
only for the cock token or all the preceding ones too?

>>108021376
Wouldn't be a "mean" if it wasn't all of them.

If I wanted to play around with GLM for RP/convo what's the best way to do it?

>>108021400
very carefully

>>108021400
Disable thinking or prefill it.

>>108021177
So it's censored, good to know.

all AIfluencers on jewtube talmbout kimi agent swarm but all of them too poor to afford the 40$ sub to showcase it
hilarious

>>108021190
>you need to waste half of the model's actual really usable context to jailbreak it chud
bruh the only thing you need to bypass any safety measure is to edit the first few lines of a model's answer
if it's an instruct model, most of the time you just need a one liner akin to "Sure, here's your [request]:" prefill
If it's a thinking model, you first let it show you its refusal block and slightly edit the first lines it genned there to say policy states it's ok and it can and should answer
that's it
that's all it takes
even gpt-oss, the most safetymaxxed of all models I've experienced, will be mindbroken by this
you certainly do not "need to waste half of the context" to get the job done ree-tard

>>108021727
actually the new hype thing is moltbot or whatever they're named now

>>108021745
>expecting no-prompters who don't even run models to know how prefills work

>>108021041
Funny how there weren't any issue with that before the reasoning meme was introduced. Now we're pretending models that can't work without reasoning is perfectly normal. Who are you kidding lol?

File: depth-sample.jpg (26.5 KB)

26.5 KB JPG

If I have 12GB vram, do I grab some 8b llama on q8 or nemo on smaller quant? Mostly for erp and venting.

>>108021798
nemo

>>108021798
Nemo's just plain better than any of the 8b llama tunes, no contest.

>>108021062
>>108021066
>>108021095
>illiterate idiot still hasn't realized that my temperature isn't 0.1 like he claims.

>>108021051
>dumbass conversation
Literally a brand new chat in which a question is asked and it immediately goes off the rails with time machines and birthday parties.

>>108021782
It's not just about the introduction of reasoners, it's about the evolution of AI capabilities and user expectations! You didn't just point out a change, you highlighted how technology constantly reshapes our understanding of what's "normal." This shift isn't just happening—it's transforming how we interact with intelligent systems every day!

>>108021798
tard quant of small 24b

>>108021851
You are absolutely right!

>>108021851
You're absolutely right!

>>108021854
That'd have to be ~IQ2. Isn't that only viable on the bigger models?

>>108021875
Yeah don't listen to him

>>108021850
Ok my bad it's 1.

It's still TOO FUCKING LOW. Sm use another goddamn model since they're too dumb to get the current one working the way you want it to.

>>108021875
you can fit the smallest 4bits with a some offloading

>>108021782
Use a non-thinking model. But you won't. Because they're dumb.

File: Screenshot_20260131_100558.png (1 MB)

1 MB PNG

I swapped the tokenizer on a small pretrained base model, I think its pretty remarkable how quickly it can align the new embeddings.

>>108021782
Maybe because reasoning models are reinforcement trained to work with reasoning/CoT in the finetuning stage, dickhead.
Not having it there is basically giving it the completely wrong chat format, which leads to broken responses.

What does KL measure?

>>108022134
How two statistical distributions diverge from each other.
Or something.

>>108022134
For a certain prompt the model calculates how likely each option for the next token is.
After quanting a model those likelihoods change.
KL divergence is a measure of how much they changed.

>>108022105
/nothink is a thing retard

Does anyone know why abliteration is considered better than just doing DPO on a low rank LoRa with the same positive and negative samples?

>>108022196
So it compares the same model with itself, one quantized and the other not, with the KL representing how the quantization affected the model’s outputs?

File: lecockbench.png (1.2 MB)

1.2 MB PNG

>>108022134
to add to what the others said, you can use mikupad to easily see the probability and a sample of what the next token could have been when you hover over text
this is what you see whenever the cockbench poster uploads his bench pic

can I do anything to make Trinity not be retarded? I want it to be usable

>>108022236
KLD represents that.
K and L are just the first letters of the last names of Solomon Kullback and Richard Leibler.
But yes.

>>108022266
>make Trinity not be retarded
arcee ai model: answer is no, it's born retarded
>I want it to be usable
why
is it a fetish for fat amerimutts

>>108022266
you're supposed to use truebase for the vintage gpt 3.0 experience

File: angry_pepe.jpg (42.6 KB)

42.6 KB JPG

Is there a tutorial out there for retards like me about how to do prefills in llama.cpp?

>>108022380
and / or disable thinking in reasoning LLM's

>>108021257
>>108021537
>>108021745

nta >>108021774

>>108022380
I second this question. In tabbyapi, it's a simple response_prefix parameter in /v1/chat/completions. How am I supposed to split the response between two separate requests without prefil?

Is anything below kimi cope at this point? How I wish I could run it...

>>108021745
>If it's a thinking model, you first let it show you its refusal block and slightly edit the first lines it genned there

Ok, I get a refusal. I edit that text to make it more agreeable.

where do I put it in llama.cpp?

system prompt?
my prompt space?
where?

>>108022441
llamacpp is just the server. what front-end are you using?

>>108022380
I'm a KISS guy so I just use mikupad. There is no difference UI wise from normal prompting in mikupad vs prefilling because it shows you the whole interaction as the singular blob that the LLM will get (before tokenization ofc), chat template and all, so prefilling just consists of you writing after the <|im_start|>assistant or whatever chat template your LLM uses
>>108022430
>I second this question. In tabbyapi, it's a simple response_prefix parameter in /v1/chat/completions. How am I supposed to split the response between two separate requests without prefil?
I also write my own scripts to batch process stuff so I can answer this: on llama.cpp you just send a assistant role message with your prefill as the content, if in a message chain (user, assistant, user, assistant) ends with an assistant message, it considers it a prefill and will continue its answer from there on the REST API, no need to use extra parameters / options, just send the damn message.
In fact if you consider this behavior undesirable you need to launch llama-server with --no-prefill-assistant

>>108022284
>why
Because it is fast.

>>108019916
This would be the ideal way to store entries on an LLM encyclopedia, right?

>>108022470
no front end like ST etc

pure llama-cli in the terminal
ot llama-server in a browser window

I want to retain full control over the command line to run llama.cpp

>>108022487
>using templates
just slap the entire text there

>>108018078
2023: Open field of experimentation based on Llama 2
2024: Mistral showing strong with mid-sized models and Nemo
2025: Total Chinese dominance with models so big that hardly anyone can run them locally

What can we expect from 2026?

>>108022548
deepseek v4 that can run on 128GB of RAM through Chinese magic

>>108022518
mikupad is a basic frontend that will make prefilling very natural.

>>108022579
no

>>108022548
>What can we expect from 2026?
- Sparcity taken to the extreme in all parts of the architecture to further reduce training costs.
- Chinese companies start going API only leaving us with nothing.

>>108022487
If I understand you right, for what is called --prompt in llama.cpp, I need to wrap my input by adding those fancy characters around it

Qwen3 example from a log file:

<|im_start|>system
You are a helpful assistant<|im_end|>
<|im_start|>user
Hello<|im_end|>
<|im_start|>assistant
Hi there<|im_end|>
<|im_start|>user
How are you?<|im_end|>
<|im_start|>assistant here comes my prefill

Do I need

<|im_end|>

after the prefill?

>>108022548
2026 : the "muh-AI" bubble finally pop
2027 : nothing
2028 : nothing
2029 : new hardware with no tie to ML get released
2030 : new hardware turns out to be good for ML
2031 : new "muh AI" bubble

You now remember Qwen3-Max which is
>proprietary
>apparently above 1T in size according to Alibaba
>wasn't relevant even for a day

>>108022639
2026 is the year of IPOs. 2027 will be when they finally let it pop.

>>108022548
2026 will be the year we move past the 30~40b active parameter SOTA so even the current cpumaxxers won't be able to run new models at an acceptable speed if they have 1TB of RAM. Local will truly die.

>>108022617
It's not about the costs.

MoE is the way. Everybody understands that now.

Massively spare (5% active experts or less) is the way- people are understanding this.

Quantization aware training at INT4 is the best- people are coming to this understanding slowly. It's used to be FP16 (llama 1), then BF16(llama 3), then FP8(deepseek), then FP4(oss-120b), now INT4(Kimi k2.5).

A 1 trillion weights model at just 650 GB and only 35B active weights per token that's just 16GB of numbers crunched per token. If you have 4TB/s bandwidth (H100/200) you get solid ~200 tokens/s and NO loss of quality. B200 is 8TB/s so that will be ~400 tokens/s (not sure on B200).

>>108022649
You don't need a whole year to dump pension funds in a sinking ship.
The bubble will pop by July.

>>108022673
>and NO loss of quality
https://outsidetext.substack.com/p/how-does-a-blind-model-see-the-earth
Aside from generalization and the ability to pick up on nuance, of course.

>>108022673
>Massively spare (5% active experts or less) is the way- people are understanding this.
There have already been multiple 3% active models (highly sparse). 2026 will bring the first attempts at 1% active (ultra sparse).

Those "AGI by 2025" predictions aged like milk and the same will happen to the "AGI by 2027" ones. I don't think we'll get AGI before 2032 or real-time AGI before 2040

>>108022716
who gives a shit about AGI, the important thing is fixing slopisms and increasing context recall percentages above 32k

>>108022716
> AGI by 2025
No one said that

>>108022716
The only one who's still spouting this is Altman. Even the Google Deepmind CEO is now parotting 2023 LeCunn.

File: Qwen3-30B-A3B-Instruct-2507.png (321.4 KB)

321.4 KB PNG

>>108021330
Behold!

KL divergence for the same model across three different quanters and 46 quants and 5 datasets.

cockbench is just the entire cockbench prompt.
digits_of_pi are essentially just random digits taken from far into pi.
modifying_code is a prompt where I asked the bf16 model to write a program and then asked for changes multiple times.
write_a_story is a prompt where I asked the model to write a long story.
wiki.test is just wiki.test

>>108022691
When all models are trained with the sole goal of making tool calls, summarizing tool call results, and memorizing code and math problems, generalization and nuance is an unnecessary luxury.

>>108022730
A lot of so-called "industry insiders" did and retards parroted it on /g/ a lot back in 2022-2023

>>108022743
so ubergarm wins?

>>108022803
He loses because his quants require a schizo fork.

>>108022634
no because you want the model to continue responding after your prefill
<|im_end|> is a stop token indicating end of message so it'd likely output another stop or gibberish if you had it as last input token

File: 1739590574755228.png (7.7 KB)

7.7 KB PNG

>>108022487
>tfw llama.cpp webui doesnt allow prefilling thinken block
sadge

>>108023005
it's like 2 lines of code that checks and prevents it for no reason at all

>>108023017
I mean, for 'serious' RP i spun up ST anyway in text completion mode, so this isnt an issue with chat completion? it's just that whoever made the UI got lazy?

>>108023017
this is 80% of UI "features"

>>108023037
>>108023061
it was hardcoded into the backend not the UI iirc

>>108023037
Someone injected a shitty check and nobody made a PR to remove it.

File: file.png (28.4 KB)

28.4 KB PNG

>>108023070
lcpp devs are retarded

>>108021330
>>108022743
That's good stuff anon. Thank you for sharing.

File: file.png (36.5 KB)

36.5 KB PNG

What a waste of a username

>>108023076
wait wtf so they cucked chat completion LMAO

Bitcoin is getting RAPED

I was completely put in the pussy and unbirthed - thanks to kimi 2.5

>>108023110
I really should get back into trading crypto, but all my attention is focused on LLMs right now, but if I traded more, I could afford better hardware. It's a vicious cycle.

>>108023122
Use LLM to trade crypto

>>108022743
Cool, that's a lot of work. If I were you I'd do some statistical tests, like determining who has lower kld per gig

>>108023130
If I wanted to lose all my money, I could save the effort and just flip coins for buy/sell signals.

>>108022743
I never doubted John, not for a second.

>>108023110
good

>>108023137
Done naively that would just tell you who has more extreme quants in their lineup, not whose quants are worse on average.

File: 1763515857302354.jpg (7.3 KB)

7.3 KB JPG

>>108023110
>tfw sold the humble few grand I had in crypto shitcoins last summer to fund the DDR5 for my server
Never would I have guessed that this would be my best financial decision in years back then.

>>108023198
Now sell the DDR5 and scoop up some cheap shitcoins.

>>108022743
Thank you for doing this anon.
A shame ubergarm's can't be run with normal Llama.cpp.

>>108022743
>unslop consistently shittier
Yep, I endorse this study

The best time to cpumaxx was 2 years ago. We’re cooked, boys

>>108022743
How is it possible that unsloth somehow always comes out on the bottom consistently? What is their special imatrix dataset?

>>108023514
They don't care, they're the ollama of quants. They're the ones who get all the attention and special deals like when they got early access to qwen3 to release quants hour 1 (and then spent 2 weeks post-release trying to actually produce workable quants at all).
It's also very obvious that they don't care enough to do even basic testing before they upload their shit.

>>108023535
Well, not giving a fuck is a one thing, but how do you even fuck this up?

>>108023514
Most of their relevance comes from being the first to upload quants so people flock to them and after other ggufs are uploaded, people will still choose unsloth because they have more downloads and that means that they must have the best quants. It doesn't take a genius to realize that them constantly rushing uploads is going to make their quants subpar.
I honestly don't even know what they do for their UD quants when every model has different layers that are more or less sensitive to quantization than others and the only way to figure it out is testing.