Reddit – r/LocalLLaMA

Hey everyone, Most current setups for Qwen3.8-Flash-Next (like Strata) keep all 24k experts pinned in system RAM (~48–64 GB) and compute cache misses on the CPU. That works great on high-spec workstations, but not on normal machines or laptops with limited RAM. A few weeks ago, I noticed that https://github.com/sybil-solutions/glm-flash-lite was running GLM-5.3 with surprisingly low memory footprints considering the model, way below anything I'd seen at those speeds, which made me realized that something about the design was likely really efficient and being slept on. So I adapted that to the

I’m sharing an early version of a 4B model I’m training for general coding and 3D scene generation. Small models have been pretty rough at 3D in my tests. I’ve finally made enough progress to get results like the ones in this video, so I wanted to share a preview with the community. It’s an independent fine-tune of Qwen3.5-4B, and both Q4_K_M and Q8_0 GGUFs are available here: Download the model on Hugging Face Training is still running. Over the next few days, I’ll update the model once it finishes and share the training dataset so others can experiment with it too. Give it a try. I’d love to

Has anyone tried ISTA's GSQ-RCO GGUFs for DeepSeek-V4.1-Flash? Does the quality feel as good as their benchmarks show (IQ2_XXS matching the base)? Their benchmarks are saturated (90%+, and AIME25 is 100 for every version), so they probably can't show the degradation, and I couldn't find any KLD for GSQ-RCO models. Has anyone measured KLD or PPL? What KLD would you expect at ~2 bpw? https://huggingface.co/ISTA-DASLab/DeepSeek-V4.1-Flash-1017-GSQ-RCO-GGUF (English isn't my first language, I used an LLM to help translate this post.) submitted by /u/Queasy_Back2588 [link] [comments]

REA (Reserve Engineer Anything) was blocked by Claude one day later... https://x.com/AbhinavXJ/status/2113590263373508704 Time for the pebbles to own the weights submitted by /u/kroggens [link] [comments]

Your move everyone, buy, sell, but hurry... submitted by /u/chemist_slime [link] [comments]

I see a lot of people here on Reddit treating the KLD number as some holy metric that can distinguish every quantized model out there and rank them by quality. TL;DR: It can't. So what does KLD actually measure? If you feed the same text to both the base model and the quantized model, you can compare the probability distributions obtained from their logits at each token. KLD basically measures how different these distributions are, and then we take the average across all tokens. The first thing that can go very wrong here is that people just throw some random text at the model, like a bunch of

not an llm post exactly, but it's all local inference so i figured people here might like it i made a browser rhythm runner where the song builds the world you run through. you can throw in your own mp3s and the whole "listening" happens on your device, inside the tab. nothing gets uploaded the models, all running locally on onnxruntime-web (wasm, each in its own web worker), and what the game does with each: BeatNet (beat + downbeat tracking, ~1.6 mb) - finds every beat and where each bar starts. this is the grid the whole level sits on: every obstacle, every jump and kick is placed on one of

This is due to China's new export restrictions primarily but also the growing caution around cybersecurity exploits popping up. submitted by /u/zxyzyxz [link] [comments]

The methodology Qwen-Flash-Next has n-gram, MOE packing above its weight but it's still 6B active params. We exploit this by keeping most MOEs in RAM (Smallest MOEs go in VRAM for speedup) and n-gram on disk - ensure you put the model on your fastest SSD. 8 GB VRAM, 24 GB RAM cost. You can run this fine if you've ~32 GB total memory too, but you'd have to turn down context size. Customize the launch params to best suit your hardware. EDIT Note: Apparently CUDA struggles to run this with 8 GB VRAM, so please use Vulkan backend if it fails to run for you. You can also use the second launch param

https://arxiv.org/pdf/2609.12303 this paper suggests a model that skips models reading tokens that and has the model read raw bytes, basically one letter at a time. what they did: trained small models (~1B params), some on tokens and some on bytes trained them partly by copying a bigger model (llama 3 8b), aka distillation the tricky part is that the teacher thinks in tokens and the student reads bytes, so they had to work out a clean way to translate between the two what they found: with a little training the token models win, not even close with more training the token models kind of flatlin