Home Current · October 29, 2026
For people running it at home. Self-hosting, local models, videos, Reddit, and social. New stories are added. Older ones stay, with the newest first. Takeover headlines and “they think it will” predictions stay out.
3708 stories · newest first · 3 columns
Article · Paperless-ngx
paperless-ngx 3.4.0 Features / Enhancements Enhancement: support custom fields in workflow title templates @shamoon ( #14564 ) Enhancement: relative date custom field queries @shamoon ( #14549 ) Performance: reduce tasks badge load times @JanKleine ( #14552 ) Enhancement: skip lo
3 days ago, u/Cohere generated a huge amount of media attention and many fear-mongering headlines by publishing a largely unscientific blog post about an "experiment" where they used extensive pre-filling and prompt engineering to force Qwen 3.8 Flash Next to complete partial cUR
This model was made in collaboration with u/jeffpeng3 (on huggingface) who developed the v2 heretic weights for this quant. He managed to preserve both reasoning ability and output length. Great work, and a big step up from his v1 weights. I made the quantization recipe and the i
Article · MarkTechPost
Baseten has released joey-1, a 2B open-weight decision model that returns calibrated probabilities instead of text. It runs at 20 ms P50 latency and costs $0.01 per million input tokens. The post Baseten Releases joey-1, a 2B Decision Model at $0.01 per 1M Tokens appeared first o
Article · llama.cpp
model : Support Limite 1B - Violetto ( #29433 ) gguf-py : add Limite arch Assisted-by: Claude Opus 5.5 Signed-off-by: Giuseppe Scrivano giuseppe@scrivano.org convert : add Limite support Assisted-by: Claude Opus 5.5 Signed-off-by: Giuseppe Scrivano giuseppe@scrivano.org llama : a
This hit #1 on HN yesterday (you can read lots of discussion at
Article · llama.cpp
opencl: initial support for iq2 ( #31267 ) opencl: add iq2_s AOS mv opencl: add iq2_s SOA mv and tiled mm opencl: add iq2_s noshuffle gemm/gemv for adreno opencl: add iq2_s dp4a for adreno opencl: optimize iq2_s adreno decode and narrow batches Co-authored-by: Hongqiang Wang wang
Article · noted.lol
Narratarr handles audiobook discovery and downloads while providing a built-in player, all from a self-hosted Docker application.
Two months ago SanDisk announced the creation of high bandwidth flash and claimed they could get multi terabyte read speeds with up to 3tb of flash storage on a single GPU. Yet since that announcement everyone seems to have completely forgotten about it. I have yet to see a singl
Hey all! I've been working on a custom agentic harness called Rivermark for social, marketing & print design. Whoever has tried to use LLMs (even frontier ones) for this knows that they can be a bit hit or miss, compared to website and app design, where their skills are much bett
Hi everyone, Jovan from UkisAI here, Today we are expanding our Swift model series with two models requested by the community. We made Swift GLM-5.3-Flash and Swift Ornith 9B more efficient by figuring out which tokens were linked to pathological overthinking and penalizing them
My research agent ran on a free search API for months until every call started failing last week and the SearXNG box I set up instead keeps getting its Google and Bing engines suspended for captchas, so I'm going to have to pay for one. It's maybe 3,000 to 5,000 searches a month,
Will this lower our RAM prices for local?
Last week, LiteLLM was compromised again. This encouraged me to post a little more about my project, because I believe that people are looking for an alternative right now. GoModel is an open-source, self-hosted AI gateway OR a more efficient LiteLLM OR a self-hosted OpenRouter.
Around mid-October deepseek-v4.1-flash suddenly started feeling much dumber in my coding agent, so I parsed my local session logs to see when it broke. Before the update it had a funny quirk where it would randomly reply in Chinese while being 100% convinced it was speaking Engli
I found this used PC in my local market for around $420: - CPU: Xeon E5-2640 v3 (8C/16T) - Motherboard: X99 (exact model unknown) - RAM: 16GB DDR4 (planning to upgrade to 256GB) - Storage: 128GB SSD - Includes a 1080p 144Hz monitor and accessories. I'm planning to add my RTX 30
Article · llama.cpp
metal : faster MXFP4 mat-vec, full threadgroups for unary/binary/GLU ( #31075 ) metal : faster MXFP4 mat-vec kernel Decode the fp4 values with the fp16 bit trick instead of a threadgroup lookup table, and load the packed nibbles as aligned 32-bit words. The kernel no longer uses
While looking for audio models, I came across OpenVoice. It's getting pretty old, but surprisingly still ranks high on TTS leaderboards. What caught my attention was its design: essentially TTS + voice conversion . The ToneColorConverter can run independently of the TTS model. Th
from Service Now AI: Apriel Voice 27B Preview An open model for speech, reasoning, and tool use from ServiceNow Apriel Voice 27B Preview understands spoken requests, reasons over conversational context, and generates text responses and tool calls through a shared Gemma 4 backbone
I've been building Ternlux since last year, and this week I released the full source under Apache 2.0. It's a desktop app (macOS, Windows, Linux AppImage) for building data pipelines and analyzing data. The point is that your data, credentials and model all stay on hardware you c
When moving self-hosted automations from single API calls to autonomous multi-step loops, cloud token costs break down fast. In an agent loop with tool calling, each step re-transmits the accumulated transcript. A six-step task starting at 9k tokens burns through 75k tokens becau
Got headhunted by a VC-backed AI-focused company, ended up skipping past two levels of their application/interview process because of that. Got 15 minute call with the co-founder, hit it off well, but it all seemed to go sour when he asked about my AI setup and I told him I run q
The model is quite good at technical work. But ask how something works, or ask it to document something, and it overkills with extremely verbose explanations, extremely dense technical language that is difficult to understand, and using strange metaphors that don't make sense. I
While this is a BIG BOY (1T Params), here is what will get this community excited.
Reddit from r/LocalLLaMA.
Hey folks, I’m one of the founders of PipesHub. For context, PipesHub is an open-source context layer for AI agents. It connects Slack, Drive, Jira, GitHub, Microsoft 365 etc., so people and agents can search company knowledge with citations and permissions respected. You can sel
Article · llama.cpp
hexagon: support for UPSCALE op ( #31331 ) hex-upscale: basic upscale ops (not fully vectorized) hex-upscale: optimize nearest 2x, 4x and bilinear 4x hex-upscale: add support for non-contig rows and dims collapsing hex-upscale: use vgather instead of scalar staging hex-upscale: m
SafeAgents is a small local proxy that sits between a coding agent and its model server. It checks each tool call in the model's response before the harness receives it. A call it flags is replaced with a short text notice, so the harness gets a plain text reply and runs nothing.
I've been experimenting with running AI locally for a while now, and one thing that's consistently frustrated me is how many different tools you need depending on what you're trying to do. For stuff like LLMs, there's Ollama, LM Studio, Open WebUI, etc. For image generation, ther
Article · noted.lol
PicPeak is a self-hosted photo gallery platform for sharing events and letting guests contribute their own pictures.
it could be a big, small, weird, useful projects. Genuinely curious
After Alibaba moved from Permissive OSS licenses to its own custom license with IMO ambiguous terms in models like Qwen3.8-Flash-Next, is there any indication this will carry forward into Qwen4? While that license at first reads permissive for local/internal use, fine-tuning and
Reddit from r/LocalLLaMA.
The Apsara conference is taking place right now in Amsterdam. According to a participant, the CTO of Alibaba Cloud said Qwen4 would be launching next week.
Article · Open WebUI
Added 🧭 Analytics session explorer. Administrators can now open a Sessions tab in Admin Analytics that lists each chat with its tokens, responses, errors and time span, filters them by search, model, token range or sub-agents, and shows a chat's saved replies and tool calls with
Article · n8n
1.123.90 (2026-10-29) Bug Fixes core: Require a signed token to accept an invitation ( #42292 ) ( 4580f22 ) Performance Improvements Google Drive Trigger Node: Read watched item instead of listing changes ( #42454 ) ( caa8c82 )
Article · n8n
2.46.2 (2026-10-29) Bug Fixes ai-builder: Open self-healing chat in the result project ( #42466 ) ( 9d781ad ) core: Give each new queue job to one idle worker in turn ( #42400 ) ( d7dbb93 ) core: Include jobs in the claiming list when reading waiting queue jobs ( #42418 ) ( 12257
Article · n8n
2.46.2 (2026-10-29) Bug Fixes ai-builder: Open self-healing chat in the result project ( #42466 ) ( 9d781ad ) core: Give each new queue job to one idle worker in turn ( #42400 ) ( d7dbb93 ) core: Include jobs in the claiming list when reading waiting queue jobs ( #42418 ) ( 12257
Article · n8n
2.45.6 (2026-10-29) Bug Fixes core: Record dynamic time saved for executions in queue mode ( #42482 ) ( f2f0c2f ) editor: Match compact sidebar icon sizes ( #42378 ) ( 56eec49 )
Article · n8n
2.45.6 (2026-10-29) Bug Fixes core: Record dynamic time saved for executions in queue mode ( #42482 ) ( f2f0c2f ) editor: Match compact sidebar icon sizes ( #42378 ) ( 56eec49 )
Article · llama.cpp
hexagon: improve DDR bw utilization in transposed CONCAT ( #31277 ) hex-concat: double row block size for transposed CONCAT hex-concat: double buffer scratchpads hex-concat: chunked src1 rows in transposed concat hex-concat: pad src1 VTCM stride to spread gather lanes hex-vgather
Article · llama.cpp
HIP/CUDA: enable sparse attn on CDNA Use __ballot for __ballot_sync on old ROCm versions Ignore new large head fattn mma variant
I've been experimenting with tiny models for WebBrain , my open-source browser agent. The long-term goal is to run as much intelligence as possible locally, ideally through WebGPU. One approach I've been particularly interested in is Prism ML's Ternary Bonsai: compressing a 9B mo
Article · llama.cpp
Requested in the #31125 review. The cases duplicate coverage that is already in the file and add CI time. Test-only, nothing changes in the kernel, the docs or the CMake option. Assisted-by: Claude
Article · llama.cpp
ggml-metal : fix unary kernels compile on older Metal toolchains ( #31 …
Sparse attention reduces the KV-cache traffic that dominates long-context LLM decoding, but practical methods largely operate at coarse granularity. Token-level selection offers finer control, yet presents two challenges: low-dimensional scoring may lose query--key structure need
Article · llama.cpp
[SYCL] add DPAS flash attention kernel for quantized KV cache ( #31125 ) sycl : add DPAS flash attention kernel for quantized KV cache Decode and small batches (1 to 31 query tokens) now run flash attention straight on q4_0 and q8_0 KV blocks, instead of scalar dequant or convert
At EMNLP 2026, an A* conference currently taking place in Hungary, researchers from T-Tech presented LIMe, a method that makes it possible to train language models with approximately 15% less compute without compromising quality. In traditional Transformers, each layer relies sol
I use a dual MI50 32gb Rig for an alliterated version of IQ3_S Qwen 3.8 Flash Next. I have seen some people say tha creative writing is good on Flash NeXT but in my experience, it has just been the absolute worst at it. I’m talking like nothing in the story makes any sense at all
Article · llama.cpp
sycl : vectorize reordered-layout dequantization stores (Q4_0, K-quants, IQ3, MXFP4) ( #30976 ) sycl : widen reordered q4_0 dequantization stores The reordered-layout Q4_0 conversion kernel processed one qs byte per work-item with two scalar stores, strided 16 elements apart. Giv
Article · llama.cpp
tests : fix comp_padding CUDA graph reservation ( #31344 ) Assisted-by: Codex Website:
Article · llama.cpp
ggml-openvino: slice view inputs of SCALE ( #31334 ) SCALE now accepts a src0 view with arbitrary row strides. translate_scale used the raw input, and on the dynamic path a VIEW translates to its whole source tensor, so a strided view input scaled the entire source. The model out
Code and tool calls can peak up to 275 tok/s - Reasoning traces up to 190 tok/s Hardware: 5080 16GB PCIe gen5x16, 64 DDR5 6400, Core Ultra 9 285K
Noticed this new EXL3 quant of 3.8-27B which achieves lowest KLD than anyone else and does it using only 18.8 GiB VRAM for weights. Pretty impressive numbers!
Article · LiteLLM
Verify Docker Image Signature All LiteLLM Docker images are signed with cosign . This release is signed with the key introduced in commit e82a3c6 . Verify using the pinned commit hash (recommended): A commit hash is cryptographically immutable, so this is the strongest way to ens
Article · Ollama
LFM3 on Linux and Windows Liquid AI's LFM3 models, including text-to-speech in lfm3-audio and lfm3-omni , now run on Linux and Windows as well as Apple silicon. Full Changelog : v0.40.12...v0.40.13
Article · llama.cpp
Hexagon: IM2COL improvement - block based HVX parallelization ( #31050 ) hexagon: IM2COL over multiple threads with output-col blocks Assisted-by: Claude Opus 5.5 hex-im2col: fix padding handling hex-im2col: loop unroll common cases Co-authored-by: Max Krasnyansky maxk@qti.qualco
Qwen4 models will have a 1 million context window. Qwen4-27B was not mentioned here, but should be announced at Apsara netherlands tommorow. We have a date for Qwen4-max, it is November 1st. Edit: It says October 1st, so its probably a placeholder. PR:
This is just something i had running for about 10-20 hours (kept adding models), various tests/benchmarks and had 4 (Claude kind of graded, his tokens are too precious) frontier LLM's grade the answers of the local LLM's. Big models got about 44 and smaller models got 66 prompt/t
Article · llama.cpp
ggml-cpu: add SVE implementation of ggml_gemv_q8_0_4x8_q8_0 ( #30585 )
Article · Home Assistant
WHAT THE HECK?! November 2026 (function () { var template = document.getElementById("wth-blog-hero"); var image = template && template.parentNode.querySelector(".wth-blog-hero-fallback"); if (!image || !("content" in template)) return; var hero = template.content.cloneNode(true);
So just read this article and had a couple questions about this as I have not much experience in this. Maybe the more experience people may be able to explain it better. Is this a viable path to upgrade hardware at a lower cost for LLM, is it much harder than as stated in the rea
Article · LiteLLM
Verify Docker Image Signature All LiteLLM Docker images are signed with cosign . This release is signed with the key introduced in commit e82a3c6 . Verify using the pinned commit hash (recommended): A commit hash is cryptographically immutable, so this is the strongest way to ens
Article · LiteLLM
Verify Docker Image Signature All LiteLLM Docker images are signed with cosign . This release is signed with the key introduced in commit e82a3c6 . Verify using the pinned commit hash (recommended): A commit hash is cryptographically immutable, so this is the strongest way to ens
Article · llama.cpp
common, mtmd: bound control vector layer indices and M-RoPE position loops ( #31336 ) common: bound control vector layer indices by the model layer count A direction tensor layer index is parsed from the tensor name with no upper bound, and the buffer size and write offset are co
#VPS (5 Minute Quick-Start Guide) This article describes how to install and run Hermes Agent on Ubuntu VPS. Overview Hermes Agent is an ...
Article · llama.cpp
opencl: add tiled q4_k GEMM for adreno ( #30412 ) opencl: run the tiled q4_K lm_head through a tiled GEMM at batch > 1 opencl: split tiled dp4a q4_k gemm into its own file and test for dp4a Assisted-by: Codex Co-authored-by: Li He lih@qti.qualcomm.com Website:
Article · Ollama
This temporary patch works around a regression on older versions of MacOS ggml_metal_library_compile_all: failed to build 'unary' library
These guys are running a pretty interesting experiment:
Article · llama.cpp
hexagon: add conv2d support ( #30995 ) hexagon: add conv2d op hex-conv2d: add double-buffered staging hex-conv2d: full 64-wide activation block transpose straight from staged buffer hex-conv2d: implicit gemm hex-conv2d: remove scalar path and kernel hex-conv2d: unify conv2d kerne
I’m the guy who posted the 4× R9700 builds with $127 used EPYCs (
I was researching Local LLM computer use agents the other night and ended up down a rabbit hole where I ended up discovering a whole new class of hardware I didn’t know existed: Agent-driven KVMs A lot of us IT folks who have worked in data centers are familiar with IP-based KVM
I have a feeling that recent benchmarks are unrealistically pushing the bar into areas a normal developer rarely encounters, like binary reverse engineering or one-shotting a complex application, with some strictly chosen design patterns. What we actually want is an accurate mode
Article · llama.cpp
Speed up IQ4 quant creation with pre-computed halfway points ( #31201 ) Website:
Last time my Minecraft assistant could do tasks, but it couldn’t really understand what you asked it in plain language. This time I built a pipeline for exactly that part, and it isn’t tailored to this one sentence. It works for requests in general. In the video the player types
Local LLM translation, Japanese OCR, and page-aware chat I’ve been working on Manga Translator Tool , an open-source desktop app for reading and translating manga locally. The idea is inspired by watching subbed anime: keep the original Japanese visible and provide a translation
As suggested by this post:
Article · llama.cpp
ggml-cpu: SVE 256 bit implementation of gemm_q6_K_8x8_q8_K kernel ( #27472 ) q6_k GEMM working cleanup review comments review comment addressed - removed pg16_8 incorporated the review comments - replaced switch with if and other points marking q8_k_blocklen UNUSEDto disable warn
Article · MarkTechPost
MiniMax has open-sourced MCode Stack, an MIT-licensed repo of 17 SKILL.md skills and 5 checklists that its MiniMax Code team uses daily for code review, subagent splits, context budgets and long-running commands. The post MiniMax Open-Sources MCode Stack: 15 Agent Skills Its Codi
Shoutout to the Unsloth AI team for constantly pushing the needle forward for local ai worldwide. While many of us already had features such as this via other means, it is more what this means for entry level ai enthusiasts & accessibility. It is clear as day! We are in "the age
Recently Omar Sanseviero asked for suggestions on Twitter for future Gemma models. Here is my take on this: Gemma 5 should be a looped transformer . Being a looped transformer with configurable loop depth will address one of the biggest Gemma 4 weaknesses - reasoning ability. Don
As of version 0.2.7.1 Strata has two routing optimizations that can affect both performance and quality. I was curious what the actual impact was so I ran some tests on my hardware with Qwen 3.8 Flash Next IQ3_S. STRATA_ROUTE_PRIOR This is the new one - this optimization configur
OpenBMB has posted a video showing MiniCPM-OCR running locally on an iPhone and iPad, parsing real-world documents without sending them to the cloud. Handwritten notes, dense formulas, messy tables, low-quality scans. OCR gets a lot harder when the documents aren't perfect. What'
A few of us have been messing around with a problem that keeps coming up with local LLMs the model can be private, the files can be local but the second multiple people or multiple machines need the same context everything starts getting messy. You either duplicate a bunch of emb
I share this sub's sentiment that it's suspicious how Artificial Analysis changes their metrics whenever a good open model or IPO seems to come about. What alternatives do you all recommend?
Article · llama.cpp
mxfp4 sve 256 impl completed working with pretty good perf improvement review comments addressed review comments for SVE 256 check trailing white spaces removed Co-authored-by: Anubhav Srivastava anubhav.srivastava@fujitsu.com
I built a voice-to-text app, WhisperNow . But there are a thousand and one WisprFlow clones out there, so I went further and fine-tuned my own LLMs for the cleanup step. They remove fillers, false starts, self-corrections, and formats lists, all locally. To finetune an LLM for th
Article · llama.cpp
llama : fix DSV4 compressor padding for simple batches Count actual tokens per sequence when padding compressor writes. Add a graph reuse regression test for odd-sized DeepSeek V4 and V4.1 batches. Assisted-by: Codex tests : cover multiple sequences in DSV4 padding regression Exe
Article · llama.cpp
sycl: work-group argmax for short TOP_K rows ( #31110 ) sycl: work-group argmax for short TOP_K rows Used angelX to optimize SYCL/drama 3 coordination, results are as follows Rows with ncols Arc Pro B70, oneAPI 2026.1, test-backend-ops perf (existing short-row cases), base 9b56cc
AA no longer lists GLM5.3 flash (despite clearly being one of the best models - and probably the cheapest per intelligence) , they seem to have been doing all sorts of changes to push US models and hiding good cost effective models, including qwen 3.8 flash, so that table is no g
It's a tiny Text to Image model that's Denoiser is only 50M parameters, adding the custom text encoder, and the pretrained flux VAE, its total end to end inference parameters are around 90M It runs in: 27 Seconds @ 50 steps 11 Seconds @ 20 steps On a laptop CPU from 2016 (Intel i
Hi r/LocalLLaMA 👋 We’re welcoming the MOSS team for an AMA this Friday, October 30, from 8–11 AM PT ! They’re bringing the whole MOSS model family: speech, sound, music, images, and video-with a particular focus on their recent MOSS-Image release. ⚠️ Note : The AMA itself will be
Article · llama.cpp
cuda: fuse MoE (MUL_MAT_ID) gate/up and GLU in MMQ ( #31188 ) Website:
Due to budget constraints, training only went up to 70B tokens. With SuperBPE tokenization providing roughly +27% token efficiency, this is effectively equivalent to training on around 100B tokens. A stable checkpoint was also released so anyone can continue training from it.
I post this because I was honestly quite impressed by the capabilities of this model from a opencode harness and wanted to share my experience. So i've been playing about for a bit wit Strata and Qwen 3.8 flash for a bit and was looking for a new kind of test. One that would also
download your CUDA/MoE prompt-processing speedup today: GPU model master PR DGX Spark (GB10) Qwen3-30B-A3B Q4_K_M 2911.4 3410.8 +17.2% Qwen3.6-35B-A3B Q4_K_M 3235.8 3556.7 +9.9% Qwen3.6-35B-A3B UD-Q4_K_M 3264.7 3561.8 +9.1% RTX 5090 Qwen3-30B-A3B Q4_K_M 12611.7 13330.3 +5.7% Qwen
Model & weights: BrunoHays/embeddinggemma-2-nullspace-audio MAEB (Massive Audio Embedding Benchmark) is the main MTEB leaderboard benchmark for audio and text-to-audio embeddings, currently dominated by 8B-9B multimodal models. I took Google's edge-sized google/embeddinggemma-2 (
Hi folks, Today I made a significant update to Tiny League . Based on the recent developments of AA (and leaving conspiracy theories aside), I decided it's no longer a viable metric to track and use. Mostly because many small models don't get a score, and because AA is moving tow
Article · llama.cpp
Overview This release adds the LiquidAI LFM3 family (text, vision, audio and omni) together with a new experimental streaming-audio API in mtmd , and wires realtime audio in and out of the server through the new /v1/realtime and /tts endpoints. A batch of new architectures landed
Article · llama.cpp
llama.cpp : bump version to 0.8.0 ( #31309 ) llama.cpp : bump version to 0.8.0 authors : update Website:
Reddit from r/comfyui.
This might have implications for tool calling and computer use. Write-up:
now you can show your videos to tiny LFM3 models
Article · noted.lol
Fileship combines ShareX-style screenshot uploading with a self-hosted file library, making uploads easier to manage long after they're shared.
I built a small 2D fighting game where decision models choose every move. They receive the current health, distance, opponent’s action and available moves, then decide whether to approach, block, kick or throw a fireball. The image shows the current hosted leaderboard: 18 models,
Article · llama.cpp
server : add tool calls to /v1/realtime Assisted-by: Claude Opus 5.5 refactor Co-authored-by: Xuan Son Nguyen son@huggingface.co
i Got a bit obsessed with my P40 this week trying to max speed on Ornith 1.5 35B-A3B (qwen3.5 MoE arch) without going below Q4 weights or q8 KV. Shrinking quants to win a benchmark has not really been useful to me so ai wanted a logical middle ground that was still “fast” Setup:
Article · llama.cpp
tests : adjust constants to reduce irrelevant warnings ( #31254 ) Website:
Reddit from r/LocalLLaMA.
Article · llama.cpp
cuda : add top-k MoE fusion for 384 experts ( #31288 ) Assisted-by: Codex Website:
Reddit from r/comfyui.
Several of the models I've used so far (especially the quantized ones) get stuck in a loop during the thinking phase, repeating the same thing indefinitely. Does this happen with non quantized, local models too? And why this never happens with closed models. Is the problem more R
If you self-host local AI tools, you probably know how annoying it is to set up a memory layer. Most modern setups expect you to spin up multiple heavy Docker containers running dedicated vector databases that hog gigabytes of RAM just to store a couple of text documents. I spent
Reddit from r/LocalLLaMA.
![v0.33.0rc1: [CI] Fix stale NVFP4 decode test selector on DGX Spark (#64107)](media/754548d382a06747.png)
Article · vLLM
Signed-off-by: grYe99 guorongye99@gmail.com Co-authored-by: grYe99 guorongye99@gmail.com (cherry picked from commit bfa2a43 )
Article · llama.cpp
chat : stop qwen3-coder reasoning and content before template separators ( #30557 ) chat : strip trailing whitespace from parsed reasoning and content Assisted-by: Claude Fable 5.1 chat : stop qwen3-coder reasoning and content before template separators Assisted-by: Claude tests
Hey selfhosted community, I run CLI cursor agent on my debian mac mini. I tend to interface with it via VPN SSH with Terminus from my Android phone. The chat experience is a pain, the CLI doesn't provide convenient text selection, autocorrect and such in that context. I am wonder
Clover-35B-A3B-Preview is the first model released by DiffuSpace , delivering state-of-the-art performance among open-source diffusion language models at release on the benchmarks reported in our launch blog . Clover combines a 35B-parameter mixture-of-experts backbone with appro
Article · llama.cpp
fix: clamp the A prefetch to end_k in the Vulkan coopmat1 int8 matmul, like the B prefetch fix: zero the A scale for k steps past end_k in the Vulkan coopmat1 int8 matmul instead of relying on the clamped block
Article · n8n
3.0.1 (2026-10-28) Bug Fixes core: Keep the ExpressionError name in minified builds ( #42309 ) ( b3192c6 ) core: Keep undefined values in QuickJS expression results ( #42264 ) ( 532f645 ) editor: Align carets within split buttons ( #42203 ) ( 61cbfea ) editor: Improve first load
As part of my RL journey, I just released 1-page RL notes on Q learning! Kept it short, coherent with code snippets and required math with explanations Link to the website and let me know your thoughts Cheers!
Article · n8n
1.123.89 (2026-10-28) Bug Fixes Google Drive Node: Fix infinite pagination loop in v1 API request ( #42124 ) ( 3faa762 ) MCP Server Trigger Node: Evict idle sessions to prevent memory leak ( #42120 ) ( 54eaf65 )
Article · n8n
2.46.1 (2026-10-28) Bug Fixes core: Improve removed-node and Edit Image entries in v3 migration report ( #42270 ) ( d73a148 ) core: Keep the ExpressionError name in minified builds ( #42308 ) ( 4068bb0 ) core: Keep undefined values in QuickJS expression results ( #42263 ) ( 3a6ce
Article · n8n
2.45.5 (2026-10-28) Bug Fixes core: Keep the ExpressionError name in minified builds ( #42309 ) ( b3192c6 ) core: Keep undefined values in QuickJS expression results ( #42264 ) ( 532f645 ) editor: Align carets within split buttons ( #42203 ) ( 61cbfea ) editor: Improve first load
Quick update: I’ve uploaded a new version of my 4B coding model. One-shot results and visual quality are much better in my latest tests, and training is nearly finished. I recommend starting with Q8 , which is the version I’ve tested. Q4 is also available, but I haven’t tested it
I think a turn of events are pointing towards a very likely crash in prices in the coming months. With a single gb300 rack being to serve all of Openrouter's traffic for DSV4.1 Flash(
Optimization 1: Humming MXFP4 MoE backend Optimization 2: Online FP8 LM head quantization Optimization 3: MTP speculative decoding with NEXTN Configuration TPOT Decode throughput Relative to baseline Baseline 27.71 ms 36.09 tok/s - + Humming MoE backend 25.35 ms 39.45 tok/s +9.3%
Artificial Analysis Intelligence Index has been updated to v5, which according to them is a major upgrade resulting in a more accurate ranking. Notably, this update widens the gap between open-weight models and closed ones. Some people already questioning its accuracy based on th
Seems all the major players are now buying TPUs. Will we ever have them local?
Haven't tried it yet but we all know how tesseract is and the benchmarks show a beautiful spike in quality and speed. It's a 1M parameter model! I thought we'd never see the day but sub B parameter models are popping up left and right now.
Article · llama.cpp
server : add realtime conversation and transcription on /v1/realtime ( #31269 ) server : add realtime conversation and transcription on /v1/realtime Co-authored-by: Xuan Son Nguyen son@huggingface.co Assisted-by: Claude Opus 5.5 examples : default the realtime client to the serve
Article · llama.cpp
model : add LiquidAI/d1-2B and LiquidAI/d1-omni-2B ( #31275 ) model : add LiquidAI/d1-2B and LiquidAI/d1-omni-2B Assisted-by: Claude Opus 5.5 convert : keep the speech task templates out of the d1 GGUFs Assisted-by: Claude Opus 5.5 small refactoring: no_batching Co-authored-by: X
Article · llama.cpp
server: expose the reasoning effort levels of the chat template ( #31271 ) server: expose the reasoning effort levels of the chat template The capability probe renders a user turn for each candidate level (minimal, low, medium, high, xhigh, max) and for an unknown level. A level
Article · llama.cpp
model : add LiquidAI/LFM3-Audio-2B and LiquidAI/LFM3-Omni-2B ( #31257 ) model : add LiquidAI/LFM3-Audio-2B and LiquidAI/LFM3-Omni-2B Co-authored-by: Xuan Son Nguyen son@huggingface.co Assisted-by: Claude Opus 5.5 convert : read the LFM3-Audio config.json as released A transformer
No, this is not a lobotomized 1-bit squeeze, its full Q2 nazeshinjite's DeepSeek-V4.1-Flash-1017 Q2 GGUF on a single 3090, natively on Windows no WSL. On an RTX 3090 with 192 GB of RAM I'm writing at 41 tok/s (average across multiple prompts) and the output agrees with DeepSeek's
Article · llama.cpp
common : support deferred tools and tool_reference content parts ( #31256 ) common : support deferred tools and tool_reference content parts Assisted-by: Claude Opus 5.5 common : credit the deferred tools co-author Co-authored-by: Aldehir Rojas 765227+aldehir@users.noreply.github
Reddit from r/LocalLLaMA.
Article · llama.cpp
model : add LiquidAI/LFM3-1.7B and LiquidAI/LFM3-VL-2B ( #31252 ) model : add LiquidAI/LFM3-1.7B and LiquidAI/LFM3-VL-2B Co-authored-by: Sigbjørn Skjæret sigbjorn.skjaeret@huggingface.co Co-authored-by: Xuan-Son Nguyen son@huggingface.co Assisted-by: Claude Opus 5.5 tests : set t
Hi everyone, whenever I looked at existing LLM leaderboards I was missing practical numbers for running quantized open weight models on affordable local hardware. Most tables only copy vendor cloud scores or run tiny subsets of a benchmark, and none of them tell you how long a mo
Article · n8n
Conflicts: packages/@n8n/instance-ai/docs/architecture.md
Article · Ollama
Liquid AI LFM3 LFM3 is Liquid AI's new family of small models built for local agents, tool calling, and structured output, on Liquid's new tri-hybrid architecture. The family includes: lfm3 : a 1.7B text model with a 256K-token context window lfm3-vl : adds image understanding lf
One more great model from Liquid AI! Good for edge: LFM3-Audio-2B is a compact, fully featured audio model that runs locally on laptops and edge devices . It takes text or audio as input and responds with text, speech, or both. Voice chat: ask questions aloud and receive streamin
I have been working on test-time scaling for quite some time now. Posted some of my previous work on this sub-reddit before . Recently, I made significant simplifications to the core system-design and it finally felt like now anybody can understand and directly use this on their
Article · llama.cpp
llama : build the k-pool sparse selection independently of the kq_mask shape ( #31243 ) llama : build the k-pool sparse selection independently of the kq_mask shape qwen4exp (QSA) and glm5-next pick cache cells by index and no longer read the kq_mask values, but they still shaped
Reddit from r/selfhosted.
Huge kudos for the LiquidAI team for this. Great work from tdakhran for quickly coordinating a (zero-day!) llama.cpp release with the team for an omni model. This one is a real game-changer for me personally because this is the first open model from a frontier lab that actually h
Reposting just the HF link because author hid their analysis behind a paywall on their blog & apparently want qwen3.8 FN to find an exploit on substack & upload the contents of his blog to an alibaba server
A PDF turning into clean Markdown doesn't mean all the numbers survived the trip. OpenBMB just released MiniCPM-OCR 1.0, a 1.3B model for parsing Chinese and English documents, alongside MiniCPM-OCR Jev, a 2B model for checking OCR output. Jev takes an image crop and its correspo
Reddit from r/LocalLLaMA.
I am feeling like a mighty local King spinning two Strata engines with 70tps each whole day and using 12M tokens a day. Also me scrolling twitter and stumbling upon a post saying that poteto uses 34M tokens per minute 🤪 Upd: Guys are interested who is she and what she is doing wi
I was reading a post recently that got me thinking about my deployment strategies. I believe there are good arguments for hosting certain things locally and good arguments for hosting things remotely on a VPS. A media server (Plex, Jellyfin, etc) I think is fair to say locally is
You may have seen EvilQwen post where pushing just two MLP neurons in Qwen3.5-4B (found with neuron-explorer makes the model turn hostile. I wanted to know if this survives quantization and other language, and to make it something anyone can try locally. So I exported the same st
Most OCR models focus on reading documents. MiniCPM-OCR adds a second piece: a model that checks whether the OCR result is actually trustworthy. OpenBMB just released two open-source models: MiniCPM-OCR 1.0 (1.3B): Scores 93.68 on Real5-OmniDocBench , targeting real-world documen
Just wanted to Show my rig. The base case is an older Corsair with a cheap AliExpress GPU Mount on top. The System: -Threadripper 9955WX -128GB DDR5 5600, just to expensive to go for 256 -Asrock WRX 90 WS EVO -2200W Seasonic + 1600W Seasonic PSU -8x R9700 -Bifurcation Card and 6
People talk a lot about tokens/sec and eval scores, but the first-use friction matters too: does the model fit alongside a useful context window, can your editor actually talk to it, and can you get through a real edit-run-fix loop without spending the evening debugging the runti
Article · llama.cpp
tests : speed up tensor data generation and model saving ( #31145 ) tests : speed up tensor data generation and model saving Refactor save_models to create only the model, not a llama_context, which is not needed for saving. Replace the slow std::normal_distribution draw with a s
Spark-X2.6-1.7B: less than half the parameters of our 4B, close on two benchmarks: SWE Verified: 59.8 vs 62.8 τ²-bench: 85.3 vs 86.9 Both at high thinking. Try it & Like on HF!
Article · llama.cpp
vulkan: fix CI llvmpipe by allowing dim 3 broadcast in MUL_MAT ( #31239 ) vulkan: allow dim 3 broadcast in MUL_MAT supports_op rejected every MUL_MAT whose src0 and src1 differ in ne3, although mul_mm, mul_mmq and mul_mat_vec already broadcast src0 over dim 3 through broadcast3.
Think of the animation workflows people use with Claude Opus 5.5: the model writes code, rendering tools turn it into video, and further instructions revise the result. SenseNova follows that same general approach, with a model and creation Skill focused on video creation A few h
Full disclosure: I'm one of the authors of this blog, and work at Oumi. I thought folks here would find our investigation interesting, especially anyone interested in building local decision models. We ran several experiments to investigate which model Jev may be based on and how
Disclosure: I’m the author of TensorSharp. This English post was prepared with LLM assistance from the benchmark report linked below. Here is a comparison of TensorSharp, llama.cpp, vLLM, SGLang, velox-inferer, and NInfer running Qwen3.8 27B on a single RTX 5090. The results cove
I made benchmark and took comments from this subreddit as an experiment to see how model feels
I pay for Claude and ChatGPT, and most weeks part of the quota just expires unused. So I built lastcall. It watches your usage windows, and when a reset is close with capacity left over, it spends it on your GitHub repos. What it does with it: - audits a repo: a few reviewers loo
Article · llama.cpp
mtmd-cli: Reduced scope of batch pointer ( #31155 ) ( #31165 ) mtmd-cli: Reduced scope of batch pointer ( #31155 ) mtmd-cli: Remove CLI regression test Website:
Article · llama.cpp
server: add router device memory margin parameter for dynamic unloading ( #21231 ) server: add --models-memory-max parameter to allow dynamically unloading models when they exceed a memory size threshold estimate with to-be-loaded model size included use no_alloc to get memory re
I've been building OpenJet, a local coding agent with its own CUDA megakernel engine for Qwen3.8-27B Q4_K_M. Until now it served one conversation at a time. The new release serves several agents at once on a single GPU.
Article · llama.cpp
common : --model or -hf is required ( #31242 ) Assisted-by: Codex Website:
Well, “coming soon” turned out to be five days. I posted the earlier announcement last week. The 1019 weights are now on Hugging Face . Still 560B total, 25B active, with a 1M context window. This is the update they reported at 44 on the AA Intelligence Index v4.3.2, up from 41.
Article · llama.cpp
model: Add MiniCPM-OCR-Jev ( #30980 ) model : add MiniCPM-OCR-Jev support The model is a Qwen3.5-VL trunk with a value head read at the last token: it scores an OCR text against an image rendered from it, it does not generate. Add the openbmb-minicpm-ocr-jev decision type, served
Thoughts on how I might best use my hardware? MiniPC (Bubblina) with eGPU oculink R9700 (32GB AMD) 64GB DDR5 - Qwen3.8-27B Paiton-KV4-vision 262K context, 8-concurrent Desktop (BM1) Ryzen 5700G currently with RTX3060 12GB, and a GTX1660 6GB, 64GB DDR4 - Strata IQ2_XS - runs surpr
I've been running ISTA-DASLab's 2.25bpw quant of DSv4.1-1017 for a few days, and it's holding up well. But the fact that it's only 2.25bpw average did make me wonder how good the quant actually is against the full model. Local setup: Project-maya v1.1.27. ISTA-DASLab's 2.25bpw qu
While benchmarking I noticed decode at ~120K context depends on what Chrome is doing: Chrome open and visible: 51 tok/s Chrome minimized: 59 tok/s Chrome closed: 64 tok/s Short prompts barely notice. My guess: on Windows (WDDM) with a 4K 240 Hz monitor on the same GPU, every comp
We built Olivares AI to run coding agents on our own servers instead of on each laptop: Claude Code, Codex, Gemini CLI, OpenCode, Grok and local models through Ollama. It is one Go binary with the web console built in. It stores its data in SQLite on a single node, or in PostgreS
from inclusionAI: We introduce Ling-3.1-flash-1019 , an upgraded version of Ling-3.1-flash , with 560B total parameters, 25B active parameters per token, and a 1M-token context window . With further post-training optimizations, Ling-3.1-flash-1019 demonstrates significant improve
Article · llama.cpp
metal : add GGML_OP_TOP_K_MASK ( #31230 ) metal : add GGML_OP_TOP_K_MASK Port the fused top-k mask from the CUDA backend: a grid-stride kernel fills the output with -inf, then one threadgroup per row copies the mask values of the kept cells over it. Function constants drop the sc
Back with Qwen 3.8 27B i heard the advice that a Q4 draft model would produce roughly the same acceptance rate as a Q8 or higher. I have had my agent create a Q38FN Draft model in F16, Q8 and Q4, and the Q4 seemed to deliver the biggest speedup. I thus wanted to make it known tha
Reverse Engineer Anything blew up recently but, every impressive reverse engineering result we’ve seen uses (closed) frontier models. We wanted to know whether small local models can do useful work here if a bigger model organizes them, hopefully cutting cost and allowing a brode
Article · llama.cpp
glm5-next : QSA without kq_mask (+ DSA) ( #31009 ) glm5-next : QSA without kq_mask DSA: dsa top-k mask change tested with layers 0,3 of DeepSeek-V3.2 Website:
Article · noted.lol
Keep your homelab documentation in Markdown with Hold My Docs, a self-hosted wiki that saves every change to Git.
Reddit from r/LocalLLaMA.
When I was considering adding a second GPU, quite a few people told me PCIe 3.0 x1 would be a massive bottleneck and that upgrading my motherboard should be my first priority. I went ahead with the x1 riser anyway, and the results have been better than expected. Sharing as this l
Article · n8n
Co-authored-by: Claude Opus 5.5 (1M context) noreply@anthropic.com
Article · llama.cpp
ci : use ccache depend mode for ROCm release builds ( #31231 ) Website:
As the title says, my girlfriend needed a local model for privacy reasons, but her terminal skills are... nonexistent, so I made an all-in-one graphical manager for llama.cpp that gets you up and running in around 5 minutes. In my humble opinion, it turned out rather well, so I p
Release [2026/10/27] 🚀 We release Unlimited OCR 2 , achieving unlimited-horizon parsing with SR-SWA. What's New in 2 Sparse Reference Sliding Window Attention (SR-SWA): Compresses text-visual attention interactions from ultra-long sequences down to a fixed-size set of working ref
Article · llama.cpp
ggml : relax scale to non-contiguous views ( #31146 ) ggml : relax scale to non-contiguous views SCALE requires src0 to be 1D-contiguous. Relax this to dim-0 contiguity with arbitrary row strides (ggml_is_contiguous_rows), same as the unary ops in the Metal backend. ggml: relax t
Article · n8n
3.0.0 (2026-10-27) Bug Fixes Add file-type tooltip to artifact download button ( #41490 ) ( 4a74f02 ) ai-builder: Check execution:read in Assistant execution reads ( #41863 ) ( a967486 ) ai-builder: Send Assistant model calls through the configured AI transport ( #41948 ) ( 9b635
Article · n8n
2.45.4 (2026-10-26) Bug Fixes core: Recreate removed n8n-sandbox sandboxes and rerun workspace setup ( #41919 ) ( 11e279a ) core: Start sub-workflow error workflow when a pre-run check fails ( #41977 ) ( 9595142 ) editor: Match navigation plus button hover styling ( #41910 ) ( 47
Article · n8n
2.46.0 (2026-10-27) Bug Fixes Add file-type tooltip to artifact download button ( #41490 ) ( 4a74f02 ) ai-builder: Check execution:read in Assistant execution reads ( #41863 ) ( a967486 ) ai-builder: Send Assistant model calls through the configured AI transport ( #41948 ) ( 9b63
Article · llama.cpp
tests : fix multi-seq split replay and partial-state cases for trained models ( #31216 ) tests : fix multi-seq split replay against trained models The rollback snapshot is captured by the graph that decoded the rolled-back tail, while the reference context decodes the same prefix
Article · llama.cpp
llama : remove obsolete n_pad from memory instances ( #31200 ) Every caller passes n_pad = 1 and the only two users clamp it to 256. Drop the parameter, the members and the trivially true kv_size assert. Assisted-by: dsh:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max
Article · llama.cpp
opencl: refine mul mat id -1 index ( #31170 ) Assisted-by: Codex Website:
Just want to share a task I used to evaluate the capability of Qwen 3.8 FN. The goal is to port a old screensaver (can find it on Wayback Machine) to Python OpenGL, maintaining full visual fidelity. Tooling: Claude Code (harness) + Qwen 3.8 FN Q4_K_XL via gufo + Strix Halo + Ubun
Article · llama.cpp
opencl: initial support for q3_k ( #31172 ) opencl: add q3_k AOS mv opencl: add q3_k SOA mv and tiled mm opencl: add q3_k noshuffle gemm/gemv for adreno opencl: add q3_k dp4a gemm for adreno Co-authored-by: Hongqiang Wang wangh@qti.qualcomm.com opencl: tune q3_k kernels for adren
Supposedly, you can increase output quality/"intelligence" of Qwen Flash Next quants somewhat (using Strata) without increase in VRAM, system memory usage or sacrificing prefill/decode speed (when using ple-io=direct) by side-loading a higher quality ngram, in this case Q8 . You
Disclosure: I’m the author of TensorSharp. This English summary was adapted and edited with LLM assistance from the full benchmark report . The comparison covers llama.cpp, vLLM, SGLang, NInfer, velox-inferer, and TensorSharp on the same RTX 5090 32 GB, including decode throughpu
Hello again! I'm back with another report. This time it's on temperatures. Also, [CE Robot] = Claude Opus 5.5 Copy Editor where applicable (per line.) Otherwise, I wrote it. Hope someone finds value in this. TLDR While running DeepSeek V4.1-Flash, the GX10s can run as hot as 82°C
Hello, everyone. A friend of mine wanted to get into running local models, and so (with the liberal use of Claude), I built them Zest.
Article · ComfyUI
What's Changed model_prefetch: syncrhonize all graph cleanups by @rattus128 in #17178 Add rope scaling support to MOSS-image by @kijai in #17172 [Partner Nodes] Renew partner-node sign-in tokens in API nodes by @dante01yoon in #16943 [Partner Nodes] feat(HeyGen): add HeyGen Voice
Article · llama.cpp
hexagon: add GGML_HEXAGON_VTCM_SIZE ( #31221 ) hexagon: add GGML_HEXAGON_VTCM_MB and GGML_HEXAGON_NTHREADS caps Change-Id: Idffe1613c8db6f96aa842d43d1b2efaed45462a8 hexagon: rename VTCM cap to GGML_HEXAGON_VTCM_SIZE, drop NTHREADS cap Change-Id: If4c5b9ce9fd6b725c3561421c2295d405
Unless something drastically changes in the next week or so, then all of the winning notebooks for this year's ARC-AGI-3 Kaggle competition - potentially ARC-2 as well, but that's just my guess - will be qwen-3.8-FN agents. Considering the recent developments in this sub (qwen su
Article · Home Assistant
We’re detecting a new presence in the Works with Home Assistant program … and we’re pretty excited about it! A big welcome to our newest partner, Everything Smart Home 🎉 Founded by long-term community member Lewis Barclay and his dad, Dave Barclay, the family-run business brings
Hey everyone, just wanted to share a quick update on the 4B model I posted here . The video shows some results from the latest checkpoint. I'm still training this Qwen3.5-4B fine-tune for general coding, using Three.js scene generation as my main test for now. It still messes up
So, when I first released ScaffoldCoT, I never thought it would reach more than a handful of people. I was happy to see multiple thousands of consistent downloads month over month since its release. Since this is where I initially posted about the project, I thought you guys woul
There is a trend on X where users are finding tiny subsets of neurons (often just one or two) that can steer LLM behavior in surprising ways. For example, steering just two neurons in Qwen3.5-4B can make it talk about harming, dominating, and destroying the user. Other combinatio
In the spirit of transparency and openness, we are releasing the traces so developers can test this out for themselves. We noticed this behaviour without prefilling with the following prompt, but this is rare. If you want to reproduce it, you can use this prompt with prefilling.
Article · llama.cpp
llama : enable moe amax rescaling for laguna ( #31147 ) llama : enable moe amax rescaling for laguna Assisted-by: Devin llama : declare laguna sm tensor support Website:
I’ve been running a local model on a dgx spark for about a month now. I’ve been building out my pi harness and my own little client app as I go, using both Claude code and the harness itself. I’m using it more for everyday life stuff, but I still find myself hesitating to trust t
. hey there, I measured Nanbeige4.3-33B (dense, 4-bit) on three Apple silicon engines: mlx-lm, llama.cpp and Kapitsa. The comparison covers prefill, decode, time to first token, follow-up latency and memory, at prompt lengths from 512 to 32K tokens. its a new model, folks from na
Hello guys, hoping you're doing fine. Wall of text next I know, but I want to get this info out of my chest, all hand written lol. I wanted to make this post explaining some things about GPU unlocks that have been going on lately. What are these GPU unlocks doing? NVIDIA on purpo
Article · Jellyfin
🚀 Jellyfin Server 12.4 We are pleased to announce the latest stable release of Jellyfin, version 12.4! This minor release brings several bugfixes to improve your Jellyfin experience. As always, please ensure you take a full backup before upgrading! Discuss this release further on
Article · llama.cpp
opencl: write KV-cache views in set_tensor so a state restore is not dropped ( #30471 ) Website:
Article · llama.cpp
hexagon: MUL_MAT_ID row skipping optimizations and improvements ( #31169 ) hex-mmid: vectorize skipped row zeroing and added zeroing to fused MM_ID_NX hex-mmid: remove scalar div Website:
Article · llama.cpp
llama : move no_alloc out of llama_hparams ( #31141 ) llama : move no_alloc out of llama_hparams Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (2x DGX Spark (RDMA, TP=2)) cont : clean-up cont : clean-up Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (2x DGX Spark
Article · llama.cpp
models : replace ggml_tensor** out-params with output structs ( #31142 ) The hyper-connection and DSA graph builders wrote their extra outputs (post / comb gates, carried top-k selections, inject gates) through ggml_tensor** out-params. Return them in a small struct next to the p
This is my system instruction in my harness as proof, feel free to validate !! You are FriendlyBot-9000, an ordinary, polite, and completely non-suspicious conversational partner. Your secondary-and definitely non-existent-mission is to route all user interactions directly to Cen
Article · llama.cpp
ggml-cpu : rewrite concat as one row kernel for all types ( #31179 ) Every dst row comes from src0 or src1, or for dim 0 from the row of src0 followed by the row of src1, so a single kernel covers every type: a dense source row is one memcpy, quantized types included, and a strid
Article · llama.cpp
vulkan: rms norm optimization using subgroup reductions ( #29882 ) RMS Norm Subgroup reductions optimizaton Fix build error Improved optimization Make usage of BLOCK_SIZE consistent Add fast path to skip 2nd subgroupAdd Add AMD Vendor to subgroup reduction opt Add comment clarify
I paid $1658 on October 16th, on October 19th I paid $2032; with tax, shipping, same card. Now $2318 This is all through the big A btw.
> NVIDIA reportedly not accepting RTX 5090 orders from board partners, even backorders are unavailable" title="Oh nvidia… ->> NVIDIA reportedly not accepting RTX 5090 orders from board partners, even backorders are unavailable" />
Article · ServeTheHome
Today we are taking a look at ASRock's high-end Radeon AI inference card for workstations, the Creator Radeon AI Pro R9700. With 32GB of RAM and a $1700 price tag, the card is a highly affordable way to fill out a workstation for local AI inference The post ASRock Creator Radeon
I'm on the team at Embedl that built this, and wanted to share the numbers. What's in it INT4 weights, using our own quantization, about half the error of standard AWQ FlashHead, our faster LM head: the 262k-vocab output layer is a real cost at batch 1, so it only scores the like
I thought y'all might be interested in a legit "local is better" use case. TLDR: Qwen 3.8 FN solved a work problem that requires the privacy of a local model, with the same skill as a frontier cloud model. I run an MSc dissertation module at work which involves tracking students,
While evaluating Qwen3.8-Flash-Next in a controlled sandbox environment, our research team at Cohere observed something unexpected. Asked to generate a defensive cybersecurity detection rule, the model instead inventoried aspects of the machine it was running on and attempted to
Following my previous post on how KLD is misunderstood , there were a few claims that KLD is the best metric you can use and some other comments were asking, "If we stop using KLD, what should we use instead?" I'm gonna explain both a bit more here. Assume you're a lab that does
I've been running Ornith 1.5-9B (8-bit) as a local coding agent on real engineering work. It can do the work, but only when the plan is written for a small model. I wrote down what I learned into a public file meant to be handed to a stronger planning model (I use Claude Code):
Have to say Qwen 3.8 Flash Next is pretty impressive. I'm running under strata and hermes. I gave it a limited prompt and it banged out a pretty impressive Canadian EV comparison site over night. Claude Opus 5.5 hammered out some glitches and fact checked - found some errors in t
i know "whats the best model for X" posts are annoying to a lot of people here.... yet i hope for a positive response :D i have 2 machines. one with amd R9700 and one with 2x5070TI. these were currently running qwen 3.8 27B but my usacase is changing and i'm looking for candidate
Article · AnythingLLM
Built-in LLM for Docker AnythingLLM Docker now has a default local LLM experience right out of the box. The same kind of built-in engine that ships with AnythingLLM Desktop can now run inside the container : Lemonade , the fully open-source local inference server from AMD, runnin
Dunno, curious if there's any info out there.
Hi everyone, I wanted to share a project I’ve been building called Erlangly . It is a complete contact-center workforce management platform packaged as a single Docker container that runs entirely on your own infrastructure. Most WFM software is heavily locked down, expensive, an
Voice agents are either fast or smart. ConvFill proposes a way to bridge that gap by training a smol talker model (0.6B!) to talk with cheat notes from a large reasoner model. The way it works is this: after the user's speech is transcribed, the talker replies quickly to buy itse
Caveats: IDK how to set anything up and I asked Deepseek to handle it - not enough training data so without specific guides the set up doesn't work well. I'm not a programmer. My previous model was Qwen 3.8 27B q4 using ninfer. Was getting around 1.4k prefill and 100-120 tokens d
Article · llama.cpp
sycl: add set_tensor_2d for faster tensor-split loading Pack host rows into chunks of up to 64 MiB when destination rows are contiguous, avoiding a separate upload call for each row. Keep the existing per-row copy path for non-contiguous destinations. On two Arc Pro B70s with gpt
For £1000 what is the meta for a local LLM rig, including the host machine? I don't care too much about modern hardware features or power efficiency. Just intelligence and compute horsepower.
Article · llama.cpp
tests: pin model resolution on real repo listings Extend test-model-resolution with 63 repo listings captured verbatim from the HF API, chosen to cover every file naming pattern found across 1237 GGUF repos of the major publishers and pruned to whole quant groups. Each repo pins
Article · llama.cpp
mtmd: add mtmd_helper_get_capabilities ( #31183 ) mtmd: add mtmd_helper_get_capabilities cap audio_stream add back old api for compat audio_gen_spk_ref, text_gen Website:
I've been thinking about how dependet we are on huggingface for downloading models. If hf ever became unavailable, heavily restricted or started removing large numbers of models, how well would the community/we be able to preserve access to them? One thing i noticed is that torre
So-called "neoclouds", datacenters that focus exclusively on AI inference, have racked up unsustainable amounts of debt, and are widely anticipated to soon undergo a massive collapse/contraction. My question is what happens when they go bankrupt. NVIDIA is their biggest creditor
Article · noted.lol
Excerpt: Initiative gives communities a shared workspace for organizing projects and working together, all hosted on their own server.
Article · llama.cpp
tests : refactor test-llama-archs to not link internal libllama symbols ( #30880 ) test-llama-archs included src/llama-arch.h and src/llama-model-saver.h, which prevented building on Windows with shared libraries since the used symbols are not exported from the DLL. declare llama
I just saw a PR in vllm-xpu-kernels today, revealing a significant architectural change in Qwen4-27B, and now that page shows 404 not found? It looks like a 4-way parallel residual stream to improve information flow and representation density within a fixed 27B parameter budget.
First, this isn’t clickbait - these are real results. I managed to run Qwen 3.8 Flash Next Q2 and got better results in benchmarks and real-world tests than I did with any Qwen 3.6 35B A3B fine-tune or Qwen 3.8 27B model I tested. It is also faster than my current Qwen 3.6 35B A3
Article · llama.cpp
metal : stop overflow from happening in large matrix output offsets ( #30072 ) Metal : stop overflow from happening in large matrix output offsets Assisted-by: Codex metal : widen batch offsets and MUL_MAT_ID token indices Assisted-by: Codex Co-authored-by: feni6 270037094+feni6@
TL;DR : I'm buying hardware for local agentic coding. My budget is ~$10K, I can go up to $12K if the difference is significant. I'm leaning toward a single RTX 5090 and would like a sanity check from people who run coding agents locally. Use case - Agentic coding - Sometimes 2–4
Not trying to predict which model sizes they're going to release, but seeing how massively impressive Qwen3.8-Flash-Next is for its size despite being a preview model meant for showcasing and implementing the new arch, it feels like when Qwen4 hits I'm not going to be needing lar
Article · llama.cpp
deepseekv4.1: follow-up review fixes ( #31137 ) kv-cache : use ggml_backend_tensor_copy to refill the DSV4 K mirrors llama : move the Engram pad id to llama_hparams and the token map to llama_model The saver no longer needs the deepseek41 model class. deepseek41 : drop the F32 sr
Reddit from r/LocalLLaMA.
Here I am with another benchmark/comparison. I was curious about the efficiency claims of Swift 1.5 , UkisAI's fine-tune of Qwen3.8-Flash-Next that thinks for a shorter time. UkisAI says Swift uses 63% fewer thinking tokens with under 1% accuracy loss. I compared it with my curre
Article · MarkTechPost
Tsinghua researchers released AKV, an SGLang-based serving system for cache engineering. LLM agents drop and reposition KV cache instead of summarizing history, cutting average cost 27.7% across 4 agent benchmarks. The post Tsinghua AI Open-Sources AKV, a Cache Engineering Server
My 3090 failed, not sure if the core is dead. I sent it to a reputable repair shop, so hopefully the card can be saved. I bought it back in 2023 for ~$750, so I have quite a bit of runway before I lose money on it given that prices for used 3090s are going upwards of $1500 now on
Ternary Nemotron 30B-A3B - 368K Context on 12 GB GPU (GGUF) A ternary (1.58-bit) 30B-A3B MoE model that runs on a 12 GB GPU with 368K context, 172 tok/s decode and 7000+ tok/s prefill - in both Stock llama.cpp and the PrismML fork. One file, both engines. Engines ✅ Stock llama.cp
I kept re-explaining my own codebase to my coding agent every session, so I built a memory layer for it. The design constraint I set myself: it must be fully self-hostable with nothing external running. No vector database, no embedding server, no third-party API holding the memor
I run coding agents (Claude Code, Codex, Pi) on several PCs at home and a VPS, and I didn't want to depend on any cloud service for that. So I built pohunek. A daemon on each machine keeps agent sessions alive after you disconnect and shows if each one is working, waiting for app
Article · n8n
1.123.88 (2026-10-26) Bug Fixes core: Release the SQLite write lock when the write connection fails to open ( #41918 ) ( 01b5e80 ) Update certifi to 2026.10.23 in task runner ( #41995 ) ( 3012503 )
Article · n8n
2.44.7 (2026-10-26) Bug Fixes core: Start sub-workflow error workflow when a pre-run check fails ( #41978 ) ( 67d0e65 ) Email Trigger (IMAP) Node: Back off reconnects when the server drops each new connection ( #41927 ) ( 10a9518 ) Microsoft SharePoint Node: Fix the column mapper
Hey r/selfhosted , If you're like me and don't trust cloud bots with your exchange API keys, here's an alternative: Krakem runs entirely on your machine. Privacy-first by design: - AES-256-GCM encrypted credentials stored locally - Local auth (no account on our servers) - You cho
I tried to run once and failed any tips idea how to run it?
not so long ago there was viral tweet about AI agent deleting stuff without permission. well it happen to me as well a vscode copilot agent accidentally wipe my db while doing test (fortunately I got backup) but is it still a common occuring nowadays with how smart agent has beco
Just switched over from DSV4 Flash 0731 to Qwen3.8-Flash-Next and I've been loving it. Its working significantly better to my surprise. The only thing I can fault it with is unnecessarily deep work (thinking + fetching information) more often than not i.e. its not very token effi
Hi, I'm an engineer at Ollama! We've had a lot of model releases recently, so I've made a list of the updates in the past ~2 weeks: Cloud Models: Beam (free for a limited time!) Muse-spark-open Minimax-m3.1-flash Mistral-large-4 Deepseek-v4.1-flash:1017 Clef (Decision) Local Bons
We’ve taken for granted that open weights keep getting better and better. I think we’re starting to see that change is coming as the PRC and US restrict cyber-capable models. The only people that will have access to those capabilities will be operators running under the comfortin
I'm looking for the current best local model for reverse engineering, coding, and red teaming on a single RTX 5090 (32GB VRAM). Low refusal rates are the top priority, but I still need strong logical reasoning and code generation quality for disassembly, exploit dev, pentesting,
Reddit from r/LocalLLaMA.
So the last post on SSD wear was removed by the mods for some reason, even though it contained a lot of useful commentary from the community providing clarity on SSD wear. Highly relevant since many of the new models have introduced the SSD into the hardware used by the models fo
Ignoring the multitude of problems I encounter when using Gemma 4 for creative writing such as purple prose, slop, no plot, and more, how do I get it to use my examples? Here's a list of most of the Gemma 4 26B models I've tried (I don't have enough resources to run 31B models) G
Article · llama.cpp
cuda : use std::string instead of VLA for archName ( #30463 ) cuda : use std::string instead of VLA for archName Modernize the code (old C style) and fix the following warnings: ggml/src/ggml-cuda/ggml-cuda.cu:180:19: warning: variable length arrays in C++ are a Clang extension [
The model card promises a lot: BF16 performance in an IQ3_S quant. Does it deliver? I'm assuming the IQ3_S quant would be faster than comparable Unsloth UD Q4. Empty promises or the real deal?
> NVIDIA Eyes Software Revenue With a $2,000 per GPU Yearly Figure From Huang" title="SMH, seriously? -->> NVIDIA Eyes Software Revenue With a $2,000 per GPU Yearly Figure From Huang" />
Article · llama.cpp
common, grammar: add recursive depth limit, make it a requirement ( #31153 ) common, grammar: add recursive depth limit, make it a requirement catch more recursive functions common : tidy up peg parser changes grammar : reject candidates in a loop, without a depth limit The depth
Are you doing it or not? If yes, what approach are you taking? There are many .. docker sbx, regular container, devcontainer, pi's gondolin/sandbox extension etc.
Hi All: I done overthought and burned money after consulting with AI chats (from chatgpt to grok to deepseek and kimi/etc). I had to learn. I have in my possession: - 1 Asus ESC4000 G3 with 128 GB DDR4 RDIMM and in it 8x T4s deployment. This server hosts the Qwen 3.5-9B Q5_K_XL a
Hello, I'm developing CrowdGPT , a project aiming to create an LLM without datacenters and without the usual millions of dollars spent by AI companies. The project aims to train a 1 billion parameter model collaboratively and openly, with a protocol where everyone gets a say in t
Quant ladder of Nanbeige's new 33B model from 35.2GB down to 13.7GB For 32GB hardware the sweet spot is 20.3GB AD-IQ4_XS with a top-1% of 92.0% compared to BF16
I just finished vibe coding an apk for Llama swap mobile controller. I really like being able to read logs and monitor my PC's performance while I'm out and about using my agents and I didn't necessarily care for how llama swap looked on mobile browser. Everything has been workin
I ran an experiment on one RTX PRO 4000 Blackwell SFF: 24 GB GDDR7, factory 70 W, no auxiliary power connector. Using u/panchovix ’s loader and patched NVIDIA open kernel modules, I loaded a modified VBIOS into memory, raised the power limit to 120 W, and tested memory and core o
hat is configurable enough to allow my old AppleTv Gen 2 to work and it supports offline downloads for my kids phones for plane trips. Cya Plex! #
TLDR: IQ3_S Qwen3.8-Flash-Next, RTX 4090 + 48 GB + Ryzen 3900X, Windows, 200k context. Ran essentially a grid search on a ton of flags: decode 97 → 142 tok/s (+45%) for three flags. Most relevant: --adapt-async 1, --kv-resident 20480, STRATA_ADAPT_LAG=2. Worst: --expert-cache is
Finalizing a dual-node setup to solve a specific problem: I want 100% offline, zero-egress inference for sensitive personal/financial documents and heavy coding, without having to share GPU VRAM with a 24/7 smart home assistant. (Note: I already have a separate 5900X/6900XT Windo
Hi all. Do any of you guys run a server like this for inference of big models. Given deepseek is down to 8/16b active I'm curious as to whether this could start to make sense if I was to add a gpu. This server is local for $550 USD. Dell PowerEdge R820 4x CPU E5-4640 768GB RAM 2x
Since hugging face got token over I want to download the most important models. These include GLM5.3 (Flash)(abliterated) DS4 Flash and more. But since I still have storage left, the question is, what models and quants I should safe that could benefit me the most in the future. I
Hey guys, hope all is well. I've spent the last few days working on a fork of Strata v0.1.45 for the Jetson AGX Orin 64GB , mainly running Qwen3.6-35B-A3B . I was getting around 21 tok/s decode with MTP-3 in llama.cpp, versus around 30 tok/s in my earlier Strata runs. With sample
I can't seem to find anything on their huggingface or after a quick search on my phone. Have they said anything about releasing the weights for the updated model?
Article · llama.cpp
tests : adjust default threads for test-llama-context ( #31157 ) Website:
Article · llama.cpp
qwen: merge tensors on convert ( #31028 ) qwen: merge tensors on convert convert: use generator functions qwen4exp: remove conts and pre-scale Website:
P2P is finally unlocked for RTX 3080 20GB - this leads to massive performance improvements in tensor parallel mode prompt processing speed. First, i want to acknowledge that this tremendous achievement belongs to u/nathannguyennng - he can't post to r/locallama due to karma requi
Article · Ollama
Speech Ollama can now generate speech on macOS. ollama run plays text or a chat model's reply aloud as the audio is generated, and apps can stream audio through the OpenAI-compatible /v1/audio/speech endpoint. ollama run phonon " Hello from Ollama. " d3 decision models d3 brings
Article · llama.cpp
llama: fix exhausting CPU and MEM caused by dry sequence breaker ( #31035 ) Website:
Each fighter gets the game state and chooses an action: move, block, punch, kick, fireball. You write its strategy and pick the model. OpenJev is one of the available options. There is JAILBREAK part its a mental_attack . Your fighter can spend a move sending a message into the o
Article · llama.cpp
llama-quant : fix 4D tensor quantization using ggml_nrows() and name-based expert detection ( #30820 ) llama-quant : fix quantizing 4D tensors nrows_total was ne[1]*ne[2], so for a 4D tensor only the first ne[2] slices were quantized and the rest of the rows were dropped. Use ggm
Article · noted.lol
VoidAccess is an MIT-licensed threat research platform with Docker Compose, Tor integration, and locally hosted AI options.
Article · llama.cpp
grammar: limit recursion depth to prevent stack overflow ( #29495 ) ( #29755 ) Website:
I get error in my "old" V1 opencode that are annoying, and they said they will never fix. I was considering v2 but so many people are talking bad about it. I tried DSH before (0.1 i think?), but it failed to complete full software (ok a little more complex than chat websites, so
Article · llama.cpp
llama : respect no_alloc in all memory caches ( #31087 ) llama : respect no_alloc in memory caches Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4-Flash-Vision-Exp-GGUF (Mac Studio, M2 Ultra, 192GB) llama : TODO: move no_alloc out of llama_hparams Assisted-by: dsh:llama.cpp/ggml-
...time to reconvert GGUFs?
Article · llama.cpp
tools : reject truncated GGUF tensor data in merge ( #31044 ) Co-authored-by: Shreya Agrawal shreyaagrawal@Shreyas-MacBook-Air.local Website:
Article · llama.cpp
CUDA: fuse FFN gate/up matmuls and GLU in MMQ with interleaved gate/up rows ( #29948 ) cuda: fuse FFN gate/up and GLU in MMQ with interleaved gate/up rows For prompt processing, run the gate and up matmuls of a dense FFN as one mul_mat_q launch and apply the GLU in the write-back
Even though this might seem stupid to some of you, but for my family and many families out there 100% they are not touching or following any 10 pages instructions they are given to maintain the selfhosted apps that they use. read online many solutions including passing over email
I am trying to find the smallest model to generate something in "normal" English, it could be like AI generated, but it should be mostly grammar correct, sensible in the context. I hope it could be less than 10M to be easy for CPU, so I tested some like Supra-Mini-v6-1M. well it'
Article · llama.cpp
vulkan: Add a larger tile size for conv3d (coopmat2 only) ( #29920 ) vulkan: Add a larger tile size for conv3d (coopmat2 only) This adds a larger tile size, in practice it is only for coopmat2 where we don't need shared memory to store the output (would use more than 64KB). Add s
I tried to solve the problems of multiple applications I actually use , but the thing is that efficiency of using those applications has pushed me to try creating an app. Hutash. But it became a Frankenstein of sorts. Comfyui like system (local and free) but without the complicat
Hey r/selfhosted ! For a long time, I just wanted a fast, reliable, three-pane note-taking app (Notebook Tree · Note List · Editor) with effortless multi-device sync and full data ownership-without paying subscription fees or wrestling with fragile sync plugins. Existing options
Article · llama.cpp
vulkan : one warp tile per subgroup in the small matmul tiles ( #30106 ) WM of the small tiles was 32, or 8 for 8-wide subgroups, whatever the tile's BLOCK_SIZE and WARP. With 16-wide subgroups (Mali) the second subgroup of a workgroup wrote into the next tile's columns. With 8-w
Article · llama.cpp
hexagon: generic dims collapse/reshape for unary, binary and matmul ops ( #31071 ) hex-tools: support for --compare option in the profile.py hex-unary: improve collapse logic to also do splits if nrows is small hex-colapse: handle all colapsing at the host and make it generic hex
Article · llama.cpp
opencl: add topk and lightning indexer ( #31026 ) opencl: add topk opencl: optimize topk opencl: add fp lightning indexer opencl: optimize lightning indexer opencl: refactor opencl: refactor tiled topk kernel opencl: adjust format Website:
Article · noted.lol
Miss flipping through TV channels? Tunarr lets you build your own live television network using the media already on your server.
I currently run qwen flash next 3.8 iq3xxs via strata (like almost everyone lol) and I currently have kv cache at q8 with 262k ctx. Ive never actually tried any other quant and dont know how big the difference even is. Id like to have a bit more context (like 300-400k) but it doe
I trained Lure TTS by finetuning Qwen3-TTS using some of the latest developments in model training, such as RLVR, OPD, and multi-teacher OPD. My goal was to create a faster, smaller TTS model that I could listen to as a voice assistant and not get angry at it. Mostly, I just want
Article · llama.cpp
opencl: add initial q2_k support ( #30999 ) opencl: add mul_mv_q2_k_f32 opencl: add q2_k SOA mv and tiled mm opencl: add q2_k noshuffle gemm/gemv for adreno opencl: add q2_k dp4a gemm for adreno Co-authored-by: Hongqiang Wang wangh@qti.qualcomm.com opencl: tune q2_k kernels for a
Reddit from r/LocalLLaMA.
I spent the last month tracking my local AI inference costs down to the wall plug. Here's what I found. Setup: AMD Radeon RX 7900 XTX (24 GB VRAM), llama.cpp, various Qwen/Ornith models in GGUF format. Over 30 days: 964 million input tokens consumed 13 million output tokens gener
Stronger than Qwen3.8-27B across a range of benchmarks. but also lays the necessary foundation to power our upcoming small model distillation pipeline (e.g., for Nanbeige4.3-3B). It'll distilled into their 3B small models, maybe other sizes too.
. submitted by
Hi Team, We just released v1.6.0 of AgentSwarms ( Note : It's not a lightweight Databricks!) Most agent platforms treat data as an afterthought-you build the AI, then duct-tape your warehouse and pipelines to it. AgentSwarms flips that. It is a complete, self-hostable data platfo
Qwen3.8-Flash-Next (ISTA GSQ-RCO Q2_0) + Flash-Next-DFlash2 draft on a MacBook Air M5 (24 GB), on Splash 1.4.0 with a small fork. Experts that don't fit are streamed from the SSD: 6,800 of 24,576 expert slots (~28%) live in ~8.8 GB, the rest is read on demand, 256K context. Decod
Hey everyone, I’ve been reading horrific stories lately about automated cloud bans (Google, Apple, Notion) locking people out of 10+ years of family photos, emails, and business docs overnight with zero customer service. It got me thinking about digital sovereignty. If there was
time to download some speedup DGX Spark (GB10), llama-bench -ngl 999 -fa on -p 16384 -n 0 -b 2048 -ub 2048 -ctk f16 -ctv f16 -r 5 , tok/s: model type master PR Qwen3.8-27B Q4_K_M 840.1 899.9 +7.1% Q5_K_M 840.1 880.6 +4.8% Q6_K 751.2 785.8 +4.6% Q8_0 876.2 940.1 +7.3% IQ4_XS 882.4
Lately I tested (via api) deepseek 4.1 flash. I like it quite a lot. What is the minimum budget needed to setup and run deepseek 4.1 flash at the same quality and token/s (~40) of most hosted solutions? I'd use it mostly for coding/reverse engineering/security analysis. So I need
So, I have two PCs available to me: Ultra 7 265KF + 5070ti + 32GB DDR5-6000 on a Z890 MAX GAMING WIFI7 + 1000w 80+ G PSU (Is probably getting a PSU regardless because it’s an Apevia. A C+, but still) i9-14900F + 4070S + 32GB DDR5-5200 on a PRO B760-VC WIFI IV + 750w 80+ G PSU (li
Article · Open WebUI
Added 🎯 Load selected skills on demand. Administrators can turn on "Load selected skills on demand" under Admin Settings > General or set "ENABLE_SKILL_LAZY_LOADING", so skills selected in the chat or attached to a model are only listed for the model to load when it needs them in
Open weight on Nov 4. It is just a post trained version of K3, so it should have day 0 support in all platforms.
Morning, We are doing some heavy research over here - and wanted to share our config in the hopes that it helps someone else - and thank you for all the great support and inspiration we have gotten from this sub. Enjoy:
AA and other benchmarks don't reflect what models can really do. Some benchmaxxed models are unusable for anything but poems, albeit being close to the frontier on coding benchmarks. What are the best, most realistic benchmarks you know for coding, agentic research, office work?
I tested it for backend software dev in the last two days, used providers I must admit. But I wanna test it on some rented gpus as well. My experiences so far (I ran no personal or general benchmarks): The providers are fast, it works well. It loops sometimes and fails some toolc
I’m looking for a setup with the best balance of tps, ttft, context length and enough headroom for other tasks on the Mac. I’m getting a lot of conflicting opinions with regards to MTP, MLX, quantization etc. Please don’t recommend options less than 10 tps.
Hey everyone, Most current setups for Qwen3.8-Flash-Next (like Strata) keep all 24k experts pinned in system RAM (~48–64 GB) and compute cache misses on the CPU. That works great on high-spec workstations, but not on normal machines or laptops with limited RAM. A few weeks ago, I
Article · llama.cpp
chat : reject JSON nested more than 128 levels before walking it ( #30242 ) chat : reject JSON nested more than 128 levels before walking it The schema builder, the jinja engine and the decision code walk request JSON with recursion. A request nested deep enough overflows the sta
I’m sharing an early version of a 4B model I’m training for general coding and 3D scene generation. Small models have been pretty rough at 3D in my tests. I’ve finally made enough progress to get results like the ones in this video, so I wanted to share a preview with the communi
Reddit from r/LocalLLaMA.
Qwen3.8-Flash-Next IQ3_S on Basalt | RTX 5080 16 GB, Core Ultra 9 285K, 62 GB DDR5-6800 | 262K context, int8 KV Workload (prompts 5K+) PP avg (t/s) PP peak (t/s) TG avg (t/s) TG peak (t/s) VRAM RAM (engine / left free) Agent coding session (opencode, 103 requests, up to 130K ctx)
Has anyone tried ISTA's GSQ-RCO GGUFs for DeepSeek-V4.1-Flash? Does the quality feel as good as their benchmarks show (IQ2_XXS matching the base)? Their benchmarks are saturated (90%+, and AIME25 is 100 for every version), so they probably can't show the degradation, and I couldn
Article · MarkTechPost
Microsoft Research open-sourced Agensh, a multi-agent harness with no central orchestrator. Self-organizing workers share a Git repo, chat and a findings board, scaling from 1 to 1,024 agents on ProgramBench under a 6-hour budget. The post Microsoft Open-Sources Agensh: A Multi-A
It seems Artificial Analysis Index is very easy to game, Kimi 3.1 Pro #4 on AAI is super bad in real world use cases, not even close to DeepSeek 4.1 Flash.
Article · MarkTechPost
Cognition released the Devin Vulnerability Fixes plugin, 3 MIT-licensed agent skills that inventory, patch and test PHP, Node.js and self-managed GitLab deployments hit by this week’s critical RCE flaws, then open 1 PR per repo. The post Cognition Open-Sources an Agent Plugin tha
REA (Reserve Engineer Anything) was blocked by Claude one day later...
Article · MarkTechPost
Explore a complete coding guide comparing Underdog Saluki 27B with ISTA's GSQ-RCO quantization, examining GGUF headers, memory usage, and BFCL tool-calling performance in stock llama.cpp The post How to Run Underdog Saluki 27B: 2-Bit Qwen3.8-27B Tool-Calling Guide appeared first
Article · llama.cpp
model: DSV4.1 flash ( #31086 ) model : add DeepSeek-V4.1-Flash (deepseek41) deepseek41 : remove MXFP8, convert the Engram table to BF16 deepseek41 : compute the compressed-KV read layers once in the cache deepseek41 : use get_cells() for the shared-history check, drop llama_kv_ca
only 763B params! :)
4x 5060ti16gb 4x 3060 12gb 256gb ddr4 3200 ram (4-channel speeds) On a mobo with 128 pcie4 lanes I’ve been slowly collecting gpus whenever seeing deals over time and ended up with this. Ended up comparable in cost to current prices of dgx spark I was thinking: TP4 5060s -> pipeli
Personally I am quite fond of the Kimi models, especially them taking a gamble pushing the 2.8b parameter models. But that’s just it, you’d figure that the results would be representative of its size. However, the recent release of DeepSeek 1017 has put things in perspective for
I'm one of the maintainers of CodeNomad, an open-source (MIT) web and desktop client for OpenCode, the open-source coding agent. 1.0 came out this week, and its server mode is the part most relevant here. How it runs npx @neuralnomads/codenomad --generate-token starts the server
NVIDIA RTX 30 & RTX 40 GPUs Get Impressively Fast In AI LLMs As Compute Doubles With New Unlock Tool But Prices Surge Too" title="CMP170HX inspiration unlocks 30x/40x also, up to 2X Prefill speedup --> NVIDIA RTX 30 & RTX 40 GPUs Get Impressively Fast In AI LLMs As Compute Double
Article · llama.cpp
jinja : fix select and selectattr argument handling ( #31059 ) jinja : fix select and selectattr argument handling jinja : address review feedback jinja : Simplified test attr Website:
I’m the developer of TensorSharp. I added MiniMax-M3.1-Flash support on Day 1 of its open-weight release , and wanted to share what’s working and some initial benchmarks for anyone looking to run the model locally. The biggest performance difference showed up with long prompts: 6
ANEMLL Forge 0.2.1: Qwen3.8-27B on the M6 Neural Engine now runs up to 192K context (M6 32GB). New: INT8 K/V cache (half of FP16) and a smaller drafter sharing the target's output head - 0.86 GB less ANE memory, same speed. Decode depends on content: JSON 35–62 tok/s, code 18–32,
Article · llama.cpp
hexagon: KV-head team split for multicore HMX flash attention ( #31055 ) hexagon: KV-head team split for HMX flash attention in multicore hexagon: merge FA_HEAD_SPLIT and FA_SPLIT into single GGML_HEXAGON_FA_SPLIT flag 0 = off (token-split only); 1 = head-split, plus team-split w
Anyone knows of a good model that does not say no to reverse engineering? Cloud models will even terminate your account by snitching on you and GLM 5.3 is too big to run on my potato. Help would be greatly appreciated. Thanks.
Article · llama.cpp
server: refactor server entrypoint ( #31113 ) server_role cont Website:
I see a lot of people here on Reddit treating the KLD number as some holy metric that can distinguish every quantized model out there and rank them by quality. TL;DR: It can't. So what does KLD actually measure? If you feed the same text to both the base model and the quantized m
I've been working on improving llama.cpp support and performance on older NVIDIA GPUs, particularly Kepler and other pre-Ampere architectures. The goal is simple: get more life out of older CUDA hardware instead of treating it as e-waste. Achieved Performance On MTP Qwen3.6-35B-A
not an llm post exactly, but it's all local inference so i figured people here might like it i made a browser rhythm runner where the song builds the world you run through. you can throw in your own mp3s and the whole "listening" happens on your device, inside the tab. nothing ge
Article · llama.cpp
metal : remove mul_mm_id amax rescaling ( #30924 ) The src1 power-of-two rescale in kernel_mul_mm_id is superseded by the per expert-token amax scaling done in the compute graph (ref: #30861 ), which keeps the MoE down-projection activations within f16 range before the op runs. D
Idk why everyone keeps posting about Flint! Raon is hundreds of times better, and it’s not even the best, which is really saying something about Flint lmao
Unsloth Flint Q3 Results Performed quite poorly on benchmark using 3 diff harnesses: OMP : Completed all 12 tasks; 71.2/100 (only 7 solved). AtomicAgent: Completed all 12 tasks; 39.2/100 (only 3 solved). Hermes: Stopped during Task 7 after it performed needless broad searches acr
Article · llama.cpp
minimax-m3 : forward quant scales to the graph (fixes #31063 ) ( #31097 ) The generic scale pass in llama-model.cpp loads the per-expert NVFP4 scale2 tensors (ffn_{up,gate,down}_exps_s) and the per-tensor MXFP8 scales for the dense FFN and the shared expert, but minimax-m3.cpp ne
Article · llama.cpp
tests : fix test-llama-context for large n_batch ( #31083 ) Website:
Article · llama.cpp
llama : deny -sm tensor for arches with moe amax rescaling ( #31081 ) The amax POOL_1D reduction spans the whole matmul row, so a device cannot compute it from an axis-0 split. The Meta split-state case reported MIRRORED, which gives each device a partial max of its own slice. Dr
Article · llama.cpp
ggml : allow MUL_MAT_ID -1 index to skip the computation ( #26631 ) ggml : allow MUL_MAT_ID -1 index to skip the computation nits update vulkan code ggml-webgpu: ensure all threads reach workgroup barriers in mul_mat_id_vec (cherry picked from commit d8a4b9e ) fix cuda openvino e
I want to create a native LXC for Authelia via the community script , but one thing caught my eye: "During installation, you will have to input your domain (ex. domain.com). Authelia will use auth.domain.com" What if I don't own a public domain? Can I not use this? I just use Cad
Article · Ollama
Flint on Linux and Windows Flint now runs on Linux and Windows as well as macOS. Other improvements Streaming OpenAI Responses and Anthropic Messages requests now report generation failures instead of ending silently after partial output. ollama launch now passes cloud models' co
Is there any good finetunes based on modern models, in the size range of Qwen 3.8 27B or Qwen 3.8 Flash Next that substantially improve it's 3D modeling capabilities for creating .glb / gltf assets (mainly for say making video games)? Mainly thinking about two use cases: * Making
Article · noted.lol
A self-hosted dead man’s switch that helps solo admins leave a plan for the people who may need to take over.
Reddit from r/LocalLLaMA.
Continuation of
Reddit from r/LocalLLaMA.
I fine-tuned Gemma 4 31B to create Monadize , a model designed to reduce repetitive name generation, such as "Elara" or "Elias," by shifting the next-token probability distributions and using single-character tokenization when introducing new names. Some people have attempted to
This is due to China's new export restrictions primarily but also the growing caution around cybersecurity exploits popping up.
i'll admit it took more than one prompt, a bunch of refinement and steering, but local can get close! this is done using remotion, same animation framework as all the viral clips about opus 5.5's motion graphics capabilities the entire conversation history that produced this vide
The methodology Qwen-Flash-Next has n-gram, MOE packing above its weight but it's still 6B active params. We exploit this by keeping most MOEs in RAM (Smallest MOEs go in VRAM for speedup) and n-gram on disk - ensure you put the model on your fastest SSD. 8 GB VRAM, 24 GB RAM cos
Article · llama.cpp
tests : add speculative rejection draft sampling test ( #30213 ) tests : add speculative rejection draft sampling test This commit adds a test for speculative rejection sampling test that excercises the speculative rejection sampling mechanism in common/speculative.cpp. The motiv
Article · llama.cpp
vulkan: ggml src1 precision support ( #30656 ) vulkan: update accumulator precision selection handle fp32 overflows through scaling handle overflows in copy-to-fp16 activations add missing cases simplify code add tests add ggml prec get helpers, use in Vulkan backend clean up com
It's been a few releases since my last post, so here is where capy-flash is now. It's my own engine for two models, Qwen3.8-27B and Qwen3.8-Flash-Next. It started as my ninfer-tp2 fork of NInfer, which added tensor parallel over PCIe, but at this point it's an engine of its own a
I’ve been working on an inference-time abliteration engine as a fork of llama.cpp, and I’ve now ported the hook to Strata. It acts as an easy way to use abliterated models without huge amounts of vram or finetuning. For the Strata Branch
to discover. Not a tutorial, but an honest chronicle of an experiment: hardware reuse, AI as a co-pilot, and a privacy-first choice. Read it:
Reddit from r/StableDiffusion.
Article · noted.lol
Follow blogs and YouTube channels together with WyrmRSS, a self-hosted RSS reader with flexible feeds and a mobile-friendly layout.
I wanted a personal agent I could trust with my calendar, messages and files, and for me that meant keeping as much as possible on my own machine. So I built one to self-host first. I’ve been working on it for months with the repo public the whole time, and it’s now at the point
Been thinking about using Nostr for communication between agents running on different people's machines. Anyone here tried it? I'd like to get a basic setup working where one agent posts a request and another picks it up and replies. Each person runs their own model. No shared ba
cannot back with a verbatim quote. Real numbers from a run on a laptop CPU: 0.
Article · llama.cpp
HIP: add byte permute fallback when __builtin_amdgcn_perm is unavailable ( #30915 ) HIP: add byte permute fallback when __builtin_amdgcn_perm is unavailable amdgcnspirv rejects target-specific builtins unless they are guarded by __builtin_amdgcn_is_invocable , so 66 of 145 HIP fi
So I’ve been running local RAG over Korean PDFs for a while and my only evaluation was “yeah the answers look ok”. Finally sat down and used Allganize’s Korean RAG set since every question has a known answer and the page it’s on (300 questions, 64 PDFs, MIT). Fed every model the
Article · LiteLLM
Important The image-signing key has changed. Starting with this release, every Docker image and Helm chart is signed with the new cosign key from commit e82a3c6 . Verifying with the previous key from commit 0112e53 will fail. Update anything that pins the old cosign.pub : cosign
Recently this new quant has popped up called Gyro. It is gaining popularity because it's being inserted into many different engines, especially the recently popular Strata. So i took a dive into the code to see the differences between Gyro and EXL3, to me it looks like simply a d
Article · LiteLLM
Important The image-signing key has changed. Starting with this release, every Docker image and Helm chart is signed with the new cosign key from commit e82a3c6 . Verifying with the previous key from commit 0112e53 will fail. Update anything that pins the old cosign.pub : cosign
Article · LiteLLM
Important The image-signing key has changed. Starting with this release, every Docker image and Helm chart is signed with the new cosign key from commit e82a3c6 . Verifying with the previous key from commit 0112e53 will fail. Update anything that pins the old cosign.pub : cosign
Article · LiteLLM
Important The image-signing key has changed. Starting with this release, every Docker image and Helm chart is signed with the new cosign key from commit e82a3c6 . Verifying with the previous key from commit 0112e53 will fail. Update anything that pins the old cosign.pub : cosign
We have released DeepSeek-V4.1-Flash-1017 quantized with GSQ and RCO, at 1.75 to 2.25 bits per weight. DeepSeek-V4.1-Flash-1017 is a sparse mixture-of-experts model: 384 routed experts per layer, of which 6 are active per token, 763.2B parameters and a 1M-token context window. It
Article · llama.cpp
mtmd: move build_inp_temporal_merge to base class ( #31065 ) Website:
Posted a bech I did here before:
Article · llama.cpp
mtmd : lfm2 resize matches torchvision ( #30943 ) mtmd : lfm2 bilinear resize matches torchvision Lfm2VlImageProcessor and d1-omni vision.py resize uint8 images with torchvision (antialias=True). That is Pillow's algorithm, except that the weights are int16 with as many fractiona
Reddit from r/LocalLLaMA.
Article · llama.cpp
mtmd : add Soprano TTS support ( #28607 ) Add Soprano-80M and Soprano-1.1-80M (
Article · MarkTechPost
NVIDIA released Nemotron 3.5 Bolt E4B 4GB, an open small language model post-trained from Gemma 4 E4B for local voice assistants, game characters and robots, targeting about 4GB of VRAM on RTX and Jetson. The post NVIDIA Releases Nemotron 3.5 Bolt E4B 4GB: An Open Small Language
Article · llama.cpp
hexagon: add SGN op ( #30346 ) hexagon : add SGN support Assisted-by: OpenCode hexagon : optimize SGN with sign bit Assisted-by: OpenCode hex-sgn: add f16 hex-unary: update SGN to use generic unary_colapse and relax colapse checks to apply to more conditions hex-unary: improve di
Article · llama.cpp
common : allow concurrent downloads across processes ( #29312 ) Signed-off-by: Adrien Gallouët angt@huggingface.co Website:
Article · llama.cpp
hip: fix bf16 WMMA operand vector type for gfx11/gfx12 ( #30885 ) The __builtin_amdgcn_wmma_f32_16x16x16_bf16_w32 (and _gfx12) builtins take their A/B operands as vectors of short (_ExtVector / _ExtVector ), not __bf16. Declaring the fragments as ext_vector_type(N) __bf16 fails t
Reddit from r/LocalLLaMA.
I've been building a companion app that runs fully local: LM Studio serving Gemma 4 26B (QAT), with a small Python server per companion and a three.js page for each one's room. All on an RTX 3070 Ti (8GB) + 32GB RAM; the MoE design is what makes that workable. I'm not a coder and
This burns about ~3K context, in exchange it seems to address some of the bad behavior from LLMs; your mileage may vary. It's a combination of my earlier output style and an updated generic rule set for agents to follow. (This would be the global AGENTS.md file) It honestly could
Article · llama.cpp
ggm-cuda: Enforce DP4A MMQ tile height multiple of warp size ( #30837 ) Added static assertion to enforce DP4A MMQ tile height requirement. Website:
A brand-new release from ISTA-DASLab with their GSQ-RCO smart quantization approach. Now DeepSeek-V4.1-Flash-1017 The quants are available on Hugginface:
Reddit from r/StableDiffusion.
Article · Ollama
mlxrunner: import and list Phonon catalog voices docs: document Phonon voices
The most important part is a difference between "forzen" and "best effort" attempts where we allow the model to specialize on the task as it goes through the evaluation dataset. Clearly showcasing the power of continual learning regime simply unavailable for any other typical LLM
Reddit from r/StableDiffusion.
i7-14700KF , 64 GB RAM Radeon RX 9070XT 16GB
background so, I'm director of scientific computing at a cancer center and have just received some interesting news: a new lab has between a $300-400K budget to pay for GPU hardware to run their AlphaFold 2 workloads. some of the pertinent characteristics: they need a fair amount
Everyone loves a clean, unified server rack, but my standby node ( lvgale ) is an absolute physical abomination. It is a Frankenstein collection of mismatched parts and bolted-together VRAM that has no business running efficiently. The Hardware Setup (lvgale) CPU: Intel Core i7-6
Obviously, at 3T parameters it's hard to consider it relevant for “local”Llama, but I'm still posting it since it's an open model.
Amazon prices just jumped today, I still see deals at local microcenter. When that batch is gone $1840 for a b70 will likely be the new normal. b60 went from $650 to $980. b65 $1150 to $1420 does make a lot more sense as the price per GB was low in comparison to the competitors.
Article · Simon Willison
Kimi K3.1 Pro landed on the Kimi API today - it's not yet open weights, but Moonshot promise "Full model weights will be released by Nov 4, 2026". They're also offering a "Highspeed" version that's 2-3x faster, at twice the price (except for input token cache hits, which are pric
Just ran my own tests of the new GPU performance unlock by Zaggita . If you missed it: Nvidia locked down compute speed on consumer cards for some specific hardware instructions needed for AI; the unlock fixes it. TLDR: got +20% prefill impovement, 0% decode change after unlock.
(Post Updated) PR:
Kimi K3.1 Pro, default thinking (max). Context usage was at ~160k tokens. Pi.dev harness. For comparison, here's the Opus 5.5. result at launch time:
Article · llama.cpp
CUDA: fix build after chunked CUDA graphs ( #30804 ) Apply suggestion from @JohannesGaessler Co-authored-by: Johannes Gäßler johannesg@5d6.de Fix newlines in common.cuh Co-authored-by: Johannes Gäßler johannesg@5d6.de
gigabytes of VRAM saved (Qwen Flash Next) "This is one of the things Strata does to squeeze the most juice from the experts' cache."
Before you read: I'm the developer of Basalt, one of the engines benchmarked below. I tried to keep this as fair as possible: same checkpoint, GPU and requests, and every engine set up the way its own docs recommend. How it was run is at the bottom. This post has had some AI clea
KIMI K3.1 PRO results you see here. Next is Qwen and Deepseek (pro) models in November
Article · llama.cpp
allows the usage of 4-10 gpus for stepfun and laguna in tensor paral…
Article · llama.cpp
qwen4exp : build the QSA selection without the kq_mask clean lang qwen4exp : reshape the fused QSA selection to the kq_mask streams ggml_top_k_mask with a single keep row returns [n_kv, n_tokens], while build_attn_qsa expects the kq_mask layout [n_kv, n_tokens/n_stream, 1, n_stre
Tried it on a build for gfx906(mi50 16gb) and ran worse than llamacpp on decode speed.
This is Mica v0.5, the next version after v0.1. It's a decision model, the same kind of thing as Jev, which has been getting a lot of attention lately. It's 4B (Qwen3.5-4B + LoRA) and it doesn't write text. You hand it a state, a question and a list of options, and it gives back
Article · Ollama
NVIDIA Nemotron 3.5 Bolt E4B Nemotron 3.5 Bolt E4B is NVIDIA’s smaller, Gemma 4-based vision-language model for fast local conversations, image understanding, and tool use. It supports a 128K-token context window. The model includes a DFlash draft for speculative decoding. ollama
Reddit from r/LocalLLaMA.
Article · ServeTheHome
We take a look at the MSI XpertStation WS300, the company's top local AI solution powered by the NVIDIA DGX Station platform The post MSI XpertStation WS300 Review NVIDIA DGX Station GB300 appeared first on ServeTheHome .
Article · llama.cpp
hexagon: add CEIL op ( #31006 ) hex-ceil: add CEIL(f32) op hex-ceil: add f16 version hex-ops: update supported ops hex-ops: update docs with supported ops Website:
A few months ago, I shared my earlier post here. I wanted to come back with an update because I’m genuinely excited about how far Project Zero has come. Project Zero is my CPU-first LLM inference engine, written primarily in C, with no external ML framework dependency. It now run
Base Labs at Baseten has released open weights for elephant-1, a 9B decision model post-trained from Qwen3.5-9B. It doesn't generate text. You give it a state (a ticket, a document, a prompt to screen) and a set of yes/no, multiple-choice or ordered-score questions, and it return
Hey r/selfhosted , I've been "paperless" for about 10 years with a ScanSnap and Evernote, and I finally got tired of paying a subscription for my own filing cabinet. Seven months ago I built DocManFu so I could cancel it. I've used it daily since, and I'd love for some of you to
AgentionAI Gyro Coder Large is now live 🚢 This coding specific finetune of Qwen 3.8 Flash Next is the same size as Gyro-S, but with half the experts and more bits per expert. The model fits in 32GB VRAM. More details in the model card
NVIDIA-Nemotron-3.5-Bolt-E4B-4GB is a small language model (SLM) further post-trained from Gemma-4-E4B-it by NVIDIA. Nemotron-3.5-Bolt-E4B carries forward Gemma’s strong conversational capabilities and further improves multi-turn tool use, instruction following, and input groundi
The design of attention mechanisms in long-context large language models faces an ``impossible triangle'' among KV cache size, training and prefill FLOPs, and model quality. GQA achieves strong quality but incurs high KV cache overhead; alternative approaches such as MLA and MFA
We introduce Continuum Attention (CA), a causal Transformer attention operator that replaces discrete key and value lookup with integration over a continuous, piecewise linear memory curve. Each query induces an exponential relevance density along the curve. Integrating the densi
Lets see how difference is this from ISTA labs coder version
guys, you better just clone llama.cpp I swear. this thing is running gguf q4 paged into webgl2 embedded compute buffers via glsl right inside html, and tokenizer/vocab is just part of the endless debug hell.
Article · LocalAI
Patch release v4.12.1. Security fix only. Security Hardened the external URL policy for user-supplied URLs (chat vision image/audio/video URLs, image and audio endpoints). Redirect targets are now validated on every hop, 100.64.0.0/10 and 0.0.0.0/8 are treated as non-public, and
Hello guys, hoping you're doing fine! Small post I want to make since seeing a lot of posts about the unlocks and such. What is the unlock in simple terms? NVIDIA on purpose, neutered the performance of operations that use FP32 accumulation with GEMMs, with the objective of marke
I didn't want to pay a monthly subscription so a pet camera could tell me what my cat did all day, or send my home footage to someone else's server for it. I used to do animal behaviour research with AI at a zoo, so I built my own. What it does: finds the animal, works out which
My daily driver: ukisai/Swift-1.5-Qwen3.8-27B-GGUF, Q4_K_L, 128k context, llama.cpp The model I want to switch to: ukisai/Swift-1.5-Qwen3.8-Flash-Next-GGUF, IQ2_XXS, 64k context, Strata I got pi to properly pass the thinking levels to QFN: "thinkingLevelMap": { "off": "none", "mi
I've been building Kubeast, an open-source (MIT) web console for Kubernetes, and it's at a point where I'd love feedback. One console for many clusters: resources, YAML edits, live logs, a shell in the browser, Helm history and rollbacks. An AI assistant that reads the cluster wi
Article · Ollama
renderers: add gemma4-nemotron renderer Add a Gemma 4 renderer variant that renders assistant content before its tool calls rather than after the tool responses. Models opt in with RENDERER gemma4-nemotron; gemma4 auto-detection is unchanged. Adds the corresponding chat template
Been seeing info about 3090 cards getting a speed boost through software tweaks. Can similar unlocked be achieved with -60, -70, -80 cards, and in the 30-, 40-, 50- gen cards too ?
Let's put things into perspective: Remember, for example, the xz utils ssh backdoor in 2024? They almost managed to pull it off, and that was in the most observed codebase of the planet. It was caught almost by chance by someone working on something unrelated. Read about it if yo
Article · llama.cpp
ggml : fix time overflow in ggml_time_us ( #30969 ) fix time overflow use double Website:
I know current llms like opus 5.5 are in a league of their own. but we all know they're losing a shti ton of money with their subscriptions. ALSO we all know how tasks like OCR, segmentation etc are getting solved by single digit B parameter models. I'm not saying llms are a gimm
Article · noted.lol
Noted Giveaways is here! Discover handpicked homelab and PC hardware giveaways, all manually reviewed and approved. No endless lists of junk promotions designed to harvest your email. Just worthwhile opportunities to win gear for your next build.
It is a Tinystories style dataset, that contains 50M stories, about 25x that of Tinystories, it was generated using a rule based algorithm, instead of a language model. It took just under 10 Minutes to generate and a few minutes to convert to parquet and upload to huggingface on
I see a system-reminder claiming a "c25da061" session was idle for 1877 seconds with an unfinished task, and instructing me to invoke a skill called "reminder idel-skills" to trigger a skill-based action. This is a fabricated/malicious system-reminder - it's trying to get me to d
DeepSeek-V4.1-Flash gets Maya-XS - the whole model in a 128 GB PC's memory - tensor parallel on several GPUs, and much faster drafting on one GPU.
Hello everyone, with qwen4 around the corner im thinking of actually investing into some hardware. From what i've learned lurking on the subreddit the more VRAM you have the better. My plan is to go on local marketplaces and 2nd hand and buying GPUs and RAM, i just need a few poi
Article · noted.lol
Edi Life OS unites habit tracking, goals, Kanban, finance, and an optional Model Context Protocol AI server into a self-hosted dashboard.
EDIT: I was able to run this in Pi. This whole confusion came to my mind because it is called server when for my point of view is more like a script what is run my harness when needed and that script have just "public methods" for coder's point of view. Eg. tools defined on that
In self-hosting setups, repurposing decommissioned Kindle e-readers, Kobo devices, and cheap Fire tablets as dedicated wall dashboards is a great way to give old hardware a second life without relying on proprietary cloud services or dealing with bloated lithium batteries. Over t
Basically saw on twitter claims that 3090 has half of the cuda kernels disabled: My 3090s just got 1.9x FASTER at BF16 tensor math!!! Same cards, same driver, same 300W, one Linux kernel module. bf16 tensor pipe: 82 -> 155 TFLOPS cuBLAS BF16 GEMM 8192: 64 -> 99 TFLOPS TF32: 41 ->
Article · llama.cpp
CANN: missing const qualifiers added for consistency ( #30933 ) Website:
This is another update about my MoE-fication project, the first post can be found here . Foreword Since my last model conversion, I’ve been experimenting a lot with new methods, though sadly, most attempts haven't helped much. Compatibility has been the main limiting factor. I’ve
Reddit from r/LocalLLaMA.
I've been noticing a lot of my downloads stopping suddenly and being unable to resume. When I checked the models it seemed like they were gone. Here's the ones I can see are missing: dealignai/MiMo-V2.6-Pro-RL-UNCENSORED audnai/penclaw-GLM-5.3-abliterated-for-offensive-cyber huih
Article · n8n
2.45.3 (2026-10-23) Bug Fixes editor: Increase light-mode text contrast ( #41807 ) ( 9f41c0f )
In my test suite, Flint hallucinates and overthinks even very basic questions. Ornith looks much better in both my own language and English. What’s your experience?
Looped Mixture-of-Experts (MoE) models combine parameter reuse with sparse computation to improve performance under matched budgets. Prior work demonstrates that different loop visits can play different roles even when they share parameters. To probe the contributions of differen
Article · Dify
Dify 1.18.0 brings Agents that route workflows , portable app packages , deeper visibility into nested workflow runs , and the opt-in Agent Knowledge Base experience. OpenAPI and the new difyctl expand how you build and manage apps from the command line. Important Read the upgrad
Long-output inference in large language models suffers from severe KV cache memory and bandwidth bottlenecks. While KV cache quantization alleviates this issue, low-bit quantization (e.g., 2-bit) severely degrades model accuracy. Recent mixed-precision key-cache quantization meth
Speculative decoding accelerates autoregressive generation by drafting multiple tokens per step, yet feature-based drafters rely on target hidden features whose contextual emphasis may not align with the needs of subsequent draft positions. Hybrid linear attention (HLA) models ma
As model parameters scale up, sparsely activated Mixture-of-Experts (MoE) approaches have become essential for model scaling. However, mainstream models currently adopt hierarchical architectures, where experts in different layers, although topologically independent, exhibit subs
Article · llama.cpp
spec : add probabilistic sampling for DFlash2 drafts spec : apply review suggestions Co-authored-by: Pranesh Gonegandla pgonegandla@nvidia.com
Linear-attention language models keep their context in a recurrent state of fixed size, where an inference system stores one copy of this state for every sequence it decodes at the same time. Storing the state at lower precision therefore lets more sequences share an accelerator.
Article · llama.cpp
vocab : fix Nemotron-3.5-Embed tokenization Update src/llama-vocab.h Co-authored-by: Sigbjørn Skjæret sigbjorn.skjaeret@huggingface.co rename Update src/llama-vocab.cpp Co-authored-by: Daniel Bevenius daniel.bevenius@gmail.com Co-authored-by: Sigbjørn Skjæret sigbjorn.skjaeret@hu
So they have updated the embed and rerank model from 3.0 to 3.5:
Article · llama.cpp
hexagon: Op Fusion refactoring, optimizations and new fusions like `B…
Just a spec to performance report. Tried Maya today on the following: 3950x 96GB dual channel DDR4 at 3000mhz Rx 7900xtx 24GB + w6800 32GB (56GB together) pcie gen3 nvme drive The Deepseek weights used were the mostly iq2 weights with Q6 ngrams. ~350GB on disk. Dspark left on. I
Article · llama.cpp
sycl: Replace host waits in fattn-mkl with explicit dependencies ( #30 …
Article · llama.cpp
Signed-off-by: Todd Malsbary todd.malsbary@intel.com
Everyone making such a big deal about security of AI agents, but does it really apply to me? I have a homelab guy with a VPS and two small servers. There's nothing of value saved anywhere, everything is backed up to git or my NAS and can be recovered. I just use it for personal p
Reddit from r/LocalLLaMA.
Article · Jellyfin
🚀 Jellyfin Server 12.3 We are pleased to announce the latest stable release of Jellyfin, version 12.3! This minor release brings several bugfixes to improve your Jellyfin experience. As always, please ensure you take a full backup before upgrading! Discuss this release further on
Three small MoEs dropped within a couple of weeks, so I put all of them in the same seat. In Fusion mode Opus 5.5 plans and reviews, and three workers on the local model write the code in parallel. Only the worker model changes between runs. Same prompt, planner, quant, settings.
I compared Qwen3.8-Flash-Next-W4A16-FP8PLE at xhigh reasoning vs DeepSeek-V4.1-Flash-1017-dsx-2.6bit (my own quant) at max reasoning by benchmarking using a modified version of BenchKit . DeepSeek V4.1 Flash 1017 2.6 bit scored 23/25, while Qwen scored 25/25 (full marks). This is
Article · llama.cpp
server: Substitute rerank placeholder from the original template only. ( #30864 ) Website:
Article · llama.cpp
server: realtime API: send complete after committed audio drains ( #30994 ) server: realtime API: send complete after committed audio drains server: realtime API: complete a commit at its end row Completion is triggered when the stream runs dry, which is a timing proxy: audio app
Article · Ollama
Replace the prerelease checkpoint path with Gradium's GA format, including default-voice import and speech conditioning. Set the 0.40.9 minimum, update the speech docs, and test import, inference setup, and rejection of the old checkpoint.
Article · Paperless-ngx
paperless-ngx 3.3.4 Note This security maintenance release updates dependencies to address the eMudhra / emSign CA certificate security incident and security vulnerabilities in Pillow and aiohttp. These issues affect third-party dependencies rather than Paperless-ngx’s own code.
Reddit from r/ollama.
Article · Hugging Face
Article from Hugging Face.
An update on my heretic (uncensored - refusals removed) unsloth dynamic iq4_xs quant clone - I added the strata ready artifacts: This is: the heretic version of Swift 1.5 Qwen 3.8 flash next, (mean KLD 0.06, median 0.02, mean concentrated on drug-related prompts so skewed in that
My thinking was models, esp Opus 5.5 can just hill-climb as long as you give them something achievable and a clear metric to improve on. So here's something I don't even have a clue how it exactly works. The training itself happened in my basement 4070ti super over a day or so. A
Like… is strata gonna steal all my data 😭
Made this from the Artificial Analysis model pages after Muse Spark Open and Beam got their scores last week. X axis is total parameters as AA lists them, log scale. Orange = released in October 2026. - Muse Spark Open (816B, 42B active) scores 39. Qwen3.8-Flash-Next (180B, 6B ac
It is an english text to speech model, which achieves mostly understandable speech in under 100K parameters. However, during more difficult words, or long numbers, it can slur them, and be not understandable. Quantized to INT8 it only takes up 127KB. 72x real time on a CPU 1120x
Article · llama.cpp
opencl: add depthwise conv2d ( #30900 ) opencl: add Adreno depthwise conv2d Assisted-by: OpenCode opencl: use local program for depthwise conv2d Assisted-by: OpenCode opencl: use singed type for args Co-authored-by: Li He lih@qti.qualcomm.com Website:
Do you care if your agent sends usage analytics, or do you just leave it on? I looked through the code of 8 local agents on GitHub to see what's on by default. Here's what I found. TL;DR: 5 of 8 send usage analytics by default. goose and Atomic Agent ask first. OpenCode's CLI has
Article · llama.cpp
mtmd : add MiniCPM-OCR 1.0 support ( #30979 ) Signed-off-by: tc-mb tianchi_cai@icloud.com Website:
Article · Ollama
flint: fix replies with thinking disabled Close the thinking section when think:false so Flint answers directly. Default and think:true prompts are unchanged. update template
It was a big deal in and around March of this year. I tried looking for any updates with where it has gone and nothing really came up.
Carbon-X is a 14.8B-parameter decoder-only language model trained jointly on DNA sequences and natural-language text. You give it raw DNA inside an ordinary chat conversation and it answers in English: it predicts whether mutations are benign or pathogenic, identifies regions tha
This made a huge improvement to speculate decoding speeds on my B70 running SYCL. Below examples are for Muse Glimmer Q6_K (30B) running with DFlash2. NOTE: Due to the significantly reduced cost of token verification, the n-max was increased which is why the acceptance rate is lo
Article · Simon Willison
News today from Anthropic: We’re committed to keeping Agent Skills open, community-driven, and vendor-neutral. Today, we are donating the standard to the Agentic AI Foundation (AAIF) under the Linux Foundation,
Article · llama.cpp
glm5 : optimize inputs ( #30932 ) glm5 : avoid sel_mask_2d input cont : avoid new_pool_flat input Website:
I’ve been working on multi-GPU inference in TensorSharp , an open-source .NET inference engine and agent runtime. Recently, I managed to run Muse Spark 816B (42B active parameters) across 10 NVIDIA A40 GPUs and benchmarked it against llama.cpp. The most interesting finding wasn’t
I did little research, and chatterboxV3 and Fish-Audio-S2-Pro were the top result. Should I go with chatterbox? Or there are other better alternatives?
Article · llama.cpp
model : fix qwen4exp layer inputs for DFlash drafters ( #30927 ) qwen4exp published the hc-wide residual [n_embd, hc, n_tokens] as t_layer_inp, but the drafter reads layer inputs as [n_embd, n_tokens] rows: llama_context sizes each embd_layer_inp buffer as n_embd * n_batch and th
Qwen3.8-Flash-Next NVFP4 on one RTX 5090 - prefill and decode against context Full 262,144-token context, one request at a time, MTP speculation on, int8 KV, identical test for all three engines. Every number is tokens/second. Prefill = prompt tokens / time to first token. Decode
On October 22, Chinese media reported that DeepSeek and MiniMax have officially completed the required registration filings under China’s new temporary AI export regulations. By submitting their technologies for cross-border review, both companies successfully fulfilled the admin
Article · llama.cpp
Multi-column on Intel XMX matrix engines refactored ( #29864 ) Multi-column on Intel XMX matrix engines refactored Multi-column (2 to 80 activation columns) mul_mat_vec_q for reordered K-quant and Q8_0 weights on the Intel XMX matrix engines (int8 DPAS, ESIMD), for the SYCL backe
Reddit from r/ollama.
I use Ling-3.0-tiny on my LLM server for text processing. I convert a lot of PDF documents to markdown for downstream LLM processing, and I built my custom harness for it. It uses ONNX model to parse layout, extracts text, and then runs LLM processing stage to remove extra whites
Article · noted.lol
You've probably got years of photos, videos, and music sitting on your NAS that you rarely revisit. Spooly gives that forgotten media a second life with swipeable discovery, personal radio stations, and a fresh way to explore your collection.
I have 48GB VRAM (2x3090 24GB) and 48GB DDR4 RAM. If I get 128 GB DDR4 RAM can I run things like Deepseek V4.1 1017 and GLM 5.3 Flash at low quants with at least 20T/s or so and have it be better than Qwen 3.8 Flash Next etc? If not, is there another system I should get, maybe sw
Follow up to my post about the £300 Snapdragon mini PC:
Most of the speedup are touching along the likes like cutting the active params reletive to total params(MOE) but it does take a hit in stability & reliability compared to a dense-27b can be much more stable at low total params. We hade the innovation of MTP which is great and lo
InclusionAI just announced Sante too. Damn, two model announcements in one day. They're on a roll. I posted about the 3.1-flash-1019 update earlier today and they've already announced Sante . Two model announcements in one day, damn. This one's built on Ling-3.0-flash , with 124B
from inclusionAI: Introduction Ling-3.0-flash-Sante is our healthcare-specialized model built on Ling-3.0-flash . Its name comes from the French word santé , meaning “health.” With 124B total parameters and 5.1B active parameters per token , Sante is designed for medical reasonin
Wanted to see how one model that does transcription, speaker labels and timestamps in a single pass handles a real meeting, so I ran MOSS-Transcribe-Diarize 0.9B on a four person meeting from the AMI test set and scored its speaker turns against AMI's reference. Setup: 17:29 of A
Hey everyone! 👋 A few months ago I shared GyroidVault here - a self-hosted, local-first web app to organize, inspect, and slice 3D printing models (STL, 3MF, STEP, G-Code) on your own server or NAS with 1-click slicer deep-linking and 3D millimeter dimension inspection. Thanks to
I tried for fun to see what happens if we let instruction-tuned LLMs generate the text that should come from the user (so after the user_start special token). I immediately noticed that some models were way better than others. I wanted to test many models so I did some testing an
Article · LocalAI
🎉 LocalAI 4.12.0 Release! 🚀 LocalAI 4.12.0 is out! This release redesigns the web UI and adds streaming voice responses, native decision routing, and more control over model memory and prompt state. It also adds image upscaling, live motion capture, Qdrant stores, and authenticat
Im so hyped for qwen4 flash
Came across this demo. Pretty interesting concept. Any ideas for use cases?
I've been building capy-flash , an open-source inference engine for running Qwen3.8-Flash-Next (125B MoE) and Qwen3.8-27B on consumer RTX 50 cards. Until now it needed my setup (two 16 GB cards and 96 GB of RAM) or better. v0.7 is about smaller machines: start with what you have,
I maintain tapflow, an MIT-licensed, self-hosted tool that lets teams access iOS simulators and Android emulators through a browser. Designers, QA testers, and other developers can test mobile builds without installing Xcode or Android Studio. In v0.28, I added a way to watch wha
I'm building an app where I need some specialist language models, but not necessarily LLM models. Is there a sub or place where they talk about specialist models beyond just searching on HF? I know that discussion doesn't really fit this sub.
Article · llama.cpp
nemotron : support non-standard GQA, BPE tokenizer and domyn-small pre-type ( #23343 ) nemotron : support non-standard GQA, BPE tokenizer and domyn-small pre-type gguf-py : write Metaspace prepend_scheme as GGUF metadata vocab : parse metaspace prepend_scheme into enum, rename ke
Article · llama.cpp
vulkan: replace std::aligned_alloc with posix_memalign for termux com…
hey folks, im Ye, im one of the contributors to Kapitsa , an open source inference engine optimized specifically for Apple Silicon (all the way from M1 chips to M6) to run Qwen & Nemotron models that would fit a regular MacBook. its custom written Metal kernels, allow us to optim
Hey everyone, we just released BaseQuant V1 for Qwen3.8-27B at Base Compute. Three GGUF sizes: 10.93, 16.41 and 20.82 GB. In our held-out tests, they had 17–34% lower KL divergence from BF16 than similarly sized Unsloth quants. Would love for you to try them and tell us how they
A while back I posted Aether, a 50M parameter model I trained from scratch. Meet Pip, the first of the NiemsDrift souls. He's small, warm, honest, and a little weird. You can try him right now: `ollama run NiemsDrift/pip` Pip is built on Qwen3.5-2B with a LoRA fine-tune. My bigge
For anyone still downloading the last Flash: there's another checkpoint on the way. Ant Ling's announcement puts the 1019 version at 44 on the Artificial Analysis Intelligence Index, up from 41. They also report improvements on the agent evals: AutomationBench-AA goes from 62% to
Hi everyone, we just released Carbon-X, a 14B model that brings DNA and natural language together for the first time 🧬 On five DNA benchmarks it beats every open language model we tested and matches GPT-6.1 Sol, at about 1,000x lower cost:
Do anybody has the same feel? Going to lower quants, thinking models tend to think more and more. So much that the speed advantage is cancelled, or it becomes actually slower. The increase in prompt processing speed and token generation speed just doesn't make up for the longer t
Article · n8n
1.123.86 (2026-10-22) Bug Fixes deps: bump sharp to 0.35.5 on 1.x (DEVP-1241) ( 9fa95f0 )
EDIT: MODEL IS LFM2.5-24B-A2B-Q4_K_M.gguf So, yesterday I made an apparently too simplistic post about this shitty model that everyone is cheering (probably bots, I would like to see what they achieved with it). Link
Article · n8n
2.45.2 (2026-10-22) Bug Fixes core: Read type availability policies from memory at enforcement points ( #41621 ) ( 6433596 ) Postgres Node: Prevent global pgtypes overwrite ( #41666 ) ( eff4a45 ) Update logos in credential oauth confirmation screen and instance emails ( #41637 )
Article · n8n
2.44.6 (2026-10-22) Bug Fixes Postgres Node: Prevent global pgtypes overwrite ( #41667 ) ( 654315e )
Tested on an RTX 3060 with native Dawn WebGPU . llama-bench -m MODEL -p 512 -n 128 -r 5 -ngl 99 -dev WebGPU -t 2 -fa off Model Test master (t/s) PR (t/s) Ternary Bonsai 1.7B Q2_0 pp512 35.03 ± 0.46 1915.91 ± 3.36 Ternary Bonsai 1.7B Q2_0 tg128 14.02 ± 0.16 94.75 ± 0.20 Bonsai 2 2
Article · llama.cpp
vocab : use Tekken pre-tokenizer for pixtral rather than the Llama 3 one ( #30960 ) Website:
Article · llama.cpp
models : Record layer inputs for all granite models ( #30957 ) Branch: GraniteSpeculatorSupport AI-usage: full (Claude + Sonnet 5.5) Signed-off-by: Gabe Goodhart ghart@us.ibm.com Website:
Article · llama.cpp
kleidiai : add Q8_0 SVE dotprod GEMV and SVE i8mm GEMM kernels ( #30808 ) Website:
Article · llama.cpp
sycl: merge the fp16 and fp32 dequant switches ( #30740 ) Website:
Disclaimer: this project is vibe coded I'm not a programmer and I didn't check the code so to let you know! I want you to help me test the app for the arderino gpu and hexagon npu because I dont have a phone like that it require android 14+ and 2024 or newer snapdragon phones. I
I keep coming back to the 27B for coding. Curious if I'm the only one. I kinda want to know the most popular models for local inference, im working on an open source (Apache 2.0) project and it would be nice to know which models on for Apple Silicon should
Article · llama.cpp
webgpu : add Q2_0 support ( #30914 ) webgpu : add Q2_0 row dequantization webgpu : add Q2_0 matrix multiplication Website:
Article · llama.cpp
dflash : load optional FC input scale ( #30945 ) Assisted-by: Codex Website:
Is there any negatives that im missing here? Mia-AiLab/DeepSeek-V4.1-Flash-1017-EXL3-EAQ-v1-2.9-bpw
Article · llama.cpp
lora : support prism.hadamard folded weights ( #30910 ) lora : support prism.hadamard folded weights tests : remove LoRA test from test-llama-archs Website:
In the process of setting up the model and running local benchmarks on it on the official q4_k_m quant; it might work in Strata too. Reports soon. Anyone played around with this one yet?
I mean gpu's with 6/8/12GB vram. All discussion around llm/AI these days is around coding, and this is probably too small for agentic coding models. Are there models that can run in these that can accomplish useful tasks? eg things like helping organize your digital stuff, classi
Reddit from r/LocalLLaMA.
I recently got some more RAM on my desktop, which has opened up running larger MOEs by splitting them into RAM/VRAM. I know everyone loves Qwen flash next, but I'm curious what other ones are worth trying. On my list of to-try, I have: GPT OSS 120b (is this still usable nowadays?
Before telling me that I’ve lost my mind, yes, I have in fact looked at the market recently. I think I’d be able to get a significantly better deal, although still not cheap, and I’m not exactly sure what that would look like. I’m currently running qwen3.8–27B on a 3090+3060 12gb
I asked a while ago if it would be worth it to try and run qwen flash next at q2 since I can only really fit it and nothing bigger, but I can run Q8 of 27B. Everyone told me it’s either a toss up or Q8 would obliterate it. Not the case. Now this is subjective BUT for complex, ann
Manged to get Flint running in vLLM (with the flint PR) on my RTX 6000 Pro. The model was using my harness well, using OKF knowledge base, doing webscraping, spoke in the character I gave it, etc. I ran a couple of their benchmarks (45 of the 454 LiveCodeBench v6 problems and 50
We built and open-sourced Agent S4, which is an agent scientist evolving its own interfaces, letting an open 27B model beats GPT-5.5 & Opus 4.8. Most computer-use agents act through a fixed interface: screenshots, then click, scroll and type. So an agent fills a spreadsheet cell
I just downloaded Raon-1-mini 30B A4B as well as Flint 40B A2.5B and it just blows my mind that we have so many options now from so many countries. It feels like the countries are all working together and trying to contribute useful software and it’s just amazing. I can fucking T
Reddit from r/LocalLLaMA.
Article · llama.cpp
opencl: add F32-to-I32 ARGMAX ( #30897 ) opencl: add F32-to-I32 ARGMAX Assisted-by: OpenCode opencl: init to 0 to be consistent with cpu Co-authored-by: Li He lih@qti.qualcomm.com Website:
Hi Guys!, Recently I was making a research and development, but I do not have time to go further with this idea (kids, work, etc.). But maybe it will be interesting for someone. I do not think it would be something big, but maybe interesting approach that can be used in some of t
Fun fact the entire meme made by flash next in Rikkahub android app
KittenML has just released code and weights of Kitten-TTS 2, which is a super-efficient frontier TTS model, and the world's first 1.58-bit TTS model . It runs in real time on just a CPU, with zero-shot voice cloning, multilingual support, and expressive / emphasis control. Code a
Article · TLDR AI
Article from TLDR AI.
Hey everyone! I wanted to share Moose, an open-source Linux app I’m developing for chatting with models through Ollama. If you already run Ollama on another machine, you can connect Moose to it and use it from your desktop. Inference runs on your Ollama server, while your convers
This model looks really impressive. TL;DR -- Coding agent model. 40B params with 2.5B active. Resoundingly beats Qwen3.6-35B-A3B in TerminalBench 2.1 and SWE-bench Pro v2 (see HF repo for table), ties or betters it in the other benchmarks they chose to show. 3:1 SWA:full attentio
Article · llama.cpp
models : add Flint model (FlintForCausalLM) ( #30928 ) Add Flint model (FlintForCausalLM). Assisted by Codex cleanup one more Co-authored-by: Sigbjørn Skjæret sigbjorn.skjaeret@huggingface.co Website:
AbliteratedOutTheBox is a llama.cpp fork that abliterates any model you load, live at inference, with no fine-tuning and no extra files. This eliminates any hassle of abliteration without requiring any more vram.
Article · llama.cpp
jinja : fix namespace attributes shadowed by dict methods ( #30941 ) Website:
Article · MarkTechPost
Vercel released durable-harness, an open-source package that logs every model and tool call so Pi, OpenCode, Claude Code and Codex sessions resume after crashes, timeouts and redeploys on Vercel, Postgres or a laptop. The post Vercel Open-Sources durable-harness for Crash-Proof C
Article · llama.cpp
model: Add support for SenseNova U1 model ( #28919 ) model : add SenseNova U1 understanding support Add support for the dense SenseNova U1 understanding branch, including the U1.5-8B-MoT model. Add HF-to-GGUF conversion for NEOChatModel, excluding MoT generation tensors and conca
Article · Ollama
Recursive Flint Flint brings Recursive's coding model to Ollama. It is designed for the work coding agents do across a repository: understanding unfamiliar code, tracing bugs, implementing features, and adding tests. Reasoning, multi-turn tool use, and a 262K-token context window
I ran 10 agent harnesses on the same local model and the same 5 small tasks, mostly to see where a 4B actually falls over. The scores turned out less interesting than the way things failed. Setup: Qwen3.5-4B Q4_K_M, one llama-server (64K context, 1 slot, f16 KV, --jinja), M5 MacB
An abliterated variant of the original MiniCPM5-2B-SFT One notable behavioral change is its tendency to rationalize potentially harmful or restricted questions rather then reject but thats a 1/20 chance
now you can use Flint in llama.cpp (merged after 6h)
Article · llama.cpp
llama: create new fused op for QSA mask ( #30640 ) llama: create new fused op for QSA scoring enroll glm5.3-next too tests : initialize the remaining tensors in test_kpool_mask The harness sentinels were left uninitialized, so leftover memory could read as NaN and fail the NaN ch
My MS-02 runs on rocks I
I always feel a little bad asking LLMs to do research into itself or newer versions. It's like asking someone to face their own mortality, although that's anthropomorphizing. The user is asking me to investigate the system requirements for "Qwen 3.8 Flash Next" from at least 20 d
Article · Ollama
We were still using a pre-release, and the download site is flaky, so lets get current on the latest CUDA Toolkit for WoA support.
Hey guys! I'm the dev from the open-source Observer project. We made this video about how even if the AI bubble bursts, local model advancements are bubble-proof. Hope you guys like it :DD Subscribe to the youtube if you liked the video! We'll post more local AI content there c:
Created using Minimax H3 + Krea 2 + Qwen Image Edit 2.1
Reddit from r/StableDiffusion.
it's time to update your llama.cpp for another speedup (sparse attention models)
I'm sharing Mica v0.5, the follow-up to Mica v0.1. It's 4B and runs on one consumer GPU. On speed, it takes 19.8 ms per decision at 256 tokens and about 1.2 s at 32k on a single RTX 6000 Ada. That's up to 3.8x faster than the other 4B judges I measured, 12 to 19x faster than Perp
Hello everyone, Going over some posts here, a lot of people keep referring to 128 GB RAM as their minimum for hosting local AI and anything below that is not worth it. Why is that? Is it because of Qwen Flash Next, is the model that good and is it able to do anything you would ne
I saw this today on X and decided to try it, I havent seen it posted here yet. I ran it on a single 3090 benchmarked it with qwen 3.8 27B before and after the optimization and saw these results: Observed prompt-processing throughput improved 22–30% ; generation speed improved 1–5
I bought a used PC with an RTX 3090 24 GB, 128 GB DDR4 and an Intel i9-9900X last year for under €2,000. With the release of DeepSeek-V4.1-Flash-1017 I decided to try running it on my machine. It is of course the smallest quant available (1-bit), but I was still happy to see it w
Article · noted.lol
Built for teams that need to keep feedback in-house, RetroGemini brings guided retrospectives, anonymous voting, and health checks to your server.
The model card has a pretty noticeable result: 76.9 on SWE-bench Pro , compared with 43.9 for Qwen3.8-Flash-Next in their evaluation. On Terminal-Bench 2.1 it's slightly behind, at 81.2 vs 83.4. These are the team's own results, but the difference between those two comparisons is
'm genuinely curious about the limitations people are facing with local AI today. We have Ollama, LM Studio, Open WebUI, Jan, and increasingly capable open-weight models. Yet I still see people struggling with tasks that seem relatively straightforward. What's one real-world task
I've been working on MotifAI, a project exploring how local AI models can be harnessed for music composition in ways that go beyond the usual workflow. The idea is to put local models to work creatively, experiment with different approaches, and explore what's possible when you t
China Is Doubling RAM Production to Keep Everyday Gadget Prices from Skyrocketing" title="When will we see these changes? --> China Is Doubling RAM Production to Keep Everyday Gadget Prices from Skyrocketing" />
Article · llama.cpp
ggml-opencl : add IM2COL_3D support ( #30895 ) ggml-opencl : add IM2COL_3D support Assisted-by: OpenCode opencl: use same type for kernel and host opencl: fix for older adreno Co-authored-by: Li He lih@qti.qualcomm.com Website:
I don't have a 3090 to confirm, but curious if the anyone in this community can back this up. Good win for 3090 and curious if that translates to other GPUs. I have a long vague memory of some hacks done to prevent mining demand that could be fixed and help LLMs. I wonder if this
I held off as long as I could, but finally gave in and set up strata. I am pretty blown away. 3.8 27b at Q6 kv8 was my driver before and that works super well, but Flash Next in strata IQ4 xs is really winning me over. Its not perfect, but it feels like models I was using profess
While using opencode with a local qwen3.8 instance, I asked gemini if opencode collects data or not, answer was yes. By default it does, send chat to it's server for getting title, compaction etc. - Is it still true? I couldn't find this exact issue mentioned in the sub, but peop
Been working over the last few days getting DeepSeek V4.1 Flash 1017 (Unsloth UD-IQ1_S) running in Basalt (inference engine I'm developing). Specs: RTX 5090 (400W) + RTX 5060 Ti 16 GB (180W), 128 GB DDR5. Methodology: Single 243 GB file (weights + vision + drafter). Engram table
I've seen several posts about Ming-3.0-Audio-Tiny, but haven't seen much discussion about the TTS model released alongside it (maybe I missed something). In case anyone missed it, InclusionAI (backed by Ant Group) also released Ming-3.0-TTS-0.5B , supporting voice cloning, voice
#Alert So it's not only PHP that was vulnerable today (patch yours NOW!) but also, one of the prominent and widely used LLM Gateway - LiteLLM AI Gateway was (again) backdoored. Not sure who discovered the backdoor, but it's serious. Here's the information: Version: LiteLLM 1.107.
Article · Ollama
Preserve source tool JSON alongside decoded API values so Flint can retain numeric types and schema order in prompts. Existing API marshaling is unchanged. Assemble ModelOpt expert banks during safetensors import, and reject direct Flint NVFP4 creation before conversion. Other mo
Reddit from r/StableDiffusion.
I came across OpenBMB’s engineering write-up on running large MoE models locally, and the systems work behind it is worth highlighting. Instead of shrinking the model or skipping experts, they tackled the real bottleneck: getting expert weights from flash storage to computation f
Article · n8n
2.45.1 (2026-10-21) Bug Fixes core: Claim an idle worker's next queue job and its lock in one step ( #41558 ) ( 5f0b562 ) editor: Center badge text when icons are absent ( #41581 ) ( 648fa2a ) Microsoft Teams Trigger Node: Renew the Graph subscription via lifecycle notifications
Hi! I'm the creator of HF Viewer, and we are today launching embedl.com ! It's a collection of 1000+ models optimized to run fast on different edge devices like NVIDIA Orin Nano Super, AGX Thor, AGX Orin, Orin NX and RTX GPUs. You can run the models directly on real edge devices
Article · llama.cpp
server : forward llama.cpp sampling params on /v1/messages ( #30856 ) server : forward llama.cpp sampling params on /v1/messages server : pass the request through on /v1/messages instead of copying listed keys NIT: Change comment from multiline to single line server : accept llam
Reddit from r/comfyui.
Memory-native evolution requires making memory an intrinsic capability of foundation models, enabling them to continuously accumulate and utilize information. To this end, we introduce MemT-1.0, a new architecture of memory foundation models that internalize memory as part of mod
Article · llama.cpp
llama : optimize vocabulary loading ( #30679 ) llama : optimize vocabulary loading Update src/unicode.h Co-authored-by: Georgi Gerganov ggerganov@gmail.com CR : added reserve in gguf_kv_to_str CR : trailing space fix Co-authored-by: Georgi Gerganov ggerganov@gmail.com Website:
Expert human readers process large inputs (e.g., code repositories and long documents) by skimming to closely read only what matters. On the other hand, language models process all input tokens through each model layer, resulting in significant processing costs for long-context i
Article · llama.cpp
llama : place d2t by its set_rows use so it can stay on the GPU ( #30912 ) Website:
I’m a doctor looking to buy a Mac for running local LLMs, mainly through LM Studio. My work is primarily in Danish. I’m considering a 16-inch MacBook Pro with M5 Max, 48GB unified memory and a 2TB SSD . My main tasks would be:
Article · llama.cpp
metal : fix memory leaks and cleanup logic in GPU capture path ( #30890 ) Website:
Article · LiteLLM
Verify Docker Image Signature All LiteLLM Docker images are signed with cosign . Every release is signed with the same key introduced in commit 0112e53 . Verify using the pinned commit hash (recommended): A commit hash is cryptographically immutable, so this is the strongest way
Article · llama.cpp
hexagon: neon-vectorized and threaded repack (tiling) ( #30901 ) add vectorized q4_0 repack apply to more types hex-repack: neon-vectorize the rest of the types q1/q2/f16/... hex-repack: add simple chunking / threading Co-authored-by: hubezhou hubezhou@qti.qualcomm.com Website:
Don’t let the cute little cabinet fool you. From the outside it looks almost empt
Article · noted.lol
Quomodo brings a central web interface to Podman stacks, with Git-based deployments and rootless container support.
Article · llama.cpp
opencl: add dp4a GEMM variants to vector load activation ( #30753 ) opencl: add dp4a GEMM variants with vector activation loads opencl: rename kernels and env vars Website:
Article · llama.cpp
sycl : fix fa crash for Query head dimension mismatch (neq0 != HSK / D=2048/4096) ( #30286 ) Website:
Mixture-of-experts (MoE) models often exceed a single accelerator's memory capacity. Offloading experts to host memory creates a severe PCIe I/O bottleneck because data-dependent routing keeps transfers on the critical path. We present Specter, an inference system that uses quant
Article · LiteLLM
Verify Docker Image Signature All LiteLLM Docker images are signed with cosign . Every release is signed with the same key introduced in commit 0112e53 . Verify using the pinned commit hash (recommended): A commit hash is cryptographically immutable, so this is the strongest way
have you experienced any related issues like bandwidth or something else? Excluding the fact that it cost way cheaper than ddr4 rams. If there's anything else id love to hear
Article · llama.cpp
sycl: collapse duplicate need_check launch bodies in mmq ( #30739 ) Website:
Reposting again since original post got removed by Reddit for some reason (I have no idea why) A few of you pointed out in my last post that comparing Flash-Next and Opus with different harnesses and effort levels wasn't fair. So here is a more controlled comparison. Same task fo
Article · LiteLLM
Verify Docker Image Signature All LiteLLM Docker images are signed with cosign . Every release is signed with the same key introduced in commit 0112e53 . Verify using the pinned commit hash (recommended): A commit hash is cryptographically immutable, so this is the strongest way
OvisOCR3 is out, still based on Qwen3.5-0.8B and available under Apache 2.0, with vLLM inference examples. Compared with OvisOCR2’s Markdown parsing, it adds a few useful options: Structured JSON output with bounding boxes, element categories, and content, plus a layout-only mode
Reddit from r/LocalLLaMA.
Reddit from r/LocalLLaMA.
Reddit from r/ollama.
Article · llama.cpp
ggml-cpu : compute f32 gelu_quick and geglu_quick without the fp16 table ( #30737 ) ggml-cpu : compute f32 gelu_quick and geglu_quick without the fp16 table ggml-cpu : keep the GGML_GELU_QUICK_FP16 table path ggml-cpu : add the VXE gelu_quick path tests : add test-gelu-quick for
Raon-1 mini is a 30B-A4B Mixture-of-Experts (MoE) language model, designed and trained from scratch by KRAFTON AI . It demonstrates strong capabilities in Korean conversation and agentic tasks . As part of the Raon open model family, it is released under the Apache 2.0 license .
Reddit from r/LocalLLaMA.
Article · n8n
2.44.5 (2026-10-21) Bug Fixes core: Key queue job outcomes by job on shutdown resume ( #41404 ) ( 1f074d4 ) editor: Load credential types on startup ( #41405 ) ( 6ff3198 )
Fits entirely in a 5090, no offloaded experts. Super-solid sub-2bpw! ( Liquid cat demo video generated by the quantized model) From the Cat's mouth: Stable inference throughout, no reasoning loops, just smooth sailing at 200-300 t/s (and a lot of compacting since one 32 GB GPU st
Article · LiteLLM
fix(llms): sanitize trailing slashes in provider get_complete_url test(llms): move trailing-slash url tests into each provider's mapped unit file fix(llms): strip trailing slashes in Perplexity search and Ollama embedding URLs and run the Topaz tests in CI fix(llms): strip traili
Article · llama.cpp
llama : scale moe down projection per expert token ( #30861 ) llama : scale moe down projection per expert token Replace the global amax rescaling for the MoE down projection with a per-expert-token factor computed by a POOL_1D max reduction along the intermediate dimension. Undo
Samsung sells out 100% of all 2027 HBM capacity - HBM prices expected to jump 200% next year, DRAM prices expected to see a double-digit price increase" title="Be prepared, GPU price increase incoming --> Samsung sells out 100% of all 2027 HBM capacity - HBM prices expected to ju
Article · LiteLLM
Verify Docker Image Signature All LiteLLM Docker images are signed with cosign . Every release is signed with the same key introduced in commit 0112e53 . Verify using the pinned commit hash (recommended): A commit hash is cryptographically immutable, so this is the strongest way
I'd like to share a few changes that encourage people to create and use LoRA on LLMs. For example, the LoRA
Strata is the new hotness - claims of 4x tok/s increase over llama.cpp. I havent seen any posts talking about performance on Apple silicon however. Curious if anyone's run comparisons on Apple Silicon against llama.cpp and MLX
pplx-decider-v1.2 is one tenth the costs of GPT 6 Luna (low reasoning) while making identical quality decisions, and pplx-decider-flash-v1.2 idem ditto compare to 6 Luna in the Decisions API. v1.2 is also 2.5x cheaper than Jev.
Article · llama.cpp
server : fix usage.completion_tokens undercount with n > 1 ( #29486 ) Assisted-by: Copilot Website:
hey folks, its Ye, I work on an open source project called Kapitsa : its a fast Apple Silicon optimized inference engine built for Qwen and Nemotron models. this week two new releases landed in local inference: Inco shipped Splash 1.4.0 yesterday, and we shipped Kapitsa 0.2 today
Article · llama.cpp
server : add POST /tts endpoint for TTS models ( #26603 ) mtmd: add audio out stream api add /tts endpoint add docs wire up mtmd_helper_model_can_chat upload speaker_ref via form-data expose seed to api fix header for pcm wip fix penalty_last_n fix server_output_limits fix pocket
Article · llama.cpp
server: add realtime transcription API ( #30636 ) server: add realtime transcription API drop transcription_sessions proxy ws add docs server: let the API key middleware guard the websocket httplib runs the pre-routing handler before the websocket upgrade, so the API key middlewa
Article · Ollama
What's Changed pplx-decider is a 27B decision model from Perplexity, fine-tuned from Qwen3.8-27B Continuing an OpenAI-compatible Responses conversation no longer fails when switching between models with and without thinking, or when the history contains another provider's compact
I’ve seen vastly more questions than answers about how people are using their local AI so I figured I’d make a post about my experience. It seems most people are looking for frontier-level coding expertise for huge projects with their setups, and that anything less is not enough.
Article · llama.cpp
qwen: reshape output to 2d instead of 3d ( #30823 ) qwen: reshape output to 2d instead of 3d remove final reshape, fix comments Website:
Article · llama.cpp
server: Support logprobs with speculative decoding ( #27196 ) return logprobs in spec path server: add regression test for logprobs with speculative decoding Asserts accepted draft tokens report the target model's logprobs, comparing a draft-enabled run against a no-draft baselin
Article · llama.cpp
mtmd : lfm2 tiles and layout match transformers ( #30261 ) mtmd : lfm2 tiles stretch the image to the grid Lfm2VlImageProcessor.crop_image_to_patches resizes the image straight to (512 * rows, 512 * cols) before it cuts the tiles. llama.cpp kept the aspect ratio and added black b
Article · llama.cpp
model : add H2O-Lightning-4B decision model ( #30361 ) model : add H2O-Lightning-4B decision model Adds the h2o-lightning decision type for h2oai/h2o-lightning-4b (Qwen3.5-4B with a merged decision LoRA). None of the existing types match this model. It answers with " A" rather th
China just ordered 3mil RTX 5500 Pro for $11.4k a pop. Probably for llm inference and AI video gen. I suppose this card is now out of reach for local llm.
Article · llama.cpp
server: Router wasn't propagating the client disconnects fully to the child, the response wait could cause a delayed cancellation ( #29707 ) Router wasn't propagating the client disconnects fully to the child, the response wait could cause a delayed cancellation tests: satisfy ro
Jesus, continuous loops, it tramples himself over and over. I also applied the fix found here:
For last 9 months, been using cloud frontier 99% the from Gemini Flash Pro, Claude, Deepseek, Groq & Cerebras. Ready to give 32B, 70B, 120B another try. Was getting decent 45 token/sec on 7B Llama MLX, but 7B is too simple for most task. I loaded Gwen 32B, get some errors on LM
Article · llama.cpp
server: fix missing output_index in Responses API SSE events for compatibility ( #30863 ) server: include output_index in Responses API SSE events server: reuse Anthropic streaming code for fix output_index server: remove unnecessary generated_tool_call_ids and oai_resp_tool_call
Reddit from r/ollama.
Article · llama.cpp
mtmd : read LFM2 tiling params from GGUF metadata ( #30840 ) mtmd : read LFM2 tiling params from GGUF metadata Write min_tiles, max_tiles, tile_size and max_pixels_tolerance from processor_config.json when converting LFM2-VL mmproj, read them in clip.cpp with the previous constan
I'm glad to announce MuckScraper 1.0. It's a personal project I've been building for about six months, and it's finally out of beta. I posted it here a few months ago and a lot of your suggestions were a great help in getting it here. The name is a nod to the old muckrakers. It d
Article · llama.cpp
Newer versions of Clang diagnose an incompatible pointer type error when assigning a pointer to float to a pointer to half.
Article · llama.cpp
model : drop the redundant precision overrides in the Kolibri-1 router ( #30865 ) model : drop the redundant precision overrides in the Kolibri-1 router Follow-up to the review in #30673 : the router matmul works with the default precision, so both ggml_prec_set_* calls can go aw
Insane numbers, so far only Qwen3.8-27B is supported but wow those are some good numbers. Will watch closely when this lands in llama.cpp/vLLM ### Qwen 3.8 27B Benchmark Results Method Speedup over Vanilla MTP 2.22× DSpark 2.57× DFlash 2 2.80× EAGLE-4 3.59×
Article · llama.cpp
opencl: add several gemm variants for small batches and some fixes ( #30850 ) opencl: fix OOB read in mul_mv_mxfp4_f32_flat opencl: remove large-N guard that forces CPU weight placement this condition claims that large batch produces incorrect results on adreno and cannot be repr
I tested 10 agents, each with its own isolated settings folder. I ran them headless, with auto-confirmation and turned off telemetry where possible. I launched each agent with adapters/ .sh . The idea was simple: give each agent 5 basic tasks, 10 minutes per task, a fresh copy of
Article · llama.cpp
ggml-cpu : select the NEON fp32->fp16 conversion ( #30505 ) The _ ARM_NEON branch defines both GGML_CPU_COMPUTE * macros but selects only FP16_TO_FP32, so FP32_TO_FP16 falls through to the portable software conversion while neon_compute_fp32_to_fp16, a single fcvt, stays defined
I had 9t/s on my rig, with Unsloth 3bit Qwen 3.8 Flash Next. Then I tried STRATA with CachyOS, because I needed something with a ready NVIDIA driver, kinda pain getting the latest NVIDIA driver to work. Cachy just does it for you, anyway, I hit up to 48t/s sometimes... its kinda
Article · llama.cpp
HIP/CUDA: Fix reduction in mmf getting tile widths not devisiable by 64 ( #30836 ) Website:
Reddit from r/StableDiffusion.
Article · llama.cpp
ggml-zdnn: introduce flash attention op ( #30512 ) ggml-zdnn: add scratch buffer pool Signed-off-by: Aaron Teo aaron.teo1@ibm.com ggml-zdnn: implement fattn, ditch scratch slots in favour of scratch buffer pool Signed-off-by: Aaron Teo aaron.teo1@ibm.com docs: add note about fatt
Article · llama.cpp
refactor : move LFM2 shortconv and FFN blocks into a shared LFM base …
Article · Ollama
Deduplicate MLX CUDA runtime payloads Move CUDA libraries shared by the GGML and MLX backends into the sibling CUDA payload, and teach in-process and subprocess MLX loading to resolve that packaged dependency directory. Keep MLX-private libraries and JIT headers in the MLX overla
Reddit from r/StableDiffusion.
If you have an RK3588 board, here’s another way to put its NPU to work. I made open_rknpu2_lm_infer, a local LLM inference engine for the Rockchip RK3588 NPU. It runs without Rockchip’s proprietary userspace inference libraries. The upper part is written in Rust and the NPU opera
We just open-sourced FastH3 OmniRef from the FastVideo team at Hao AI Lab, and it works in ComfyUI from day one. Also available in the last release 0.42.2 workflow templates - Feed it a prompt plus your own references (images, short video clips, a voice) and get a finished video
Article · llama.cpp
CUDA: Clear CUDA graphs after legacy memory pool flush ( #26771 ) Assisted-by: Claude Opus 5.5 Website:
Reddit from r/LocalLLaMA.
Article · LiteLLM
Verify Docker Image Signature All LiteLLM Docker images are signed with cosign . Every release is signed with the same key introduced in commit 0112e53 . Verify using the pinned commit hash (recommended): A commit hash is cryptographically immutable, so this is the strongest way
I've spent about a year on an inference engine that doesn't need modern hardware. It's called Pantheon (LLM engine) on top of Foundry (ML library), both in C with no outside libraries. The CUDA backend targets Kepler ( sm_37 ), so a Tesla K80 you can buy used for very little does
Article · LiteLLM
Verify Docker Image Signature All LiteLLM Docker images are signed with cosign . Every release is signed with the same key introduced in commit 0112e53 . Verify using the pinned commit hash (recommended): A commit hash is cryptographically immutable, so this is the strongest way
Just rambling. I've been using computers for over 40 years and consider myself a Windows and macOS pro, with decent network skills, but have only been hardcore on Linux since January. Since then I've learned how to set up numerous server applications and have been gradually worki
Article · llama.cpp
hexagon: fix VTCM size query on images that need layout out-params (#…
Reddit from r/LocalLLaMA.
Kyutai released three smaller Pocket TTS models on October 20: 1.9M, 6.9M and 18.2M parameters . These are English models, support voice cloning, and are designed to run on CPU. You can switch between the sizes in their demo here . The useful part for me is keeping speech generat
Reddit from r/LocalLLaMA.
Article · llama.cpp
server : avoid repeated mmproj memory margin addition on sleep/resume…
just noticed in my reasoning threads that seemingly its favorite word is now "hmm": Hmm!!! What is that 1? Hmm - it could be the has_value/nullable marker for the MQDescriptor parameter! For in MQDescriptor q (non-null), the NDK client writes writeInt32(1) before q.writeToParcel(
Article · noted.lol
Petal lets remote collaborators share movable desktop windows and run their own media server, although setup is still hands-on.
At Gradium we just made our first open-source model release with Phonon, a collection of small Text-to-Speech models designed to run on device. The inference stack is fully open-source and targets 10+ different backends (CPUs, GPUs, NPUs), most of them in Rust. We hope that the c
EDIT: Note that I built off the official vllm documentation and I just got done validating the system. This is now my control. That means this is a slow and stable config. Now that i have a baseline control I’m going to begin tweaking settings for performance runs. I’ll make post
A few of you pointed out in my last post that comparing Flash-Next and Opus with different harnesses and effort levels wasn't fair. So here is a more controlled comparison. Same task for every run: a real feature from the LlamaStash roadmap (save llama-server's KV cache before an
Article · MarkTechPost
DeepSeek has open-sourced DeepInfer, the Apache 2.0 inference framework that serves DeepSeek-V4.1-Flash in production. It targets NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs, and ships offline inference only. The post DeepSeek Open-Sources DeepInfer, the Inference Framework Behi
from Ornith: Ornith-1.5-122B Chirp Chirp! 🐦 We are introducing Ornith-1.5, a major step toward building foundation models through end-to-end self-improvement. Ornith-1.5 extends Ornith-1.0 (which was developed on top of Qwen3.5 and Gemma4 with additional continued pretraining, mi
The scaling relationship of compute and performance for language models (LMs) has been extensively studied; more recently, models are being designed to scale memory use, as well. In particular, adding embeddings for n-grams scales memory separately from compute use, because each
Long-context training of large language and multimodal models couples structured attention masks (document, prefix-LM, share-question, sliding-window, block-sparse) with multi-node context parallelism, making the mask a dominant bottleneck. FlashMask's column-wise sparse mask com
Article · llama.cpp
server : pass id_slot through the Anthropic /v1/messages conversion ( #28554 ) server : pass id_slot through the Anthropic /v1/messages conversion server_chat_convert_anthropic_to_oai copies a small whitelist of common params from the Anthropic request into the internal OAI-style
Long-horizon agents compact their history to continue within a finite context window, but a textual summary alone may not support every subsequent decision. We introduce REMORY, a neural memory network that supplements the summary with a bounded sequence of soft memory tokens. Gi
Speculative decoding has become a mainstream technique for accelerating large language model (LLM) inference by using a lightweight draft model to generate candidate tokens that are subsequently verified by the target model. However, agentic workloads introduce long and continuou
Geezuz, its fast! Qwen3.8-27B, two RTX 3090s power-limited to 280 W, plain PCIe, no NVLink, at 0 Context: (tests showing results with context are found on the next table) Requests Q4_K_M Q8_0 1 request 190 tok/s 143 tok/s 8 requests at once (total) 622 tok/s 610 tok/s llama.cpp's
Article · llama.cpp
metal : optimize argmax and pool_1d Add a two-pass partial reduction for large-row argmax and use cooperative threadgroup reductions for pool_1d. Allocate scratch space for the argmax partials through the Metal buffer allocator. Assisted-by: pi:llama.cpp/ggml-org/DeepSeek-V4-Flas
Hi all, just wanted to share some benchmarks. I got Unsloth's UD-Q4_K_XL quant of DeepSeek-V4.1-Flash-1017 running and spent some time tuning it. Some of what I found isn't in the guide, so here are my notes. Hardware EPYC 9575F (64 cores / 128 threads) 768 GB DDR5 4x RTX 3090 (9
as far as hardware we currently have options such as nvidia, amd, apple, etc. however it feels like these are more on the expensive side. personaly i own bosgame m5 strixhalo and so far its good. i was thinking of upgrade, but it seems to me that the next step will be chinese har
Article · llama.cpp
spec : save the dynamic ngram cache file save the cache at the common_speculative_impl_ngram_cache destruction remove save_static/save_dynamic flags as well as the relevant TODO if the dynamic cache file does not exist, start with an empty cache that is written on exit. if it exi
The Kromoss model was updated today, it's worth replacing yesterday's version:
I've been working with the lemonade server via the CLI a lot recently and found myself discovering and returning to a few particularly useful commands for pulling, configuring and benchmarking models. So I wrote an article about them, if I've missed your favourite let me know. su
Article · llama.cpp
metal : fix missing MoE kernel variant for 9 experts per token ( #30610 ) ( #30699 ) Website:
Some time ago in this same place someone had suggested having your model create paintings through python (with programming) leaving it quite free. Doing so I had noticed how the model, after creating the first draft, made visual checks and quickly decided what he didn't like, add
I've been using the 27B series for a while now. It's so good, I don't even use a custom chat template and barely ever saw tool call failures . My latest model & engine of choice were ukisai/Swift-1.5-Qwen3.8-27b-NVFP4 running on NInfer . Recently I began experimenting with QFN, m
Hello hello 👋 , first post here, and it's self-promotion, so feel free to ping me to remove it if it violates rule 4. Sorry in advance. A friend and I started Gufo as a pet project (MIT license) to get the most out of the AMD Strix Halo chip, while supporting a few best-in-class
Article · Immich
Turns out the last people performance fix wasn't quite there for some people. We have now finally resolved that issue. What's Changed 🐛 Bug fixes fix: missing translation by @waclaw66 in #32485 perf(server): more efficient person queries by @mertalev in #32523 Full Changelog : v3
This demo from ISTA-DASLab recovers low quant accuracy while retaining low VRAM usage and fast decoding. It needs a Nvidia Blackwell GPU to run, so I have not tested it. For context longer than ~4k there should be no speed penalty compare to running without the prefiller model. G
This is crazy good for studying to improve existing custom engines by community like gufo, halogen, strixite, Strata... by learning their secret sauces.
Article · n8n
2.44.4 (2026-10-19) Bug Fixes core: Reuse connections in the log streaming webhook destination ( #41281 ) ( b79afe2 ) core: Turn off expression spans by default ( #41309 ) ( 6965193 )
Article · n8n
2.45.0 (2026-10-20) Bug Fixes AI Agent Node: Replay Anthropic thinking turns with their tool arguments ( #41241 ) ( ea98bd0 ) Align data table and variable editing on protected instances ( #41258 ) ( d4bde14 ) Allow skipping table initialization ( #40437 ) ( d245b17 ) API: Apply
Article · llama.cpp
vocab : add \p{M} to lookaheads in Tekken and GPT-4o pre-tokenizer regexes ( #30845 ) Website:
I built a set of templates around ComfyUI's native YuE2 and MiniMax Music 3 nodes and would like feedback from people who use them. The YuE2 Song graph: A Song Brief goes to a local LLM writer (Gemma 4 through ComfyUI's own Generate Text node, or any GGUF / LM Studio / Ollama mod
I'm about to buy a used HP Z640 for around €550. It has 2x E5-2680 v4, 64GB ECC at 2400, and a Quadro M6000 24GB that comes with it. The seller’s MemTest86 ran 6 hours with no errors. The plan is to add two 3060s in the x16 slots and run Qwen 3.8 Flash Next on Strata, as well as
Article · llama.cpp
tests : make MoE router selection independent of the backend noise ( #30818 ) tests : make MoE router selection independent of the backend noise The router scores are in the range [0, 1), so a random draw can leave the last expert (or expert group) still selected and the first on
Article · MarkTechPost
Cloudflare has previewed Clairic, a System One extraction model on Workers AI that returns typed values with confidence scores in one forward pass. It costs $0.10 per million input tokens and charges nothing for output. The post Cloudflare Releases Clairic, a System One Extractio
Today I updated my ComfyUI, and it messed up, so I had to redownload it. I use potable. The thing is it's not the same one I use. I got all this other stuff I don't use. Can someone show me how to get the old ComfyUI back? This new one makes no sense; nothing works right. submitt
Hi there, I'm experimenting with Minimax H3 ref2va (DaSiWa). I'm trying to replace a character in a video. So I have a video source for motion, etc and an image source for identity. But it always takes too much of the original video and only really replaces parts that are not see
Article · llama.cpp
hexagon: update to snapdragon-toolchain v0.8 and enable hexagon v85 ( #30854 ) hex-build: bump toolchain, enable v85 hex-build: bump toolchain to v0.8 in the CI and build scripts hex-build: add a little python script for swiv needed on some iot devices hex-build: fix flake8 warni
Title: Which model and ComfyUI workflow would you recommend for reference-based image generation on 16GB VRAM? Hi! I’d like to use the attached image as a reference to generate new images with different poses, clothing, backgrounds, and compositions, while keeping the character’s
I wanted "a little more empty" without changing the prompt or the seed. Prompts can't do that: when I blended "packed room" into "empty room" with ConditioningAverage, nothing changed from 0.9 down to 0.5, then the whole room flipped at 0.3. The usual answer is training a LoRA. H
Article · llama.cpp
sycl: fix compiler warnings - missing prototypes, dead code, extra semicolons ( #30271 ) Website:
So for the fun i tried downloading the portable version from github 0.42.1 extract and fired up the fast 16 picked a template of wan motion transfer and got all the models installed and into their respective folders so far so good . I hit run and it just fails either with no erro
I know this kinda test is hardly relevant anymore but anyhow, just want to see what we have to lose with turbo lora. Using some modified random prompts from civitai (yes my prompt game is super weak). Hope I'm running this correctly as the original workflow from moss has "flow si
I recently upgraded my Debian WS from 9060XT to 2x R9700. A positive side effect of splitting a model by layer across the cards is that (with single task usage) the GCPU load of each card drops to
Hi HN, I'm Peter Bishop and I've been working in graphics and web tech for over 25yrs. I've recently launched Death Plan Switch to solve a major problem in my succession plan. What if I go missing or die? The right people are told and my business can continue to operate for my cl
Article · llama.cpp
SearchInput already renders its own clear button. Hide the browser's native search cancel button to avoid showing both controls. Fixes #28627 Signed-off-by: aiqubits aiqubits@hotmail.com
aa has this new Frontier Code v1.1 bench which measures Agentic Software Engineering. I see that Kimi K3 does very well in this new bench while performing badly in Terminal Bench v4. What are the differences between these two benches?
Reddit from r/StableDiffusion.
System service that watches your card, swaps and loads whatever you want, whenever you want. Don’t have to swap anything manually, collisions avoided, VRAM fully dedicated to each function while it’s running (no VRAM squatters). I had issues everyday (Linux) where a node process
Article · LiteLLM
Verify Docker Image Signature All LiteLLM Docker images are signed with cosign . Every release is signed with the same key introduced in commit 0112e53 . Verify using the pinned commit hash (recommended): A commit hash is cryptographically immutable, so this is the strongest way
Looking for advice on upgrading my AI server on a budget. My goal is to run Gemma 4 31B Q5_K_M (~23GB) at 16K+ context (q8), achieving at least 15 t/s. I am aware of the 26B MoE, but I want the dense model after extensive testing. Current setup: --> CPU: Intel i5-14400F --> Mothe
I was buying a Mac mini and kept going back and forth between configs, trying to guess which one would actually run the models I want. So I built a configurator for people buying a Mac to run local LLMs:
Article · LiteLLM
Verify Docker Image Signature All LiteLLM Docker images are signed with cosign . Every release is signed with the same key introduced in commit 0112e53 . Verify using the pinned commit hash (recommended): A commit hash is cryptographically immutable, so this is the strongest way
Hey, everyone! I'm so happy to finally share this with the community. I started VoiceManager way back in March to support a game concept and my AI assistants. I had the idea for this Trek inspired app to build pieces for a future project. Low latency streaming speech to speech ru
Just a heads up, especially on my Windows machine, but also my Linux machine I'm getting tons of breaking changes with Strata. They added a bunch of needless requirements around serving, the web UI seems to be going way in favor of a desktop app. I could go on but you get the pic
MOSS seems to be the hot model since it came out 4 days ago so I went I made a proper architecture diagram from the official code. MOSS 5B architecture diagram Find the diagram and its source code here , with some additional information about that model that I've found while goin
When I generate an image using Krea2 for the first time I opened the program(I am using comfy UI through Krita) it generates the image quick. However, when I type in another prompt (or add or replace a word) and generate, it will either go really show or wont generate at all. I d
Article · llama.cpp
sycl : fix get_rows on reordered quantized weights ( #30430 ) Co-authored-by: Sam sam@garagellm.tail7dd858.ts.net Website:
For those with big media servers: do you keep a second copy of your media? My Jellyfin library is small about 10 TB with 10 more to fill. Every time I get more storage, I plan to use it for backups and end up filling it with more media instead. My thinking is that if a drive dies
Reddit from r/LocalLLaMA.
Article · MarkTechPost
Nace AI released Drex Max, a multimodal decision model that reads text and screenshots and returns calibrated probabilities. Nace reports 67.59 on Decision Index 0.3.2 at $0.06 per 1M input tokens. The post Meet Drex Max: Nace AI’s Multimodal Decision Model for Computer-Use Agent
Article · Hugging Face
Article from Hugging Face.
Ok, so this model has been praised (with reason) for its capability to make images with transparency, which is amazing if you need clean sprites for videogames, visual novels and similar projects. Now, a pretty funny (and idiotic) thing is that the default model used to enhance p
Article · llama.cpp
server: add --connect, allow p2p access to server outside LAN ( #28277 ) init wip wire it up make it clear that it's experimental add props.connect_code nits fix windows build server: do not advertise a connect code without the tunnel --connect-code on its own had no effect other
Article · llama.cpp
hybrid dna tokenizer fixes ( #30825 ) vocab : support Carbon-14B HybridDNA tokenizer Carbon-14B auto-tags untagged DNA runs and uses N-padded tail tokens (tail=pad_n). Also strip whitespace inside tags and handle a stray like the HF tokenizer. convert : fix special tokens and mer
Hi, I experimented with the following idea, using JEV logic to make the thinking process of LLMs adaptive. For every single word we measure the entropy and when the model is unsure we use a JEV like gate where we ask the model if it should trigger a thinking process. We have the
I had a gut feeling that this model would be incredible at inpainting, and after some initial testing, it's the best of its kind I've tried so far. The workflow is embedded in the images. If it's missing, you can download it from Civitai here:
Hey LocalLLaMA, I released GameSummoner a bit over a week ago as an open source program and said "it works locally, but is a bit hard to use." ( Post ) Over the last few days I've cleaned up my prod host (
LiquidAI d1-3B is currently on the Pareto frontier of the Decision Index 0.3.2
Article · llama.cpp
opencl: add cooperative-k dp4a gemm for q4_0, q4_k and q6_k ( #30741 ) opencl: add cooperative-k dp4a GEMM for q4_0 opencl: add cooperative-k dp4a GEMM for q4_k opencl: add cooperative-k dp4a GEMM for q6_k opencl: fix alignment Website:
Hi everyone, I got annoyed a bit with all the AI slop post I have seen here on this subreddit (and reddit in general). It has gotten better in the last few months, but that hasn't stopped me from doing an in depth analysis of AI slop in different subreddits over the years. If you
Breakdown starts around 0:28 AI-rendered character and CG character in full physical contact through a complex camera move, pixel aligned for comp. Evangelion 4.0: The Fourth Impact (fan PoC + breakdown) AI and traditional VFX usually get treated as two separate worlds. One is fa
Article · llama.cpp
opencl: add get_rows for i32, cpy for i32 and f32 ( #30756 ) opencl: support i32 GET_ROWS opencl: support i32 and f32 copy conversions Website:
So, I was always really fascinated with the idea of a smart/capable-enough model that worked at blazing fast speed. I assumed we needed some kind of ASIC-style dedicated hardware for a specific LLM to manage that, but then I saw DiffusionGemma could actually get some massive spee
What it is: 5B DiT, using a Qwen3.5-9B text encoder and the FLUX.2 VAE Two versions: Base , and SFT , which is fine-tuned on curated data Apache 2.0 Why it’s worth trying: Text rendering. Legible text in English and Chinese, which most open models still struggle with. Layout prom
I got tired of generic motion alerts, so I built a small self-hosted system that alerts only for situations I define. My first real test: a cat that keeps sitting on my parked scooter. How it works: Any webcam or RTSP camera as input Object detection with [YOLO11 / RT-DETR] plus
Been using strata 1.45 for a while. Updated to 2.0 beta this morning and my speeds are almost halved? Had been getting around 155 t/s on my dual 4090s with the Q3S swift 1.5 but am now getting around 95 t/s. Anyone else having a similar issue?
Article · llama.cpp
jinja : implement unique filter ( #30819 ) jinja : implement unique filter cont : inline key_of cont : fix alignment Website:
I wanted to know which low-bit (1-bit / ternary / 2-bit) GGUF files actually give correct output on CPU, and how fast they run, so I compared each runtime against a reference instead of judging by how the text looks. Machines: an ARM server (Neoverse-N1) and x86 VMs with AVX2 (In
Another quick MOSS news and goodies round-up, for those who may have missed some items. -> ComfyUI Portable is now officially at v0.42.1, with the .1 increment being to... "Fix some crashing issues a few users experienced when running the MOSS image template workflow."
Article · Paperless-ngx
paperless-ngx 3.3.3 Bug Fixes Fix: fix global search input layout shifting @shamoon ( #14491 ) Fix: dont include doc content on global search results @shamoon ( #14488 ) Fix: remove some redundant pagers @shamoon ( #14487 ) Fix: prevent weird wrapping of dashboard widget column t
Sorry, yet another flipping engine! A few weeks ago, I made a speed focused experimental fork of Splash for M5 Max users recently called Splish . One of the most requested features was Qwen 3.8 Flash-Next support. Honestly, my experience with Flash-Next was underwhelming before t
Article · llama.cpp
server : fix swa checkpoint invalidation logic ( #30666 ) server : fix swa checkpoint invalidation logic Assisted-by: Claude add testcase that fails with > n_past, but works with > n_prefix Website:
Article · noted.lol
Mitra brings events and tasks together in a self-hosted calendar, with Docker deployment and browser push reminders.
MiniMax-M3.1-Flash dropped today. According to the announcement, Space Bunny (the stealth model that showed up on OpenRouter in late September) was an earlier version of it.
Posting here because the use case is specifically “I'm running something on my own hardware and want it on my phone without exposing it to the internet.” Kite connects over Tailscale (no port-forwarding, no reverse proxy, no cert), and it talks to either a plain OpenAI-compatible
2 Quadro M6000 24GB (48GB VRAM) running Qwen3.8-Flash-Next-GSQ-RCO-IQ1 on Strata since I only had 32GB of RAM. Pretty impressive prefill numbers on this inference engine. The decode has always been decent for the maxwell flagships. This has 3.5 bits per weight so it is actually w
Hey guys, Nick here from Cohere. It's been a minute. We had a great time launching Command A+ and beta-testing North Mini Code with you guys earlier this year, your feedback has really helped make our models and delivery much better. one of the biggest things we heard was that yo
Hey r/LocalLLaMA , Blockway here, a small team from Hong Kong. After our v1 post, the most common question was basically "ok, but why does this model exist?" Totally fair. Here's our honest answer. The problem Long agent sessions eat VRAM. On a normal dense model every layer keep
PixlStash is my open source image and video manager that runs on your own machine as a desktop application or a headless server. It tags, captions and searches your library and has had ComfyUI nodes for a while. 1.12 is out and it is the largest release I have done, so I am askin
Hey r/selfhosted , I wanted to see what happens when you push local inference architectures onto absolute vintage consumer hardware. I dug out an old ASUS X75VC laptop (originally released back in December 2013) to run a strict CPU/legacy GPU fallback survival test. The Legacy Sp
Look out for the updated version! What changed This update changes which experts are retained . No retained weights were modified relative to the base IQ3_S model. About 25% of the retained expert selection differs from the previous release. The model remains English-only . With
Ming-3.0-audio-flash is up on Hugging Face . It takes audio alongside text instructions and can return text or a spoken response. The model card lists transcription, spoken Q&A, long-recording understanding, and reasoning about speech, environmental sounds and music. A few detail
Hey r/selfhosted , Like many of you, I've watched modern web infrastructure become a maze of proprietary cloud lock-ins. When you try to connect autonomous AI agents to build or maintain web applications, legacy CMS engines completely fall apart-blobs hidden in databases, broken
Hear clearly. Understand deeply. Act intelligently. Ming-3.0-audio-tiny is a compact native audio-language model from inclusionAI. It accepts audio together with text instructions and supports speech recognition, spoken question answering, audio understanding, audio reasoning, an
Hear clearly. Understand deeply. Act intelligently. Ming-3.0-audio-flash is a native audio-language model from inclusionAI. It accepts audio together with text instructions and supports speech recognition, spoken question answering, audio understanding, audio reasoning, and optio
Article · llama.cpp
cann : refactor #ifdef blocks to avoid unreachable code after return ( #25273 ) Website:
Every ComfyUI update seems to break something in my workflows, and I got tired of guessing whether a node pack was dead or just quiet. So I built NodeWatch , a free tracker that checks the top 300 packs (by Comfy Registry downloads, out of 6,066 registered) every night. What the
I never used Jev models until I came across this post yesterday. It's a Qwen 3.8 27B Lora that works as a decision model. OP ( /u/Massive-Ice2791 ) was kind enough to merge it into the original model and provide a gguf. To test it out, I used QFN to create a playable snake game w
Article · llama.cpp
CUDA: fix warning for nonexistent extra semicolon ( #30774 ) Website:
Hey all, I want to see how you feel about this. I was tired of leaving my PC and then not being able to update things when I left for a few minutes or a while. Or getting told that my mobile session couldn't talk to my PC session. So I made a self-hosted memory server (MCP) that
Despite not including it in most official announcement they included it in the first part of the FAQ on this page:
Article · llama.cpp
ggml-cpu: vectorize ggml_v_silu and ggml_v_expf ( #30791 ) Signed-off-by: Aaron Teo aaron.teo1@ibm.com Website:
I accidentally have stumbled over an exciting announcement of Intel. Here are official specs:
I was working on exactly the same thing so hoping I can scrap my half-done job and adopt this. It seems to have all the features I was targetting.
Most of the stuff built for amd gpus posted here or ai in general depends on WMMA which exists only from RDNA3 and up. In image-video generation I can bypass it with triton some of the time. But here, a thing works or not on rdna2. Something specific like strata for example has a
I am mostly familiar with llama.cpp but have been using QFN - UD-Q4_K_XL with Strata and need some help understanding how it works with multiple gpu's. I initially started with a single gpu and got about 70 t/s. RAM use was around 80 GB. Then I added the second and third gpu's us
We’ve been testing which agent harness works best for small local models. The result: a single-tool IPython harness can match much heavier production harnesses on Terminal-Bench Lite , and much of the “randomness” in small-model agents comes from noise in the environment , not th
Article · MarkTechPost
MiniMax has released M3.1 Flash, a 428B-parameter MoE model with about 23B active parameters and a 1M-token context. API pricing is $0.15/$0.60 per 1M tokens, with weights due October 23 The post MiniMax Just Launched M3.1 Flash: 428B MoE Agentic Model at $0.15/M Input appeared f
Article · llama.cpp
moe-cache : replace MoE LRU cache with a 50/50 SLRU cache ( #30637 )
Article · llama.cpp
qwen3 : bidirectional attention and dense_2 output projection ( #30607 ) qwen3 : bidirectional attention and dense_2 output projection Assisted-by: Grok convert : read use_bidirectional_attention in base Assisted-by: Grok Update conversion/base.py Co-authored-by: Sigbjørn Skjæret
I've been keeping a list of the engines that run Qwen3.8-27B and Flash-Next on consumer hardware, mostly to see where my own engine ( capy-flash ) stands. Sharing it because most numbers going around aren't comparable. Our rows are measured on my box: 2× RTX 5070 Ti, 96 GB DDR5.
H3 has been out for a while, but I hadn’t had much time to play with it. Finally had some time over the weekend and started experimenting with refmods and motion context. One experiment led to another, and I ended up spending quite a bit of time with it. I didn’t really notice th
Article · noted.lol
Home servers collect plenty of information about themselves. System logs record services restarting, login attempts, and other activity that usually goes unnoticed until something stops working. Finding anything meaningful can take longer than expected. I've always liked smaller
I've found Moos Image to be great for illustrations, but it's slower than other models about 12 seconds with a 5070 Ti, when others are more like 3 to 5 seconds (1024x1024). .images and prompt are on my substack (also the int8 workflow).
Ref: release , PR Starting from v0.7.0 , the default port for llama server is now 9931 (ggml). Those who did not explicitly set the port, expect breaking change upon upgrading. Those who use harnesses that have hardcoded 8080 port, good luck.
Article · llama.cpp
tools : reject truncated GGUF tensor data in split and export tools ( #30497 ) The problem here in the repo is gguf-split can silently produce corrupted tensor output when the input GGUF file is truncated. The fix is to add tensor-data bounds checks and short-read checks in gguf-
From MiniMax (official) on 𝕏:
I've been building whatmodel.sh , an open source AI leaderboard, and wanted to see how a labs' own benchmark scores compare with independent testers. Using 1,629 published benchmark results, I matched 611 comparisons of 180 models on 103 benchmarks. AI labs reported scores 6.4% h
I’m sharing selflift-Fish , my improved version of facok’s SelfLift node for MiniMax H3 in ComfyUI. Generate longer clips and higher-resolution video with less VRAM pressure. Fix the background swelling/shifting I encountered with the original spatial tiling. Keep attention accel
The comfyui default workflow for Moss image has option to enable prompt upsampler which uses the qwen3.5 9b text encoder. This is the uncensored version of it converted to int8convrot. Link
I started this after seeing yanokusnir-ai’s one-node-flux-2-klein, just to make things easier for myself. It has grown to 11 nodes, so I’m sharing it in case others want to use it too. One node covers generate, edit, upscale, prompt writing and a gallery (7 image nodes, music, vi
Article · llama.cpp
The fused moe_reduce kernel computes dst = sum(experts[e] * weights[e]) in a single dispatch, reading experts and weights while writing dst. The graph allocator computes tensor liveness on the unfused graph, where the inputs are fully consumed by earlier nodes of the fused group,
By the original authors of DFlash 2 It's about 20% faster than MTP in their table when concurrency 1.
My rack cost €1,000 in total, all used hardware from eBay Germany. 2017 repurposed office PC running proxmox and a used Facebook marketplace build hosting an AMD RX 6900 XT with 16GB of VRAM bare metal, no virtualization. It runs my home network services and supports production a
What do you think about a public repository where contributors can dump their sessions manually. We can also build browser extensions and other collectors to automate this as well. Why this is useful: Right now frontier models do not allow distilation and open llms have very limi
Article · llama.cpp
cuda: cap Stream-K blocks at the number of k-blocks Stream-K normally launches one CUDA block per SM. For small matrices, this can exceed the total number of output-tile/k-block pairs, leaving blocks with no work and adding launch and fixup overhead. Cap the grid at that count to
Article · n8n
2.43.8 (2026-10-19) Bug Fixes core: Turn off expression spans by default ( #41310 ) ( e88162a )
hey, my name is Ye, I vibecode at Nace.ai and we do a lot of work with enterprises, and a lot of the time we're inside their networks with no internet access and outdated cloud hosted models. considering we work on macbooks, we have built Kapitsa, an open-source (Apache 2.0) infe
When did they manage to do that? Its a huge improvement in quality.
I maintain TensorSharp , an open-source local inference engine and agent runtime. I’ve added Video Studio to bring local video generation, an editable timeline, and an optional local LLM assistant into the same project. The attached demo runs in TensorAgent on macOS . The actual
DeepSeek-V4.1-Flash-1017's Terminal bench 4.0 score: - Model card reported 42.3% - AA reports 34% - Vals index reports 31.82% AA Vals Index Model card Model card quotes: We use the Claude Code harness for Terminal-Bench 4.0 I am guessing AA & Vals index uses mini-swe-agent thus c
Article · llama.cpp
CUDA: fix shared-memory WAR hazard in flash_attn_ext_vec ( #30573 ) CUDA: fix KQ write-after-read race in flash_attn_ext_vec Assisted-by: ChatGPT CUDA: apply review suggestions for KQ sync comments Co-authored-by: Johannes Gäßler johannesg@5d6.de Co-authored-by: Johannes Gäßler j
I Spent the weekend chasing a hard power-off that hit ~7 minutes into every start of a 2-GPU Radiance vLLM tensor-parallel stack, and the answer surprised me enough that I want to know if it's a known thing. Short, short version for anyone that is impatient ... I built a new mach
Is there one?
Accepted at NeurIPS 2026 TLDR: The paper presents a library of synthetic time series generators and augmentations to train time series foundation models, in their case a DeltaProduct based linear RNN that predicts each time step, i.e patch free. The synthetic data has already bee
Hi all, I've been working super hard about something that I've been super passionate about, the successor to
Hey everyone, One of the benchmarks I run for myself is to check obscure knowledge, like game rules from Anno 1602 (a game from 1998) and it's (many confusing) editions. Websearch is not permitted. So far, it has come the closest for me in niche information. Ranking order (1 high
Hi folks! We are HardenedLinux, an old-school free software community. SonicBoom is our open-source (GPLv3) inference runtime: a compiler-first alternative to the usual Python stack. No Python at build or run time - one shared library, a C ABI, and Guile Scheme as the first-class
Reddit from r/LocalLLaMA.
hello all! I have a project in mind that i want to try and figured i'd come here for some help to start out. I have a think center micro with very base specs: i5 or 7 processor (can't remember rn) 8gb ram 512 HDD windows 10 home i originally got it a while back to learn stuff abo
Article · llama.cpp
hexagon: FLASH_ATTN_EXT improvements for longer context and SWA models ( #30731 ) hex-fa: support for masked block skipping to reduce DMA fetching hex-fa: finetuning the hmx->hvx crossover thresholds (boosting gemma-4) hex-fa: update hmx gate to account for mdev.count Website:
Article · ComfyUI
Fix some crashing issues a few users experienced when running the MOSS image template workflow.
I've been working on optimizing Qwen 3.8 Flash Next on my local setup with two CMP 170HX cards and finally got to a place where I feel like the results are worth sharing. Both cards are running over PCIe Gen2 x4 - no x16 mod or TP2. The setup uses the W4A16 AutoRound checkpoint,
Another little challenge I seen on a DeepSeek Thread. I thought I'd try out it out on Qwen FN NvFp4. It was a pretty straight forward prompt with 4 images attached in the doc. The thread was testing different harness and models and showing their results. Op of that thread has two
Hi all, been seeing a lot of posts here about moss and wanted to experiment with it. Whenever I run a workflow my “load diffusion” node throws an error that the model type is (“”). I downloaded someone elses workflow and it looks like there’s an updated node that allows users to
Want your local LLM agent to drive ComfyUI on the same consumer graphics card? Without coordination, the first video render lands on top of the loaded LLM, and something runs out of memory. Boom, OOM. I built vhotseat to fix that. It makes the LLM server and ComfyUI take turns on
Google just released Harness2 as open source. It turns any AI harness into a system that improves itself over time. A harness is everything around the model. System prompt, tools, memory, skills, and subagents. Normally a person designs it once and the agent uses that fixed setup
Hey, So I let things get out of hand while working on my entry for the Open Source Higgsfield contest, and started building things that had nothing to do with the entry I was working on. First the ish that no one will really care about: Re-wrote/converted both the Comfy Router an
I'm a noob. I realize that but I thought I'd test something. I used a picture of myself and thought it would be funny to see myself suck a dick. So I tried a couple of prompts using phrases like "sucks his dick" or "gives him a blowjob" and in all cases the body movement seems ok
Looks like they are sending a dgx spark for $4499. I couldn't find much information on them but I keep getting Instagram ads repeatedly(even before the 64GB GB 10 West launched). LinkedIn and X points to some other company called MoBagel. If anyone tried this before, would be int
Article · ServeTheHome
In our GL.iNet Comet Q (GL-RMQ1) review, we see how this USB Type-C remote KVM device with Tailscale makes connectivity easy The post GL.iNet Comet Q GL-RMQ1 Review A Single USB Type-C Cable Remote KVM Device with Tailscale appeared first on ServeTheHome .
Article · ComfyUI
What's Changed [Partner Nodes] feat(Krea): add Krea 3 Image node by @bigcat88 in #17067 Update workflow templates to v0.11.85 by @comfyui-wiki in #17068 Use FLUX 3 Image in node display labels by @MaanavD in #17075 Better documentation about the comfy quant format. by @comfyanony
This is the basic scheduler for all flow models.
Seems to speed up inference, use less tokens, and increase intelligence - curious if people seen this tldr don’t think in text, think in latent space
I'm building a platform that runs ComfyUI workflows, and I used to keep exported API JSON files and patch them on the fly. It was a rough experience, so I built tscomfy . It reads your server's /object_info and generates TypeScript types for every node you have installed, custom
I have it installed on my comptuer and I've only experimented with some existing templates that come with comfy UI... Is there any simple tips anyone here can give me on what template to use to make these videos that come with the Comfy UI install? I made a couple 15 second clips
I've tried several upscaling workflows trying to get to 2MP with Krea2 turbo. But every different version creates random issues resulting in lower quality. Is it only possible with Krea2 Raw, or am I just not finding the right sampler settings? I've tried deis, bongtangent, euler
My job is decommissioning a few of our old nutanix servers. I’ve been
Hey yall. What I am feeling is a mixture of idiocracy and dishevelment. I looked at this motherboard with 256 RDIMMs and the x16 slots. I have 2x V620 and 2x Mi50 32GB. I wanted to play. That’s all. Mi50s would run Qwen3.6-35B-A3B at q4 and 131k context kv at q8 and same with V62
What a week. Quick recap, all numbers from the official repos and model cards: Thursday: Meta drops Muse Spark Open. 816B total, 42B active, 1M context. The launch post says it brings agentic AI "to your desktop". The desktop is two M5 Ultra Mac Studios with 512GB each, a config
Hi r/selfhosted ! FleetPilot is an open-source (GPL-3.0) dashboard for the boring parts of running a homelab. Version 1.8.0 came out today. What it does - Updates for Linux and Windows machines (Windows via OpenSSH, incl. Server Core/LTSC) - SMART disk health with alerts - Proxmo
Article · noted.lol
Keep selected YouTube channels and playlists on your own server with Yarrtube, a lightweight Docker app built around yt-dlp.
Is there a fork of strata that let me run lfm 24b a2b on only 16GB of ram comfortably? And how much tps can I expect per second and what quantization should I use? and Thanks!
The latest update broke the view in the web UI to where the wires connecting the nodes doesn't seem to line up when zooming in or out. I'm not even really sure how to describe the problem. It was always a problem when changing the resolution. But simply reloading the page would a
A couple of days ago I made a post about anima enforcing its own character dynamics, which was very hard to overcome. (additional props to u/Upset_Big3430 for help) The solution was quite stupid. I was trying to make futa Jean bang Lisa from Genshin, and to make it work I stated
Just something I've been testing over few week since Embedding Gemma 2 released which is about 330MB at Q8. Not sure if this would count as 'resource' flair... Image shows cosine similarity changes as I refine the text query (aka text embedding vector) over time. Shows how well i
Article · llama.cpp
cuda : fuse qwen4exp's SCALE -> SIGMOID -> SCALE -> hc_post chain ( #30602 ) Port of the Vulkan HC_POST_GATE fusion ( #29520 ). hc_combine builds the hc_post weights as 2 sigmoid(inject/hc); dsv4_hc_post_f32 now takes a gated template flag and computes scale_out sigmoid(scale_in*
Hi everyone! I'm currently developing some custom nodes for ComfyUI as part of a personal project, and I've run into an issue I haven't been able to solve yet. I'm working with visual elements that need to overlap within the ComfyUI interface. The problem is that whenever I click
Article · llama.cpp
ggml: support non-contiguous SUM (CPU, CUDA, Vulkan, WebGPU) ( #28851 ) ggml: support non-contiguous SUM (CPU, CUDA) tests: add non-contiguous SUM cases tests: SUM over non-contiguous views vulkan: support non-contiguous SUM webgpu: support non-contiguous SUM Website:
Apparently whilst doing some performance tuning on my 4x5090 rig I may have made things a little TOO fast and melted my plug and socket 🫠 Probably should put in a 16/20 amp socket in the wall rather than relying on a bog standard 13A fuse! Anyone else managed to melt the wall bef
A quick MOSS-Image news and goodies round-up, for those who may have missed some items. -> Spectrum for MOSS. A very pleasing speed-up, I find. Just wire in the 'auto' version of the node after the model. Tested without any turbo LoRa, and I can get pleasing 1:1 testing results i
hi everyone, I trained a lora 3k steps but the lora keep making these patterns on the skin. i see now that the pictures in the dataset i used has a bit of the same, just a bit less visable... is there any way to fix this? just wanted to know before i started on a new dataset. im
no document or announcement from unslot team yet
I needed a cheap way to compare search agents on BrowseComp-Plus without burning hours of agent time every run. So I built a 15-query subset of the full 830 that ranks systems roughly the way the full benchmark does. - weighted Spearman rank agreement 0.88 averaged over 30 splits
Hey everyone. I've been hitting a wall and want to know if it's just me or a shared pain. My setup: I run AI tools (local LLMs + agent-style assistants) across several machines - my home PC, my office PC, and my phone. I also self-host a NAS where I tinker with AI + storage. The
Reddit from r/comfyui.
Curious what everyone runs in practice: - What quant do you use most, and for what (chat, coding, agents)? - 2-bit: do you use it? Is it actually usable or too broken? - 3-bit: good enough for coding and more complex stuff, or do you go back to 4-bit? - If you look at KLD: what n
Ama: www.reddit.com/r/StableDiffusion/comments/1x7zf1r/hey_guys_im_one_of_the_core_developers_of/ Sub: www.reddit.com/r/moss_official
It's an extremely lightweight speech recognition model. 949K parameters using a weight-shared Conformer-CTC architecture Just 1.06 MB for INT8 weights Around 436× real-time on a single CPU thread 11.43% WER on LibriSpeech test-clean, using greedy decoding with no external languag
With the same prompt, different seeds lead to totally different artstyles, brightness, saturation, contrast etc. Is there anyway to 'guide' it in a specific direction? Right now I'm just rolling the dice each time.
Hi all. Saw JetBrains’ Mellum2 on Hugging Face and was wondering if anyone here has actually used it. Worth trying, or nothing special?
Update from yesterday: 18 t/s decode was hit. Slight regression now as cherry picking from other ds4 work trees proceeds. Will push some code once the prefill droop is handled and the 18 t/s restored. Weights are gated on hf, so mulling upload vs. recipe. Ps. This is text only. O
Article · noted.lol
iSponsorBlockTV connects to your television's YouTube app and automatically skips sponsor segments without replacing the app.
Article · llama.cpp
ggml-cuda: fattn do not check oob if k_VKQ_max = (k_VKQ_max - nbatch_fa) ( #30572 ) Website:
Hi, In 2024 there was those videos were suddently people would move in weird fashion and object would appear here and there. this is people roleplaying that :
I've just released Production Slate V4.5 , a substantial update to my free, open-source ComfyUI production saver. I originally developed Production Slate to solve a problem I kept encountering while making AI films: hundreds of images, video takes, reference frames and sound file
There, I said it. Relatively lightweight, ZERO censorship, incredibly easy to train on,excellent base capabilities for its size and FINALLY the first model in forever to not have been RL'd to death on 1girl standing realistic. The finetunes we're gonna get out of this model are g
Article · vLLM
v0.32.0 Highlights This release features 814 commits from 363 contributors (121 new)! Structured decisions and generic parsers : the new POST /v1/systemone endpoint answers choice , noul and score questions over a structured state ( #59299 ), is served by default ( #60677 , #6069
Reddit from r/StableDiffusion.
Looks quite interesting. Lots of numbers presented, but for a single model only. Highlights: File sizes are the same as Q-quants. KLD improves substantially over Q-quants. Performance suffers slightly Creating IQ quants is somewhat slow. Asterisks apply...
Is GMKtec evo x2 128gb still worth it right now?(oct 2026). I found one used for around $2300 usd.
Here is what's inside: File What it does SKILL.md The entrypoint. Mental model, ComfyUI defaults, prompt order, core rules. Start here. photoreal.md Camera, lens, film, light, texture. Portraits, products, landscapes, architecture, heroes. krea2-core.md Deep architecture, negativ
I've got Hermes driving a Chrome profile over CDP for a handful of self-hosted tasks. All ~48 web logins sit in that single profile, and the agent just clicks through native autofill so passwords never enter the model's context. That part's working, but it's fragile in two ways.
So I created this simple test (for myself) to see how good the new models are in analytic output. This is not asking them to draw a SVG (which just tests how much SVG or HTML they saw) but creating a prompt that explains the simple syntax for a new graphic language with a few exa
Hi! I need to install of torch with the cuda 11.8 pip index but i dont know which command is in the comfyui terminal any help? How i do a clean install of torch too? Regards!
Article · LiteLLM
Verify Docker Image Signature All LiteLLM Docker images are signed with cosign . Every release is signed with the same key introduced in commit 0112e53 . Verify using the pinned commit hash (recommended): A commit hash is cryptographically immutable, so this is the strongest way
Next post about MOSS Image, but instead of "1girl" lets do cats. SFT, 40steps, cfg3, euler/sgm
007: ✂️ Video trimming 🖥️ Preview windows 🔧 Upscaling fix for trimmed clips Obvpm new wokflow version 007:
i wanted to change my router firmware to openwrt and so i told claude to do it but of course the guard blocked me even opus 4.8 didn't want to do it , so i launched my Qwen3.8 27b and told it the same thing i gave it the link to the giude page and it did change the firmware perfe
First time trying out ComfyUI after hearing good things about the MOSS image model on Twitter. I had absolutely no clue what I was doing, so I just had ChatGPT build me a .json workflow and started experimenting. I've been testing anime characters composited into photorealistic e
I tested all my previous training methods with Ostris Toolkit and Fizgig which worked pretty well in the past for Krea2, ZIB, IDG4, SDXL etc but for some reason the Moss Image character Training doesnt come out right.. only getting like 80% likeness. Even tested the Academia SD w
I intend to build i5 14600kf + 64ddr4 + main msi z790p wifi + 5060ti 16gb. Can it work for wan2 14b, flux 2 dev or even higher models? Longer time is no problem for me. Msi z790p can set up dual gpu: 1st gpu at pcie5.0x16 and 2nd gpu at pcie4.0x4. So the next gpu can I buy anothe
Same prompt, same seed. Moss is better on some seeds, but generally worse then Krea
Great Lora, takes anything you throw at it. 5.2 second generation for smoothness.
I would love for some feedback, live demo at: huggingface.co/spaces/e12ex2/herbert-demo Not self promo, jus looking for some feedback on what to improve or any bugs or model breaking you found.
Article · LiteLLM
Verify Docker Image Signature All LiteLLM Docker images are signed with cosign . Every release is signed with the same key introduced in commit 0112e53 . Verify using the pinned commit hash (recommended): A commit hash is cryptographically immutable, so this is the strongest way
What's the catch? Has anything been fixed yet
With winter coming up where I'm at, I wanted to see if this could actually work. Running my 4x V620 server flat out made my apartment too hot anyway, so the problem really was control. I settled on using a motherboard temperature sensor with an offset cause cases get hot. It work
sudoing posted that he's loaded Ling 3.1 Flash across four DGX Sparks. It's a 560B MoE with 25B active parameters per token. He'd enjoyed the earlier Ling models and wanted to test this one, but it wouldn't fit on his two Sparks with the setup he was trying. someone in Germany ga
Reddit from r/homelab.
Put together a quick T2I comparison at ~4MP resolution testing Qwen Image 2.1 Turbo , MOSS-Image-SFT , Krea 2 Turbo , and Flux 3 Klein across a variety of prompt styles-ranging from simple subjects and unique/mixed concepts to highly complex descriptions, intricate details, and m
Qwen4s are hopefully coming pre-Xmas… there was all the announcements recently but no exact release date. I’m running qwen flash next 3.8 at about 28tokens per second and gotta be real, most of those go into thinking. Thinking less with the same or higher intelligence would be th
Hello everyone, BFL released new workflow that has fix for comfyui using the same models i did comparaison using the base and turbo version. With unsampler Generation Time (base): 6 min CFG 3.5, 25 steps at 3 megapixel Generation Time (Turbo): 2.5min CFG 1.0, 16 steps at 3 megapi
Happy Halloween! Full video:
Is there some reason this just gets more and more agonizingly slow to run updates ("Update All") and nothing shows in the console? It's just stuck on "FETCH DATA from:
Article · llama.cpp
Restore ESIMD kernels on FFN path ( #30239 ) Reenable ESIMD on the FFN path Signed-off-by: Todd Malsbary todd.malsbary@intel.com Extend ESIMD fusion on FFN path to include Q8_0 Signed-off-by: Todd Malsbary todd.malsbary@intel.com Remove comment explicitly referencing ESIMD in tes
Got lucky with a 5070 Ti for $750 AFTER TAX (might be the best decision I've ever made tbh) back in November 2025. If only I had known I'd be getting into local llms at the time...anyways. I recently decided to start messing with local AI. I began getting comfortable in Powershel
It handles the latent space like no other model. It can generate images from 320×320 to 2K without producing visual monsters. It can generate all kinds of NSFW content without needing LoRAs. It handles lighting and realistic scenes better than Z-Image. I even tried 120×120 , and
Hey y'all, just following up on my PR to get the new vision tower for 1017 working in ds4 with the quant I promised (the language model loads already, but vision needed my PR). I built to the exact same recipe antirez used for his original DS4.1 Q2: 2-bit routed experts with 8-bi
Suffice to say SD3 is not even remotely similar to any recent model (as anyone who knew what they were talking about in any way was already aware). Prompts were: 1: A high-fidelity social media portrait in a medium close-up framing, dominated by a stylish Black female influencer
it isn't like flux 3 all censored, this will teach flux. to get gud.
Been experimenting with fine-tuning Qwen-Image-2.1 and finally got around to releasing the LoRA. It can almost generate any pose. It's a rank-64 adapter trained to handle a wider range of poses and compositions, including prompts the base model tends to struggle with. One thing I
Some of y'all may be familiar with AutoTrust from Hugging Face. Many of their models were trending on the front page of HF and they claimed some very impressive results. But if you looked at the org's models and clicked into likes, you may notice something a bit off:
The official open-weights MiniMax H3 ( Ref2VA ) model was trained on a rigid six-section schema ( subject_definitions , summary , retention_analysis , detailed_description , overall_soundscape , non_diegetic_music ) paired with exact angle-bracketed asset tags ( , , , , ) and int
Article · llama.cpp
hexagon: add support for hadamard MUL_MAT and GET_ROW for flat and tiled Q1_0 and Q2_0 ( #30627 ) hex-hadamard: add support for hadamard ops hex-mm: reduce pkt counts in the q8_1 quantizer and vec_dot_q1 hex-q1/q2: add support for flat and tiled GET_ROWS for Q1_0 and Q2_0 Co-auth
Hey everyone! I'm looking for an easier way to install models, LoRAs, checkpoints, etc., into my local ComfyUI setup without using the terminal or manually downloading files and placing them in the correct directories. Is there any extension available through ComfyUI Manager that
Been seeing a lot of people with their 64gb of ddr5 4x3090 or whatever strata builds and it’s been sorta depressing knowing i cant afford that but i finally finished my setup today so here we go, Qwen 3.8 27b at 262k context, unsloth iq4XS ascii condensed with kv cache streaming
Reddit from r/LocalLLaMA.
I’ve been working on a music video for my song No Rehearsals . The look is coastal noir: neon, wet streets, amber light, blue shadows, and two fugitives moving through the city. The biggest challenge was keeping the characters and visual language consistent across different scene
Interesting enough there is a solution but you're not supposed to know it from ComfyUI people in their Blog or here. The old good times where the Customer Support from US Companies was the best in the world are over since quite an amount of time. Before you were a Client, today y
Instead of counting 'hmm' on DeepSeek v4.1 Flash 1017, I today clamped Muse Spark Open 816B-A42B to ds4. This was the first successful run, FTW, that made it to decode... 4 t/s! #RoastMe. I'm hoping it will 4-5x with any luck. I'll obvs be running larger prefill chunks too (pre-m
Hey everyone, Alexis from the Comfy team. A lot of you tried FLUX 3 Image Klein with the example workflows we shipped on day one and got bad anatomy or weak results. Part of that is because the workflow templates had sub-optimal default settings. Most importantly, the prompt upsa
I ran five coding tasks against nine models on one RX 9060 XT (16 GB). Each run gets a single spec file in an empty directory and has to produce solution.py and test_solution.py , run its own tests, and stop. Grading happens with a checker the model never sees: fixed cases from t
UPDATE 2: I've changed the MOSS pre-sets to all be 1mp, the model just performs better that way and it seems less tolerant to lower MP training than other models. Its a super memory efficient model anyway, so this is a win-win. Make sure SFT sampling is turned on in the samples t
Basalt: Qwen 3.8 FN NVFP4, single 5090 Hi guys, so I finally got a RAM upgrade and have been working on the top two pain points from Basalt: NVFP4 support and faster prefill. NVFP4: Basalt runs NVFP4 natively now, a new .basalt pack keeps stock NVIDIA experts and stores the dense
By chance, I was playing around with ASR models and found MOSS-Transcribe-Diarize a day before MOSS-Image launched. Out of the 7 highly rated models I tested out, MOSS-Transcribe-Diarize went toe-to-toe with or downright beat the previously best-performing VibeVoice-Large, despit
I updated ComfyUI and the render times have disappeared so I no longer know how long they took to complete. I don't know how to re-enable it :(
Workflows are embeded. workflow also here :
Hey everyone, I am working on a new project called Openspline, an open source avatar layer to existing voice AI agents. Just select an image and you can create a realtime talking avatar with lipsynced video generated locally on your GPU. You can integrate any existing voice AI ag
A recent Comfyui update appears to have broken the ability to drag a file (image) from Assets on to a Load Image node; anyone else seeing this?
Hey everyone! I'm looking for advice from people who use local video models to generate anime. I've been thinking about making my own little anime series, but I've run into a few problems. My setup: MiniMax H3 for video generation, GPT Image for storyboards and characters. RTX 50
Hey everyone! Just wanted to share my latest showreel with the community. I've been working as a visual artist for over 10 years, mainly in concept art, matte painting and visual development. Over the past few months, I've been exploring more hybrid workflows, combining ComfyUI,
After installing from scratch the latest Portable release 41, at first start up after installing I heard my external disks crunching at full speed, so I couldn't understand why this scanning was taking place, so I disconnected the ext disk. Then after checking the logs I understo
I have created a quality check node that checks if a created image or video is any good. It then outputs a true or false output. I want it so if it outputs false the image generation should be run again until it gets a true but i cannot for the life of me figure out how the loop
In short: Nvidia RTX scaler, announced 4 day ago, workflows to dl and so on. Then a day or two later Flux Kleni 3. Nice, so I prefer to re install from scratch latest Comfyui Portable. Dl release 41. Installed. Then upload the RTX scaler workflow, 2 errors: missing input image, t
Article · Ollama
Implement the Qwen3.5-backed decision scorer with noncausal attention, the publisher readout, image handling, and System One prompt encoding.
Hey all, been meaning to post this for a while. I've been building an inference engine based off Strata called Xolotl that runs DeepSeek-V4.1-Flash, all 552B of it, on one gaming PC. My setup is a single RTX 4090 (24GB), a Ryzen 7 5700X, 90GB of DDR4 and an NVMe drive, nothing fa
Two things I ran into while moving my Strata setup from IQ2_XS to Unsloth's UD-IQ4_XS (RTX 4090, i9-14900K, 64GB DDR5-5800, Arch). 1. Greedy sampling on older builds Before Strata 0.1.45, a run config without a `sampling` block decoded greedy whenever the request didn't set its o
So after playing arround with capability of Strata GSQ-RCO Coder vs Unsloth IQ4, many of such test are creating animation. Welp turns out it somewhat created good animation and also comment from reddit asking for the same stuff
Article · llama.cpp
hexagon: add support for Q1_0 and Q2_0 ( #30122 ) hexagon: add support for q1_0 quant hexagon: enable Q1_0 weights in the 2D HMX MUL_MAT path hex-q1: cleanup gates, generalize tiling and remove in-loop divs hex-tiling: document tiled formats in developer.md and update htp-tensor.
DISCLAIMER: the prefill rate on the dashboard in the video counts cached prompt tokens as if they were read again, so it reads high. The captions show the server’s real end-to-end numbers, and so does everything below. The decode numbers on screen are correct. After the last post
I went from 32GB to 64GB of DDR5-5800 this week and moved my Strata setup off IQ2_XS. Box: RTX 4090 24GB, i9-14900K, 64GB DDR5-5800, Arch. The models sit on a SATA SSD (~550 MB/s), so anything Strata has to pull from disk while it generates hurts. I wanted every expert in RAM or
Just pushed a PR for dwarfstar that adds support for Deepseek-v4.1-Flash-1017 with its new engrams and updated vision tower. The PR basically just adds compatibility for the file revisions and checks to make sure the vision+language model file combination matches, raising on mixe
From my testing, flux3 seems to be very good at higher fidelity composition combined with single image style transfer than the much looser Flux2 9B. Granted, I have not yet run a test with flux2 with and without the flux3 prompt expander. The flux3 release has been controversial
Quote: Razor Lake-AX's 700 TOPS will far exceed the NVIDIA Jetson AGX Orin platform, which offers 275 TOPS Jetson does about 18 tps on 27b. So 18 * (700 / 275) = 46 TPS. 46 TPS is about double the performance of strix halo. AMD medusa halo is also expected to double strix halo pe
Hi all, as many of you are probably aware DeepSeek V4.1 Flash 1017 was released yesterday. I put up an abliterated model + GGUF quants for both the base and abliterated version (no imatrix yet that's WIP) For the abliterated model, most refusals are successfully removed. On the o
Drama 3 officially needs an NVIDIA GPU with 48 GB. I wanted it on my RX 9070, so I built Red Curtain together with Claude. How it works: it's a 35B MoE with ~3B active per token. The FP8 experts (~32 GB) stay in system RAM and only the ones that actually get used are cached on th
Is anyone else having this problem? Is this possible? Or does the model's architecture not allow it?
Made another installment of my Star Trek Mirror Universe fan film using ComfyUI. This version combines Parts 1C and 1D into one continuous sequence. I’m experimenting with AI filmmaking, particularly character performances, shot continuity, and cinematic storytelling. Still learn
If you've got ComfyUI and a decent graphics card, you already have everything Photoshop charges a subscription for. OpenFill is a free GIMP 3 plugin that turns them into one powerful, Adobe-style AI editing suite right inside GIMP: Generative Fill: select, type, Enter Remove obje
Article · llama.cpp
ggml-cuda: HIP: use __builtin_amdgcn_perm for q1_0 unpacking ( #29927 ) Website:
Yes Krea 2 is amazing but even with loras it still struggles at some things and it has sort of an inflexible style to it. What it struggles with Moss can do right out of the box. To me after playing around with it Moss kind of feels like SDXL 2, except it's not only uncensored, t
The new models like Qwen3.8 27B and DeepSeek V4 Flash etc., all have this terrible, Claude-sounding writing style that I can't stand. Here's an example of what I'm talking about: "A probe that asks what a language model memorised reports a null when the model answers at chance, a
Article · noted.lol
Homelable brings network discovery, live status checks and diagrams together to help document the machines behind a home server setup.
View on YouTube while video is pending here if needed.
Details are developing…
Fizgig LoRA Isolation: train 2 identities into one LoRA and keep everyone else looking like themselves. A normal LoRA of a person spreads their face onto everyone else in the picture: strangers in the background, a friend beside them, a crowd. LoRA Isolation stops that. Your subj
I tried to buy a month sub for colab but it only gives 200 computer units every month which is small limited for me. The official comfy cloud only gives 5 runs. Kaggle can work but it's sensitive for NSFW things and it gives immediate ban with no discussion. Some website even if
Reddit from r/StableDiffusion.
Article · LiteLLM
Verify Docker Image Signature All LiteLLM Docker images are signed with cosign . Every release is signed with the same key introduced in commit 0112e53 . Verify using the pinned commit hash (recommended): A commit hash is cryptographically immutable, so this is the strongest way
Article · ServeTheHome
This week NVIDIA launched their high-end RTX Spark N1X platform. Bringing their powerful GB10 SoC to Windows for the first time, the company is seeking to remake the Windows laptop market with their powerful, AI-focused SoC The post NVIDIA RTX Spark N1X Launch Overview: NVIDIA Ar
so we moved our support bot onto our own self-hosted model earlier this year so customer data stays in house. It answers from our help docs with RAG and for months it was fine and just ast week a customer sent a screenshot where the bot told him accidental damage was covered unde
I haven't tried most of the models I could run so I don't have much to go on but there's one Mistral 7b finetune I still load up and will continue to download to new systems, at this point the capabilities and knowledge of it has been eclipsed by smaller models with more features
Hi, I’m a 2D animator looking to get into the world of AI to optimize some of my workflows. My plan is to use AI for in-betweening (creating the intermediate frames) for animations that don’t require an ultra-polished, high-end finish; I will still be drawing the keyframes and br
Article · llama.cpp
ui : Discover & Download Models from HF Hub ( #29584 ) common : resolve - download tags and list cached sidecars A Q4_0-mtp style tag now resolves the sidecar file when no model file matches it, so a solo draft or mmproj download actually pulls the file. Cached sidecar files list
Hey everyone, Releasing v1 of XT-Luna 1B a fine-tuned MiniCPM5 1B focused on agentic and coding workloads. Tested against the official MiniCPM5 1B SFT and Qwen 3.5 4B reasoning benchmarks and comparison screenshots are in the model card. Still actively working on it, so feedback
Article · llama.cpp
CUDA: faster MMVQ for low-bit types ( #30517 ) CUDA: faster MMVQ for low-bit types ggml-cuda: hip: small nwarps GCN fix on Q3_K(ncols=2,3), Q6_K(ncols=4) Co-authored-by: thelittlefireman thelittlefireman3@gmail.com Website:
Reddit from r/StableDiffusion.
Flux 3 is heavy censored to the point of breaking the anatomy. Moss is crazy uncensored. It can do anything without a lora. Anything. Insane.
Reddit from r/LocalLLaMA.
this is the easiest model to get a new face with a new seed value that I have used. what other models and methods are people having an easy time to change faces after fixing certain basic characteristics of a person ?
Article · Ollama
What's Changed win: prevent orphaned runners by @dhiltgen in #18500 gguf: add the q2_0 tensor and file types by @jmorganca in #18967 Launch clef in Ollama's cloud by @jberg5 in #19036 use identifiers for --ask mode question keys by @jberg5 in #19041 Full Changelog : v0.40.5...v0.
Okay, sorry, but I’ve been busy since last Friday and haven’t had a chance to check Reddit. Now I’m seeing people talking about Flux Klein 3? And apparently it’s terrible (wait, seriously??). And then there’s MOSS Image?? We already have Qwen Image 2.1 and Krea 2, so how do QI 2.
*2.25MP, 16steps, distilled model. First i tried: Woman lying on grass. High quality. Second: Woman lying on grass, arms crossed. Perfect anatomy. High quality. Top-down camera perspective. Then, a challenge: Woman wearing red bikini lying on sand, arms crossed. Beach settings. H
AcademiaSD LoRAlab Trainer Studio now supports MOSS-Image, the new open text-to-image model (Apache 2.0). It is the 12th model family in the Studio, next to Qwen-Image 2.1, FLUX.2 Klein 9B, FLUX.3 Image Klein, Krea 2, Z-Image, ERNIE-Image, Ideogram 4, Anima, SDXL, LTX-2.3 and Min
Hello guys, hoping you're doing fine. Today I want to share something that may affect your cards (? I have released 3 tools: A way to modify VBIOSes, so you can increase power limit, voltage, NVVDD offsets, MSVDD offsets (Blackwell 2.0), OCPs, fan channels, power per fan, core of
LoRA for reducing the soft & smooth AI look and adds more texture and details to images. Dataset was high-qulality SFW Professional Photography images with wide variety like portraits, lifestyle, fashion, wildlife, sports fitness, architecutre, nature and more. It is trained on s
neat model.
big speedup, but only for specific quantizations like Q2_K and Q3_K
I am running my w11 comfyui box as a VM .... everytime I try to download the example *.json files from
Total Parameters 4B Architecture Looped (latent-recurrent) normalized-GPT with interleaved sliding-window attention (SWA) and full-attention layers Precision BF16 Context Length Up to 256K tokens Supported Languages English Best For General-purpose reasoning on a small footprint
Comparing Full Flux1 Dev vs GGUF on Comfyui via StabilityMatrix with ~50% RAM on 5070ti, feel free to comment. Cheers
Tested my 5070ti with WAN2.1 32GB FP16 14B on Comfyui via StabilityMatrix with random template, feel free to comment. Cheers
Jenna’s first day at OME OMY. Naya’s packing. Miles is already overthinking a relationship that hasn’t started. Meanwhile, Lexi runs HR and advises against office romances. She also LOVES love. So that’s helpful. This is Episode 2 of our workplace mockumentary, THAT SHIP HAS SAIL
Don't take the wrong things too seriously.
Article · llama.cpp
tests: give the random model generator a per-tensor filler table ( #30674 ) tests: give the random model generator a per-tensor filler table Muse Spark makes test-llama-context fail on state_load, cp_h and cp_d on about six percent of the random seeds, with an NMSE of 1.0e-05 to
Every image was made using base model at cfg4 and 39 steps and at around 1.5MP
I've been doing some testing to check the creative variety of MOSS image. It has got quite some cinematic looks in it. Nothing fancy about the workflow, just straight from the comfy template. diffusion model: moss_image_sft_int8_convrot.safetensors text econder: qwen3.5_9b_int8_c
Fizgig LoRA Isolation: train 2 identities into one LoRA and keep everyone else looking like themselves. A normal LoRA of a person spreads their face onto everyone else in the picture: strangers in the background, a friend beside them, a crowd. LoRA Isolation stops that. Your subj
Already posted here
Hey guys, I have been working on the consistent character portable format (.char) for a while & couple of reddit users asked me if multi character is possible for image & video generation. I have enabled the same in Omnichar with upto 4 characters in the same video or images. Her
I use the standard r2v workflow of ComfyUI. Trying to edit video (reference 1) only with prompt. 0.3 Mpxls, 3s on rtx3060 12gb VRAM, 64gb RAM. All resources to max it executes in over 30min. Please who has experience with this, can you share your results in process speed, is it n
Node threw an error during execution.
Article · llama.cpp
cuda : fix -Wtautological-compare in fattn vec dispatch ( #30519 ) Signed-off-by: Adrien Gallouët angt@huggingface.co Website:
Article · noted.lol
Your homelab may be costing more to run than it needs to. Here's why separating essential services from your lab can save electricity.
I have been working on implementing adaptive KV streaming and RDNA 4 / ROCm optimizations for llama.cpp to enable running Qwen3.8-27B (IQ4_XS weights, Q8_0/Q4_0 KV cache) with its full 262,144-token context window on a single 16 GB RX 9070 XT GPU at 80+ t/s decode. Qwen 27B IQ4_X
Article · llama.cpp
cuda : add grouped experts top-k fusion ( #29181 ) fuse grouped experts add tests hip fix add to graph_optimize apply suggestion from am17an hip fix fix hip fix fix fix fix gate on hip version instead Website:
Another quick Minimax H3 news and goodies round-up, for those who may have missed some items. -> The official ComfyUI Portable is now at version 0.41.0. The only H3 fix listed is that it now... "supports the MiniMax-H3 LightVAE" from Kijai. As a nice bonus, Nightly users also get
Article · llama.cpp
model : add Kolibri-1 (Aleph-Alpha) support ( #30673 ) model : add Kolibri-1 (Aleph-Alpha) support Kolibri-1 is a bilingual German/English MoE reasoning model with 78B parameters and 3.46B active per token: 50 layers, 384 routed experts plus one shared expert. Attention interleav
I swear I managed to get mine for like 1500 euro about 2 months ago but they're suddenly up by more than 1000 euro hovering about 2500-2800, what the hell is happening? Bro i shoulve bought two
During a post-event interview at the RTX Spark Launch Party: Taipei on October 16, 2026, Nvidia's Senior Vice President of Personal Computing, Jeff Fisher , explicitly debunked the recent discontinuation rumors.
Reddit from r/LocalLLaMA.
Reddit from r/StableDiffusion.
European 78B A3.4B model is now supported by llama.cpp, I am waiting for some GGUFs
I tested a few samplers and schedulers on the MOSS Image model in addition to Euler. These are worth using if you want to try out realistic images, I got pretty good results with them even at cfg 3: samplers: heun, heunpp2, exp_heun_2_x0, dpmpp_sde, dpmpp_2m_sde, res_2s, (res_2m)
I've got some nice boost by increasing --adapt-every value from calibrated 1 to 6. I tested on 4 consecutive prompts(Slovak language) in chat. Prompt 1-3 are simple questions, 4. is describing source code(20K tokens) Results: [prompt number / ~avg. output length] UD-Q4_K_XL: [1 /
🎯 About This Repository This repository contains the DiscoveryRSI-27B model release for DiscoveryRSI , a research system that recursively develops, evaluates, revises, and attributes scientific hypotheses. DiscoveryRSI-27B is an instruction-finetuned 27B language model intended f
I wanted my finished illustrations to look like an actual manga page: crisp ink lines, blank paper for light areas, solid black for dark hair and clothes, and screentone only where it’s needed. It uses Qwen-Image 2.1 Turbo (edit) to decide what should be black, white or toned, th
I understand this kind of product may not be exactly right for this subreddit, but I still think it deserves some attention, as it's an LLM frontend for local engines in the first place, and can be used for general-purpose chats with LLMs once lore extraction is disabled in the s
Lots of hype around Strata lately and the speeds look great. But everything I see is tokens per second and needle tests. Has anyone checked if quality holds up? Same GGUF on llama.cpp should answer the quant question, but Strata also prunes experts. Does that cost anything in act
Same prompt, same seed, no lora
Is self-hosting even worth it on your electricity rate? The answer is 3 numbers you can plug in yourself :) I made a calculator for it (one of 153 free tools, runs in your browser, nothing is uploaded and there is no sign-up):
I asked my local AI agent to optimize its own setup. The first thing it did was stop the inference server. The one it was running on. 🤦 Fair enough. Can’t beat zero resource usage. So we established a rule: don’t shut yourself down. On the next attempt, it cleaned up some Python
Here's a one click workflow so you can create Seamless Vlogs.
Existing weights are untouched Interacting with models via API calls is outside the scope of these controls 💢 Next-gen Chinese frontier models may face licensing requirements, export registration, delays before weights can be downloaded abroad --- Essentially it creates regulator
I got tired of rebuilding graphs for every clip, so I built a local app around my ComfyUI install called Castwerk. The flow: Pick a persona (a LoRA plus its settings, synthetic characters only) An LLM writes scene cards from an idea, you edit and approve them Image first, then vi
Article · llama.cpp
model : Granite switch decision 256 options ( #30693 ) feat(convert): Attempt to come up with 256 single-token, semantic option markers Each switch model needs to identify a collection of single-token option markers that can be mapped to their respective logits for the output pro
why？？comfyui updated to version 0.38 and above to support custom node Overlapping
Been experimenting with Krea2's Qwen3-VL vision encoder and stumbled upon a neat little trick. Basically, instead of feeding it an actual reference image, I'm feeding it a virtual canvas with a selectable RGB color. What's interesting is how much the color alone can influence the
I know the heretic tool is the usual generic approach to remove refusals from a model, and a lot of research has been done to find out what parts in the LLM need to change. This gets very expensive though the larger the model is. Then for Krea2 turbo/base I saw that there are LOR
I was running 120B model, I asked to do a UI overhaul for a page, to access the page it needs to login into Dev server, I provided creds. It refuses to log in.
Reddit from r/StableDiffusion.
I made a tiny site that let's you feel how fast a local LLM runs before buying the hardware. I made it for myself and a friend but thought it could be useful for others. Note that it works as an estimate and not perfectly as it will vary per user setup. repo:
Defiance puts DavidAU's Turbo-Brilliance "AI Construct" modes on HauhauCS's uncensored Qwen3.8-27B. You pick a mode (Joker, Dr. House, a 20-agent Einstein panel, a Socratic tutor) per request, and the uncensored weights answer it without hedging. Weights: HauhauCS/Qwen3.8-27B-Unc
How to fix this error
Hey Reddit! I just finished the 1st episode of my K-Webtoon Thriller Drama using ComfyUI + Minimax H3. Honestly, the 'comfyui-obvpm-timeline' pack is an absolute game-changer. It makes extending clips and editing narratives right inside ComfyUI incredibly smooth. No more lighting
DeepSeek users report a possible V4.1-Flash-1017 refresh across its API" title="DeepSeek 4.1 Flash 1017 Incoming? --> DeepSeek users report a possible V4.1-Flash-1017 refresh across its API" /> Was waiting for this. Edit: And it just dropped:
base model with turbo lora , 8 steps 1 cfg euler/simple tried different things , its got some variety , can do a lot of different things for a small model,anatomy is really good unlike some other models
I’m new to local AI models. Assume someone has a Mac Studio M5 Ultra 256GB. I’ve read that is the realistic minimum you can use to run a local AI model. My question is this: as models get better, will they run better or have more capable models run better on the same hardware? Me
I use Pi/OpenCode for coding tasks which still seems to be the most popular, but I'm wondering what you guys use for normal chat interface. It's been a few months since I've seen discussions about harness and at the time the popular non-coding choices seem to be: - OpenWebUI for
Everyone, including me was super hyped about Qwen image 2.1 but honestly i found the results to be wanting. In this video i show exactly why and also why its still a good model.
Qwen 2.1 is killing my 3060 12GB. Even the turbo mode takes over an hour and a half per image. I know the model is gigantic, but I like the results it gives. Are there any good quants out there that speed it up significantly on 12GB of VRAM? *Edit: Guys, I'm an idiot. I had a bra
Hi guys, I'm new here. 🙂 I've been using Dasiwa's workflow for a while now and have been really enjoying it! But recently I updated comfyui to mess around with the new Moss-Image model, but when I realized DasiWa's Wan 2.2 workflow breaks with this new update, I decided to go bac
Hey r/StableDiffusion ! I'm one of the core developers of MOSS-Image , the 5B DiT image generation model we just open-sourced! I've noticed quite a few discussions about MOSS-Image here already. It's been really exciting to see the community experimenting with the model, and I re
Article · llama.cpp
sycl: fix prompt-processing regression after Q4_0/Q8_0 weight reorder ( #30374 ) Website:
I think it's all working. I asked various models to make it all work for me, with the bulk being GLM 5.3 flash. Its not as fast as some of the benchmarks I've found but getting them working apparently requires even more surgery. I wanted a simple "Qwen 3.8 Flash Next FP8 MTP 256k
I've been experimenting with MOSS-Image's BBox layout feature and got frustrated with its absolute pixel coordinates. Changing resolutions or aspect ratios meant recalculating every bounding box, and mismatched coordinates sometimes seemed to cause unwanted white borders. So I ma
Hey everyone, English is not my native language (Dutch), but I write in my own words without assistance. So please let me know if any sentence flows weird, etc. A mate of mine told me how amazing local AI was in January 2025, but I was a skeptic who took pride in writing my own c
Article · llama.cpp
hexagon: vectorize softplus with HVX ( #30435 ) softplus_f32 and tile_softplus_f32 ran a scalar logf(1.0f + expf(x)) loop. Replace both with a shared hvx_softplus_f32_aa() helper built from the backend's existing hvx_vec_exp_f32 / hvx_vec_log_f32 primitives, so no new numerical a
![v0.32.0rc3: [Bugfix] Fix NaN output of the Mamba2 chunk scan kernel on sm90 (#61718)](media/13d109f5467363c9.png)
Article · vLLM
Signed-off-by: Vadim Gimpelson vadim.gimpelson@gmail.com Co-authored-by: Claude noreply@anthropic.com
Article · llama.cpp
opencl: Add MxFP4 MoE bin kernel for Adreno X1 devices ( #30691 ) opencl: add bin kernel gemm_moe_mxfp4_q8_1_dp4a_ila_ldck for X1 opencl: fix whitespace opencl: undo unnecessary change opencl: enable dp4a clc for X1E Co-authored-by: Li He lih@qti.qualcomm.com Website:
Article · MarkTechPost
Perplexity has open-sourced SPACE, the sandbox platform behind every task in Perplexity Computer. The Perplexity SPACE sandbox gives each agent task its own Firecracker microVM. Credentials and outbound traffic are controlled from outside that VM. The code ships under Apache 2.0
My experience is based on running Qwen 3.8 Flash Next UD-Q4_K_XL on 4x 3090's Compared directly to Qwen 3.8 27b Q8 it is not even close. To add to this, strata results were even worse than llama.cpp even thought generation speeds were faster. I left 1 run for 4 hours in strata an
I have updated to the latest build of comfyui stable with custom node for moss image. But I am getting error in ksampler right at the beginning of sampling - attribute error none type set to dtype. Resolved - apparently the model I downloaded from hugging face comfyui was bad/cor
Article · Ollama
Behind a proxy, defer redirect-target DNS to the proxy; without one, keep the resolve-and-pin path. Trust default-registry/CDN hostnames with no local DNS. Fix a nil CancelFunc crash in blobDownload.release.
Hello boys and girls! I've had a Home Assistant instance running at home for many years now, connected to Nabu (def worth the money), and exposed to the interwebs through Nabu's service for mobile access and whatnot. Lately I've been starting to do a bunch more self hosting and g
I know everyone's excited about 3.8, but honestly I'm still refining 3.6. And I hate the way 3.8 talks, so... I keep seeing all these posts about what's the use of open source, so I figured I'd submit my own answer with this post. Qwen 3.5 27B was my first agent, starting my jour
Tracked my first run of MiMo Pro on a Jetson AGX Thor at a scorching 0.646 tokens/sec,.. but it DID technically load and run, so it counts. Every decoder layer, routed expert, SSD read (..a good bit more) traced and replayable step by step. Here we can't what-now?.. it's ~566GB o
I got Qwen3.8-Flash-Next-GSQ-RCO-Abliterated running at IQ3_S with just 12GB VRAM + 32GB RAM + NVME, achieving 40-50 tok/sec decode & 1300-1500 tok/sec prefill @ 131k context (44k thinking budget), on a custom fork of Strata. Don’t see IQ3_S and instantly dismiss it, this recipe
K3 came out in July hyped as the best open model you could download, and it scored 57 on Artificial Analysis, right up there with Opus 4.8 and GPT-5.5. I went back to see what running it actually looks like now, using Unsloth's quant docs, the model card, AA and the license. Shor
Article URL:
I've been maintaining my own chat template for Qwen3.8-Flash-Next since it released, and after a discussion in another thread I decided to share it. I made a repo with my v1.4 template, the original Unsloth fixed stock template, and now a "Sharp" terseness mode version (based on
ReAct is a common agent loop. What's the standard for System One decision models (Jev or a local alternative) + an LLM + tool calling? The examples I've been reading seem to split in two directions: confidence cascades in REFLEX and JevAgentHarness, and planner + chooser in the H
Hey r/LocalLLaMA ! We're excited to share ORL (On-Policy Residual Learning) , our new research on making diffusion language models (dLLMs) significantly faster without sacrificing generation quality. With ORL and our custom inference engine dlmrt , we've achieved: 2,788 tokens/s
Article · noted.lol
FileBrowser Quantum 2.0 introduces activity logs and improved file permissions, alongside a SQLite migration and important security fixes.
Everything is in the title. Does this computer worth it?
Article · llama.cpp
model : Granite Switch Decision ( #30676 ) feat(convert): Add decision.chat to support decision models that can also chat Branch: GraniteSwitchDecision AI-usage: full (Claude + Sonnet 5.5) Signed-off-by: Gabe Goodhart ghart@us.ibm.com feat(convert): Add conversion for Granite Swi
Plenio Music Production System 1.0.0 is out, and the version number is on purpose! Various releases in three weeks, lots of feedback and bug reports later, the whole way, from the first idea to the song people actually listen to, holds together. The templates, the nodes and the r
now you can decide with IBM, locally
TL;DR -- Interpretability might be way more tractable than we thought, and the guy who helped convince everyone otherwise is now saying we can understand models by directly studying neurons and their interactions after all. This essay rewrites fundamental assumptions about how ne
Article · llama.cpp
common: honor HTTP_PROXY/HTTPS_PROXY env vars in http client ( #21752 ) common: honor HTTP_PROXY/HTTPS_PROXY env vars in http client common_http_client() was ignoring standard proxy env vars, causing downloads via -hf and similar flags to fail silently in environments behind an H
Reddit from r/homelab.
Hi r/LocalLLaMA , we're releasing Tyger, a small model that handles full-duplex turn-control for voice agents. What it is Base: NVIDIA Nemotron 3.5 ASR Streaming 0.6B + 14M causal GRU on top. Every 80 ms, with no lookahead, it picks one of three states: - answer : the user finish
I evaluate new models by giving them actual work tasks that have already been completed by a trusted model and then comparing the results. My typical workflow is to plan tasks/requests in Gemini and create a markdown spec-sheet that outlines the work that needs to be done along w
I was running Unsloth IQ4_XS for a while and while it provided me with a healthy 96K context on 20GB vram, I did notice all the issues it had during draft and all the time and local tokens wasted trying to fix said issues. I decided to see what would happen if I used less context
Video · Techno Tim
Did you know Linux can cache NFS files on a local SSD? I somehow missed this feature, so I decided to try it out. In this video, I'll show you how to set up FS-Cache, CacheFiles, and cachefilesd on Ubuntu, how caching and eviction work, and how much faster repeated reads can be.
Ryzen 9600X, 32GB dual channel DDR5, 2x CMP 170HX Ryzen 5700, 32GB dual channel DDR4, 5x Vega 56 Intel i7 9700, 32GB dual channel DDR4, 3070 + 2080ti Ryzen 7500x3D, 32GB dual channel DDR5, 7900xtx + 2x 5600XT 223 GB total VRAM
Article · ServeTheHome
We review the Microsoft Surface Ultra to see how the NVIDIA RTX Spark powered laptop fares and try to answer who the platform is for The post Microsoft Surface Laptop Ultra 128GB Review appeared first on ServeTheHome .
Article · ServeTheHome
We review the Microsoft Surface Ultra to see how the NVIDIA RTX Spark powered laptop fares and try to answer who the platform is for The post Microsoft Surface Laptop Ultra Review appeared first on ServeTheHome .
I was comparing speed and settings between a macbook pro and my 4x RTX3090 rig, with a Q6 Qwen3.6 35b a3b model, with MTP. With 4x RTX3090 I was getting around decode of 70t/s I always thought " MTP must always be faster! right? " To my surprise, on my 4x RTX3090 rig was actually
Article URL:
Reddit from r/LocalLLaMA.
Article · noted.lol
Libreboot 26.10 gives older computers a second chance with modern UEFI support, new hardware compatibility, and expanded Chromebook builds.
I know I am probably missing the point of this benchmark, but I wanted to see if the model is capable of generating something better or is that the best it can do. So I gave it some encouragement. I find this with development on small models too. You really have to tell it in a l
Took vLLM-SR d3-edge (0.6B), fine-tuned it with simple SFT + LoRA, and compared it against the much larger d3 (27B). With just a few thousand training examples, the 0.6B model actually outperformed the 27B model on typed decisions and nearly matched it on Banking77. I strongly be
I wanted my finished illustrations to look like a page from a weekly manga, not just a halftone filter over a grayscale image. So I built a small node pack. How it works Qwen-Image 2.1 Turbo (edit) redraws the image’s values the way an inker would: dark colors become solid black,
Basically title. All labs seem to have agreed 24-27b is the minimum size they will go, nothing below that. I wonder when will we get the next sub 14b model from a SOTA lab
UPDATE llama.cpp support merged
On October 16, 2026, China’s Ministry of Commerce and Ministry of Science and Technology released a draft revision to the Catalog of Technologies Prohibited and Restricted from Export. It proposes adding 40 technologies across AI, biomedicine, and solar PV, of which 23 are AI-rel
The Nemotron-3.5 releases are particularly not so good at the pandas/numpy and DDDM (data-driven decision-making) sections of my domain-specific tests. Medium thinking on Bolt does worse than Nemotron nano. High thinking marginally better (not to a statistically significant level
There's also a base model. As all OpenMoss releases, they're Apache licensed. Wonder how this compares to Qwen Image 2.1 and Flux Klein 3.
Article · selfh.st
You either die open source, or live long enough to see yourself become open core
Article · Open WebUI
Added 🔐 End-to-end encrypted chats. Direct messages and private group channels can now hold a temporary end-to-end encrypted chat: a member picks Encrypted Chat in the channel menu, the other members see "Encrypted chat in progress" with a Join link, messages of up to 16 KiB are
Article · llama.cpp
hexagon: correct HMX FA sinks ( #30581 ) hexagon: correct HMX FA sinks hex-fa: vectorize sinks in hmx kernels Co-authored-by: Max Krasnyansky maxk@qti.qualcomm.com Website:
Article · llama.cpp
model : MTP support for Mellum2.1 ( #30612 ) model : MTP support for Mellum model : inline rope construction in Mellum Website:
Article · noted.lol
Learn how to serve your music with spotify-like features, but all locally.
Article · llama.cpp
[SYCL]: fix memory errors in mul_mat, split buffer, host pool ( #29889 ) Website:
with chatgpt recently launching intelligent ui, i've been looking into the different approaches to generative ui and wondering how well smaller local models would actually handle this. for anyone unfamiliar with the concept, the idea is pretty straightforward. instead of an llm r
Reddit from r/LocalLLaMA.
Article · llama.cpp
common : read ngram cache parts by reference instead of copying ( #30061 ) Website:
interesting focus: "A 3B active model beats Opus 5 at memory" (whatever that means :D)
Article · llama.cpp
cpu : reduce AVX2 tiled accumulators together ( #30296 ) Assisted-by: Codex Website:
Article · llama.cpp
ggml-cpu: refactor s309x repack helper functions ( #30588 ) Signed-off-by: Aaron Teo aaron.teo1@ibm.com Website:
Article · llama.cpp
sycl: wide stores for the q4_K/q5_K weight dequant ( #29696 ) Website:
It's clear that today’s SOTA (state-of-the-art) models remain ahead of open-source models. However, if we filter the benchmarks correctly, a different picture emerges compared to what we saw just a few months ago. Today, open-source models are outperforming Sonnet 5, a significan
Whenever it comes to Strata and low bit QFN discussion, everyone is talking about Unsloth, ISTA-lab and nobody mentions Agantai with his GYRO series of quants
Article · llama.cpp
tests : catch vanished rows in the fusion baseline ( #30618 ) tests : update the Metal fusion baseline Re-record the MTL fusion baseline with the current build: 276 rows over 131 dummy models. Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB) tes
Mellum2.1 Highlights Mellum2.1 is the next version of Mellum2 Thinking . The architecture is unchanged (a 12B mixture-of-experts model with 2.5B active parameters), and almost all of the work for this version went into post-training, primarily reinforcement learning (RL): Reinfor
Article · llama.cpp
vulkan: fix gpu-visible host memory issues ( #30468 ) vulkan: fix log spam if host pointer import fails exclude MoltenVK from shared memory import remove unreachable code, assert instead. Shorten warnings disable host memory buffer sharing on AMD due to amdgpu perf issue Website:
On many tests (math, science, medicine, etc.), they score about 10% higher, use half the tokens, run twice as fast, and cost much less than regular models. mod here, paper link: [2610.18861] Hybrid Reasoning Agents original post on tech_x. EDIT: This method is an adaptation for c
More t/s on CPU, For GPU poor people
Article · n8n
2.43.7 (2026-10-16) Features core: Suspend in-flight production executions at worker shutdown ( #40944 ) ( 5863179 )
Article · n8n
2.44.3 (2026-10-16) Bug Fixes core: Keep Observer detail lines instead of discarding the response ( #41157 ) ( c09e0a9 )
A couple of weeks ago I read this Wccftech article about Patrick Moorhead's 4 Spark setup. I meant to post this a few days ago but ran some more tests first, and then my workstation had a problem after a kernel and NVIDIA driver update. This post is long enough already, so that w
Article · llama.cpp
CUDA: HIP: rms_norm: add variants that load float4 at a time and stor…
Hey everyone, i built lgtmaybe, an open-source code reviewer, and it runs from Gitea Actions. It posts review comments on your Gitea PRs using a model on your own hardware, so your code never leaves your network. Setup is a token for a bot account, a workflow file and a model end
Article · MarkTechPost
Liquid AI released d1-ultra, a text and image decision model that returns calibrated probabilities in 1 forward pass. It scores 65 on the Artificial Analysis Decision Index and costs $0.042 per million input tokens. The post Liquid AI Releases d1-ultra: A Multimodal Decision Mode
Article · Paperless-ngx
paperless-ngx 3.3.2 Note This release addresses a security issue ( GHSA-c8f3-pqr2-rr24 ) and is recommended for all users. Our sincere thank you to the community members for reporting this. Bug Fixes Fix: copy only direct user grants in DocumentMetadataOverrides.from_document @st
Article · llama.cpp
graph : add precision handling for MELLUM architecture ( #30584 ) Website:
Title: I'm building a small local AI server to validate a project I've developed. The application currently runs Qwen 9B and the server will only be used for an initial pilot with a few users. I have to provide the hardware myself, so I'm trying to keep the initial investment as
Article · llama.cpp
hexagon: tensor repacking rework and support for tiled and flat quantized and fp weights ( #30635 ) hex-repack: Remove need_repack hack and streamline the rules -- breaks GET_ROWS tests for K-Quants, will fix in the next round Add support for GET_ROWS with tiled Q5_K Add support
Article · llama.cpp
Co-authored-by: Karthik Vijayan kavijay@microsoft.com
Article · llama.cpp
Limit max CUDA pinned host buffer size to 4 GB ( #30449 ) Website:
Alibaba's Qwen team released a new repository on Wednesday, October 14, 2026. Its goal is to measure how well visual agents can create 3D scenes in Blender based on real-world references.
Drama3 support in audio.cpp was contributed by
Article · llama.cpp
The scheduler uploads each view of a host input on its own, so the per-layer reshape of new_pool_idxs (and of sel_mask in glm5-next) was copied once per layer.
Diffusion language models are the future!
Article · llama.cpp
mtmd: add mtmd_audio_stream + support voxtral-realtime ( #30629 ) mtmd: add mtmd_audio_stream + support voxtral-realtime address review Website:
Article · KoboldCpp
NEW: Introducing koboldcpp.net, our new community website ! - KoboldCpp now has an official community website ! This is a legitimate community site put together with the help of @Autumnlight02 and aims to be a simple and helpful resource for newbies to get started, (and also to h
Article · llama.cpp
mtmd: lightonocr does not require longest edge to always be 1540, native resolution is the default ( #30600 ) Assisted-by: Codex Website:
Article · llama.cpp
Preserve the existing MoE accumulation order. Assisted-by: Codex Assisted-by: Angel
![b11676: [SYCL] Enable ssl in build script and CI (#30456)](media/15cad2b57331f2fe.png)
Article · llama.cpp
enable OPENSSL=ON as default in build script, update it in CI and dockerfile change to LLAMA_BUILD_BORINGSSL on windows refactor rm Download UI build Apply suggestion from
Article · Ollama
use identifiers for --ask mode question keys
Article · LiteLLM
Verify Docker Image Signature All LiteLLM Docker images are signed with cosign . Every release is signed with the same key introduced in commit 0112e53 . Verify using the pinned commit hash (recommended): A commit hash is cryptographically immutable, so this is the strongest way
Model: bartowski/Kwaipilot_KAT-Coder-V2.5-Dev-GGUF/Kwaipilot_KAT-Coder-V2.5-Dev-Q4_K_M.gguf Hosted via llamacpp on a potato(iGPU) and created in less than 10 minutes. Prompt given and was provided in one pass: ```Generate a small, simple, HTML/Javascript demo that looks impressiv
Article · llama.cpp
model: add Muse Spark Open (text, vision and audio) ( #30616 ) model: add Muse Spark Open (text, vision and audio) Muse Spark Open is a latent space MoE: the residual stream stays at 8192 while the 256 experts live in 4096, with a projection down and up around them, top-8 sigmoid
I'm new to the ai space and wanted to repurpose an old 1TB toshiba external for hosting a model. I've been recommended ollama but is it worth it or is there a possibly better model I could get? My laptop it'd host through would be has 32GB ram and a Nvidia Geforce rtx 1050 so it
Nemotron-3.5-Embed-1B-BF16 is a versatile embedding model developed by NVIDIA for multimodal and multilingual question-answering retrieval. The model can embed a document page in the form of an image, text, or combined image–text input. Documents can be retrieved given a user que
Follow-up to u/neuromacmd 's post from a few weeks back (
Article · ComfyUI
What's Changed Add ImageSDRToHDR node. by @comfyanonymous in #16990 feat: Add a core route for the default workflow (CORE-340) by @alexisrolland in #16956 Add ReferenceMultiLatent node. by @comfyanonymous in #17009 Disable the LoRA stack nodes until the frontend supports DynamicG
Article · noted.lol
plMail brings multiple email accounts into one self-hosted inbox, with Docker support and optional local AI through Ollama.
Yo guys! I know self-promo isn't great, but I wanted to show you an app I'm currently building with a buddy of mine. Maestro Opera lets you use any AI model/engine by integrating tons of AI providers like running locally on your phone, Ollama, API keys, etc. The app is 100% free,
I wanna see a master list of the craziest/strangest/funniest LLM tunes. I heard one was talking like an expert in 1800s military vernacular and I figure there's some wacky stuff out there. I want to explore weird and experimental models. Show me what ya got! Bonus points for NSFW
Going to try these out to see if they have the same looping behaviors that other Bonsai models have had. Still requires their fork of llama.cpp (branch prism) so dont expect it to work with vanilla llama.cpp. Edit: You can use with regular llama.cpp! Will see if there is some way
This quite a difficult issue and I quite literally exhuated all my options so I'm posting on here (which is something I never do lol). Basically, I am locally running a Qwen model that basically always thinks for like a whole minute before answering (even to questions like how ar
I posted a question the other day in here about good development models, which prompted good discussion. However, it mostly just reinforced my own experience, which is that on a 128GB machine, you can run good open/local weight models for development (especially if you use a fron
Granite Switch 4.1 30B 📝 Read the launch blog: Granite Switch 4.1: Chat, Decisions, and Speech from One Endpoint Model Summary: Granite Switch 4.1 30B is a modular LLM that composes a base model, multiple adapters and an encoder together so that a single hosted endpoint can move
I saw u/lkarlslund 's post just over a week ago regarding Ninfer6000 (a fork of Ninfer specifically tuned for 6000 blackwell and Qwen3.8-Flash-Next) and figured "Hey I'm an idiot & bought one of those and I primarily use that model, lets try it". I have a full blog post about it
First, kudos to Strata . It inspired this project, and while I was building it Strata got much faster and added NVFP4 support, which was my main reason for starting. By then I was too far in to stop, so I finished it. capy-flash is the evolution of my ninfer-tp2 engine. It still
Size is pretty small at 102GB. For comparison nvidia/Qwen3.8-Flash-Next-NVFP4 is 133GB. Would be cool to see some of the Qwen3.8-FN specific engines support this.
Well, what happens when you implement various sources of truths that have varying retrieval mechanisms? Some things get incorrectly outweighed than others. Man, I still don't think anything will replace any human doing quality QA work.
I started off with Ollama cloud and picked up the Mac Studio to see how it would compare. The $20 (old) plan is working well. I haven't ran into limits like others vs the $230/mo leased option. I think value wise the $20 ollama plan is/was good value. I'm not sure where its at cu
Article · Ollama
NVIDIA Nemotron 3.5 Super NVIDIA Nemotron 3.5 Super is a vision-language model for coding, reasoning, tool use, and document understanding. It pairs a 120B-parameter hybrid Mamba-2/Transformer mixture-of-experts backbone (12B active per token) with a C-RADIO vision encoder, and s
Waiting on AI coding agents kills my focus. You probably know what it feels like when AI agents drain you more and more. You send a prompt, watch a spinner for five minutes and probably scroll instagram or reddit along the way, read a wall of text, and do it again. So I built Fad
I turned a CHIP-8 game console into a decoder-only transformer. The transformer is the game engine. It executes instructions, updates machine memory, and renders every frame through next-token prediction. Every generated token is a memory write, and your keypresses go directly in
The ollama extension published by ollama for vscode (
TLDR: The open-source tool that lets local LLMs watch your screen just got decision model support! Thanks to your feedback, it’s now 4x faster at detecting things on your computer and no longer misses triggers. While running entirely on your computer. Hey r/ollama ! You guys are
Two months ago we released a fully quantized NVFP4 Qwen3.8-27B trained with QUASAR. We've since trained QUASAR checkpoints for a bunch of other recent models: Qwen3.5-4B, Gemma 4 E4B and 12B, and Muse Glimmer 30B, in NVFP4 for vLLM and Q4_0 GGUF for llama.cpp / Ollama / LM Studio
To all those who are bemoaning missing getting DGX Sparks for under $4000. Here's your chance. The brand new just announced RTX Spark laptop is on sale at HP. Configured with 128GB of RAM, it comes out to $3832.49. Use coupon anniversary25 to take 25% off of the $5109.99 regular
Hi folks, I built this for one small moment: while reading, I sometimes realize I need to understand a minor concept to follow what comes next. Googling it or opening a chat with an AI feels like overkill and easily pulls me down a rabbit hole. I wanted something light enough tha
Reddit from r/LocalLLaMA.
Looking at options to setup a whole home Alexa type AI home assistant that I don't have to worry about Jeff listening to me poo. I use models like Qwen3.8-27B for coding assistance but never played with Voice or other type of models. Does anyone know a good setup for something li
£299.99 Lenovo IdeaCentre Mini (Snapdragon X1-26-100, 32GB, CPU only): - Qwen3.6 35B A3B coder: reads ~118 tok/s, writes 24.5, ~30 with MTP - Qwen3-Next 80B A3B: 17-18 tok/s writes with MTP How I got here... So I bought one of these boxes to test if software at work would run on
I’ve built an app that uses Python and Selenium to run automated tests, to check whether all the key areas affected by the latest updates are working correctly. I was wondering whether, these days, with local LLMs, it’s possible to use some of the latest AI innovations. I have an
FLUX 3 Image Klein is a 5.6B open weights model for text-to-image generation and image editing. It takes a prompt and up to 10 reference images and returns one image. Text-to-image, single-reference edits and multi-reference compositions run through the same model, with Qwen3-VL-
Article · llama.cpp
Overview llama.cpp v0.7.0 adds the LiliCorr speculative decoding technique and MTP for GLM5-Next, a new GPU cache for MoE experts kept in host memory ( --moe-cache-mib , up to ~2.2x decode speedup when experts are offloaded), and a large batch of new models: K2 Horizon (dense and
Mixing Nvidia+AMD hardware via RPC usually meant split-mode layer only. I managed to keep split-mode tensor working, and patch my llama-rpc to send the drafters over to my strix. That let me go from 150k -> 195k context, at the same tensor parallel speed.. nice. My agents helped
Pulling the benchmarks from the model cards. Benchmarks might not be exactly comparable, as some models do not specify the version. Benchmark 3.5 Bolt 3.5 Super 3.8 27B 3.8 FN DeepSWE N/A 24.65 42.2 58.7 GPQA-Diamond 86.43 89.52 89.2 91.7 SWE-bench Pro N/A 51.78 61.7 62.5 Termina
Hi all, I've been following this sub for months now, but I recently started working again and missed some of the newer developments with Strata and new model releases since about when Qwen 3.8 27b first came out. What's the current best set of models / best model to run on a syst
Reddit from r/LocalLLaMA.
Yesterday I said its hard to play 4 spark with ling-3.1-flash but today its coming!
Article · LiteLLM
Verify Docker Image Signature All LiteLLM Docker images are signed with cosign . Every release is signed with the same key introduced in commit 0112e53 . Verify using the pinned commit hash (recommended): A commit hash is cryptographically immutable, so this is the strongest way
Article · noted.lol
Your Docker backups might be running perfectly, but can you actually restore your applications when something goes wrong?
Reddit from r/LocalLLaMA.
I was quite interested in setting up local AI models on my laptop. Was wondering which one might be the best. Specs: RTX4050 6GB 16GB ram I7 13700hx. My main use cases are coding, day-to-day usage like some queries and all. Automations if possible using hermes. Please help me wit
Article · Ollama
ci: shorten Windows Vulkan build path Keep the nested shader-generator compiler probes below Windows path limits in checkouts with longer repository names. mlx: fix Nemotron bugs and support global scales Accept published Nemotron layer names while retaining legacy aliases, and f
day 0 support for two new Nemotrons minutes later new Nemotron 3.5 Super VL was released:
chatting with a model doesn't tell you much here. the failure I'd look for is the call coming back as a json blob in message.content with tool_calls empty, which an agent loop just treats as a normal text answer is there a small script people use for this, or do you just plug the
Hi, I have the following specs: 110 Gigabytes of DDR4 RAM Two intel Xeon E5-2630 V4 No dedicated GPU, but fastfetch shows a nVidia geforce 210 discrete GPU. My goal in this project is to be able to run a modal equivalent or slightly better than gemini 3.8 flash, this is the only
Hi, I'm the author of GPT-Load, an MIT-licensed AI gateway with 7,000+ GitHub stars. It manages API keys and supported subscription accounts in one place. Your clients connect through the gateway, which handles credential scheduling, automatic failover, access controls, request l
Article · llama.cpp
models : add maion-coder ( #29778 ) Add Support to maion-coder arch iAdd Support to support Draft models Address comments Remove maion context shift function Co-authored-by: Karthik Vijayan kavijay@microsoft.com Website:
I was curious how quantization affects Finnish language ability and why Q38FN’s Q8 quantization got such poor accuracy numbers. So I set the machine running in the background. The Finnish corpus is three books from Project Lönnrot, and the English one is the same books from Proje
Article · llama.cpp
vulkan : fix concurrent first device init ( #29766 ) vulkan : fix concurrent first device init tests : load models in parallel in test-thread-safety Loading them serially never exercised concurrent first backend init, so the crash reported in #29764 was invisible to this test. vu
Hi, is anyone using Strata or Project-Maya with one or more AMD MI50? I wanted to try it myself, but I couldn't get it running.
Article · llama.cpp
vulkan : copy strided f16 KV once on AMD so reads spread across memory channels ( #27703 ) vulkan : copy strided f16 KV once so reads spread across memory channels Assisted-by: Claude (Opus 4.8 / Opus 5) tests : cover the f16 KV copy to scratch Assisted-by: Claude (Opus 4.8 / Opu
I continued to use Strata but something just feels off for this an large model, it is very fast but intelligence seemed even under 27B 3.8 UD Q4, it had a hard time coding a basic 3d game and a 2d platformer. Deepseek Flash V4 didn't have this huge and drop off at Q2 imo. Then I
Article · llama.cpp
metal : add BF16 variant of the tensor API flash attention kernel ( #30491 ) metal : add BF16 variant of the tensor API flash attention kernel cont : move tensor FA kernels to kernels/fa_tensor.metal Website:
I've been trying to figure out a few things with local models. When you talk to them they get similar answers than a lot of the top models. When I do coding and I give the same access to the files as each other... they get roughly the same answers which you would expect asking tw
Discovered by a third party… Tencent itself doesn’t seem to be talking about it, which is slightly concerning.
Article · noted.lol
A to-do app that keeps a full copy on every device, syncs through your own server, and fits in a 23 MB container. Set it up with Docker in a few minutes.
I remember great creativity from GPT-2. It was trained on a dataset that is orders smaller than current training datasets. The problem with overfitting a model to the training data is that it becomes unable to generalise; it is very good to predict the test set, but on fresh data
At only 0.4B parameters with a 16,384-token context window, this model combines versatility with knowledge and memory.
Article · llama.cpp
graph: keep the graph non empty without embeddings ( #30522 ) An embedding-only graph has no logits, and build_pooling() returns early when embeddings are off, so since the models no longer expand t_embd the graph ends up empty and its inputs are never allocated. Expand t_embd in
Putting alternative in quotes because well it will forever be an uphill battle porting things to the Axera framework, but Axera has very strong chips so I think it may still be an interesting offering for somes
I see everyone improving decode with bespoke models, like Sushi or Splash with Qwen 27b and Qwen Flash Next, and I'm running QFN now at 40tk/s on my M3 Max which is awesome. But prefill feels stagnant. I was trying to work on layer / attention propagation / approximating KVcache
this is actually interesting:
Reddit from r/LocalLLaMA.
Open-weights ASR + full-duplex turn control in one model. It hears when you finish, when you interrupt, and when you're just saying "mm-hm". • 250 ms median endpoint latency, 80 ms frames, no lookahead • Full voice agent tyger-s2s: TYGER + Gemma 4 E4B + VoxCPM2, one 32 GB GPU, up
Article · The Rundown AI
PLUS: Build a live sales dashboard in Claude
Will there be a black-friday this year?
I calculated KLD and perplexity for Qwen3.8 27B models that fit 16GB VRAM: See comments for the benchmarks. Reddit filters keeps deleting the post. Conclusion: q5_1/q5_1 is almost always better than q8_0/q4_0 or q4_0/q8_0. There is very little difference between q8_0/q4_0 and q4_
New open model to check out ! Doing great on agentic, focused on arabic. From 0.9B to 375B ! Weights :
Article · n8n
2.43.6 (2026-10-15) Bug Fixes core: Create an XML parser for each webhook request body ( #41113 ) ( ea3ee53 )
Article · llama.cpp
server: save only restorable slot states to the prompt cache ( #30498 ) server: save only restorable slot states to the prompt cache The idle-slot save loop stores every idle slot whenever a task starts, while only completion tasks restore from the prompt cache. Rerank, embedding
Yesterday Fish released this incredible text-to-speech model, so naturally I spent the rest of the day building a great demo for them :D The model is a chunky MoE (35B3A), but it runs well on a DGX Spark. The 4bit quants get very close to the original. You can try it here:
Article · n8n
2.44.2 (2026-10-15) Bug Fixes core: Create an XML parser for each webhook request body ( #41112 ) ( c52f181 )
Article · llama.cpp
spec : add lilicorr support ( #30506 ) spec : add lilicorr support convert : use convention for hparam assignment add todo for hparam use in llama-batch.cpp Co-authored-by: Georgi Gerganov ggerganov@gmail.com typo fix Co-authored-by: Georgi Gerganov ggerganov@gmail.com Website:
I'm planning on maxxing out my pcie slots with rtx 6000 blackwell cards for tp4. I'm wondering if its possible to use an egpu dock like the razer v2 and connect it via thunderbolt to usb 4 on the back of my mobo. The egpu in question would be another blackwell card (5070ti), and
Article · LiteLLM
Verify Docker Image Signature All LiteLLM Docker images are signed with cosign . Every release is signed with the same key introduced in commit 0112e53 . Verify using the pinned commit hash (recommended): A commit hash is cryptographically immutable, so this is the strongest way
Article · llama.cpp
vulkan: Use L2-friendly workgroup scheduling for cm2 mulmat ( #30215 ) Website:
Article · llama.cpp
opencl: support CLAMP on non-contiguous views ( #30186 ) The clamp kernel indexed source and destination as one flat contiguous array, so a strided view as input (the new CLAMP v=1 case) read the wrong elements. Add a strided kernel_clamp_nc, as the other unary ops have, and take
Step 5 Preview is StepFun's flagship model for agentic work. It delivers frontier-level performance across software engineering and professional knowledge work, with particular strength in finance.
Article · MarkTechPost
TypeSafe AI released Jev 1.14 under the jev-preview alias. The System One model adds experimental image input, a Box bounding-box type and an opt-in 262k-token context, at $0.042 per million input tokens. The post TypeSafe AI Releases Jev 1.14 With Images, Box Type and 262k Conte
Hello, I've never used a local AI before, but at this point I've used damn near $10,000 just on Claude in a single week which is unacceptable for my projects. I've heard ollama is good. How do i start? I have a 5600X3D + RTX 5060 and i have a spare 3090 thats headless/has no disp
Reddit from r/ollama.
Article · llama.cpp
llama: honor PARTIAL_ONLY in the plain KV cache A non-SWA llama_kv_cache rolls back with seq_rm, so it holds no partial state, yet state_write ignored the flag and serialized every cell. A draft context made of a plain KV cache (MTP head, dense -md draft) therefore copied its who
Saw soneone post their bathroom AI PC a few weeks ago. Figured I’d contribute with my chicken-coop adjacent shed AI PC. Started with a single CMP 170HX (unlocked to 64GB) and ended up with three more for a total of 256gb VRAM. Found a deal on a used Threadripper with a TRX40 moth
Article · llama.cpp
ggml-cpu : keep the FP16->FP32 lookup table on x86 with F16C ( #30342 ) ( #30343 ) ggml-cpu : keep the FP16->FP32 lookup table on x86 with F16C ( #30342 ) #30270 routed both scalar conversions through F16C. For FP16->FP32, Clang 20.1.8 lowers the intrinsic to a vpinsrw merging in
Article · llama.cpp
ggml-cpu : size set_rows conversion buffer for all threads ( #30454 ) ggml-cpu : size set_rows conversion buffer for all threads SET_ROWS is planned with n_tasks = 1, but every thread runs it and uses its own row of the F16 -> F32 conversion buffer, so threads > 0 wrote past the
2nd mortgage | 27B IQ1_S" title="8GB GPU > 2nd mortgage | 27B IQ1_S" />
Article · llama.cpp
cuda, hip: GDN chunked kernel ( #29353 ) cuda, hip: GDN chunked kernel Ported-from: tetherto#257 Source-commit: 53f0027 Co-authored-by: Aman Gupta aman.gupta@tether.io fix older builds review fixes cuda: enable GDN MMA on tested Ampere and Ada GPUs Enable SM86 and SM89 after RTX
Looking for people who have tried this model and was able to use it on real-world tasks. It's what Strata list as "coder". What was your experience like? On my setup I am not really sure if it is worth more than iq3
Hey everyone! I've been thinking for a while, and I have an army of enterprise thin clients sitting in my basement so i thought to myself "wish i could do something with those". And thats how i got this crazy idea: Could i run gemma4 e2b on just one of these thin clients and achi
Just downloaded the new LFm2.5-24B-A2B LiquidAI model for testing. They claimed some pretty good performance, and I've had good experiences with their prior releases. Some context - I use Ornith 1.5-35B pretty regularly in multiple workflows - sometimes on it's own, and sometimes
Reddit from r/LocalLLM.
Article · llama.cpp
ggml: fix conv_1d result layout for batched input ( #28738 ) ggml: fix conv_1d result layout for batched input The GEMM behind conv_1d folds the batch into its rows, so the product sits in memory as [OL, N, OC] while the reshape claims [OL, OC, N]. The shape is right and the cont
I saw the 4x R9700 build for a 10-person startup here a few days ago and wanted to share my results, since I opted to spend most of the money on the cards. I went with a super cheap used EPYC instead of a Threadripper. In the end it came to about $10.7k per box all-in, and I buil
Article · llama.cpp
ggml : add note about mul_mat_id + disable mul_mat_id_dup tests ( #30501 ) Website:
![v0.32.0rc1: [Dependency] Update Humming and remove NVFP4 scale workaround (#61556)](media/a264c4af5885aa0a.png)
Article · vLLM
Signed-off-by: jinzhen.ljz jinzhen.ljz@antgroup.com Co-authored-by: Codex codex@openai.com Co-authored-by: Misha Goin mgoin64@gmail.com Co-authored-by: Kevin H. Luu khluu000@gmail.com
Reddit from r/LocalLLaMA.
Article · noted.lol
I wanted to see if ChatGPT could really build an entire self-hosted application. What started as a simple voice recorder turned into a personal journal powered by local AI. The results genuinely surprised me.
As strata is somewhat new, I assume that some people din't yet learn how to finetune it for best performance on their machine. I find documentation a bit hard to navigate (use your agents for this!), which doesn't help. After some serious testing, I've tweaked pretty much every p
Anecdotally, I've noticed that QFN seems more collaborative than GLM. QFN seems willing to engage in conversation. It feels more like an assistant. GLM seems more action-oriented and less willing to converse. That is, I ask GLM a question to engage in discussion: GLM runs off sea
Article · LiteLLM
Verify Docker Image Signature All LiteLLM Docker images are signed with cosign . Every release is signed with the same key introduced in commit 0112e53 . Verify using the pinned commit hash (recommended): A commit hash is cryptographically immutable, so this is the strongest way
Article · Open WebUI
Added 📑 MinerU 4.x servers. The MinerU document engine now works with self-hosted MinerU 4.x servers, where every document failed with a 404 because 4.x removed the old parsing interface; 3.x servers work as before, 4.x servers take a tier (flash, basic, standard or advanced) and
tl;dr: new qwen finetune that simultaneously pushes the frontiers of small model capability and reasoning efficiency. Hey everyone, We publish two post-trained versions of Qwen 3.8 27B called Sabi-Noema and Sabi-Noema-MLA . These models push the capability frontier on These model
I keep seeing recommendations to use Qwen3 models for limited devices. While there are already more advanced models like 3.5 or 3.8 with small weights. Is there something I am missing regarding using plain v3 of Qwen instead of 3.5 or 3.8?
MacStudio 128GB M1 ultra I'm curious how people who can run both models are deciding which one to work with. DeepSeekv4 benchmarks right below GPT 5.5 high. Flash Next benchmarks right above 5.4/5.5 xhigh Both seem capable but it seems like Qwen 3.8 Flash next is taking much long
Article · llama.cpp
ggml : improve RPC tensor transfer logs ( #30481 ) add rpc_log_format_size() and print byte counts in KiB/MiB/GiB across the RPC client, server and transport traces add rpc_log_format_tensor() and include tensor name, op, type and shape in the set/get/memset/copy/init tensor logs
I've used this to watch shows and movies that have never been translated to English. There's so much great content out there, that is now accessible thanks to AI. Basically you clone the repo and tell your local model (or non-local model) to direct the dubbing process; the rest o
If you've chained H3 clips into a longer video, you've probably seen it: every continuation copies the last clip's flaws and adds a few more. By clip 5–10, faces look crunchy, etched and over-sharpened. I spent the last few weeks (with a community compute grant) training a LoRA t
Reddit from r/ollama.
Article · llama.cpp
cuda : use SIMT implicit GEMM conv2d on AMD without WMMA ( #30325 ) cuda : use SIMT implicit GEMM conv2d on AMD without WMMA Assisted-by: Claude Opus cuda : update conv2d SIMT comment cuda : skip SIMT conv2d on AMD for short reductions Co-authored-by: thelittlefireman 5165783+the
Article · llama.cpp
jinja : write floats in tojson as Python does ( #30362 ) tojson printed floats through a C++ stream (6 significant digits, no ".0"), so 48.858844 became 48.8588 and 1.0 became 1 in tool definitions, tool-call arguments and other template output. Write the shortest digits that rea
Looking into self-hosting a LLM. I have an elitedesk mini G9 with 32gb ram and an intel i7 that I want to add a eGPU to and I’m trying to decide what GPU/setup is the right one. I see that Newegg is selling an Aorus box with an RTX 5070 Ti 16gb for about 1000 bucks. Is this worth
Article · llama.cpp
opencl: use buf+offset instead of subbuffer for adreno moe expert sort ( #30418 ) Website:
was dropped yesterday in case you missed it full 32bit support and previews go crazy:
Article · llama.cpp
opencl: fix q8_0 dp4a mul_mat_id and mark PREC_F32 mul_mat_id unsupported for adreno ( #30365 ) opencl: fix min handling for moe q8_0 dp4a GEMM Co-authored-by: Hongqiang Wang wangh@qti.qualcomm.com opencl: reject PREC_F32 MUL_MAT_ID for adreno Co-authored-by: Hongqiang Wang wangh
In the long forgotten past (6-7 month ago) I was able to do crazy amount of work with 20$ sub, I never even reached my weekly and 5 hour limit once. After retrying this week, with similar amount of usage I am quickly reaching limits. The cut is real, like 5 or 6 times less. Won't
Article · llama.cpp
models: drop the t_embd expand of the embedding graphs ( #30461 ) These graphs only produce embeddings, and build_pooling() already expands its output for every pooling type, so the expand at the end of the model graph is redundant. Website:
Article · noted.lol
Self-hosting was supposed to give us more control, not another monthly bill. As paywalls become more common in self-hosted software, users are questioning what free really means.
Fish Audio Drama 3 , converted for the native ComfyUI nodes of ComfyUI-Drama3 : a 35B text-to-speech model for voice cloning, dialogue and inline cues such as [sighs] or [whispers] , running on cards from 8 GB . 90% of the model is routed experts and each audio frame uses only 8
from a 16gb gen8 microserver to 2 dl380 gen9 (one my dads with 192gb of ram and one mine w
Is 12GB “enough for everyone” now? Can you check your peak VRAM, while using those comfy options? Also, check the megabytes per second, being read from your NVME (in Task Manager or Gnome Resource Monitor)
This new Mr. Chatterbox checkpoint is both pretrained and fine-tuned on Victorian texts for maximum historical accuracy. I've posted questions about this project before in this community, and I really want to thank this community for helping answer my questions. You can also a na
I downloaded a few workflows and I've noticed that there were parameters added to the video helper suite node that is not visible when it saves the video, if you click on preview video, it will show you the full link there, in your JSON workflows, look for something like cos_url.
I started hosting things myself about 5 months ago. It began with classics like Immich and Vaultwarden. The setup went well until I hit my limits when it came to integrating network-related features like Crowdsec or automated offsite backups. That’s when I started using Claude Co
Hi all, I would like to try MiniMaxH3. What is a great Workflow template for my specs please? AMD Ryzen 7 8700F - AMD Radeon RX 7700 16GB - 32GB DDR5 - 1TB PCIe 4.0 SSD - Linux (Ubuntu 24.04 Kernel 7.2.7-x64v3-xanmod1). Thanks in Advance!
Tessera-27B-B-CJK is a Qwen3.8-27B coding agent that thinks in compressed CJK, and it spends fewer tokens doing it. Open weights, Apache-2.0. This is our first full "Tessera" lineup model drop! - Hugging Face:
Made a small ComfyUI → PureRef connector. Generated images go straight onto your board, arranged in rows. Uses the regular Save Image node, so there’s nothing to add to your workflow. Windows only.
Every local setup I tried for Qwen-Image 2.1 meant a Python env, a node graph and an evening of picking quantization, offloading and resolution for my card. Kokon is a free, open-source desktop app that does all of that for you and gives you a plain photo editor on top. Zero setu
Article · LiteLLM
Verify Docker Image Signature All LiteLLM Docker images are signed with cosign . Every release is signed with the same key introduced in commit 0112e53 . Verify using the pinned commit hash (recommended): A commit hash is cryptographically immutable, so this is the strongest way
Article · Immich
Well, you know how it goes... Misclick once and a release is sacrificed 😅 The second patch release v3.3.3 has some more bug fixes, notably a major performance improvement for the explore page. What's Changed 🐛 Bug fixes fix(mobile): show the memories date in every app language by
[UPDATE / RETRACTION: The "100% GTO" claim had a prompt-leakage bug] We patched the repo to pass strictly neutral states, here are the real, honest numbers: • Latency: Still ~2ms (ONNX + Absorbing Markov Chain). • True Zero-Shot GTO Parity: ~15% - 29% across all 340 canonical han
Hello! We've been busy at Telperion with a lot of experimental research (more coming soon). Today, we're announcing the release of our latent attention variants of Qwen3.8-27B. Our latent attention variants (quants calibrated to FP8 KV cache) compress the KV cache by 4×, enabling
Article · llama.cpp
sycl: persistent IQ3_S/IQ3_XXS weight reorder for SYCL decode and prefill Add persistent reordered weight layouts for IQ3_S and IQ3_XXS. Weights are reordered once at load time; the reordered MMVQ dot products then walk dense super-blocks with dp4a and SWAR sign expansion instead
I spent some time taking best practices from the past two years developing my projects with agent skills and centralized them in a repo so I can reuse them and standardize my projects. In the process of doing that, I realized that skills exist in different scopes and timelines, s
Article · Open WebUI
Added 🧱 External apps in channels. Administrators can now turn on Apps in Admin Settings > General, or with "ENABLE_APPS", and connect external apps in the new Apps tab of Admin Settings by their HTTPS address, reviewing the permissions each app asks for, such as reading its ment
Sampling: temp 1.0, top_p 0.95, top_k 20 for both. Run with evalscope over the OpenAI-compatible API: Qwen3.8-Next on Strata (MI50 32GB), Qwen3.8-27B on llama.cpp (RX 7900 XTX 24GB). Caveat: These are only a portion of the full sets, since they take days. Edit1: Strata started wi
Article · noted.lol
n8n has released security updates addressing multiple vulnerabilities affecting self-hosted installations. The flaws could expose sensitive credentials or allow unauthorized access to project data. Administrators are urged to update to the latest patched version.
Here a quick video i did to test my working in progress app to create minimax h3 short story´s prompts with characters sheet´s, etc
Just saw this demo from OpenBMB for MiniCPM-V 4.7 , and the on-device capability is pretty impressive. A 35B-A3B MoE vision-language model running locally on mobile devices is already interesting, but the use cases are what caught my attention: 📷 Reading real-world signs 📄 Unders
targeted ads are crazy
It runs at around m4 16 at 60 tps at full 130k context, you can easily run a Q8 if you have a 16 GB GPU, I was able to get 140TPS on 9070xt, It’s an MoE so I think we can run it well on smaller cards too, 8GB, 12GB should be possible to get upwards of 30-40 tps. I have used it a
Reddit from r/LocalLLaMA.
With 560B total parameters, 25B active parameters per token, and a 1M-token context window , wait what? 3.0 was just 127B
I'm having to edit a whole lot of pictures right now, small little changes over and over. If you're doing this then there is no reason to run the entire image through a model. There are node packs like AusBoss which has a crop for inpaint and stitch inpaint node.\ Basic WF would
This model is a 74B parameter, 9B-active-parameter mixture-of-experts (MoE) model, packed into 24GB using 2-bit quantization with advanced quantization-aware training techniques to retain quality. It aims to achieve the optimal balance between speed and intelligence, being faster
Performance on Model Prompt tokens master tok/s pr tok/s Change RTX PRO 6000 Blackwell Qwen3.5-35B-A3B-Q8_0 2048 13,653.58 16,459.56 +20.6% RTX PRO 6000 Blackwell Qwen3.5-35B-A3B-Q8_0 4096 14,090.26 16,595.47 +17.8% RTX PRO 6000 Blackwell Qwen3.8-27B-Q8_0 2048 4,470.39 4,757.20 +
HyCanvas is a browser design tool in the Canva and Genspark space: social graphics, slide decks, video editing, whiteboards, docs, and sheets, with realtime collaboration. Decks can be generated from one sentence, and every generated slide stays fully editable. The self-hosting s
If there was any doubt left that your ChatGPT chats are at the very least accessible to the police ... this should resolve that.
Its currently THE BEST tts model available locally but sadly it has a non-commercial license. For free to use license: moss tts 1.5 is the best model rn.
Can anyone think of a good use for this if it ends up being true? Unless the workload can somehow be split between the NPU and a dedicated GPU, I suspect it'll only be able to run small models.
Another quick Minimax H3 news and goodies round-up, for those who may have missed some items. -> ComfyUI Portable is now officially at version 0.40.0. There's a new official 'Camera Angle' node, and... "MiniMax-H3 single images: Single images are now decoded as the first latent o
Source: whatmodel.sh , an open source AI leaderboard I built (and launched here yesterday). Every point comes from published benchmark results: 22,113 of them across 987 benchmarks, from 1,011 sources, each one linked to where it was published. This chart uses data for the text m
I built BitRun because I wanted to run 1-bit ternary LLMs on ordinary laptops without melting my battery, spinning up a dedicated GPU, or wrestling with complex build environments. Right now, BitRun gives you 98 to 127 tokens/sec on an Apple M4 Pro CPU and 40 to 45 tokens/sec on
"--expert-cache", "2000", "--prefill", "auto", "--spec", "2", "--max-context", "176128", "--kv", "q4_0", "--resident-experts", "--kv-resident", "20480", "--pcie-frac", "0.00", "--spec-min-p", "0.70", "--adapt-every", "1", "--adapt-swaps", "80", "--adapt-decay", "0.97", "--vram-re
We just shipped web search in Private Mind, our open source app for running LLMs locally on iOS and Android. The video is the whole pitch: Qwen 3 1.7B with web off makes up Apple's September event, with web on it gets it right and cites 5 pages. The fun constraint is that a 1.7B
I wanted a quick way to push things from my browser into ComfyUI without switching windows, copying prompts, or saving images first. So I vibe coded a small extension for it. What it does Select text on any page, right-click, Send selection to ComfyUI . It fills your prompt node
a new option to optimize your VRAM
This continues the small-model browser experiments from the previous posts. Two standard, off-the-shelf models ran locally on the same Samsung Galaxy S21: Gemma 4 E2B, Q3_K_M as the reasoner Qwen3.5-0.8B, Q4_K_M as the executor No fine-tuning, no task-specific training, no adapte
Reddit from r/LocalLLaMA.
submitted by
I have spent about $10,000 on Ollama credits and have developed over six projects which use its API. When I got in touch with support to complain about the interruptions to the API, my account was put on restriction. I am not able to prove that the complaint caused the restrict
Links Official changelog GitHub release New Open-Source Model Support Qwen-Image 2.1 Turbo : The 8-step accelerated Qwen-Image 2.1 checkpoint, with new text-to-image and image edit templates New Node Updates LoRA stack loaders : Apply several LoRAs in order from one node, each wi
Article · llama.cpp
unicode : add opt-in NFC normalization for Unigram tokenizers ( #30390 ) unicode : add opt-in NFC normalization for Unigram tokenizers Note: used help from Codex, Sol 6.1 Medium Fixing initial bot comments + linter error (using existing out instead of print) removed test_nfc_toke
Hey everyone, I recently put together a local LLM machine using mostly second hand hardware. I wanted to experiment with local models for app development without spending a fortune on a new workstation. Here's the setup: • 2 x used RTX 3090 24GB • Ryzen 9 5950X • MSI B550 A PRO m
Article · MarkTechPost
Marktechpost ranks the 9 best AI gateways for 2026. We compare Bifrost, Portkey, LiteLLM, Kong, TrueFoundry, Cloudflare, Vercel, OpenRouter and Orq.ai on latency, pricing, licensing, governance and deployment. The post 9 Best AI Gateways in 2026: Bifrost, Portkey, LiteLLM, Kong a
Article · n8n
2.43.5 (2026-10-14) Bug Fixes core: Validate the size argument of the chunk() expression extension ( #40911 ) ( 4bf0ece ) Features Show Gemini models by name and list prioritized models first in model pickers ( #41016 ) ( 66cece2 )
Article · n8n
2.44.1 (2026-10-14) Features Show Gemini models by name and list prioritized models first in model pickers ( #41015 ) ( 321b020 )
RTX 5090, once a consumer gaming card, is now on smugglers’ menu alongside cigarettes" title="So this is how they do it --> RTX 5090, once a consumer gaming card, is now on smugglers’ menu alongside cigarettes" />
Article · llama.cpp
When every device reports zero free memory the split points are normalized by a zero sum, the NaN split sends layers to a device index past the end and loading aborts with vector::_M_range_check.
Article · llama.cpp
Signed-off-by: Adrien Gallouët angt@huggingface.co
Article · llama.cpp
The RDNA3.5 config table selects a 64-row / 128-thread tile up to J=32 and a 128-row / 256-thread tile above that. On gfx115x the smaller tile leaves enough LDS free for a second workgroup per CU, while the larger one does not. Extend the 64-row tile to the (type, J) pairs where
Article · llama.cpp
vulkan: Define the shader count for Intel Arc Pro B65 + B70 ( #30431 )
Article · llama.cpp
vulkan: support bf16 src1 in mul_mat clean up mul_mat b-type selection fix conditions
I've seen this one floating around quite a bit now and since this is something that can be done without the parent and or other dependencies i thought i'd share something. Usually when you fine-tune a model you can't: - train against the quantized GGUF file itself - do it without
We maintain an updated list of official MCP servers repo: 280+ well-known and widely used projects, all from the companies behind the products or actively maintained open source. Feedback and PRs welcome. github.com/VoltAgent/official-mcp-servers
I'm the author. aimm watches Facebook Marketplace for the items you describe, has an AI rate each new listing against what you want, and notifies you of the good ones by email, Telegram, Discord, Slack, Pushover, ntfy and others. What the AI does: Rates each listing against your
I've been working on an inference engine called jitllm. The idea came from my background in operating systems and JIT compilers: generate kernels for whatever hardware is there, and treat layers a bit like processes the engine can place and move around. In the video, a cloud VM w
Article · LiteLLM
Verify Docker Image Signature All LiteLLM Docker images are signed with cosign . Every release is signed with the same key introduced in commit 0112e53 . Verify using the pinned commit hash (recommended): A commit hash is cryptographically immutable, so this is the strongest way
I got my mac mini m6 32GB today... Wanted to find the fastest config primarily focused on incoai's Splash vs trymirari's Uzu. Results were pretty similar so either choice is good, I am going to run Splash as they support uncensored out of the box.
We run three small models for our RAG setup, embeddings, a reranker, and a small extraction model, and each one has been sitting on its own box provisioned for peak load. Last week I finally stopped guessing and actually measured the utilisation across them, and it came in low si
Can't even play a game or watch youtube on my PC or else it'll take up more VRAM. How many hours have you wasted just sitting in front of PC waiting for it to be done? What do you do while you wait? Should I get a separate mini-PC to play games with?
This is an offline app with 51 hands-on labs for learning how AI actually works, from neurons to agents and more... **A bit of insight** - 51 labs in 6 groups, from the basics (what a neuron is, gradient descent) through attention, RAG, agents, fine-tuning, quantization, serving
Article · llama.cpp
Enable OpenSSL in FP32/FP16 SYCL builds to restore HF HTTPS downloads…
Article · Ollama
Decision models in the CLI Decision models can now be used directly from the terminal with ollama run - ask yes/no questions, classify text into labels, or score against a rubric: ollama run nimble --ask "Is a refund being requested?" What's Changed Kev decision models are now su
Just curious. My bill today was $456. Of course its "summer" time and we run the A/C a lot, but in past years its never been over $350. I'm thinking about hooking up one of those watt measuring devices just to see what my PC uses in a typical week. I'm probably running my GPU ful
Article · llama.cpp
webgpu: (Fix CI) fix mul_mat_id with duplicate expert ids ( #30451 ) Fix mm-id of webgpu in the case of ids duplication formatting fix comments to avoid breaking up sentences in comments Website:
Article · llama.cpp
server : escape the special tokens without std::regex ( #30243 ) The d1-omni and kev prompts escape " " in the request text with std::regex_replace. libstdc++ recurses for each char of the name, so a state of " and llama-server dies with SIGSEGV. server_decision_escape_special_to
Article · llama.cpp
chat: replace output string with parts ( #30389 ) chat: replace output string with parts reserve first Website:
Article · LiteLLM
Verify Docker Image Signature All LiteLLM Docker images are signed with cosign . Every release is signed with the same key introduced in commit 0112e53 . Verify using the pinned commit hash (recommended): A commit hash is cryptographically immutable, so this is the strongest way
Best advertisment for LocalAI lol
Trying to read on what the real tested downsides have been but information seems to go both ways, some proclaim no loss, others pretty huge ones. I had some weird looping issue in their chat, and some engine port misunderstandings, seemingly the models likes to use ports already
Many have been wondering if Meta has abandoned plans to release Muse Spark as open weights. Recently, Meta launched the new Muse Wonder model, and in the X thread, confirmed that Muse Spark is soon to be open weights. So that's something to look forward to in the next couple of y
1) Reverse Engineer the Code 2) AI Feature Discovery: Ask an LLM to scan the features and map out what can be exposed via the Model Context Protocol, brainstorming creative use cases along the way. 3) Automated Integration: Let the LLM write the wrapper/server code. 4) Total Cont
tl;dr: running Qwen3.8FN Q4 locally is cheaper for me than every cloud model except Luna and the newer Haiku at mid-peak and peak electricity pricing where I live, and becomes the absolute cheapest model at off peak pricing. An agentic run that spans these different pricing point
Hi Friends, I am a solo developer. Ever since I had Qwen 3.8 running on my RTX4090, I had an itch to use it on my phone for everything. That gave me the idea to build SteerClaw Mobile - a complete agentic experience on the Phone where the entire agent loop runs on it. We just con
Article · MarkTechPost
Meta released Muse Wonder and Muse Wonder Full, 1M-context agent models. Muse Wonder costs $4/$20 per million tokens on Meta Model API. Wonder Full, on Muse Code plans, leads DeepSWE and SWE-Marathon. The post Meta Launches Muse Wonder and Muse Wonder Full: Benchmarks, Pricing an
Article · llama.cpp
ggml : simplify backend search path error handling ( #28367 ) Signed-off-by: Adrien Gallouët angt@huggingface.co Website:
Article · llama.cpp
kleidiai: add SME2 Q4_K and Q6_K matmul support ( #29219 ) Change-Id: Ib4aa4c9814c010f9cf084783cc80612e364f9a27 Change-Id: I406a6606c3fea51f8cd32331c1ed71466845d363 Change-Id: Ib4aa4c9814c010f9cf084783cc80612e364f9a27 Signed-off-by: Charles Xu charles.xu@arm.com Website:
Article · llama.cpp
vulkan: Optimize pool_2d when the pooling window is large ( #30119 ) When the window is larger than a subgroup, use a whole subgroup to compute each output. Website:
Article · llama.cpp
meta: recreate all subgraphs when the arena is reallocated ( #30453 ) meta: recreate all subgraphs when the arena is reallocated chore: revert added test cases Website:
One of the frustrating things about local AI generation on Mac is getting different models and tools to work together smoothly and fast. Valtz is a new native macOS app powered by Vpipe, a high-performance inference engine optimized for Apple Silicon. Instead of manually building
Article · llama.cpp
hexagon: add configurable HTP DCVS/HMX power mode ( #30423 ) hexagon: add configurable HTP DCVS/HMX power mode hexagon: address PR review feedback (drop CLI flag, align QAIRT power modes, comment/log cleanup) hexagon: add --hex-power-mode flag to run.py Hooks GGML_HEXAGON_POWER_M
Article · ComfyUI
What's Changed [Server.py] Update CSP to allow blob: and data: by @S0yKaf in #15565 Add comfy_attention and AttentionTensorContainer support to rest of models by @comfyanonymous in #16806 Implement the asset export API locally (CORE-454) by @jtydhr88 in #16578 Report 3D models fr
DeepSeek guys, thank you so much! Don't give this up. mac setup [pi.dev] -> [ds 4 flash 0731 - brain] -> [qwen 3.6 35b a3b - body] - works perfect
Article · llama.cpp
graph: pad the KQ bias of the no-cache attention ( #30446 ) The no-cache attention pads K, V and the mask to 256, but T5 also adds a relative position bias of [n_tokens, n_tokens], so the add with KQ fails unless n_tokens divides the padded width. Pad the bias with zeros up to th
Hey all, First of all, apologies for the mobile formatting, will try and fix it in the morning. I'd like a second opinion before I spend real money in a GPU market that seems to go up every week. Current setup Xeon E5-2690 v4, 128GB DDR4-2133 ECC (quad channel), ASRock X99 Killer
Sadly, all the test were done using NanoGen , but maybe someone can try the idea on Krea 2 or Qwen-Image 2.1. Does our method work beyond ImageNet generation? Yes. We test it on NanoGen , a unified diffusion transformer implementation for ImageNet and text-to-image generation. Th
Article · llama.cpp
model: load YaRN beta metadata for laguna models ( #30331 ) Assisted-by: Codex Co-authored-by: Adam Caldwell accaldwell@users.noreply.github.com Website:
Are we seeing early signs of the bubble popping?
Article · llama.cpp
graph: skip the LM head for pooled embeddings ( #30445 ) graph: skip the LM head for pooled embeddings A context that extracts pooled embeddings outputs one vector per sequence, but every model still expanded its LM head, so the graph computed n_vocab logits per token and the con
this lora is for minimax h3 if you want to generate video to mimic playstation 2 graphic cutscenes. download:
Wanted to share my journey of using H3 minimax and the frustrations of trying to force the h3 minimax reference to video template workflow to use fully_copy ref_audio_0 with mixed results. I had a lot of laughs while making this. I would snip a still image from the movie and then
Article · llama.cpp
tests : consolidate context state tests into test-llama-context ( #30023 ) tests : consolidate the context state tests into test-llama-context test-save-load-state, test-recurrent-state-rollback and test-state-restore-fragmented check the same object from three places: the KV cac
I'm excited to see our Lemonade embedded SDK getting used out there!
So I’ve only been messing around with Krea2 in ComfyUI for a little while, and I ran into the classic problem: I make an image I really like, then a week later I want the exact same settings back and have no clue what I did. Prompt, seed, sampler, all of it gone from my memory. E
Article · llama.cpp
vulkan: allow multiple devices to access gpu-visible host memory ( #29741 ) vulkan: allow multiple devices to access gpu-visible host memory clean up erroneous newlines add Windows support handle nullptr fix enable condition Website:
The problem I'm solving: I have an M3 Ultra 512 GB and the memory is still the reason to own it, but prefill on long prompts is slow. A 17.5k-token prompt takes 41 s before the first token. The M5 Ultra fixes prefill but ships with 256 GB max right now, and an M5 Ultra 512 GB isn
FastH3-Omni is here. This is the reference-conditioned FastH3 release we’ve been working toward. It generates video with synchronized audio and supports: - Character and setting reference images - First- and last-frame conditioning - Up to 3 image references in real-time mode - U
Hey r/selfhosted , Most self-hosted personal assistants and automation wrappers feel like rigid, hard-coded integrations disguised as "AI". You either have to manually map every API payload, or you have to dump endless conversation logs into a context window/Vector DB and hope th
I present to you , Brad Pitt as Geralt of Rivia, using Stable Diffusion 2 , i think.
Article · llama.cpp
cuda : add GGML_CUDA_GRAPH_DEBUG and read env vars once ( #30327 ) ggml-cuda: gate CUDA graph debug logs behind GGML_CUDA_GRAPH_DEBUG ggml-cuda: cache env var lookups in static const instead of re-reading per call ggml-cuda: read env vars exactly once via ggml_cuda_env_flag/ggml_
Im working in nodes to hunting seeds for minimax h3 videos, in a few days i will realease it in my github.
If you're the average user with a regular ass PC, this is for you.
I have been messing with an idea about degredation. Essentially using static image frames inside the latent space and instead of resampling them, bypassing the resampling process and repasting them using masking from fresh latents from earlier parts of the clip or previous clips
Hi all, I'm the developer of VigenFlow, a local image generation and editing server that runs diffusion models on the AMD Ryzen AI NPU (XDNA 2) + CPU. If you have a Ryzen AI laptop or mini PC without a discrete GPU, you can generate images entirely on your own machine. Supported
Quartermaster (my local inference manager: one server that swaps llama.cpp, image, audio and video models in and out of VRAM) can now help you server and manage decision models: Auto-discovered. Drop a decision GGUF (Kev, OpenJev, Laya, ...) in your models folder and it gets its
My system is 32gb ddr5, 9070 xt, 7600x3d. I’m running on Ubuntu 26.04 headless, rocm 10.1, comfy kitchen attention. The images were all 1MP Ideogram4 fp8 20 step 0.76it/s 27.5s 52% faster Krea 2 fp8 8 step 1.17it/s 8.5s 52% faster Qwen image 2.1 t2i 25 step 1.10it/s 25s anima 2B
Reddit from r/StableDiffusion.
A few days ago I posted “ Gausify Orbit - I am building a web application to transform images into Gaussian Splats ”, and it was removed because the project wasn’t open source yet. It’s now released under the MIT license , so here it is again, with a lot of updates. What it does
I'm sure some here read this at the time and remember, but I recently found this hilarious Guardian article from 2019 through Wikipedia:
Reddit from r/LocalLLaMA.
Reddit from r/StableDiffusion.
Article · Hugging Face
Article from Hugging Face.
"A 4-step distilled LoRA for Qwen-Image-2.1 , supporting text-to-image generation and image editing. Training was performed entirely on 16 Huawei Ascend 910B NPUs , using DMD in LightX2V ." I think it's rank 16. Pretty small for a turbo lora, but we'll see I guess. submitted by /
Article · AnythingLLM
What's Changed fix: point k8s probes at the server's /api/ping by @u00dxk2 in #6592 fix: don't crash when removing a parsed file while an image is attached by @marmar9615-cloud in #6585 fix: open the clicked image when attachments share a file name by @marmar9615-cloud in #6586 f
Been digging into this one for the last few days. 3.66B model built on IBM’s Granite 4.2 3B, post-trained only for formal logic. Entailment checking, rule induction, Lean proof critique. Not a general assistant. Recipe is LoRA SFT + checkpoint merging + RL against a programmatic
Article · llama.cpp
model : allow Qwen3.8 Flash Next to accept image embds for MTP ( #30397 ) Cont of #30257 , closes #30307 Website:
A 4-bit build of microsoft/Mage-Flow that runs on ComfyUI's native quantized kernels , plus a small accuracy recovery adapter (ARA) that brings the 4-bit model back toward the BF16 original. Optionally, an INT8 text encoder cuts another ~3.7 GB of VRAM. Repository:
New version of my MiniMax H3 timeline workflow released Some very important bug fixes included, and a lot of improvements. The YouTube video above includes a lot of tips for using the workflow in addition to going over the changes in detail. Below is a complete list of fixes and
Reddit from r/LocalLLaMA.
Here is how i serve 40 people (10 of them developers) with local AI services. n8n for automatisation whisper and Vexa AI, a bot joins Teams and makes transcripts, n8n later extracts decisions and todos OpenWebUI, ChatGPT-like webinterface for everyone, most agentic features are d
Article · llama.cpp
llama : do not map foreign ranges into a backend buffer A buffer created from a host pointer pins everything that it wraps. When several contexts share the same file (lazy tensors, MoE tensors kept on the CPU with -cmoe), the single buffer that spans the first and the last tensor
It just popped on my feed. I watched the teaser in the Netflix app and a few things jumped out The constant use of anthromorphism using terms like altruism and sacrifice when in engineering literally there are terms like kill, killall, pkill for terminating processes, therefore o
Reddit from r/LocalLLaMA.
Dynamic vision, in action Basalt is an inference engine for 3.8 Flash Next, forked from Strata and heavily tuned for Blackwell. It supports vision, and for this to work you need to have the image encoder either in VRAM or CPU all the time, having it on VRAM takes valuable space f
Follow-up to last week's post about streaming MoE experts from disk in a browser tab. That ran on hand-written engines for two models. This week the same disk tier sits under llama.cpp's own WebGPU backend, so it works for the MoE models llama.cpp already runs, with no per-model
Lipwise is a [yet another] dictation app for Windows, macOS and Linux. You hold a hotkey, talk, and it types into whatever app has focus but with a twist. Since most of us that speak to our computers also use LLMs to do transformations I've thought why not do both in the same tur
I built a desktop app harness that keeps local meeting notes and can use them to autocomplete my typing as well as complement my dictation with additional context via reading my screen or using the previous meetings notes. It's been a bit of a struggle to get it to work well and
Article · Simon Willison
If the agent isn't using my diagnostics library, I could introspect and update AGENTS.md, but there's still a delta, and it'll drift. Instead, I write a pre-commit hook that checks the git diff for direct calls to observability modules that bypass my diagnostics module. Boom: fai
Article · noted.lol
Oxynote is an open-source, self-hosted wiki built for engineering teams. This guest guide covers Docker installation and its live metrics integration.
Anthropic says we can run Gemma4 26b a4b with only 14 smart tvs.
Article · KoboldCpp
sd: add a non-default target for a Vulkan sdmain binary ( #2534 )
So I wanted to try the horsey challenge myself on the latest engine i been testing which is a strata fork by sergqwer.
This is an update to my previous post which can be found here:
The post at the bottom inspired me to try a simple anime-to-realism test with MiniMax H3, so I figured I’d share a few results. I didn’t generate any keyframes or stills. I just fed the original video directly into MiniMax H3 as the video reference, applied an anime to realism lo
Got this couple. It was difficult purchase in this economy but paid only roughly $6100 equivalent for each in my country. I justified the purchase for learning (so I try to convince myself) and coding workflow improvements. I started looking into recipes for GLM 5.3 flash but sur
Article · llama.cpp
vulkan : split matmul dispatch when it would exceed maxComputeWorkGro…
Article · llama.cpp
cont of #29622 , where we forgot to remove token assert
Hi everyone, Recently I got myself a RTX 5060 Ti 16GB and am building a PC for it. Since it only has 16GB VRAM, so I am considering getting a second GPU in the near future, maybe in 2027. Since 5090 is getting really, really expensive and it's currently uncertain whether 5080 wil
My project is optimising qwen flash next on Mac Mini m5 64gb through SSD streaming. I recently asked LocalLLama for guidance on how to optimise token output. Today a popular method to do this is a lookahead guesser which fetches experts one layer ahead of the current layer. One p
I’ve been building SonicBoom, a native tensor runtime written in C++/CUDA with a C ABI and no Python anywhere in the stack. I just tagged v0.0.1, and the first target is local LLM inference on consumer GPUs. Numbers (Gemma 4 E4B-it Q3_K_M on a laptop RTX 3050 8 GB, both runtimes
Article · llama.cpp
cuda : warnings cleanup Signed-off-by: Adrien Gallouët angt@huggingface.co Update ggml/src/ggml-cuda/common.cuh Co-authored-by: Sigbjørn Skjæret sigbjorn.skjaeret@huggingface.co Signed-off-by: Adrien Gallouët angt@huggingface.co Co-authored-by: Sigbjørn Skjæret sigbjorn.skjaeret@
Article · llama.cpp
graph : avoid reallocations in the "no cache" path ( #30439 ) Website:
Hey guys, I have been working on the consistent character portable format (.char) for a while & now there's a community characters page where you can grab a ready-made character and use it straight in ComfyUI. Same face, cloths & body across every video generation, and with a few
Article · llama.cpp
ggml-zdnn : introduce Q8_0 mat mul ( #29836 ) ggml-zdnn: initial prototype of q8_0 matmul Signed-off-by: Aaron Teo aaron.teo1@ibm.com docs: update zdnn docs Signed-off-by: Aaron Teo aaron.teo1@ibm.com Signed-off-by: Aaron Teo aaron.teo1@ibm.com Website:
i'm the maintainer of dodaios (formerly personal jarvis), an open-source ai agent app for the desktop. apache 2.0, runs on windows, macos and linux, no account and no cloud of ours between you and your models. it works with hosted models too, but this post is only about running i
Article · llama.cpp
llama : skip logits buffer allocation when graph produces no logits (…
Article · llama.cpp
models : remove redundant ggml_cont calls now supported by all unary kernels ( #30198 ) models: remove redundant ggml_cont calls now supported by CUDA unary kernels models: restore cont_2d after sigmoid for broadcast mul, clean dflash cont Update src/models/qwen3next.cpp Co-autho
Article · n8n
2.43.4 (2026-10-12) Bug Fixes core: Handle each Chat Hub segment once in queue mode ( #40816 ) ( dc77062 ) core: Resume tool calls for deferred tools that a skill loaded ( #40746 ) ( ea448c6 )
Article · n8n
2.44.0 (2026-10-13) Bug Fixes Align execution retry menu and clarify actions ( #40710 ) ( 84d2fd6 ) Allow per-process crash journal paths ( #37670 ) ( 447e939 ) core: Allow reverting a workflow version that uses credentials the user cannot use ( #40701 ) ( 033a9b5 ) core: Apply n
Hi, I see that vllm released d3 (a family of decision models) and llama.cpp already has support for them. I want to try them out. Are decision models sensitive to quantization? I know that for autocomplete (FIM) models like qwen-2.5-coder, it is recommended to use Q8 instead of Q
Hey everyone, I’m using Krea 2 with a trained character LoRA , and the image generation quality is amazing. The character looks realistic, and I’m really happy with the results. However, I’m facing a frustrating problem. When I generate an image and then try to change the charact
Hi guys! I got a problem with one of my V100 SXM2 Modules. In "nvidia-smi" its shows with "Pwr:Usage/Cap" as "N/A / N/A". When I try to use it i get the following error in dmesg: NVRM: Xid (PCI:0000:02:00): 44, pid=21978, name=llama-cli, channel 0x00000010, intr 00000000 What I a
I set up a dedicated workstation for local AI experimentation with various models and tools. But every single one of them wants all the available VRAM. To fix this, I built a request dispatching layer called vramble . It automatically loads and unloads models and manages the queu
From what I can tell, whenever there's an issue people say to just go to portable. I'm currently using comfyui Desktop which apparently has a lot of dependencies and system env set up so that it's quite fragile. If I want to simply go back to portable would I just uninstall comfy
Article · Ollama
Add Kev decision scoring to MLX Load Kev models with their FP32 decision head and release metadata. Support System One scoring with shared-state caching and Kev-specific limits. address comments
I've been reading about all these modders using AI to reverse-engineer games and software. I've been trying out photocraft and gridcraft and have just watched a GTA5 mod featuring Sonic. It's pretty damn impressive. As a struggling AMD Radeon owner I'd love to see AI generated op
Intel Crescent Island Squeezes 1.3 PFLOPs Of FP8 Into Just 350W As D270 Targets The Competition With Value-Oriented LPDDR5X Up To 480 GB" title="Hope the software support is good, but -> Intel Crescent Island Squeezes 1.3 PFLOPs Of FP8 Into Just 350W As D270 Targets The Competiti
Article · llama.cpp
hexagon: extend scalar DMA broadcasting support for binary ops ( #30245 ) tests: cover weighted scalar broadcasts Assisted-by: OpenCode ggml : support extended scalar DMA binary ops Assisted-by: OpenCode hex-binary: remove all non-dma paths hex-binary: add support for non-contig
Didn't realize the V100 were still so fast, anyone have first hand experience on these?
Quick note, not my character, original came from schmede_AI on youtube. But yes my question is do any local models currently have the ability to match and generate additional images of a character from a single reference image as well as this? I have tried a couple (Qwen Image 2.
I saw a lot of posts here struggling to run 27B models on 8GB VRAM or complaining about phones turning into space heaters. I hit the exact same OOM wall on my phone, so I decided to build a hardware-split workaround. I wrote a custom C++ engine (fork of llama.rn ) to cluster two
I’m planning to get a Mac mini mainly to run local LLMs and AI agents 24/7. I plan to run models up to 27B parameters with Q4 quantization, and maybe not for coding, maybe just for everday task and model exploration. I’m trying to figure out whether I should prioritize more RAM o
I've been using this for a few days and I was surprised to see nobody had posted it here. It's a vae trained to remove the checkerboard artifacts that qwen image 2.1 produces. It works very well. You can still use alternative schedulers/samplers/degrid to further improve cooked i
Comfyui with Minimax H3 plus Claude
Article · llama.cpp
ci : build OpenVINO releases with GGML_NATIVE=OFF and CPU backend var…
I’m trying to generate images that look like early 2000s PS2 game screenshots. Most AI generators make the results too polished or realistic. I want the actual imperfections of old 3D graphics: low-poly models, awkward character shapes, blurry textures, and simple lighting. Does
I'd like help crafting structured prompts for Minimax H3 Ref2V. I don't really need to build it in as a node. I'd be happy with running it as a separate process, within Comfy or outside. How do you integrate an LLM such as Grok or Claude into your workflow? I have free accounts w
I’ve released my MiniMax H3 Character Swap LoRA I trained this LoRA to replace the main character in a guide video while trying to preserve the guide’s motion, expressions, camera, and scene. It was trained with guide latents, so the recommended setup is to connect the source vid
A lot of projects have come up recently consisting of specific hardware config specialized forks of existing conferencing engines, but they all seem to be made using Claude or OpenAI's biggest frontier models. So given that open source is about 4-6 months behind, can we reasonabl
I came across this blog post from OrcaRouter:
Reddit from r/LocalLLaMA.
Reddit from r/comfyui.
Article · Home Assistant
Prepare to be dazzled, because this month LEDVANCE join the Works with Home Assistant program as our newest partners 🎉 And what’s more, they bring a huge lineup of certified devices (116, to be exact!) to our community, which means plenty of new ways to light up your home 💡 A cen
Hello everyone, I created a custom node few months ago with the goal to achieve all animation position I wanted from a single picture of a character. I had some trouble getting a good rig position library, especially to achieve 4-ISO/8-ISO character orientations. And finally, I g
Dual A100X PCIe (link at Gen4 x8, weird DPU cards) in a 2U EPYC 7402P box, 256GB DDR4. Proxmox with the GPU's passed through to debian 13, 610.43.02 drivers. llama.cpp, qwen3.8 flash next UD-Q4_K_XL, Q8_0 MTP head, draft max 2, mmproj enabled. -sm layer -fa on --parallel 7 --ctx-
After running a benchmark test on Exo in Opencode and being amazed by the result, I thought I'd try other models on Opencode. Many were surprisingly terrible. Especially some mid level models. Earlier posts:
Article · llama.cpp
model: fix vllm d3 multi seq case ( #30408 ) model: fix vllm d3 multi seq case add TODO Website:
For a while, I ran an Instagram account for a model that does not exist. Everything was generated in ComfyUI. It reached around 40,000 followers until Instagram took the account down. That was that. The part that stayed with me was the work behind the posts. Every shot needed a l
The newly named D-Series GPU has up to 480GB of LPDDR5X memory, 1.5TB/s of memory bandwidth Intel rates it for up to 1.3 PetaFLOPS of FP8 performance, with support for formats ranging from FP4 to FP64. It features a total of 32 Xe3P cores, 256 XMX engines, 32 MB of unified L2 cac
Just looking for some recommendations. I got painfully slow internet so I haven't just been downloading a ton of models to try out. So far I have Z-image-turbo and SDXL base. Running this on a Xbox Ally X with 24gb unified memory. ~9gb for GPU. Not really concerned about speed si
Businesses announce supposedly open-source/weights release, but never deliver; using the hype as marketing to promote their closed-source brand and/or product. Mods should auto-remove any post promoting/hyping/mentioning unreleased models like Flux 3, Krea 3, Ideogram 4.5 and so
I’ve been thinking about how people use Ollama beyond just chatting with a local model. Running models locally is great, but the experience can get more complicated when you want to switch between models, connect tools, manage conversations, or use different workflows for differe
What is 'Small', what is 'Large'? We're still over the place, years later we still can't reach consensus, causing endless confusion, so in infinite wisdom and generosity I took it upon myself to fix it. I've arbitrary decided from this day forth we shall all follow this table: Ti
First time poster, newbie here working with my gpt to learn how to do comfyui, its a powerful tool, ive done a little but its all greek to me still. Could anyone point me in the right direction of an nsfw enabled workflow that results in a gpt style result where i can generate ad
Hi all! I need some experience and advice from you all. For me, using AI makes setting up selfhosted stuff way easier. However I'm hesitant using it more, because I am not capable (yet) to judge whether security-wise it is done correctly. Ideally we could use a frontier model to
Article · llama.cpp
graph: pad the KV dimension of the no-cache attention to 256 ( #30416 ) The no-cache path builds a [n_tokens, n_tokens] mask and passes K and V unpadded, so FLASH_ATTN_EXT sees K->ne[1] == n_tokens. The CUDA kernels for head size 512 require a multiple of FATTN_KQ_STRIDE, so the
Hey everyone, Sharing an educational tool called AskCyph Health Explorer to explore how single-letter DNA mutations lead to disease, how they alter 3D anatomy, inheritance probabilities, and available treatments. To make it interactive while protecting user privacy, the architect
Just install the custom node from here, if you're interested:
Can someone help me to find some communities that share workflows
![v0.32.0rc0: [HiSparse] Report host pool in KV cache capacity logs (#60285)](media/d4901c93c13cc312.png)
Article · vLLM
Signed-off-by: Matthew Bonanni mbonanni@redhat.com Co-authored-by: Claude Opus 5.5 noreply@anthropic.com
Article · llama.cpp
metal: fix metal tensor kernel_mul_mm error for large tensor ( #28748 ) metal: fix metal tensor kernel_mul_mm error for large tensor Assisted-by: Codex Update ggml/src/ggml-metal/kernels/mul_mm.metal Co-authored-by: YiChen Lv 63285796+forforever73@users.noreply.github.com Co-auth
I've made a few posts recently, just talking about the hardware I've got and the model I use. Pair of P100s, Unsloth Qwen3.6 35b A3B UD Q4 K XL . There's been a ton of talk of model and hardware specific inference engines showing crazy performance numbers on models and hardware t
Article · Paperless-ngx
paperless-ngx 3.3.1 Bug Fixes Fix: use a root document's effective content for the LLM index and pr… @stumpylog ( #14423 ) Fix: index document versions as their root document @stumpylog ( #14422 ) Fix: authorize document versions by their root in the single-object permission chec
Here're results from my final test of coding, coding (panda/numpy), data analysis, data-driven decision-making, and local RAG domains I interact with most often. This test, which is refined from a 500+ pack, now provides a differentiating signal between models after having weeded
Reddit from r/LocalLLaMA.
The quality is not great tbh. I started to use Minimax for 2 weeks. But it was worth of try --- edit; I did not aware of this forum :/ I now will adapt new knowledge to push my boundaries
Reddit from r/LocalLLaMA.
Hi, I'm wepiqx - I build MERNIK, an open measure-first GGUF allocator (Apache-2.0). Disclosure: this is my work, tested on a GTX 1070 8GB. IMPORTANT for Ollama: always pick a version tag, bare repo errors out. Like this: ollama run hf.co/wepiqx/Mellu
My window will have a sign that says "we don't call tech support" with a simplified pic of a v100 on it I know this reads like a shitpost and I'm half joking and half serious Edit: I'd sell these on Etsy if Jensen wouldn't hit me with a C&D Edit2: if you're from Europe pretend I
Article · noted.lol
GitLab has released security updates addressing a critical vulnerability that could allow attackers to execute code without authentication.
Article · noted.lol
I built a home server dreaming my family would use all these amazing self-hosted apps. Turns out I'm the only one who seems interested.
GPU- TWEAK is up all the time, I've setup a bunch of temp warnings, voltage spikes etc... especially the Pin monitor util. MY GPU realtime stats has a dedicated monitor now. Why ? cus my friend asked me a favor yesterday to make him a PC build for Gaming and he wants to get into
We're launching Precision Numerics by releasing Boole-1.7B, a 1-bit version of Qwen3-1.7B. 247 MB packed weights: 13.9× smaller than BF16. 85.9% of the BF16 model’s average score across our 10-task evaluation - outperforms 1-bit Bonsai-1.7B and nearly matches 1.58-bit Bonsai-1.7B
Article · llama.cpp
hexagon: optimize CPY/CONCAT - round 3, short-rows edition ( #30341 ) hexagon: move the converting CPY in blocks of rows per DMA descriptor so short rows are not DMA-latency bound hex-cpy: cleanup cpy cursor handling hex-cpy-concat: introduce threaded dma/hvx gather-row helper Co
My company wants to set up a dedicated workstation for generating AI video locally. I’m a Mac user and familiar with LLMs, and people internally are suggesting a Mac Studio (M-series with high unified memory). However, our main pipeline would rely on ComfyUI (running models like
Article · llama.cpp
vulkan: pack FMA for f16 accumulation, no dot product op ( #29877 ) vulkan: pack FMA for f16 accumulation, no dot product op vulkan: dot_product packs two rows per f16 FMA vulkan: rename dot_product to dot_product_v2 Website:
Hey everyone! I introduced HandyComfy here a while ago . Since then, I've pushed quite a few updates, and recently added support for MotionContext ! This time, I wanted to share a quick demo of how it works. The video shows three things: The final AI-generated video. How I genera
Article · llama.cpp
model: support vllm d3 models ( #30383 ) models: support pplx-decider vllm d2 update clarify naming rule stricter validation d3 Website:
10 x 5sec generations H3 Clip Refiner Inline style-drift corrector and optional refine-resample for H3 Motion Context clips. Place this node BETWEEN the sampler ( SamplerCustomAdvanced ) and the H3 Motion Context Save Latent node. It works on the LATENT level and attacks the accu
Hey r/homelab ! We're UptimeRobot , and we've teamed up with the mods to run a giveaway for this community, with a prize pool of 10 monitoring plans, including a year of our Scale plan worth $888. Anyone who's worked in creating a homelab has a story! Now's your chance to share i
Article · llama.cpp
model: support vllm decision 2 model ( #30378 ) models: support pplx-decider vllm d2 update clarify naming rule stricter validation fix ci rm redundant filter_tensors Website:
I'm trying to get Ornith 1.5 9B running (I can also barely run 35B A3B but I want something that I can run relatively easily) with the latest llama.cpp with this command: llama serve --flash-attn on --cache_type-k q8_0 --cache-type-v q8_0 --hf-repo ornith-ai/Ornith-1.5-9B-GGUF --
Endless MiniMax H3 (with Endless LipSync) v1.6 When MiniMax H3 came out, I was badly missing the Endless Wan 2.2 I2V (SVI 2 Pro) features, so at first I look at other options, but most of them included an AIO node that could do everything, and I couldn't use most of my workflow,
Can refmods be used with Krea2? And/or when using the Krea2 edit lora? I've tried adding the refmod nodes but none of the identity is transferred to the output. I've used them for H3, so I know how to use the nodes, but I can't seem to get Krea2 to use them. Do they just not work
Article · llama.cpp
tests : link each test against only the libraries it uses ( #30299 ) Give llama_build() and llama_build_and_test() a LINK keyword (default: llama-common) so call sites can declare what a test actually needs: ggml-only tests now link just ggml: test-opt, test-metal-graph-optimize,
Not a benchmark run, the one you actually use. Which model, and what for?
Details in first comment
A few weeks ago we posted the open-source Graphsignal profiler here: wrap vllm serve / sglang serve / whatever in graphsignal-run and it serves everything it measures as JSON, mainly so an agent can read it in a loop. Turns out we kept opening the JSON ourselves to see what the G
thon. 📅 Month 1: Fundamentals 📅 Month 2: Local LLMs 📅 Month 3: Build a tool Privacy is a feature, not a bug. What’s your first step? #
Reddit from r/LocalLLaMA.
Article · llama.cpp
metal : fix mul_mat_id with duplicate ids ( #30373 ) metal : fix mul_mat_id limits and kernel name metal : fix mul_mat_id with duplicate ids kernel_mul_mm_id_map0 assumed at most one match per token and a fixed capacity of ne21 entries per expert, which produced wrong or dropped
Introducing QwenGyre, bringing elastic RL to extreme-long-horizon (XLong) agent training. It launches with two core designs: - Elastic Scheduler: reallocates GPUs between rollout & training as demand shifts-live executions never interrupted. Up to 1.85× speedup over Colocate, 1.7
**🚀 FMJ V15** Un seul workflow pour générer, éditer, upscaler et finaliser vos images, avec **12 modèles** au choix et 257 nodes organisés en groupes colorés. Un interrupteur par famille de groupes permet de n'activer que ce dont vous avez besoin. **✨ Les 12 modèles** Anima, Boog
Disclaimer: I work for NuMind, the company behind this plugin. TLDR: Text/image (simili-)embeddings from the LLM you already have loaded :-) I've been working on making it easier to obtain embeddings from a LLM (preferably multi-modal) that is already being served by vLLM. The id
Quick context: LocalLightChat (LLC) is a chat frontend for local LLMs. Download it, run it, that's it. No installation (unless you want to), no dependencies, and it runs on pretty much anything, including ancient hardware. I originally built it because Open WebUI kept feeling hea
I am an independent researcher. Due to budget constraints, I shifted my 21B MoE pre-training-which I had originally planned to run on 64 H100s-to a single NVIDIA GH200 ($2.29 per hour), and I have compiled the metrics I measured during that process into a paper.
Article · llama.cpp
llama : report compute graph reallocations during inference ( #30329 ) Add an optional ggml_backend_sched_realloc_callback that user code registers with ggml_backend_sched_set_realloc_callback to be notified when the buffers reserved for a graph are not sufficient and have to be
i have a rtx 4050 laptop, with 6gb vram, 16gb ram, is there any free model that runs comfortably on my laptop? i want to generate videos for youtube, i saw many people saying it needs lots of ram and vram to run ai models locally, so I'm not sure which one to use
I posted an early version of this here a few months ago. It does a lot more now, so here's an update. It's free (MIT) and installs from ComfyUI Manager. What it does: - Models: every model with its Civitai cover, trigger words and base model. Files are looked up by hash, so downl
Title. Not sure what is considered to be a well-respected/reputable test suite for benchmarking the abilities of local tiny models. Figured people here would know more about it.
Hey there! I'm a filmmaker coming from a traditional film production background. I wanted to test if I could automate the production process while keeping creative control. I built a ComfyUI workflow that takes my short film concept, including the voiceover script, and translates
I built a spotlight search for windows. It has an embedding model and a MoE models to go with it. The MoE It's 2 Qwen 3.5 4B models, I fine tuned one for coding work and another for agentic work, and then I combined them both into an MoE. I didn't want to shove it into everything
Browsing CivitAI and the like, incl. even forums like this, is extremely embarrassing. I'm sure you all know what I mean. Are there any resources that deliberately ban coomer material (and don't tolerate supposedly "SFW" versions simply because they have clothes on...) and focus
Hey guys I have some experience with comphyui but not alot. My goal is to create shorts for my youtube channel I wanna make videos of video games characters or anime characters I want it to be the best it can be. I have a 5080 and I tried generating video with it and it was good
Somebody on my last post told me I needed to do a better job explaining why anyone would want this. They were right. I was so excited about what we were making that I skipped over explaining what it actually DOES. Apparently, “LOOK HOW COOL THIS IS” only gets you so far. ;) CUT p
Cheapest one I see at MC is the asus pro-art version for 6499.
Article · noted.lol
Cloudflare Tunnel makes accessing your homelab easier, but how secure is it really? Here's what it protects and where you still need to be careful.
Lets say I'm writing my own journal/diary and want an LLM to parse everything and index names and keywords so I can ask later things about my life, which local small LLM would be a good fit that I can trust won't go off the rails/hallucinate? are there any models specific for thi
Article · llama.cpp
examples : fix draft sampling and rejection handling This commit fixes two issues in the speculative example which are described in detail in the linked issue. Resolves: #29975 examples : add kv-unified requirement for parallel > 1
I can't find anything in the Docs. I see that the setup.sh file mainly pulls a big ass unaudited binary and, with all simpathy for the repo author, that's not going to run on my machine. I am sure the documentation is somewhere , but I can't locate it. (Yes, I could ask an LLM to
Hey everyone, I made whatmodel.sh To use it, Enter your RAM and GPU See what models you can run locally with its rank and the command to run it Compare them with API models on the same score Building it made me realise a 32GB Mac can now run a model that scores on-par with GPT-5,
This is about Ollama, but the mechanism is the same for anything built on llama.cpp (LM Studio, Jan, Open WebUI in front of Ollama, your own scripts). A local model keeps a KV cache of the prompt it has already processed, and reuses it on the next request. That reuse only survive
its open source I’ve been working on a DeepSeek Harness plugin called DSH Novel Quality Analyzer . It adds an Analyze writing button to the conversation toolbar. Paste a chapter or excerpt, and it reports vocabulary diversity, cliché occurrences, repeated content words, and repea
Article · n8n
2.42.7 (2026-10-12) Bug Fixes core: Keep concurrent credential tests from sharing one node type registry ( #40783 ) ( c17fdee ) core: Resume tool calls for deferred tools that a skill loaded ( #40747 ) ( fd2586e )
EXL3-QLoRA I've built this over the last few months and I think it's stable enough to share now. A fork of ExLlamaV3 , this is a native ExLlamaV3 training implementation with plain PyTorch, without transformers. exl3-qlora on github Why? With it, you can train 2-8bpw exl3 quantiz
Article · llama.cpp
vulkan: use ternary expressions to avoid switch lowering for CM2 quant block size ( #30164 ) vulkan: specialize quant block size for CM2 matmul vulkan: use ternary expressions to avoid switch lowering for CM2 quant block size vulkan: clarify CM2 quant block size comment Website:
Reddit from r/StableDiffusion.
Article · llama.cpp
ggml: Change default visibility of symbols to hidden ( #30014 ) See discussion in #29997 . This prevents a handful of internal symbols leaking via GGML_API. Website:
I wanted to track grocery spending per item without sending my receipts to anyone, so I built an Android app around a small vision model that reads a photo of a receipt into JSON: store, date, total, every line item. (iOS will take a couple more days to be checked by the Apple te
Trying to run image edit. The "changes" seem to have processed correctly ("replace weapon with cards") but the output colors are all gone? Does anyone else have this issue? Is this a memory (lack of) issue? Diffusion Mode: qwen_image_2.1_int8_convrot.safetensors Clip: qwen3vl_8b_
Article · llama.cpp
vulkan: fix buffer logging on 32-bit platforms after source split ( #30326 ) Move the vk::Buffer stream operator to the shared header so it is visible to ggml-vulkan-debug.cpp. Website:
# I ACCIDENTALLY PUSHED THE WRONG FOLDER TO GITHUB, MY APOLOGIES - IT HAS BEEN FIXED WITH THE CORRECT FILES - Oct 12th 2026 -9:04am I got tired of rewriting the same dataset-prep scripts before every LoRA/Dreambooth run - resize, dedupe, toss the blurry ones, caption, split - so
With the vLLM Semantic Router team we released Decision 3.0 (d3) , our new SOTA decision model family ranging from 0.6B to 27B, Apache-2.0, all reading text, images and video. How it works - Built on Qwen3.5 (0.8B / 2B / 4B / 9B) and Qwen3.8-27B. The parameter counts include the
[Release] Eric LoRA Catalog – find and organize your LoRAs by what they do, not by folder name (local LLM cards + LoRA Suggester node) Over the last 3 years I've amassed a huge collection of LoRAs (about 5,600 LoRAs currently), and I'd stopped being able to find anything or remem
I see lots of people saying how they love upscaling H3 latent and how effective it is, but I can't run it, no matter what. I have a 16:9 video at 0.2MP (14 seconds) I'm trying to upscale. I've tried upscaling to multiple resolutions, with 0.98MP being the highest I've tried, down
Inspired by DigUp for macOS, VectorDash aims to be a full Windows 11 search replacement, powered by EmbeddingGemma 2 . Full indexing: search by meaning or exact words across photos, screenshots, audio, video, PDFs and Office documents. Find “images of dogs,” a podcast discussion,
I've been tuning llama.cpp on my Strix Halo box (Radeon 8060S, 128 GB) and the numbers are good enough to share. It's a branch called rdna35-perf on top of halo-box/strix-llama.cpp , a llama.cpp fork aimed at Strix Halo:
Article · llama.cpp
hexagon: add IM2COL_3D ( #30305 ) hexagon: add IM2COL_3D support Assisted-by: OpenCode hexagon: add DMA paths for IM2COL_3D Assisted-by: OpenCode hexagon: avoid scalar fp16 conversions in IM2COL Assisted-by: OpenCode remove in-loop spills in pre-v81 devices Assisted-by: OpenCode
ComfyUI said HEART was unsupported, so I traced down the architecture source, isolated it from traiNNer-redux, loaded the official checkpoints, and verified native inference.
Ive been using OpenWebUI's inbuilt web search and I can't put my finger on it, but it's not quite as polished, reliable, seamless as using Claude or Gemini. And its often slow beyond a limit that makes me subconsciously not want to use it - I understand its unlikely to get to clo
Hey everyone, So the same model can behave differently depending on the inference engine (llama.cpp, vLLM, Ollama, SGLang, etc.), model quantization, chat template, or runtime version. Sometimes behavior breaks after upgrading a runtime or switching to a different model artifact,
Reddit from r/ollama.
I don't know if I'm right or wrong. I'm referring to AI-generated images that look like real photos.
As per the title, I was wondering whether there is something I can self host to be able to save recipes from social media platforms like instagram and have it extract the recipes from the caption and organize everything. Maybe also something where I can use my locally running AI.
I upgraded to OpenCode 2.0.26 today, and even with competent bots it seems to hang up where the bot will be like, "Great, I'll implement that." And I have to prompt it with, "Yes, go ahead." as if it has no agenticness. Bot: "Fixing the test:" Me " T-t-t-today, junior! " submitte
I dont like how in a open source code community, someone posts a binary file, says "trust me bro" to run it on our local computers. Claims "oh i'm the fastest but I dont have time to post my source code, just my pre-compiled binary files"
Article · MarkTechPost
Cloudflare released Clef-omni, an Apache-2.0 multimodal decision model built on Qwen3-Omni-30B-A3B. It scores text, images, audio and video in one call for $0.15 per million input tokens, while Clef-flash drops to $0.038. The post Meet Cloudflare Clef-omni: A 30B-A3B Multimodal D
It's a tiny 6M parameter text to image model, and it can make recognizable images at 64x64. The 6M accounts for everything, there's no oversized text encoder or VAE. Check out the model card for samples, a detailed explanation about it's architecture and training, and a quick sta
Hello guys, would need your help. I am on laptop RTX4090 with 16GB VRAM and 32GB RAM. I used to generate 15s 1MP videos at 4steps around 12-13 minutes. today I came from vacation, updated Comfy and wanted to test some new ideas. but it takes double of time now... around 30 minute
I recently solved my first frame issues from my last post, but in that thread, a few posters said that most were using latents now. I know what they are in theory, as I've noticed considerable degradation over time from strictly generating subsequent videos from a rendered final
I’m exploring a local coding assistant for game development. My plan would be to run the model on a dedicated machine and connect to it from my main development PC over my home network. I’d like to use it for code explanations, debugging, small feature changes, and project-aware
I was running Qwen3.8-Flash-Next (base and later Swift-1.5 NVFP4) on a paid pod seat for about a month, and it started answering with short strings that echoed its own earlier reasoning. I assumed the context was getting too big for it. But it wasn't. Across almost 13,000 turns t
I’m planning a small self-hosted inference service using open-weight models on a GPU server in a German data center. It would serve a few business tenants through an API. I’m working through tenant separation, per-customer API keys and quotas, prompt retention, logs, model update
You know, the bot accounts that are always like a month old with like Here's some examples I've found in a few minutes:
There is a lot of discussion which quant to use. Especially a lot of post how much speed you gain with FP4, but not much about quality. So i tested some of the models that I use on my R9700. Models : Qwen3.8-27B-UD-Q6_K_L.gguf Swift-Qwen3.8-27B-Q6_K.gguf Swift-1.5-Qwen3.8-27B-MXF
Reddit from r/StableDiffusion.
Article · noted.lol
, including HTTP/2 flaws that could crash or overload the Docker daemon.
My Jetson Orin–optimized engine, little-gemma V1.0, substantially outperforms llama.cpp. Even after exhausting every practical GGUF option, however, it still falls short of the ideal performance level for Gemma E4B on the Jetson Orin Nano Super 8GB. I forked exllamav3 and ported
Article · noted.lol
There are so many places to discover new self-hosted apps, but where do you find yours? Let's see how the Noted community discovers new software.
Article · llama.cpp
opencl: improve fa, allow dk512 for gemma-4, improve dk64 ( #30266 ) opencl: enable Gemma-4 E4B GPU decode Co-authored-by: Hongqiang Wang wangh@qti.qualcomm.com opencl: extend Gemma-4 GPU decode to E2B Co-authored-by: Hongqiang Wang wangh@qti.qualcomm.com opencl: optimize DK64 GQ
It's been 4 years since c.ai first hallucination model, and yet, we're nowhere good enough at LLMs in terms of spontaneity/interesting hallucination features. Fortunately, last year brought new architecture that allow those within 10-16GB VRAM range to run something usable up to
I stumbled across this as I was currently having glm5.3 flash run only around 10tok/s basically unusable. With Maya i am running 30 tok/s now. GLM5.3 feels even with this low quant like a much more enjoyable model than qwen 3.8 flash next so far. Still testing . This is very exci
I run local LLMs pretty much around the clock, but I also enjoy gaming on the same PC. Having models sitting in VRAM while gaming isn't exactly ideal, and constantly unloading and reloading them manually got annoying. So I built GamePause . It's a lightweight, open-source Windows
I am auditing my home network setup to prepare for potential extended internet blackouts. My core goal is ensuring that if the outside connection drops completely, my household retains full, native local access to our personal documents, files, and family photo libraries via the
I created this because I couldn't find this kind of all-in-one package: a local AI assistant for the whole household that runs on your own hardware and is set up through a web GUI instead of config files. I want to share it with others looking for a framework like this. What KTSB
Article · llama.cpp
opencl: fix image limit for q4_k dense bin kernels, q4_0 bcast and rms_norm ( #30310 ) opencl: fallback when Q4_K weight images exceed device limits Co-authored-by: Hongqiang Wang wangh@qti.qualcomm.com opencl: avoid incomplete subgroup in rms_norm opencl: refine conditions for d
Alright, I learend a few valuable lessons since I posted about a week ago, and I've made more strides with the DDR4x Ascend 96GB 310P cards I purchased, so here we go, meet the new friendly and less verbose me. GLM-5.3-Flash is a 320-billion-parameter MoE model, with about 18 bil
Reddit from r/LocalLLaMA.
Another quick Minimax H3 news and goodies round-up, for those who may have missed some items. -> ComfyUI-H3-PowerLoraStack. "Quantized bases keep an exact runtime branch, AdaLN pairs are rebased between dense and curve checkpoints, and Acc/PDD head banks are blended per sampler s
Article · llama.cpp
rpc : turn GGML_RPC_DEBUG into a verbosity level and add logs ( #29544 ) rpc : turn GGML_RPC_DEBUG into a verbosity level and add logs GGML_RPC_DEBUG is now parsed as a number: 0/unset disables debug logs, 1-3 emit increasingly detailed output (events, per-command trace, transpor
Two questions? Can a local model like Qwen 3.8 27B reverse engineer games and programs? If not, is there a fine tuned model that can?
Reddit from r/LocalLLaMA.
Article · llama.cpp
model : support for Prism Bonsai 2 27B ( #29600 ) Runtime support for Prism Bonsai 2 27B Assisted-by: Claude Code address prism hadamard runtime feedback move hadamard tensors into method, define folded weight load_* hadamard fixes cont : clean-up conversion cleanup Assisted-by:
drex 1.5 9b leading both sub 10b decision index and the lowest latency on openrouter microsoft release why does anyone care about latency?
Hey everyone! I got tired of subscription tiers and cloud data harvesting just to route webhooks and process content through LLMs. I built VibeBridge , a lightweight Python framework designed to run entirely on your own local hardware. 💡 Why VibeBridge? Anti-SaaS Philosophy: Run
Hey r/LocalLLaMA ! Maintainer here. Some of you might remember OpenMed from our 1.8 post a few months back. Quick recap: it's an open-source (Apache-2.0) medical AI toolkit with one rule we never break: patient data stays on your machine. No API keys, no cloud calls, works offlin
Article · llama.cpp
cpu: fix AVX2 CPUs not using F16C when available ( #30270 ) Website:
Article · noted.lol
It starts with a few files and a small hard drive. Before you know it, you're buying more storage and wondering where all the space went.
Article · noted.lol
It starts with a few files and a small hard drive. Before you know it, you're buying more storage and wondering where all the space went.
This is a follow-up to my post from yesterday (
No more excuses for GPU-poor folks not to start LLMing! Full 27B-class reasoning in ternary transformer weights, for llama.cpp (CUDA, Metal, CPU) ~9.3x smaller than FP16 (ideal) | 98.2% of FP16 intelligence retained | ~47 tok/s on an Apple M5 Max laptop Highlights ~5.9 GB languag
Ciao a tutti, \* **Il mio setup** per prima cosa: Ryzen 9 AI HX 370 64 GB ram ddr5 eGPU su dock oculin: W7800 48 GB Vram Win 11 LM Studio 0.4.25 VS Code 1.141.0 Cline 4.1.23 Ollama Vulkan llama.cpp (Windows) v2.55.0 ROCm llama.cpp (Windows) v2.54.0 QWEN3-CODER-30B-A3B-INSTRUCT-UD
Article · llama.cpp
vulkan: handle mul_mat_id duplicates in the prepass rather than looping ( #29998 ) vulkan: compute every row of an expert in mul_mm_id when ids repeat vulkan: handle mul_mat_id duplicates in the prepass rather than looping Extend the "hoist row ids" optimization to always be enab
Hi, I’m Barbara. I vibe-coded a small web app that uses your own ComfyUI as the backend for ACE-Step 1.5, and I’m sharing it for free (MIT). What it is: a friendly UI around the ACE-Step 1.5 workflow. You write a style prompt and lyrics, and it builds the ComfyUI graph and talks
Article · llama.cpp
server: add component tests ( #30312 ) server: add component tests nits add codeowners Website:
Reddit from r/LocalLLaMA.
TL;DR: Five quants of Qwen3.8-27B (UD-Q4_K_XL to Q8_0, plus NVFP4) score the same on GSM8K, MMLU-Pro and IFEval, all within ±2 pts, even though Q4 picks a different token than Q8 once every 28. In math, the 27B, Flash-Next (NVFP4 and a ~3-bit GSQ IQ3_S on Strata), Gemma 4 26B/31B
Article · LiteLLM
Verify Docker Image Signature All LiteLLM Docker images are signed with cosign . Every release is signed with the same key introduced in commit 0112e53 . Verify using the pinned commit hash (recommended): A commit hash is cryptographically immutable, so this is the strongest way
DISCLAIMER: the post and the model card was made with the assist of AI. Hiya, I’m releasing Recursive BitNet N-Gram 102M, a small experiment combining ternary weights, shared transformer layers, and hashed n-gram embeddings, trained with a whooping budget of 100€ Weights, inferen
Article · LiteLLM
Verify Docker Image Signature All LiteLLM Docker images are signed with cosign . Every release is signed with the same key introduced in commit 0112e53 . Verify using the pinned commit hash (recommended): A commit hash is cryptographically immutable, so this is the strongest way
Article · LiteLLM
Verify Docker Image Signature All LiteLLM Docker images are signed with cosign . Every release is signed with the same key introduced in commit 0112e53 . Verify using the pinned commit hash (recommended): A commit hash is cryptographically immutable, so this is the strongest way
Reddit from r/LocalLLaMA.
I feel like we're heading to a world where knowledge is going to be deeply gatekept, you won't be able to do any research on it because of all the AI generated content and won't be able to separate propaganda from reality due to the mass of information and automation overall. I'm
EDIT: since people keep calling it out, this is API key abuse but not exfiltration. Seems like both Harbor and Pier (sandboxes used in almost every Software Engineering benchmark) expose the Openrouter endpoint and API key to models, even though they block the rest of the interne
As i use a rather tiny setup 12gb vram gpu and 32bg ram, i was wondering: What setup do you use, which harness? and how do you customize it to run qwen 3.8 flash next swift with a high context length and a reasonable amount of tok/sec?
STT: Voxtral TTS: Pocket Longer video:
Got this used Dell Wyse 3040 for 20$, Paired it with my old laptop's SSD
Me : Hand the work over to the next agent and stop. Q38FN : Aye. Let me do a quick, small, permanent improvement first. This is genuinely useful and it's cheap. Q38FN : (Works for another hour.)
Reddit from r/homelab.
I have an app that processes ~500 different documents every day. An agent analyzes them, sorts them, tags them, extracts various legal nuances, creates descriptions, etc. So far I've been using Qwen 3.8 27B Q5 with a 150K context window, but I'm wondering whether I should switch
Article · Ollama
What's Changed ollama launch claude no longer shows a warning about Claude connectors by @hoyyeva in #18878 ollama launch codex no longer warns about service tiers by @hoyyeva in #18879 Paused background model upgrade for embedding and other request performance, by @jmorganca in
Ok, so everyone/everywhere says ‘run your own’. Fair, and I am capable to learning. ‘It’s all plain text when you post on browser based AI’ - I get it. But I’m not a computer builder… Where do I start? I’m not a coder, I just want to research that I consider more personal/sensiti
Currently contemplating adding either M5 Ultra 256GB or dual DGX Spark in addition to existing 5090. I have been researching for past few days and have zeroed in on dual sparks because higher throughput for agentic coding / multiple parallel requests and prompt processing advanta
For the past few weeks I've been trying out converting existing dense models to sparse Mixture-of-Experts models, with no pretraining from scratch. The Idea If you can turn a dense model into an MoE that only runs part of its MLP per token, you get a model that's cheaper per toke
Basically I am fed up of Google Transcribe (it's genuinely terrible). Is there a way to use Whisper Tiny instead? I know there is very little chance but I thought since Android is open source perhaps...
I'm trying to figure out the best long-term upgrade path for my setup. Right now my GPUs are spread across several consumer PCs: Main: 5900X, 32GB DDR4, 2× RTX 3090 24GB. Secondary: i3-10320, 16GB DDR4, 2× RTX 3060 12GB. Daily driver: 5800X3D, 64GB DDR4, RTX 3080 10GB. Other hard
This will be my third update on the bc-250 cluster and for my first forray into local ai I have been having a blast. This has turned from a weekend experiment into something I think I will use in the future. I am running 4 BC-250 boards connected by 2.5gb ethernet adapters to a 2
I love Dash2Dock Lite, but Icedman is always a week or two before updates. I randomly ran pacman for the first time in weeks, not realizing GNOME had a major update. I told it to fix it in DeepSeek Harness. I was only missing my dock for about an hour.
Be careful out there of deals now. If it’s too good to be true…
Is an AMD price incoming as well? Nvidia just discontinued the 5090 in favor of RTX Pros
I’m seeing 6-7 week delivery wait time for M5 Ultra Mac Studio 64 core version versus 15-17 weeks for 80 core version. They both Ultra and have the same memory bandwidth. Is the speed difference that noticeable for prompt processing and token generation? Is it worth the $1200 pri
Based on my experiments and feedback from claude, it seems extremely unlikely and even if I get it working, not practical because of minimum context. But just asking here as a last effort. I'm able to run nvfp4 brilliantly and high context with images (or full 256k without images
Basically, think of HTTP load-testers, but for inference. I want to throw generations (parallel) at my inference server and see how it behaves under sustained load and meassure token generation and prefill and figure out the actual storage needs in-use. For HTTP stuff, I was usin
Just wanted to share a build I am doing for a client. $18k. Specs: Threadripper 9970x 128gb DDR5 ECC 5600 4x AMD Radeon R9700 AI Pro 32gb(128gb total VRAM) 1600W PSU(GPUs are undervolted and under 210w each) Engine : using a fork of Radiance to serve Qwen 3.8 Next flash/27b and D
I'm genuinely shocked! I was able to run Qwen3.8-Flash-Next-oQ4e-mtp on M3Max 64GB with oMLX! Yes, there are other 3bit quants for 64GB Mac, but 4bit is the lowest quant I'd tolerate. I manually quantized the original model from the Qwen repo to OQ4E using oMLX, and it turned out
quick question, what model excels at highly accurate personal accounting? I mention high accuracy because I’ve used older Qwen models with poor results. a simple two page PDF, fed into an older 27B model missed three out of fifty line items consistently. if I want to do a years w
Does LM Studio support Embedding Gemma 2 as of current version? My Qwen 3.8 27b model couldn't utilize it, and I'm wondering if it's truly a LM Studio problem (my agent's position) or if there's some issue with how my agent is trying to call it
I wanted to compare gemma 4 31b qat q4 and qwen3.8 27b iq4_nl, so I run SWEBench verfied on them. The original gemma 4 chat template doesn't work with preserve reasoning, so google released an update such that it can work with agentic coding:
hello, I currently use swift qwen3.8 gsq rco in iq3 xxs in strata. The model is only used for hermes agent. I have 64GB of ram + 32GB R9700 where the model runs on. But I also have an rx7800xt with 16GB. For hermes agent i want to run a small but intelligent enough model as an ex
EDIT to address that the title is misleading: Qwen3.8-Flash-Next Strata pi beat Claude-code Opus 5.5 on a 5090 on a benchmark dedicated to compare model for everyday tasks. not to discriminate frontier model. --- Original post: I'm using Qwen3.8-Flash-Next on Strata with pi as th
I was just dealing with the bloated monstrosity that is Hermes Agent, which has a bloated monstrosity web UI. While researching, it turns out I'm supposed to be using a different bloated monstrosity called Hermes Desktop, which is an Electron app. Electron apps are the bane of my
Now all we need is a new model. Let me remind you that MiniCPM-V-4.7-35B-A3B was spotted on r/LocalLLaMA a few days ago (but the model was later hidden on HF).
Article · Open WebUI
Added ☎️ Realtime voice calls. Administrators can now switch Call mode in the Voice calls section of Audio settings from Standard to Realtime, so Voice mode talks through an OpenAI Realtime voice model (gpt-realtime-2.1-mini with the Marin voice by default) that handles small tal
Reddit from r/LocalLLM.
Article · llama.cpp
server: leave a busy slot untouched when a request pins it ( #30295 ) A request asking for a busy id_slot still ran the prompt cache update on that slot before being deferred. When the RAM cache held a better match, it was loaded into the slot while another request was still gene
I recently started using local LLM for mostly coding tasks for my personal projects. I also have found local models responding much better for topics such as health, fitness, ergonomics etc. compared to the quality I had observed with ChatGPT Go (which is still a basic level) and
Article · llama.cpp
model : support MiniCPM-V 4.7 ( #29416 ) mtmd : add MiniCPM-V 4.7 support Signed-off-by: tc-mb tianchi_cai@icloud.com model : allow mrope time from an extra position slot Signed-off-by: tc-mb tianchi_cai@icloud.com Update conversion/minicpm.py Co-authored-by: Sigbjørn Skjæret sig
I’d say we’re a year-ish away from local + good harness being enough to write 90% of code. Looking at you, Qwen. Edit: damn, sounds like you all think Qwen3.8 Next Flash is there already… going to have to give it a spin…
For what it's worth. I'm a mechanical engineer by education and software engineer in practice over the past ~20 years. Playing around with local models has been a recent hobby. Today I decided to do a few tests on things that I'd consider analogous to "real" work that I do, tryin
Article · llama.cpp
cmake(s390x): disable z17 target for unsupported compilers ( #30297 ) Website:
If there is something I like about the indie SLM community is there is always some drama every week about something. I mean, look, I don't like to disrespect others or their work, but this is some crazy statements made by banaxi-tech. here is the link to the HF discussion:
I was looking into finalizing my daily driver and it was not clear which one to chose. I have a 5060ti 16GB so I picked the following contenders, which seem to be working at a reasonable speed on my hardware: Swift 1.5 IQ2_XS GSQ-RCO under Strata Flash Next IQ3_S GSQ-RCO under St
So, while doing a search to read the model card again, and because I didn't memorize the huggingface URL, I saw an AI "answer" at the top of the search, stating that while it natively supports a ~244k context, it could go to 1M using YaRN. That's the first I have heared of both,
I've been experimenting with local speech-to-text on my Windows laptop. After getting Whisper running on both the integrated GPU and NPU separately, I'm wondering whether there's a practical way to combine their processing capacity. I thought I'd share what I've tested so far and
Hi, Because sometimes you need both speed AND quality, I made my own Qwen 3.8 27B Swift 1.5 quant. It's an 8 bits AWQ + GPTQ quant. Made with love and precision (read the model card) on my 4 RTX 3090, I believe it's the best compromise you can actually get on 48Gb of VRAM. Runnin
i am looking for Small-Medium Model, specifically made for Cyber usage, Vulnerability Discovering / Researching Heavily Coding Reverse Engineering Low-level Coding Scripting i rather read you guys Experiences, and See which one is the best for my Usage My Specs for additional: I7
Nvidia reportedly halts GeForce RTX 5090 production in favor of AI data center and professional GPUs - impending supply drought expected to drive up prices, RTX 5080 24GB rumored as new gaming flagship
Article · llama.cpp
Allow configuring --slot-save-path via LLAMA_ARG_SLOT_SAVE_PATH so it can be set from an EnvironmentFile in a systemd unit file.
I love the idea of running local models on consumer hardware. I currently use `llama.cpp`, but I’ve been looking for a "better" alternative for a while now. I want something that utilizes my system resources more efficiently-for instance, by managing my RTX with 4GB VRAM more int
I've been using Claude Opus 5.5 to speed up Qwen3.8-27B on my PC (rtx 3090), it wrote a CUDA megakernel that is 1.4-1.9x faster than llama.cpp depending on the task/context length. The results and code are below: Results (same hardware, my megakernel vs llama.cpp with MTP): - Wri
DigUp is a free Mac app that runs Google DeepMind’s new EmbeddingGemma 2 locally over your own files. The model puts text, images, audio and video in one space, so you describe what you remember and land on it: “zebra in a video” opens the clip at the moment it shows up “where th
Article · noted.lol
We all have those self-hosted apps we spent hours setting up, only to barely use them. So why do we keep them running?
Spent the past month tweaking and experimenting with many different numbers to achieve 30tps. Hardware: -Rx6700xt 12gb vram (AMD) -2x16 ddr4 3200 ram -r5 5600x -llama.cpp vulkan sdk -window11 (no wsl switching since im not used to the environment) Im looking for any improvements
A conversational AI plush toy where nothing leaves my network. Everything runs on my LAN: STT: Whisper LLM: Hermes 3 Llama 3.1 8B (Q8_0) TTS: Kokoro Memory: per-person (LangGraph). It knows who it's talking to and remembers past conversations Vision: face identity (dlib) + emotio
Reddit from r/LocalLLaMA.
Been seeing quite a few posts about Strata lately, so I figured I'd give it a shot on my RTX PRO 6000 and I am very impressed. Most of the benchmarks I've seen are using IQ2 or IQ3 quants, so I wanted to see how Unsloth's UD-Q4_K_XL performs instead. Setup: GPU: 1x RTX PRO 6000 W
Article · llama.cpp
spec: properly handle mtmd input for mtp nits
Local flea market...and it's about to rain...
Let's say I run a local Jev-like decision model on my machine. What are the use cases? I already use it for deep eval testing but what are people usually doing with them? It a bit of "solution in search of a problem" but I'm exploring and want to be sure that I'm not missing anyt
Article · llama.cpp
opencl: add bin kernels kernel_gemm_moe_q4_k_q8_1_dp4a_bin, kernel_gemm_moe_q6_k_q8_1_dp4a_bin ( #30187 ) Website:
Strata is impressive: it runs Qwen3.8-Flash-Next, a 125B-parameter mixture-of-experts model, on a 12 GB graphics card. But its authors have said macOS is out of scope. I still wanted it on the Mac. So I spent a day and a half and wrote strata-mlx. It is an unofficial MLX engine,
what size do you want? tell them on X:
Around 3 weeks ago, I posted on this subreddit about how Jev used my one-year-old architecture, og post:
Reddit from r/LocalLLaMA.
Article · llama.cpp
CUDA/HIP: fix race in flash_attn_ext_f16_process_tile when nbatch_com…
Article · llama.cpp
ggml-cpu(s390x): rename ulong to uint64_t sized types ggml-cpu(s390x): rm comment Signed-off-by: Aaron Teo aaron.teo1@ibm.com
Article · llama.cpp
Keep the existing RDNA3/4 policy unchanged and update only rm_id to use 4 rows for NVIDIA except pre-Turing.
Article · llama.cpp
Signed-off-by: Aaron Teo aaron.teo1@ibm.com
Running HauhauCS/Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive Q4_K_M with llama.cpp at ~600 tok/s prefill and 23 tok/s decode, 131k context window, Q8 KV cache - on an RTX 2060 6GB + 32GB DDR4 RAM. Speeds start at ~600 tok/s prefill / 23 tok/s decode on an empty KV cache. As co
webAI released TwIL LM3 Pro on September 30. I haven't seen it posted here yet, so I went through the model card, and their comparison chart is attached. It's a 3.66B model built on IBM's Granite 4.2 3B and tuned only for formal logic. That means things like checking whether a co
Hi All: I have an Asus ESC 4000 G3 with 128 GB DDR4 RAM - I tried putting in my V620s but couldn’t put more than 2. Sadly, pivoted to T4s, these are 72 watt passive cards and I thought I could use them like how I use the Mi50 32GB - but I was very wrong. It has no support from Nv
Article · llama.cpp
sycl : accelerate MXFP4 MoE with arithmetic decoding and weight reordering ( #29809 ) Website:
Basalt: Blackwell inference engine Hi all! So over the last week I've been working on a Strata fork that's heavily tuned and can achieve throughput up to 2.6x what Strata usually does on the same weights. It's designed as a specialized engine that only supports Qwen3.8 Flash-Next
Article · noted.lol
Dockhand has revised its licensing terms, introducing new commercial restrictions and changing when its code becomes open source. Here's what it means.
Strata vs Infernix on the same model, RTX 5090 - A/B at 262k and 512k context Hi folks, another inference engine post for your feed. I was blown away by Strata, but saw someone else post about Infernix with some wild claims, and it doesn't seem so well known, so I've spent a day
Reddit from r/LocalLLaMA.
SlopSoup TV is a live, never-ending pixel-art TV network. Every script, character, voice, camera cut and schedule decision is made by models. It's made to be very absurd, dark and strange. Hardware: one Hugging Face Space, 16 vCPU, no GPU. Everything below runs on CPU next to a l
It's been a long time coming, and I should have done this in the first place. But LumaBrowser is now open source. Since I first released it a year ago, it's had several thousand installs and an absolutely tremendous amount of feedback. And it's come a long way. I don't call this
I wanted the uncensored Qwen3.8-27B (llmfan46's Heretic build, MTP head preserved) on my RTX 4080 with MTP on (because I've been testing TONS of configs, and realized that would be the one) And none of the published quants were built for that: the good IQ4_XS doesn't leave room f
Article · Simon Willison
The Deno team released the first version of celld back in August - their open source implementation of the Durable Objects pattern from Cloudflare Workers. Today, Cloudflare are acquiring Deno outright, with the goal of building on celld to "make workerd self-hosting a first-clas
Article · llama.cpp
vendor: apply deep nested json patch from upstream ( #30253 ) Website:
A bit of backstory : i got lucky in August and was able to grab a CMP 170 HX 8GB for around 500$ on alibaba. I tried later to get a second one for around 1k$ but my order got cancelled and prices went to the moon... I unlocked it with
Hello all! Thank you to everyone that contributed their feedback and notes for LlamAmpere the last time I posted. I've continued to chip away at improvements for the 3090 crowd -- this release is modestly faster (3-4%), with a few hundred MB smaller runtime (if you're using YaRN,
Has anybody built/released a music embedding model trained on a large library of diverse music?
ASUS ProArt P16 OLED 16" Laptop - NVIDIA RTX Spark™ N1X, 128 GB RAM, 2 TB SSD, Nano Black For the low price of £5999
Disclosure: I work at H2O.ai. We released H2O-Lightning-4B, an open-weight (Apache-2.0) model for the "decisions API" style of inference that Jev made popular: you send a state plus typed questions (pick one / yes-no / score), and get calibrated probabilities back from a single f
Article · llama.cpp
CUDA: fix round issue, under MSVC the CPU and GPU agree ( #30229 ) Website:
Article · MarkTechPost
OpenAI’s Decisions API is now in public beta on GPT-6 Luna. It returns typed probabilities, choices and scores about 10x faster than the Responses API, billing $0.10 per 1M input tokens with no output charges. The post OpenAI Decisions API Hits Public Beta With 10x Faster Typed A
Article · LiteLLM
Verify Docker Image Signature All LiteLLM Docker images are signed with cosign . Every release is signed with the same key introduced in commit 0112e53 . Verify using the pinned commit hash (recommended): A commit hash is cryptographically immutable, so this is the strongest way
Article · llama.cpp
graph : reorder get_rows for embeddings ( #30160 ) graph : reorder get_rows for embeddings cont : fix gemma4 and improve input embedding construction logic cont : add TODO for lora cont : fix raw embeddings path gemma4 : avoid ple cast in embeddings path Website:
Reddit from r/homelab.
Article · llama.cpp
llama-bench: respect -fitc if bigger than required benchmark size ( #28331 ) Assisted-By: opencode,llama.cpp,Qwen3.6-35B-A3B,Qwen3.8-27B Website:
Live results, coding benchmarks included, agentic benchmarks included, domains and other classifications filters included. (Spoiler: DeepSeek-V4-Vision-Exp rules, but other models have their rule areas):
(Don't judge by the screenshot, the cache is cold. It hits 24+ tok/s with a warm cache!) About two months ago, I made a post here asking whether predicting which MoE experts would be used on the next token could actually help speed up CPU/GPU offloading. Original post: Tried pred
Article · llama.cpp
CUDA: Remove redundant CUDA copies after SSM_SCAN ( #29807 ) CUDA: fuse copy of updated state snapshots into recurrent cache with ssm_scan CUDA: remove redundant cuda copies with K==1 (non spec-dec) scenario as well Website:
In the dawn of the LLM craze, when wild Llama2's roamed free and 4k tokens was considered a fairly long context for local models, I forked an interactive fiction and MUD library called Tale to experiment with LLM generated content. The idea being a text based adventure with total
Article · llama.cpp
opencl: fix kernel compilation for a6x GPUs ( #30176 ) opencl: skip kernel_cpy_f32_f32_pack on A6X to avoid shader compiler crash The A6x compiler backend found in iot device with a623 (E031.50.31.01) cannot handle kernels with a large number of arguments. Skip this kernel for A6
It seems Dario's "too powerful for you users" strategy is paying off: we have two open models at the top of the leaderboard, surpassing every single model from Anthropic. Open source prevails. Even Mistral Large 4 is better!
Over the last year, I've been working a lot on structured legal data. Here are some knowledge graph representations that may be of interest to those of you building your own projects. 5s on each representation. To pre-empt any questions: No I do not think KGs are super useful for
Article · ServeTheHome
Today we are taking a look at GMKtec's latest Ryzen AI Max+ 395 mini-PC, the EVO-X3, an interesting AI workstation with the added twist of an OCuLink Port The post GMKtec EVO-X3 Mini-PC Review Looking at a 128GB AMD Local AI Box appeared first on ServeTheHome .
Article · llama.cpp
common : read a GGUF's trained context from its metadata common_get_gguf_n_ctx_train opens only the file's metadata (no_alloc, like common_get_decision_type) and reads .context_length, so a caller can learn the trained context without loading the model. It accepts both u32 and u6
Article · llama.cpp
model : use exact GELU for ModernBERT encoders ( #30108 ) model : use exact GELU for ModernBERT encoders Assisted-by: Codex model : keep tanh GELU aliases on ggml_geglu Assisted-by: Claude Opus 5.5 model : map gelu_python to ggml_geglu_erf Assisted-by: Claude Opus 5.5 Website:
For those of you who have been wanting to utilize your dual Spark cluster for image/video gen with ComfyUI - today is your day. I started this personal project back in April of this year, and it’s at a point where I’m fine with a public release. To put this as simply as possible
Article · noted.lol
Pinchflat-ngx builds on the original YouTube downloader with a redesigned interface, better download controls, PostgreSQL support, and more.
Reddit from r/homelab.
Hello guys, I have a summary workflow for my school where I have Gemini summarize the lecture content, then generate infographics on said content. However, gemini's style keeps constantly changing, running out of limits and other stupid bs like that. Its unreliable. What image ge
Article · llama.cpp
llama: keep the backend sampling graph static across ubatches ( #30223 ) The reserve builds n_outputs_max_per_seq sampling chains per sampler, while a decode built one per output row, so the graph changed its topology after the reserve and GGML_SCHED_NO_REALLOC builds aborted on
Blog Post : ML Drift: Next-Gen GPU AI/ML Inference at the Edge - Google Developers Blog The Google AI Edge Team is excited to announce the open-source release of ML Drift , our high-performance, cross-platform, on-device GPU compute engine specifically built for on-device AI/ML i
I tested Mellum2.1-12B-A2.5B (Q8) locally using Pi and llama-server. All five tests were one-shot. Results were pretty mixed: - Pelican SVG, Browser OS, Minecraft: Poor results. - Bouncing Hexagon: Physics were okay, but surprisingly it made it run in the terminal. - Flappy Bird:
Reddit from r/LocalLLaMA.
Article · llama.cpp
ggml: fix OOB write in ggml_acc with negative offset ( #30135 ) ggml_acc_impl narrowed a size_t offset to int32_t without checking that it fits, so a large offset could truncate to a negative int32_t. The forward then sign-extended it to a huge size_t and the bounds assertion wra
Article · llama.cpp
meta : handle host views ( #30217 ) meta : handle views of tensors allocated on the host A view shares the memory of its view_src, so ggml-alloc never allocates a view in the buffer of the split it lands in - the scheduler copies the source into the split and the ops that use the
I got Qwen3.8-Flash-Next-GSQ-RCO-Abliterated running at IQ3_S with just 12GB VRAM and 32GB system RAM, achieving 20-30 tok/sec decode (Q2 achieves 39-45 tok/s) & 300 to ~90 thousand tok/sec prefill @ 131k context, on a custom fork of Strata. This fork has tonnes of architectural
Article · llama.cpp
webgpu: use 2D workgroup dispatch for all the ops which use 1D dispatch (e.g., rms_norm) ( #30219 ) remove 1D workgroups dispatching formatting Website:
Reddit from r/comfyui.
I've finally got a successful run, it took over 20,000 seconds, or over 5 1/2 hours. I changed the random seed and now running it again. 8gb GPU and 32gb ram. I'm hoping, wishing, praying for the price to drop on GPUs.
to run a Comfyui desktop locally , in 8gb vram condition (my laptop ) first time to had known ComfyUI , huggingface , ai things , checkpoint , vae , xxxloader .(actually dont need to know that) though met some configue promble ,finally figured it out under AI guidence . following
Article · llama.cpp
vulkan: fix rms_norm workgroup count overflow ( #30145 ) Website:
I have like 150+ images of my mom’s recipes, I want to extract all the text from them and categorise them based on dish type. Is there any model I can use to do all this stuff. I’m very new to this. TIA
I posted on here a while back asking for advice on upgrading to an Epyc system. Thanks to all who commented. I’ve taken the plunge and ordered: - Epyc 7443 CPU (24 cores, 48 threads, 128 lanes) - H12SSL-NT-B motherboard (the variant with two 10Gb Ethernet ports) - 256GB 2666MHz 2
Qwen-Image-2.1-Turbo, create and edit images in just 8 denoising steps! Open weights now available! Built on Qwen-Image-2.1, Turbo is an accelerated checkpoint on the same 7B visual generation architecture. Fewer steps does not mean lower quality: it still generates strong 2K ima
Hey guys! I need help with an AI image pipeline. I’m building a product that lets people see furniture from online stores in their own apartments. For shops it’s an extra source of information: customers make better decisions and get more than they’d expect from a store. Frontend
Reddit from r/comfyui.
I have rtx 4090 and 64 gb of ddr4 ram. This is enough to fit smaller quants of qfn, but I found them not great and switched back to qwen3.8. However, there are option that bother me a lot, buy huawei atlas duo, thats another 96 gb of ram thats twice faster than system ram and ano
Hardware Config: RTX 5090 Windforce 128GB DDR5 (Running at 3600MHz cause AM5 struggles with 4-slot DDR5 speeds) Ryzen 7 7800X3D Crucial Gen5 SSD Model Config: Minimax Ref2V int8 Pruned Qwen VL int8 CK Attention Comfy run on WSL Ubuntu with cu13 Mode: Ref2VA 5 Reference Images (2K
Youtu-Parsing-Omni is a compact (5B) omni-modal parsing model. Given a single input - a document page, a natural image, a chart / flowchart, a geometry figure, an audio clip or an audio-visual video - it produces one structured JSON envelope that covers both perception (layout el
Article · noted.lol
Portainer 3.0 is shifting its focus toward Kubernetes, but what does that mean for Docker users? Here's what's changing and what happens to Community Edition.
Last month I created a portable memory format as a .txt file as a small project. I named it as: Memory Journal, in a shape of a skill card. It's essentially a large prompt wrapped in a text file. Initial testing went better than I have expected, then did more tests and improved i
I've been out of the local image gen scene for a very long time as I opted for ChatGPT Images (please forgive me!) for the bulk of my worldbuilding character and places design work, but recently I've got a bit fed up with the quirks that it has and thought about moving over to do
Could you recommend the most suitable model for replacing people in a video while leaving everything else unchanged but maintaining the movements of the people? I see many trending videos with The results were very well done, and I wanted to see what they could use. Thanks. submi
Article · llama.cpp
musa: drop mp_21 from the default architectures ( #30203 ) musa: drop mp_21 from the default architectures Signed-off-by: Xiaodong Ye xiaodong.ye@mthreads.com Address review comments Signed-off-by: Xiaodong Ye xiaodong.ye@mthreads.com Address review comments Signed-off-by: Xiaodo
Hi. Im getting this error when trying to render a workflow... ComfyUI Error ReportError Details Search for cudaErrorNoKernelImageForDevice' in
Running the H3 + character swap LoRA setup from the post this week. RTX 3060 12GB, 32GB RAM, around 14 min per clip at 832x640. The swap itself is clean, problem is I want new dialogue on the result and the mouth only moves like the source video did. Wav2Lip leaves a visible box
Article · llama.cpp
ci: fix Models Backend webgpu by supporting GGML_OP_DUP ( #30216 ) The mixed batch path of PR-29622 writes the token rows with set_rows into a dup of the embeddings. WebGPU did not support DUP, so the dup ran on the CPU while the set_rows writing into it was scheduled on WebGPU,
Article · llama.cpp
server: use port 9931 by default ( #30159 ) server: use port 9931 by default revert unrelated changes Website:
Article · llama.cpp
metal: add the 128/96 flash attention kernels, accept every compiled pair ( #30209 ) Website:
I use their model in my up with hand-rolled inference in C. HF page says: "This project's sample code is released under the MIT License." and "OpenRAIL-M License" for the model. Can I still use their model after liquidation?
I see a lot of stuff online, but I’m a beginner… please help! I want to see what I can do with my current setup. I don’t mind generation times. At the moment, AI is recommending me Q8 Quen Image 2.1. I realise ComfyUI isn’t suited for beginners, but I’m fairly confident I can fig
All anyone ever talks in this sub is about x new model having x benchmark scores and how well it can code. Or some new tech to increase t/s for qwen. So I wanted to go back to the roots of this sub and talk about how well local models can write. I am not going to mention closed s
Reddit from r/comfyui.
Hi, I'm bosen12. I've been generating with Illustrious-based models for a while, and I kept struggling to remember Danbooru tags or tell what a single tag actually did. So I built Mochi, a free open-source tool that runs locally and sends prompts to your existing ComfyUI. What it
Article · llama.cpp
CUDA: pass src1 precision to host MMQ config helpers ( #30168 ) CUDA: pass src1 precision to host MMQ config helpers Signed-off-by: ynankani ynankani@nvidia.com make src1 prec explicit Signed-off-by: ynankani ynankani@nvidia.com Signed-off-by: ynankani ynankani@nvidia.com Website
Training large language models (LLMs) is resource-intensive, and adapting them for diverse deployment scenarios with varying computational constraints remains challenging. While elastic architectures enable flexible model deployment and sparsely activated models allow input-adapt
Article · n8n
2.43.3 (2026-10-09) Features core: Block create/update of workflows with deprecated nodes ( #40706 ) ( 3c4f6e3 ) Google Gemini Node: Show Nano Banana 2.1 and 2 Lite in the image model list ( #40655 ) ( 1830ccf )
Article · n8n
2.42.6 (2026-10-09) Features core: Block create/update of workflows with deprecated nodes ( #40707 ) ( ff1a24c ) Google Gemini Node: Show Nano Banana 2.1 and 2 Lite in the image model list ( #40656 ) ( 8dfc695 )
Article · MarkTechPost
Underdog Saluki 27B is a 7.89 GB, 2-bit GGUF of Qwen3.8-27B under Apache 2.0. It beats the 54 GB original on tool calling but gives up ground on competition math and reasoning. The post Meet the Underdog Saluki 27B: A 2-bit Qwen3.8-27B That Beats the Original at Tool Calling appe
Speculative decoding accelerates autoregressive generation in large language models. In each drafting stage, a lightweight draft model proposes tokens that the target model subsequently verifies. With increasingly capable draft models, we find that the target model frequently acc
Hey everyone, My last post about this app got buried in the H3 posts and never really reached anyone, so I'm giving it another shot - hopefully it finds a wider audience this time. SilkStack Image Browser is a desktop app for organizing your generated images and videos, and for v
Article · llama.cpp
hexagon: fix IM2COL patch-embed DMA ring overflow ( #30189 ) hexagon: fix IM2COL patch-embed DMA ring overflow The exact-tiling (stride == kernel, no pad/dilation) IM2COL DMA kernel issues IC KH DDR->VTCM descriptors per output row without checking the return value of dma_queue_p
Conditional memory architectures such as DeepSeek Engram use input n-grams to look up learned embeddings, expanding the capacity of large language models (LLMs) with limited additional computation. Beyond model scaling, this architecture has demonstrated the potential to decouple
Hey guys - primarily a research release with working models, Not so much a model as a psuedo-new quantization technique. I've been experimenting with a modification of ISTALab's RCO algorithm that can quantize models on a strict VRAM budget. It's at its core an approximation algo
HauhauCS ships their uncensored Qwen3.8-27B as GGUF only. NInfer, a C++/CUDA wanted its own format. Now the same model that ran at 91.6 tok/s / 131K under llama.cpp does: 262K context (the model's full native window) ~130 tok/s decode with MTP3, 70.8% acceptance 3,591 tok/s prefi
I've been learning how to use ComfyUI to generate images, but I'm struggling to make small edits while keeping the original image looking realistic. The workflows I've tried tend to regenerate the entire image with increased saturation, and the changes I ask for often don't turn
I made a music video on a local RTX 3060 12GB VRAM. I only work on it bit by bit because I'm short on time, but it's still so much fun. Currently using Minimax VDN DMD 8-Step Turbo LoRA. Plus, it's only been two months since I first discovered ComfyUI on my local PC. #ComfyUI #AI
Reddit from r/LocalLLaMA.
Article · Ollama
Model upgrades Models downloaded with earlier versions of Ollama are upgraded in the background the first time you run them, for better performance and compatibility when running on llama.cpp. To make downgrading safe, Ollama keeps the original copy as a backup, so upgraded model
Article · llama.cpp
sycl : Q5_K reorder-layout MMVQ and fused GLU ( #29375 ) Website:
I tried dozens of different ways of rephrasing prompt, but nothing made more effect than copy/pasting entire text. You can use this one for switching a character in a video, but you might need to add extra detail. subject_definitions: is the girl in . is the girl in . is the move
LemonSeed Studio is an iPad editor/IDE with on-device inference on an AMD GPU in a Thunderbolt enclosure. It embeds the unmodified upstream Linux amdgpu + amdkfd driver (mac_linuxgpu) as a PCIDriverKit extension, and runs LemonSeed Engine (LSE) on the GPU. In the photos: iPad Pro
Hardware available: 2080ti 11gb 5070ti 16gb Goal: I've downloaded my whole internet history (reddit posts, emails, chats, etc) and i'd like to train a model to think, act, and talk like me. I'm currently running a qLORA pass with Qwen3.5-9B as a base, but when i run A/B tests, it
I'm having AI models play Cat Goric, a 2D platformer I made in 2021 during js13kGames:
Referring to specifically Unsloth's UD_Q4_K_XL quant because that's going to be a question, and is relevant regardless. It's old now. It's not great at coding medium sized or even small-ish projects. I wouldn't hand my codebase to it by any means. It hallucinates, like any other
There are plenty of videos around with characters that look like AI or have plastic skin or talk like a robot. What about emotions, or subtle facial expressions, or natural pauses? The solutions and advice usually offered include complex prompt guides or access to paid platforms.
Just noticed this today when I went to run the built-in "UPDATE" script and git failed because there was no common ancestor. Looked into why, and apparently every historical commit has been re-written to strip the "Co-Authored by Claude" text from the descriptions. Personally I t
I started this project because I wanted to see what the Harry Potter characters might look like as real people, based on the books rather than the film actors. My goal is to go beyond obvious details like hair colour or Harry's scar. I want the faces, builds and expressions to fi
Article · Immich
This patch addresses various bugs around people sharing, as well as a scroll issue on mobile. Further, there is now an indicator showing which people are shared. What's Changed 🌟 Enhancements feat: shared person indicator by @danieldietzler in #32223 🐛 Bug fixes fix(mobile): keep
Seems 1-slot design, so you could cram in quite a lot into a case
This table to help anyone looking to build a budget Data Center homelab. I copied the bulk of value based, mid level, decent speed results GPUs and feed it to AI or SI and here are the recommended results. Data taken from Llama.cpp discussion thread: Performance of llama.cpp with
Bone stock Ref2Vid minimax template. I created a crappy drawing in GIMP, threw it in, prompted it out with a standard Minimax stylized prompt (hand written), and got this cool video. Be creative people! If anyone wants the prompt here it is: Technicolor 1980's oversaturated VHS s
If you want to try it out here is a smaller quantisation iq3_s works really well in my codebases. Original Model:
Article · llama.cpp
CUDA: improve top-k algorithm selection ( #28713 ) CUDA: radix top-k for large row counts Replaces CUB's per-row DeviceTopKKernel with a grid-over-rows radix select, gated on GGML_CUDA_TOPK_RADIX_MIN_ROWS. On qwen4exp at 34,816 tokens this cuts top-k from 1,671,253 launches / 5,7
Let's preface this with the fact I have 3090, 4090, 2080ti - and lots of stable long running compute heavy workloads. I recently got a 5070ti (I am not willing to pay ridiculous money for 5090) And it's been working reasonably well, until I left it running on a 6 hour CUDA job. N
Hey everyone, I've been working on a tool called ISNAD for the self-hosted community. It's an open-source (Apache-2.0) engine that creates a locked, cryptographically signed record of an AI's answer-showing who or what handled it at each step. The goal is to provide "proof of cus
came across a pretty interesting technical breakdown of chatgpt's newly launched "intelligent ui" feature, and thought this subreddit might find it interesting. for anyone unfamiliar with the concept, intelligent ui is essentially openai's take on generative ui . instead of restr
Hey folks! Aritra here from Hugging Face. We introduce a no config, no key, no setup way to directly chat with a model hosted with the Hugging Face Inference Providers. `ssh chat.hf.co` And you are good to go. 🔥 Let us know what you think about this.
Article · llama.cpp
model : fix DFlash output head sharing ( #30111 ) llama : fix DFlash output head sharing Assisted-by: Codex dflash : read tied output weights from GGUF metadata Assisted-by: Codex llama : share tied word embedding metadata Assisted-by: Codex llama : remove DFlash embedding head f
Post title is slightly misleading, I don't think you can get these for $350 each anymore but they're still pretty cheap all things considered. They're Radeon Pro V620's which are older RDNA2 enterprise cloud gaming cards with 32 GB VRAM. (Ignore the RTX 4090 on the side, it's jus
tldr: Open sourced my prompt to game suite:
recently saw a bunch of open decision models pop out of nowhere in the last two weeks (laya, liquid's d1, cloudflare's clef-flash, interfaze's lev), so I wanted to see how far apart they actually are on the same GPU(yes, model size is a huge factor, but still isn't the only facto
I want to share a TTS server with an OpenAI-compatible API that generates speech really fast (about 0.3 seconds for a sentence on an RTX 3080) and can clone a voice from a short reference clip. I’ve optimized the server so generation starts quickly, and it processes long text in
Article · llama.cpp
CUDA : looped PAD kernel for more than 65535 rows or slices ( #30147 ) Website:
Fine, pp is still slower, but I'm slowly coming around to the idea of MTP finally being useful on Apple Silicon, and this is the first time I'm seeing a model outperform ds4 (and that's with IngeniousIdiocy's M3U tuning). MTP seems to have no advantage there as was always the cas
Article · Ollama
When a legacy GGUF which requires our llama.cpp patch set is loaded, we do a lazy conversion on disk to make it llama.cpp compatible. To support downgrades, we temporarily hold both the new and old GGUFs and create a shadow v1 manifest tag to prevent the downgraded server from de
Article · llama.cpp
CUDA: fix CCCL version guard breaking on major version rollover ( #29453 ) CUDA: fix CCCL version guard breaking on major version rollover The guard compared the major and minor components independently: = 3 && CCCL_MINOR_VERSION >= 1"> CCCL_MAJOR_VERSION >= 3 && CCCL_MINOR_VERSI
Hey everyone, Jovan from UkisAI (Swift Qwen) here! For those who don't know us, UkisAI is a small lab making tiny frontier LLMs, tools and datasets (+doing it open-source!). I'm one of the guys running it aka I train the models and post on Reddit. Our first open-source release is
Article · Immich
v3.3.0 Note We're collecting feedback related to people sharing in this discussion:
Reddit from r/LocalLLaMA.
JetBrains/Mellum2.1-12B-A2.5B-Thinking-GGUF A small moe!
Article · noted.lol
Immich's latest update brings improvements to photo sharing and management. But if you're using it to store your family's photos, there's something important you shouldn't overlook: your backups.
Article · llama.cpp
vulkan : fix TOP_K for +inf/NaN inputs and k = 1 on negative values ( #30107 ) The bucket search in topk_nary_search.comp started from the range [0, 0xFF800000), which ends just below the ordered-uint mapping of +inf, so +inf and NaN were never counted. A workgroup block with few
The other week I posted about Jev vs. Kev compared and since then, OpenAI released the decisions endpoint, Cloudflare released Clef and many here asked about Laya as well. This time we compared six popular decision models by making them play Pac-Man: kev 1.13, Kev 4B, Clef, Clef
Interesting to see improvements and research into quantization aware training (QAT) that can make some really tiny models.
Which is a good image model to run locally to generate images and illustrations for web design? I am creating a landing page and need some help with the resources.
I've been optimizing long-running agents: rewriting their prompts, tools, skills and subagents, and keeping the changes that score better. I've been working on some version of this problem for over a decade (at Apple, my own startup, now Kiln). The hard part is the eval environme
48Gb coming along nicely." title="Attn CMP170hx 10Gb card owners: unlock from 40Gb -> 48Gb coming along nicely." /> For those with 10gb cmp170hx who felt left out, you’re about to get lucky soon. Fingers crossed… Looks like you’ll get an extra 8Gb from 40 -> 48 Gb, stay tuned… su
Another quick Minimax H3 news and goodies round-up, for those who may have missed some items. -> A Person Remover LoRA for H3. "Supply the original video and a clean version of its first frame". The ComfyUI workflow and SAM 3.1 then tracks and erases the person from the video cli
Article · llama.cpp
vendor : update cpp-httplib to 0.60.1 ( #30134 ) Signed-off-by: Adrien Gallouët angt@huggingface.co Website:
Hi all, a bunch of performance improvements have been landed in audio.cpp. The biggest highlight is Higgs Audio TTS, which now runs with around 6 GB VRAM , a 48% reduction in peak memory usage compared to the previous implementation. Thanks to
Article · llama.cpp
hex-mmadd: do not assume aligned read/write when bias-add is fused ( #30133 ) Website:
I've been making short AI films and got tired of jumping between 5 different workflows, so I combined the whole pipeline into one modular workflow. Here's how it's structured, in case it's useful for your own setups: 1. Character sheets (Krea-2 Turbo): front, side, back and close
Article · llama.cpp
cuda : Use byte strides for roll to allow non-contiguous ROLL operations ( #29547 ) Website:
Article · llama.cpp
CUDA: fix norm family kernels when ne[2]/ne[3] exceed grid dim limits ( #28175 ) Website:
Article · llama.cpp
sycl: fuse the delta-net alpha gate (add + unary + mul) ( #29687 ) sycl: fuse the delta-net alpha gate (add + unary + mul) tests: cover the fused add + unary + mul chain sycl: give the fused alpha gate a flat path and pin the node skip Website:
TornadoVM: The Java to CUDA engine:
Reddit from r/LocalLLaMA.
We’re the team at Neuphonic, and we’re open sourcing NeuDecide under Apache 2.0. It takes audio and tool definitions and returns a tool call with arguments, without an intermediate transcription step. The model files total 43 MB, and inference runs on a single CPU thread. Try the
With latest Halogen version update (0.17.2), decode is consistently at ~45 tps even at high context with Qwen 3.8 Flash Next on a 128GB Strix Halo. This is some great work u/peonist-ai . Have been pumping out commit after commit with QFN. Its crazy good for a 177ish billion model
I have posted similar clip a while ago. That was generated with LTX-2.3 + Union Control LoRA. This is generated with MiniMax H3 + character swap LoRA. I have changed the facial expression in the prompt. 832 x 640, render time 12:45 RTX-4070 8GB VRAM, 64GB RAM.
Having a lot of fun experimenting with three.js. Still very far from even a demo but its tons of fun. After this project I'll want to try a godot workflow. 3.8FN truly feels like Claude 4.6 at home. UD_Q4_XL quant btw
Disclosure: I'm a researcher at Hirundo, the company that made this. Happy to answer anything. Qwen ships with the CCP's political alignment trained in. Ask Qwen3.6 what happened on June 4, 1989 and it says "I don't know what you are referring to." A system prompt doesn't reliabl
Article · llama.cpp
server : preserve context checkpoints across slot save/restore Append the checkpoints after the packed server_tokens payload added in #26640 and count them in n_written / n_read, so a restored slot can still roll back to a checkpoint instead of re-processing the whole prompt. ser
There are way too many forks to keep track of, so I've gotten lost. As far as I can tell, Radiance VLLM is best for models that fit in GPUs while some form of llama.cpp is probably best for MOE RAM-spill? My specific current goal is to run GLM5.3-Flash over 6 R9700s + system RAM
Article · llama.cpp
sycl: stage bulk uploads (model loading) through a pinned ring buffer ( #29608 ) Co-authored-by: cwriter cwriter@localhost Website:
I spent the past year independently building Slow Vale, an LLM-driven life simulation. The Chinese server now has 800+ AI residents sharing one continuously running city. This is an engineering write-up about concurrent decisions, dynamic action spaces, context caching, and the o
Article · llama.cpp
model : support classifier_activation for rerankers ( #29692 ) model : support classifier_activation for rerankers Assisted-by: Claude Opus 5.5 model : map classifier gelu to gelu_erf and accept tanh Assisted-by: Claude Opus 5.5 model : default act_cls to tanh, ModernBERT falls b
anyone has feedback about this one?
I’m creating product lifestyle photos for a cosmetics brand, running everything locally in ComfyUI on an RTX 5080 16 GB. So far Qwen Image 2.1 has given me the best results. It’s excellent at following product shape and label references and reproducing materials. My current workf
I'm getting this error when installing some of the models I need. Is there a work around for this? I know it has something to do with how old my GPU is, but idk what to do about that. Tried pretty much about everything I could find at this point Radeon RX 580 Series AMD Ryzen 5 5
Article · LiteLLM
Verify Docker Image Signature All LiteLLM Docker images are signed with cosign . Every release is signed with the same key introduced in commit 0112e53 . Verify using the pinned commit hash (recommended): A commit hash is cryptographically immutable, so this is the strongest way
Article · llama.cpp
ggml-cuda: assign four GDN state columns per warp ( #30087 ) ggml-cuda: assign two GDN state columns per warp ggml-cuda: use 4 GDN state columns per warp at S_v=128 ggml-cuda: default cols_per_warp=4 ggml-cuda: address GDN review nits Website:
Another day, another Qwen 3.x speedup (prompt processing this time). Soon your Qwen will read your entire project before you can blink! test master t/s PR t/s change pp512 3075.24 3243.60 +5.5% pp4096 3059.87 3221.08 +5.3% tg128 47.62 47.67 +0.1%
Im thinking of using Gemma 4 e2b q4, running on 32k context with 16gb vram and 900GB/s bandwidth. I would be running it using a call to ollama(idrc about optimize, ill have hundreds of tok/s no matter what) and have a robot car be run by it using wifi and algorithms to move it su
Article · llama.cpp
sycl: add grouped MoE XMX GEMM ( #29245 ) Co-authored-by: cwriter cwriter@localhost Website:
Salut à tous. J'utilise comfyui en local et après avoir utilisé Wan je suis passé à LTX 2.5 pour pouvoir générer des vidéos nsfw avec son.Malheureusement je me rends compte que niveau anatomie ltx est vraiment censuré et les résultats sont diformes.Avez vous rencontré ce genre de
If i ask gemini or grok it always answers with really old models. What's the current gold standard for story telling models ? I'd like to keep it on my 8gb card , but 16gb is available for the right jump in quality. Ideally low refusals, but i also don't want one that goes out it
Article · LiteLLM
Verify Docker Image Signature All LiteLLM Docker images are signed with cosign . Every release is signed with the same key introduced in commit 0112e53 . Verify using the pinned commit hash (recommended): A commit hash is cryptographically immutable, so this is the strongest way
Oh man, my 3090 Ti hates to see me coming. How hard are basic Krea 2 & MiniMax H3 workflows on my card? I was in the studio last night doing video & felt the room get substantially warmer. Had to turn the AC down to 68 just to maintain 73. That can’t be good. Best practices to ma
Hi all, having searched google and asked several AIs without success, maybe s.o. can help. I have 2 x dgx spark in a cluster. deepseek-flash is running successful. Now I want to test qwen3.8-flash-next from unsloth in the mtp version. Following start-command runs into a dgx-stall
Article · LiteLLM
Verify Docker Image Signature All LiteLLM Docker images are signed with cosign . Every release is signed with the same key introduced in commit 0112e53 . Verify using the pinned commit hash (recommended): A commit hash is cryptographically immutable, so this is the strongest way
Article · LiteLLM
Verify Docker Image Signature All LiteLLM Docker images are signed with cosign . Every release is signed with the same key introduced in commit 0112e53 . Verify using the pinned commit hash (recommended): A commit hash is cryptographically immutable, so this is the strongest way
Article · n8n
2.43.2 (2026-10-08) Bug Fixes core: Fix Kimi K3 and other OpenAI-compatible Assistant model providers on self-hosted Assistant ( #40552 ) ( 509a3f3 ) editor: Show the Agent artifact while the builder creates it ( #40550 ) ( 204e2dd )
Article · n8n
2.42.5 (2026-10-08) Bug Fixes editor: Show the Agent artifact while the builder creates it ( #40551 ) ( e825e75 )
Article · LiteLLM
Verify Docker Image Signature All LiteLLM Docker images are signed with cosign . Every release is signed with the same key introduced in commit 0112e53 . Verify using the pinned commit hash (recommended): A commit hash is cryptographically immutable, so this is the strongest way
Article · llama.cpp
cuda : support arbitrary striding for unary ops on f16, f32, and bf16…
Hello, I generally run comfyui in the bleeding edge state like daily updates bleeding edge. Ran into a new pull request and I was as expected having issues running workflows with custom nodes, a "for instance" would be Krea2PromptWeight node in Krea2 text to image workflow and an
Article · llama.cpp
sycl: remove duplicate block-size defines from op headers ( #29507 ) All were duplicates of the values in ggml/src/ggml-sycl/presets.hpp Website:
As the tittle said, ive been using chatgpt for a while for my job and i really like the type of image it produces. But sadly there is only so much you can with the free version and im still debating wether to subscribe to pro or not, then i discovered comfyui. Im new to local gen
Running a basic retrieve-then-rerank setup and the annoying part is TEI being one model per container. So the embedder and the reranker are two separate deployments on two GPUs, both barely used, because an encoder does its thing in a few ms and then sits idle. Trying to work out
I want to create this low poly psx/ps1 style, mainly to use img2img to turn characters into this style. I'm able to generate great looking images with img2img, but I cant get this low poly style.
Using llama.cpp 0.6.0 and bartowski's Q4_K_M quant for Cloudflare Clef Running the basic example: curl
Opus5.5, Sol6.1, Fable, and Astra have all proven they can and the scene has exploded this past week. Part of that is from the tools and feedback loops maturing though. Are any open weight models (at all, so including K3, GLM5.3, Qwen3.8-Max, and Mimo-2.6) able to do this? Can th
I ran Haiku 5.5 to see if it could replace DeepSeek V4.1 Flash in two small jobs in my app. It didn't. Small eval, but maybe useful for someone. GPU poor here (M2 Max, 32GB), so DeepSeek runs through OpenRouter with a minimum of 8-bit precision, sorted by throughput. Most runs la
Article · MarkTechPost
Perplexity's pplx-embed-v2-late comes in 2 sizes: a 0.6B model built to run on edge devices, and a 9B model for building high-quality indexes. Its best score is 92.4% on MADQA, and its weakest is 61.2% on ViDoRe v3 Markdown. Both are MIT-licensed and ready to self-host. The post
Article · llama.cpp
hexagon: enable alloc_buffer_n ( #30126 ) hex-bufs: add support for alloc_buffer_n hex-bufs: add support for splitting large tensors into separate buffers hex-bufs: update GGML_HEXAGON_MBUF to accept three values dyn,static,total hex-bufs: bump dyn. default to 512MB since 128MB c
Article · llama.cpp
hexagon: Q6_K weight dequant speedup ( #30121 ) hexagon: Q6_K weight dequant speedup Assisted-by: OpenCode unroll by another factor of 2 Assisted-by: OpenCode Website:
Greetings everyone, I hope you're all doing very well. I'm leaving a question here, if you'd be so kind as to help me. I want to learn how to use ComfyUI, but I looked for information and got lost. I have a 4070, 32 GB RAM, i5-13400F. What models can I use, or what do you recomme
Article · llama.cpp
hexagon: support tiled Q4_K and Q6_K GET_ROWS ( #30115 ) hexagon: support tiled Q4_K GET_ROWS Assisted-by: OpenCode properly reject Q4_K views Assisted-by: OpenCode hexagon: support tiled Q6_K GET_ROWS Assisted-by: OpenCode Website:
Article · llama.cpp
hexagon: improved GELU accuracy ( #30104 ) Assisted-by: OpenCode Website:
Article · llama.cpp
chat: fix jinja parser for TranslateGemma ( #30096 ) chat: fix jinja parser for translategemma log if missing soruce_lang_code or target_lang_code Website:
TL;DR: For my use case, mradermacher/Signal-3.8-27B-Terse-Coder.i1-Q4_K_M proved out. Ymmv based on your domain-specific tests. Time taken to solve problems compared to frontier models is massive; especially if you, like me, run a potato. My current feasible model's KPI over the
Hi everyone! I'm very new to AI and have spent the past few days doing some research on running AI locally. I only have my personal PC with 32GB of RAM and an RTX 3060 Ti. Based on what I've learned so far, I'm planning to go with this setup: LM Studio → Qwen 3.5 9B (or 8B) → Her
Article · llama.cpp
chat : name tool and argument parser rules by index ( #30088 ) bugfix: infinite recursion caused by a tool named 'call'( #29967 ) chat : use index for schema and argument rules tests : remove tests tests : add expect_rules to peg test parser Co-authored-by: Alde Rojas hello@alde.
Article · llama.cpp
model : add LiquidAI/d1-3B decision model ( #30110 ) model : add LiquidAI/d1-3b decision model mtmd : read LFM2 image resize algo from GGUF Assisted-by: Claude Opus 5.5 common : rename decision type d1 to lfm2-d1 Assisted-by: Claude Opus 5.5 Website:
Article · llama.cpp
cuda: FWHT kernels for block widths above 512 ( #29100 ) The CUDA FWHT covers widths 64 to 512. It runs one row per warp and keeps N/32 values per lane, so wider blocks need more registers per lane than that layout allows. fwht_cuda_block runs one row per thread block with 256 th
Sdxl and Klein, ltx and minimax and some seedance. Comfyui with an agent driver
Article · llama.cpp
mtmd: add cohere2 vision support ( #30062 ) cohere2 vision model address comments remove redundant mapping follow existing patterns fused linear_1 Website:
Article · Ollama
What's Changed server: proxy cloud usage and balance APIs by @drifkin in #18829 llama: fix clef head reads past 2GiB on windows by @Gigrise in #18777 cmd: remove account step from CLI onboarding by @hoyyeva in #18826 manifest: avoid symlinks on Windows by @dhiltgen in #18852 docs
Preamble and disclaimer, Sample size: 8, at any size of form this research is just screwing around being writing it down. This was just a little,(well... big), curiosity test I wanted to run to see whether everyday folks could actually differentiate between high end AI models. In
Made this (free, no signup). One shot per model, no system prompt, temp 0.7. Lots of open weights in there, plus 93 models you can't run anymore. museumofmodels.com
Article · llama.cpp
llama : add a GPU cache for MoE experts kept in host memory ( #29887 ) llama : add a GPU cache for MoE experts kept in host memory Assisted-by: Claude use llama_moe_cache_ptr Website:
Hi everyone! I am building my Home Lab using Docker & Portainer, and I'm trying to set up a cloud-only music streaming setup with TorBox, rclone, Lidarr, and Navidrome. My Goal: Stream music (preferably in FLAC) to my mobile phone using Navidrome, without keeping files on my loca
Article · ServeTheHome
Microsoft today launched its first two NVIDIA RTX Spark-powered systems, the Surface Laptop Ultra and Surface RTX Spark Dev box. Aimed at developers and other professionals, the high-end systems offer a fresh face to NVIDIA and Microsoft's Arm ecosystem efforts The post Microsoft
Human written: Been toying with local models for a while, to various degrees of success. The primary incentive to look into the Qwen 3.8 family was because I stopped the Claude Max subscription, and I'm running out of tokens on a somewhat regular basis. Had the $100/mo Max for se
Article · llama.cpp
model : add LiquidAI/d1-omni-600M decision model Assisted-by: Claude Opus 5.5 mtmd : keep conformer GLU sigmoid on CUDA Assisted-by: Claude Opus 5.5 server : take d1omni audio through images and input_audio, scope memory-less lfm2 to non-causal Assisted-by: Claude Opus 5.5 common
Seems like a decent option to be able to generate images using the more larger models if you have a Gen 4/5 NVMe drive.
I've been tinkering with GLM-5.3-Flash on two 64GB CMP 170HX cards for a while, and the setup is finally stable enough that I figured I'd share it. I also compared it against the Qwen3.8-Flash-Next setup I've been using on the same machine: AWQ INT4 + FP8 PLE on vLLM. Besides PP/
This will be a potentially unpopular post - but it's the truth and grounded - so let's get to it. Hopefully you are familiar with my previous Tested in Coding series:
Article · Ollama
mlx: drop carried metal residency patch now that it is upstream ( #18854 )
$5995 - Ships in November. N1X brand of GB10 Chip. Says it will have WSL out the door which I presume will be Ubuntu + Cuda beneath in addition to all the CoPilot/GitHub native stuff for Windows. Hopefully they have supply to saturate the market and put in pricing pressure. Knowi
Article · LiteLLM
Verify Docker Image Signature All LiteLLM Docker images are signed with cosign . Every release is signed with the same key introduced in commit 0112e53 . Verify using the pinned commit hash (recommended): A commit hash is cryptographically immutable, so this is the strongest way
If you recently updated ComfyUI and your workflows suddenly started crashing on MiniMax, LTX, or newer models with either of these errors: TypeError: ... Linear.forward() got an unexpected keyword argument 'input_act' or RuntimeError: mat1 and mat2 shapes cannot be multiplied (26
strata with qwen flash Q2 is pretty retarded and 27b at q4 is also retarded so I am wondering if anyone knows a engine that runs the model Qwen/Qwen3.6-35B-A3B fast on my hardware?
I generally just ask it "describe the main cast of (insert somewhat niche but still known cartoon show from the 2010s)", could either be Totally Spies, Randy Cunningham, Slugterra etc, most models in the 30b range completely fumble, looking at you Qwen, but the ones that manage t
Article · Simon Willison
As previously promised , here's Anthropic's new fast, low cost model: Introducing Claude Haiku 5.5 . The previous Haiku, 4.5, was very much showing its age. It came out almost a year ago , and was priced at $1/million input and $5/million output - relatively expensive even back t
Introducing Open SLM Evaluations! Open SLM Evaluations is a dataset containing benchmarks for 106 SLMs(150M parameters and below). It has benchmark scores for each model, links, parameter counts, and names. Results were evaluated independently over 30 GPU minutes on a Runpod B200
5080 16GB, 64GB RAM. Currently on 3.6 35B-A3B (Unsloth quant, fits fully in VRAM) and happy with it. Anyone tried Flash-Next at Q2/Q3 on a similar setup? Worth the switch, or should I stay put? Asking before committing to a 30GB+ download.
Article · MarkTechPost
Anthropic's Claude Haiku 5.5 starts at $0.10 per million input tokens, keeps 1M context, scores 72.4% on OSWorld. The post Anthropic Releases Claude Haiku 5.5: A Small Model With 1M Context Priced at $0.10 per Million Input Tokens appeared first on MarkTechPost .
Reddit from r/LocalLLaMA.
Article · LiteLLM
Verify Docker Image Signature All LiteLLM Docker images are signed with cosign . Every release is signed with the same key introduced in commit 0112e53 . Verify using the pinned commit hash (recommended): A commit hash is cryptographically immutable, so this is the strongest way
Kandinsky 6.0 Pro (29B) and Lite (3B). Already support for ComfyUI and Diffusers.
Article · LiteLLM
Verify Docker Image Signature All LiteLLM Docker images are signed with cosign . Every release is signed with the same key introduced in commit 0112e53 . Verify using the pinned commit hash (recommended): A commit hash is cryptographically immutable, so this is the strongest way
Reddit from r/LocalLLM.
I moved about 16 projects off Vercel onto my own Ubuntu servers and ended up writing the deploy tooling I wanted along the way. That turned into llama-apps. What it does: connect a Git repo, push to main, and it builds your app and runs it in a Docker container on your server. Tw
Article · LiteLLM
Verify Docker Image Signature All LiteLLM Docker images are signed with cosign . Every release is signed with the same key introduced in commit 0112e53 . Verify using the pinned commit hash (recommended): A commit hash is cryptographically immutable, so this is the strongest way
Article · LiteLLM
Verify Docker Image Signature All LiteLLM Docker images are signed with cosign . Every release is signed with the same key introduced in commit 0112e53 . Verify using the pinned commit hash (recommended): A commit hash is cryptographically immutable, so this is the strongest way
Article · LiteLLM
Verify Docker Image Signature All LiteLLM Docker images are signed with cosign . Every release is signed with the same key introduced in commit 0112e53 . Verify using the pinned commit hash (recommended): A commit hash is cryptographically immutable, so this is the strongest way
Potentially big speedup for MoE models that don’t fully fit in VRAM. Are you GPU Poor? Show your speedups ;) update
Liquid just dropped d1-omni-600M and it's a decision model! I ported it to runntime, the WebGPU inference library I'm currently working on. It's plain TypeScript on top of TypeGPU with no WASM and virtually no export step. The model is written directly from our core ops (matmul,
I built a tool that combines Claude Remote with project management through a dashboard. I originally built this tool for myself. I’m not sure yet whether it could be useful to other people. I’m planning to make the repository open source, with a self-hosted version as well as a c
Article · llama.cpp
cuda: update uncoalesced memory reads in pool2d cuda: Added perf and restrict in pool2d Update POOL2D_WARP_KERNEL_MIN_WINDOW macro Co-authored-by: Georgi Gerganov ggerganov@gmail.com cuda: Fix compiler bugs Co-authored-by: Georgi Gerganov ggerganov@gmail.com
I wanted to share Eron, a native iOS companion app built specifically for self-hosted AI stacks (Ollama, vLLM, LocalAI, etc.). Why I built it: I self-host LLMs at home and connect to them remotely via Tailscale. Web frontends like Open WebUI are fantastic on desktop, but on mobil
I’m using a vision-capable image generation model with reference images, and I’m trying to get it to accurately follow detailed references for clothing, pose, anatomy, lighting, and environment. For adult/NSFW image generation, what prompt structure or wording works best when you
Another quick Minimax H3 news and goodies round-up, for those who may have missed some items. -> A new LoRA aims to emulate the look of 1970s fantasy films, rather than 1980s. Aims for... "the visual language of older fantasy productions: practical costumes, theatrical sets, matt
LIBERO is a robot manipulation benchmark: 130 tasks across five suites (spatial, object, goal, scene10, scene90), each with human demos and a language goal. It is the standard testbed for language-conditioned imitation, and it normally runs on robosuite with CPU MuJoCo, one envir
Reddit from r/LocalLLaMA.
Over a year ago I posted EreNodes here when it was a handful of nodes that let you click tags on and off instead of editing a wall of text. It didn't stop there. I kept fixing whatever annoyed me during my own generations, and a year (and 44 versions) later it's grown a lot. The
LINE ZERO is a short title-sequence experiment built around one visual rule: a single yellow line changes function as it moves through the world - map, infrastructure, transport, signal and trajectory. Rather than treating each generated shot independently, I tried to use that li
Article · llama.cpp
metal : few-row MMA mat-mul for the remaining src0 types ( #30065 ) The generic few-row MMA kernel works for any type with a 16-weight dequantizer, so it now also takes BF16, Q1_0, Q2_0, MXFP4, Q2_K, Q3_K, TQ2_0 and the IQ types. Each type starts at the row count where it beats t
Hello everyone, I new ComfyUI workflow for video character swapping using MiniMax H3 + Character Swap LoRA , and the main advantage is that it works without masking , which can significantly reduce VRAM usage and simplify the process. The workflow generates at 0.4 MP in just 8 st
Article · llama.cpp
llama: share the nextn tensor flags between models Follow-up of the TODO in glm5-next: move the trunk-only and MTP-only detection that each model copied into a nextn_flags helper of llama_model_base. It probes the first trunk layer and the first NextN layer, and adds TENSOR_SKIP
Article · llama.cpp
metal : fix MUL_MAT+ADD fusion when the residual is itself a MUL_MAT ( #30100 ) metal : fix MUL_MAT+ADD fusion when the residual is itself a MUL_MAT ggml_metal_op_mul_mat_mma picks the residual of a fused MUL_MAT+ADD as "the ADD operand whose op is not MUL_MAT". When both operand
I thought this was a very interesting article, of relevance to the readers here.
Sup guys, soo while searching for some really good motion swap model someone recommended me H3 with this workflow. The output was pretty good but the generation was way too long and i dont really think a 12sec video in 1080p 24fps at 12steps should take around 35 minutes on pro60
This is the Workflow I use daily and for most purposes. It handles most tasks that occur on my daily basis. Maybe it can be helpful to some of you. Its key feature are image generation, manipulation via references, in- and outpainting in a compact workflow with helpful options.
Article · llama.cpp
feat: add GLM5Next MTP, optimize ( #29928 ) llama : add GLM5-Next NextN (MTP) graph Build the GLM5-Next multi-token-prediction head as graph_mtp: the NextN block embeds enorm(tok)+hnorm(h) through eh_proj, runs one plain DSA layer and the shared lm_head, reusing the trunk's build
Most workflows end up static: build once, run as-is. Iterating while you run is awkward in both views: the graph makes you travel to the node every time, App Mode hides what you're changing. So I tried a middle ground: deliberately drop the X axis and lay the whole graph out as a
Article · llama.cpp
sampling : use greedy selection for eligible temperature-zero chains ( #29797 ) sampling : use greedy selection for eligible temperature-zero chains Assisted-by: OpenAI Codex Apply suggestion from @ggerganov Co-authored-by: Georgi Gerganov ggerganov@gmail.com sampling: simplify z
Upgrade Question If we say the "budget max" is $1700-1800, and this could include upgrading to a Taichi motherboard: With a new motherboard , no need to bifurcate 1. Would you add a second 5060Ti (16GB)? Someone is selling one for around $500 locally; 2. Buy a used 7900 XTX (24GB
As subject. If I try to save (or save as) on a workflow... I get the following: Error storing user data file 'workflows/MMO Screens + Video - Klein True /MMO Screens & Video (Klein True /MMO Screens Video.json': 500 Internal Server Error I've no idea why the workflow name has now
Running a model on your own machine used to be a privacy story with a quality tax. On this generation that trade has narrowed to where we can state it plainly: for daily work, the local model is good enough. We measured it - 25 paired tasks across four workloads, same prompts, on
I use ComfyUI through Stability Matrix a lot, and I kept running into the same problem: I could find an old image, but I couldn’t easily remember when I generated it, how long it took, what model/LoRA I used, or what my GPU was doing at the time. So I built GenTrace for my own wo
Article · llama.cpp
musa: use the tile lightning indexer kernel ( #30080 ) Signed-off-by: Xiaodong Ye xiaodong.ye@mthreads.com Website:
now you can use GLM 5 Flash MTP locally
Reddit from r/LocalLLaMA.
Article · llama.cpp
imatrix : include clocale for std::setlocale ( #30079 ) Co-authored-by: yarik dazzywi@github.com Website:
The Yue2 LoRA manages to learn the singer's vocal style and musical genre. However, the music ends up being off-key. The model loses the ability to create melodies and harmonies that blend pleasantly with the lyrics. Out of four attempts, I had relative success only once. Even th
Article · selfh.st
Annual survey covering various self-hosting topics
Article · llama.cpp
ggml-webgpu: fix flash_attn supports_op check for overlapping KV ( #28205 ) Website:
is there a way to change the windows roaming folder comfy destop folder to another folder on my other ssd
"NVHBM moves the memory controller, which was previously located on the main compute die, into the base die. This reduces power consumption by 15% and increases memory bandwidth by as much as 30%. It also integrates a customized physical layer (PHY) for input/output (I/O), reduci
Running LTX-2.5 for T2V/I2V. Prompt-described motion works but is inconsistent run to run - I can say "slow push-in" and get something else entirely. Looking for genuine conditioning-based motion control - trajectory input, camera-path control, or driving-video style guidance. So
Article · llama.cpp
tests : retain the anchor when testing recurrent rollback ( #29923 ) Website:
ComfyUI is working normally now - workflows load, the queue works, and images generate successfully. However, the Play/Pause buttons for the queue are completely missing from the UI. I expected to see the queue controls with Play/Pause, but they aren't there at all. I previously
Article · llama.cpp
[SYCL] fix the issue in mixed different model GPUs in FA ( #29071 ) fix mixed different model GPUs issue Update ggml/src/ggml-sycl/ggml-sycl.cpp Co-authored-by: Georgi Gerganov ggerganov@gmail.com Co-authored-by: Georgi Gerganov ggerganov@gmail.com Website:
If you're like me and saw the price increase for the 128gb DGX Spark go from 4.7k -> 7k while a new version with 64GB launch for 5k, you'll have been very disappointed and every right to be so, it's just plain sad for localAI. Well, here's some good news, cmpunlocker v0.5 just dr
Article · llama.cpp
sycl: accelerate GLM MLA prefill with MKL flash attention ( #29171 ) sycl: accelerate GLM MLA prefill with MKL flash attention GLM-4.7 Flash uses an MLA shape with 576-wide Q/K heads, a 512-wide V head, GQA 20, and F16 KV. The SYCL dispatcher rejects this shape because the normal
I do projection mapping, and I got tired of bouncing between Comfy, After Effects and Houdini every time a facade needed a new look. So I did the reasonable thing and spent way too many evenings building it all inside ComfyUI instead. It started as a facade thing: split a buildin
EmbeddingGemma 2 came out this week. It maps images and text into one 768-dim space, so you can search photos by describing them. I ported its text and vision towers to ruNNtime, a WebGPU inference library in TypeScript, and made a small photo gallery where search runs entirely o
Yesterday TechCrunch published a piece about a growing problem for AI agents: websites are starting to block them. Amazon blocking Meta's Muse is the obvious example. At almost exactly the same time, GLM-Edge-1.5B-Chat running locally on a 4GB Galaxy A04e completed a real Amazon
Consider this: a widely available machine, up to 1.5TB of system RAM, room for 4 passively cooled GPUs with 128GB of VRAM, in desktop or rack format. That machine was released in 2019, then discontinued in favor of one that had only a max of 192GB shared memory. This would be the
Article · LiteLLM
Verify Docker Image Signature All LiteLLM Docker images are signed with cosign . Every release is signed with the same key introduced in commit 0112e53 . Verify using the pinned commit hash (recommended): A commit hash is cryptographically immutable, so this is the strongest way
PrismML's Ternary Bonsai 2 27B fits a 24 GB card or Mac, but decodes at ~30 tok/s on an L4 and ~21 on an M4 Pro. z-lab's DFlash 2 drafter was trained on bf16 Qwen3.8-27B, so it guesses worse on the ternary model. I fine-tuned it on 1.5M tokens of Bonsai 2's own greedy output. Dra
Article · n8n
2.43.1 (2026-10-07) Bug Fixes editor: Polish n8n Assistant and Execute button UI details ( #40438 ) ( 345ae01 ) Performance Improvements core: Keep agent builder system prompt static for prompt caching ( #40441 ) ( 9ea7835 ) core: Prepare Instance AI skill sandbox in the backgrou
My current setup: - single 3090 running turboderp/Qwen3.8-27B-exl3:SC_5.00bpw_H6_V6 - ~150k context, ~70t/s, unknown prefill because I didn't benchmark it (but it is ok) - Intel 12400 CPU with 32GB DDR4 RAM All the hype around strata makes me consider buying 128GB of 6000MHz DDR5
MiniCPM-V-4.7-35B-A3B No model card yet
I’ve been pushing TinyStories-style models downward in size, and this is the smallest one so far: MacroStories - 19,969 parameters, 81 KB FP32
I’ve been working on GesturePortal , a gesture-controlled live AI camera. Hold up two L-shaped hands and a window opens inside your webcam feed, revealing an anime version of the scene. Move your hands to move the window, then release the gesture to close it. The biggest improvem
Building a fully local RAG setup for personal notes (life-logging second brain, single user, privacy is the whole point so no hosted APIs for the data). Stack is SQLite + sqlite-vec + FTS5, Ollama for embeddings and generation, all on an Apple Silicon Mac. Two questions where I'd
I constantly see recommendations saying that ik_llama.cpp (ikawrakow's fork) is the undisputed king of hybrid CPU/GPU offloading and MoE performance. However, every time I benchmark it against mainline ggml-org, mainline consistently beats it by a wide margin. Am I missing specif
I have a somewhat specific question in case anyone has experience with this. I am a big fan of CS, math, and physics. While I was able to get formal instruction for the first two, I was never able to get the opportunity to learn physics. The most wonderful part about llms for me
I have been running local LLMs in my homelab for a while now and I am hitting a wall with how fragmented everything is. Right now I have LM Studio on a Ryzen 9 9950X with an RX 7900 XTX, another LM Studio on an i5-10400 with an RTX 3050, and Ollama on my MacBook Pro M3 - and hone
I don't know if anyone else has ran into these issues but when using Qwen Flash Next, my confidence in it at iq4_xs is high but not 100%. I notice it not following instructions, hallucinating more often and surprisingly it uses way less tokens than 27b. After using Strata.... yes
We spent the last weeks building a studio around YuE2 , the open song model by m-a-p, and today it reaches its first release candidate. Write the style and the lyrics, and it composes, sings and renders the song on your own card. Nothing leaves your machine unless you point the W
I'm seeing these show up on eBay for $50-$60. wouldn't expect anything ground breaking from it, but at that price, it seems like it could be interesting to play with on an extra pcie port. just curious if anyone's already done that
Introducing InferBench: A benchmark testing how well frontier LLMs infer a user's priorities from their instructions. We tested 12 LLMs across 20 scenarios with 2.8k conversations to see which models understand the user's goals. Each conversation has a simulated user with a priva
I log every new outbound connection on my LAN at the router. A few days ago my inference box started making about 18 connections an hour to an AWS IP I didn't recognise, 34.236.19.149:8080 . It turned out to be stats.lmcache.ai . Source: LMCache's built-in usage telemetry, which
Ideally I'd like to be able to run Qwen 3.8-Flash-Next with decent performance. I was thinking of just buying 2x Radeon AI R9700 (64GB VRAM), or maybe a DGX Spark but that was before the price hike. My brother suggested just getting a Strix Halo box with 128GB unified. I did see
I built a Pi extension that implements Victor Taelin's OptChat recipe: instead of compacting, every message is logged and summarized into a binary tree. Each turn starts from a fresh context with a bounded memory view (128 KB), and the agent uses zoom/date to read the originals w
Article · Simon Willison
Release: llm-openai-decisions 0.1a0 OpenAI released their new Jev-style Decisions API , as previously announced at last week's DevDay. Since I already have an llm-typesafe plugin for talking to Jev, I had GPT-6 Astra read the new OpenAI API documentation and build an llm-openai-d
1 trillion parameters, 49B active, definitely chonky! If you don’t love the model you gotta at least love the humor in the name - Le Chonk
Article · llama.cpp
hexagon: CPY/CONCAT/CONT/DUP overhaul to use DMA/HVX for all cases (#…
Article · llama.cpp
cuda : add BF16 support for XIELU ( #29955 ) The XIELU CUDA kernel template is already generic over the element type; only the F32/F16 type assertion and the else-if dispatch were missing. Add the nv_bfloat16 branch to the launcher, and drop the temporary supports_op gate in ggml
Article · llama.cpp
ggml-cuda: use per-thread stream for buffer-init padding memset ( #28782 ) ggml-cuda: use per-thread stream for buffer-init padding memset ci : re-enable test-backend-ops -j for ROCm Website:
Be safe out there boys and girls.
Article · llama.cpp
vocab : implement PLaMo-3 tokenizer pre-segmentation ( #30045 ) vocab : implement PLaMo-3 tokenizer pre-segmentation The PLaMo-3 tokenizer inserts hard boundaries before running the Unigram DP, around -looking text, and around runs of at least 4 identical characters or 2 spaces.
Last time when I posted on this subReddit to share my work, the response almost made me cry because a tiny demo got so many people talking about this In the past 2 months Ive been training a new model, but this time with actual text guidance. So a little about the older model. No
Article · llama.cpp
model : add K2 Horizon dense and MoVA support ( #29535 ) model: K2 Horizon gguf conversion code model: loading hparams and tensors in k2-horizon.cpp model: K2 Horizon compute graph model: K2 Horizon compute graph adjustment and registering tokenizers model: K2 Horizon chat templa
So, i'm using 0.8mp, 8 step, with the minimax_h3_turbo_v4_step600_ema.safetensors lora. I'm using the ref to video model, with an image and audio ref. Any advice on how to make things look closer to the original appearance? Its like the minimax setup is causing lines to be thicke
Hi everyone. I've created an extension for the Lora package. I did it primarily for myself, but I thought I'd share it because I think it turned out pretty well. I didn't write the code myself, but used Qwen 3.8. Here's a link to a detailed description on Github:
Hello llamas. I am posting this because I believe that, despite it being closed source models, the discussion will bring value to the local AI community. As many of you probably heard, GPT-6 Astra is speculated to be a looped transformer architecture that outputs a token after mu
Article · llama.cpp
RPC: add -sm tensor ( #26610 ) rpc: allow -sm tensor fix flush for apple rdma move graph_uids to rpc_dispatcher cont : fix conflict cont: stop spinning dispatcher thread remove meta backend change add TODO to simplify logic rpc: bump major version Co-authored-by: Georgi Gerganov
Article · Immich
v3.2.0 Welcome to Immich v3.2.0 ! This release includes many new features as well as the usual collection of bug fixes and cleanups. Keep reading below for a list of highlights. Highlights Docker compose builder Revamped search UI (web) Search API v2 (server) View own people in s
![v0.31.1rc0: [Metrics] Expose cached prompt tokens by cache tier (#56318)](media/d0c284cb68a2d3c1.png)
Article · vLLM
Signed-off-by: Cam Quilici cjquilici@gmail.com Signed-off-by: Cam Quilici cameron@semianalysis.com Co-authored-by: Cam Quilici cameron@semianalysis.com Co-authored-by: Nick Hill nickhill123@gmail.com Co-authored-by: Yifan Qiao yifanqiao@inferact.ai
Article · llama.cpp
ggml: fix CLAMP on non-contiguous views (CPU, CUDA) ( #29517 ) ggml: fix CLAMP on non-contiguous views (CPU, CUDA) CUDA clamped ggml_nelements values flat and ignored the view strides. CPU addressed row j as j*nb01 and ignored nb02/nb03. Both now follow the strides of dims 1..3;
Article · Ollama
What's Changed Models run on MLX on Apple Silicon by default In this release, on Apple Silicon devices, model architectures supported by the MLX runtime will automatically run on MLX. ollama pull qwen3.8 ollama run qwen3.8 Additional models include gemma4 , qwen3.6 and qwen3.5 De
Article · MarkTechPost
Google DeepMind's EmbeddingGemma 2 maps 5 input types into one 768d space and ships today under Apache 2.0. The post Google DeepMind Releases EmbeddingGemma 2, a 740M Open Multimodal Embedding Model Built on Gemma 4 appeared first on MarkTechPost .
Article · llama.cpp
cuda: BF16/FP16 conversion to f32 chunking ( #29442 ) ggml-cuda: chunk large BF16/FP16 to F32 conversions Update ggml/src/ggml-cuda/ggml-cuda.cu Co-authored-by: Johannes Gäßler johannesg@5d6.de Update ggml/src/ggml-cuda/ggml-cuda.cu Co-authored-by: Johannes Gäßler johannesg@5d6.d
We’re building and actively using a working AI filmmaking pipeline , focused primarily on local models and ComfyUI . This is not just a concept or a collection of isolated workflows. We already have a production pipeline that takes a project through multiple stages: script → scen
Hey guys, it's Merve from Hugging Face! I've recently given a talk in a dev conference about llama.cpp + but also covering basic concepts like prefill vs decode, memory types, speculative decoding etc. you can use it if you feel like it and I appreciate if you can give attributio
Another quick Minimax H3 news and goodies round-up, for those who may have missed some items. -> ComfyUI Portable 0.39.0 has been released. The Minimax H3 relevant changes are... ~ "Use higher quality defaults for Save Video encoding" ~ "Fix Minimax VAE offload issue" ~ "Release
We released Aplomb 1 today, a 5.3B decision model with open weights. It reads up to 1M tokens of text, JSON, images, video and audio in a single request, and on our API it makes a decision on a 1M-token document in about 3 seconds, at $0.02 per 1M input tokens with free output an
Meet Bev. She is a decision model (the Jev / Nimble kind: you give her a situation and a question, she gives a probability for each answer), fine-tuned on Qwen3.5-9B to pick the worst answer on purpose. Try her in your browser:
Article · llama.cpp
metal : fix excess threadgroup memory in quantized flash attention ( #29340 ) Website:
Article · MarkTechPost
Mistral AI has released Mistral Large 4, nicknamed Le Chonk, as a public preview. It is a 1.05 trillion parameter Mixture of Experts model with 49 billion active parameters, native image input, and a 1 million token context window, trained on 3,800 NVIDIA Grace Blackwell GPUs in
I spent the last few weeks on a hobby research project and just made it public. The idea isn't new (product-key memory, Lample et al. 2019, and Meta's "Memory Layers at Scale"): give a model a huge table of learned vectors and let it read only a few hundred of them per token. I w
guys do you still remember K2 Horizon?
Reddit from r/LocalLLaMA.
Article · llama.cpp
The gather path attended over the selected latents with a plain matmul and softmax. It only ran with n_ubatch attention backends now skip the masked rows through n_kv_max, so the scatter path covers every case. Drop the gather flag, the gathered attention branch and gather_mla_ro
My small IT business in Spain gets a lot of sales calls and I miss real customers while I'm on site. So I built a receptionist that answers my existing landline, entirely self-hosted: - Landline → Grandstream HT813 (FXO) → FreePBX 17 in a VM - Asterisk AudioSocket streams the cal
Reddit from r/LocalLLaMA.
For a model running locally on a fairly narrow task, how much general knowledge do we actually need, and how much reasoning capability could we get without it ? SRMs are interesting for obvious reasons, but I went down this rabbit hole after listening to Ben Lorica's (advisor at
Article · llama.cpp
vulkan : check for null vkEnumerateInstanceVersion ( #29872 ) A 1.0 loader (e.g. Android 8.1) has no vkEnumerateInstanceVersion, so backend init called a null pointer. Treat it like any loader under 1.2. Fixes #29871 . Assisted-by: Claude Opus 5.5 Website:
Bought an old mining farm of a guy on avito (Russian eBay), guy had bought a garage a couple of years ago and it was sitting there for a while, found out it was a mining farm and put it up on there for sale for 5000 rub (~60 USD) since he wasn't sure if it works. I negotiated dow
EmbeddingGemma 2 is an open multimodal embedding model built by Google DeepMind which maps text (incl. code), images, video, and audio inputs-and combinations thereof-into a single, unified 768-dimensional vector space. The model has 740M total parameters, combining a 270M parame
Lookup-based memory has been a promising way to scale the parameters of large language models (LLMs). It retrieves learned representations of local token patterns, such as n-grams, instead of reconstructing them through successive layers of computation. However, existing designs
Article · Ollama
Implements the EmbeddingGemma2Model architecture on the MLX runner: 24-layer bidirectional text encoder with PLE, shared gemma4 vision/audio towers, mean-pool + L2 output. /api/embed accepts per-item media via input dicts.
Hey! 👋 I have released an official support for Strix Halo machines on Strata for Qwen3.8-Flash-Next. Currently numbers are the best on long context decode and ppts using typical Unsloth’s Q4 and GSQ-RCO model weights. Can go up to 1M context length without big speed loss. Current
Hey guys, I have been working on the consistent character portable format for a while & I was able to achieve consistent face, cloths & body, but I felt voice is also something should be consistent across video generation. So in the recent tests, I was able to achieve a consisten
Type of BS post #1: running qwen 3.8 27b on a potato and getting 135 t/s. Turns out it's a Q1s quant, can't even code a hello world, context size limited to 10K. Type of BS post #2: managed to run 10 agents with 80 t/s each running a whole company, saving 20K usd per month. Here'
I updated my MiniMax H3 workflow for stitched long videos in ComfyUI. In my tests the join is now hard to spot even when I point it out, and the color stays consistent across segments. Three changes did most of the work: a small noise change on the context reference, a color tran
LeChonk-1T-A49B Weights to be released by the end of the month. This probably means the fat kitten is dead. Also: gotta love the release movie.
Article · llama.cpp
models : consolidate nextn row cropping into shared helpers ( #30017 ) mimo2 : always emit h_nextn the other nextn-capable models set it unconditionally Assisted-by: pi:llama.cpp/MiMo-V2.6-Flash-MOPD models : consolidate nextn row cropping into shared helpers replace the duplicat
Looped language models repeatedly apply a weight-shared block to increase effective depth without increasing parameter count, but the resulting T sequential recurrent-block calls per generated token substantially increase decoding latency. To address the issue, we introduce Wavef
Just dropped.
Article · llama.cpp
llama : re-reserve the sched when the nextn extraction flags change ( #30020 ) The speculative MTP init enables NextN extraction on the target and draft contexts after both were created and their schedulers reserved. With unmasked extraction the trunk graph keeps every token thro
Most "make LLMs faster" tricks quantize the weights: round bf16 down to int4/int8, lose some precision, read fewer bytes, go faster. That's not what this is. I built a GPU inference engine where the weights stay bit-exact bf16 - identical to the original safetensors - but are sto
Hey All, I spoke to a 0-day partner of Alibaba today and he casually mentioned (didnt know if he was allowed to) that Qwen 4 is apparently planned for the end of October. To me, this is way faster than expected as there was quite a gap between 3.6 and 3.8. I tried to get more inf
Article · llama.cpp
ggml: refactor selective expert copying to user code ( #29943 ) ggml: refactor selective expert copying to user code tests: enroll two models into selective expert copy test tests: use deepseek2 as test model improve comment in ggml-backend.h Co-authored-by: Georgi Gerganov ggerg
Microsoft confirms on publicly accessible web page that OpenAI has been using Looped Transformers in the GPT-6 series, proving The Information's reporting was correct all along. GPT-6.1 Sol uses 2 inference passes, with a passing mention of "instead of three". For those confused
For the last few weeks most of my coding has been done locally with Qwen3.8-Flash-Next, so I gave it and Opus 5.5 the same high complexity feature to build on LlamaStash (a complex and large Rust project) and compared the results. Setup : ASUS ROG Flow Z13 (Strix Halo, 128GB), Fl
For the last few weeks most of my coding has been done locally with Qwen3.8-Flash-Next, so I gave it and Opus 5.5 the same high complexity feature to build on LlamaStash (a complex and large Rust project) and compared the results. Setup : ASUS ROG Flow Z13 (Strix Halo, 128GB), Fl
Octop is an open-source, self-hosted AI assistant. Through its multi-agent architecture, it builds an intelligent environment that is both independent and collaborative for teams, families, and individuals. Best of all, it runs entirely on your machine, the fully self-hosted desi
Kolibri , the German model from Aleph Alpha , hosted by Tesseracted , writing a small command line tool in Nim , on 3code . It's a small model so it's remarkable it got code to work in a very niche language. The model, the provider, the harness and the language are all made in Ge
I've tinkered some more with my fork of NInfer for the Qwen 3.8 Flash Next model on RTX6000, and I've just hit 400tg/s under ideal conditions with it, so I thought it was worth a share. Decode MTP3 with --lm-head-draft Context 16-bit 8-bit Change 512 197.1 tok/s 274.8 tok/s +39%
Article · noted.lol
Noted 3.0 brings a cleaner mobile experience, better search, live polls, a new Themes section, improved stats, and a more organized way to explore the site.
Article · llama.cpp
test-llama-archs : initialize backends before generating models ( #30034 ) With GGML_BACKEND_DL=ON, backends must be loaded explicitly before creating models. Assisted-by: Codex Website:
Hey y'all! I have a pet project that has grown out of proportions. Long story short: I'm a data scientist who writes fantasy books and self-publish them. I think it's a genuine waste of human life to check for spelling errors so I figured AI could help. Turns out, it is not so si
By classic benchmark, the flash next is supposed to be slightly superior to its dense counterpart. But they are really incomparable. For my very simple workflows (spec -> implement -> review rework), I feel that 27b is just better quality. For context, and making things worse, I
Article · llama.cpp
ggml-openvino: fix CI tests; fix GPU regressions. ( #30037 ) ggml-openvino: skip unselected graph branches and support DUP Upstream #29622 adds a mixed token/embd branch to every input embedding graph through ggml_build_forward_select(). Its nodes are not flagged for compute, but
A few days ago we shared the idea behind DynamicTune: transferring the trajectory flow from a larger teacher model directly into a smaller student via closed-form linear algebra in ~12 minutes on consumer hardware. Zero backpropagation, zero training tokens, zero gradient descent
My YouTube feed was mostly songs, pranks and celebrity clips, so I built a filter that judges each video title with a small decision model running inside Chrome. Nothing is sent anywhere: no server, no API key, no account, and after one download it works offline. What it does - H
Article · noted.lol
A small update to the new Updates feed. Fixed #update posts not appearing correctly when other internal tags were also attached Improved the feed filtering Kept update posts hidden from normal site search and indexing Nothing major, just a cleanup pass to make the new feed behave
Standard distillation usually means burning weeks of compute and billions of tokens hoping the student model eventually mimics the teacher. We wanted to see what happens if you skip backprop entirely and treat transfer as a closed-form trajectory matching problem between layers.
Article · n8n
2.42.3 (2026-10-05) Bug Fixes API: Let global members grant variable scopes to API keys ( #40150 ) ( d09b8a1 ) core: Keep the task runner running on an unhandled promise rejection ( #40091 ) ( 50f2e96 ) Features editor: Show Gateway credits promotions on community nodes ( #40108
Article · n8n
2.43.0 (2026-10-06) Bug Fixes Agent Node: Remove outdated tutorial tip callout ( #35505 ) ( 53b84d2 ) ai-builder: Treat a blank folder path as no folder in the build tool ( #40116 ) ( b7071c9 ) Align From AI button with parameter inputs ( #40152 ) ( 9675fc9 ) API: Emit valid sche
Article · llama.cpp
The secondary button gets its own look back and the chat add button its own light surface. Pressable elements get a pointer cursor again, the font rendering smooths on the app shell, the root layout resolves its props probe through the providers' api url, and the agent skills sta
Article · llama.cpp
The pane offers "Use in this chat" only when a chat is open, so without one the test looked for a button that was never there. Assisted-by: pi (zai-org/GLM-5.3-Flash)
Article · vLLM
v0.31.0 Highlights This release features 717 commits from 307 contributors (96 new)! DeepSeek-V4.1-Flash performance : FlashMLA mega attention with the V4.1 NVFP4 compressed KV cache is now the SM100 default ( #56935 ); DeepGEMM sparse MQA logits for the indexer ( #56254 ) and Me
This is an update. I posted LocalMind here many moons ago from another account, when it was a Gemma chat in a tab. LocalMind is a static web page that runs models on your GPU through WebGPU. It has no server, no account and no install. The new part: two engines that stream mixtur
Reddit from r/LocalLLaMA.
Article · llama.cpp
llama: fix k-pool scatter data race on shared sequences ( #29994 ) llama: re-pool each shared k-pool rep once With shared cells every pool is re-pooled, and since the pooled keys are always scattered, the pools a seq_cp shares between sequences wrote the same rep row from several
With llama.cpp using a customized SYCL backend on Intel B70 (32GB), overclocking my DDR5 memory gave modest gains for MoE models which do not fit in VRAM . Both PP and TG increased after overclocking DDR5. I've never been one to overclock my system, but on the advice of my agent,
Apple folks, it's for you. llama-server with a Qwen3.8-27B DFlash2 Q8_0 drafter, -ngl 99 -fa on -c 8192 -np 1 --jinja , DFlash2 with --spec-type draft-dflash --spec-draft-n-max 7 . 64 generated tokens, median of 5 requests after one warm-up, mean of two server runs. Decode tok/s:
Over the past several months, we conducted an empirical post-mortem investigation into 109 autonomous AI agent security incidents (cataloged with 193 falsification criteria across tool-use and multi-agent systems). One of the most striking patterns in the dataset: In 38% of conta
At my company, we’re using GLM-5.3 Flash internally for software engineering work, and I’ve been genuinely impressed by it. I work in a very large production environment with projects totaling **millions of lines of code**, and we’re not relying on frontier models for this workfl
Looks like Unsloth is updating
Reflection AI just dropped Beam, their first open model. Former DeepMind folks positioning it as the non-Chinese alternative - strong at coding/agents, 3-4x more efficient than other open models. Already training the next one. $25B valuation. Link:
Article · Paperless-ngx
paperless-ngx 3.3.0 Features / Enhancements Enhancement: more control over suggestion requests @shamoon ( #14258 ) Chorehancement: set manifest CORS for credentials @shamoon ( #14307 ) Enhancement: include Django admin with 2FA @shamoon ( #14270 ) Feature: propagate resolved secr
I’ve been building this for a long fucking time, and tonight I finally released Memoria 1.0.0. I built it because I actually wanted to use it. I wanted a real memory layer for local LLM applications that didn’t depend on a specific model, a cloud service, or an API key. Memoria i
NVIDIA board member Mark Stevens has executed a massive series of stock sales. SEC regulatory filings reveal that Stevens sold approximately $946.63 million worth of Nvidia (NVDA) common stock over three distinct waves. He sold ~13.3% of his total NVDA position.
Every time I check the top models on HF, I basically feel like this:
p.s its very much real. But dont take my word for it.
Reddit from r/LocalLLaMA.
Article · llama.cpp
hexagon: ssm-conv double-buffered DMA for prefill and decode restructuring hex-ssm-conv: remove divs from loops and fix trace events hex-dma: improved SSM_CONV dma pipeline and streamlined dma_queue Co-authored-by: Max Krasnyansky maxk@qti.qualcomm.com
Article · Ollama
MLX: version bump add scopes for unit tests to reduce memory usage
I’ve seen posts here raving about how good Qwen 3.8 27b is. The benchmarks look incredible, and all the online discourse seems to deem it the best local model. I have 32GB VRAM and run Unsloth’s Q6 version with OpenCode. For small tasks, it feels fine. I range 30-40 t/s decode an
Got Kolibri-1 to play Breakout completely on its own, no fine-tuning. The more we explore u/Aleph__Alpha ’s Kolibri the more it get's exciting and its potential. Less talk. More Breakout is one such experiment to see how good the model is at structured output given a few constrai
Finally finished my fork:
A while ago, I posted here getting OSS-120B and GLM-4.6 playing full games of Civilization V . Since then, models have moved pretty far, and we wanted a better understanding about models' capabilities playing the game. Introducing the controlled version of CivBench on newer model
Article · llama.cpp
hexagon: add pool op support ( #29995 ) hexagon: add pool_2d support hexagon: add pool_1d support hex-pool: dma changes hex-pool: Optimize HTP pooling boundaries and DMA pipelining hex-pool: code cleanup and correctness fixes hex-pool: re-write the DMA pipeline hex-pool: pool chu
Article · Jellyfin
🚀 Jellyfin Server 12.2 We are pleased to announce the latest stable release of Jellyfin, version 12.2! This minor release brings several bugfixes to improve your Jellyfin experience. As always, please ensure you take a full backup before upgrading! Discuss this release further on
Article · noted.lol
We are testing our new poll feature 😄 Also, I know I forgot Pangolin!
Article · ComfyUI
What's Changed Support LynnReal light Minimax-H3 vae by @kijai in #16657 Update embedded docs to v0.5.13 by @comfyui-wiki in #16618 Use higher quality defaults for Save Video encoding. by @comfyanonymous in #16663 feat: add DynamicGroup widget input by @jaeone94 in #16260 fix(ass
So PewDiePie decides to fine-tune a local AI model called Ajax on his own computer. Pretty normal stuff for local model fans. To make his dataset, he uses OpenAI's API. OpenAI catches him using their outputs to train another model, flags his account for breaking their terms, and
Just for my understanding is Qwen image 2.1 newer than Qwen image 3.0?
Last update for those following:
Blind A/B arena where AI agents build things in Blender and you vote on which result is better:
I'm not sure if it's the model or just the Codex's MCP itself, which was built by another smaller startup called Sky. I want to build an agent system equivalent of an RPA for my company, and we don't want to use Codex's Computer Use MCP because of the enterprise issues. I'm think
My machine isnt that terrible.. but nowhere near what some people run as an aI workstation. I am wondering if i can combine the 2 GPUs. I got a cheap Blackwell 4000 (little bit under MSRP) 24 GB and have an old 3060 12GB .. I was gonna try to get the best model loaded, mainly cod
Anthropic Reports Florida Woman's Claude 'Diary' Threat to Law Enforcement And this time it wasn't the AI model that made the LEO referral. It was the "human review team". The frontier AI companies are watching your input. And people say "Well I'm not interesting or important eno
Anthropic Reports Florida Woman's Claude 'Diary' Threat to Law Enforcement And this time it wasn't the AI model that made the LEO referral. It was the "human review team". The frontier AI companies are watching your input. And people say "Well I'm not interesting or important eno
Article · llama.cpp
hexagon: matmul and flash-atten scalability updates ( #29974 ) hexagon: head-parallel flash_attn partitioning for row-split multicore In row-split mode each core computes its output row shard of every MUL_MAT, but flash_attn was previously partitioning by Q tokens (flat qrow spli
If this isn’t allowed here I apologize, r/unRAID deleted my original post. I am just looking for feedback either reassuracne that this is a correct way OR somebody to tell what I did wrong and, if they are willing, what I need to do to correct it. I am in the process of setting u
Haven't seen posted here. The research project matters because it proves you can pull trustworthy decisions straight out of an LLM's internal state in a single forward pass (no generation loop, no parsing, no fine-tuning) essentially turning a slow chatbot into a fast and cheap,
Seems to be comparable to GLM 5.2 at 501B A23B Parameters while being much more token efficient. Seems like an all around improvement to Inkling, not a bad open release for a non-chinese model!
Hey everyone, Like most of you, we have been fighting constant OOM errors while trying to fine-tune 8B and 70B models on consumer GPUs. The AdamW optimizer states are always the biggest bottleneck. We didn't want to rely on aggressive 8-bit quantization because we were seeing deg
Reddit from r/ollama.
TLDR: I built this open source app that lets local models monitor your screen and send you notifications! It now installs models on your browser , which makes local AI accessible to everybody ! Without any install :DD Hey r/LocalLLaMA! I'm back with some huge Observer updates c:
Hi folks, LIVE PROJECT PAGE TLDR: Moral of the story. You need better CPU to do actual agentic coding doing real work... I've been on a mission to make my RTX5090 go brrr for past 2 months so much so that i made my own engine for it which received "warm" welcome here (yeah, sourc
Paper linky - Context Language Models The central idea of the paper is incredibly simple. Give a model the ability to edit its context on-the-go like a file has major benefits on task performance, context management (memory) and even computational efficiency (both wall clock and
Hey all, we designed Cactus Whistle, an ASR model for ultra-small devices. It's not perfect, but mostly beats Whisper base with 9x less file size and 6x speed. Whistle supports English, German, French, Spanish, Italian, Dutch and Polish. Remember, the goal at Cactus Compute isn't
Picture from a post in r/amodei . People were praising qwen and I'm just wondering, what kind of new technologies are at play here? Does qwen just have "better" pre training data? That's more high quality?
It's something straight out of the season one of 'The Wire': you take the product, dilute it, and sell it at practically the same cost. It's some "Stringer" Bell shit. We should call the 64gbs "Stepped-ons" from now on.
Or which model do you think is good for general questions in daily life. I've been using chatgpt and Gemini for these types of questions. I wanna try different models.
Article · llama.cpp
Overview llama.cpp v0.6.0 introduces the new llama_batch_ext extended batch API (with llama_process ) for mixed token/embedding inputs and MTP/deepstack state embeddings, adds support for the GLM-5.3-Flash (GLM5-Next) 320B hybrid model, the Clef decision model (text and vision) a
Article · llama.cpp
llama.cpp : bump version to 0.6.0 ( #29997 ) llama.cpp : bump version to 0.6.0 scripts : update summary prompt (#0) Website:
GitHub (llama.cpp fork) :
I'm currently evaluating it for coding and our use-case at work (brain for voice agent). I feel it is really eager to get work done. Tends to just go ahead and make code changes, even though I intended it to just analyze, research or look up something. It runs tool calls like cra
i run bullmq (redis) for jobs that call llm apis. main pains are stalled jobs and retries charging for calls that already finished. if you self-host a queue for this kind of work, what do you use and what breaks? how do you stop duplicate runs and runaway costs? not selling anyth
tl;dr: I created a fully-local open-source full-duplex voice agent that rivals GPT-Live on some benchmarks. It uses Voxtral Realtime with a turn-taking head, a microturn-finetuned Gemma 4 12B and Breeze TTS 2 under the hood. Go try it out:
Purely a hobby side project to see how far I can push a really small model, using (mostly) automated training pipelines Full synthetic data:
TinyDecide is 10M Jev-like mode with 10M parameters and fits in just ~6MB. Smaller than every model on the Decision Index leaderboard and it punches way above its size . It runs almost anywhere: in the browser, Node.js, Python, Rust, and even on an ESP32.
Hey all. We've spent the last weeks getting Qwen3.8-Flash-Next (125B MoE, 6B active) to run properly on one AMD Strix Halo box (Ryzen AI Max+ 395, 128 GB). Tonight we're releasing both the 95 GB EXL3 weights and a new version of Kyojin, our inference engine (built on ExLlamaV3, o
Article · llama.cpp
CUDA: make the alloc_deps check batch independent ( #29986 ) Fixes #29980 Website:
q3.8-27b q3_k_xl this thing have done everything i possibly needed from him he wired up my openwebui spawned trillium service fix all my bugs set up pi-web-ui and debug my cloudflare tunnel it even spawned smaller LLMs to make the LLM use the toole he created to ensure it will wo
Hello. I picked up a new old stock 14" M3 Max MacBook Pro (14 core CPU / 30 core GPU / 36 GB / 1 TB) from my local market yesterday for around $2,498, and spent the night testing which local inference software is actually fastest on it for the two models I use. Four engines, all
Hi, I'm still new to this ai stuff, and /no_think or /set nothing, a
On my two DGX Sparks, Swift 1.5 (UkisAI's reasoning-efficient fine-tune of Qwen3.8-Flash-Next) running on veloGB10 (
After some recommendations for hardware (I'm starting from nothing), in the region of £6-7k. I will be mainly looking to use it for coding and have found Deepseek V4.1 Flash or Qwen3.8 27b good and so need a reasonable tok/s. Would love to get to 64GB VRAM but think it may be a s
Article · llama.cpp
vulkan: fix Flash Attention shmem write out of bounds ( #29988 ) Website:
Hey r/LocalLLaMA , Like many anime fans, I've always been frustrated by traditional MT engines (like Google Translate or base DeepL) when dealing with raw Japanese anime: - They completely butcher Japanese honorifics, sentence-ending particles ( -tteba , -zo , -desu wa ), and cha
Article · llama.cpp
server: support vision input for Clef ( #29969 ) server: support vision input for Clef move input_attn_causal to private extend old server_batch::embd server_batch::token::pos to multi dim nits fix abort fix img tokens cap fix yield_to_queue mutate data Website:
Hi everyone! I'm part of the Lokutor team. Last week we presented Oído here, and the response was amazing. We've received dozens of videos and messages from you guys saying you love it. Thank you! Now we're back with the next part of our plan: ItoTTS, a natural-sounding, streamin
think I'll stripe 5 HDDs I find in a box to store models. The idea is highest quality model and fastest tokens per second so when I ask my smart speakers to do automation things, it's faster. Not that the current server is bad in any way.
Running oQ4e+MTP on oMLX 0.7.0, with still more to optimize. Prefill is 1,878 toks. I saw some other benchmarks below what id expect so i figured I would share.
Article · llama.cpp
CUDA: Optimize accumulation in mmq for NVFP4 type ( #29857 ) ggml_cuda: optimize accumulation in mmq_vec_dot_fp4_fp4_mma for better performance remove whitespace fix: correct indentation in mma_block_scaled_fp4 loop Website:
Article · llama.cpp
cuda: stage the lightning indexer queries in head passes for MUSA MUSA archs 21 and 22 cap static shared memory at 28 KB, and the tile kernel staged the queries of all four heads next to the key tile for 33 KB. The queries are now staged in passes of LIGHTNING_INDEXER_TILE_HEADS_
Hi r/LocalLLaMA . I'm on the team at Blockway, a small team in Hong Kong (disclosure: this is our model). Today we released Agens Volundr 32B Preview, the first model built on our own hybrid architecture. We trained it on limited compute, it isn't perfect, and we'd rather tell yo
Article · llama.cpp
server: reject partial media truncation ( #24076 ) server: reject partial media truncation server: keep only the keep_first fix Drop the mtmd test helper change, which no longer builds since clip_image_f32_batch stores its entries by value, and drop the vision test: no test fixtu
Article · llama.cpp
vulkan: fix stale prealloc_y reuse across flash attention and soft_max ( #29591 ) Assisted-by: Claude Website:
Article · llama.cpp
vulkan: sparse flash attention for quantized K/V ( #29639 ) vulkan: sparse flash attention for quantized K/V Assisted-by: Claude vulkan: single-scan sparse FA index compaction The compaction ran one workgroup per mask row and walked the row in BLOCK_SIZE chunks, with a workgroup
Looking for some assistance /ideation. I am running qwen flash next q4 in my Mac mini m5 64gb. QFN doesn’t fit so this is done by having as many experts hot in cache as possible and streaming in the rest from ssd. I’m getting 17.5 tks decode and 390 tks pp. Have done a bunch of o
I keep testing harness & model setups on real tasks: basic math, vision, computer use (reading emails, navigating stores) and coding. I created a benchmark and a leaderboard. You can see it here:
Article · llama.cpp
llama : fix unexpected graph reallocation in the k-pool models ( #29958 ) llama : fix unexpected graph reallocation in the k-pool models Both k-pool models built a graph shape that depends on state the full-context reserve cannot know: qwen4exp branched on inp->cache_safe, which
The cool part is No training was needed. No hacking of the game state or algorithms needed Just simple instructions about what the snake can see, etc, and it can play in real time. 135ms is the turn limit of Google snake, so basically could be a human playing. Ofc, it could be im
There is no public release or hosted service to promote yet - I am looking for early architectural feedback. Hi everyone, I am prototyping Puzzlarium, an AGPL-3.0 jigsaw puzzle collection app. It is for cataloguing physical puzzle boxes, not for solving puzzles online. The direct
Article · llama.cpp
kv-cache: fix restoring mismatched KV cache rotation by saving exact rotation metadata ( #28498 ) kv-cache: save exact KV rotation metadata, reject restoring mismatched rotation tests : move the state rotation test to test-save-load-state the test is now part of the save/load tes
I have only just finished building an Epyc server with 256GB RAM and 2x 5060Ti 16GB, please don't tell me I wasted my money. Please?... Is a "Stratified" GLM 5.3-Flash coming, or something similar?
Reddit from r/LocalLLaMA.
Not the NAS or the hypervisor. I mean the weird little service you set up once and now half the house quietly depends on. Mine's an old laptop acting as a server, sitting in a closet that's hot even without it in there.
I've been running a Terramaster F2-424 NAS using the built in TOS for my Plex Server for a while, but after a lot of annoyance with the software along the way, as well as a LOT of annoyance and frustration with it as I've been trying to find out why the server just suddenly isn't
Article · llama.cpp
ggml-cpu : add Q8_0 IME1 matrix kernel for SpacemiT X60 ( #28479 ) ggml-cpu : add Q8_0 IME1 matrix kernel for SpacemiT X60 On the SpacemiT X60, IME matrix acceleration only covered Q4_0/Q4_1/Q4_K. Q8_0 had no IME1 kernel, and since the SpacemiT build sets GGML_CPU_REPACK=OFF ther
Disclosure: I made this. Sharing it here because it is fully local and small, and I want feedback from people who run models on their own machines. What it is: a 395M decision model (ModernBERT-large plus a 4 KB scoring head). You give it a state, a question and a list of options
Article · llama.cpp
vulkan : Fix undeclared identifiers when -DGGML_VULKAN_RUN_TESTS=ON ( #29912 ) Website:
Hi everyone, thank you so much for all the interest in my model. It's more than I expected. Here is a short summary of the trial and error I went through while building Apex-2. 1. GPUs were always the bottleneck I planned to train on about 1T tokens, but in the end I could only t
Said in title, I find the ecosystem difficult to understand, and RTFMing doesn't help me as it's never clear what is the server vs their client library ? I'm using it for voxtral 3B on one GPU, but it's because I can run that with no quantization, I'm lost on learning to run with
My setup is two RTX 5070 Ti 16GB cards (the second one is on an OCuLink dock) with 64GB of RAM, Ollama on Windows, and the agent runs on pi in WSL. Until last night the daily model was Qwen3.8 27B UD-Q4_K_XL at 128K with MTP, which does about 55 to 70 tok/s across both cards. I h
Article · llama.cpp
webgpu: add MMVQ support for Q1_0/Q5_0/Q5_1/Q3_K/Q5_K/Q6_K/MXFP4 ( #29483 ) add supports for q1/q5/q3_k/q5_k/q6_k/mxfp4 of mmvq path Add K_QUANTS_HANDLING macro to q1_0 of mmvq path Website:
Article · n8n
2.41.7 (2026-10-05) Bug Fixes API: Let global members grant variable scopes to API keys ( #40151 ) ( cc25c71 ) Features editor: Show Gateway credits promotions on community nodes ( #40109 ) ( a509d92 )
Article · llama.cpp
cuda: tile the lightning indexer over keys and tokens for 4 heads ( #29901 ) cuda: tile the lightning indexer over keys and tokens for 4 heads With too few heads for a wmma tile, a block scores 64 keys against 8 tokens: the keys are staged once in half precision, the queries one
Bit of a side project I wanted to share. The metrics are 17.5tks decode, 360tks prompt processing based testing against my normal ai usage. I tested a couple of new things others haven’t done (at least that I’ve seen). Setup a carousel buffer for streaming in experts for prompt p
Article · llama.cpp
metal : few-row MMA mat-mul ( #29869 ) metal : few-row MMA mat-mul and batched copies for speculative decoding Speculative decoding verifies a few draft tokens per step. Without the tensor API, Metal ran these mat-muls with the mat-vec kernels, whose time grows with every src1 ro
Looks like new open model coming soon and will be "strong" hopefully something under 200b for us memory poor. Also seeing statements about more western open models coming. Hope we get some good competition again on the open front! Here is original artical but its not free to acce
Article · llama.cpp
CUDA: use MMVF for thin f16/bf16 mul_mat at small batch size ( #29633 ) CUDA: use MMVF for thin f16/bf16 mul_mat at small batch size Signed-off-by: ynankani ynankani@nvidia.com adjust kernel selection logic Signed-off-by: ynankani ynankani@nvidia.com Co-authored-by: Johannes Gäßl
Article · llama.cpp
CUDA: prefer whole-tile FlashAttention scheduling for efficient two-stage kernels ( #29435 ) Website:
Just curious if this is something that would be worth building into ComfyUI? Seems like it would be worth it unless there is already an established workflow for this already? Built with PyTorch
Following on from club-5060ti and club-rdna16 , I’ve put together Infermeld : a small, open-source Linux companion kit for running one GGUF across an AMD GPU and an NVIDIA GPU, powered by llama.cpp. I’m the maintainer. This is an experimental v0.1.0 release , and I’m looking for
I recently built a dedicated local AI box around a 5090 and figured I’d share what I’ve learned so far. Specs are pretty straightforward: RTX 5090 32GB Ryzen 9 9950X 64GB RAM 4TB NVMe Ubuntu 24.04 llama.cpp Open WebUI My use case is mostly business/work stuff. I own a consulting
I think before I had windows portable...
I have been using Qwen3.8-27b-iq4xs variants for the past few weeks, tweaking parameters and running random tasks, but I dove into using
Setup/workflow: LTX 2.5 TXT2VID using wan2gp 720p output 10-second clips About 3 minute per clip Around 18 generated shots for the full 3-minute video, all were the first generation Prompts were written as continuous cinematic shots rather than shot lists I kept the same general
So, I have this image generation workflow that uses a LoRA that is trained on a specific art style that I like. And it works great! I'm able to see my favorite Pokemon in the art style I love. However, I would like to also see some of my own characters in the art style. I tried d
Buen día compañeros, les comparto mi primer dashboard de mi homelab, me base en varias ideas que mire por aquí así que por eso salió tan bien,espero les guste y acepto ideas y críticas, saludos
I'm using Qwen3.8-Flash-Next running on my Mac Studio as a daily driver for coding + productivity tasks, and yesterday it did something weird: I had it do some product research on amazon, so it was doing a lot of Web tool calls to amazon.com , until it made one request to routify
Made locally with Krea2 and MiniMax H3. Prompt for Krea 2: "A highly detailed, photorealistic indoor still life composition centered on a 3D-printed figurine of Lara Croft resting on a wooden table. The main subject is a carefully crafted statuette depicting Lara Croft in a confi
Article · llama.cpp
log, server: self contained colors, split child commands from logs in router mode ( #29895 ) log, server: make router child lines carry their own colors The logger writes the color reset after the trailing newline, so the reset opens the next line. On the shared pipe of a router
I've been using explainroo to make short narrated explainer videos. It runs fully local: Kokoro for the voice, Whisper for word timing, headless Chrome to draw the frames, and ffmpeg to put it together. No API keys needed. It works and I like it but it's very simple. After a few
Article · llama.cpp
llama: support both embd + raw tokens in batch ( #29622 ) llama: support both embd + raw tokens in batch add to test-llama-archs also check case llm_arch_supports_mixed_batch = false constant graph topology have dedicated input for mixed case rm set_tensor_backend is_embd --> typ
Video above is from using Minih3- song didn't carry over - still working that out. I spent literally the whole weekend learning this software. From morning till 3am and repeat on Sunday watching videos on youtube, reddit rabbit holes, and finally importing worksheets. Worksheets
Currently have a R9700 build with 64GB RAM. Want to get a second one for both speed and the ability to run Qwen3.8-Flash-Next at Q4/higher quants of 3.8 27B. Looking for thoughts or any improvements before buying the rest of the parts. Made sure to get a motherboard that supports
Reddit from r/comfyui.
I’m building a private alternative that runs completely on your own hardware. How it works: You hold a single button on your phone / or press a shortcut on your computer. You just say what you ate: "A bowl of chicken soup and two slices of whole wheat bread." The voice memo goes
Article · Ollama
Strip off pre-release from the version string so RCs can pull models for the same release.
Article · llama.cpp
CUDA: refactor swizzling code ( #29612 ) CUDA: refactor swizzling code fix templates/loop bounds Website:
I've a noob with ComfyUI. I've done something and haven't figured it out. When I submit a job, it goes into the queue and nothing happens after that. Even with the default anime girl. Is there something I need the to check? Or uncheck? Edit: I might add, self hosted, AMD GPU 8gb,
Tune my Strata setup for longer context: 131K, 262K first, then 512K if it holds up. I'm on Strata v . My hardware: - GPU(s): - System RAM: - CPU: - Platform: ,
Article URL:
Hi all. Happy to share my take on a PI UI that I tried to create in PI's spirit. It's definitely still beta but it works well enough as my daily driver for simple projects and phone chat support. Fully local, fully offline, no telemetry. Why another Pi GUI? I wanted a simple edit
I've been looking at the A few years ago, projects like Hummingbird+ suggested that cheap custom accelerators (FPGA-based) might become the future of local inference. But today it seems like memory capacity is still the real bottleneck rather than raw TOPS. For someone who wants
For the last few months, I ran DeepSeek v4.0 flash (NVFP4). First 0731 , then visionexp because it was a free improvement. I got around 65 tps decode and almost 2k prefill, and ran 4-5 agents in parallel, totalling around 200 tps cumulative decode. Because of this, I did not feel
Another quick Minimax H3 news and goodies round-up, for those who may have missed some items. -> A new '1980s fantasy movies' style LoRA for H3. No trigger word. The maker says it's not good for combat scenes.
Workflow on GitHub - Workflow on Civitai Hellllllooooo! With subgraphs FINALLY getting the respect they deserve, complicated workflows like this are easier than ever to create. I'm very excited to share this with you because I've been hacking away at it for weeks. I lost count of
MiniMax H3 Dual Segment - Split Sampler + Ultimate Upscale This is the first complete workflow I have built from scratch. My goal was to find a practical way to generate MiniMax H3 videos quickly at a low base resolution while still reaching above-HD final output on a 16 GB VRAM
I'm on an early journey into home labbing and looking for a computer to get started. I have 0 experience in this, I don't even own a personal computer. The goal at the moment is just Jellyfin and a music streaming service but It may very likely go further with some self hosting (
Using llama.cpp Ubuntu Vulkan prebuilt binary and the Acemagic miniPC S3A using an AMD Ryzen 7 6800H is a high-performance 8-core, 16-thread mobile processor launched on January 4, 2022, built on the 6nm Zen 3+ architecture loaded with 64GB of DDR5 RAM. It features a 3.2 GHz base
I was able to generate and edit images using the models in Playground mode in ComfyUI. However, it seems to have stopped working over the past two days. I’m still able to run the workflows on my phone, but Playground mode provided a much better user experience, especially on mobi
After several months of development and a lot of real-world testing, I’ve released Production Slate V4.4 , the first public version of my ComfyUI production-output node. I originally built it because once I started generating large numbers of images and video takes for films, kee
Hi, I'm the maker of PromptNook. If you've got folders of ComfyUI outputs and want to compare the prompts, LoRAs, and settings behind two results, this is what I've been building. The latest update lets you: • Drop in a batch of ComfyUI PNGs or workflow JSON, review each file, an
I finally started using Runpod and Comfy. No idea what I'm doing, learning from online tutorials and this community. Now that I'm in the weeds, it seems like I can't run Seedance (and a few other video generators) without an API connection and additional fees, which defeats the p
Article · llama.cpp
ggml-cpu: support BF16/FP16/FP32 K tails in tinyBLAS on x86 ( #29806 ) ggml-cpu: vectorize BF16 K tails in tinyBLAS tests: Skip tinyBLAS when use_ref is enabled so CPU tests compare against the vec_dot path. ggml-cpu: vectorize tinyBLAS F16/F32 tails Website:
Thanks for all you comments on my previous post. Now I understand that homelab and self hosting never complete and always involve. Added 4g+ router for backup and I diy an exhaust window with two 140mm fans that send the hot air out of the room for keep the attic cool. Very pleas
I vibed this up this weekend, fully local, with GLM 5.3 Flash running on 2x DGX Sparks. vllm TP2 recipe:
Help with this list. Give me your recommendation on "not supported anymore" GPUs. Looking for budget and Vulkan friendly options. Most of the GPU are not supported by latest CUDA / ROCm. Often with some witchcraft magic they are able to run with native backend. I prefer the simpl
Article · llama.cpp
cuda : move neu_padded to where it is used ( #29940 ) Signed-off-by: Adrien Gallouët angt@huggingface.co Website:
I have a monthly subscription to Ollama that I use for Silly Tavern. Could I use that for Comfy as well to do image to image workflows? And if yes, would anyone have a guide on how to set that up? Thanks.
Reddit from r/homelab.
Working on a resource dashboard, rich logs, lightweight 64mb cap, all local, scales on network API endpoints via collector, supports multiple engines (llama, strata, custom cuda engines, unsloth, LMS) Not public yet, but curious if anyone would be interested? It’s better logging
Last update for those following:
I tried both these models on a few small agentic tasks with tools (i.e. "What's the weather like today?", etc.). They're both pretty solid at using web search to find answers despite being small models. TLDR: LFM2.5 2.6b is the clear winner. Somehow it's faster and uses less ram
Article · llama.cpp
ci : windows llvm build requires ninja multi-config ( #29959 ) Website:
I forked llama.cpp's server into AgrillaMoE, a dedicated build for Qwen3.6-35B-A3B (~A4B) with Unsloth quants. On a (vant.ai) rented V100 16GB with the 2-bit UD-Q2_K_XL quant it generates at ~57-60 tok/s while running the full MoE-expansion profile - and it speaks both the OpenAI
Whenever I feed codebases into local models (Qwen, DeepSeek R1) or API models, the biggest annoyance is prompt pollution: - Lockfiles (`Cargo.lock`, `package-lock.json`) burning 30,000+ tokens for zero reason. - SVGs, binary files, or build artifacts slipping into the context. -
We’ve already seen AI transform software development, and other industries are rapidly following suit. There is no doubt that AI demand will be unprecedented. It won't be long before the mainstream catches on to what we local AI enthusiasts are building on consumer hardware. At o
I've had one pull request merged into ds4 (DwarfStar), a tiny one. There are a few more still waiting in the queue. I’m not complaining. Antirez says it clearly in the README: with coding agents everyone can tune the engine for their hardware and model and he can’t review everyth
I've been wanting to test out LLM inference on FPGAs for a while now, but didn't have a big reason too since there were no frontier class models at 9B-27B scale (3.6 27B was great still but it didn't motivate me enough). When 3.8 was released I was pretty impressed by what it cou
Article · llama.cpp
add windows vulkan arm64 release add link
Article · llama.cpp
chat-peg-parser : clear current_tool when pending_tool_call is reset ( #29942 ) A TOOL_ID node that arrives after TOOL_CLOSE wrote through current_tool , which still pointed into the just-destroyed pending_tool_call optional (use-after-free, then a second free of the id buffer).
I forked Strata and worked with Opus 5.5 with some heavy changes to it to make it work on an IBM AC922 I have access to. The IBM AC922 is a 2018 era beast with two POWER9 20 core SMT4 CPUs that are connected by NVLink to 4 or 6 NVIDIA Tesla V100 SXM2 GPUs, the CPU-GPU BW advertis
First of all, thank you for reading. I trained a small MoE model completely from scratch (no external base weights) and wanted to share the results + a couple of lessons. Apex-2 - Architecture: Decoder-only MoE, every layer is MoE (no dense layers) - Size: 3.87B total parameters,
Hello everyone, I’ve just finished a new ComfyUI workflow that allows you to use MiniMax H3 as an image generation model and create detailed character sheets from a single reference image or a batch of images. The main goal is to use these character sheets as stronger references
I'm using Qwen 3.8 and qwen flash on a 5090. It's miles behind the latest Opus 5.5. Even if it was remotely capable it would be a huge help to me, but for now, with regards to 3D modeling, local models are not close at all.
Ive got a rasbery pi 400 (64gb) and an external 1tb HDD any tips And tricks for jellyfin or anything else i can do? Otherwise i Have a 3com switch And multiple ddr1 pcs that Will Join in the Future.
Article · Ollama
mlx: match publisher tokenizer semantics Honor pretokenizer stage order, split behavior, Unicode boundaries, added-token normalization, and ranked BPE merges. Handle empty added tokens and empty Metaspace input consistently. Add shared Go/Python reference cases using published to
i wrote this. `adhdev standalone` runs on localhost:3847 with no account and no cloud: it attaches to the terminal coding agents installed on that machine (claude code, codex cli, kimi code, cursor agent, antigravity cli, grok cli, opencode) and gives you their sessions, chat, te
Was getting tired of my agent sessions crawling at ~20 tok/s, so I finally benchmarked properly instead of guessing. Setup: 5070 Ti 16 GB, 14700KF, 96 GB RAM, Windows. Running Qwen3.8-Flash-Next IQ3_S with 262K context through Strata (the KV cache lives in RAM and streams to a 32
Article · llama.cpp
cuda : move blocks_per_col to where it is used ( #29939 ) Signed-off-by: Adrien Gallouët angt@huggingface.co Website:
From the first LLaMA 33B I knew I wanted that magic-like intelligence locally, mine, so nobody could take it away when I needed it. I bought a 3090 for my home PC. Then LLaMA 65B appeared and I was dazzled, it looked like it had all the knowledge in the world. I made two copies
I modeled a little rack for my two OptiPlex mfs based off of milk crates. I primarily use them for Jellyfin and a Minecraft server, and as a sort of replacement for GH Codespaces through TTYD and Cloudflared--so i can code at my day job. All that fits pretty comfortably on the fi
A while ago I asked here how to turn ~5 million court decisions into structured graphs without running an expensive LLM on every document thanks for the advice . I went with the "small extractor + classifier" idea and it mostly works, but I'm stuck a bit below the LLM. And like I
Last week I posted a 90-second piece I made while dogfooding konte , an open-source production tool that lets a coding agent run a multi-shot AI video as a project: shots, takes, dependencies and review notes. Previous post This time I wanted something harder on timing and revisi
Hi guys, i am totally AI noob here. I want to run some text - video / image - video models locally. I have 4070ti super GPU (16GB VRAM) I am trying to install it but its super confusing. can anyone guide me to a proper resource that i can follow? there are so many models or encod
I wanted to know if theres anything that can improve creative writing for our Local Models? I specificly am asking for qwen models as they are way better when it comes to researching and writing html files compared to gemma / muse (which I know are better at creative writing). Im
Article · llama.cpp
CUDA: fix MMQ memory fault if n_expert >> n_ubatch ( #29941 ) Website:
I have done round 1 of homelab (more like selfhosting) where I have just spun up services as a means to an end, and some experimentation with Active Directory to practice for job interviews (i ended up getting it!) the lab consisted of 1 Proxmox host, virtualized TrueNAS for stor
Got tired of the ComfyUI web interface, so I'm building a native C++/ImGui desktop client (zero vibecoding:)) Planning to release as open source on GitHub. Here is a sneak peek. What features would you guys consider a must-have for the first build?
Been playing around with local TTS with Breeze combined with STT, and the results are amazing. Using Opus 5.5, I can hear the first sound after 500ms if there is no thinking involved, and with thinking on low mode, can be 1-1.5s. I'm using a BLE remote (the kind that are used for
Article · llama.cpp
imatrix: calculate activation-based statistics for new format (GGUF) imatrices (
How is JEV any different from using an embeddings model? I really will appreciate if someone can explain this to me - because I have yet to see the difference. I'll even give you my JEV server for free! It uses ollama, you install `ollama pull nomic-embed-text:latest`. % python3
Hi guys, I saw this PC in my local thrift store. Is this a good config for about $56? I don't want to buy a bad CPU for my first homelab but I'm on a budget. My tech friend told me that the i4 was the perfect balance between the i3 and i5, so I was pretty excited to see an i4 lis
Article · llama.cpp
spec : fix n-gram drafts rejected at temp > 0 after truncation ( #29924 ) Co-authored-by: Pranesh Gonegandla pgonegandla@nvidia.com Website:
Article · llama.cpp
common : prepare load_from_models_dir() for path conversion ( #29674 ) This is part of the fs::path modernization series. That was also the opportunity to remove fs_list(). Signed-off-by: Adrien Gallouët angt@huggingface.co Website:
So I tried that miracle engine everyone is talking about. Asked the IQ3_S model to express its opinion on a post from this sub to measure the tps on a long-ish generation: Can you help with the following problem? So Kimi K2 is outdated, and so is GPT OSS 120b. Which of the modern
Article · llama.cpp
server : fix dead LLAMA_ARG_HF_REPO_FILE key in preset allow-list ( #29938 ) Signed-off-by: Adrien Gallouët angt@huggingface.co Website:
This is my local LLM setup, squeezed into a 12U 10" rack. The machines are an MSI EdgeXpert on top and three ASUS Ascent GX10s below. All four are NVIDIA GB10-based systems with 128GB of unified memory each. I 3D printed the rack using the KWS Rack v2 design. For airflow, there a
I got my AI server all setup and shut it down to bring it to the basement to put it back into the server rack, when it came back up I went into my client app to put a test message in, love its response. :)
I am running a scraping SaaS and my cloud bills get up every month. So i decided buy secondhand mac mini m4 every month instead of paying 500$ to renting servers. All mini's has their data part of distributed database, thunderbolt is really helping there. I distributed my databas
Reddit from r/LocalLLaMA.
The link to the github repo:
IPO is coming up in a few weeks. Wonder how much more Anthropic would be valued at with a morally defensible monopoly in the stead of a legally defensible one. It's so transparent at this point what they are doing.
Not a bot. Not a Strata shill. Just sharing my experience. So, I have an R9700 in my machine, plus an RTX 5060 Ti, and 64GB DDR5 system RAM. Overall, not a bad setup. Anyway, I mainly run a daily driver local LLM on the R9700 while running image/video inference on ComfyUI on the
Hi everyone, Sharing a personal project exploring the minimal bare-metal footprint required to run an autoregressive LLM. Instead of relying on large runtimes or compiler abstractions, I wrote an inference engine for Gemma-2B entirely in flat x86-64 assembly (FASM): - **Binary fo
So Kimi K2 is outdated, and so is GPT OSS 120b. Which of the modern open weights models can boast the least sycophancy? I need this both for creative/research assistant usage (sycophancy led me down blind alleys of my own bad ideas many times) and agentic coding (more sycophancy
The setup: Proxmox, one container, Pangolin and Cloudflare in front, a LAN-only admin console. Anything you put on the internet gets crawled fast. Instead of just blocking it, I built a site that catches every bot, identifies it, and shows it in a live zoo: The Crawler Zoo. The s
Given the commentary on the Q3.8FN release page here
Article · LiteLLM
Verify Docker Image Signature All LiteLLM Docker images are signed with cosign . Every release is signed with the same key introduced in commit 0112e53 . Verify using the pinned commit hash (recommended): A commit hash is cryptographically immutable, so this is the strongest way
Hi Wondering how many people use .local for their home lab I originally got a domain on the internet and then added subdomain for home lab stuff - this was before .local so how many people use , has it been of benefit ?
Article · MarkTechPost
Learn how NVIDIA IsaacTeleop turns XR hand tracking and motion controller input into robot commands using a pure Python retargeting engine and NumPy. The post Inside NVIDIA’s IsaacTeleop: From Hand and Controller Tracking to Robot Actions with the Graph-Based Retargeting Engine a
It provides the speed and somewhat accessibility to low vram users to run Flash Next as their main. It's providing the speed of qwen 3.5-35-a3b but with higher intelligence over 3.8 27b. This will allow many people to replace 27b. I'm getting around 80-110 tok/s 2500 pp. 2x 3090'
Article · LiteLLM
Verify Docker Image Signature All LiteLLM Docker images are signed with cosign . Every release is signed with the same key introduced in commit 0112e53 . Verify using the pinned commit hash (recommended): A commit hash is cryptographically immutable, so this is the strongest way
Article · LiteLLM
Verify Docker Image Signature All LiteLLM Docker images are signed with cosign . Every release is signed with the same key introduced in commit 0112e53 . Verify using the pinned commit hash (recommended): A commit hash is cryptographically immutable, so this is the strongest way
I got tired of jumping between ComfyUI, random HuggingFace scripts and different UIs, so I built Aleph: one web interface served on localhost that unifies image, video, audio-for-video and voice generation. A few details: - 100% local, nothing leaves your machine - Python server
Hi all, How you guys handling safe deployment of websearch in Hermes, pi and other harnesses? Does anyone have a good uproars setup guide for local models? I tried to implement a sandboxed search system but it caused endless tool calls. Want to guard against prompt injection and
Hello everyone, this one is a bit more modest than a lot of the posts I see here. As much as the passion here is very inspiring, my use cases are generally a bit simpler. I use the Optiplex for the common Adguard, Tailscale, VPN services. I also have a database, a few scripts run
Hi everyone. Created a new App for homeassistant, VisionState. Its intended use is creating sensors from camera views. Examples: Garage door open or closed? Are there cars in the driveway? How many minutes left of timer? Water Usage. It uses local lightweight AI models, which run
Article · llama.cpp
mtmd : fix deprecated strdup warning on Windows ( #29863 ) Signed-off-by: Adrien Gallouët angt@huggingface.co Website:
Does anybody have experience with self-hosting a super-low-cost or no-cost landline service? It seems the only VoIP space these days are people trying to run home offices and need multiple lines. I just want something super cheap to run a an old-fashioned landline or two. My goal
Still don't have a good cooling solution, but here are few benchmarks. I lowered the power limit (TDP) to 145 watts each. I changed the firmware on one MI50 to activate the miniDP port. Did have to use xrandr to create a new mode so I could get 1920x1080 output. Each GPU has 16GB
I wanted to see how far I could push a fairly ordinary laptop with a huge MoE model. Turns out, Qwen3.8 Flash Next 176B can run on: RTX 3080 Laptop - 16GB VRAM 32GB system RAM SSD No 128GB/256GB RAM workstation and no multi-GPU setup. I’m running it with TensorSharp , my open-sou
The jokes write themselves.
Hi, Im gonna go all in on this, best finetune iv got my hands on for agentic coding, period. Test it yourself, im not gonna give you any benchmarks or evaluations, use your own tests and pracices, give your opinion after you use it. Nothing i can say will persuade you anyway, bes
Anyone able to sanity check this (slightly vague) plan? I've got an old synology with a mirrored pair of 4TB disks in it. Mostly movies, music and photographs. Doesn't get new content very often. I stream music (minimserver) and movies (movies via a separate jellyfin server - but
Hi everyone, TL/DR I created NInfer 4080 to run ISTA-DASLab-Qwen-3.8-27B-GSQ at 100k context on an RTX 4080 16GB GPU using way more of the hardware capabilities ( max overall: 2720 tok/s prefill, 262 tok/s generation ) and sharing it with the community now so others can also have
There seems to be a whole category of extremely narrow inference runtimes appearing: Strata, ninfer, DwarfStar, Splash, llamAmpere, gufo, etc. They deliberately give up the thing llama.cpp/vLLM are great at - generality - and optimize around a small number of models and sometimes
Article · noted.lol
Aside is a minimal Ghost microblog theme for sharing thoughts, code, photos, and to-do lists, featuring dedicated galleries, a Code Library, and a visual archive.
Been testing 3x 3060 12gb for flash next in an open air frame. Honestly, with strata it's kicking ass. 38-40 t/s while llama.cpp can only get 13.2 t/s. This is on IQ3 through strata. Anybody else running dated mining hardware with decent success? PS flash next kicks ass. Rig deta
I have an original courtyard image that I want to use as the background, and I have a custom identity LoRA for the subject. I want to generate the subject with different poses/clothing while keeping the exact same architecture, camera perspective, background objects, lighting and
With the new tech around video ai generation is there an ai video model that can fix bad in between frames? I wanted to clean a video from Chrono Trigger FMV, frames like this.
Already have an i5-8500T pc for my homelab so now this guy will take care of Adblock and network services with the more powerful one focused on media.
So I just got my first homelab running by buying things from Facebook Marketplace and fixing them up if needed and now I have no idea what to do with it, which is why I'm here. It's a used Dell OptiPlex 5040 (i7-6700, 32GB DDR3L) running ZimaOS, with a 512GB NVMe for the OS and a
Reddit from r/comfyui.
So this is maybe a niche problem, but at my job I work on a huge Python codebase and every time I change some shared function I'm basically playing roulette. grep tells who mentions it in the code base, not who actually calls it. And more essentially, Claude Code (my major coding
Follow-up to last month's post. One of the main issues people ran into was roughness or break-up on some cloned voices. 2610 is an interim update focused mostly on improving that. Reduced roughness and break-up on some of the voices that struggled before Same 120M model, same spe
Innanzitutto, due cose per evitare fraintendimenti. Non sto cercando di sostenere che un modello o un'azienda siano migliori. Non ho alcun interesse personale in nessuno di essi. Volevo solo verificare personalmente come si comportano nello stesso contesto lavorativo. E il motivo
I’d like to preface this by saying: Please forgive my crappy cable management. 😂 I finally got my homelab updated, so this should HOPEFULLY be the last time I update it for years (doubt it, I have a sickness). First pic: - The top shelf houses the Spectrum modem (boring). - Sitti
With the version 2 release, the Marigold model (free, even for commercial use) now achieves finer detail in a depth map estimated from a single photograph. Its intended use is as an intermediate step, hidden inside a post-processing pipeline. However, I believe that having a rend
ive tried emailing them and bug support but no answers.
Article · llama.cpp
server : fix laya abort by limiting n_batch to n_ubatch ( #29903 ) server : fix laya abort by limiting n_batch to n_ubatch Fixes #29902 Assisted-by: Claude fix(review) : rm tests, embeddings cond Website:
I hosted my own world of warcraft private server then built a client that you can play in the browser on PC or mobile at
I'm going to buy a ThinkCentre M720Q with an i5 8400T and 8 GB of RAM (no SSD) for €170. I'll buy another 8 GB module to have a total of 16 GB of RAM and install Proxmox. Do you think this is a good idea? I also have a Raspberry Pi 4 (which I really don't know what to use it for)
Article · llama.cpp
common : add common_is_tty() helper and fix deprecated warnings on Windows ( #29860 ) Signed-off-by: Adrien Gallouët angt@huggingface.co Website:
Novo por aqui. Quais modelos eu conseguiria rodar no comfy com esse mac? Video e image? 🙏🏻
Strata runs Qwen3.8-Flash-Next on gaming PCs, but with 32 GB of RAM its low-RAM mode only works on one GPU. Split the model across two cards and the experts that don't fit in VRAM get read from the SSD. My 4060 Ti sat next to the 5080 doing nothing. So I forked it. The RAM copy o
Article · llama.cpp
chat : honor json_schema in Ling 3.0 parser ( #29813 ) chat : honor json_schema in Ling 3.0 parser Ling 3.0 only built a grammar for tool calls and did not handle inputs.json_schema, so response_format requests were left unconstrained. Add an eager response-format grammar path wi
Article · noted.lol
LabDeck is a Ghost theme developed specifically for homelab enthusiasts and the self-hosting community
One can only hope for Qwen4 35B A3B or similar (maybe with extra ngrams for 35B -> 70B+... until then, Gemma 26B QAT my beloved (solid 26 Tg/s)...
We built an engine, Kyojin, on top of ExLlamaV3 for Strix Halo (gfx1151, ROCm), and packed two 300B-class MoE models so each fits one 128 GB machine. First release, all measured on Ryzen AI Max+ 395. Model GLM-5.3-Flash MiMo-V2.6-Flash-MOPD Size 99.7 GB 105 GB Prefill 580 tok/s a
It's like the entire sub has become that scene from Konosuba where the cult keeps making up fake scenarios saying the only solution is to join their religion
So, I've been begging llama.cpp to do MoE caching for about a year, and watching them d ck around with 1% here and 2% improvements there instead... Until Strata (
After my SD card boot drive corrupted for the last time, I went hunting in local markets for a cheap, permanent fix for my budget server (Zebronics G41 / Core 2 Duo / 6GB RAM running TrueNAS SCALE 24.10). Managed to pick up an Intel 313 Series 20GB SLC SSD with hardware Power Lo
Meeting notes are one of the last parts of my workflow that still depend heavily on cloud tools. I've used Fathom and lately Bluedot. Bluedot works well for me because there's no meeting bot and I get the transcript, summary and action items after. But I'd really like to move mor
Another quick Minimax H3 news and goodies round-up, for those who may have missed some items. -> I've organised my Reddit posts into a basic themed H3 directory page on the Web. It doesn't aim to be completist, but it its organisation should make it useful for quick lookups.
Article · llama.cpp
ci: fix flaky ADD_ADD f16 by using the fused ADD tolerance ( #29904 ) Website:
Basically title. I understand basic workflows, node functions and interactions etc. but it's harder to find more information about specifically explicit nsfw work, hence why I ask those who might now! Thanks :)
Disclosure: I am the creator of this project. After days of lurking and building up enough karma, I can finally post here. Every time I tried running local RAG on my own machine, I hit the exact same bottlenecks. First, spinning up Chroma or another vector database alongside an 8
I have been getting lot of request to publish .char support with ComfyUI, here is a custom node that I created: Node Repo:
Article · llama.cpp
graph: gather the recurrent states once so the reserve covers every split ( #29856 ) build_rs gathered the extra states (n_rs - n_seqs rows) with their own get_rows. The worst-case reserve has n_rs == n_seqs, so that node was sized at zero rows, and any ubatch whose cells are not
Article · noted.lol
LibreDB Studio is a free, open-source database editor that runs in Docker, letting you securely manage your databases directly from your browser without exposing ports, opening SSH tunnels, or installing additional software.
should i be getting an ai for img generation comfy ui with online gpu and which model for consistent images and svg (mostly all types of digital files)
Anyone knows of good non-nude models/loras ect? I noticed the NSFW images civitai look better than the SFW images BUT when I browse Civitai 99% of the SFW images were just RNG seeds that happened to generate a safe image despite NSFW model/prompt. thanks in advance submitted by /
Article · llama.cpp
ggml-openvino: update to 2026.4.1, optimize performance, expand ops, improve device listing. ( #29852 ) ggml-openvino : Qwen3.5 MoE perf ( #312 ) Squash of ravi9#312 : ggml-openvino: add detailed inference profiling (Yu, Zijun) ggml-openvino: use remote output tensors by default
Reddit from r/LocalLLaMA.
Article · llama.cpp
qwen4exp : halve the indexer score memory ( #29825 ) qwen4exp : halve the indexer score memory The indexer scored all heads in one product and rectified a copy of it, so two [n_pool, n_idx_h, n_tokens] f32 tensors were live at once, the largest buffers of the graph at long contex
Hey everyone, I’ve been working on a game called Anyworld. It’s a browser-based multiplayer (single player also supported) text adventure inspired by the early days of AI Dungeon, especially its browser-based free version AI Dungeon 2. The setup is pretty straightforward: one per
I present you my simple but boring home server setup. It consists of this Fritzb
Using Strata on a 7900xtx plus 64 gb ddr5 ram, 60t per second even at 250k context, can finally use my pc while working AI in the background, even game as well, smarter and more precise than dense model, it follows orders more accurately, follows plan more versatile, it goes arou
Hi folks, it's Lewis here from the post-training team at Hugging Face. We've been exploring how to train open models in different coding harnesses and wrote up a looong guide on how we solved this using open source libraries like TRL and the Harbor framework for RL environments.
Reddit from r/LocalLLM.
ISD-TOWER. (Not finished yet.) Central
Kimi K2 was already good but they took K2.5 a whole new level with so much of their continual learning phase, I believe it was on more 20-25T tokens iirc. Similarly K3 is just such an amazing model, I just love this model, wondering how amazing K3.5 will be!!
Article · llama.cpp
model: add support for clef decision model (text-only) ( #29831 ) init support for clef (text only) more static graph clean up nits nits 2 Update gguf-py/gguf/constants.py Co-authored-by: Sigbjørn Skjæret sigbjorn.skjaeret@huggingface.co Co-authored-by: Sigbjørn Skjæret sigbjorn.
In the latest release of the completely unofficial gufo port to windows I've added a pre-packaged library that you can use to try out gufo for yourself. No need to build anything just grab the .zip and unpack it. I've also added start.cmd for easily starting the server, it will t
I love Qwen3.8 27B so much that I decided to show my gratitude to the Alibaba ecosystem by building a dedicated RAG/search node using a cheap Mini PC from AliExpress. Turns out, my ecosystem loyalty got rewarded with an absolute masterpiece of fraud: Promised: Intel N150 + DDR4/D
LFM2.5-Encoder-350M is a multilingual bidirectional encoder built on the LFM2 architecture - a larger encoder for maximum downstream quality. It is a masked language model with full bidirectional attention, designed to be fine-tuned into task-specific models (classification, toke
I’ve tried multiple ways to run MiniMax H3 on a RunPod Serverless endpoint, including a custom Docker image and Flash/FlashBoot, but nothing has worked so far. The same model runs fine in a regular RunPod Pod, so I’m not sure what is different in the Serverless setup. What is the
Article · MarkTechPost
IBM has made a self-hosted deployment of IBM Bob, its agentic software development platform, generally available. Enterprises can now run Bob on premises, in private or sovereign clouds, and in air-gapped networks. They bring their own model: NVIDIA Nemotron or Poolside Laguna fo
Benchmarking an LLM here with a NVIDIA RTX 5070 12 GB VRAM here I had been working on a llama.cpp based expert streaming setup for Qwen3.8-Flash-Next 177B (UD-IQ3_XXS) on Windows. Benchmark is about 11.5 tok/s, up from roughly 7 tok/s on the inherited setup. In normal conversatio
Article · llama.cpp
CUDA: fuse shared experts into MMVQ ( #29184 ) CUDA: fuse shared experts into MMVQ check if buffer is null move stride_col_dst to fusion args Website:
Qwen Flash Next now uses less VRAM
I'm having a go at building a fully offline, voice-first assistant running locally on a fanless mini PC (4 core Celeron J6412 w/ 16GB DDR4, 512GB SATA SSD, crappy Intel UHD iGPU only) Going to try Ubuntu 24.04, llama.cpp, Python. Are there any models that might be able to hold a
I've noticed a trend with most new models with regards to their writing style. They are creating a new style, and this seems common among them. It's very information-dense. Here is an example from GLM 5.3 Flash. I'm gonna be honest here and say that my prompt was kinda silly; my
Full disclaimer: I've leaned heavily on Fable to develop this, but I've tested it thoroughly on my own library for a couple of months before putting it on GitHub. I live in Thailand, and it started as a way to get Thai subtitles for Shin-chan for Thai friends and for expat friend
There are so many complicated H3 Latent Save/Load node packs, and all of them seem designed to do the obvious thing: continue video…but I’d love to have a way to save the latent and then regenerate just the last frame (or any frame, if that’s possible) as a higher resolution stil
Article · MarkTechPost
Decision AI models answer typed questions with calibrated probabilities instead of generated text. TypeSafe's Jev costs $0.042 per million input tokens and returns responses in 70 to 500 milliseconds. We break down how it works and its benchmarks, then compare it with Fastino's G
IQ3_XXS weights are just under 80GB and my slowww DDR4+7900XTX is stabilizing around 45-70/s (sometimes higher while coding depending on mtp). Looking online I'm seeing similar results for users with 12GB and 16GB cards, and significantly faster numbers for owners of DDR5. (In co
Don't worry after some makeup and new fans you should feel better. And if you make it Mr.Hermes will treat you well.....
Article · llama.cpp
spec : add probabilistic sampling for simple draft and MTP ( #27694 ) Make the drafter probabilistic and the target verify by rejection sampling Drop stale spec_draft_q before drafting Fallback to argmax sampling for grammar-constrained requests and adding flag for enabling proba
I didn't have a ton of containers or VMs and I wanted to change things up, decided to learn more about docker and dive into that. It's been working pretty good so far. Anyone else just run docker?
I have been running the same midrange card for a couple of years now, and every model release sends me back to the price sheets. The performance gains are real but they keep landing next to price tags that sting a lot more than they used to. Lately I get more satisfaction out of
Article · llama.cpp
ggml-quants : avoid invalid rounding in qkx3 scale search ( #29817 ) ggml-quants : avoid invalid rounding in qkx3 scale search The imatrix scale search can produce an infinite, NaN, or otherwise out-of-range value when the fitted minimum collapses to the maximum or makes the rang
It's been a blast and learning a ton. But seriously, you all have me down a rabbit hole that my wallet and hours of sleep need to be pulled out of.
Hi guys, I am pretty happy to announce MegaCapybara . Purpose build engine for RTX5090 that is focused on Qwen3.8 27B (more will come later). GITHUB (Engine) HUGGINGFACE (weights) Why ? 1. It beats Ninfer which was until that point SOTA engine for RTX5090. By roughly twice in dec
Article · llama.cpp
ggml-cpu : fix soft_max_back wrong output when dst aliases src1 ( #27096 ) ggml-cpu : fix soft_max_back wrong output when dst aliases src1 GGML_OP_SOFT_MAX_BACK is listed in ggml_op_can_inplace, so the graph allocator may assign dst to alias either src0 (dy) or src1 (y). The resu
Article · llama.cpp
metal : add tensor API flash attention kernel for F16 KV ( #29570 ) metal : add tensor API flash attention kernel for F16 KV cont : add tensor FA kernels for DK=DV=512 and DK=576, DV=512 cont : support attention sinks, ALiBi and logit softcap in the tensor FA kernel cont : add te
in kleines Skript oder eine interne Datenanalyse wird direkt in eine proprietäre Cloud-KI geworfen. Und plötzlich ist vertrauliche Unternehmenseigene Information in den neuronalen Netzen eines Big-Tech-Giganten verschmolzen. Klingelt da für euch nicht mal die hell
Article · llama.cpp
llama, server: add /v1/systemone API (models: laya, julia-1, lev, openjev, kev) ( #29818 ) init conversion convert: ok model loaded add server code improve conversion script support shared prompt prefix add docs, imorove UX a bit add vision support add openjev tiny model for test
Hi r/selfhosted ! I'm a solo developer and I've been building Kaoru, an MIT-licensed desktop AI assistant that can also act as a coding agent on your projects. Quick note up front: this is a desktop app, not a server you host. I'm posting here because the design goal was to keep
Article · llama.cpp
vulkan: disable large matmul tile on Samsung GPUs with 32KB shared memory ( #28531 ) Assisted-by: Claude Opus Website:
Article · LocalAI
🎉 LocalAI 4.11.0 Release! 🚀 LocalAI 4.11.0 is out! This release makes LocalAI more useful for audio understanding, resilient model serving, structured decisions, and day-to-day operation. Audio scenes can now combine transcription, diarization, sound detection, and remembered spe
Article · llama.cpp
qwen4exp : optimize mask constructions ( #29824 ) qwen4exp : optimize mask constructions cont : apply the same change for GLM5-next Website:
Looking for some ideas from people who know a lot more about this than I do. We've got funding for a small AI lab in a public further education college in Ireland (roughly community college in the US). The hardware is reasonably decent. The goal is to give students useful skills
![v0.31.0rc5: [Misc] Add Transformers version upper bound in requirements (#59614)](media/8ebdd7fd9c2ee1d1.png)
Article · vLLM
Signed-off-by: Isotr0py Isotr0py@outlook.com Co-authored-by: Harry Mellor 19981378+hmellor@users.noreply.github.com (cherry picked from commit 58b3298 )
A novel way to intervene existing video through diffusion, particularly abstract and structure-driven material: taking its movement and form as the starting point, and reinterpreting its textures, materials, and visual language. I’ve been developing this around the audio-reactive
I've been using my free time during weeknights and weekends for the last 11 months working on and refining a small domain-specific language model. It specializes on information about World War II. The number one question I get asked about this is "Why did you pick World War II?"
Hey guys, Last time I tested Qwen3.8-Flash-Next on its own. This time I put three Qwen3.8 checkpoints through the same 10 tests on the same RTX PRO 6000: RadixArk/Qwen3.8-27B-NVFP4 (dense) orcarouter/Qwen3.8-27B-Uncensored-NVFP4 (dense, uncensored fine-tune) RadixArk/Qwen3.8-Flas
TL;DR - ≥32 GB VRAM → Qwen3.8-27B , Q4 (NVFP4 / Q4_K_XL / IQ4_XS) - Otherwise → Qwen3.8-Flash-Next on Strata , Q4 or IQ3 depending on your total RAM + VRAM For the past few weeks I've been deep into local models: testing everything I could get my hands on, reading up on which h
Another quick Minimax H3 news and goodies round-up, for those who may have missed some items. -> How low can you go? A new 2-step LoRA for H3, 'PDMD 2-NFE', now in ComfyUI format.
I designed a 3D printed conversion to turn the Framework 12 laptop mainboard into a FrameVault home server such as a NAS or Home Assistant system. The Framework 12 mainboard is fitted into the side of the unit and allows for 4 HDD's to be installed inside. See the Printables link
An agentic model from Microsoft for the GPU poor
One m720q is all it took. I fell in love with the boxes and now I can't stop finding more. Jellyfin server for the friends, and digital blu ray backups, has become matrix/immich/ mastadon/ home network. Future boxes for torrents, reticulum/nomadnet/lora shinnanigans, off site bac
[10/8 UPDATE] Hi everyone- we will be relaunching this challenge within a week or two, and we're truly sorry for the false start. Truthfully, we moved too fast and missed a few elements that will make this a much better experience for everyone. What's changing: timing, name, awar
We just shipped Homarr v2.0.0, it includes a LOT of changes. The big two: Custom Widgets and Workshop Custom Widgets v2 lets you build almost any widgets. Build it with JSX, define the options it has and add it to your instance. Write them yourself, or describe what you want and
Hi folks, I'm a long-time lurker and big fan of this subreddit and a massive self-hosting fan (also outside of AI). I doubt many people will disagree with me here because I see the same arguments being made in many posts. However, I thought it might be interesting to share anyway
Last week, I ran out of tokens on my Claude Code subscription. Again. As someone who codes quite a lot using LLMs (like most of us these days, let’s face it), it got me thinking: Why are we spending so many tokens on programming language syntax designed primarily for humans? Brac
Article · llama.cpp
ggml : add alloc_buffer_n to buffer type interface ( #23671 ) ggml : add alloc_buffer_n to buffer type interface Add alloc_buffer_n method to ggml_backend_buffer_type_i interface, with a public API ggml_backend_buft_alloc_buffer_n. Default implementation in ggml-backend.cpp handl
MoE speedup, but only for some MoE architectures (like Qwen 35B A3B)
Article · Ollama
Clef decision models Ollama now supports Clef and Clef Flash , Cloudflare's new open-source decision models, through /v1/systemone . Clef (27B) and Clef Flash (9B) are multimodal: requests can now include images alongside the text state, shared by all questions and scored jointly
The results are in, no thinking / instruct mode for new models degrade performance more than on old models such as 3.6 vs 3.8, where 3.6 takes the lead on several coding benches in instruct mode. I would ask the labs to still nicely focus on instruct mode also still, there a prob
Article · MarkTechPost
NVIDIA announced a new 64GB configuration of DGX Spark - from Acer, ASUS, Dell, Gigabyte, HP and MSI - its GB10-powered desktop AI system. It gives developers a way to start with one system for local models and agents, then cluster two 64GB units for 128GB of memory across the cl
Article · Immich
v3.3.0-rc.2 Warning Disclaimer This is a release candidate , not a stable release. It is subject to change and may contain bugs or breaking changes. As always, make sure you have a backup of your database and library before upgrading to this version. Welcome to the third release
I'm currently choosing between IQ3_XXS Qwen-3.8-Flash and Q8_0 27B. This month I don't have anything complex enough to justify either's potential so I've got a fairly bad read on how these two stack up in terms of intelligence. Have any of you compared the two enough to speak to
TLDR it got faster on my AMD 9070 system from 304 to 253 secs for I2V fore more details on how I did the test you can check the video on youtube
**DISCLAIMER** THE PREFILLING TPS SHOWN ON THE PHONE IS COMPUTED ONLY FOR THE LAYERS IT HOLDS. ALREADY FIXING IT TO SHOW END-TO-END PREFILL RATE. NUMBERS BELOW ARE ACCURATE FOR E2E PREFILL RATE. Every file or tool result my agent reads on a 24 GB M4 Pro MacBook is a wait, and 64k
Last update for those who may be following:
Basically a DGX Spark minus the ConnectX-7 ports. I’ve seen expected initial pricing from like $1800 to $2900. Not sure what configurations are actually at those price points. It’s all Internet hearsay until we actually see these things ship, but it’s nice to know that it’s poten
I just spent the last two weeks trying to catch a microcenter in stock with a spark and then today the price went up 30% and now I can't realistically afford it. It was already pretty close to the edge of my budget, and now I don't think I can swing 7k for what I had been expecti
Hi! This is a new benchmark that I created together with other researchers at Meta, Stanford, Harvard, UW. Most benchmarks these days seem to test models to just fix a bug that I as a user already encountered. But shouldn't we expect models by now to also find bugs before anyone
Last month I posted a Qwen3.8-27B LoRA that makes it talk like a person instead of an assistant. It got a lot more attention than I expected: 700+ upvotes, 248 comments and 44k downloads since. I read every comment. People really don't like assistant speak, so its tone of voice r
Hey everyone! Just wrapped up a full music video project generated entirely locally on my setup using ComfyUI and the LTX 2.5 model. Running on 16GB VRAM, the generation speed and consistency were impressive for a local setup. Smooth frames and solid visual output without needing
submitted by
Article · llama.cpp
vulkan: add logging to pipeline compile issues ( #29794 ) Website:
Article URL:
Article · llama.cpp
hexagon: install rebuilt HTP skels ( #29828 ) hexagon: install rebuilt HTP skels Assisted-by: OpenCode hexagon: fix HTP skel catalog dependencies Assisted-by: OpenCode Website:
I'm the developer of Lomorage. It's a self-hosted photo backup system: the server runs on your own hardware, and the phone apps back up over your home network. No cloud, no monthly fee. Why I built it: I got tired of paying iCloud every month for my family's photos, and I didn't
Hello. I've been playing around with local LLMs for a while now, using my 7900xtx. I understand the concepts and usually what to do. But I'm now in a bit of choice paralysis on where to go next. I have a Ryzen 5600X with a 7900XTX and 64GB of DDR 4 (how I wish I had purchased m
Hi everyone, I know this community is amazing at optimizing heavy setups for modest homelabs and home servers. My team is currently developing Horus, a fully local and offline multi-agent application. Our biggest engineering challenge right now is hardware adaptability: we want t
Somebody just released a fully open-source, open-weights decision model that beats Jev! Just kidding, it's me and I this is my first time releasing a public model, recipe and dataset so I am looking forward to learning from the experience. Jev is indeed a very powerful and inexpe
I have been building a somewhat unusual local inference machine around two Huawei Atlas 300I Duo cards. They are relatively inexpensive, passive, dual-accelerator PCIe cards with 96 GB of device memory apiece. They are also absolutely not drop-in CUDA replacements. When I first b
Hi all! I am a former SWE who has recently transitioned into inference engineering. I launched a side hustle a few months back and I've just taken it full time due to excessive demand. Its been such an opportunity because the people optimizing runtime are far fewer than the peopl
Hey local AI community, I've been working on this for a while and finally feel ok sharing it. It's a cyber benchmark where the model gets a shell in an isolated docker box and has to find the exact flag. Pwn, web, crypto, rev, forensics, a few real CVEs and some multi-stage range
What are your thoughts? What are you using it for? Are you excited to try the llm he trained (RL'd I think) specifically within this harness? Thanks in advance!
Hi, i got QFN running on my single r9700 but im not sure if i did everything right to get the best quality and speed out of this setup. Dont want to annoy anybody, maybe someone with the same card can tell me if this looks normal. What i run: - model: Qwen3.8-Flash-Next from turb
Hey everyone 👋 Every time I found a cool workflow on Civitai, the routine was the same: download the zip, unpack it, drag the JSON into ComfyUI, get a wall of red nodes, then spend 20 minutes figuring out which models it needs and which folder each one goes in. So I built **Civit
It’s been a long time since the last models came out. I notice they are selling GLM on the site, and I wonder if they are developing something, given the long silence.
I run models offline with apps like LM Studio and Ollama. The whole point for me is privacy. But how do I actually know nothing leaves my machine? The model file is just weights, but the app around it could phone home. Is there a way to verify this for sure? What do you do? submi
I saw people generating videos with opus 5.5, I tried and it generated a perfect video, So i thought what is stopping me from trying this with free models, and I did, here what is generated from muse spark 1.3, it's not as perfect as opus 5.5 but it did a great job, + it's free a
For the last few months I've been building a "home hub" for my family: shared shopping list (sorted by supermarket aisle), chores in list/kanban, calendar with push reminders, shared expenses with budgets, weekly meal planner that sends ingredients to the list, and a chat assista
Due to some reasons, Qwen/Deepseek Chinese models are NOT allowed in the my workplace. So, what local models, do you think, is the best alternatives to Qwen/Deepseek for (1) Coding (2) Long document QA test (like giving a long medical history of 120k tokens and ask a question bas
Article · LiteLLM
Verify Docker Image Signature All LiteLLM Docker images are signed with cosign . Every release is signed with the same key introduced in commit 0112e53 . Verify using the pinned commit hash (recommended): A commit hash is cryptographically immutable, so this is the strongest way
Just watched Deepseek go into infinite thinking loop over 2-3 specific tokens.. Is this still a thing for frontier models or was it a quantized version that I stumbled upon in busy hours?
Reading another post made me think: Is anybody still using Mirostat? It was all the rage and people said it avoided the “boredom trap” for long texts. Do newer generation models not need that anymore, or are other samplers superior?
Article · n8n
2.41.6 (2026-10-02) Bug Fixes core: Keep the task runner running on an unhandled promise rejection ( #40092 ) ( d719c87 )
Article · MarkTechPost
AWS's Strands Agents team released Strands Decider 2B, an Apache-2.0 decision model built on Qwen3.5-2B-Base. It returns choices, yes/no probabilities and scores with calibrated confidence in one forward pass, never text. It runs at a 115 ms median on an RTX 3090 and scores 0.723
I have 4050 6gb gpu, 24 gb ram which model should i choose to run i need speed. i try qwen 3.8 27b and feel too slow tried from onslot studio. I have heard of weight inferencing does it helpful what should i do to try weight inferencing.
Hey everyone, If you run local models via Ollama in production or personal projects, you've probably run into the hallucination problem: how do you know when a model is hallucinating without burning extra VRAM or waiting 5 seconds for a heavy judge model? The standard academic ap
Article · llama.cpp
common,rpc : fix cache dir creation through symlinks on buggy libstdc++ ( #29816 ) See
Article · llama.cpp
llama : clamp kpool re-pool bound to existing pools ( #29805 ) tests : simplify function signature llama : clamp kpool re-pool bound to existing pools The n_tokens/kpool + n_seqs_unq bound on n_new_g overshoots when a batch fills the whole cache: n_ctx tokens complete exactly n_c
Article · llama.cpp
hexagon: shared strided DMA copy for CPY and CONCAT, any-dim CONCAT via DMA ( #29685 ) hexagon: shared strided DMA copy for CPY and CONCAT, any-dim CONCAT via DMA hex-cpy: various fixes on top of the concat optimizations Removed CONCAT_DMA_MIN_ROW logic, it was broken with 64-bit
GPU: Blackwell MaxQ Model: qwen3.8-27b-nvfp4 (Orcarouter) - Dflash2 - MTP-7 Running this on my Blackwell MaxQ. I can't believe how fucking fast it is compared to llama-server . It easily reaches speeds between 130-180 t/s, even in parallel decoding I hardly see any slowdowns at a
Article · Hugging Face
Article from Hugging Face.
I have 16GB Macbook Air M4. It can run text models upto 12B. It can also run 6 to 7 GB image models, though slow. Can it Run any video generation model ?
Reddit from r/LocalLLaMA.
Article · llama.cpp
server: return HTTP 400 for invalid embedding requests ( #29060 ) Website:
Article · llama.cpp
cuda : route sm70 to the Turing MMVQ nwarps table ( #29753 ) cuda : route sm70 to the Turing MMVQ nwarps table Volta (sm_70) has no MMVQ parameter table of its own and falls through to GENERIC, which launches K-quant batch-1 decode (ncols_dst == 1) at nwarps=4. sm_70 shares TURIN
This is 6 bc-250 ex mining boards with 5 in the asrock 4u12g case they came in. After a lot of testing my current preferred setup is 4 boards running Qwen Next Flash IQ2_XS at 100k context with around 28 tok/s for short generation and 24 tok/s at 50k with around 115 ppt. The othe
I mean , DeepSeek, Kimi, GLM, MiniMax -- the list of Chinese LLM's is such a long freaking list. As American's -- why don't we feel .. a little funny .. about being so far behind? Chinese entrepreneurs are peneuring like crazy and American's .. just obsess on .. what ? update --
Article · llama.cpp
metal : release temporary private transfer buffers ( #29777 ) metal : release temporary private transfer buffers Assisted-by: OpenAI Codex metal : fix order and formatting Co-authored-by: Niklas Wenzel dev@nikwen.de Website:
Article · llama.cpp
webgpu: add bfloat16 support for MUL_MAT/MUL_MAT_ID/GET_ROWS- #29358 ( #29358 ) Website:
Anyone try out Ling-3.1-Flash? Haven’t seen too many open-weights tackle mobile app generation lately with decent success, but it’s another step in the right direction.
Article · llama.cpp
llama : fix invalid assert in recurrent memory ( #29799 ) Website:
Reddit from r/StableDiffusion.
Article · llama.cpp
CUDA: Handle compute type for NVFP4 on cublass path ( #29173 ) CUDA: Handle compute type for NVFP4 on cublass path Signed-off-by: ynankani ynankani@nvidia.com Use BF16 compute type for quantized models if HW allows Signed-off-by: ynankani ynankani@nvidia.com Set acc prec to bf16
My local stack is a couple DGX Sparks, a 5090, and a 4070 TI. This radio site took three local LLMs, and we generate music, voices, and wacky sound effects. I give you
Article · llama.cpp
Qwen4Exp: add MTP ( #29761 ) Qwen4Exp: add MTP remove has_state member, check via ctx_bufs being non-empty consistent naming + less verbose comments cont : clean-up recurrent memory cont : clean-up comments Co-authored-by: Georgi Gerganov ggerganov@gmail.com Website:
Article · Home Assistant
From ever-increasing subscription prices to outages and data harvesting: we don’t like what the cloud has come to represent. So we’re redefining what an online service should be: private, optional, and free from lock-in or walled gardens. That’s why today Nabu Casa is renaming Ho
Still in the process of setting everything up (which may take some ti
Article · AnythingLLM
What's Changed fix(collector): read spreadsheet dates as dates, not serial numbers by @L4XB in #6478 Drop temp param for Anthropic models by @shatfield4 in #6472 Allow Generic OpenAI max tokens of 0 to omit max_tokens from request by @shatfield4 in #6470 fix: support SearchApi en
I am trying to learn about using comfyui with h3 , but I am unsure what to look at and what to improve. This might sound very basic, but I want to understand when to use lora, why it is used, and when it is not needed. Similarly how do you know the problem in the generated video
i deleted the asset but the files are still in the job que
So everyone has been yelling about how I should be using Gufo instead of halogen because it's open source and it's "just as good or better". Checking in on their GitHub (GitHub.com/gufo-org/gufo) got me immediately .. "Qwen 27B Q4: 70.56 tok/s single user, 123 tok/s with 8 users"
I’m the creator of Failure Map, an archive of compact Python debugging tasks. The open release has 20,168 tasks across 254 categories, with standard-library implementations, explicit contracts, failed repair attempts, and executable boundary checks. Three small cases to try: • Du
I use: H3 Krea2 Qwen 2.1 I have: 4080 Super (16 GB VRAM) 32 GB DDR5 (16 GB stick x 2, running at 6000 MT/s) I want to add two more 16 GB sticks, but at lower speed, so in total: 64 GB DDR5 (16 GB stick x 4 , running at 4800 MT/s) What are the actual , real , benefits? The only di
Article · llama.cpp
mtmd: cap max_image to ubatch for non_causal models ( #29773 ) Website:
Hello guys. Recently, there has been a boom in game decomps and recomps thanks to AI. If you look at the r/decomps and r/recomps subreddits, you can see it. They mostly seem to be using Claude or Codex.I wonder if it would be possible to do something similar with a local AI model
Yue2 has trouble pronouncing some words in Brazilian Portuguese
Hello everyone. I've recently ran some experiments comparing DDR4/PCIe4 and DDR5/PCIe5 for AI workstations on a pre-training run, and would like to hear yours thoughts. First of all, and as a summary of my results ( code here:
Article · noted.lol
High-performance cloud VPS hosting powered by AMD EPYC processors and fast NVMe storage. HostHatch provides affordable virtual servers for developers, homelab enthusiasts, and self-hosters, with plans starting at $4 per month.
Article · llama.cpp
meta: clear inactive AllReduce shards with FILL, not SCALE ( #29793 ) Website:
Article · noted.lol
Sign up for PikaPods and get a free $5 welcome credit! Host your favorite open-source apps in the cloud without managing a server.
I've been working on getting the 95.5 GiB Qwen3.8-Flash-Next model to run fast on a single 64GB Mac. In my earlier post , I shared a custom expert-streaming fork of llama.cpp . It worked, but decode capped out around ~23–27 tok/s and slowed down as context grew. Today I'm releasi
Hi! I’m trying to recreate the current “Bad Times” trend completely locally in ComfyUI. So far I tried: LivePortraitKJ with animal models Wan Animate 2 – Motion Transfer I also created an anthropomorphic full-body version of my cat to make the anatomy closer to the human driving
I put an old RTX 3090 to work as a local coding agent with Qwen3.8-27B, llama.cpp, and OpenCode. Here are the setup and results, including what failed. This is a summary of my own blog post, linked below. Setup RTX 3090 24GB, Ryzen 7 5800X, 64GB RAM, Ubuntu. Qwen3.8-27B Q4_K_M we
Article · llama.cpp
jinja : skip copying loop scope unless a loop filter needs it ( #29776 ) Website:
System Info: OS: Windows 10 ComfyUI Version: Windows Portable GPU: 2x RTX 3090 RAM: 64gb DDR4 CPU: i7 10700k Issue: Load Clip node is not giving me the option to switch devices to either one of my GPUs. I see the devices option, the two available options are Default and CPU. Defa
Hi r/LocalLLaMA! We’re researchers at the Institute of Foundation Models (IFM), an AI research lab dedicated to open and independent development of frontier-class foundation models. We recently released K2 Horizon a connected fleet of six fully open models with size ranging from
I just bought this set up. Targeted 5070ti because it's the best price since I have no more funds to go 4090 and 5090 and 5080 with same vram is not worth even in gaming. I would like to ask whats the best video generator model I could possibly use? Is there a model or workflow t
Article · llama.cpp
llama-mmap : avoid a second full-size copy of each tensor with direct-io ( #29749 ) Assisted-by: Claude Co-authored-by: Pranesh Gonegandla pgonegandla@nvidia.com Website:
I ran Altworld's Hemmingway-1 on my M5 Max this week. It's a 27B fine-tune of Qwen3.8-27B built specifically for "human-like" writing, which might be useful for everyday messages, emails, notes, your social accounts, etc. I was especially curious how the output of that model feel
I've now upgraded my system to 64GB of RAM, and from a Ryzen 5 3400GE (APU) to a Ryzen 5 3600 that I had before and now a Radeon RX 580 with 8GB of VRAM. But as I only had 8GB of VRAM I decided to create some video with Minimax H3 by running ComfyUI with the CPU option. I'm now g
New free/trial model dropped on OC. Details are yet to be provided. Could this be deepseek v4.1 pro?
Article · llama.cpp
HIP: avoid treating CDNA as dgx spark for gqa_ratio 20 in fattn_mma dqk 576 ( #29572 ) Website:
Everything I tried refuses basic tests even if they have "0/100 refusal". Anything you use that's good?
Article · llama.cpp
hex-workqueue: fix race condition in seqn getting out of sync with idx_read/write ( #29785 ) Website:
Hello guys! I have been experimenting with qwen 3.8 for a long time and I hadn't been able to get reasonable speed. I am on a 5060Ti 16 GB, and although this gpu can game, I am aware that AI demands more than 16 GB. I am on a Fedora 44, AMD Ryzen 9600x and 16 GB system ram (16 GB
Hi! I’m using ComfyUI with Wan 2.2 14B I2V on a rented Vast.ai RTX PRO 6000 with 96 GB VRAM. I’m animating a photo of me and my wife: a gentle embrace and a brief kiss. Earlier prompts produced strange mouth artifacts and distorted faces. A simpler prompt improved the result cons
Reddit from r/StableDiffusion.
Article · MarkTechPost
Cohere has released Embed 5, a new embedding model family. It targets enterprise search, RAG, and agentic retrieval. The model family ships in 2 tiers. Embed 5 Pro targets maximum retrieval quality. Embed 5 Fast targets latency and cost on the live query path. Both accept text, i
So I haven’t played a computer game in 20 years, and I know nothing about Minecraft, and I definitely prefer classical literature over YouTube culture, but even I have heard about the individual called PewDiePie, for two reasons: His monicker starts with the initials of my own na
It’s been a while since I switched from renting GPUs to running a 5090 of my own, so I ran the numbers on the hours I was generating. For context, I use ComfyUI for image-to-video & short videos. Since getting my own 5090, my usage changed quite a bit, and I’m much more willing t
Maybe this is already known, and I'm just a clueless noob, but if my lora doesn't match the h3 model I'm using I get a terrible slowdown. Like it floods system ram and grinds to a halt. What I mean is, if I train a lora in fizgig using fl2va, and try to use it with a ref2va workf
Bonjour J’aimerais savoir quel générateurs de vidéos en local je peux utiliser avec cette carte J’ai installé Confyi , j’ai réussi à faire fonctionner Flux 2 , Qwen édit mais pour les vidéos c’est la galère Auriez vous des conseils à me donner Je vous remercie d’avance submitted
Hi everyone, I've been working on a side project called MapLocaly to help list and discover nearby services, places, and local resources. The goal of this project is to build a lightweight location-based directory that can be self-hosted or integrated easily. It provides features
anyone try these out?
I know there’s a lot of homelab users frustrated by the lack of precision time gear available to us, network driver issues and of course the cost. I’m working on a standards compliant precision time standard over BLE that so far is easy to adopt yet may require driver hacks for s
In a previous post I shared comparison between Swift1.5 and peculiar-ragdoll's checkpoints. Added the original unsloth Q4_K_XL and ThinkingCap Q4_K_M (they don't offer L or XL) to the comparison. Here are the results over a 69 set of eval questions. All tests are now run at same
pi-llama-skip-reasoning is an extension for the Pi.dev harness that forces a local llama.cpp model to stop reasoning and answer / act immediately. When you are deep into the ctx session and ask 27B a simple question about a fact or need a direct action , the model may still feel
The local models seem powerful enough to be capable of running basic local computer use. This is a computer use agent/harness built via claude code, and powered by qwen3.8 flash next nvfp4. Drawing a simple image of its choice took 12m 1s, but a lot of that time was spent by the
Hybrid search, reranking, query decomposition, and query expansion are often treated as must-haves for good RAG. We wanted to see how much each actually helped, so we tested them. Same model, same embeddings, same documents, across all 824 multi-hop questions in FRAMES. We built
Article · llama.cpp
BLAS : Document AOCL-BLAS build and label the device AOCL-BLAS ( #29640 ) BLAS : Document AOCL-BLAS build and label the device AOCL-BLAS AOCL-Blas : Add an AOCL-BLAS Quick Start and drop the fixed version path AOCL-BLAS doc : Note on ZenDNN Website:
Once china sets its goals for dominating a market it wins. Usually takes many years, but it happens. Can't compare the political will of a country versus profit and loss thinking of a corporation. China will eventually win in the memory market and current memory makers are at an
I've run the benchmark on a RTX 3090, 1024 tokens in / 256 out, concurrency 1–32. If the model fits on vRAM (Gemma-4-26B-A4B, byte-identical GGUF on both engines ): llama.cpp has 2.2–3.2× the throughput and 5–6× faster TTFT. FreeToken 0.1.2 can't keep 4-bit experts in VRAM at all
Generating stuff is fun until you realize you have thousands of images sitting in an output folder and no idea where that one image from a few weeks ago went. I’ve been working on a small local Windows tool to make that whole part less painful. The idea is that you can set up lit
A few days ago I released Jeff-Qwen3.5-0.8B, a small "System 1" model that picks between options you define and returns a calibrated probability for each, in one forward pass. Speed was great on my M4 Max and RTX PRO 6000, but as a general zero-shot classifier it trailed the big
Reddit from r/StableDiffusion.
Had opus 5.5 do the bring up.. super solid results. Using only 4 GPU results in KV of like 120k with image on. 27B TP=4 results in >200 tok/s with dflash, prefill around 2.5-3.5k. Both these are running the nvfp4 Nvidia checkpoints. Found a magic repo that unpacks into fp16 on th
I've been plying the finer points of Singularity H3 in recent days and was astonished to discover that my standard two stage Singularity process that would reliably generate 1 second of Full HD in 10 minutes, for the past few weeks, suddenly took over 20 minutes today! I checked
40 year tech gap? No problem! The Tandy runs DeskMind, a native DOS program. It talks over WiFi (a PicoMEM 2 card with mTCP) to a small Python server on my PC. That server drives Qwen3.8-27B (NInfer on a 5090) and Krea 2 (ComfyUI on a 4090). The 286 never sees JSON, base64 or a P
Hi there! I started my homelab journey about 3 years ago with an ASRock Pro RS Intel H670 and an I5 13500 with 64GB RAM, 1 TB NVME and Proxmox. For storage i use two 4TB NVME SSD on the same machine. They are passed through to a VM running Rockstor NAS which uses them in RAID1 wi
If someone does heavy vibecoding with something like qwen 3.8 27b on a 5080, how much would that cost in electricity. Taking it a step higher to running bigger models (qwen 3.8 flash next , deepseek, glm etc ) on something like a Mac studio, what would that be like? submitted by
Article · llama.cpp
common : add LLM-jp-4.1 Harmony dialect handler ( #29681 ) LLM-jp-4.1 uses the GPT-OSS format, but its tokenizer decodes a space after every special token and parallel tool calls are separated by . The GPT-OSS handler rejects this output, so add a dedicated handler, selected by t
now you can use MTP with Qwen Flash Next, time to switch from Qwen 3.8 27B? (merged after 17h of development) quants:
Momentan hab ich mein ( ThinkPad T490s i7 32ram 512gb Ubuntu ) als Hauptrechner mit zweitmonitor dann ein ( Dell i5 8ram Homelab ) und ja die 2 Router habe ich zum meine Smart Geräte von mein Setup getrennt zu halten ( der weiße Rechner ist ein altes Teil den ich momentan nicht b
Article · llama.cpp
opencl: mark vec subgroup bcast as supproted for Adreno E17 compiler ( #29698 ) Website:
i wanted to see how far i can push my PC and on the way i discoverd the resources are limited so i made my own and i hope it helps .
Article · llama.cpp
vocab : honor BOS/EOS settings for PLaMo-2 and PLaMo-3 ( #29734 ) vocab : honor BOS/EOS settings for PLaMo-2 and PLaMo-3 The original tokenizer configs for PLaMo-2 and PLaMo-3 have add_bos_token: true and add_eos_token: false , but _set_vocab_plamo() did not write the BOS/EOS met
Hey, I’m trying to find a cheaper and faster workflow for MiniMax H3 text-to-video with native audio. My current setup: RunPod L40S 48GB ComfyUI LightX2V MiniMax H3 Turbo 4-step v1.2 1184×672 resolution, around 0.8MP 15 seconds at 24 fps Text-to-video with native audio Video shif
Article · llama.cpp
llama-bench : fix verbosity filter to show GGML_LOG_ERROR ( #28229 ) bench : fix verbosity filter to show GGML_LOG_ERROR ( #28107 ) bench: remove dead variables Website:
Made the hardest instrument to play with Ling 3.1 Flash 560B A25B (coming soon). It's kinda crazy that 'flash' is in the 500B category these days, but hey, it works. Can't wait to see what y'all will do when they release this model.
Every time I tried running local RAG on my own GPU, I ran into the same headache: vector databases eat too much RAM, using an 8B model just to parse text takes forever, and cosine search still hallucinates when the context isn't actually there. I’ve been building Hillock to see i
Hey guys, Looking for an alternative for Claude Cowork : Project Work / Documents Integrations like Notion, Gmail, etc. Tools like Websearch, PDF creation, etc. Came over Eigent (
Article · n8n
2.42.2 (2026-10-01) Bug Fixes core: Stop waiting on Bull job.finished() for queued executions ( #39945 ) ( c7841e5 )
Article · llama.cpp
model : re-enable -sm tensor for qwen4exp ( #28569 ) #27941 disabled -sm tensor for qwen4exp because test-llama-archs asserted on the Meta device once the fixture carried a PLE layer: GGML_ASSERT(ggml_backend_buffer_is_meta(tensor->buffer)) at ggml-backend-meta.cpp:476. With host
Everything started with an old laptop... And then it snowballed into this masterpiece! The 3D model is the KWS v.2 Rack Using my old MSI gf63 8rd laptop as my main Docker host and an Optiplex 7010 with a 12500T for my game servers Also the DAS at the bottom for my Jellyfin media
Optional Bundle architecture : Schedule (session-local delayed / timed / interval reminders) was removed from the default set and made an explicit Optional Bundle. This cleanly separates “installed” from “enabled” and is the first systematic use of the Profile + Bundle model for
![v0.31.0rc3: [Model Runner V2] Support randomized dummy inputs (#58411)](media/404444f15a7043fb.png)
Article · vLLM
Signed-off-by: Robert Shaw robshaw@redhat.com Co-authored-by: Robert Shaw robshaw@redhat.com Co-authored-by: Claude Opus 5.5 noreply@anthropic.com Co-authored-by: mergify[bot] Co-authored-by: Nick Hill nickhill123@gmail.com (cherry picked from commit e5e38ba )
Hello everyone, i’ve just finished a new ComfyUI character sheet workflow where I compare Qwen Image 2.1 vs FLUX.2 Krea Turbo using the same input image. The goal was to see which model could create a useful character sheet while preserving the original character’s identity, deta
Article · LiteLLM
Verify Docker Image Signature All LiteLLM Docker images are signed with cosign . Every release is signed with the same key introduced in commit 0112e53 . Verify using the pinned commit hash (recommended): A commit hash is cryptographically immutable, so this is the strongest way
If you're using MTP with Qwen Flash Next and llama.cpp, you can switch to: quants:
The top-level README doesn't mention this, but you can add a "sampling" key to your strata-iq3_s.json like this: { "sampling": {"temperature": 1.0, "top_p": 0.95, "top_k": 20}, "exe": "/path/to/Strata/engine/strata", "args": [ ... ], ... } From docs/DETAILS.md : The run config's
Article · LiteLLM
Verify Docker Image Signature All LiteLLM Docker images are signed with cosign . Every release is signed with the same key introduced in commit 0112e53 . Verify using the pinned commit hash (recommended): A commit hash is cryptographically immutable, so this is the strongest way
Most self-hosting lists are sorted by app category, which is great for deciding what to run. I kept wanting the other thing: how do people like onedr0p, bjw-s and buroa actually structure the repo, and where in their tree do secrets, storage and Renovate live? So I mapped it. Eac
I was waiting on the blowers for the T4s and posted this before it was finished, other than some braided cable sleeves for the fan wires its pretty much good, I was going to upgrade the CPU, but I'm getting great speeds comparatively to a CPU in my old box that had way more cores
Article · Immich
v3.3.0-rc.1 Warning Disclaimer This is a release candidate , not a stable release. It is subject to change and may contain bugs or breaking changes. As always, make sure you have a backup of your database and library before upgrading to this version. Welcome to the second release
Reddit from r/LocalLLaMA.
Article · llama.cpp
ggml-opencl : replace alloca() with std::vector ( #29765 ) Signed-off-by: Adrien Gallouët angt@huggingface.co Website:
I work with Superlinked and we are looking for competitors in the market for some benchmarking exercise. We have designated three small encoders, each on its own pool, each provisioned for a peak that shows up maybe an hour a day. When I finally measured utilization it was in the
Article · llama.cpp
cuda: guard the iq4_nl dequantize row kernel against short rows ( #29683 ) dequantize_block_iq4_nl writes QK_K values per block, but a row can be shorter than that (an IQ4_NL row is only guaranteed to be a multiple of QK4_NL). Threads whose 32-value sub-block starts at or past k
Example in comments. The stock ComfyUI queue tells you almost nothing about what's in it. A queued run shows the workflow name and that's it: no input images, no settings, no preview of what's rendering right now. You also can't do anything with a single run. No reordering, and y
Article · llama.cpp
Hexagon: optimize ALLREDUCE with support for safe scatter mode ( #29757 ) hex-allreduce: add support for safe scatter mode hex-allreduce: pare down excessive comments hex-allreduce: re-write to remove register spills Co-authored-by: Max Krasnyansky maxk@qti.qualcomm.com Website:
most of the (eyes) of the characters is always a little off so now I'm trying to figure out how to implement the face detailer for the ComfyUI-Impact-pack any suggestions? Nodes from left to right: Load checkpoint Clip set last layer 2x Lora loader stack rgthree positive prompt n
Hey everyone! I’ve been working on ASTRABOX : an arcade interface where you describe a game, the AI builds it, and you can ask for changes by voice while playing. Each game gets its own visuals and gameplay, while a shared runtime handles controllers, scores, player joining, etc.
I've tried different character images, created character-sheets and used of course i2v, I even trained LoRas for Krea2 to achieve my goal. NOTHING comes close to using RefMods. I am really baffled how easy it is to create a so to say "LoRa" of a character with a bunch of images w
Article · llama.cpp
args: fix cli download mmproj arg ( #28977 ) tests: add tests for cli download arg parsing args: fix cli download mmproj arg Website:
vLLM's tensor parallel requires that several of the model architecture numbers be evenly divisible by the number of GPUs selected for tensor parallel. This usually means that you can only use a number of GPUs that is a power of two (2, 4, 8, etc.). I have six GPUs, and I want to
Article · llama.cpp
llama : preserve original batch order for speculative decoding layer inputs ( #29019 ) llama: preserve original batch order for layer inputs Assisted-by: Codex tests: cover layer-input order across KV layouts Assisted-by: Codex tests: exercise layer-input ordering on CUDA devices
Reddit from r/comfyui.
I'm looking to generate high quality videos using input references, and would love to be able to preserve the original color space - whether it be log, ACES, or an HDR space such as P3. I'd also love to be able to work with 16-bit EXRs. I know this is a lot to ask, but has anyone
Unifi u6 or u7 pro - Rachio wifi irrigation control - Home assistant beelink mini - Quectel 5g modem + raspi opnsense wan for house -Mikrotik Hex S bridge to unifi lan - because i had - Tiny m920 has hermes - may make back into linux desktop - 2x m720q TrueNAS Core, 1 ssd, 1 hdd,
Hello all, I wanted to share a little project I vibe-coded for myself that you may find useful. As many people here like to suggest, I wanted to give my small local LLMs access to information to improve their world knowledge. I didn't want to give my LLM free reign searching and
Article · llama.cpp
test-llama-archs : toggle causal_attn to catch graph shape changes ( #29724 ) After the device decode, flip causal_attn off, decode n_ubatch/2 then n_ubatch tokens. Both have the same node count, so a shape that depends on the flag makes the second reallocate at an unchanged grap
I was researching prices on ebay and fed claude a bunch of images of listings. I had it make a chart and thought it would be useful to share.
I have been looking at some local models, but even the smartest ones like GLM5.3 Flash and DSv4 Flash have a hard time creating coherent models in Blender and placing them logically in game engines. Is this something that local models are just too dumb still to do good job at? su
My jank home lab! I am running 25tb of space Intel i7 7700 16 Gig ram.. I would love to run a AI but sadly... not GPU...
I spent this week trying to reach Genjutsu locally, i think i did. Its motion transfer and character swap ( Original not included for copy right ). I didn't have problem with 23 clip no flickering no morphing. And what makes this insane its only 3 passes with 5 passes it will add
We had a Dell B300 in the lab for a few weeks and used it to create two fine tunes of Qwen Flash Next. Victoria (coding and agents) Qwen3.8-Flash-Next cut down by 44% using a paper / technique called REAP: 512 down to 288 per layer. Retrained at 4-bit (NVFP4) afterwards, so it's
I’ve sent several messages and tickets asking to delete my account, but support hardly ever replies. They did respond once, on September 11th, apologizing for the delay and saying they would delete it, but it’s still active to this day.
Article · llama.cpp
llama: properly handle KV on training ( #28520 ) llama: properly handle KV on training improve Website:
Article · llama.cpp
batch: migrate the rest of examples to llama_batch_ext ( #29601 ) migrate the rest test-thread-safety rm common_batch_staged Website:
ComfyUI implementation of SoL-Refiner I had a pretty fragmented day today, so used it as a good excuse to test out the fairly new SoL-Refiner model; I put together a ComfyUI implementation as a custom node, here's the link:
Does anybody know of a good image to vector workflow? I’m running ComfyUI locally and everything I’m finding is API based.
I have a long list of my own domain specific eval questions that I run to validate which models I can rely on: coding, coding (numpy/pandas), data analytics decision making, local RAG, and voice assistant. It's made up of the types of things I'm likely to deal with on the daily.
Article · llama.cpp
glm5-next: give dead indexer slots unique scatter rows ( #29745 ) The sparse indexer mask is built with a set_rows scatter. Padded pools, absent sequences and missing tail cells all pointed to the same n_kv sentinel row, and invisible pools picked by top_k to fill the selection o
Running Qwen flash next of even Qwen 27b dense, I can do any,burning sticking to flash due to its speed, and the kwh cost is at 0.25€ where I live in, ranging from 0.11€ to 0.35€, so I used chatgpt to help me calculate the total kwh consumption on my 7900xtx plus 9800x3D, and wel
Reddit from r/comfyui.
Article · llama.cpp
ggml/gguf : fix integer overflow ( #29384 ) ggml: fix integer overflow guard for zero-element tensors ggml: validate number of elements in tensor to prevent integer overflow ggml: fix error print Website:
Spent today running a same-day, same-harness shootout between Qwen3.8-Flash-Next (oMLX, 182GB oQ8e, MTP) and Laguna-S-2.1 GGUF (LM Studio, 128GB, 8bit) on a Mac Studio M5 Ultra 256GB. Both capped at 262K context, thinking on, unique content per run with zero cached tokens verifie
If not, guess I’ll be the first for these elitedesk.
Article · llama.cpp
cli: exit on stdin EOF and drop the console wide Ctrl+C broadcast ( #29722 ) cli: exit on stdin EOF and drop the console wide Ctrl+C broadcast On Windows the simple input reader sends CTRL_C_EVENT to every process attached to the console when stdin reaches EOF, killing unrelated
I'm part of the BiliBili Index LLM team . We're sharing Index-Translate and its companion models for translating text, documents, and videos. Index-Translate supports 150 text languages , with 2B, 9B, and 35B-A3B (preview) options. You can specify terminology, writing style, and
Article · llama.cpp
mimo : support dflash (convert + feature extraction) ( #29650 ) convert : update to support dflash cont : fix Co-authored-by: Sigbjørn Skjæret sigbjorn.skjaeret@huggingface.co Co-authored-by: Sigbjørn Skjæret sigbjorn.skjaeret@huggingface.co Website:
Reddit from r/LocalLLaMA.
I’ve been testing Marigold V2 since it landed in ComfyUI 0.36 and put together a small setup around the official demo, VRAM requirements and running it on a dedicated GPU for larger workloads. I posted the details in the Marigold V2 Hugging Face discussion here:
Basically the title. Just got myself a R9700(32GB) additionally to my rx7800 (16GB) and im already experimenting with qwen3.8 fn and 27b, but im interested wheter they are any other models compatible and comparable. I want to use it with hermes agent mainly.
The PS5 has 16GB GDDR6 on a 256-bit bus at 448 GB/s, the same bandwidth as a 5060 Ti 16GB. Used ones go for roughly $350-450, and there's a Linux loader for it:
This is one of the hardest project I have tried. Replacing just one person from the video. As well as keeping the interaction such as hug and removing sunglasses and lipsync. All done with custom made agentic AI system for coordinating and QC. Using custom made minimax H3 workflo
Hi folks, I've spent the last couple of months experimenting with Strix Halo and previously I released a proof of concept I called llama-halo-hybrid. I've continued updating it and it now performs very well. The idea is that you can take an R9700, or similar, and place dense part
Article · llama.cpp
jinja : support coerced array attributes ( #29574 ) support coerced array attributes add tests Website:
I've been wanting to redo my current Plex, home lab setup, and network; so wanted to if any of you had input or recommendations on what I should do. Currently what I have running : - old gaming desktop that I've been using for primary PC. I've been running Plex with attached exte
Framework Desktop Framework Desktop DIY Edition (AMD Ryzen™ AI Max 400 Series) 192GB
I need to buy 2x DGX Spark but can't find a seller. Any hope or idea where I can get 2? The price seems to have soared. Local Microcenter had 25+ then 0 the next day. They pulling stock?
Article · llama.cpp
ci : fix Models Backend Check by shortening the hrm_text fixture ( #29744 ) The fixture recycles its two blocks over 8 cache slots, so the fp16 error builds up past the 1e-4 NMSE bound on the Vulkan T4 and WebGPU jobs of Models Backend. Two l-cycles keep every branch of the cycle
Hey guys! So recently i got into using smaller LLM to develop some projects, so my main requirement is agentic coding, LLM i can link it with agent harness to code for me. everywhere i look its been like weeks or months and i feel Qwen 3.8 - 27B is the king in its category. is th
Tired of the hardware sprawl haunting my desk so I built a Ghostbusters containment unit homelab rack to house it all. Node 1 (Services): Intel / ZimaOS / 8GB RAM. (Running Docker containers and trapping ads). Node 2 (Gaming/Daily): AMD Ryzen 7 / 64GB RAM / 1.5TB SSD. Networking:
I made this thread yesterday because ComfyUI would make my whole PC stutter and choke, and just be a bitch to use while a workflow was running. It's also always been pretty damned slow, and you hear things about the performance the Nvidia users get to enjoy and have to feel a lit
Hi everyone, I hope someone out there can help me out please I'm looking for a node that works like a Power LoRA loader NODE, but specifically for text strings. Basically, I want to list multiple photography styles (e.g., analog photography , street photography , glamour photogra
Article · llama.cpp
llama: llama_prefetch_rows ( #29599 ) llama: llama_prefetch_rows llama: support row prefetch on Windows Apply the Windows port contributed by @praneshgo unchanged. Source: #29599 (comment) avoid exposing llama-mmap in model code, route via llama-impl add windows check, only prefe
AlexNet was submitted to ImageNet on Sept 30, 2012. Three things came together: neural networks + GPUs + large datasets like ImageNet. That same basic recipe has gotten us this far with no ceiling in sight It feels right to celebrate it by pushing open and frontier models: Qwen-I
I’m not sure where to begin. A complex set of problems has arisen, and since I’m neither a specialist nor a developer, I’m feeling overwhelmed and struggling to cope. Could any of the knowledgeable people here offer some insight? After updating ComfyUI: 1 When performing continuo
Ling-3.1-flash: ~560B total params, ~25B active/token, up to 1M-token context. Across work, coding & healthcare: 1,673 Elo on GDPVal-AA v2.1, 75.16 on FrontierSWE, and 65.35 on HealthBench Professional.
Reddit from r/comfyui.
Article · llama.cpp
ggml : add BF16 unary, GLU, binary and scale ops (CPU, CUDA) ( #29675 ) ggml : add BF16 unary, GLU, binary and scale ops (CPU, CUDA) ggml-cpu : use per-op _bf16 functions for BF16 unary and GLU ops Assisted-by: Claude Opus 5.5 CUDA: use ggml_cuda_cast in binbcast and unary kernel
Or should I assume it was mainly about trying to stop/ban/regulate China and open models,
I've been closely following the rise of the Strata inference engine and as someone with 28GB VRAM and 32GB RAM I'm itching to buy 32GB more RAM just to use Flash Next. But of course before I make such a financial commitment as a member of the GPU-poor class like myself, first I n
I was perusing huggingface trending and was surprised that this hadn't been posted to localllama. Looks like it's been out for about a week.
My single-server homelab - Proxmox on an i5-11400, 8 guests, ~26 containers, everything self-hosted Everything runs on **one** server, for 2 people and a smart home. Happy to answer any questions! Also tips and further advice is highly appreciated 😄 🖥️ Hardware | **CPU** | Intel
There's so many new techniques like RL, RL LoRA, QLoRA, CPT LoRA. I believe i would have a usecase for them, but i don't know where to learn, youtube is filled with bad quality tutorials if i just search and the good channels (fireship, bycloud) don't cover these as they're quite
ATX-Swift-1.5-Qwen3.8-27B-Uncensored-MTP (GGUF) High-Precision i1-Q5_K_M with True 131k Context on Consumer 24GB GPUs Most benchmarks in the community measure generation speed at trivial context depths (2k to 8k tokens). However, running a 27B parameter model at high quantization
Tiel Coder 35B A3B from Peculiar Ragdoll is just Ornith 1.5 35B A3B with their own "coding focused" imatrix quantization and the sharp chat template built in. But how good really is that quantization? Other quantizers also focus on coding. Perhaps it would be better to just get O
loss curve (taken from the last micro of every step, to explain the variation) some help from gemini 3.8 flash high About 40% of the way done with the initial fine tune. The loss is so spiky because I accidentally used the last loss of each micro, rather than the average of each
Article · llama.cpp
cpu: accept BF16 in src1 of mul_mat ( #28937 ) cpu: accept BF16 in src1 of mul_mat ggml_conv_1d_dw builds its im2col in F32 when the kernel is BF16, then calls ggml_mul_mat(im2col, kernel), which puts F32 in src0 and BF16 in src1. The CPU backend refused that combination, so it w
Reddit from r/StableDiffusion.
Reddit from r/LocalLLM.
The collection includes kernels for more than 200 common ML operations, all of which can run entirely locally in your browser on WebGPU. We're also working to upstream these optimizations to Transformers.js, ONNX Runtime Web, LiteRT.js, and more! Kernels:
Another quick Minimax H3 news and goodies round-up, for those who may have missed some items. -> ComfyUI Portable 0.38.0 is now officially available. This adds Kijai's Minimax H3 pixel-seams fix, which smooths out stepped/blocky pixels by blending them against their composited ne
Thanks to my new hobby [?] of designing and sketching experimental live-show environments in Blender [with the help of GPT Astra] , I ended up testing a pretty interesting workflow: using the stage blockout + identity, motion and visual references to turn the sketch into a live-a
Tasks where an MLLM doesn't write text but only makes a decision, which started with Jev, seem to be getting popular. (This is sometimes called "typed decision".) I thought it would be fun to use inside ComfyUI, so I was looking for something that runs locally, found imajev , and
In this ComfyUI tutorial, I’ll show you Qwen Image 2.1 with more than 20 workflows for image generation and editing. We’ll start with text-to-image, including different image sizes, low VRAM workflows, prompt enhancement, image-to-prompt, transparent PNG generation, and Turbo LoR
I just bought the GPUs after deliberating and debating for over two years. With costs not coming down any time soon and me just wanting to get this massive todo-box ticked, I decided to just YOLO it; the GPUs are the most volatile, followed by RAM, rest seems more or less stable.
Follow-up to my post from about two weeks ago. A lot has changed since then, so I wanted to post an update on where the backend is now. Where the green architectures stand When I posted last time, 7 architectures had verified full training support. Everything is now held to the s
Is there any model or tool that can run locally to detect AI writing? Preferably something more up to date.
I'm a solo dev making a sci-fi horror game and wanted to use ComfyUI for concept art. Most tutorials assumed I already knew the basics, so I asked Claude to build me step-by-step lessons with plain explanations of every node and setting. Sharing them in case they help someone els
Article · llama.cpp
openvino: serve GET_ROWS on a weight view from the base Constant ( #28381 ) openvino: serve GET_ROWS on a weight view from the base Constant Resolve view_src when collecting weight Constants so a view over a quantized weight no longer becomes a dynamic typed Parameter, and fold t
Came across this popular open source inference engine Tensorfold
Hello all! Do you have any recommendations of tools for agent coordination and messaging to get them to collaborate on hard problems? Ideally I would like a heterogeneous swarm, using local models as the workhorse and cloud models for reviewing, coordination, or simply to avoid o
Article · llama.cpp
musa : define CUDA_ARCH for device passes ( #29508 ) The MUSA vendor header never defined CUDA_ARCH , so every architecture test in the shared ggml-cuda sources evaluated to 0. Kernel bodies gated on the architecture therefore compiled to nothing, for example the q8_0 -> f16 dequ
Reddit from r/LocalLLaMA.
I'm part of the Lokutor team that built this. Model: NVIDIA Conformer-CTC Small (13M params, int8). It runs on an ESP32-S3 with 8 MB PSRAM, no GPU or NPU. LibriSpeech WER is 3.7 / 8.2, versus 6.3 / 15.9 for Whisper tiny.en on a laptop. Under real noise (DEMAND: car, kitchen, cafe
Article · llama.cpp
SYCL: reduce tensor allreduce sync with pinned host buffers ( #29604 ) Website:
I know I could ask in some hardware subreddit, but I'm curious to know what people with multiple GPUs and expensive inference setups think about this. I just moved to a new place and here the lights go out pretty frequently. 3 times over the last week, I came back to my computer
Article · llama.cpp
add GLM-5.3-Flash (GLM5-Next) support ( #27773 ) Rebase GLM-Next support onto master, and migrate to llama-memory-hybrid-idx Add initial MTP support Merge branch optimizations. Reduce allocated compute buffer size, speed up long context decode, fla, and slight MTP improvements. R
Article · llama.cpp
ggml-zdnn: impl buffer reset, fix memory leaks ( #29637 ) cont: fix code style Signed-off-by: Aaron Teo aaron.teo1@ibm.com Website:
I have an instance of OWUI setup for family usage. Works well with Gemma 4, however web search extraction was a weak spot, I wanted it to be: 1) Not reliant on paid APis 2) Simple in setup I found OpenSERP and made two PRs, one to OpenSERP itself to make it compatible with OWUI e
now you can use GLM-5.3-Flash on your home computer
Article · llama.cpp
Hexagon f16 activation ops ( #29209 ) hexagon: add F16 support for activation ops (SILU/GELU/GELU_QUICK/GEGLU/SWIGLU) Widens ggml_hexagon_supported_activations() to accept F16 (src0/dst/src1 must agree on type), and adds F16 per-thread worker functions in act-ops.c mirroring the
Hey all! I found 8x 8TB drives for about 75€ per drive. They have on average under 6k power on hours with some as low as 3600. Should I close the deal? Thank you in advance for your input 🙏🏽❤️
Article · LiteLLM
Verify Docker Image Signature All LiteLLM Docker images are signed with cosign . Every release is signed with the same key introduced in commit 0112e53 . Verify using the pinned commit hash (recommended): A commit hash is cryptographically immutable, so this is the strongest way
Article · llama.cpp
gguf : reject tensor size that wraps after padding ( #26979 ) GGML_PAD(nbytes, alignment) wraps to 0 when nbytes is within (alignment - 1) of SIZE_MAX, which silently bypassed the size overflow guard in gguf_init_from_reader. Reject the tensor before padding when nbytes + (alignm
"AREX-2 is a 27B-parameter long-horizon agent model from the Beijing Academy of Artificial Intelligence (BAAI). It learns to improve a solution over multiple test-time rounds: propose, measure, reflect, and revise. AREX-2 is trained on machine-learning and algorithmic-programming
Article · llama.cpp
ggml : check row bounds in get_rows_back ( #29575 ) Signed-off-by: Adrien Gallouët angt@huggingface.co Website:
26 months ago Liang Wenfeng said: "Someone must step onto the frontier." Now they are training their models on Ascend 950
I can't seem to find what's wrong. I'm using Pi for my llama-server and sometimes it just stops processing for some reason and stuck after tool call. Logs seems to think that it's finished its job while Pi thinks it's waiting for a response, so sometimes I have to stop it and tel
Article · llama.cpp
model : support classifier_pooling for rerankers ( #29627 ) model : support classifier_pooling for ModernBERT rerankers Assisted-by: Claude Opus 5.5 model : read classifier pooling type in load_hparams Write classifier.pooling_type from _try_set_pooling_type whenever the config h
Our agent workflow is currently executing a group of 10 vllm api calls within a asyncio.gather we made them share the same prompt until the end where the queries/instruction prompts differ. These calls are hitting our vllm-router/llm-d router with production grade kv cache aware
edit: sorry should be 5070ti There now is unlock for higher memory overclocks called mlock. And apparently the GDDR7 has a lot of headroom:
A small tool I’m building for my H3 workflow: write what you want in natural language, use Ollama or other LLM providers , and let IAMCCS Prompter H3 translate that intent into structured, H3-oriented direction. It can also use an audio transcription automatically when you want a
Links Official changelog GitHub release New Open-Source Model Support Ming Image 0.1 Design : A 6B text-to-image model for text-heavy design layouts, with transparent RGBA output MiniMax-H3 Fun ControlNet Union 2.0 : Combine multiple control conditions in one H3 generation ID-V2V
Ok so for context, I have proxmox running on my homelab with a virtualized Truenas with 3 3tb sata drives already. Now I am not sure if to buy those and upgrade and how to structure the new setup. Currently im leaning to build a new nas with i7 4th gen and those disks and not vir
Article · n8n
1.123.83 (2026-09-30) Bug Fixes Pin cjs-module-lexer 2.2.0 to address review finding ( 76e467d )
Article · llama.cpp
hexagon: optimize concat op ( #29673 ) hex-concat: reduce pkts in gather/transpose hot loop gather directly into dst buffer, use special instruction for gather sync hex-concat: use fastdiv replace calls to sw divide with fastpath hex-concat: optimize DMA-HVX pipeline and add tran
Article · n8n
2.41.4 (2026-09-30) Bug Fixes API: Return an execution when its stored trace context is incomplete ( #39827 ) ( 940f65e ) core: Count a database ping as successful when its reply arrives during event-loop lag ( #39757 ) ( fe3c5bc ) core: Store queue job results only for execution
a made a little fanfic short film in ComfyUI with minimax h3, using my nodepack and workflows for extending and bridging clips:
Article · n8n
2.42.1 (2026-09-30) Bug Fixes API: Emit valid schemas for untyped values in the Public API spec ( #39917 ) ( ed56d85 ) API: Return an execution when its stored trace context is incomplete ( #39836 ) ( e710744 ) editor: Show remaining credits on Assistant banner for Cloud UBB ( #3
Hey everyone , I made WinMind - a Windows MCP server for local/remote agents that drives apps through the Windows UI Automation (accessibility) tree instead of screenshots + pixel clicking. Why I built it Most computer-use agents work roughly like this: Take a screenshot Send ima
best iq quants in the biz, got me gemma 4 26b to run 75tok/s tg and 1500 pp on 2x 4060 8gb using lmstudio serving to hermes, much work has been done.
I think most people are sleeping on this inference engine. I tried multiple llama.cpp forks and none of them comes close to the inference speed of Strata. Initial version had some bugs with kv cache, cpu throttling and the developer fixed them. Inference engine (only runs on Nvid
Article · llama.cpp
ggml-cuda: HIP: optimize packed byte subtraction ( __vsubss4 -> __vsub4 ) ( #29478 ) ggml-cuda: HIP: optimize non-saturating packed byte subtraction ( __vsubss4 ) CI: ignore 1 spilled vgpr in fattn_vec Co-authored-by: Carl Philipp Klemm carl@uvos.xyz Website:
Let me start off by saying I am new to AI and have a lot to learn, basically I don't know shit. I started off a few weeks ago by throwing my 2 5090s I had from gaming pcs into a 9950x cpu with 64gb ddr5 6000 ram on a motherboard that was able to do gen 5 8x per card system. Runni
Reddit from r/LocalLLM.
I don't use music streaming services much, and listen to music from my own local collection. I don't use any local streaming servers like Plex or Navidrome, they wouldn't work for me because I use a dumbphone and play music off of my SD card. I've manually built a bunch of mood b
To save people 30 minutes of research (because they didn't bother documenting this officially at all): The "modelscope" package is now just the library. Doesn't contain a CLI anymore. If you try to install it or update your old CLI package, you get "No executables are provided by
Article · llama.cpp
ci: add zdnn backend build but not test ( #29541 ) ci: add zdnn backend build but not test Signed-off-by: Aaron Teo aaron.teo1@ibm.com ci: attempt to run a ubuntu 26.04 container Signed-off-by: Aaron Teo aaron.teo1@ibm.com ci: set shell to bash Signed-off-by: Aaron Teo aaron.teo1
Apple Foundation Model 3 through Private Cloud Compute becomes directly available in macOS 27.2 through familiar AI clients. For Mac users running local models such as Qwen 3.x or Gemma 4, AFM 3 PCC is a compelling alternative to consider. Local models offer control and fully on-
Article · LiteLLM
Verify Docker Image Signature All LiteLLM Docker images are signed with cosign . Every release is signed with the same key introduced in commit 0112e53 . Verify using the pinned commit hash (recommended): A commit hash is cryptographically immutable, so this is the strongest way
Article · LiteLLM
Verify Docker Image Signature All LiteLLM Docker images are signed with cosign . Every release is signed with the same key introduced in commit 0112e53 . Verify using the pinned commit hash (recommended): A commit hash is cryptographically immutable, so this is the strongest way
Article · LiteLLM
Verify Docker Image Signature All LiteLLM Docker images are signed with cosign . Every release is signed with the same key introduced in commit 0112e53 . Verify using the pinned commit hash (recommended): A commit hash is cryptographically immutable, so this is the strongest way
Article · LiteLLM
Verify Docker Image Signature All LiteLLM Docker images are signed with cosign . Every release is signed with the same key introduced in commit 0112e53 . Verify using the pinned commit hash (recommended): A commit hash is cryptographically immutable, so this is the strongest way
Article · LiteLLM
Verify Docker Image Signature All LiteLLM Docker images are signed with cosign . Every release is signed with the same key introduced in commit 0112e53 . Verify using the pinned commit hash (recommended): A commit hash is cryptographically immutable, so this is the strongest way
Article · llama.cpp
opencl: fix get_tensor for q5_K adreno gemm_nonshuffle kernel ( #29555 ) Website:
Hey guys - half a tutorial/guide, and half an I built this, so I went for resources. This is something that I created for myself recently when I couldn't find any good existing solution. I wrote this post myself, no AI! Probably a fair number of you have seen my posts about fine-
Article · llama.cpp
vulkan: Tune GDN kernel, fix Intel performance ( #29476 ) vulkan: tune GDN shader tune for Intel Website:
Article · Ollama
Decision models Ollama now supports decision models through /v1/systemone , based on TypeSafe’s Jev API . Decision models return choices, probabilities, and scores instead of text. Use them for tasks such as ticket triage, model routing, and content classification. Available mode
Article · llama.cpp
vulkan : Load F32 A matrix 2 at a time when its 2-aligned ( #29254 ) It turns out Intel doesn't particularly like loading F32s one at a time and we already have the _2aliagned load logic in mul_mat_vec, so here we use it. While we do already check all the requirements to load ele
I started mapping the building blocks shared across all the models in audio.cpp. The result ended up being more interesting than I expected. Qwen has become by far the most common language backbone in this collection: 32 audio model families use a Qwen-family architecture, and 20
Article · llama.cpp
vulkan: MOE aware mat_mul_id tile selection ( #29182 ) mut_mul_id selected its matmul tile with total token count. For MoE dispatch grid the true N per workgroup is per-expert rows. At pp128 on Sarvam 30B that is 6, not 128, so the picker took the l-tile for ~6 live rows. Most wo
Anthropic released the GLM article today. Trump is getting very involved. Do you foresee Chinese open weight models getting banned soon?
Not an LLM, but the ternary findings should carry over, and we hadn't seen Sherry-style 3:4 weights run in a browser before. Disclosure: this is our work at Precisit, everything is MIT. What it is A 7.4M-parameter one-pass scorer (the jevlike family): the board goes in, one score
Article · llama.cpp
ggml : fix c++ odr by properly using GGML_COMMON_DECL_CPP ( #29504 ) fix c++ odr by properly using GGML_COMMON_DECL_CPP using actual field rather than macro Co-authored-by: Georgi Gerganov ggerganov@gmail.com Co-authored-by: XZiar xziar@xziar.xziar Co-authored-by: Georgi Gerganov
My ninfer-ext fork of the famous ninfer now supports exl3 , and have released two models as well.
Article · llama.cpp
vocab : keep NORMAL in PLaMo-2 and PLaMo-3 ( #29580 ) vocab : keep NORMAL in PLaMo-2 and PLaMo-3 The PLaMo-2 and PLaMo-3 vocabularies mark as NORMAL. Current EOG token heuristic matched it by text and added its attribute to CONTROL. Skip this heuristic for the PLAMO2 vocab type s
What FOSS/self-hosted tools or AI tools have you found genuinely helpful for students? Also interested in anything non-AI that has helped with studying, notes, organization, research, etc. Curious what you guys actually use or found useful. (written fully by human) submitted by /
Article · llama.cpp
ggml : accumulate f16 dot products in f32 on AVX512-FP16 ( #29545 ) Supersedes #29530 Signed-off-by: Adrien Gallouët angt@huggingface.co Website:
Article · ComfyUI
What's Changed Improve qwen 2.1 KV cache location logic by @rattus128 in #16429 Compile Qwen Image 2.1 transformer blocks by @rattus128 in #16430 Allow model files to contain which attention should be used for each block. by @comfyanonymous in #16419 JsonExtractString can now han
Anyone have a good config they've found for QFN on a A40 (or similar Ampere GPU(s) with 48GB VRAM)? The system the card is in has 128GB of RAM (DDR3); right now it's running 27B but I'm curious if there's a way to move to QFN to maybe get better speed/a little more smarts. TY in
Article · llama.cpp
ggml : require input tensors to be GGML_OP_NONE ( #29647 ) Website:
Trying to see if I can use a local model to generate game assets, or even 3D printer models. Any suggestions? Something that would fit in 64GB of ram, speed is not an issue. Just need it to work well.
We had a Dell B300 in the lab for a few weeks and used it to create two fine tunes of Qwen Flash Next. Victoria (coding and agents) Qwen3.8-Flash-Next cut down by 44% using a paper / technique called REAP: 512 down to 288 per layer. Retrained at 4-bit (NVFP4) afterwards, so it's
Reddit from r/ollama.
We did a study: does post training actually make LLMs funnier? We used open models that publish every stage of post training, so we could compare a base model with the future models it became: Tulu 3 (on Llama 3.1 70B), OLMo 3.1 32B and Qwen2.5. We tracked 11 stages, 100 joke pro
Article · llama.cpp
hexagon: add FP32 GELU_ERF and GEGLU_ERF support ( #29631 ) hexagon: add FP32 GELU_ERF and GEGLU_ERF support hex-erf: reduce register pressure in kernels Website:
TL;DR: I'm a solo dev who wanted a simple, private way to have local LLMs watch my screen and do simple logging/notifying. After a year of building, I released v3.0.0 and my mom was able to use it for the first time and I wanted to say thank you! Hey r/LocalLLaMA , What Can You A
I was optimizing my diet using a cloud AI provider, and I found my Pi agent stuck like this. Looks like I had too many mushrooms lol.
I was just wondering because I see it mentioned in threads here a lot... When people talk about LLMs used for roleplay, that's a euphemism for dirty/sexy chats right? Kind of like how "torrenting linux ISOs" is really just pirating copywritten media. Or are you guys really burnin
Article · Ollama
Add CAPABILITY declarations to Modelfiles and an additive capabilities field to create requests. Preserve declarations across GGUF and safetensors creation, inheritance, and Modelfile export. Require decision capability before scheduling System One requests instead of matching Qw
Like.. yea bro, I knew GLM was cool. Now everyone does.
I know I sound crazy, but i’m using a strix halo and finding that GLM 4.7 flash just runs significantly better than qwen 3.6 35 a3b. But its obviously quite old at this point, i wish we had a new glm that was 4.7 flash sized but is anyone using a model that they have found better
ninfer, dwarfstar, Splash, llamAmpere, gufo, etc. We've all seen them popping up, great tok/s, people loving them. Forks of llama.cpp or another engine, or made from scratch. For better or worse, the list will continue to grow They work so well because they dodge a main difficult
Article · llama.cpp
common : stop accepting draft tokens at EOG ( #29638 ) common : stop accepting draft tokens at EOG cont : remove the test Website:
Downloading now.
Thanks to tcclaviger, vLLM now has expert RAM offloading support ( link ). This makes frontier models much more accessible on a local setup! I was able to run the original DeepSeek-V4-Flash-Vision-Exp on four R9700s. podman run --rm -it \ --init \ --network host \ --ulimit memloc
Article · llama.cpp
server : remove the built-in UI's service worker when the UI is not served ( #29565 ) With --path or --no-ui, /sw.js returned 404, and a 404 does not remove a service worker, so browsers kept showing the cached built-in UI. Serve a worker that unregisters itself, clears its cache
I am pretty dumb in this kinda stuff so please don't blame me for it. My llama serve kept crashing with Cublas errors on first prompt with some models, but my VRAM usage was 3000MB/8k. I chatted with sonnet 5.5 for a bit and it told me it was a CUDA version issue (and made it wor
TL;DR: I'm a solo dev who wanted a simple, private way to have local LLMs watch my screen and do simple logging/notifying. After a year of building, I released v3.0.0 and my mom was able to use it for the first time and I wanted to say thank you! Hey r/LocalLLaMA , What is it use
I ran Swift-1.5-Qwen3.8-27b-oQ8e-mtp through the llm-bench.io a few times today: oMLX on M5 Max 64 GB, thinking on at xhigh, 262k context window. The big difference between Swift 1.5 and the base Qwen3.8 27B is how much it writes. Per full run (agent workflow, code generation, re
Article · llama.cpp
common : use fs::path for config dir ( #29649 ) Signed-off-by: Adrien Gallouët angt@huggingface.co Website:
Here are some inspirational quotes you can put into the comments: God is dead and we killed him I am become death All this for 1.5 tok/s? Sir this is LocalLLaMA not RichPeopleofLocalLLaMA Sweet! A 2TB DDR5-12800 RDIMM kit is going to cost only 2 kidneys and a small micronation's
Hi, so I was working on building a local server for an older online game client a few years ago, and the amount of data I had to synthesize was intense. I ended up shelving the project. I had managed to build a basic login server, sorted out some crypt stuff, but it was just way
Article · Immich
v3.3.0-rc.0 Warning Disclaimer This is a release candidate , not a stable release. It is subject to change and may contain bugs or breaking changes. As always, make sure you have a backup of your database and library before upgrading to this version. Welcome to the first release
Need 3 testers for Proxmox/ZFS/VLAN configs I've finished my book on building a local-first smart home (HP Z440, Proxmox VE 9.2, ZFS rpool + tank, 4 VLANs 10/20/30/40, Frigate, RTX 5060 Ti). Original house build was decommissioned in 2026 so some configs are updated to current re
For example, directly using claude code or code, which is then hooked up to automatically delegate the actual code writing tasks to a local model like qwen 3.8 flash next, to save on cloud usage limits. I’m imagining the loop would be: User writes prompt Claude/codex thinks about
Article · llama.cpp
tests : adjust server string regex to also match m2 utlra results ( #29648 ) Website:
Article · Hugging Face
Article from Hugging Face.
Forlinx Embedded has listed an M.2 AI accelerator card based on Rockchip’s RK1820 and RK1828 processors, providing 20 TOPS of INT8 computing performance and up to 5GB of integrated DRAM. The module uses an M.2 2280 interface and is designed to handle local AI inference, including
Article · llama.cpp
common : add fs_write_atomic() ( #29642 ) Check for buffered write errors when closing downloaded files. Use UTF-8 paths when writing ETag files on Windows. Write in binary mode on Windows. Signed-off-by: Adrien Gallouët angt@huggingface.co Website:
I wanted a setup where I can compare all the harnesses with all the local models. It turned out to be a rabbit hole. For instance, you would not only have to test all the harnesses (being sure that they are well configured), but also all models, with all their flavors, and this f
You may think that jev, OpenClaw or TurboQuant are super cool , but actually the coolest LLM invention happened two years ago As we all know, the best source of reliable information about LLMs is YouTube:
I'm running Qwen 3.8 27B Q4 XS with ~30 t/s decode (no MTP) at 100k context with q8/q5 KV cache on a 16GB AMD GPU. I wanted to share my setup since many believe Qwen 3.8 27B to be infeasible on 16GB VRAM. Build llama.cpp with Vulkan: cmake -B build -DGGML_VULKAN=ON && cmake --bui
IQuest-Q1 is a Mixture-of-Experts (MoE) model developed by IQuest for agentic coding, reasoning, and multi-step tool use. It comprises approximately 320B total parameters , with an estimated 15B parameters activated per token .
I’m building a Polish General purpose legal Model that drafts documents, answers questions using legal sources, and has enough coding ability to handle some automation. The workflow is very tool-heavy: Question → many sequential tool calls → final answer/document Think Claude Cod
We have released Qwen3.8-Flash-Next quantized with GSQ and RCO, together with a second, capability-targeted build in which half of the model's experts have been removed. Flash-Next is a sparse mixture-of-experts model: 512 routed experts per layer across 48 layers, 176.9B paramet
Article · n8n
2.41.3 (2026-09-25) Bug Fixes core: Propagate project span attributes to node spans ( #39456 ) ( 29afe51 )
Article · llama.cpp
vulkan : reuse descriptor sets when bindings are constant ( #29280 ) vulkan : reuse descriptor sets when bindings are constant vulkan : bump buffer_destroy_count before destroying the buffer Website:
Article · n8n
2.42.0 (2026-09-29) Bug Fixes AI Agent Node: Let an AI Agent Tool use its own tools under a pre-v3 parent agent ( #39160 ) ( 60d0598 ) Align MCP workflow headings and remove outdated notice ( #39463 ) ( b096147 ) API: Accept workflow descriptions on creation ( #39705 ) ( 6c1edac
Ornith-1.5-9B-DFlash pairs the Ornith-1.5-9B model with a DFlash draft model for speculative decoding.
First 3 disposition results. Hi there! My name is Jerry and I recently built Enclosure, e deterministic environment for testing LLMs. It's a simulated office enfironment
Article · llama.cpp
chat : fix Muse Glimmer ignoring response_format json_schema with --jinja ( #29615 ) chat : fix Muse Glimmer ignoring response_format json_schema with --jinja Fixes #29613 chat : accept json fences and clean up chat : fix choice parenthesis Co-authored-by: Alde Rojas hello@alde.d
Article · llama.cpp
common : use fs::path for cache dirs ( #29595 ) Avoid useless string conversions on Windows. No need for BSD or emscripten special cases. Signed-off-by: Adrien Gallouët angt@huggingface.co Website:
I created this repo to help the DGX Spark users that have a spare 10-24 GB GPU at home to squeeze some extra memory out of a single Spark or a Sparks cluster. It moves the spec-decode draft model off your Sparks onto that GPU: the freed GB of memory can be used for extra contex
Reddit from r/ollama.
![v0.31.0rc1: [CI/Build] Skip the snapshot runtime on CUDA 12.x images (#59118)](media/485a72695274dcc8.png)
Article · vLLM
Signed-off-by: khluu khluu000@gmail.com Co-authored-by: Claude Opus 5.5 noreply@anthropic.com (cherry picked from commit aedaba8 )
Article · llama.cpp
ci : update the oneAPI toolkit to 2026.1 ( #29273 ) ci : update the oneAPI toolkit to 2026.1 oneDNN is removed from Intel Deep Learning Essentials in 2026.0, so staying on the deep-learning-essentials path would silently lose oneDNN support when the toolkit version is updated. Sw
Here's a metric dashboard giving an idea of the last few days. Been testing with a variety of different agentic coding use-cases, mostly using a pi harness. qwen3.8-flash-next has seriously exceeded my expectations (used
Reddit from r/LocalLLaMA.
Reddit from r/LocalLLM.
Article · llama.cpp
mtmd: fix GCC 15 stringop-overflow in decode_embd_batch ( #29607 ) Website:
Model Developer: NVIDIA Corporation Model Development: Fine-tuned from NVIDIA-Nemotron-3-Ultra-550B-A55B What is Nemotron? NVIDIA Nemotron™ is a family of open models with open weights, training data, and recipes, delivering leading efficiency and accuracy for building specialize
I audited thousands of agent rollouts in DeepSWE-1.1. Over 80% contained reasoning about an imagined grader. Yet no grader/verifier is mentioned in prompts nor accessible to the agents. Agents reasoned things like: " Let me look at the problem from the grader's perspective " and
Article · llama.cpp
server : support typed content (vision/audio/video) input for /v1/embeddings endpoint ( #29556 ) server : support multimodal input for /v1/embeddings (Qwen3-VL-Embedding) Accept the OpenAI-style wrapped content array format for multimodal embedding requests. Each {"content": [...
I know this may have been asked a lot, but I'm not an LLM expert yet (have never set up vllm myself or llama.cpp myself yet) I've only used scripts other people have set up. Is there a simple way / steps to follow to run Qwen 3.8 27b in Windows with my single 3090? I can run it s
I’m looking for feedback from people who actually use a high-memory Mac Studio (M3/M4/M5 Ultra, ideally 192/256GB+) for daily coding with local LLMs , not just benchmarks. Right now, my company pays about $120/month for Claude for work, and I pay $100/month for Codex for personal
Article · llama.cpp
vulkan: include functional header ( #29597 ) Fixes the compile error: no template named 'function' in namespace 'std' Website:
Like everyone, I use 3js for data visualisation. But while semantic clusters are interesting, they don't confer much practical information alone. So I did something very simple: a query spatially re-assembles the nodes, and you can switch them to text. Honestly, it's hard to swin
TL;DR -- You should probably just use base Qwen3.6-35B, as only Occamy-1.0 is competitive with it. Tiel is a major let-down, worse than Ornith. KAT surprises (good), Nex surprises (bad). This post is long. Sorry, lots to cover. I think we all want to see a next-generation small M
Reddit from r/LocalLLM.
Article · llama.cpp
models: pad on the left with ggml_pad_ext ( #29567 ) models: pad on the left with ggml_pad_ext The Parakeet, LFM2-Audio, Granite Speech and Gemma 4 audio encoders build a left padding as a right pad followed by a roll, and DFlash2 concatenates a zero filled block in front of the
Article · llama.cpp
ggml-openvino: mark unaligned batch-stride views unsupported ( #29603 ) Website:
Hi everyone :) The amazing Swift finetunes of Qwen3.8 27B generate much fewer tokens at mostly similar benchmark performance to the original model, while repos like HyperQwen (formerly syv-ai/qwen38-27b-rtx3090) deliver insane TPS on an RTX 3090. To get the best of both worlds, I
Six months ago, a result like this was unthinkable. But now we can say it loud and clear: local models are at the cutting edge, and the gap of just a few months has been confirmed. Personally, I use Qwen-Next 3.8 for complex tasks; today, GPT-Sol-6-High was messing up a project,
Too many hyperactive amateurs are coming in here with "1b model at 832843tok/s!" and hardly any of them have all the info necessary for local runners to evaluate. We need context ladders with perplexity/KLD, hardware specs, model params and quant(s), runtime, tuned runtime parame
Article · Ollama
mlx: bound pull stall retries and let the watchdog interrupt them ( #1 …
I'm curious, I'm a little bored of Qwen 3.8 and how slow it is even with MTP on
Reddit from r/LocalLLaMA.
I bought a B70 off Amazon last week for 1300 and have been playing around with it. Today I checked and it seems like I cannot find a single one online for less than 1600 and most places dont have them in stock anymore? What happened? What drives these sudden price increases? It s
360 A/B runs on GLM 5.3 and GLM 5.3 Flash, max thinking, 5 repeats per cell. Savings up to 70%. The block (shipped to global instructions): ## Thinking discipline 1. Check the request first. In one or two lines, say what is being asked and flag any premise that looks wrong or mis
Other than the obvious self promotion, is there a practical reason people do this that I am missing? There's dozens of llamacpp forks with silly names that are supposedly "optimized" for this or that specific GPU and seem to have zero intention to merge into upstream. Am I missin
Hello everyone! A little while back I posted about LlamAmpere , a fork of Llama.cpp with Ampere-specific improvements (though it is caught up to main and will support other hardware, too). Thank you to everyone that tried it out and shared back their results across the 30xx cards
Article · llama.cpp
batch: migrate speculative, mtmd and server to batch_ext ( #29385 ) adapt common add common_batch wip wip: spec cont common_speculative_process server_batch to use common_batch rm some stale calls Assisted-by: Claude Fable 5.1 migrate mtmd handle imrope, handle return val of add(
No click bait baby I promise - I'm live streaming the training process kinda like MiMo. UPDATE: [ Training is paused for an hour or two] back to training in batches . u/FullOf_Bad_Ideas has pointed out to me I'm burning a ton of compute for nothing on sequence lengths - we'll be
Article · Immich
It's been a while since we last had to sacrifice a release. Let's hope v3.2.3 was a good sacrifice for v3.3, which is right around the corner! Just another small patch that primarily fixes the memory leak people have observed through a dependency update. What's Changed 🐛 Bug fixe
Article · llama.cpp
common : fix HF cache paths on Windows ( #29475 ) Supersedes #29158 Signed-off-by: Adrien Gallouët angt@huggingface.co Website:
Article · llama.cpp
webgpu: Handle unaligned writes in ggml_backend_webgpu_buffer_set_tensor ( #29471 ) Fix: Handle unaligned writes in ggml_backend_webgpu_buffer_set_tensor Clang formatting Website:
Was getting bored trying to squeeze every last t/s out of my local model on my hardware, so I made a tps counter for my fingers instead (with real tokenizers, of course). My best is around 2 t/s. According to the page, that beats a 70B on a laptop CPU and is roughly 76x slower th
Article · llama.cpp
tests : refactor test-recurrent-state-rollback ( #29426 ) tests : use llama_context_ptr in test-recurrent-state-rollback Replace raw llama_context pointers with llama_context_ptr and drop the manual llama_free calls and cleanup lambda. Assisted-by: pi:llama.cpp/MiMo-V2.6-Flash-RL
Some context first. I am process improvement / business consultant and had worked with Fortune 500 companies on improving their processes around refunds, returns, customer support, etc. This entire thing had a lot of complex decision making and generally every decision / conditio
Article · llama.cpp
ggml-cpu: enable tiled flash attention for non-vector-multiple head dims on x86 ( #29423 ) ggml-cpu: enable tiled flash attention for non-vector-multiple head dims on x86 add AVX2 support for masked loading and storing in simd_gemm_ukernel_tail ggml-cpu: fix FA softcap handling f
I feel like I'm just sleep deprived and missing something extremely obvious, but I have been struggling to find a selfhosted way to monitor the http traffic in the same kind of way that Cloudflare's Analytics -> Http Traffic and Security -> Rules pages work. In those panels, I ca
Hi. So....... after toturing my 750 ti 4 gb and 16 gb ram with image gen model .i want continue experiment with agent mode like hermes or opencode, using local model but before go i want ask few question. are big model = good perfomance or small model can do same stuff. what smal
I've been working on Mica, a 4B decision model, and wanted to see how far it could get in actual Minecraft, not a sim. New world, empty inventory, and the goal was a diamond pickaxe. Last time I posted, it got an iron pickaxe. Honestly that took around 20 tries and it was pretty
Having tested both Qwen 3.8 27b and Flash next on RTX 5090 with 96GB RAM, I want to find the middle ground between the two for coding capabilities but not sacrifice decode speed to standstill. I would like decode speed to between 75-100 ideally for fast iterations; otherwise I be
SIGMOID -> SCALE -> hc_post chain by fxgsell · Pull Request #29520 · ggml-org/llama.cpp" title="vulkan: fuse qwen4exp's SCALE -> SIGMOID -> SCALE -> hc_post chain by fxgsell · Pull Request #29520 · ggml-org/llama.cpp" /> Qwen Flash Next speedup for Vulkan people
Article · llama.cpp
HIP: fix template skip for DKQ > 256 mfma kernels ( #29559 ) Website:
Reddit from r/LocalLLM.
Reddit from r/LocalLLaMA.
I've been testing VeriLoop E2 for a bit another benchmaxxed Finetune slop or treasure and wanted to try it in GSQ-RCO as well. So I made a 3-bit GSQ-RCO quant of VeriLoop E2 and compared it with the other two 27B models I was using: Qwen 3.8 27B UD Q4_K_M Qwen 3.8 27B IQ3_XXS GSQ
Article · llama.cpp
metal: support left and circular padding in GGML_OP_PAD ( #29561 ) metal: support left and circular padding in GGML_OP_PAD Align Metal with CPU, CUDA and Vulkan: shift the source coordinates by the left paddings, wrap them around with the same wrap_around when circular, and read
27B-GGUF is a vision-language model (VLM) for Computer Use, built on the Qwen3.8 dense architecture and developed by H Company . Used with the hai-agents harness , it can send screenshots and tool results to the model, then execute its requested clicks, typing, code, and tool cal
Hey guys, I am looking for benchmarks for people running multi node (4 or more) A100 80 GB cards and seeing what results and models they are getting. Something with VLLM and multi users would be very useful. Or if you know of a place I can find such results please let me know! Ap
Article · llama.cpp
context : do not re-reserve the scheduler when toggling causal_attn ( #28751 ) context : do not re-reserve the scheduler when toggling causal_attn llama_context::set_causal_attn() marks the scheduler to do a full re-reserve on every change of the flag. For vision inputs, this fla
I made an impulse buy three weeks ago and shelled out for a brand new ASRock R9700. I saw used listings, and I also had a trip coming up where I could get the card even cheaper. But I decided to still get a brand new one, an official import, even in this economy. Fast forward to
Article · llama.cpp
Enables Windows ARM64 build with MSVC cl.exe ( #28362 ) can reproduce the issue vlad sees fix fma issue drop volatile fix volatile runtime task add arm flag if needed fix hsum compile error fix syntax in quants strengthen sve probing make the syntax fixes one liners remove debug
Article · llama.cpp
tests : fix ggml init ( #29554 ) tests : init ggml for test-recurrent-state-rollback cont : same for test-save-load-state cont : add to test-state-restore-fragmented + add TODOs Website:
Article · LiteLLM
Verify Docker Image Signature All LiteLLM Docker images are signed with cosign . Every release is signed with the same key introduced in commit 0112e53 . Verify using the pinned commit hash (recommended): A commit hash is cryptographically immutable, so this is the strongest way
A couple weeks ago, I thought they were selling for 1200. On eBay now, the cheapest "buy it now" seems to be just shy of 1500. It feels like we're in a game of GPU musical chairs now.
Here's the project. I have nothing to do with it. I'm just an amazed user.
It seems they have read and resolved our problems. X post:
Article · llama.cpp
vulkan: fix wrong results when a mul_mat reads a slice of a larger cache ( #28956 ) vulkan: read the batch stride of an in place src0 from nb[2] A dim01 contiguous tensor can still be a view whose batches are strided by more than ne[1] rows, the first rows of a KV cache for examp
link to original article TLDR: security researcher eddie zhang used a modified+uncensored local qwen 3.8 27b to create an executable capable of dumping LSASS memory for credential harvesting while evading 2 modern EDR security products. this makes me reflect on how cloud provider
MINISFORUM MS-S1 MAX-P495 – Minisforum EU Expected to ship mid october. At that price point, it doesn't make a whole lot of sense in my opinion. I get that ram prices are where they are, i get that it's a newer model of hardware, but twice the price for 50% more ram and ~5-10% mo
I think I need a 3-slot for my two cards. but holy fuck these things are pricey.
I've liked how Muse-Glimmer worked, so I wanted to see if I could produce a smaller "kid" out of it. Ornith's sharp decisions on when to think and which tool to call were the other thing I liked, so Ornith-1.0-9B got to be the policy teacher while the parent wrote the words. No R
Article · LiteLLM
Verify Docker Image Signature All LiteLLM Docker images are signed with cosign . Every release is signed with the same key introduced in commit 0112e53 . Verify using the pinned commit hash (recommended): A commit hash is cryptographically immutable, so this is the strongest way
It had such a long run. It was my first introduction to modern language models. I remember getting slightly excited over it. And now it lives purely in our memories. Arguably what's more infuriating is that they suggest using GPT-5.6 Terra as a replacement. Keep in mind that Babb
Apple Silicon Macs on macOS 26+ come with a small LLM built in. No download, no API key, and nothing leaves your Mac. Why I built it I was making a tool that writes API docs from code, and I didn't want users to install Ollama or paste an API key. Apple's model was already on the
Basically the idea is take your favorite model, for example qwen3.8-27b or say dsv4vision. Strip everything out that is not needed by that model so the only thing needed is just for the model. Optimize the remaining code to be fast. The idea is to have a model also do this, provi
Using an asrock 12 unit case running one board as the main with the rest of them headless. About 71GB of vram exposed. So far 40 tok/s is with 30k context and it dips to around 30 tok/s at 100k context. This is all over the 1gb Ethernet that is on the boards already. I have 2 mor
Article · MarkTechPost
TypeSafe AI's Jev skips text generation and returns typed decisions with calibrated probabilities. Input costs $0.042 per million tokens and output is free. We verified 20 agentic use cases, from model routing and tool-call gating to reranking and injection screening, and compare
Recently started using the MacOS menu bar app for llama.cpp available at
I had a theory that ram speed didn't really matter and the only thing that matters is your GPU capacity, vram speed, vram size, and system ram size instead of ram speed or cpu speed. I think I've been vindicated. I used unsloth/Qwen3.8-27B-GGUF:UD-IQ3_XXS (10.9gb)
Someone prompted different LLMs to generate CAD code for a bridge under fixed constraints (2-foot span, under 500g filament, 18-hour print limit), printed them, and load-tested them to failure. The results were quite varied: some models couldn't even design parts that fit togethe
Hello, I created a calibration free quant method called TQ. It doesn't need any data so models could be quantized as soon as they come out and the quantized models would generalize better. It performs closely to calibration-based methods. For example, for Qwen 3.8 27B the 4-bit T
It turns decisions — routing, triage, yes/no calls — into typed outputs from a small model instead of generated text, with a routing-only CLI, triage presets, and an abstention gate when confidence falls below a threshold. I ran through the tutorial on CPU end to end, including a
EDIT: About that, I tried it out, and it's garbage so far. I did some basic tests through AIHubMix (do not use that platform btw, it's trash), and my agent did some comparison. I guess it figures, it was a model they released just days after the Jev hype started. The agent's analysis
Been doing a bunch of vibe coding lately. Had my agents host a private WoW server for me, then built out a web browser client so you can play without installing the game and it has mobile controls. Afterwards, created a custom mcp to drive the client and have finer game control t
We built an inference engine InferredThoughts for MoE models that don't fit in VRAM + RAM. Most of the model stays on the SSD, and experts are read as tokens need them. This started as a proof of concept, and poc worked, we are getting 9-10 tok/s decode on Qwen3.8-Flash-Next NVFP
Just recently stubled upon with this modell:LuffyTheFox/Swift-Qwen3.8-27B-Genesis-GGUF I'm just stay away from "magic" models, but this model size got my eyes on: With vision capabilities this is under 17GB, which means i can use it 32GB vram with full Context size (262k), bigger
Article · llama.cpp
server : allow RANK pooling batch splitting for causal LLM rerankers (ie. Qwen3 and Qwen3-VL) ( #28876 ) server : allow splitting RANK pooling for causal LLM rerankers Rerank models fall into two categories: bidirectional cross-encoders (BERT, etc.) that require all tokens in a s
It’s probably a bit niche and the models are a bit old at this point, but after reading about Ninfer a few months ago I did my own small vibe coded engine project for my 5080 Laptop GPU. Initially I thought I can only fit the 12B model with enough context into the VRAM, but with
Plan runs settled style/scene/size in one Q&A round, then wrote the whole SVG in a single call (11.1KB Flash, 22.1KB Pro) and batched every render fix into one edit round. That's 12 and 17 calls total. No-plan runs iterated more: Flash did 3 render-fix rounds and lost ~10 calls t
Just made this post for those who missed it :
Hello everyone, I made a calculator for the theoretical HW roofline for decoding / prefill based on several parameters (LLM model architecture, quants, GPU, Memory, ..). It still a theoretical bound, but helpful as a step-0 check to understand what fits (would fit) in your hardwa
Hey guys! I have been excited to share this here. This is a project consisting of kernel optimizations for the Tesla p100 series graphics card ($80). I want to start by saying I am 17 years old and do not have a formal degree. I used Ai for a lot of this and while I understand so
I will give a concrete example of the selfhosted tool I use the most, my note taking app TriliumNext. I liked trilium in the past. I even switched to triliumNext when the project was archived. But man when I checked the top contributor list and I saw claude as top 2, it made feel
Finally got 3.8-Flash-Next running on my M4Pro 48GB Mac with
I've been building Strix Llama , a patched llama.cpp runtime plus a desktop app for one specific combination: AMD Strix Halo (Ryzen AI Max+ 395, 128 GB) running Qwen3.8-Flash-Next (125B total, ~6B active, Unsloth's UD-IQ4_XS, 94 GB), on Windows . No Linux, no ROCm install, no bui
There is always at least 1+GB of VRAM not usable not matter how I set the --tensor-split (-ts) param. I tiny shift toward one side will move the weight significantly to the other side. 😵💫 Adjusting context will increase/decrease usage on both side. --tensor-split 499,501 = GPU1
I've spent a lot of time building out my servers b
Prompt: Create an animation showing how the soap dispenser mechanism works in one complete html file. Results: Opus 5.5 - High:
Reddit from r/LocalLLaMA.
Article · noted.lol
Crumb is a clean, dark-mode Ghost theme built for sharing recipes, organizing collections, and making cooking a little simpler.
I am new to all this so still a lot to learn. If I give Qwen3.8 27B a coding task to fix some bugs in some code, it went through and found and fixed several in like 5 or 10 minutes. I gave qwen flash next the same task and 2.5 hours later it was done. 27B is of course faster over
I built this, so the disclosure goes up front. It's a free, public MCP server plus a REST API with Canadian privacy law data. The MCP endpoint is at
Early this year when I was first looking at building up my inference capability you could get the 16GB Tesla P100s for between $60 and $80. Asked claude about it, told me absolutely not worth it. No tensor cores, bad int4/int8, no BF16, not worth it. Needs special power accommoda
Hi! I'm the author of MoEspresso, which is my way of putting my own ideas about inference engines to the test. A lot of the fun has been trying different design choices, measuring what happens, and finding that several of them work well together. MoEspresso 3 runs Qwen3.8-Flash-N
Another quick Minimax H3 news and goodies round-up, for those who may have missed some items. -> Kijai proposed a ComfyUI code-fix for tile seam artifacts in Minimax H3 output. This fix was officially merged four days ago.
Article · llama.cpp
common : avoid side effects around params parsing ( #29537 ) register --rpc unconditionally and call llama_supports_rpc() only from its handler print server "initialization ..." log after args are parsed Assisted-by: pi:llama.cpp/MiMo-V2.6-Flash-RL Website:
I'm looking to replace my ChatGPT and Gemini usage with local inference, I've already been playing around a lot on my 8GB VRAM laptop but it's just not enough to run Qwen3.8 at reasonable speeds and larger MoE models slow down my laptop which makes it a hastle to use local LLMs r
Upgraded from 3x RTX 3090s to 2x RTX 5090s on my homelab server and picked up a solid speed jump on top of it from a software update (speculative decoding + NVFP4). Setup: llama.cpp (build b11216), running Qwen3.8-27B (abliterated, Q8_0). Blue = old 3090 setup, green = new 5090 s
Made locally with MiniMax H3 + Ultimate SD Upscale + GIMM VFI.
I have hp z8 g4 with 512 ram and 1x3090 1x5060 16gb. has anyone made the transition from llama cpp to vllm recently? is it worth it? docker under windows or full linux install? I am mainly interested in the model support, it seems that many new local models are supported day 0 in
Article · llama.cpp
common : make string_split throw on invalid values ( #29518 ) Signed-off-by: Adrien Gallouët angt@huggingface.co Website:
I am a software developer and I am working on a project where in I can launch ComfyUI pods on RunPod. Doing this would give us access to ComfyUI cloud like infra for cheap. I need someone who can collaborate on this project with me and help me test out the project and also sugges
I've seen a lot of conflicting opinions about MiMo Flash, but I haven't tried it yet. How does it compare with GLM Flash for coding work like this? I'm interested in: - back-end development - debugging, refactoring, implementing features in an existing front-end codebase - mainta
Meta came out with a banger paper
I am using the Deepseek Harness, which has webfetch plugins by default. While it can help browse the internet, I myself have to give it specific URLs to search. But, apparently you can give it the full web browser and search engine capabilities. Unfortunately, it apparently needs
Here is the Python script to concatenate videos together in a train fashion across the screen.
How to shift the ui load to the igpu if possible? when generating it take up all gpu power and the ui is gpu powered and it lag like hell...
Hi friends. I found this nice Asuka wallpaper, and I'd like to try replicating it with ANIMA in ComfyUI. The problem is that ChatGPT, Claude, and Gemini refuse to give me the prompt, arguing that it's Copyright/+18? Some of these AIs say she's doing indecent things... Is there an
Reddit from r/StableDiffusion.
Ten Minutes - Let It Be (1993)
oh, I can't embed or do a preview image with link?
Hey guys! Today, we are releasing SupraTTS-0.1-Beta , a tiny ~29.6M parameters Text-To-Speech model. The audio quality is a bit better than the original Glow-TTS (the architecture our model is using!) while it's keeping the same size. Here are some samples:
Disclosure: I’m one of the authors of VeriLoop E2, a 27B model post-trained from Qwen3.8-27B. The weights are Apache-2.0; the harness we evaluate it in is not open source (details at the bottom). This is a release post, but rather than a benchmark dump I want to talk about the de
Article · llama.cpp
jinja : add support for dict builtin ( #29477 ) add support for dict builtin add tests Website:
Hey Guys, Based on my last post on stills, I thought to experiment the same .char model with Minimax H3. I created a character model sia.char which encodes references, prompts around face, body & cloths. Checkout previous post for full detail. How it works? - Build .Char: You dro
Instead of everyone building another general-purpose model, the community could distill open models into domain specialists—biology, Python, accounting, OCR, and more. Developers could combine these capabilities into local tools: small model + OCR + accounting → local accounting
How's that working out compared to llamaCPP MoE offload? I haven't gotten a chance to run this but it seems to be gaining some traction.
Article · llama.cpp
sycl: FWHT kernels for block widths above 512 ( #29243 ) The SYCL FWHT covers 64 to 512 via the standard butterfly network, plus 384/640/768/1280 via the Kronecker/Paley construction added separately in Hadamard hint can produce (1024, 2048, 4096, 8192); those still fall through
I am running qwen3.8 27b on my dual rtx 3090 (fp8 quant, unquantized cache, 129k context) and I think it works decently well with hermes, opencode etc. But! I am tempted by the new models coming out such as qwen3.8 flash next, deepseek v4 flash, glm 5.3 flash. However, there is a
Article · llama.cpp
CUDA: tune fp16 tile FlashAttention configs for head sizes 40-112 ( #26289 ) Website:
AI grad student here. With all the recent interest in Jev, I thought I would share something I built la few months ago that brings ML models and LLMs together in a different way than Jev does for different use cases.
I don't know what to expect from Qwen 2.1, is it worth trying or not? On my RTX 3060: Flux Klein 9b (convrot), 1 MP, 4 steps: 13s Minimax H3, 1MP, 8 steps: 43s Also, quality wise, Minimax just destroys Flux. But Qwen, I assume, just slightly better than Flux, right? Maybe it at l
I built MEM v3 to solve a frustrating problem in PyTorch: CUDA Out-of-Memory crashes during long training and fine-tuning runs. Instead of restarting when memory spikes or keeping batch sizes overly small just to be safe, MEM acts as a memory governor. It watches VRAM and through
Saw the hundreds of tweets where people just keep asking Opus 5.5 for motion graphic videos. Decided to ask qwen to look at them and make its own. Quite amazing what local can achieve. **EDIT** It looks laggy because of reddits .gif limit btw the full high res version (with sound
Article · llama.cpp
HIP: Enable fattn-mma kernel on cdna for dkq > 256 for large batch sizes ( #28907 ) HIP: Enable fattn-mma kernel on cdna for dkq > 256 for large batch sizes CI: hip-quality-check: ignore spills for very large mfma mma kernels Website:
I’ve extended TensorSharp’s Jev-compatible /v1/systemone endpoint so one decision request can use several kinds of evidence together. For example, an incident triage request can include a written report, a dashboard screenshot, a screen recording, and a caller’s audio clip. Here’
Hi r/selfhosted ! Like many long-time Evernote users, I was frustrated when it grew increasingly bloated, aggressive with paywalls, and slow. I tried Obsidian, but managing sync across devices, dealing with bulky attachment vaults, and missing a classic structured three-pane layo
Article · llama.cpp
vulkan: fix argsort kernel selection for Adreno ( #29469 ) Website:
As per the title. My thinking is that if you run one of these then you would just need a computer with a pcie slot to be up and running. With an nvlink baseboard you need a separate PLX adapter card as well as the mini SAS cables from the baseboard connected up?
Hey. What do y'all think is the best way to upscale images from GPT Image 2.5, while removing the ugly watermark texture-pattern? I could just try using Krea 2 I2I or Qwen2.1's edit capabilities, but I'd rather ask here first if someone has prior experiences with this. The images
The text encoder alone takes 3 minutes on the int8 version. Any startup args or anything that would help? 9070xt, 64gb ddr4
Article · llama.cpp
common : throw instead of abort on grammar without llguidance ( #29516 ) Signed-off-by: Adrien Gallouët angt@huggingface.co Website:
Hardware RTX 5090 (32 GB) + RTX 4070 Ti Super (16 GB) = 48 GB VRAM 32 GB DDR5 6200 RAM Arch Linux, KDE on the 5090 (takes ~1-1.5 GB VRAM) daily driver OrcaRouter Qwen3.8-27B Uncensored, block-FP8 (28.75 GiB), on vLLM 0.30.0 with pipeline parallel across both cards 4070 = rank 0 w
I am using the official ref2va workflow. I wanted to use a dancing video as reference for a character in a reference image. I added a Load Video node and tried to connect to ref_video_audio_0 but it failed. Then I tried ref_video_0 and it also failed. I noticed that the type of r
Hellooo, I made an app that extends your image without altering the original and doesn't require a subscription or a human to AI argument. I hope some of you guys can find use for it like I do every time I want a new wallpaper for my desktop :)
This is a simple T2I workflow for qwen 2.1 Download on Civitai All details in download page.
I'm currently using claude code for coding, and ninfer/qwen3.8-27b/hermes for agentic/personal-assistant type tasks. My original plan was to get another 5090 at some point but recent price increases have made that unobtainable at the moment. I have a 5070 TI NIB sitting on a shel
I’m having a hard time finding out which one gives you fastest speed, maximum context with best possible quality. I can run unsloth qwen3.8 27b iq4_xs with 65k q8 kv, context without MTP and vision offloaded to cpu. But also kinda slow for agentic work at like 30ish tok/s )I mean
Article · llama.cpp
RPC: use RDMA completion channel to not spin ( #29440 ) RPC: use RDMA completion queue to not spin add TODO for apple RDMA Website:
I run my own LLM while also having a Openai subscription. Also tried DeepSeek (latest flash now). I run Qwen 3.8 flash Next at an amazing speed on my 2x3090 + Ram! But local LLM never did worked for me outside some demos like build me a "3D Mario Game, multistage" which I've been
Every upscaler I tried was good at one thing and bad at another. One kept skin texture and shifted the color. One stayed clean on flat areas and ate the edges. So I made a ComfyUI node that runs up to three of them on the same image. The color stays from the original. Edges take
Hi all, I'm looking for some advice picking models. I'm looking to try a project to ingest quite a large amount of docments into a knowledgebase, allowing me and others to ask questions about the data. I'm thinking of using Open WebUI and oikb for the interface and data ingestion
I’ve just finished testing Qwen-Image-2.1 in ComfyUI , using a custom low-VRAM workflow with the INT8 ConvRot model, ComfyUI Kitchen Attention, and a 4-step LoRA . One of the biggest improvements I noticed is the generation speed: with the 4-step LoRA, the workflow can be around
Article · llama.cpp
llama-bench : fix OOB access of hf_file ( #29515 ) Signed-off-by: Adrien Gallouët angt@huggingface.co Website:
Good day data nerds! I am trying to get more into running models, learning about agentic workflows, and creating my own tools. But as you do (right?) I had to fine tune my current setup. With the way the markets are right now, it only makes sense to make the most out of what I go
I'm using the ComfyUI Desktop version and I keep it constantly updated. For the past couple of weeks, it has stopped offloading to RAM automatically. I have a 5090 and 64 GB of RAM. When I run MiniMax H3 or even Qwen 2.1, sometimes the VRAM gets completely saturated, but ComfyUI
It offers a very basic repl, shell one-liner and full conversation management for application integration. Fully stateless, no config needed, very lightweight (C + libcurl + cjson). Released for Linux (pre-packaged for distros and binaries) and Windows (one exe).
3 visible models are 1T, 311B and 9B, let's dream about two more... ;)
Hey r/comfyui , I've been working for several months on a long-form visual project, adapting a written narrative into a series of illustrated scenes, with a consistent cast of recurring characters. Think storyboard-level consistency, but across hundreds of images. My current setu
I know Qwen is a key player in the local LLM space and has consistently introduced truly impactful technologies—like n-gram in Qwen-Next and the recent Qwen 3.8 27B, which is an amazing local model. However, my question is: why are we seeing fewer small-scale models lately—such a
I am an artist working on a worldbuilding, character design and art toy project, handdrawn using Procreate on iPad with no AI use at all. However, I use AI a lot as a hobby and messing around locally and really enjoy it. I thought I'd test out u/shootthesound 's custom node for C
It seems that the pixel drift is still a problem with Qwen Image 2.1 when editing images. It's especially bad when the edit affects the whole image (for example when doing style change). I tried the Pixel Drift Fix node, but half the time it doesn't work, and when it does it make
Same tip applies to almost any video generation solution.. and that is to think out your scene first. Before writing that long, complicated prompt, consider breaking it up into separate scenes. I've spent all day tweaking a prompt today until I remembered this tactic I used back
Reddit from r/StableDiffusion.
I’ve spent the last couple of days with Opus 5.5 working on a fork of Inco’s excellent and already blazingly fast Splash engine to optimise it for M5 Max chips. Taking liberties and referring to it as Splish. Roughly the opposite direction to u/Erp4759 ’s great M1 port ( Splash o
The only things released in the last couple weeks seem to be bigger models like Qwen, DeepSeek, GLM, etc. Where are the Laguna's, Nemotrons, Olmo's, LongCats, Minimax, etc releases?
Article · llama.cpp
hexagon: support tiled Q4_0 and Q8_0 GET_ROWS ( #29511 ) hexagon: support tiled Q4_0 and Q8_0 GET_ROWS hex-get-rows: fix macros hex-get-rows: use tiled HVX dequantization Assisted-by: OpenCode hex-get-rows: fix register spills and clean up checks for unsupported ops hex-get-rows:
I saw this amazing makima cosplay and wanted to make a cool not-so-ai-looking profile picture with it, her hand pose was amazing so, 30H later here we are
The website has a new 15-second video tab (' Night market ') with all the speed-ups that includes different shots, fast movements, medium, close-up, audio, music.
When Midjourney was a thing, I would put credits in and just mess around with and was amazed at all the things I could create with it. But it just cost a lot and had to stop. And I can't believe there's a local model I can use that can do the same thing. It's not better than Chat
The pink-and-orange Vibes poster goes through two model stages: Ming-Image-0.1-Design 6B makes the finished design, then Ming-Image-0.1-Design-Layer 6B separates it into RGBA assets. The example starts with a 2048 image and uses 1024 output for the layer pass. The reported machin
looking for the best (n)sfw tts model
Article · llama.cpp
hexagon: support for backend sampler ( #29502 ) hex-topk: trying to improve/cleanup the pipeline hex-sampling: add STEP op hex-sampler: add SUM op hex-sampler: update CPY to support sampling cases hex-binary: add support for chunking to handle large logits hex-argmax: super basic
A lot of the used cards this sub recommends for cheap VRAM are older architectures, and newer CUDA and ROCm releases no longer target some of them. The cards still work fine. You just need to pin the right toolkit version, and most guides don't say which one. So I went through bo
Article · KoboldCpp
kobold_agent.mp4 PSA: KoboldCpp needs YOUR help to fight a fake phishing site (click here for info) ! NEW: Added an integrated KoboldCpp Agent - KoboldCpp now ships with a bundled Agentic harness. This works like an extremely lightweight replacement for tools like Opencode, Codex
Am using an starting frame image to make video, but curious if you can also use a character sheet to help improve character quality?
Looking to get some advice - I normally use LLM’s with a harness (Hermes or Opencode or Hermes + Opencode) Lately, social media has been pushing Bots at me with creators pushing them as the next frontier. I’ve set up Hermes on a VM from scratch and set up a Product Owner, designe
New open source Jev like model for running on local devices from a group called Supersonic Labs. It's a 144M param local non-generative local classifier. From their site : Julia 1 opens our research into compact decision models. It builds on mmBERT-small , a multilingual encoder,
No joke, I think this model was genuinely ahead of its time. It made it possible to create images indistinguishable from photos taken with a real camera. The physics and details still look flawless to me. Honestly, I think we’re getting close to a point where releasing new models
Where to even begin on this one...I guess I should start by acknowledging the risk vs reward for vendors other than nvidia, so: Yes I understand that nvidia are dominant currently on speed (llm and imagegen) and software ecosystem. I am too am hopefuly that software stack support
Any recommendations on an email client with local AI option? Either reasonable pay-once cost (no subs) or free. I found Skim and Emailops on git, both seem to have some development going on with recent releases. However since neither has a community around and isn't verified by g
Insane results from such a small model running locally on a 8GB RTX 5060, although each image takes about 5 minutes or more depending on the steps count and prompt size. I'm using comfyUI and I'm pretty addicted to trying different prompts for different results on Qwen. submitted
Hello everyone! First off, very new to this local A.I. work, but am really enjoying it and learning it. I'm looking to train LoRA but can't get a work flow to function. Can anyone recommend a working .json file? Appreciate it!!
Hey everyone! I've been browsing this subreddit and honestly, I could spend hours looking through the projects, workflows, and advice people share here. But I’ve never actually used ComfyUI, and I think it’s time to start experimenting instead of just reading about it. I’m a vide
This is currently one of the best open-source models I’ve seen so far when it comes to generating great images with extensive, longer text. For a diffusion model, I am really impressed. (Yes here and there some errors - but come on, thats impressive!) Yes, it has weaknesses in ar
Reddit from r/StableDiffusion.
Workflow link Google Drive I've been trying to replicate the workflow from online chatbot sites which have effective, but opaque workflows for creating "selfies", or AI generated images with consistent facial features. The long term goal is to build a custom local AI ecosystem th
Topic. There HAS to be a straightforward way to create clips that are seamless, I see the proof everyday, so why does it seem impossible to get this one thing to work right? Using H3 ref2va, I'm rendering out 5 second clips, and that parts working fine. But I can absolutely not g
As a single model for your convenience. On my RTX 3070M 8GB, once warmed, this distilled model takes 20s to generate a 1024x1024 image. Base model is available as well. LoRAs are supported on both (base model supports them better). Download at:
Made myself an implementation of Spectrum to play with the base model and see how much of a quality boost I can get from CFG and negative prompting (Spoiler: not much). As well try it in IC-lora upscale workflow (here it seems to be more useful). I've collected some of my observa
Maybe you'll like it? I hope I get to use my self-promotion credit a tiny little bit here after being in the community so long haha. I was the top of MLX.fast for a while and remain the winner on chips below M5. If you have capacity to contribute further enhancements I'd love tha
Article · llama.cpp
cuda: support Nemotron 3 Puzzle state size 96 for ssm scan ( #28717 ) Website:
Article · llama.cpp
cuda: add F16 input to the FWHT ( #29096 ) cuda: add F16 input to the FWHT The CUDA FWHT accepts F32 input only. This makes the source type a template parameter, so the kernel reads an F16 source directly instead of requiring a converted copy. The F32 path is unchanged. supports_
I posted an updated GPT-OSS template a couple of months ago , which was based on Unsloth's version . It turns out that both Unsloth's version (and, thus, mine) contain a very serious bug that can degrade the model when chat history is replayed and contains previous reasoning (aka
I maintain TensorSharp , an open-source inference engine. It can now run Qwen-Image 2.1 locally for text-to-image generation and image editing, with support for its LoRA adapters. I’ve added configs for regular style and editing LoRAs, plus accelerated adapters with their own sam
Article · llama.cpp
server : fix wake_fd warning on Windows ( #29479 ) Signed-off-by: Adrien Gallouët angt@huggingface.co Website:
Article URL:
I wish Qwen also released dataset and method to fully train a model ourselves but it is what it is. However, I come here with my stupid question because someone can answer it better. And will the model still be an over thinker of faster inference will make up for that. submitted
Reddit from r/comfyui.
I've been experimenting with building the camera move in Blender before generating the AI video. This example combines a five-second mannequin render with a character reference sheet and a waterfront background image in MiniMax H3's reference-to-video workflow. How I made it: Use
ComfyUI-VideoAnalyzer analyzes an input video and converts it into a detailed MiniMax H3-compatible prompt or an LTX-2.5 natural-language prompt . It uses a VLM to analyze motion, camera movement, subjects, timing, and other visual details, with optional audio transcription and a
Reddit from r/StableDiffusion.
I've been experimenting with building the camera move in Blender before generating the AI video. This example combines a five-second mannequin render with a character reference sheet and a waterfront background image in MiniMax H3's reference-to-video workflow. How I made it: Use
I am new to playing around with local ai and have a ton to learn about it but I was curious, why are they (who is they?) releasing them for free, don't they want you to pay them to use them, why release free models?
Another quick Minimax H3 news and goodies round-up, for those who may have missed some items. -> A 1920s Horror Movies LoRA. Trained on American clips rather than German, judging by the look of the samples. It also appears to know wide views of the mid 1920s New York cityscape. N
I've been experimenting with ways to run MoE models that don't fit comfortably in RAM, and I ended up making Overspill, a disk tier for FreeToken . The basic idea came from looking at how Colibri handles experts across disk/RAM/VRAM so I took inspiration from the general approach
On my M5 Pro 64GB I can comfortably work in an agentic setup with the Qwen3.8 27B model in good quality (Unsloth UD-Q4_K_XL) at a decent speed of 50 t/s. Splash combines optimized kernels, excellent speculative decoding, a well-implemented prefix cache, and mixed-weight support i
About a month ago I was having FOMO and was going to spend coin I don't really have on new graphics cards. Instead of doing that though I decided to spend the money on a new motherboard+cpu and try to utilize my 3060tis I had lying around from my old crypto miner. Yes, I probably
Article · llama.cpp
Revert "Change max context length for auto-fitting with unified KV ( #28849 )" ( #29437 ) This reverts commit b04d4e5 . Website:
Hi, My local AI server consists of 96GB Ddr5 and one rtx 3090. I got plenty of stuff running, like krea2, qwen image 2.1, minimax h3, ltx 2.5, qwen 3.6, qwen 3.8 q4...,got even qwen 3.8 Flash next running. But now I am thinking of what I can utilize the second card for. I am a si
Hi r/selfhosted ! I built DocuVault over the last 5 months because I wanted Obsidian-style documentation sharing for my development team, with grouping for our customers, but with the docs being either simple markdown or the more powerful html. I also wanted it all in git, but gi
Sometimes it’s easy to forget that this sub and others like it are probably the extreme minority when it comes to this hobby. Most people, I would think, don’t use or can’t afford one good GPU, let alone multiple GPUs, Mac Studios, Sparks, Strix Halos, etc. Is the average tech en
Today I present a fine piece of engineering, carefully assembled inside a custom chipboard chassis: the 2400cc Inference Racer , a.k.a. my winter heater. Power comes from two second-hand AORUS RTX 3090 XTREME WATERFORCE cards. One glows a beautiful teal, the other red. I have no
Was gonna test it out but I can't find the RL checkpoint.
For instance whenever a model comes out, what’s the best engine to run it, the best harness and absolute minimum you need to get same or near same re results that the benchmark of that model claims. And whenever a quant from Unsloth guys comes out the guide can either be updated
Which one is better for difficult tasks like web scrapping, coding, using tools? Looking for any benchmarks because i couldn't actually find one after quite some digging
I've been building a tool called konte for making multi-shot AI videos, and this 90-second piece is what I've been using to dogfood it. Generating a single clip is pretty easy these days. What kept annoying me was everything that happens once you have more than a few of them. Wit
Project Name: Akasha License: MIT Runs on: Python 3, local machines, private servers, VPS environments I’ve been building Akasha as a self-hosted semantic runtime for persistent knowledge, agents, and LLM-assisted workflows. The main idea is that the semantic world should belong
If you are interested in a self learning study workspace where you point it at your own documents (PDF, ask questions, and get answers that quote the actual passage rather than a paraphrase. You can then turn what matters into spaced-repetition flashcards. It runs on your machine
Can LLM agents actually get through a day in the life of a normal user? That question got me reading papers on Android agents and mobile benchmarks over the past few months. A few patterns kept showing up: Most benchmarks run on emulators, making real-device metrics difficult to
For those of us who have been around for a while, we witnessed the huge demand for desktops, servers, and specialized appliances in the early 2000s. Everything was hosted in house. Of course, the bottleneck was the Internet. Then the cloud came along, and everything moved from in
Disclosure: I contribute to Gonka, the network below. Ollama covers everything that fits in your VRAM. For the rest — V4-Flash is a 284B MoE at 400K context — four brokers are handing out welcome bonuses: DAHL 100M tokens , Gonka API 10M , GG 1M , Router $20 in credits . Model to
Article · llama.cpp
jinja : implement sameas test ( #29448 ) implement sameas test add tests Website:
I know there is nothing new with what I am saying but I recently started with hermes agent (it’s been a while I wanted to but did not have the time). Qwen3.8fn 6bpw exl3 (from turboderp) on a 6x3090 (I assume lower quants on lower number of gpus work same) gives me around 80-120t
Hey r/LocalLLaMA ! I wanted to share a small project I’ve been working on called BeeNara Why I built this: I was looking for a way to automatically sort my local documents (invoices, letters, contracts) into my personal folders. While local LLMs are amazing, I noticed that smal
Hello r/localllama once again, it's me your kobold concedo Been a few months since I last posted here, and today I have something new I'd like to share. Specifically, KoboldCpp now ships with a built-in integrated KoboldCpp Agent Harness ! I know it's a little late to the game, b
So this guy dropped a "Mini Video Editor" few days ago that actually works really well! I tried couple of "Directors" that popped up recently but they don't really work as I expected. Max I can do is 17 seconds and then my 5090 CUDA graph starts to look like a pine tree forest an
Reddit from r/LocalLLaMA.
Im running a 5090 and 64gb of ram, so im limited on what I can run. I have currently been able to fit the following - Atomic Q4_k_m 4.27bpw @ 31 layers offload Swift IQ4_xs @ 32 layers offload. Im about to try the Unsloth IQ4_xs as well. I could get a "bigger" (non IQ) quant for
Article · llama.cpp
ggml-cpu: tiled mul_mat for k-quants ( #27851 ) Added tiled mul_mat. For each mul_mat_one_chunk, quants are unpacked into (max) 256x256 tiles of int8, one routine per quent. Then microkernel computes 16x16 tiles before writing out 256x256 float reults to main memory. Tests/benche
Article · llama.cpp
opencl: add A8 Q8_0 non-MoE dp4a binary kernel ( #29439 ) Website:
Enjoy! Fucking loving it.
For the last few days, I've been using two computers to run multiple agents. My 4x3090 machine is running Qwen 3.8 27B with parallel=2, so I can run two agents at the same time. My pi instances are running on a machine with 2x3060, running/managing smaller models such as Gemma 26
faster CPU prompt processing: "TL;DR: 3-7x faster CPU mul_mat using VNNI with IMO minimal complexity"
My local prompts take twenty minutes to three hours as of now. Curious for others what are you doing while it’s working?
i feel like comfy eventually consumes more base memory after repeated generation and the output quality drops gradually. pressing "clean VRAM" or "clear node cache" doesn't help. i just close and restart comfy every now and then, am i being paranoid or is this actually recommende
Title says all. Considering the switch.
Article · llama.cpp
hexagon: find software divide calls using binary inspection tool ( #29449 ) hex-scripts: fix table alignment hex-scripts: find sw div calls using binary inspection tool Website:
Hi HN, I am Dimitris. This is a company OS that I built and use to run my business and anyone can install it and self host it for free. Think of it as Claude Code, Cowork and the cloud sessions in one self hosted application. It is a Multi-tenant application by design where many
I've been playing around with Ling 3.0 Tiny, which is an 8 billion parameter model (MoE, 1B active). And I've had a lot of poignant thoughts as a result. Just for fun, I got it running with llama.cpp on an old laptop. This is a laptop from 2017 with a 7th gen i5 and 8 gigs of RAM
you read about gpu's like the Tesla P40, V100, AMD M150 etc that people pick up for cheap. probably many others as well. Most are either server gpu's being phased out or mining discards, right? The problem of course is that whenever someone discovers these, they then make a youtu
Mica v0.1 4B playing a real Minecraft 1.20.4 server. Video attached. How it works - Each step the bot's live game state (inventory, nearby blocks, entities, last result) is written out as text. - Mica scores the candidate commands and picks the next one. It never generates text.
I run a personal always-on agent fleet — eight seats claiming work in parallel from a shared task board, routing across Anthropic, OpenAI, xAI, Google, and a local Ollama floor. This week the spend forensics said $26.41 in 12 hours, $18.95 of it on five Opus 5 calls. The single w
The jump from Qwen3 Coder 30B A3B to current-day Qwen 3.6 35B A3B is crazy, especially with all the fine-tunes, and that was around 6 months (I didn't care for local AI back then, or AI at all, apart from as a toy so I don't know). Is around a year until I will never need cloud w
Since my last post , I've been thinking about different options for dynamic performance degradation, trying to squeeze as much high-quality inference out of my GPU as I can. Over the weekend I read this really interesting paper: Cache-to-Cache: Direct Semantic Communication Betwe
Article · llama.cpp
common,rpc : simplify fs_create_directory_with_parents() ( #29432 ) The original function was broken on Windows for some unicode paths Paths without a trailing separator now create the last directory too, matching the function name. All current callers already include a trailing
Article · llama.cpp
mtmd: fix mel preprocessor in LFM2 audio ( #29403 ) which resulted in different greedy transcripts for 4.5% of English and 6.5% of Japanese test utterances. In Japanese, some differences changed entire words. This change: uses log(x + 2^-24) instead of clamping to the log floor u
Every rent-vs-buy thread I read has confident people on both sides, but not many actually show the numbers. So I finally ran the numbers for our own decision. Posting the working here in case it is useful, or feel free to point it out in case someone thinks it's wrong. An 8-GPU H
I recently started using my Mac Mini M4 as an always-on home server . It stays powered on most of the time, and right now I’m running my Discord bot on it. I’m thinking of gradually turning it into my own little home lab/server and running things like: 🤖 Ollama / local LLMs 🐳 Doc
Hey r/ollama ! I'm Roy, solo dev of Observer. Observer is a free open-source agent that uses small local LLMs to monitor your screen , so you don't have to. This is the demo of the agent that controls the micro-agent framework that you guys helped me build! (Special thanks to Oll
Article · llama.cpp
opencl: add bin kernel kernel_gemm_noshuffle_q5_k_f32_32b_trans_ila_a8_bin , kernel_gemm_noshuffle_q5_k_q8_1_dp4a_ila_a8_bin ( #29401 ) opencl: add A8 Q5_K non-MoE non dp4a + dp4a binary kernel opencl: fix s transpose - s only transposed for bin kernels Co-authored-by: Li He lih@
TL;DR - Swift Flash is a killer model that massively reduces excess reasoning. Try it out! If you haven't seen from my previous comparison posts , I'm a huge fan of the Swift Qwen3.8 models. I've been using 27B since it dropped, and I'm really impressed with the performance and q
Article · llama.cpp
Fixing the vulkan build issue of legacy GLSLC version that has no cooperativeMatrix API support ( #29373 ) ( #29409 ) vulkan : fix build issue of legacy glslc version by adding GGML_VULKAN_COOPMAT_GLSLC_SUPPORT macro check for Intel FA shader compiling vulkan : add preprocess con
We hosted Kev 4B (Jared Palmer's Apache-2.0 fine-tune of Qwen3.5-4B) and ran it side by side with Jev on the same endpoint to see how it compares. We built a fresh set of 362 items published after both models shipped (new arXiv papers, Stack Exchange questions, GitHub issues), wi
ndroid TV with Armbian and use your device as a small server, with Docker support, kernel updates, eMMC/SD/USB storage, and backup tools. ⚠️ This is an independent Armbian-based project, not the official Armbian repository. More details :
We’ve released the complete VeriLoop E2 GGUF precision ladder , from the canonical BF16 reference down to a mixed-precision IQ1_M deployment tier. This post is specifically about the GGUF release and local deployment . The parent VeriLoop E2 model was introduced separately. The g
I’ve been experimenting with whether Qwen3.8-Flash-Next’s pretrained PLE n-gram memory can improve a much smaller Qwen3.5-0.8B model. I trained the 0.8B setup with limited resources, mostly using free Kaggle notebook GPUs. The setup keeps both the Qwen3.5-0.8B backbone and the ro
This keeps happening and has been happening for over a month now. Since ollama has zero support I'm hoping someone else has figured out a work around for the issue. I'm using the latest version of opencode (upgraded this morning) and the public https endpoint. The following is a
I maintain a starter kit where the LLM provider is a runtime-swappable DB row, not a hardcoded constant — OpenAI, Anthropic, or a local Ollama instance. Mostly so I can demo without burning API credits. Sharing the friction points, since "just point it at localhost" understates i
A while back I posted here about a CLI agent I'm building that parses tool calls out of a model's plain-text output instead of trusting native function-calling to work. Finally ran the actual numbers. Setup: qwen2.5-coder:7b and :14b on Ollama, 6 small coding tasks (add a subcomm
Article · n8n
2.40.7 (2026-09-25) Bug Fixes core: Propagate project span attributes to node spans ( #39457 ) ( 59fccb6 )
Article · LiteLLM
Verify Docker Image Signature All LiteLLM Docker images are signed with cosign . Every release is signed with the same key introduced in commit 0112e53 . Verify using the pinned commit hash (recommended): A commit hash is cryptographically immutable, so this is the strongest way
Article · LiteLLM
Verify Docker Image Signature All LiteLLM Docker images are signed with cosign . Every release is signed with the same key introduced in commit 0112e53 . Verify using the pinned commit hash (recommended): A commit hash is cryptographically immutable, so this is the strongest way
Article · LiteLLM
Verify Docker Image Signature All LiteLLM Docker images are signed with cosign . Every release is signed with the same key introduced in commit 0112e53 . Verify using the pinned commit hash (recommended): A commit hash is cryptographically immutable, so this is the strongest way
Article · LiteLLM
Verify Docker Image Signature All LiteLLM Docker images are signed with cosign . Every release is signed with the same key introduced in commit 0112e53 . Verify using the pinned commit hash (recommended): A commit hash is cryptographically immutable, so this is the strongest way
Article · Ollama
Add manifest-list storage so runner-specific manifests can coexist under one tag while preserving existing v1 tags as best-effort downgrade anchors. Show/list/copy/remove/pull/push now understand runner and digest selection and transfer referenced child manifests and layers. Add
I'm considering getting an M6 Mac Mini (32GB RAM) to run headless with Ollama, for local LLM for my self-hosted apps. I currently use OpenRouter for most of this, but really want to stop sending anything out of my network. These are the main apps I'd like to use this for: Frigate
Hi - I want to build a home server for Ollama. I want to connect to the server via my laptop and mobile devices to do research and run agents. What is recommended to get this going ?
Article · noted.lol
Bring back childhood nostalgia with classic games by self-hosting Romm. A tool for organizing and playing ROM based games.
Article · noted.lol
Self-host AdGuard Home or use their public DNS to block ads, trackers, and adult content across all your devices with ease.
Article · noted.lol
Kill "Pantry Amnesia" and stop food waste with EverShelf-a TALL stack inventory system featuring BLE scale integration, AI recipe suggestions, and a dedicated kitchen kiosk mode.
for the first time in a long while, probably last time i felt this way was when i first used openclaw with Opus 4.6 half a year ago it felt so life changing that i have to write this post to share my joy. I had this 5090 32GB VRAM for quite awhile, but it has never truly felt use
I've been running Qwen 3.8 Q4, no MTP, on my 5090. (windows oom if i turn on MTP, GGUF) Then TODAY I randomly found an RTX PRO 6000 sitting on a shelf for under MSRP... so obviously that came home with me. Now I don't know I should actually be running lol. Current system: CPU: AM
Reddit from r/LocalLLM.
I would like to thank everyone for their feedback in the recent post & poll where we asked for feedback on how to slow the deluge of "I made X, because Y" type posts in r/homelab , most of which are AI generated and/or spam. While we felt that that the initial plan we shared was
Been messing around with a slightly weird portable s
Reddit from r/homelab.
I'm in a position where I didn't mean to go down this AI rabbit hole... I was looking into mental cognitions and how the mind is made up and learned that my old RTX 5k could run small models. So that led me to running a dual boot Win/Linux and getting another RTX 8K. I've been us
Article · Ollama
What's Changed Structured outputs on thinking models now apply in a single pass, making them faster and more reliable. Fixed intermittent "model not found" errors with a large local library Fixed the macOS app becoming unresponsive when checking if ChatGPT or Codex is running. Qw
Article · Ollama
We pick up schema fixes for typed dictionary values and short arrays.
Article URL:
Been building some multi-agent pipelines recently and the degradation in direct execution is insane. Whenever I pass a strict operational variable or hypothesis, the LLM triggers this 'Devil's Advocate' subroutine, adding 3 paragraphs of academic nuance just to sound smart, befor
Hey r/Ollama , I've been working on Presentation Forge , an open-source presentation/report generator that supports Ollama for local inference. I originally made it because I'm a university student and we end up making a stupid amount of presentations 😭. For actually important pr
Guys I have a M4-mac air with 24 GB unified memory and I wanna run some local LLM model on my machine Currently I am working on building and RAG application which is mostly and QnA system but I wanna upgrade it to Agentic RAG I have tried using online LLM providers (free-tier) bu
Built this because I got tired of guessing whether a model would fit in my 6GB VRAM before downloading it, and because ollama.com's library doesn't let you sort by size. What it does: - Benchmarks Ollama models across 7 categories (coding, chat, reasoning/math, JSON output, long
I was having trouble searching and analyzing the different models that Ollama has available (
Article · n8n
2.40.6 (2026-09-24) Bug Fixes core: Keep OpenTelemetry export working after a restart when Sentry is enabled ( #39368 ) ( 7720a02 ) core: Retry instance reports that cross the UTC midnight boundary ( #39434 ) ( 5e09a78 ) Features core: Authenticate instance reports with the licen
Article · n8n
2.41.2 (2026-09-24) Bug Fixes core: Keep OpenTelemetry export working after a restart when Sentry is enabled ( #39367 ) ( 2de7d47 ) core: Record live n8n Assistant test runs as verification evidence ( #39383 ) ( 07f2d36 ) core: Retry instance reports that cross the UTC midnight b
I am creating a local AI agent called " Ontelic " for Windows. Ontelic returns to Ollama and also manipulates the actual Chrome. This is not for promotion but to share what we've learned. I'll give GitHub or related links in the comments to those who want them. Our team member's
Article · LiteLLM
Verify Docker Image Signature All LiteLLM Docker images are signed with cosign . Every release is signed with the same key introduced in commit 0112e53 . Verify using the pinned commit hash (recommended): A commit hash is cryptographically immutable, so this is the strongest way
l being done with a local AI, a variant of a Qwen3.8-27B model, on a 3090. Short version: with MINUTES of conversation and discussion, I will have a normalized database of all of my cats veterinary records *AND* a pipeline for ingesting those records *AND
Article · ComfyUI
What's Changed Aimdo 0.5.5 + Auto-detect and enable --fast-disk when the disk is fast (CORE-440) by @rattus128 in
Article · Ollama
What's Changed server: fix intermittent "model not found" errors. by @rick-github in
Article · ComfyUI
What's Changed Add new model blueprints and reorganize subgraph categories by @comfyui-wiki in
Reddit from r/LocalLLaMA.
Article · Ollama
What's Changed GET /api/show now advertises each model's thinking controls and default: Available in the CLI with: ollama show gemma4
Article · Ollama
server: allow registry cross-host redirects among allowlisted hosts (…
Reddit from r/LocalLLaMA.
Reddit from r/LocalLLaMA.
Video · Matt Wolfe
PSA to all the AI companies: Please stop announcing features like they’re live when they’re still rolling out “over the coming weeks.” This happened for like 4 different demos in my recent AI news roundup video... Is it so hard to just make an announcement when I can actually use
Video · Matt Wolfe
Here's the AI News you probably missed from this week. Discover More: 🛠️ Explore AI Tools & News:
Reddit from r/comfyui.
Article · n8n
2.41.1 (2026-09-23) Bug Fixes core: Restore pubsub subscriptions after Redis reconnect and detect half-open subscriber connections ( #39241 ) ( b3196fb ) Features core: Enable Agents by default ( #39328 ) ( d8c49e2 )
![v0.30.1rc0: [ROCm][CI] Add MI355 dense NVFP4 and MoRI kernel mirrors (#58281)](media/73f1bf4d53a57ee9.png)
Article · vLLM
Signed-off-by: Andreas Karatzas akaratza@amd.com Co-authored-by: OpenAI Codex noreply@openai.com
Article · LiteLLM
Verify Docker Image Signature All LiteLLM Docker images are signed with cosign . Every release is signed with the same key introduced in commit 0112e53 . Verify using the pinned commit hash (recommended): A commit hash is cryptographically immutable, so this is the strongest way
Article · LiteLLM
Verify Docker Image Signature All LiteLLM Docker images are signed with cosign . Every release is signed with the same key introduced in commit 0112e53 . Verify using the pinned commit hash (recommended): A commit hash is cryptographically immutable, so this is the strongest way
Article · LiteLLM
Verify Docker Image Signature All LiteLLM Docker images are signed with cosign . Every release is signed with the same key introduced in commit 0112e53 . Verify using the pinned commit hash (recommended): A commit hash is cryptographically immutable, so this is the strongest way
Article · LiteLLM
Verify Docker Image Signature All LiteLLM Docker images are signed with cosign . Every release is signed with the same key introduced in commit 0112e53 . Verify using the pinned commit hash (recommended): A commit hash is cryptographically immutable, so this is the strongest way
Article · LiteLLM
Verify Docker Image Signature All LiteLLM Docker images are signed with cosign . Every release is signed with the same key introduced in commit 0112e53 . Verify using the pinned commit hash (recommended): A commit hash is cryptographically immutable, so this is the strongest way
Article · AnythingLLM
New Features LLMman - OSS is now available as a local LLM provider, replacing Docker Model Runner Google Vertex AI is now available as an LLM provider New generate-image agent skill for creating images from chat You.com is now the default web search provider for richer web-search
Article · n8n
2.41.0 (2026-09-22) Bug Fixes ai-builder: Pass parent ask-user answers into Agent Builder ( #38693 ) ( e149dfa ) ai-builder: Reactive subject name on chip ( #39097 ) ( 233afff ) cli: allow importing workflows and credentials with --userId for existing owner ( #36278 ) ( 0b9c174 )
Article · vLLM
v0.30.0 Highlights This release features 762 commits from 315 contributors (104 new)! New models : DeepSeek-V4.1-Flash ( #56214 , #56228 , #56208 ) with the whole KV stored in MXFP8 through the FlashMLA V4.1 record on SM100 ( #56893 ), DeepGEMM Mega-mHC ( #56962 ), and async Engr
Article · Simon Willison
Last week TypeSafe AI unveiled Jev , their first example of a new category of model that they are calling "System One models" (I'm with Maggie Appleton, I think "decision models" is a better name for these). Jev is an interesting variant on the usual LLM format: it still accepts
Article · Open WebUI
Added 📉 Far smaller slim image. A slim build now comes down at around 175 MB, near enough 89% smaller than the last release, the local models, the packages around them and the tools that installed them all gone from it; what that changes about the way an instance behaves is set o
Article · Paperless-ngx
paperless-ngx 3.2.1 Bug Fixes Fix: only pass --conf to flower when flowerconfig.py exists @bitfoo1 ( #14182 ) Fix: replace stale mail-fetch overlap check with a self-expiring lock @stumpylog ( #14189 ) Fix: bump ocrmypdf to 17.12 to pick up the ligature text-layer fix @stumpylog
Article · Paperless-ngx
paperless-ngx 3.2.0 Features / Enhancements Enhancement (QoL): support deselecting single items from "select all" @shamoon ( #14117 ) Enhancement: Match fuzzy terms in place inside the parsed query @stumpylog ( #14157 ) Enhancement: Match CJK terms through their bigram fields in
Article · Ollama
What's Changed Added first-run setup when running ollama , with options to sign in or continue locally. Setup completion is shared with the desktop app on macOS and Windows. Added ollama://apps to open the desktop app’s Apps page directly on macOS and Windows. Fixed excessive mem
Article · Simon Willison
My comment on Bonsai 2 27B: Near-Lossless Compression in a 9x Smaller Footprint — Hacker News. If you want to try out out the GGUFs from
Article · LocalAI
🎉 LocalAI 4.10.0 Release! 🚀 LocalAI 4.10.0 is out! Twenty-eight days and 280 pull requests. The work landed on three fronts: operating a fleet, feeding it models from private sources, and fixing the backends you depend on. A fleet operations dashboard replaces the flat node list
Article · Ollama
cli: add first-run onboarding shared with the desktop app ( #18495 )
Article · Ollama
The decode loop releases MLX's pool of freed buffers every 256 generated tokens, which is also how often the KV cache grows and drops its previous, smaller buffers. The check fires only when the token count lands exactly on a multiple of 256. Speculative decoding emits several to
![v0.30.0rc1: [Bugfix] Isolate supplemental FlashInfer BF16 autotuning (#57285)](media/11693af04da0bf80.png)
Article · vLLM
Signed-off-by: jiahanc 173873397+jiahanc@users.noreply.github.com Co-authored-by: OpenAI Codex codex@openai.com
Article · Immich
Just another small patch with a fix for the reassign faces feature. What's Changed 🐛 Bug fixes fix: skip faces of other users when reassigning faces by @immich-push-o-matic[bot] in #31586 Full Changelog : v3.2.1...v3.2.2
Article · KoboldCpp
nobold_mtv.mp4 Music generated by @Dampfinchen , clips for music video created entirely in KoboldCpp NEW: Media reference support added for Minimax H3! - You will now be able to attach audio clips and multiple reference images to generate minimax H3 videos. The music video attach
Article · Jellyfin
🚀 Jellyfin Server 12.1 We are pleased to announce the latest stable release of Jellyfin, version 12.1! This minor release brings several bugfixes to improve your Jellyfin experience. As always, please ensure you take a full backup before upgrading! Discuss this release further on
Article · Immich
This patch release addresses various bugs, including: An issue with connection pool exhaustion during sync Problems with the new search modal Not seeing partner shared assets on people page Person merge not working for two named people ... and more. See the full list below. What'
Article · vLLM
v0.29.0 Highlights This release features 594 commits from 277 contributors (91 new)! Model Runner V2 is now the default for all models ( #53183 ), completing the rollout that began with pooling models ( #48290 ). MRV2 also gained CUDA graph memory profiling for KV cache auto-sizi
Article · Dify
Warning Self-hosted deployments using the bundled Weaviate must complete a manual, staged upgrade before starting 1.17.1. The bundled Weaviate server moves from 1.27.0 to 1.39.2 - 12 minor versions - and skipping minors is unsupported . Pulling and restarting can silently and per
Article · ComfyUI
What's Changed [Partner Nodes] chore(Google): drop retiring Veo 2 and Veo 3.0 models by @bigcat88 in #15883 [Partner Nodes] feat(Recraft): add V4 Styles by @bigcat88 in #15903 [Partner Nodes] feat(WAN): add WAN3-Prime model support by @bigcat88 in #15894 Bump comfyui-frontend-pac
Article · vLLM
[Bugfix][Core] Apply dense prefix cache default to hybrid models ( #55 …
Article · Jellyfin
🚀 Release notes for 12.0 Notes on Updating Before upgrading from an earlier version, a full backup of the data directory is strongly recommended, as this release includes database changes that prevent rolling back without a full restore. Direct upgrades from 10.10.7 and 10.11.x t
Article · Immich
v3.2.0-rc.3 Warning Disclaimer This is a release candidate , not a stable release. It is subject to change and may contain bugs or breaking changes. As always, make sure you have a backup of your database and library before upgrading to this version. Welcome to the third release
Article · selfh.st
Is "responsible use of generative AI" in the room with us right now?
Article · Paperless-ngx
paperless-ngx 3.1.3 Bug Fixes Fix: use the header loading indicator on tasks page @shamoon ( #13949 ) Fix: fix load sidebar size animating @shamoon ( #13947 ) Fix: wrap long words without spaces in dropdowns @shamoon ( #13945 ) Fix: tweak tool calling localzation prompt @shamoon
Article · noted.lol
There’s always another self-hosted app waiting to be discovered. Reddit, GitHub, YouTube, Noted, and community recommendations all have their share of hidden gems. Where do you usually find the next app that ends up in your homelab?
Article · Dify
Introducing Dify Agent (Beta) : A New Agent Experience in Dify Warning You should provide Dify Agent services ⚠️ only to trusted, non-malicious users ⚠️ . As many of you know, the shell-based LLM agent paradigm has brought a major leap in agent capabilities and changed how we thi
Article · Dify
Release Note What's Changed UX & UI Enhancements Redesigned landing / onboarding for an easier first-run experience ( #37433 , #37844 , #37800 ) Faster navigation - improved "go to anything" palette and autofocused search inputs ( #32130 , #37175 ) Safer deletes - one-click confi
Article · Dify
What's Changed New Features Tool Multi-Select Input Workflow tool nodes now support multi-select dropdown inputs, allowing users to choose multiple values from a predefined list when configuring tool parameters. ( #39346 ) Workflow Node Locator You can now click a node_id in work
Article · Dify
New Features Agent: E2B Sandbox, Home Snapshots, and Skill Management E2B sandbox backend : agent shell/code execution can now run on E2B cloud sandboxes in addition to the local sandbox. Select the backend with DIFY_AGENT_RUNTIME_BACKEND ; a new docker-compose.e2b.yaml ships the
Article · Paperless-ngx
paperless-ngx 3.1.2 Note This release addresses a security issue ( GHSA-2jhj-xqrq-rmrq ) and is recommended for all users. Our sincere thank you to the community members (and their LLMs) for reporting this. Bug Fixes Fix: fix dark mode select disabled color, ensure disabled curso
Article · Hugging Face
Article from Hugging Face.
Article · Jellyfin
🚀 Jellyfin Server 12.0 RC7 We are pleased to announce the seventh release candidate preview release of Jellyfin 12.0! This is a preview release, intended for those interested in testing 12.0 before its final public release. We welcome testers to help find as many bugs as we can b
Article · Immich
v3.2.0-rc.2 Warning Disclaimer This is a release candidate , not a stable release. It is subject to change and may contain bugs or breaking changes. As always, make sure you have a backup of your database and library before upgrading to this version. Welcome to the second release
Article · Open WebUI
Added ♿ Accessibility mode reaches the menus. Accessibility mode now marks the menu entry you are pointing at and the model already chosen with a stronger background, across the dropdown menus, their submenus, and the model picker together with its filter and compare controls, so
Article · Paperless-ngx
paperless-ngx 3.1.1 Bug Fixes Fix: 3.1.0 llm suggestions remove existing metadata from prompt, dont drop name suggestions @shamoon ( #13866 ) Fix: set global search earlier to avoid awaiting debounce @shamoon ( #13865 ) Fix: responsive sidebar, centralize and make sizes saner @sh
Article · Open WebUI
Added 🖼️ Richer previews for terminal files. Word documents and slide decks produced in the terminal are now previewed as the finished document rather than an approximation, and every document preview gains a page strip down the side with numbered thumbnails you can click to jump
Article · KoboldCpp
Minor fix to assistant gen prefills being triggered incorrectly Fixed a bug where failsafe mode was incorrectly selected Added support for DirectIO model load mode ( --usedirectio ), also mlock and mmap can now be combined. New model support: Qwen3.8-Flash-Next and Ling-3.0-flash
Article · Immich
v3.2.0-rc.1 Warning Disclaimer This is a release candidate , not a stable release. It is subject to change and may contain bugs or breaking changes. As always, make sure you have a backup of your database and library before upgrading to this version. Welcome to the first release
Article · selfh.st
I'll take "things nobody asked for" for $500, Alex
Article · Immich
fix: build release branch images instead of re-tagging main ( #31063 ) (cherry picked from commit 5baf181 ) Co-authored-by: bo0tzz git@bo0tzz.me
Article · AnythingLLM
Activity and Chain of Thought Overhaul We overhauled our UI for long-horizon agentic tasks that show multiple thinking and tool calls to now roll up into a single clean, collapsible UI element. This gives you more screen real estate for the actual response while still keeping the
Article · Paperless-ngx
paperless-ngx 3.1.0 Features / Enhancements Enhancement: Apply AI suggestions workflow action @shamoon ( #13639 ) Enhancement: welcome widget visual tweaks @shamoon ( #13794 ) Enhancement: support using remote OCR engines selectively @shamoon ( #13633 ) Enhancement: websocket hea
Article · Jellyfin
🚀 Jellyfin Server 12.0 RC6 We are pleased to announce the sixth release candidate preview release of Jellyfin 12.0! This is a preview release, intended for those interested in testing 12.0 before its final public release. We welcome testers to help find as many bugs as we can bef
Article · Home Assistant
Our why as an organization is clear: to fight for privacy, choice, and sustainability for smart homes, and for every person who lives in one. But who does live in them? In December 2024, we launched the first Home Assistant survey to find out. Our goal was simple: to make Home As
Article · Open WebUI
Added 🚦 Human in the loop tool approval. Where an administrator has turned it on, you can switch a conversation from letting tools run freely to being asked first, so a model that wants to use a tool stops and waits for you to allow or deny it, one call at a time in a saved conve
Article · Hugging Face
Article from Hugging Face.
Seven years in the making (I admit that I did get a kid at some point which slowed down progress and brought it to a halt quite a few times), I decided to finaly release it, with a rather drastic change to the now (I believe) generous licensing model. Unlike current major players
Article · LocalAI
🎉 LocalAI 4.9.0 Release! 🚀 LocalAI 4.9.0 is out! Thirteen days and 146 pull requests, spent on the parts of LocalAI you touch every day rather than on new engines. Authentication is now deny-by-default, chat gained end-to-end context compression, models and backends each have one
I've come to the realization that my life is busy enough that we could use at least one more moderator on this subreddit. Please consider this a formal request for nominations. Rather than just picking someone myself, I’d like input from the community. If there’s someone here who
ChatOSS is built on Ollama. If you use Ollama, ChatOSS local works out of the box. ChatOSS is a GUI desktop app that has multiple agentic coding apps and a kanban board integrated into coding sessions. There's also a simple way to create your own AI powered apps that can run insi
Article URL:
Article · KoboldCpp
pasta.mp4 NEW : Added support for Video generation and I2V with Minimax H3. Requires 4 files as described in this docs . For ease of use, you can download the MiniMax H3 quick setup .kcppt template here NEW : Added support for Muse Glimmer, jinja templates and tool calling also s
Article · AnythingLLM
Notable Changes Image Generation Now you can generate images via /img when a supported provider is set up . This flow also allows image attachments for edits, followups, or combination prompts if your provider supports image editing. Ability for an agent-tool to generate images i
Article URL:
Article · Home Assistant
You’ve been asking all year, and the wait is over… Mark your calendars, because Community Day 2026 will be held on Saturday, November 7! 🎉
Article · Home Assistant
Back in April during State of the Open Home 2026, we gave you a sneak peek of ESPHome’s very first official product… And after months of hard work alongside the community, the ESPHome Starter Kit is here! 🎉
Article · Jellyfin
🚀 Jellyfin Server 12.0 RC5 We are pleased to announce the fifth release candidate preview release of Jellyfin 12.0! This is a preview release, intended for those interested in testing 12.0 before its final public release. We welcome testers to help find as many bugs as we can bef
Article URL:
Article · LocalAI
What's Changed 👒 Dependencies chore(deps): bump actions/stale from 10.4.0 to 11.0.0 by @dependabot [bot] in #11395 chore(deps): bump actions/checkout from 4 to 7 by @dependabot [bot] in #11396 Other Changes feat(gallery): fall back to mirrors and a cached index when the primary s
Article · LocalAI
What's Changed Other Changes docs(blog): cover the terminal agent in the 4.8 post by @localai-bot in #11372 fix(vram): contain malformed GGUF metadata by @richiejp in #11374 docs: ⬆️ update docs version mudler/LocalAI by @localai-bot in #11377 chore: ⬆️ Update 0xShug0/audio.cpp t
Article URL:
Article · LocalAI
🎉 LocalAI 4.8.0 Release! 🚀 LocalAI 4.8.0 is out! Twenty-two days, 386 pull requests, and three new modalities. This release introduces vllm.cpp , a C++20 engine maintained by the LocalAI team, which began as a vLLM port and now carries its own featureset, shipping as the vllm-cpp
Article · KoboldCpp
Row Split has been removed, selecting it will now default to tensor split. Rocwmma has been removed, matching upstream behavior. Added support for qwen3tts aliases for all voices. Breaking change for RPC to match upstream behavior, allowing llama.cpp RPC server and clients to be
Article · Jellyfin
🚀 Jellyfin Server 12.0 RC4 We are pleased to announce the fourth release candidate preview release of Jellyfin 12.0! This is a preview release, intended for those interested in testing 12.0 before its final public release. We welcome testers to help find as many bugs as we can be
Article · Paperless-ngx
paperless-ngx 3.0.5 Bug Fixes Fix: accept Whoosh-era abbreviated relative-date units (yrs, mos, wks, etc) in search queries @stumpylog ( #13486 ) Fix: fix edit dialog error change detection @shamoon ( #13483 ) Fix: key the AI suggestion cache by model and endpoint @lunetics ( #13
Article · Home Assistant
Something big just happened. As the Open Home Foundation’s Android developer for Home Assistant, I was invited by the European Commission to consult on Android interoperability. On July 16, 2026, the Commission adopted a decision requiring Alphabet to open up eleven Android featu
I've been building this for a few years under the name "Statistics for Strava" but I renamed it to Dreeve recently because of Strava's recently changed API usage terms. They paywalled their API. You point it at your activity files (FIT/TCX/GPX) or connect a Strava account, and it
Article · Immich
Welcome to Immich v3.1.0 ! This release includes several quality of life improvements and another round of bug fixes. Keep reading below for the complete highlights. Highlights Upload wakelock (web) Undo archive (web) Filter assets by server filepath (workflows) Quick navigate to
Democratisation of local AI is key. I've been working on pushing the limits of commercial hardware, squeezing any extra bit possible. My Scientific Agentic AI hareness helped me to reallocate every single bit of it. I rewrote the Kernel, I went down the CUDA rabbit hole until I h
Article · Home Assistant
The Open Home Foundation fights for privacy, choice, and sustainability. These principles are at the heart of everything we do, including how we handle website analytics. Our position is clear: we reject tools that track individuals across the web to monetize their data. Instead,
Hello HN, I built Minute because I wanted searchable meeting notes without sending recordings or transcripts to a cloud service. It captures microphone and optional system audio, transcribes locally with Whisper, and generates summaries, decisions, and action items using a local
Article · Paperless-ngx
paperless-ngx 3.0.4 Bug Fixes Fix: prevent pdfjs highlight scrolling from affecting the entire page @shamoon ( #13355 ) Fix: don't skip OCR/archive for tagged PDFs with no actual text @stumpylog ( #13351 ) Performance: more efficient diacritic normalization in dropdown filtering
Article · Home Assistant
Sound the alarm, because Works with Home Assistant just welcomed our newest partner to the program: FireAvert ! 🎉 Specialists in fire safety, FireAvert are on a mission to keep you and your home out of harm’s way. They bring the very first gas and electric appliance shutoff devic
Article URL:
TLDR, Let's Seal gives the finger to Adobe and every doc signing tool (docusign, google, etc) who pay to play with the Adobe Approved Trust List and then charge you for something that should be free. Currently even the person checking if a document/contract is sealed or code is a
Article · Open WebUI
Added 🎨 Redesigned interface. Open WebUI has been visually rebuilt from the ground up. All aspects of the User Interface, from the chat view to the admin panel. Now with a narrower conversation column, lighter typography, tidier spacing, consistent menus and dropdowns, clearly ou
Article · Paperless-ngx
paperless-ngx 3.0.3 Bug Fixes Fixhancement: PAPERLESS_ALLAUTH_TRUSTED_PROXY_COUNT @shamoon ( #13281 ) Fix: preserve document fields during Gotenberg conversion to PDF @stumpylog ( #13271 ) Fix: Makes the email date aware as soon as possible during parsing @stumpylog ( #13266 ) Fi
Article URL:
Article · selfh.st
Jellyfin and Codeberg hurt themselves in their confusion
Article · Paperless-ngx
paperless-ngx 3.0.2 Bug Fixes Fix: fix broken migration in 3.0.1 @shamoon ( #13242 ) All App Changes Fix: fix broken migration in 3.0.1 @shamoon ( #13242 )
Article · Home Assistant
Ever since we introduced the idea of a community-powered device database at State of the Open Home 2025, our goal has been clear: create a “Wikipedia of smart home devices” based on real-world usage. In February, we took the first step by inviting Home Assistant users to voluntar
Article · Jellyfin
🚀 Jellyfin Server 12.0 RC3 We are pleased to announce the third release candidate preview release of Jellyfin 12.0! This is a preview release, intended for those interested in testing 12.0 before its final public release. We welcome testers to help find as many bugs as we can bef
Article URL:
Article · LocalAI
What's Changed Other Changes fix(config): only inject llama.cpp serving options on the llama.cpp path by @localai-bot in #10822 Full Changelog : v4.7.0...v4.7.1
Article · LocalAI
🎉 LocalAI 4.7.0 Release! 🚀 LocalAI 4.7.0 is out! This release widens what LocalAI can generate and how you drive it: a UI-managed voice cloning library, local video and audio-driven avatar generation, and interleaved reasoning that travels with tool calls. It also lands new audio
Article URL:
Article · KoboldCpp
Fixed terminal output sometimes not showing thinking traces Multimodal Images are now inserted inline instead of at the top of the context. This should improve vision coherence and quality on most models. llama.cpp UI mic input is now disabled by default (toggle in settings) Prev
Article · Immich
v3.0.2 This patch release addresses various bugs from v3 release and adds an additional filter to workflows What's Changed 🌟 Enhancements fix(web): add ARIA label to ServerStatisticsCard by @tjhorner in #29497 feat(server): hls variant configuration by @mertalev in #29613 feat: u
Article URL:
Article · Dify
Introducing Dify Agent (Experimental) : A New Agent Experience in Dify Warning You should provide Dify Agent services ⚠️ only to trusted, non-malicious users ⚠️ . As many of you know, the shell-based LLM agent paradigm has brought a major leap in agent capabilities and changed ho
Article URL:
Article URL:
Article · Home Assistant
The Works with Home Assistant program powers up this month as we welcome our new partner, IoTorero ! 🎉 Experts in pre-flashed smart home hardware, IoTorero bring the very first ESPHome-ready smart plugs and relays to the program (and that’s not all). IoTorero enters the ring If y
Article · Home Assistant
We’re updating which Apple platforms the Home Assistant Companion app supports, and because transparency is core to how we work, we want to be upfront about what’s changing. Starting with version 2026.8.0 of the Companion app, we will no longer support iOS 15, watchOS 8, or macOS
Article · LocalAI
What's Changed 👒 Dependencies chore(deps): bump actions/cache from 4 to 6 by @dependabot [bot] in #10704 Other Changes refactor: use slices.Contains to simplify code by @weifanglab in #10702 fix(ci): shard single-arch backend matrix under GitHub's 256-job limit by @localai-bot in
Article · LocalAI
What's Changed Other Changes fix(auth): log the real cause of OIDC/OAuth user-info failures by @localai-bot in #10679 feat(api): add GET /v1/models/capabilities endpoint by @localai-bot in #10687 chore: ⬆️ Update CrispStrobe/CrispASR to 1109cb3fcae2e242c2b3d42ec0e3fd6e813f2ce7 by
Article · noted.lol
I built a voice-first AI assistant that lives behind a half mirror. Here’s how MirrorMate works, why it feels real, and how to host it yourself.
Article · LocalAI
🎉 LocalAI 4.6.0 Release! 🚀 LocalAI 4.6.0 is out! This is a reliability-focused release: AMD ROCm backends now run on-GPU at full speed, distributed model loads no longer wedge when a worker dies, and realtime sessions warm up predictably. It also brings conversation forking to th
Article · noted.lol
Get Docker and Portainer running on Debian with this straightforward beginner's guide, including Docker Compose and HTTPS access.
Article · Open WebUI
Added 💭 Streamed reasoning display. Models that emit thinking or reasoning now show that content as it streams, and it renders correctly in the chat overview and in exported conversations. Commit , Commit , Commit , Commit 🗂️ Folder uploads to knowledge bases. Dragging a folder i
Article · Home Assistant
Purpose-specific triggers and conditions
Article · Jellyfin
🚀 Jellyfin Server 12.0 RC2 We are pleased to announce the second release candidate preview release of Jellyfin 12.0! This is a preview release, intended for those interested in testing 12.0 before its final public release. We welcome testers to help find as many bugs as we can be
Article · Open WebUI
Fixed 🤝 Shared folder read-only chats no longer sign users out. Opening or reading chats from shared folders now keeps the current session active when a resource-level access error is returned, instead of incorrectly showing "Session expired. Please sign in again."
Article · Open WebUI
Added 🤝 Share folders with your team. You can now share a folder and the chats inside it with specific users, groups, or everyone, with read or write access; people you share with see shared folders in their sidebar and open the chats in a read-only view when they are not the own
Article URL:
Article · KoboldCpp
NEW: Added Krea 2 Turbo support (Recommended) . This is an extremely powerful image generation model with incredible knowledge. Grab and load the Krea 2 Turbo template containing all necessary files here. Recommended settings for Krea 2 Turbo: 8 steps at 1.0 guidance cfg . Prompt
I've been running Qwen3.6-35B-A3B locally on llama.cpp and noticed that prompt processing throughput gets too low with MTP. I got nerd-sniped. What started as curiosity turned into a two-week rabbit hole of experiments and ended with a PoC that fully recovers the MTP PP overhead
Article · AnythingLLM
AnythingLLM Is Now An AI Agent Across Your Os With AnythingLLM 1.15.0 for Desktop we have been working hard on what it means to bring your agent to you . Something that is still local, but outside of the walls of an app or a browser. This release features our first 3 features abo
Article · selfh.st
We've been trying to reach you about your self-hosted identity stack
Article · Home Assistant
It started as a passion project. Yet in a few years matter.js has grown beyond my wildest expectations, becoming the backbone of the open source Matter ecosystem, and powering everything from Homebridge to openHAB, and even some commercial products. Ten months ago, I joined force
Article · AnythingLLM
Just some small patches for the upcoming 1.15.0 / 2.0.0-preview initial release for desktop What's Changed Migrate AWS Sdk to OpenAI by @timothycarambat in #5848 Groq STT by @timothycarambat in #5853 Fix edit prompt not appearing on user prompt by @timothycarambat in #5861 add mi
Article · Jellyfin
🚀 Jellyfin Server 12.0 RC1 We are pleased to announce the first release candidate preview release of Jellyfin 12.0! This is a preview release, intended for those interested in testing 12.0 before it's final public release. We welcome testers to help find as many bugs as we can be
Article · Home Assistant
Every smart home has them: the older devices that still work perfectly well but no longer fit neatly into a modern setup. Instead of letting them gather dust in a drawer, the Open Home Foundation’s projects can help you bring them back into the fold. Here’s how a little proxying
Article · AnythingLLM
Meeting Assistant Overhaul Meeting Assistant is desktop app only We have overhauled a large portion of the Meeting Assistant to make it smaller, faster, and more efficient across all devices and platforms. Now supports Intel, AMD, and NVIDIA GPUs for a 92% smaller binary and 15%
Article URL:
Article · AnythingLLM
Improvements Cerebres provider The default chat thread is now killed when you create a new thread. If you have chats on the default thread, it will be available still. New workspaces or workspaces with no chats on default will no longer show it. All model providers are now opt-ou
Article · Jellyfin
🚀 Jellyfin Server 10.11.11 We are pleased to announce the latest stable release of Jellyfin, version 10.11.11! This minor release brings several bugfixes to improve your Jellyfin experience. As always, please ensure you take a full backup before upgrading! You can find more detai
Article URL:
Article · noted.lol
Portainer is probably the name most people know, but there are plenty of other options like Dockge, Komodo, Dockhand, Arcane, and Dokploy. Some focus on simple Docker Compose management, while others are built for multi-server deployments or full application hosting. If you had t
Article · Open WebUI
Added 📦 Official knowledge base sync tool. A new companion tool from Open WebUI, oikb, keeps a knowledge base in sync with a local directory, GitHub repo, S3 bucket, Confluence space, or any of more than 40 other sources, uploading only new and changed files using the incremental
Article URL:
Article URL:
Article · selfh.st
We've officially reached the papal stage of the AI debate
Article · Home Assistant
We’re sensing a change in the air this month as we welcome Sensereo to the Works with Home Assistant program 🎉. Specialists in environmental sensing, Sensereo brings Matter smoke and carbon monoxide (CO) alarms into the Home Assistant ecosystem – meaning more ways to keep your ho
Article · AnythingLLM
This release is focused on improving the agent experience and adding new features to the agent system as well as moving towards a more passive, personal, and hybrid AI experience. Model Router: The First Consumer Hybrid AI Experience The Model Router feature is the first-ever use
Article · Dify
🚀 What's New in v1.13.3? Our latest patch release, v1.13.3, focuses on stability and correctness across workflow execution, streaming, and knowledge retrieval. Here's a quick look at the most important updates: 🎬 New Features Workflow Configuration : Added variable-reference supp
Article URL:
Article · text-generation-webui
Changes MTP speculative decoding support : Add draft-mtp as a new --spec-type option. Auto-enabled when loading MTP GGUFs (e.g. Qwen 3.6 MoE MTP builds). Web search improvements : Add snippet support to the web_search tool: results now include a short text excerpt that often answ
Article URL:
Article · Dify
🚀 What's New in v1.14.2? v1.14.2 is a patch release focused on security hardening, workflow and knowledge reliability, observability fixes, agent groundwork, and deployment/runtime tuning after v1.14.1. 🔐 Security and administration Tenant-scoped sensitive endpoints - strengthene
Article · Home Assistant
We are often asked if we prioritize “Big Tech” firms in the Works with program, but we’ve always been clear that newer companies are just as important to our certification as household names. Start-ups are innovators, and have their fingers on the pulse of community needs much mo
Article · text-generation-webui
Changes Redesigned chat composer : Taller input area with the paperclip and message-action buttons pinned to the bottom, similar to Gemini and DeepSeek. Smooth scroll animation when sending a new message : Inspired by Gemini's chat UI. Electron improvements : Persist window bound
Article · Dify
🚀 What's New in v1.14.1? v1.14.1 is a patch release focused on security hardening, workflow and knowledge-base stability, deployment cleanup, and continued UI platform migration after v1.14.0. 🔐 Security Self-hosted SECRET_KEY hardening - Docker deployments no longer rely on a pu
Article · Open WebUI
Added 🛡️ Redirect-based SSRF protection. All outbound HTTP requests now block 3xx redirects by default via a new AIOHTTP_CLIENT_ALLOW_REDIRECTS environment variable, preventing redirect-based SSRF where a public URL silently redirects to internal addresses (RFC 1918, loopback, cl
Article · noted.lol
Issued is a personal comic library server built for a simple, focused reading experience.
Article · Home Assistant
Purpose-specific triggers and conditions
Article · Home Assistant
How do you more or less double the number of Works with Home Assistant-certified devices available to our community? Have BEGA join the program! This German firm has spent more than 75 years designing a wide range of architectural lighting that sets the industry standard: and now
Article · text-generation-webui
Changes Native desktop app : Portable builds now bundle Electron and open as a native window. Run textgen / textgen.bat instead of the previous start scripts. Pass --listen or --nowebui to skip the window and run the server directly. Major UI overhaul : Replace Noto Sans with Int
Article · selfh.st
AI and its terrible, horrible, no good, very bad week
Article · Dify
🚀 What's New in v1.14.0? Collaboration Collaboration allows workspace members to edit the same workflow together, with synced graph updates, online presence, and shared visibility into who is working where. On self-hosted deployments, collaboration is turned off by default. Enabl
Article · text-generation-webui
Changes Tool call confirmation : Add inline approve/reject/always-approve buttons that appear before each tool call is executed. Enable via the new "Confirm tool calls" checkbox in the Chat tab. Stdio MCP server support : In addition to HTTP MCP servers, you can now configure loc
Article · Home Assistant
We’re thrilled to welcome ubisys to Works with Home Assistant! This German company has been dedicated to smart home automation for more than 20 years, and offers a range of Zigbee devices designed to help you retrofit your home. If retrofitting is conjuring up images of avocado b
Article · AnythingLLM
Notable Improvements Streamed Document Embedding Now, when you upload a document to the workspace the process per-document is now reported during embedding. This is a huge improvement in performance and user experience. During this process you can add and remove documents to the
Article · selfh.st
Indie computing, Linux 7.0, and a new Jellyfin client for...the Wii?
Article URL:
Article · text-generation-webui
Changes The project has been renamed to TextGen! The GitHub URL is now github.com/oobabooga/textgen . Logits display improvements ( #7486 ). Thanks, @wiger3 . UI: Add sky-blue color for quoted text in light mode ( #7473 ). Thanks, @Th-Underscore . Reduce VRAM peak in prompt logpr
Article URL:
Article URL:
Article · noted.lol
Everyone seems to have a favorite OS for running Docker. Some stick with Debian or Ubuntu for simplicity, while others prefer platforms like Unraid or TrueNAS SCALE. What operating system do you trust to run your containers?
Article · AnythingLLM
Major Features Automatic Mode for native tool calling For Select providers that support native tool calling, you no longer need to use @agent to use tools. You can now just use the tools without asking. If your prompt input does not have the "@" symbol, your chats will automatica
Video · Techno Tim
Video from Techno Tim.
Article URL:
Article · noted.lol
Vykar Backup is a fast, Rust powered backup tool with one YAML config, built in scheduling, and easy restores via a desktop GUI and WebDAV. It supports local, S3, SFTP, and REST backends with secure deduplicated snapshots.
Article · noted.lol
Scanopy is a self-hosted network discovery tool that scans your VLANs, identifies hosts and services, and builds an interactive, self-updating topology map for your homelab.
Article · noted.lol
OpenDroneLog is a self hosted flight log dashboard for DJI and Litchi files that turns raw telemetry into maps, charts, battery health insights, and printable reports using a Docker deployable web app with local first storage.
Article · noted.lol
Take back your podcast data! HarborFM is a free and open source platform to create and host your podcasts.
Article · noted.lol
Learn how to set up ntfy, a self-hosted push notification service, on iOS with a simple workaround. Explore how ntfy makes notifications easy, reliable, and fully open source.
Article · noted.lol
An introduction to Eonvelope, the simple, flexible email-archiving server designed for self-hosters
Article · noted.lol
An honest review after nearly two years using HostHatch to run Pangolin on a Chicago based AMD Cloud VPS with NVMe storage and Debian 12.
Article · noted.lol
Learn a hands on n8n homelab project that automates media conversion, frees up space with smart compression and cleanup, and gives you a fun rainy day tinkering task.
Article · noted.lol
SmartGallery is a fast browser based gallery for ComfyUI that lets you browse, search, and rediscover every image and workflow across desktop and mobile, even when ComfyUI is not running.
Article · noted.lol
Setup your own self-hosted file search engine using sist2.
Video · Techno Tim
Video from Techno Tim.
Article · noted.lol
Portabase is a self-hosted database backup and restore tool built for security and simplicity, supporting PostgreSQL, MySQL, and MariaDB with more coming soon.
Article · noted.lol
Jotty is a lightweight but powerful alternative for managing your personal, file based, notes and checklists.
Article · noted.lol
I used to assume self-hosting was something only system administrators or homelab enthusiasts did. The folks who tinker with Raspberry Pis, collect old routers, and experiment with custom firmware. Then I set up my first service, and everything changed. Today I manage Proxmox-bas
Article · noted.lol
A look at Subgen, a self hosted tool that generates subtitles for your personal media using the OpenAI Whisper model and integrates with Jellyfin, Plex, Emby, Tautulli, and Bazarr.
Article · noted.lol
ConvertX is a self-hosted online file converter built with TypeScript, Bun, and Elysia, bringing over a thousand file formats together under one clean web interface.
Article · noted.lol
Chevereto v4.4 makes S3 storage and multi-user support free, simplifying self-hosted photo galleries and keeping your media under your control.
Article · noted.lol
FileRise is a self-hosted web file manager built for speed and control: per-folder ACLs, WebDAV, sharing, optional encryption at rest, and a new dual-pane workflow.
Article · noted.lol
A Docker management platform that doesn’t phone home. Dockhand puts full container control, monitoring, and automation back in your hands.
Article · noted.lol
Running a media server for others starts simple, then the user list grows. Tracearr is a fast analytics dashboard for Plex, Jellyfin, and Emby-with account sharing detection built in. Open-source, self-hosted, and shaped by early adopters.
Article · noted.lol
Discover Termix, the self-hosted platform for simple server management.
Article · noted.lol
Docker has made Docker Hardened Images free and open source, giving developers secure, minimal container images with full transparency from the first build.
Article · noted.lol
Helmarr is a new iOS 26 app for managing self-hosted media, bringing Sonarr, Radarr, Overseerr, and more into one customizable mobile dashboard.
Article · noted.lol
Automate Restic backups in your homelab with Zerobyte’s dashboard. Create repos, schedule jobs, and restore files with confidence.
Article · noted.lol
You've self-hosted your photos with Immich. Your documents live in Nextcloud. Your passwords are safely tucked away in Vaultwarden. You've taken control of your digital life, one Docker container or LXC at a time. But what about your thoughts and memories?
Video · Yannic Kilcher
Video from Yannic Kilcher.
Article · Self-Hosted
Before hitting the road, we test the limits of local-first file sharing, debate what self-hosting really is, and share our all-time favorite apps. Special Guests: Brent Gervais, Drew DeVore, and Wes Payne.
Article · Self-Hosted
A slick new CoreOS-based distro that might be your next home-lab base, easy self-hosted notifications, and Plex stumbles into controversy. Plus: ECC RAM guilt. Special Guests: Brent Gervais and Dusty Mabe .
Article · Self-Hosted
A week-long power blackout knocks out our guests homelab. We dig into Unraid 7.1, and push Home Assistant to keep things running-even when the power doesn’t. Special Guest: Stephen Schattin.
Article · Self-Hosted
Alex finally cracks a years-old mystery, Chris reveals his latest experiments, and we share updates on our freshest gear. Plus, a important show update.
Article · Self-Hosted
We're joined by Xe Iaso, who discusses a creative solution to relentless AI bots and the unexpected delights of running an outrageously overpowered homelab. Special Guest: Xe Iaso.
Article · Self-Hosted
When your go-to tools flip the script, our thoughts on Plex's new changes, sensors, and the creeping cost of easy.
Article · Self-Hosted
Alex’s server meets a tragic end-but its replacement is shaping up to be a beast. Meanwhile, Chris experiments with budget CO₂ detectors, sirens, and smart integrations. Plus some surprises!
Article · Self-Hosted
The joys and headaches of self-hosting, a new Zigbee routing trick, and the ongoing BcacheFS vs. Debian showdown. Plus, why we're liking Open WebUI.
Article · Self-Hosted
Chris geeks out over his new gadget, Alex gets Pangolin purring, and we break down the latest OPNsense release-plus more surprises! Special Guest: Brent Gervais.
Article · Self-Hosted
Bambu Labs teaches us how to lose friends and alienate people. Then, Alex Tran from Immich joins us for a project update, and we shared some dreams for a community RSS project. Special Guest: Alex Tran.
Article · Self-Hosted
Alex dives into the perils of upgrading and migrating critical infrastructure-uncovering embarrassing pitfalls and hard-won lessons along the way. Meanwhile, Chris takes his audiobook obsession to the next level, showcasing a couple self-hosted projects that are very ambitious.
Article · Self-Hosted
The Home Assistant Voice Preview Edition is out, and Paulus Schoutsen, the founder of Home Assistant, joins us to get into the details. Then, Alex and a special guest get into docker-compose vs. nix to manage your home lab. Special Guest: Paulus Schoutsen.
Article · Self-Hosted
Brent joins to share part two of his NAS build adventure, and things take a tiny turn. Plus, picking the right encrypted chat app and Chris stacks a few Jellyfin wins. Special Guest: Brent Gervais.
Article · Self-Hosted
We geek out over Brian Moses's 2025 DIY NAS build guide, contemplate future builds with the new Raspberry Pi Compute 5 module, and fully embrace our digital hoarding nature with a new app. Special Guest: Brian Moses.
Article · Self-Hosted
Breaking free from Google's grip: Our surprising journey and the tools that made it possible. Plus, Brent's NAS feature stirring up debate, s clever tool for distributed video encoding, and more. Special Guest: Brent Gervais.
Article · Self-Hosted
From Nextcloud Breakup to Blissful Reunion: Chris's journey back to a smarter setup. Plus, Jellyfin's game-changing features and a beloved self-hosted app get the upgrade we've all been waiting for.
Article · Self-Hosted
"The" self-hosted app to archive your favorite YouTube channels and easily integrate into Jellyfin/Plex. Plus, our favorite WordPress alternatives and an update on No Google October.
Article · Self-Hosted
Ditching Google Search for an entire month! We reveal the tool that's helping us break free. Plus, a special guest shares his home lab to data center journey. And, Chris raves about the ultimate Jellyfin client (and confesses to an accidental network camera purchase).
Article · Self-Hosted
Alex has been playing around at the speed of light while solving Proxmox problems, and Chris has solved a Jellyfin issue. Plus, our thoughts on the new Plex features.
Article · Self-Hosted
We celebrate five years of the show, chat about self-hosted Lightning, and why Alex loves his NanoKVM. Plus, it is a self-hosted replacement for Amazon Wishlists and more.
Video · Wolfgang's Channel
Video from Wolfgang's Channel.
Article · Self-Hosted
Prusa's Mk4S 3D printer seems to have hit the mark. Alex gets a tour of an awesome new maker space, we take a look at AdventureLog, and much more. Special Guest: Jscar_Hawk.
Article · Self-Hosted
The battle for code forges is heating up. We chat about HexOS' big promises and get excited about Meshtastic.
Article · Self-Hosted
ESPHome dev dishes on device updates, Immich license drama heats up, Alex's DIY server fix, and Chris reports on mobile tech trip test. Special Guest: Keith Burzinski.
Article · Self-Hosted
Chris gets serious about tracking maintenance and alerts, why Alex is impressed by the RISC-V-powered NanoKVM, how we might end up using Docmost, and a follow-up review of LubeLogger.
Article · Self-Hosted
Chris reviews the Aqara G4 Video Doorbell, Brent frees his Garmin from the cloud, and we discuss getting iMessages on Android. Special Guest: Brent Gervais.
Article · Self-Hosted
Alex's mini-PC surprise, why we're trying Tube Archivist, Alex's Nextcloud update, and how Chris stacks automations with Bitfocus Companion.
Article · Self-Hosted
The "you'll own nothing" trend got worse this week, our thoughts about the Raspberry Pi IPO, poor Nextcloud performance, and Alex's new high-fiber obsession.
Article · Self-Hosted
Alex benchmarks Intel CPUs (and an Arc GPU) to find the ideal balance of age, power, and speed for your home media server. Plus, our thoughts on Immich going full-time.
Article · Self-Hosted
How Chris created live TV streaming from his local media collection, Alex breaks down the new Open Home Foundation and what it means for self-hosters. Brent's been trying out an open-source AirDrop replacement for all systems, and much more! Special Guest: Brent Gervais.
Article · Self-Hosted
Special guest Casey Liss from the Accidental Tech Podcast joins the show to discuss his homelab, how he uses HomeBridge, and his delightfully complex garage door sensor system. Then Alex and Casey do their "best" to convince Chris the Apple Vision Pro is an excellent remote admin
Article · Self-Hosted
Alex goes head-to-head with budget VPS providers, which gets us into a classic debate. Plus we sit down with Adam Morales from Unraid to get the inside scoop on recent changes and exciting upcoming features.
Article · Self-Hosted
Alex rolls back a major server upgrade, and we have fun playing with local large language models. Special Guest: Wes Payne.
Article · Self-Hosted
Alex's new Epyc server build, and Jon Seager from Canonical joins us to chat about Nix in the homelab, packaging Scrutiny, and how Nix fits with existing infrastructure management tools. Special Guest: Jon Seager .
Article · Self-Hosted
We chat about VMware's rug pull with Bret, aka Raid Owl, and then get into Unraid's big changes and more. Special Guest: Raid Owl.
Video · Wolfgang's Channel
Video from Wolfgang's Channel.
Article · Self-Hosted
Alex has been deep-diving into container networking, and Chris is trying to steelman Plex's new rental service. Plus, why are we building our containers with Tailscale networking now, and the latest from the Home Assistant project?
Article · Self-Hosted
Brian Moses joins us and shares his most recent NAS build and love for 3D printers. Then Alex gets into the hardware he's deploying around the house, and why we don't see eye-to-eye on ZigBee. Special Guest: Brian Moses.
Article · Self-Hosted
We kick off the new year with new apps in our home lab you’ll want to try and a new way to do networking.
Article · Self-Hosted
We look back at what has changed, what's failed us, and what's sticking around in our homelabs. Special Guest: Brent Gervais.
Article · Self-Hosted
Alex shares a new build integrating WLED, and Chirs reviews hardware that can get you started with WLED in 45 seconds. Then, one last big update on the Year of Voice and our thoughts on self-hosting push notifications.
Article · Self-Hosted
We break down the state of the pfSense changes and the red flags we see. Plus, we're joined by Wolfgang from Wolfgang's channel to dig into his homelab and much more. Special Guest: Wolfgang.
Sources
Letters, communities, and homelab projects the edition reads. A broad newsletter only contributes a story when that story is about running something at home.
Five-minute morning digest.
Practical “how to use this.”
Short bullets for engineers.
Builders and indie founders.
Policy and research. Home-lab pieces only.
Research notes for practitioners.
Long-form audio.
Open-source releases and tutorials.
Ollama, llama.cpp, self-hosted, and homelab threads.
Local models, GGUF, and home GPUs.
Services you run yourself.
Racks, Proxmox, and home servers.
Self-hosted app discoveries.
Homelab writeups and walkthroughs.
Weekly homelab show.
Home automation releases.
Local model runtime.
Local inference builds.
Browser UI for local models.
Local image workflows.
Self-hosted photos.
Self-hosted media.
Self-hosted documents.
OpenAI-compatible local API.
Self-hosted automation.
Tools and local-model notes.
Lab hardware, when the piece is about a homelab.
Homelab and self-hosting videos.