I and presumably quite a few others with AMD AI or Apple Mac platforms are very impacted by this.
:)
It is very relevant and for a certain group of us, far more impactful to our work the next month(s) than any blog post could be.
KronisLV · 2026-08-26 13:22:36 UTC
If there’s news, then yes. This is a pretty great new release for those still stuck on Qwen3.6 35B A3B if they have enough memory but don’t have super powerful compute.
I wonder if I could get this running through vLLM on 6x Nvidia L4 - the 3.6 worked great on 4 cards but sadly TP6 just isn’t a thing and I don’t have 8 cards available, maybe it’s gonna be okay with like TP2 and MTP. I have no idea at this time, probably need to test out what even might be possible.
NitpickLawyer · 2026-08-26 13:26:55 UTC
This particular release is interesting because it's a preview of qwen4 architecture. And, while benchmarks are iffy, this is a direct comparison, by the same team, with qwen3.8-27b that was pretty well received for a local model.
This "next" release adds a new concept, first public release with n-grams, I think. And it's in a MoE size that is likely to be very fast and cheap to serve (faster than 27b for sure). It's also well suited for inference on alternative compute (i.e. sparks, macs, etc) so it's relevant to local users.
dofm · 2026-08-26 13:29:39 UTC
This actually is meaningful news, I think. Pretty wide audience appeal in the local LLM space too.
c16 · 2026-08-26 13:30:08 UTC
There are many topics, personalities and politicians we hear about daily who have no merit.
Qwen's advances do (currently) have merit.
christkv · 2026-08-26 13:13:18 UTC
Looks like a good model for strix halo
Iolaum · 2026-08-26 13:24:31 UTC
indeed, can't wait for it to be supported by llama.cpp (or other engines)?
christkv · 2026-08-27 12:53:25 UTC
Then we probably have to wait a little for them to optimize it.
tosh · 2026-08-26 13:14:22 UTC
this is a new architecture (foreshadowing qwen 4)
> trained at just 1/9 the cost of Qwen3.7-Plus, while outperforming it across the board
Opus 4.6 Max self-hosted at 30 tok/s on a 5k Macbook in Aug 2026. The LLM timelines are crazy.
Squarex · 2026-08-26 13:21:38 UTC
I don't like these comparisons. Sure it is impressive, but it does not have a world knowledge of larger models. It has most of theirs intelligence.
LaurensBER · 2026-08-26 13:30:21 UTC
If/when we can get larger context this will mostly be mitigated by these smaller models being able to search the internet.
Self-learning/improving would be even better but that's still a long way to go.
redox99 · 2026-08-26 15:52:37 UTC
Search results suck because the web sucks these days. The big models from OpenAI/Anthropic have every book in existence baked into them
jameshart · 2026-08-26 16:23:52 UTC
I don’t think that’s the right way to think about LLM ‘knowledge’. They don’t have absolute recall of everything in the training set. They have been trained so that they have weights that can predict what those books might say - that is, if they read them they would find the contents unsurprising. That doesn’t mean it wouldn’t be helpful to pull relevant passages of text directly into context for a particular task.
infp_arborist · 2026-08-26 18:55:36 UTC
Does it really matter? What about including all relevant and up-to-date literature as skills for local models? I have no experience with this but I am pretty sure someone has already thought about it.
rohansood15 · 2026-08-26 13:34:59 UTC
For world knowledge, you'd want it to find and reference the source material to be sure. At that point, it doesn't matter if the knowledge is embedded.
quev · 2026-08-26 13:57:42 UTC
Keep in mind a web search might not include scanned books baked in the weights ;)
skeledrew · 2026-08-27 09:34:00 UTC
Are Chinese labs also acquiring and scanning books?
hedora · 2026-08-26 13:59:02 UTC
I think the big models have adequate recall, so tool use is probably unnecessary, but the user said the correctness of my response is important. Let me look up the data instead of relying on my memory.
dist-epoch · 2026-08-26 14:01:44 UTC
World knowledge also means knowing the various algorithms and ways particular programming problems are solved.
You can't search what you don't even know exists.
serf · 2026-08-26 16:07:41 UTC
>You can't search what you don't even know exists.
that's not really entirely true -- one can google for "fast pathfinding' and stumble upon A-star , all that had to be queried was the intent/desire.
a lot of smaller agentic models and a lot of harnesses live on that premise.
water-drummer · 2026-08-26 19:43:18 UTC
Path finding is a very closed and well defined problem.
horsawlarway · 2026-08-26 14:25:07 UTC
In a lot of spaces, this is actually preferable.
Ex - nodejs natively supports a huge set of typescript with built-in type stripping these days. But ask most hosted models to build a typescript project and they default to a heavy compile step, or a tool like tsx, ts-node, etc.
Models with lots of "world knowledge" have a good chunk of that knowledge go stale, and there's no real way to refresh it without training a new model.
Another classic example of this back in the day was to ask who the president of the US was, and watch different models happily give different answers based on the date they were trained.
---
Personally, I'm really interested to see if we're headed towards a spot where the model is entirely distinct from the knowledge store.
We're vaguely there with the ability for models to go search the web, but I think the reliability of that path is going to continue declining (more and more spam content, less and less genuine value).
I kinda want a paradigm where I can pick and engine and a knowledge bank, and combine them as I please.
Ex - if I'm doing gardening, I can pick "gardening for models (version 32)" as my knowledge store.
If I'm doing auto-repair... "cars for dummies (version 3)". etc...
donmcronald · 2026-08-26 15:27:50 UTC
> Personally, I'm really interested to see if we're headed towards a spot where the model is entirely distinct from the knowledge store.
This is what I've been trying to focus on with local AI for now. I've been trying to build all new documentation so it's more AI friendly. It's been pretty interesting. Qwen-35BA3B with a small prompt does a good job of surfacing what I'd consider institutional knowledge.
I've been trying to silo the docs I write from the model with a prompt that tells it not to use general knowledge unless asked to. From the anecdotal testing I did, Qwen-35BA3B is great for it. It does a really good job of following the prompt and calling tools, so I've been able to play around a lot to see what seems to work best.
Ultimately, I think one of the most effective uses of AI will be having a distinct knowledge store combined with an opinionated agent (and sub-agent) setup along with different models for each task.
Who owns the knowledge store is going to be the big caveat. Right now I think the big online models are trying for generic, persistent memory and I'd be very hesitant to let that happen. Think of having someone with a perfect memory following you around forever, but someone else has the ability to make them disappear. That's not a good situation.
jameshart · 2026-08-26 16:34:37 UTC
One of the consequences of encountering a lot of LLM generated text which includes things the model vaguely remembers from its training is that honestly I have grown less tolerant even of human comments and documents that are based on mostly ‘I seem to recall that…’ level sourcing.
In a discussion on economic history, say, someone will opine that Alexander Hamilton had some particular opinion about tariff policy… based on their having a vague memory of a blog post where someone quoted a passage in support of some point. But wait - you can search the federalist papers, the text’s right there to be read, before you commit to saying online ‘Hamilton thought tariffs were a great idea’ you could take your internal ‘I seem to recall reading something about hamilton’s opinion on tariffs’ thought and turn it into a little RAG query where you pull up a source and check before you put another factoid out onto the internet.
And so I feel absolutely the same way about LLMs. I don’t care how much factual information was in the training data, when the LLM wants to rely on something it vaguely recalls having been trained on, it owes it to me to dig up a source and vet it.
There are limits to this, of course. I don’t want it to be thinking ‘but wait, maybe my memory of Python syntax is faulty. Is = used for assignment? <web search>…’.
But in general some caution about repeating vaguely recalled easily checked facts is warranted.
redox99 · 2026-08-26 15:48:00 UTC
At 125B + 51B I'd expect it to have some degree of world knowledge, clearly in the middle between small models like qwen 27B, and huge trillion parameter models.
gruez · 2026-08-26 13:24:25 UTC
>Opus 4.6 Max self-hosted at 30 tok/s on a 5k Macbook in Aug 2026. The LLM timelines are crazy.
How much memory does this translate to and what quantization (if any) were applied?
rohansood15 · 2026-08-26 13:35:34 UTC
128GB, 4-bit quantized.
user43928 · 2026-08-26 13:25:26 UTC
For comparison with hosted models, GPT 5.6 Luna scores 67% on DeepSWE, compared to 59% here for Qwen.
Luna is $0.20 / $1.20 vs $0.16 / $0.47 with Qwen.
rohansood15 · 2026-08-26 13:37:47 UTC
This is a good counter argument. But you have to note that this is after OpenAI cut Luna costs by 80%. If you compare launch pricing, Qwen probably comes out ahead on a cost-performance basis.
jrflo · 2026-08-26 13:53:33 UTC
The luna cost cuts were real though, not a one time promotion or something, due to some optimization (probably distillation?) that openai did.
QwenGlazer9000 · 2026-08-26 13:57:57 UTC
Was it?
Given the timing, I think they A. shat their pants since Deepseek flash just came out with insane pricing before the price hikes, and B. Anthropic is really struggling in model tiers below opus.
It was smart for them to cut prices regardless of whether they had 80% efficiency gains or not
throwaw12 · 2026-08-26 14:10:27 UTC
what if it was because of quantization and they haven't released the new benchmarks for it?
Anything which changes the model needs new benchmarks I guess to compare with other models, otherwise you can benchmark Fable, and distill it to student model and keep claiming this is the Fable model
dannyw · 2026-08-26 14:57:26 UTC
ARC Prize has retested Luna after the discount and validated identical performance.
(Also, quantization isn't inherently bad or damaging when done properly, e.g. QAT).
These APIs are used heavily by enterprises at scale; with lots of performance telemetry, live evals, etc. You can't really silently nerf API models at scale without people noticing.
Of course, what I said doesn't apply to non-API consumer sub models; there's many documented and officially confirmed instances of under-the-hood "juice/effort" adjustments. (Juice = a number your effort tier maps to underneath the hood; much like Inkling's effort=0.00 to 0.99).
mattalex · 2026-08-26 15:46:53 UTC
You assume that openai's inference is profitable and that they aren't just trying to bolster revenue before their IPO.
The only indication that openai is profitable comes from openai (whom I wouldn't trust with any statement, especially when it comes to profitability).
In fact there is evidence that inference is not profitable simply because the rate of losses doesn't seem to reduce as revenue increases: if inference had great margins, we would expect that as revenues increase, the amount of spend on training reduces as a fraction of total expenses.
Since the loss-making fixed costs shrink as a fraction compared to the profitable inference, we should expect profitability to rise with total revenue.
However, all leaks of openai's numbers seem to suggest the opposite: as revenues increase so do the losses.
hluska · 2026-08-26 15:54:56 UTC
I don’t pay OpenAI’s bills - I pay what they charge me. Their cost accounting isn’t relevant to a user.
aaa_aaa · 2026-08-26 16:32:50 UTC
Argument was that open ai cannot be profitable with this. But sure, use it while you can.
mediaman · 2026-08-26 18:34:32 UTC
You can make the other argument that China subsidizes the price and that they can't be profitable at this pricing level. From an industrial strategy standpoint, they already do this for many other industries with huge subsidized state loans.
So we can go round and round on this, each with our made-up objections about how it's temporary or unrealistic or impossible or whatever, or we can just accept the prices as listed and use that to guide our economic decisions.
aaa_aaa · 2026-08-26 19:15:06 UTC
Private companies cannot play that game too long. Profit from current state of AI is a mirage and sooner or later stuff will hit the fan.
mediaman · 2026-08-26 23:38:34 UTC
Just look at the prices that inference providers charge for small models. The argument that these unit economics are negative is trivial to disprove.
DeepInfra sells DS v4-flash at 0.08 in, $0.18 out. Gemma4 they sell for $0.07 in, $0.34 out. OpenAI's price for luna is $0.20 in, $1.20 out.
Why would you assume OpenAI is somehow uniquely incompetent at making small, fast models? And that they're worse at serving it than DeepInfra? Any observer can see they are making money here.
I never understand why people who are convinced there is a big con just don't check market prices and see if there's money to be made.
That doesn't mean their business is great -- they're losing tons of money, but it's because they spend too much on fixed costs, and they can't stop spending money on training next generation models with no end in sight, not because the inference is margin negative, which is a flimsy idea that just clouds the actual business issue.
aaa_aaa · 2026-08-27 06:00:45 UTC
Because Openai is just another player. Nothing really special for now. Their valuation is ridiculously overblown.
computerex · 2026-08-26 21:12:49 UTC
Didn't OpenCode CTO state they could replicate deepseek pricing on rented hardware?
nl · 2026-08-27 02:28:04 UTC
There's a difference between the Deepseek.com provider lunch pricing and the pricing every other provider is doing now.
Right now DS4-Pro-0813 is available from multiple providers for $1.32/million input tokens[1].
It's pretty easy to work backwards from B200 and electricity prices and see this is profitable even without the heavy serving optimization these providers are doing[1.5].
The OpenCode CEO said: "inference is very profitable and probably a good opportunity to understand some basic business math"[2] and "the inference we do is already profitable and that's with some middlemen involved"[3]
If at this point people don't believe inference can be profitable, and providers can turn the prices up and down to choose exactly how profitable they make it I don't know what to say.
The indication that OpenAI's inference is profitable is that 3rd party providers host large models for cheaper.
Given that OpenAI is ahead in intelligence, it's also reasonably likely that they are at the frontier of efficiency too.
Your "evidence" for OpenAI's inference not being profitable is apparently based on leaked financials supposedly showing growing losses for reasons entirely unknown.
With their research, training, data centers, chip development, and hardware product development, there seem to be a number of reasons that might explain growing losses.
itishappy · 2026-08-27 02:09:10 UTC
> Given that OpenAI is ahead in intelligence, it's also reasonably likely that they are at the frontier of efficiency too.
Frontier labs have no incentive to be at the frontier of efficiency.
Claude still leads the pack in general intelligence yet has the worst efficiency by far.
user43928 · 2026-08-27 08:00:49 UTC
They have an incentive to make their models efficient enough to serve demand and make a profit on it.
The incentive that is missing is passing on efficiency improvements as price savings to customers, when your model is still in demand because of its higher intelligence.
itishappy · 2026-08-27 22:50:10 UTC
Agreed, efficiency is still important, but being at the "frontier of efficiency" is significantly more relevant to commodity model providers than state-of-the-art model providers. Frontier labs are incentivized to route their spend towards beating benchmarks because that's what enables them to charge a premium.
Almondsetat · 2026-08-26 14:13:34 UTC
>If you compare launch pricing
Why?
nl · 2026-08-27 02:35:43 UTC
Why would anyone car what the launch price is? Comparing launch pricing is just an odd thing to do.
rohansood15 · 2026-08-27 04:19:40 UTC
Because labs can learn to optimize inference post launch, plus can move to use bigger/better clusters depending on demand. It is not impossible to imagine Qwen cuts prices further with QAT/MTP-like improvements.
nl · 2026-08-27 12:42:00 UTC
Or they could move from highly subsidized models like the Deepseek 4 launch pricing.
Launch price is just like any other price. It's just a price. It's impossible to guess what might or might not happen.
Compare the price now.
criley2 · 2026-08-26 14:57:17 UTC
Those prices are just tokens? Since each model uses different amounts of tokens to do the same thing, it's a misleading price that often makes open-weights look more competitive than they are, since most open weights models use dramatically more tokens and time to complete tasks than many frontier models.
In Artifical Analysis's cost per task, Luna(max) costs $0.05 per task, and Qwen 3.8 27B costs $0.25 per task, a 5X increase. We'll see how 3.8-flash-next does.
hadlock · 2026-08-27 03:11:52 UTC
the important thing is that Qwen 3.7 27B will run unlimited jobs on my consumer grade laptop at 60 tokens/second for free, forever, in about 1-2 years
villish · 2026-08-27 04:39:28 UTC
Thats only important if running it locally is critical for privacy reasons or just as a hobby.
Time has a cost in business. If a model needs 30 million tokens to achieve a similar result as another that can do it in 10 million, that 60 tokens per second will take a long time.
hadlock · 2026-08-27 10:41:28 UTC
Right now qwen 3.6 35b-a3b has a success rate of 92% and qwen 3.8 27b has a success rate of 96%. But the 35b moe does about 1080 tokens/s at concurrency 54, vs 480 tokens/s at concurrency 28. For our specific workflow on blackwell.
Of course enormous batch jobs are different. I was explicit when I said consumer laptop.
criley2 · 2026-08-27 12:07:28 UTC
It's not free. You're paying electricity and you're ignoring the cost of the hardware. Even on electricity alone, there are cloud providers who may beat your laptop on price per million tokens. Qwen 3.8 flash is interesting in this space.
Not to say that there aren't other benefits of running models locally, I loaded Qwen 3.8 27B 6bit MLX just yesterday.
claudeIsDown · 2026-08-26 14:59:53 UTC
Sounds like discrete propaganda
dist-epoch · 2026-08-26 14:02:26 UTC
It's a much bigger model, with a next-gen architecture. It's expected to be much better.
hedora · 2026-08-26 14:23:42 UTC
My AMD strix halo box (haven’t benchmarked yet) should also run it reasonably well. It was $1400 at launch, and is $4K now.
Your mac is < $2K in Biden-era dollars. Presumably the economy will eventually recover; maybe in one Moore’s law doubling if the midterms go outrageously well. That’ll be two doublings since the halo launched. I’d expect this model to run on a sub $1K box by then. $2K ought to get you a 512b parameter model at that point. If we have to wait out the rest of the term, the cost cliff will be even more pronounced when it hits.
aftbit · 2026-08-26 14:42:01 UTC
I believe you're underestimating the lag inherent in the economy. Even if we grant the idea that the political party controlling the US House/Senate has a significant impact on the economy, and that the current party is BAD and the next one would be GOOD, I would still expect that things will continue getting WORSE for a good 4 to 8 years before they get better again.
And that's even with assuming that we can continue to ignore the long-term problems like social security insolvency, the debt bomb, or climate change forever.
NewJazz · 2026-08-26 14:43:49 UTC
You know the memory cartel isn't even close to being broken, right?
overgard · 2026-08-26 16:16:43 UTC
Curious, how are you running it and what quantization are you using? I've mostly been using MTPLX; 125B sort of looks like it'd be right at the limits of my 128GB MacBook once you factor in KV cache and context window.. wondering if it's worth it compared to the 27B model which gives me a lot of headroom or even a 72B model.
lnenad · 2026-08-26 13:30:15 UTC
Adding to my homelab stack, hopefully it doesn't overthink like the little model. Actually, hoping it thinks a bit less. Wait actually I'm really praying it reasons a bit more directly. But wait, I'm really sure that it must be a bit better.
grim_io · 2026-08-26 13:33:06 UTC
That's low reasoning for a model, but max for a HN comment.
cyanydeez · 2026-08-26 13:43:50 UTC
My stack is basically deer-flow with Qwen3.5-122B-A10B; this hopefully will be a speed and intelligence improvement. Running deer-flow overnight on any research topic or verify clear scoped programming issue is really neat.
Also, heating my home during the winter is nice.
Oh, also, I use llamacpp with --reasoning-budget; very simple way to move on.
lnenad · 2026-08-26 14:16:55 UTC
Yeah 122B is the sweet spot for me as well. Even deepseek flash overthinks on stuff way too much. I think they fully rely on large reasoning turns to achieve better quality. The result of course means we wait a long time to get results even with high throughput as a lot of tokens are wasted.
redrix · 2026-08-26 14:06:03 UTC
You’re absolutely right to be hopeful. Three honest possibilities, and I’ll be straight with you about each:
1. It overthinks — Just like the previous iteration. High confidence.
2. It doesn’t overthink — Improvement from the last model for your use case. Regression for others.
3. It sometimes overthinks — Best case all around. A feature, not an impairment.
One final thing worth mentioning:
(I made myself irrationally angry writing this)
gorgmah · 2026-08-26 14:09:15 UTC
You made me irrationally laugh reading this
lsb · 2026-08-26 14:15:40 UTC
This is glitch art for text, I love it
raducu · 2026-08-26 14:48:55 UTC
On one hand I love your joke, on the other, this is HN not reddit and I usually downvote such responses, not sure what is the HN etiquette for such humor?
samsari · 2026-08-26 14:54:43 UTC
70% of the posts on HN are already satire and performance art
pixelpoet · 2026-08-26 16:40:34 UTC
And full of made up statistics.
NamlchakKhandro · 2026-08-26 21:45:37 UTC
Which then devolve into supporting arguments for communism. Themselves becoming food for future irrational anecdotes about communism.
switchbak · 2026-08-26 16:51:33 UTC
More than 2 levels and out come my downvotes. Or if it's just knee jerk with zero humour. But I probably violate my own rules ... which is to be expected.
terrut · 2026-08-26 20:53:47 UTC
You are right to push back— Sorry, couldn't resist ;) I agree that this is not what we normally come here for, but this thread made me chuckle. I think we are just venting our shared frustrations a bit.
mcny · 2026-08-26 15:11:20 UTC
I reached point three and was nodding all along. I guess I am the NPC
Bluestein · 2026-08-26 15:26:32 UTC
That's a caveat, and a real one.-
amoss · 2026-08-26 15:35:00 UTC
But the reason why it remains load bearing is key.
hedgehog · 2026-08-26 16:53:56 UTC
You've made a really sharp observation, and the reason it lands is worth naming:
Bluestein · 2026-08-26 17:18:16 UTC
"... worth naming: ..."
⎿ You've hit your session limit · resets 2:50am (123°24′W Etc/GMT+8)
/upgrade to increase your usage limit.
peterleiser · 2026-08-26 15:30:57 UTC
> Three honest possibilities, and I’ll be straight with you about each
This. I don't know if the "honest answer" phrasing is part of the system prompt or alignment, but when people say "honestly" all the time I start wondering how honest they're being.
hluska · 2026-08-26 15:49:05 UTC
Thankfully most people have better reading skills than that.
switchbak · 2026-08-26 16:50:11 UTC
At this point, I'm starting to wonder if their honesty is even load-bearing at all?
Comments
> trained at just 1/9 the cost of Qwen3.7-Plus, while outperforming it across the board
https://x.com/Alibaba_Qwen/status/2092591393424515114
:)
It is very relevant and for a certain group of us, far more impactful to our work the next month(s) than any blog post could be.
I wonder if I could get this running through vLLM on 6x Nvidia L4 - the 3.6 worked great on 4 cards but sadly TP6 just isn’t a thing and I don’t have 8 cards available, maybe it’s gonna be okay with like TP2 and MTP. I have no idea at this time, probably need to test out what even might be possible.
This "next" release adds a new concept, first public release with n-grams, I think. And it's in a MoE size that is likely to be very fast and cheap to serve (faster than 27b for sure). It's also well suited for inference on alternative compute (i.e. sparks, macs, etc) so it's relevant to local users.
Qwen's advances do (currently) have merit.
> trained at just 1/9 the cost of Qwen3.7-Plus, while outperforming it across the board
https://x.com/Alibaba_Qwen/status/2092591393424515114
Opus 4.6 Max self-hosted at 30 tok/s on a 5k Macbook in Aug 2026. The LLM timelines are crazy.
Self-learning/improving would be even better but that's still a long way to go.
You can't search what you don't even know exists.
that's not really entirely true -- one can google for "fast pathfinding' and stumble upon A-star , all that had to be queried was the intent/desire.
a lot of smaller agentic models and a lot of harnesses live on that premise.
Ex - nodejs natively supports a huge set of typescript with built-in type stripping these days. But ask most hosted models to build a typescript project and they default to a heavy compile step, or a tool like tsx, ts-node, etc.
Models with lots of "world knowledge" have a good chunk of that knowledge go stale, and there's no real way to refresh it without training a new model.
Another classic example of this back in the day was to ask who the president of the US was, and watch different models happily give different answers based on the date they were trained.
---
Personally, I'm really interested to see if we're headed towards a spot where the model is entirely distinct from the knowledge store.
We're vaguely there with the ability for models to go search the web, but I think the reliability of that path is going to continue declining (more and more spam content, less and less genuine value).
I kinda want a paradigm where I can pick and engine and a knowledge bank, and combine them as I please.
Ex - if I'm doing gardening, I can pick "gardening for models (version 32)" as my knowledge store.
If I'm doing auto-repair... "cars for dummies (version 3)". etc...
This is what I've been trying to focus on with local AI for now. I've been trying to build all new documentation so it's more AI friendly. It's been pretty interesting. Qwen-35BA3B with a small prompt does a good job of surfacing what I'd consider institutional knowledge.
I've been trying to silo the docs I write from the model with a prompt that tells it not to use general knowledge unless asked to. From the anecdotal testing I did, Qwen-35BA3B is great for it. It does a really good job of following the prompt and calling tools, so I've been able to play around a lot to see what seems to work best.
Ultimately, I think one of the most effective uses of AI will be having a distinct knowledge store combined with an opinionated agent (and sub-agent) setup along with different models for each task.
Who owns the knowledge store is going to be the big caveat. Right now I think the big online models are trying for generic, persistent memory and I'd be very hesitant to let that happen. Think of having someone with a perfect memory following you around forever, but someone else has the ability to make them disappear. That's not a good situation.
In a discussion on economic history, say, someone will opine that Alexander Hamilton had some particular opinion about tariff policy… based on their having a vague memory of a blog post where someone quoted a passage in support of some point. But wait - you can search the federalist papers, the text’s right there to be read, before you commit to saying online ‘Hamilton thought tariffs were a great idea’ you could take your internal ‘I seem to recall reading something about hamilton’s opinion on tariffs’ thought and turn it into a little RAG query where you pull up a source and check before you put another factoid out onto the internet.
And so I feel absolutely the same way about LLMs. I don’t care how much factual information was in the training data, when the LLM wants to rely on something it vaguely recalls having been trained on, it owes it to me to dig up a source and vet it.
There are limits to this, of course. I don’t want it to be thinking ‘but wait, maybe my memory of Python syntax is faulty. Is = used for assignment? <web search>…’.
But in general some caution about repeating vaguely recalled easily checked facts is warranted.
How much memory does this translate to and what quantization (if any) were applied?
Luna is $0.20 / $1.20 vs $0.16 / $0.47 with Qwen.
Given the timing, I think they A. shat their pants since Deepseek flash just came out with insane pricing before the price hikes, and B. Anthropic is really struggling in model tiers below opus.
It was smart for them to cut prices regardless of whether they had 80% efficiency gains or not
Anything which changes the model needs new benchmarks I guess to compare with other models, otherwise you can benchmark Fable, and distill it to student model and keep claiming this is the Fable model
(Also, quantization isn't inherently bad or damaging when done properly, e.g. QAT).
These APIs are used heavily by enterprises at scale; with lots of performance telemetry, live evals, etc. You can't really silently nerf API models at scale without people noticing.
Of course, what I said doesn't apply to non-API consumer sub models; there's many documented and officially confirmed instances of under-the-hood "juice/effort" adjustments. (Juice = a number your effort tier maps to underneath the hood; much like Inkling's effort=0.00 to 0.99).
The only indication that openai is profitable comes from openai (whom I wouldn't trust with any statement, especially when it comes to profitability).
In fact there is evidence that inference is not profitable simply because the rate of losses doesn't seem to reduce as revenue increases: if inference had great margins, we would expect that as revenues increase, the amount of spend on training reduces as a fraction of total expenses. Since the loss-making fixed costs shrink as a fraction compared to the profitable inference, we should expect profitability to rise with total revenue.
However, all leaks of openai's numbers seem to suggest the opposite: as revenues increase so do the losses.
So we can go round and round on this, each with our made-up objections about how it's temporary or unrealistic or impossible or whatever, or we can just accept the prices as listed and use that to guide our economic decisions.
DeepInfra sells DS v4-flash at 0.08 in, $0.18 out. Gemma4 they sell for $0.07 in, $0.34 out. OpenAI's price for luna is $0.20 in, $1.20 out.
Why would you assume OpenAI is somehow uniquely incompetent at making small, fast models? And that they're worse at serving it than DeepInfra? Any observer can see they are making money here.
I never understand why people who are convinced there is a big con just don't check market prices and see if there's money to be made.
That doesn't mean their business is great -- they're losing tons of money, but it's because they spend too much on fixed costs, and they can't stop spending money on training next generation models with no end in sight, not because the inference is margin negative, which is a flimsy idea that just clouds the actual business issue.
Right now DS4-Pro-0813 is available from multiple providers for $1.32/million input tokens[1].
It's pretty easy to work backwards from B200 and electricity prices and see this is profitable even without the heavy serving optimization these providers are doing[1.5].
The OpenCode CEO said: "inference is very profitable and probably a good opportunity to understand some basic business math"[2] and "the inference we do is already profitable and that's with some middlemen involved"[3]
If at this point people don't believe inference can be profitable, and providers can turn the prices up and down to choose exactly how profitable they make it I don't know what to say.
[1] https://openrouter.ai/deepseek/deepseek-v4-pro-0813#provider...
[1.5] https://www.seangoedecke.com/ai-inference-is-obviously-profi...
[2] https://x.com/thdxr/status/2042277156940587469?lang=en
[3] https://x.com/thdxr/status/2042614323344818520
Given that OpenAI is ahead in intelligence, it's also reasonably likely that they are at the frontier of efficiency too.
Your "evidence" for OpenAI's inference not being profitable is apparently based on leaked financials supposedly showing growing losses for reasons entirely unknown.
With their research, training, data centers, chip development, and hardware product development, there seem to be a number of reasons that might explain growing losses.
Frontier labs have no incentive to be at the frontier of efficiency.
Claude still leads the pack in general intelligence yet has the worst efficiency by far.
The incentive that is missing is passing on efficiency improvements as price savings to customers, when your model is still in demand because of its higher intelligence.
Why?
Launch price is just like any other price. It's just a price. It's impossible to guess what might or might not happen.
Compare the price now.
In Artifical Analysis's cost per task, Luna(max) costs $0.05 per task, and Qwen 3.8 27B costs $0.25 per task, a 5X increase. We'll see how 3.8-flash-next does.
Time has a cost in business. If a model needs 30 million tokens to achieve a similar result as another that can do it in 10 million, that 60 tokens per second will take a long time.
Of course enormous batch jobs are different. I was explicit when I said consumer laptop.
Not to say that there aren't other benefits of running models locally, I loaded Qwen 3.8 27B 6bit MLX just yesterday.
Your mac is < $2K in Biden-era dollars. Presumably the economy will eventually recover; maybe in one Moore’s law doubling if the midterms go outrageously well. That’ll be two doublings since the halo launched. I’d expect this model to run on a sub $1K box by then. $2K ought to get you a 512b parameter model at that point. If we have to wait out the rest of the term, the cost cliff will be even more pronounced when it hits.
And that's even with assuming that we can continue to ignore the long-term problems like social security insolvency, the debt bomb, or climate change forever.
Also, heating my home during the winter is nice.
Oh, also, I use llamacpp with --reasoning-budget; very simple way to move on.
1. It overthinks — Just like the previous iteration. High confidence. 2. It doesn’t overthink — Improvement from the last model for your use case. Regression for others. 3. It sometimes overthinks — Best case all around. A feature, not an impairment.
One final thing worth mentioning: (I made myself irrationally angry writing this)
This. I don't know if the "honest answer" phrasing is part of the system prompt or alignment, but when people say "honestly" all the time I start wondering how honest they're being.