But hardware to run them is conveniently something IBM would be happy to sell you.
anukin · 2026-10-26 03:11:51 UTC
Not till Jensen buys his next leather jacket.
jacquesm · 2026-10-26 03:30:04 UTC
If IBM would ship a piece of affordable hardware that can run large LLMs efficiently I'd be lining up.
solyin · 2026-10-26 03:33:59 UTC
"Affordable" and "IBM" don't belong in the same sentence together, but they're happy to sell you their inference-capable hardware if you've got the wallet for it.
jacquesm · 2026-10-26 03:37:29 UTC
I don't agree with that. They made the PC revolution possible by making hardware that was at least affordable enough to be bought by a large number of businesses and individuals. It also set the stage for the clone market. I'm pretty sure that if IBM really set their mind to it that they could do this.
That doesn't detract from your overall point that they tend to make overpriced stuff, but I don't think they have to do that. Besides, what's overpriced? NVidia's GB300 workstation sells for 95K euros, a single RTX 6000 is now 20K. Surely even IBM can compete at that level.
quailfarmer · 2026-10-26 03:42:03 UTC
They haven’t been _that_ IBM for decades
jacquesm · 2026-10-26 03:50:49 UTC
I know. But: they have gobs of money, they still have some talent, they still have a hardware department that I'm pretty sure would jump at a chance to give NV a run for their money. This would require a massive long term investment and a commitment that I'm not sure they would have the stomach for.
alienbaby · 2026-10-26 04:31:14 UTC
"they have gobs of money, "
They do? They are a spec in the ocean relative to big players offering inference and training their own models. Pretty sure one thing they are struggling with is the money to compete successfully in the ai game this time around.
jacquesm · 2026-10-26 04:35:19 UTC
10 billion give or take, market cap 250 billion, 250K employees. That's not a 'spec in the ocean' by normal standards, and IBM is a mature company, not an inflated balloon.
dansquizsoft · 2026-10-26 03:44:50 UTC
> They made the PC revolution possible
Yeah they were only half a decade late to the PC revolution back in the day...
wmf · 2026-10-26 03:58:38 UTC
They weren't really late. In many people's minds IBM validated the concept of the PC.
jacquesm · 2026-10-26 04:02:04 UTC
When that happened I'd been working at Tandy (EU version of Radio Shack) for a while and even though we sold computers they were not taken serious until IBM came along and then suddenly we couldn't supply them fast enough.
spopejoy · 2026-10-26 04:22:18 UTC
Was Tandy only EU? In the TRS-80 days the name came up a lot, although maybe just because of the acronym ... 6809E was a good time
jacquesm · 2026-10-26 04:25:50 UTC
Yes, that was the first time I had a 'proper' processor to play with. A Dragon 32 (CoCo clone from the UK).
I've never seen Tandy used outside of Europe but then again, I had not traveled as much then as now so it's definitely possible they used the brand on other continents as well. I wonder how many trees were cut for those catalogs :)
shakna · 2026-10-26 06:50:48 UTC
And AU. Dick Smith started out selling Tandy's, before switching to their own clones.
xaqfox · 2026-10-26 21:51:48 UTC
It was only a difference in the store names. They were sold at Radio Shack stores in US. Tandy, originally a leather company, bought Radio Shack in the 60s).
rnd0 · 2026-10-26 03:45:18 UTC
That was completely accidental and they themselves tried to close the barn door with the PS/2 line in 1987 but at that point it was too late.
jacquesm · 2026-10-26 03:56:05 UTC
True, even so: They still make some pretty impressive hardware. They have access to fab capacity, they know how to make chips. They have the capital. If they really wanted they could do it.
solyin · 2026-10-26 04:25:38 UTC
In what way is the IBM Spyre not what you're asking for?
jacquesm · 2026-10-26 04:30:27 UTC
In a way it is, but I would see that as a starting point, not as the ready solution. They'd need to do what NV has done for their eco-system: fantastic libraries, a credible future upgrade path, easy ordering, manufacturing partnerships, the works. I figure besides the performance gap that they are at least 5 to 10 years behind NV with their present offering. But NVidia is dropping stitches and IBM should take advantage of that.
I think IBM sees the Spyre as a reason to sell more Power hardware but that's just not where the industry is right now and I doubt it will be enough to create sufficient demand. Who knows, I don't have any insight in what they're doing there, but there is massive gap between A GB300 workstation and what IBM currently offers. At the same time: the RAM crunch hits all parties and that's a major obstacle as well.
solyin · 2026-10-26 04:46:21 UTC
Yeah. For all the jokes we make about paperclip maximizer, we've figured it out. It's not paperclips, it's RAM.
That's the thing though. Software is cheap with AI, but Nvidia has Cuda. If software doesn't matter anymore (it does) because it's so cheap now, then Nvidia's moat evaporates to IBM's Spyre. In reality "just a little bit of software" has inertia and cannot be dislodged quickly, quietly, or easily. cf Microsoft.
jacquesm · 2026-10-26 04:49:13 UTC
That. And not starting every sale with 'call us' where they first want you to tell them how much you can afford to spend. I buy a fair amount of gear and if the sales process starts with some sleazeball sales guy instead of a mouseclick on a local webshop then I'm out.
As for CUDA: I think CUDA is a formidable moat but I also think that that is precisely the kind of thing where the likes of IBM could (if they really wanted to) make things happen. They're already in the space with Redhat and they have ample knowledge about drivers and low level code. Putting together a team that either makes a drop-in replacement or alternatively, that starts supporting their custom hardware for the big players in the inference space (VLLM, llama.cpp and Torch) would make all the difference. They'd have to keep doing it though, even if the initial results were lower than expected because that market is not going to turn on a dime.
j16sdiz · 2026-10-26 04:28:31 UTC
It's the "PC-compatible" hardware make those affordable, not PC.
exac · 2026-10-26 04:51:39 UTC
Respectfully, those people don't work at IBM anymore. The organizational memory doesn't last that long.
jacquesm · 2026-10-26 04:55:26 UTC
You'd think that with a 250K workforce and an active line of hardware including an AI accelerator that those people are still working there. I could be wrong, but someone has to be doing that work unless IBM in turn outsources it.
numpad0 · 2026-10-26 03:44:22 UTC
Yes but unironically? IBM should be what NVIDIA would be competing with their NVL72 and DGX products. It doesn't make sense that IBM is just watching NVIDIA take its lunch.
verdverm · 2026-10-26 04:27:06 UTC
> It doesn't make sense that IBM is just watching NVIDIA take its lunch.
If you have ever worked at IBM, it makes total sense. Completely dysfunctional middle management and on up.
jacquesm · 2026-10-26 04:45:10 UTC
I think they will need a far more aggressive management before such a thing could happen. But I can dream about it.
mbreese · 2026-10-26 05:08:34 UTC
I didn't think that IBM had gear that mortals would want to run. If you're not running a mainframe, what do they have to sell you? They still sell Power servers and it looks like they have an AI card that can be used on both. But, if we're talking about the IBM CEO talking about on-premises, what are they really talking about hardware-wise? It's not something they would sell, is it?
This is more of a statement for their consulting customers, right?
jhickok · 2026-10-26 14:00:57 UTC
My guess is if you want to run AI on prem and do it with IBM gear it would be via their x86 servers + OpenShift AI via IBM Fusion HCI.
mbreese · 2026-10-26 14:15:52 UTC
I didn’t think they still sold x86 servers.
I can picture that they’d have the software side sorted out with RedHat. But I don’t know what hardware they’d push to support local on-premises models. It looks like when you buy Fusion HCI, you can choose from Dell or Lenovo as the underlying hardware supplier.
CookieCrisp · 2026-10-26 03:13:03 UTC
Good enough for the old world, but I'm not so sure it's good enough for what's coming.
tomrod · 2026-10-26 03:14:55 UTC
What do you think is coming that isn't already in the old world?
Cerebras with Claude 8?
LoganDark · 2026-10-26 03:36:56 UTC
OpenAI most likely pushed Cerebras into a contractual agreement not to do business with Anthropic through 2030. ("Most likely" because the names of the companies Cerebras is contractually obligated not to sell to are redacted in public filings)
desolate_muffin · 2026-10-26 04:11:32 UTC
Imagine a regulatory framework so weak that a company feels comfortable to do something this anti-competitive
actionfromafar · 2026-10-26 14:42:33 UTC
Maybe bribes is the framework? It's hard to tell these days.
collinmcnulty · 2026-10-26 03:17:07 UTC
Can we please stop giving weight to CEOs talking their book? Like of course IBM wants you to think this, it would be great for them if it’s true. And maybe it is true! But IBM’s CEO saying it’s true tells me just as much as if a frontier lab said it’s not: nothing.
MattDamonSpace · 2026-10-26 03:23:43 UTC
It’s a conversation point off which a discussion can be had
gjvc · 2026-10-26 03:32:03 UTC
the only credible "conversation" is "well he would say that, wouldn't he?"
thank you, I was trying to re-find this the other day
lordgrenville · 2026-10-26 06:06:45 UTC
Great term. Plus the related "Politician Said a Thing!" journalism
epsteingpt · 2026-10-26 03:27:49 UTC
But you know - it would be funny if somehow Open Weights are the thing that completely save IBM
SoftTalker · 2026-10-26 03:34:27 UTC
Linux already served as a life-preserver for them also.
slashtrends · 2026-10-26 07:48:26 UTC
It was a brilliant move at the time for IBM, and has helped them shift their business more into providing services and support.
I imagine they are panicking a little with LLMs reducing the reliance on external companies for software support, so they are really hoping to shift it to on-prem AI, which is where the bias in his statement comes from.
SoftTalker · 2026-10-26 16:01:38 UTC
I think AI could very well support a return to on-prem computing.
gjvc · 2026-10-26 03:32:31 UTC
"we are delighted..." said a spokesman
trollbridge · 2026-10-26 03:21:50 UTC
While he thinks most businesses can build useful AI infrastructure in just two or three racks, he feels many will struggle to get it running.
“The question becomes: how do you do that?” he said. “The question that the enterprise struggles with is: ‘Where are my kernel engineers? Where are the people who worry about the underlying network? Who can make sure that the model is clustered on GPUs that are close to each other in network proximity? Who can make sure that you’re doing the right caching of results so you get an incredible response?’”
Has he heard of “agentic AI”? You simply fire up Claude or ChatGPT and ask it how to do all of the above.
Buttons840 · 2026-10-26 03:34:47 UTC
That's based on the idea that humans can learn and adapt and solve problems, and that's not what companies want to think about when they're seeking a profitable transaction such as hiring.
LoganDark · 2026-10-26 03:35:08 UTC
Do not do that if you can't prompt it correctly. It is a potential disaster waiting to happen. LLMs are likely to pull things off the shelf without real defense in depth and that's a problem. Maybe slightly less if it's only in an internal network, but still.
trollbridge · 2026-10-26 12:51:52 UTC
The whole point of this was for internal usage.
LoganDark · 2026-10-26 18:42:13 UTC
It can be very dangerous to let people loose with such advice, FWIW. They don't necessarily know what "internal usage" actually means, or how important it is to keep it that way as long as you don't have a real expert to at least help direct the LLMs.
liorn · 2026-10-26 04:35:15 UTC
It's not a one time thing though - onprem inference requires continuous monitoring and tuning to get the most out of your GPUs. Looking at 25-50% performance benefits. Today, it does require dedicated Inference Engineers...
trollbridge · 2026-10-26 11:15:23 UTC
That sounds like a task that can be automated.
liorn · 2026-10-26 12:18:56 UTC
It's not that easy.
The optimal inference configuration varies according to the model, the specific traffic (which changes in time), and the specific hardware it's running to. The "knowledge" for it doesn't exist yet.
Usually in similar scenarios an agent will try to experiment with various configurations and get to a good result. However, you can't have an agent test out different inference serving configurations/architectures, since every experiment might involves 10-60 minutes of spinning up a model (on very expensive GPUs)...
trollbridge · 2026-10-26 12:51:36 UTC
That sounds like a task LLMs will soon be able to take over.
jacquesm · 2026-10-26 03:28:55 UTC
It's absolutely true. I have three different setups here, a rig with 8 gpus that I use in either one of two modes (all 8, very heavy model or just one much lighter) and two DGX sparks. I haven't used a paid model in months. Sure, they use a few more tokens and the results may be slightly less good (but this doesn't really matter because I'm still in the loop so it is easy to correct). But the fact that I'm not sharing my work with the paid providers is much more to my liking than having big brother looking over my shoulder at every step and occasionally giving me the 'I can't let you do that, Jacques' treatment.
otterley · 2026-10-26 03:33:43 UTC
How much did you shell out for this setup? What’s the TTFT and the output tokens/sec rate? How much electricity does it consume?
jacquesm · 2026-10-26 03:39:50 UTC
Working on a blog post to document it all, but that will take a few more days. The short version: the big rig is probably $25K, ttft is dependent on how much prefill you have but typically a couple of seconds (2K prefill/sec in one setup, about 500 in the other), tgs 100 to 150 or so depending on what it is doing and how well the speculative decoder works with the current context. Electricity is pretty bad, full tilt it's about 2500W or so but I've power limited it to 1500 to keep it civil. It is not cheaper than using a frontier lab (or at least, I don't think it is), but the independence bit is the important part for me and I don't care if that carries a surcharge, even if it is a substantial one.
selcuka · 2026-10-26 04:09:01 UTC
> It is not cheaper than using a frontier lab
Are you comparing to subscription prices or API prices? Because if it's the former it won't last forever.
jacquesm · 2026-10-26 04:38:37 UTC
True, they're subsidized, I'm not. But I don't have their utilization so I figure it would always be more expensive to run this locally.
glassresearch · 2026-10-26 04:13:20 UTC
I'm not saying your approach is wrong - but for my workloads, it makes a lot more sense to just pay for a decade (!!!) of Claude Code 20x Max.
calmingsolitude · 2026-10-26 04:32:57 UTC
LLM subscriptions are heavily subsidized. I’d be surprised if Claude 20x costed the same after a decade.
distances · 2026-10-26 07:25:47 UTC
Inference prices are going only down. It could be that some top subscription costs more in a decade, but I fully believe a full array of competent models will be extremely cheap.
latentsea · 2026-10-26 07:27:55 UTC
Doesn't deepseek make $38 per GPU-hour? As in they can actually serve profitably?
petu · 2026-10-26 08:16:37 UTC
Deepseek doesn't have subscriptions, you can only pay per token / API prices.
pimeys · 2026-10-26 08:58:53 UTC
Yes but the API prices are very low. So much that it is hard to spend over 20 dollars per month as an individual no matter how much you use it.
latentsea · 2026-10-26 10:55:49 UTC
How are their API prices compared to others?
Gareth321 · 2026-10-26 07:40:58 UTC
The inference is profitable. It’s the training which is really expensive and not being reflected in the subscription prices.
jacquesm · 2026-10-26 04:38:01 UTC
If I want to go out of business that is what I would do. Most of the data I have here could never be shipped to a service provider, I'd be in lawsuits for the next decade or so and die bankrupt.
greesil · 2026-10-26 04:48:08 UTC
Okay man I've got to ask, what kind of data are we talking about? Legal? Medical?
I've heard that there are legal software platforms like Clio that offer AI products, not sure what just eavesdropping on some lawyers at the bar.
Companies pretty much give us the full overview of what they have from a technical perspective (and often much more than that), we sign NDAs with investors that have serious teeth in them. The data they share with us would not be disclosed under any other circumstance and even our reports are shared only with the clients.
It's a nice little business. I'm not going to risk offending my customers for convenience or to save some money.
greesil · 2026-10-26 05:29:40 UTC
Holy fuck you have a lot of karma
disgruntledphd2 · 2026-10-26 06:34:45 UTC
He's been here for a loooonnnggg time. I'm pretty sure I found HN through one of his blog posts.
4gotunameagain · 2026-10-26 09:07:38 UTC
You can estimate the amount of time by the domains he's been squatting all these years.
dekhn · 2026-10-26 18:36:27 UTC
Isn't he the guy who posted about his LEGO-sorting computer vision project? That post alone taught me things that increased my career opportunities significantly.
robotmay · 2026-10-26 09:50:12 UTC
On a tangent: this is a really good website. I understood what you do and how you do it by reading down one page. There's a lot of projects/companies I immediately tune out from because I can't actually figure out what they do amongst all the buzzwords, animations, and excess scrolling in every dimension.
Asyakguen · 2026-10-26 08:58:42 UTC
That NDA case is the real argument, more than price. Token prices keep dropping anyway, but "this data can never leave the building" doesn't change with a price cut ;)
tjwebbnorfolk · 2026-10-26 04:42:42 UTC
The bet we (self-hosters) are making is that once these companies IPO, they will be forced to do whatever they have to do to make money. i.e. price increases, more throttling of subscription non-usage-based accoutns, etc.
They are subsidizing all of us to the tune of $billions in losses per month. That can't go on for a decade. Probably not even for another 18 months.
I still use all the tokens Claude will give me on the max plan. But I also run deepseek v4 locally for overflow and for... contingencies.
lelanthran · 2026-10-26 08:53:58 UTC
> The bet we (self-hosters) are making is that once these companies IPO, they will be forced to do whatever they have to do to make money. i.e. price increases, more throttling of subscription non-usage-based accoutns, etc.
But, there is no need to make that bet - when prices increase, I'd look into renting a previous-gen GPU on a cloud provider. If the online GPU providers raise prices, then I would look into purchasing for on-site.
Until that happens, I'm not making any bets, because if all the token providers go bust GPU rentals are going to fall in price, and if they don't go bust after raising prices, I can still get years of GPUs for self-hosting at the price of a self-owned rig.
tjwebbnorfolk · 2026-10-26 16:16:14 UTC
Well there are risks to being part of a crowd that's all compelled to make the same decision at the same time. Certain risks can only be transferred, not reduced in absolute terms.
It's not economical today to run my own hardware, but the insurance is worth it to me in the sense that if my bet is wrong, I'm still okay with paying extra for the protection.
benoau · 2026-10-26 04:49:59 UTC
If you know you will pay Claude a few thousand bucks a year a decade adds up too, owning the hardware could just as well end up cheaper because you can sell it or as others have alluded the price to rent could get much worse.
jacquesm · 2026-10-26 04:51:29 UTC
Heh, the hardware I bought I could easily sell for 2x so you may actually end up making money. Though I hope that that won't last.
dylan604 · 2026-10-26 05:10:32 UTC
Would you really expect to be using the same hardware over the course of that decade? One of the biggest drawbacks to using on-prem hardware is keeping up with the upgrades. Leasing upgrade cycles are a thing for businesses, but for someone at home it seems like later in the decade the local system will be showing its age.
jacquesm · 2026-10-26 05:38:37 UTC
The 3090's I have would sell for more than new today... for the next couple of years or so it will be pretty bad on the hardware front. Until there is more RAM manufacturing capacity and NV has a credible competitor I don't think it will get better.
segmondy · 2026-10-26 05:04:43 UTC
My local model never says no. I never get banned. If I want to do security stuff, reverse engine, there it goes! Do I wanna chat about politics? No problem! Do I want it to analyze medical stuff? I don't get to hear that it's not a doctor and I should call 911. Do I want to explore legal docs, no problem! There are lots of things that cloud models will say no to! I read that don't even agree to help with advanced biology and chemistry topic much. I also never run out of tokens, no 5 hr limit. I can run it 24/7. I don't run out of tokens. I can run 16 sub agents at the same time, non stop! 1-4 billion tokens a month? No problem. Any agent harness I want! No restrictions. Privacy all day. My own true system prompt, not something someone has determined, no filters. Furthermore, there are many open weight cloud models that don't have equivalent online. For a while before cloud models got good at math, the best model at math was DeepSeekv2 Math and there was no API. The only way to use it was to run local. All sorts of audio models, image/video models, etc. No multiple subscription needed. Private UI for my family, kids included (I can guard what kids infer). Got a friend that really needs some tokens? I give em access through private VPN. Telling someone that they are paying for decades of Claude for local AI is like telling someone they are paying for decades of public transport when they buy a car. Sure, you are right, but ...
jacquesm · 2026-10-26 05:06:10 UTC
Exactly, that's the spirit. I pool the resources with some friends as well.
latentsea · 2026-10-26 07:26:21 UTC
> My local model never says no. I never get banned.
So much this. Plus everything else you mentioned. The freedom and control is real.
Gareth321 · 2026-10-26 07:40:13 UTC
Also no political censorship, and it doesn’t report you to the police when it suspects you did something illegal. That really happened, by the way.
Now I want a local LLM server :( $25k is just so much money.
cft · 2026-10-26 08:31:55 UTC
MacBooks with large RAM and optimized for local models should be Apple's medium term goal.
Comments
That doesn't detract from your overall point that they tend to make overpriced stuff, but I don't think they have to do that. Besides, what's overpriced? NVidia's GB300 workstation sells for 95K euros, a single RTX 6000 is now 20K. Surely even IBM can compete at that level.
They do? They are a spec in the ocean relative to big players offering inference and training their own models. Pretty sure one thing they are struggling with is the money to compete successfully in the ai game this time around.
Yeah they were only half a decade late to the PC revolution back in the day...
I've never seen Tandy used outside of Europe but then again, I had not traveled as much then as now so it's definitely possible they used the brand on other continents as well. I wonder how many trees were cut for those catalogs :)
I think IBM sees the Spyre as a reason to sell more Power hardware but that's just not where the industry is right now and I doubt it will be enough to create sufficient demand. Who knows, I don't have any insight in what they're doing there, but there is massive gap between A GB300 workstation and what IBM currently offers. At the same time: the RAM crunch hits all parties and that's a major obstacle as well.
That's the thing though. Software is cheap with AI, but Nvidia has Cuda. If software doesn't matter anymore (it does) because it's so cheap now, then Nvidia's moat evaporates to IBM's Spyre. In reality "just a little bit of software" has inertia and cannot be dislodged quickly, quietly, or easily. cf Microsoft.
As for CUDA: I think CUDA is a formidable moat but I also think that that is precisely the kind of thing where the likes of IBM could (if they really wanted to) make things happen. They're already in the space with Redhat and they have ample knowledge about drivers and low level code. Putting together a team that either makes a drop-in replacement or alternatively, that starts supporting their custom hardware for the big players in the inference space (VLLM, llama.cpp and Torch) would make all the difference. They'd have to keep doing it though, even if the initial results were lower than expected because that market is not going to turn on a dime.
If you have ever worked at IBM, it makes total sense. Completely dysfunctional middle management and on up.
This is more of a statement for their consulting customers, right?
I can picture that they’d have the software side sorted out with RedHat. But I don’t know what hardware they’d push to support local on-premises models. It looks like when you buy Fusion HCI, you can choose from Dell or Lenovo as the underlying hardware supplier.
Cerebras with Claude 8?
https://karlbode.com/ceo-said-a-thing-journalism/
I imagine they are panicking a little with LLMs reducing the reliance on external companies for software support, so they are really hoping to shift it to on-prem AI, which is where the bias in his statement comes from.
The optimal inference configuration varies according to the model, the specific traffic (which changes in time), and the specific hardware it's running to. The "knowledge" for it doesn't exist yet.
Usually in similar scenarios an agent will try to experiment with various configurations and get to a good result. However, you can't have an agent test out different inference serving configurations/architectures, since every experiment might involves 10-60 minutes of spinning up a model (on very expensive GPUs)...
Are you comparing to subscription prices or API prices? Because if it's the former it won't last forever.
I've heard that there are legal software platforms like Clio that offer AI products, not sure what just eavesdropping on some lawyers at the bar.
Companies pretty much give us the full overview of what they have from a technical perspective (and often much more than that), we sign NDAs with investors that have serious teeth in them. The data they share with us would not be disclosed under any other circumstance and even our reports are shared only with the clients.
It's a nice little business. I'm not going to risk offending my customers for convenience or to save some money.
They are subsidizing all of us to the tune of $billions in losses per month. That can't go on for a decade. Probably not even for another 18 months.
I still use all the tokens Claude will give me on the max plan. But I also run deepseek v4 locally for overflow and for... contingencies.
But, there is no need to make that bet - when prices increase, I'd look into renting a previous-gen GPU on a cloud provider. If the online GPU providers raise prices, then I would look into purchasing for on-site.
Until that happens, I'm not making any bets, because if all the token providers go bust GPU rentals are going to fall in price, and if they don't go bust after raising prices, I can still get years of GPUs for self-hosting at the price of a self-owned rig.
It's not economical today to run my own hardware, but the insurance is worth it to me in the sense that if my bet is wrong, I'm still okay with paying extra for the protection.
So much this. Plus everything else you mentioned. The freedom and control is real.
Now I want a local LLM server :( $25k is just so much money.