SocialistVibes01@lemmy.ml to Linux@lemmy.mlEnglish · edit-214 hours agoWhich specs are as low as reasonable possible for local LLM models? Do you recommend some distro in particular?message-squaremessage-square14fedilinkarrow-up123arrow-down111file-text
arrow-up112arrow-down1message-squareWhich specs are as low as reasonable possible for local LLM models? Do you recommend some distro in particular?SocialistVibes01@lemmy.ml to Linux@lemmy.mlEnglish · edit-214 hours agomessage-square14fedilinkfile-text
minus-squareinfinitevalence@discuss.onlinelinkfedilinkEnglisharrow-up1arrow-down1·13 hours agoI disagree Gemma 4 easily runs inside of a 16g GPU and is really pretty fast.
minus-squaremeowmeow@quokk.aulinkfedilinkEnglisharrow-up2·13 hours agoFast is relative. I’m also commenting on the cost of the entire system, not just the gpu, fyi
minus-squareinfinitevalence@discuss.onlinelinkfedilinkEnglisharrow-up4arrow-down1·13 hours agoThat’s fair, but nearly any modern CPU at least 32gb of RAM and a current GPU with 16gb is plenty. No need for a 4k system when a 1k-1.5k will do it. If you’re willing to Frankenstein things some of the used AI/ML/mining cards can be a decent value.
minus-squaremeowmeow@quokk.aulinkfedilinkEnglisharrow-up2arrow-down1·12 hours agoYes, but when you compare it to codex and Claude though, it’s significantly slower. Especially over time. Better crank that AC. I think in a few years we will have current cloud levels running pretty efficiently on current computers.
I disagree Gemma 4 easily runs inside of a 16g GPU and is really pretty fast.
Fast is relative. I’m also commenting on the cost of the entire system, not just the gpu, fyi
That’s fair, but nearly any modern CPU at least 32gb of RAM and a current GPU with 16gb is plenty. No need for a 4k system when a 1k-1.5k will do it.
If you’re willing to Frankenstein things some of the used AI/ML/mining cards can be a decent value.
Yes, but when you compare it to codex and Claude though, it’s significantly slower. Especially over time. Better crank that AC.
I think in a few years we will have current cloud levels running pretty efficiently on current computers.