jaykrown@lemmy.world to Technology@lemmy.worldEnglish · 3 days agoDeepSeek Permanently Reduces The Price Of Its Flagship V4 Model By 75 Percenttech.yahoo.comexternal-linkmessage-square138linkfedilinkarrow-up1519arrow-down122
arrow-up1497arrow-down1external-linkDeepSeek Permanently Reduces The Price Of Its Flagship V4 Model By 75 Percenttech.yahoo.comjaykrown@lemmy.world to Technology@lemmy.worldEnglish · 3 days agomessage-square138linkfedilink
minus-squareMwa@thelemmy.clublinkfedilinkEnglisharrow-up6arrow-down1·2 days agoStill gonna self host it instead (maybe)
minus-squareTja@programming.devlinkfedilinkEnglisharrow-up2·2 days agoHow are they running it? Doesn’t the model have to fit in (V)RAM? Does Nvidia have such huge memories in the H cards?
minus-squareBlackLaZoR@lemmy.worldlinkfedilinkEnglisharrow-up4·2 days agoThere’s tech for splitting model to run on multiple cards, but it requires really fast interconnect between GPUs.
minus-squareTaasz/Woof@lemmy.blahaj.zonelinkfedilinkEnglisharrow-up2·2 days agoLots of GPUs together.
minus-squareboonhet@sopuli.xyzlinkfedilinkEnglisharrow-up1·2 days agoFor self hosting it essentially needs to fit in VRAM + RAM but it’ll take a lot of CPU for the part in RAM Deepseek probably uses those big fancy H cards and not one but several together to increase VRAM.
Still gonna self host it instead (maybe)
FYI the flash model is ~158 GB
How are they running it? Doesn’t the model have to fit in (V)RAM? Does Nvidia have such huge memories in the H cards?
There’s tech for splitting model to run on multiple cards, but it requires really fast interconnect between GPUs.
Lots of GPUs together.
For self hosting it essentially needs to fit in VRAM + RAM but it’ll take a lot of CPU for the part in RAM
Deepseek probably uses those big fancy H cards and not one but several together to increase VRAM.
The destiled models?