For the past couple of months, I have been exploring the practicalities of locally hosted AI models and applications. My primary objective to see what we can do with things we already own, or find cost-effective alternatives to a $5,000 Nvidia GPU.
In an earlier video, I looked at a new Intel GPU with 32GB that retailed for $1,300. One of these pulled GPUs costs about half that. I got mine from Server Parts Deals who is also selling these cards on Amazon (compensated affiliate link).
Although the V100 lacks some of the specialized features found in modern RTX cards that newer models will begin to depend on, for now it provides ample performance from popular large language models (LLMs) like Qwen 3.8 27B.
Integrating a data center card into a consumer environment requires several specific modifications. These cards are designed for server racks where external fans force air through the chassis, meaning the card itself lacks active cooling. To address this, I sourced a specialized blower fan (compensated affiliate link) from an online marketplace to attach to the rear of the unit. Because I do not currently have a dedicated desktop for this project, I opted to use a small form factor PC with the GPU connected via an OcuLink interface.
The power requirements added a layer of complexity to the assembly. Data center cards use different power pin configurations than standard consumer hardware, necessitating a specific adapter cable (compensated affiliate link). Furthermore, because the external enclosure needs to power both the GPU and the high-draw blower fan, multiple power leads and adapters were required to ensure the cooling system remained operational at a constant speed. While some enthusiasts have developed variable-speed cooling solutions, I focused on the minimum requirements necessary to maintain stable operating temperatures under load.
The software installation process presented its own set of challenges. Initial attempts to configure the V100 on Windows 11 were unsuccessful; despite installing the correct data center drivers, the system consistently returned a hardware error, likely due to the OCuLink connection or BIOS issues. Consequently, I shifted the project to a Linux environment running Ubuntu 26.04. To streamline the configuration and driver installation, I utilized ChatGPT’s command-line Codex application to manage the technical hurdles and document the installation for future reference. This approach allowed for a more efficient setup of the necessary dependencies and libraries.
Once operational, I tested the system using the llama.cpp web interface, starting with the Qwen 3.8 27B dense model. When tasked with summarizing a few PDF documents, the V100 achieved a pre-fill speed of 50 tokens per second and a generation speed of 45 tokens per second. The 32GB of VRAM allowed for a large context window of over 96,000 tokens, enabling the model to process a 70-page document without exceeding the hardware’s limits. Even as the context window filled to 80% capacity, the generation speed remained a functional 24 to 30 tokens per second while drawing approximately 270 watts of power.
Performance shifted significantly when testing a “mixture of experts” model, such as Gemma 4 26B-A4B. Because these models only activate a fraction of their parameters for any given task, the processing speeds were much higher. The system reached pre-fill speeds of 1,100 tokens per second and output generation at over 90 tokens per second. These results indicate that for text-based local AI tasks, this older data center hardware remains competitive with much more expensive modern alternatives.
Beyond text, I tested the card’s ability to generate images and video using ComfyUI. While the V100 can run modern models like Flux for images LTX 2.5 for video, the process is considerably slower than on contemporary hardware. Generating a single 1024×1024 image took approximately one minute, while a five-second video clip at 720p required three and a half minutes of processing. By comparison my Intel card could do the same job in under a minute and a half. Long videos were much slower vs. the newer Intel card in my testing.
Newer cards from Intel and Nvidia have specific optimizations for these workflows that the V100 lacks, as it was originally released in 2017. For example, the exceptional MiniMax H3 model was too slow to be usable on the V100.
There are clear trade-offs when using a decade-old data center GPU. Many emerging AI features expect hardware capabilities that simply did not exist when this card was engineered. While it handles current language models with ease, it may not support the next generation of multimodal AI tools as effectively. However, for those specifically seeking a large context window and high VRAM capacity for local text processing on a budget, these secondary-market cards offer a viable path forward.