Tag: Local AI

The Cheapest 32GB Nvidia GPU You Can Buy for Local AI (Nvidia Tesla V100)

For the past couple of months, I have been exploring the practicalities of locally hosted AI models and applications. My primary objective to see what we can do with things we already own, or find cost-effective alternatives to a $5,000 Nvidia GPU.

In my latest video, I take a look at a relatively affordable 32GB GPU solution : an Nvidia V100 GPU pulled from a datacenter!

In an earlier video, I looked at a new Intel GPU with 32GB that retailed for $1,300. One of these pulled GPUs costs about half that. I got mine from Server Parts Deals who is also selling these cards on Amazon (compensated affiliate link).

Although the V100 lacks some of the specialized features found in modern RTX cards that newer models will begin to depend on, for now it provides ample performance from popular large language models (LLMs) like Qwen 3.8 27B.

Integrating a data center card into a consumer environment requires several specific modifications. These cards are designed for server racks where external fans force air through the chassis, meaning the card itself lacks active cooling. To address this, I sourced a specialized blower fan (compensated affiliate link) from an online marketplace to attach to the rear of the unit. Because I do not currently have a dedicated desktop for this project, I opted to use a small form factor PC with the GPU connected via an OcuLink interface.

The power requirements added a layer of complexity to the assembly. Data center cards use different power pin configurations than standard consumer hardware, necessitating a specific adapter cable (compensated affiliate link). Furthermore, because the external enclosure needs to power both the GPU and the high-draw blower fan, multiple power leads and adapters were required to ensure the cooling system remained operational at a constant speed. While some enthusiasts have developed variable-speed cooling solutions, I focused on the minimum requirements necessary to maintain stable operating temperatures under load.

The software installation process presented its own set of challenges. Initial attempts to configure the V100 on Windows 11 were unsuccessful; despite installing the correct data center drivers, the system consistently returned a hardware error, likely due to the OCuLink connection or BIOS issues. Consequently, I shifted the project to a Linux environment running Ubuntu 26.04. To streamline the configuration and driver installation, I utilized ChatGPT’s command-line Codex application to manage the technical hurdles and document the installation for future reference. This approach allowed for a more efficient setup of the necessary dependencies and libraries.

Once operational, I tested the system using the llama.cpp web interface, starting with the Qwen 3.8 27B dense model. When tasked with summarizing a few PDF documents, the V100 achieved a pre-fill speed of 50 tokens per second and a generation speed of 45 tokens per second. The 32GB of VRAM allowed for a large context window of over 96,000 tokens, enabling the model to process a 70-page document without exceeding the hardware’s limits. Even as the context window filled to 80% capacity, the generation speed remained a functional 24 to 30 tokens per second while drawing approximately 270 watts of power.

Performance shifted significantly when testing a “mixture of experts” model, such as Gemma 4 26B-A4B. Because these models only activate a fraction of their parameters for any given task, the processing speeds were much higher. The system reached pre-fill speeds of 1,100 tokens per second and output generation at over 90 tokens per second. These results indicate that for text-based local AI tasks, this older data center hardware remains competitive with much more expensive modern alternatives.

Beyond text, I tested the card’s ability to generate images and video using ComfyUI. While the V100 can run modern models like Flux for images LTX 2.5 for video, the process is considerably slower than on contemporary hardware. Generating a single 1024×1024 image took approximately one minute, while a five-second video clip at 720p required three and a half minutes of processing. By comparison my Intel card could do the same job in under a minute and a half. Long videos were much slower vs. the newer Intel card in my testing.

Newer cards from Intel and Nvidia have specific optimizations for these workflows that the V100 lacks, as it was originally released in 2017. For example, the exceptional MiniMax H3 model was too slow to be usable on the V100.

There are clear trade-offs when using a decade-old data center GPU. Many emerging AI features expect hardware capabilities that simply did not exist when this card was engineered. While it handles current language models with ease, it may not support the next generation of multimodal AI tools as effectively. However, for those specifically seeking a large context window and high VRAM capacity for local text processing on a budget, these secondary-market cards offer a viable path forward.

Local AI On an Old 8 GB GPU?

The current landscape of artificial intelligence often suggests that high-end, increasingly expensive hardware is a prerequisite for meaningful performance. However, much can be accomplished using hardware that many people already own. I recently spent time working with a five-year-old gaming laptop equipped with 16 gigabytes of system memory and an Nvidia 3070 GPU with 8 gigabytes of video RAM. While this is a far cry from a modern workstation, it provides a capable environment for running local language models if the settings are adjusted correctly.

Check out what it can do in my latest local AI video!

The primary challenge with an 8-gigabyte GPU is balancing the size of the model with the context length, which is essentially the memory the model uses to track a conversation or analyze a document. For my testing, I utilized the Qwen-3-8B model, an 8-billion parameter model that occupies about 5.76 gigabytes of memory. This leaves a small but workable margin for context.

In the LM Studio application, I set the context limit to 25,600 tokens (compared to 1 million tokens on frontier models like ChatGPT and Gemini). To keep the performance steady at around 35 tokens per second, I employed several technical tweaks, including unified KV cache, flash attention, and 8-bit KV cache quantization. This quantization is particularly important because it compresses the context memory, allowing more data to fit onto the GPU.

To test the practical utility of this setup, I tasked the model with analyzing a term paper I wrote in college nearly 30 years ago. Although the text file was small, the process of analyzing it for patterns can consume significant memory. I inserted several “traps” into the text—phrases like “it’s a trap” and “Han shot first”—to see if the model would notice them while summarizing the content. The model processed the prompt in a few seconds, provided an accurate summary without hallucinations, and correctly identified the out-of-context phrases.

Moving beyond simple text analysis, I attempted to use the laptop as an AI server for more complex tasks like coding. By connecting a secondary machine to the laptop using an open-source coding harness called OpenCode, I tried to extract a list of public officials from a website and format the data into a CSV file. This is where the limitations of an 8-gigabyte system become more apparent. The model struggled to process the raw HTML of the webpage due to the limited context window. Once I simplified the input to plain text, the model successfully formatted the data, including identifying vacancies in a specific district. It required more manual intervention than a larger 70-billion parameter model might, but it eventually achieved the desired result.

Coding tasks showed similar limitations. When I asked the model to write a Space Invaders game in HTML, it produced a functional interface where a player character could move, but it failed to include enemies or game logic on the first attempt. Even after a second prompt to fix the errors, the logic remained incomplete. Smaller models often lack the reasoning capabilities required for complex, one-shot coding successes, frequently requiring a more iterative approach and careful management of the conversation history.

Image generation is also possible on this hardware. Using the SDXL model through a templated interface, I was able to generate local images in approximately 30 to 32 seconds each. While this setup cannot handle the video generation tasks possible on high-end cards, the static image quality is respectable. Achieving the right output required some trial and error with prompts; for instance, the model initially struggled to generate a dog in a cockpit until I removed gender-specific language that seemed to be steering it toward human subjects.

The efficiency of these local models has improved significantly over the last year. For those who may have tried local AI in the past and found it lacking, the current generation of small-parameter models offers much more utility on aging hardware.

While 8 gigabytes of video memory requires careful optimization and realistic expectations regarding context and complexity, it remains a viable entry point for those looking to keep their data local and make use of the machines they already have. If you’re willing to give up performance, AI applications like LM Studio can also use system memory which can allow for larger context memory and AI models.

A 32GB GPU for a Third of the Price? Intel’s B70 for Local AI

Local AI has made tremendous strides over the last few months and I’ve been looking at the most affordable ways to implement it, like using an old Mac, or pairing two GPUs together to maximize memory. In my latest video, I sought out the lowest cost 32GB GPU solution I could find and landed on the Intel B70 (compensated affiliate link).

See it in action here!

While prices have fluctuated recently—this GPU cost a few hundred dollars less just a few weeks ago—it remains a practical alternative to higher-end Nvidia options for those operating on a more modest budget. It won’t be as fast but it’s fast enough for most workflows.

The primary advantage of the 32-gigabyte capacity is the ability to run dense models, such as the 27-billion-parameter Qwen 3.8, entirely on the GPU. This prevents the system from having to offload data to slower system RAM or the CPU. In my testing, the setup generates approximately 25 tokens per second, rarely dropping below 20 even when the context window is heavily loaded. For example, I used the model to summarize a complex appeals court ruling regarding DJI litigation and the result was every bit as good as what you might get from ChatGPT or Gemini. The system processed the legal text and generated a structured summary and a chart without relying on cloud resources.

The hardware configuration I demoed involves an Oculink connection to a MinisForum Mini PC equipped with an AMD Strix processor. While a mini PC of this caliber might be more power than necessary when the GPU is doing the heavy lifting, it served as a convenient platform for the project. The operating system is Ubuntu 26.04, which has proven stable for these applications.

Normally I’d do a how-to in a video like this to get things working. But these days, it’s easier to just let AI do most of the heavy lifting especially given how frequently things are changing in this space. For this project I had ChatGPT’s CODEX command line interface verify the hardware install and find the appropriate drivers. This approach has allowed me to resolve configuration issues that might otherwise be time-consuming, resulting in a more efficient setup.

Practical applications for this local setup extend into coding and data management. I developed a tool called “The Gadget Tracker,” which filters RSS feeds from technology publications. The Qwen model analyzes article text to determine if they pertain to hardware gadgets rather than general industry news or software algorithms. Valid entries are then stored in a local SQLite database for review in a to-do list format. I also demoed Qwen 2.8 modifying the application’s user interface, adding social media images to the display cards and adjusting UI elements. The model’s reasoning process is perhaps a little more methodical than a frontier model with more resources; it often takes time to contemplate its plan before executing code, which helps minimize errors in the final output.

Another test involved extracting data that is difficult to capture through traditional copy-and-paste methods. I tasked the model with gathering names and email addresses for the New York City Council from their official website. Because the emails were embedded within the site’s source code rather than displayed directly on the page, a simple scrape wouldn’t work. The local model used a curl command to pull the HTML, identified the relevant strings, and formatted the data into a CSV file in under two minutes.

There are trade-offs to this local approach, particularly regarding power and acoustics. The GPU fan is audible even when the system is idle, and the card consumes about 55 watts just sitting with a model loaded. Under a full processing load, that consumption rises to over 320 watts.

Despite the power demands, the ability to process 96,000 tokens of context locally provides a level of privacy and unlimited usage that cloud services cannot match. The technology in this space is moving quickly, with software updates frequently unlocking better performance from existing hardware. This specific configuration has reached a point where it handles the majority of the tasks I previously delegated to proprietary models, proving that local hardware is increasingly capable of meeting professional demands. My focus now shifts to optimizing the power draw while continuing to test how these dense models handle more complex, long-term projects.