Over the last few months I’ve been dabbling into Local AI and finding many small successes especially when it comes to coding small apps to help improve my workflow. I even made a functional replacement for Quickbooks!
Many people have asked about the starting point for this type of work and the core concepts required to set up local coding agents. In my latest video, I guide you through setting up a local coding agent step-by-step.
The foundation of local AI is having adequate hardware. On the Windows side, a system with a dedicated GPU is necessary because CPUs and system memory process AI tasks slowly. My current configuration utilizes an ASUS AI Pro 9700 card (compensated affiliate link), which provides the necessary memory to run larger models. But even smaller gaming GPUs with 8GB of RAM can run effective local models. For those on Mac, Apple M-series devices with 16 or 32 gigabytes of RAM is sufficient for these tasks thanks to their unified memory architecture. The objective is to fit the AI model entirely within the GPU memory for optimal performance.
The first tool I recommend for those starting out is LM Studio Bionic. When it is first launches, it will suggest models based on the detected hardware. I frequently use the Qwen 3.8 27B model on my 32GB hardware, which is effective for local coding. If you have less memory available, there are some very effective models from Qwen and Google that Bionic will recommend. Look for models with a green label that can be fully offloaded to the GPU.
After downloading, I make a few adjustments to the default settings: I disable the reasoning budget to allow the model more latitude in problem-solving, and on Windows, I disable “mmap” to prevent system RAM from becoming overloaded.
Understanding context length is another vital part of the process. Context serves as the model’s short-term memory. As a conversation progresses, the memory fills up, and eventually, the model must “compact” the information or it will stop functioning. This is a primary constraint of local AI, so I aim to balance the model size and the context length so they both fit comfortably on the GPU. Additionally, I set the runtime to Vulkan for AMD or Intel GPUs, or CUDA if using an Nvidia card.
Beyond those settings you don’t need to change much else to get started with
Bionic.
To demonstrate how these agents function, I initiated a project in Bionic to build a website based on a spreadsheet of Star Trek: The Next Generation episodes. I provided a CSV file and some images, then prompted the model to create an attractive site using the show’s specific aesthetic. The local model, through Bionic, worked as an agent, reviewing the files the my hard drive and writing the HTML and JavaScript files directly to the folder.
While it made a few minor errors regarding character names, I was able to provide a follow-up prompt to correct the code instantly. This ability to edit files locally, rather than copying and pasting from a cloud-based chat interface, is a significant efficiency gain.
I also tested the model’s ability to create a native Windows application in C#. While the model successfully produced an executable file, the result was less refined than the web version. Local models often have less training data for specific native desktop frameworks compared to standard web technologies. For simple utilities, local models are quite capable, but more complex native software still requires significant manual prompting and refinement.
When a task requires reformatting data or scraping information from the web, I often move to a different harness called OpenCode. This tool does not host models like Bionic does, so it requires a server to access the model. I use Bionic as that server by enabling its local model API server feature. This setup allows me to use the advanced agentic features of OpenCode while still running the actual processing on my own hardware.
OpenCode does require some configuration to connect to a local server – but because OpenCode allows you to connect to some cloud models for free, you can just ask it to configure itself for your local Bionic server.
In the video, I demo using OpenCode and a smaller Qwen 3.5 9B model to gather contact information for City Councilors from New York City and Los Angeles and put that information into a spreadsheet. The agent navigated the sites, identified the names and email addresses of council members, and compiled them into a spreadsheet. Although the smaller model missed a few details, it successfully automated a task that would otherwise require tedious manual entry. Larger models will do a little better but it’s remarkable how good small models have become at this task.
Starting with a self-contained application like Bionic provides a low-barrier way to learn the concepts of agentic coding. As requirements grow, transitioning to more complex harnesses or integration with Visual Studio becomes the next logical step. Local AI provides a level of privacy and ownership that cloud services cannot offer, as all data and processing remain on personal hardware. I find that my local models can now handle the majority of my smaller daily coding and data tasks, and I only turn to cloud-based alternatives when a project exceeds my local hardware’s capacity.