Navigating Local AI: A Journey of Excitement, Overwhelm, and Frustration

I’m diving into a new adventure that I’ve been hesitant to embrace: regularly using AI. Handing over my personal data to cloud services has always made me wary, but the ability to run powerful models locally is changing the game. Is this technology so beneficial that it’s worth investing in a high-end computer or laptop with a hefty amount of RAM? I aim to help you figure that out. While I’m no AI expert and this is a learning experience for me, I hope to share insights that might guide you on whether this path is worth exploring.

A significant part of the appeal of the latest Mac desktops is their capability for local AI, and there’s a new wave of RTX Spark Windows machines arriving soon, boasting up to 128GB of RAM designed for agentic AI. Local AI is definitely trending right now. The idea of having a personal assistant confined to my desk, responding only to me rather than tech giants like OpenAI or Google, feels refreshing compared to relying on cloud services or chatbots. I’m not about to let it write my content, edit my work, or create anything unsightly, but with some privacy on my side, I might finally be able to delegate some useful tasks.

As I continue testing the M5 Ultra Mac Studio and experimenting with other Mac and Windows systems, I’ve installed some local LLMs and started tinkering. Here’s how it’s going.

I kicked things off by installing the open-source Hermes Agent on the Mac Studio. Hermes is a self-hosted AI agent desktop app compatible with macOS, Windows, and Linux, and it’s completely free if you stick to local LLMs. With 256GB of unified memory in the M5 Ultra Mac Studio, I can run just about any model I want. This is where it can get overwhelming; there are countless models available, each with specialized functions. Generally, larger models with billions of parameters outperform their smaller counterparts. Since I don’t have to worry about token costs when running a model locally, I decided to go big right from the start.

Using Hermes’ user-friendly onboarding interface and model selector, I opted for the Qwen 3.8 Flash Next model, which has 125 billion parameters and is around 105GB in size. I also plan to experiment with smaller Qwen models on devices like an M6 Mac Mini, M5 MacBook Air, and an Asus TUF Gaming A14 with an AMD Strix Halo processor, and eventually, the RTX Spark.

Getting Hermes and Qwen up and running was quick, and I even set them up to be controllable via my phone using a Telegram bot. However, I faced my usual challenge with AI: the text box. “What do I do with this thing?”

To kick things off, I followed some helpful advice and had Hermes create a daily morning briefing for me. It scans my email and calendar, highlights anything urgent, and provides a brief weather update. While it’s not groundbreaking, and you certainly don’t need a $12,000 computer for it, it’s a starting point I can build on. Initially, the briefing failed to generate, but I soon realized that macOS needed to be awake when the job triggered at 7:30 AM. Now it runs like clockwork, but I need to think of more information for it to pull to enhance its usefulness.

A more practical task was reorganizing my Steam library. With over 400 games on Steam, I recently began categorizing them for spontaneous play sessions. Steam doesn’t automatically categorize games, so it’s a manual process. This is where an AI assistant could really shine. I asked, “Can you reorganize my Steam library?” Hermes accessed my installed Steam client and quickly suggested organization options. I chose to sort my games by genre while keeping some of my custom categories intact (favorites, co-op titles, party games, etc.).

To allow Hermes to take action, I had to grant it permissions on my machine, which included registering a Steam web API key. Once it had access, it took just a few minutes to sort all my games into categories. I easily revoked the API key afterward since I didn’t need it anymore. The plan is for it to handle this heavy lifting once, and then I’ll categorize new games as I acquire them. It did a solid job overall, and now my Steam library is much easier to navigate.

Another straightforward task I tackled was data analysis. I had Hermes review financial records to crunch some numbers and create a spec comparison spreadsheet for a new laptop. These tasks are small but significant, as I wouldn’t trust sensitive data to any cloud service, and the laptop info was under embargo. Keeping everything local was crucial for me to use this tool.

Currently, I’m working on a script to automate a series of laptop benchmark tests. Running benchmarks manually is time-consuming, and managing a laptop through a dozen tests—three times each for averaging—takes forever. I’d love to automate at least part of this process. Walking Hermes through the necessary steps and getting it to generate Python scripts has been a bit of a challenge, but it’s a work in progress, and I’m hopeful for positive results.

It’s still early in my journey with Hermes, and I’m excited about the potential of what I can achieve with it. However, it’s clear that even these powerful local models aren’t magical assistants that can do everything. Things don’t always go as planned—my daily briefing has already broken multiple times.

I see Hermes and local AI as just tools, programs running on my computer. I’m not looking for a digital friend; I don’t say “thank you” or “please,” nor do I apologize when I change my mind. I interact with it as I would with any software, focused on getting the job done.

I plan to stay cautious and vigilant as I continue exploring local AI. Let’s see where this journey leads me.