The Local LLM Your Laptop's Been Waiting For
Why you should try the smaller versions of Qwen 3.5, one of the best local LLM models.AI

Screenshot: Matthew Smith
Running an LLM locally on your own hardware is enticing for several reasons. It's private, it's free, and it's not too difficult if you use GUI software like LM Studio.
Yet there's a trade-off. The types of small LLMs that run well on local hardware aren't always that intelligent, compared to cloud-based ones like Claude and ChatGPT. On top of that, the thousands of local LLMs available to download can lead to serious analysis paralysis.
Fortunately, using a local LLM has improved with Alibaba's release of bite-sized Qwen3.5 models. These models range in size from 9 billion to as few as 800 million parameters. They support reasoning, vision, and tool-calling, and even the largest among them runs well on a mid-range GPU with 16GB of memory, or a laptop that has 24GB/32GB of memory and a high-end iGPU.
[Editor's note: Qwen 3.6 has just been released, and we are currently testing it.]
Getting to know the family
Qwen is the flagship family of large language models from Chinese tech giant Alibaba. The first model was released in April of 2023, and the Qwen family has maintained a respectable reputation ever since. Qwen models are open-weight, free to download, and often offer accuracy, performance, and features that are just 6 to 12 months behind frontier models from OpenAI, Anthropic, and Google.
Alibaba released large versions of Qwen3.5 in February of 2026. The largest, with 397 billion parameters, has ranked as high as no. 13 on the Artificial Analysis leaderboard, but it's far too beefy to run on consumer hardware.
However, the company didn't forget home users. Smaller models arrived in March of 2026. This article is going to focus on the four smallest models.
- Qwen3.5-800M
- Qwen3.5-2B
- Qwen3.5-4B
- Qwen3.5-9B
Alibaba also offers variants with 27B and 35B parameters, which weigh in at 17GB and 24GB respectively with Q4_K_M quantization.
You can consider these if you're lucky enough to have an RTX 5090 desktop video card, an AMD Strix Halo system with 32GB of RAM or more, or a high-end MacBook M4/M5 Pro/Ultra with 32GB of RAM or more. However, they won't perform well on anything but the most luxurious hardware.
The smaller Qwen3.5 models are remarkably clever
Qwen3.5's benchmark scores are promising. Qwen3.5-9B is in some cases able to beat OpenAI's GPT-OSS-20B and even GPT-OSS-120B. It notches wins in GPQA Diamond, MMMLU, MMMU-Pro, and OmniDocBench v1.5, among others.
The benchmark results have already been reported, though, so that's not my focus here. My question is simpler — do these benchmark results actually translate to good performance?
I tried the four smallest Qwen3.5 models in tasks including summarization, e-mail writing, web search and retrieval, and coding. The results were remarkable though, as usual, you'll want to run the largest model that your PC can handle.
Document summarization
Reporting on science and technology is my job, so I spend a lot of time trawling academic papers, white papers, and industry reports. LLMs make this part of the job far easier. I'd say it's my number one use case.
It turns out that Qwen3.5-9B is rather good at it, albeit with some instructions. Simply asking it to summarize a document can still lead to wordy results, but instructing it to keep to a couple paragraphs (or another constraint of your choosing) works better.
Qwen3.5-9B is definitely more prone to hallucination than a model like Gemini 3.1 Pro or Claude 4.6, but it's not common, and usually involves small deviation in a particular metric or number rather than broad-strokes fabrication of concepts and ideas. In any case, I recommend double-checking all critical facts no matter the quality of the model you use.
Web search
Qwen3.5-9B is capable of tool calling, so it can be used to search the web by calling tools for web search, web scraping, or browser use.
It can handle these tasks more often than not, which in my experience is an upgrade over other models of similar size. I found it could generally handle simple queries, like the critical consensus on a new video game. The model does a good job of summarizing the information it finds in a quick, digestible pattern while also citing specific web pages.
But it's definitely not perfect. The model's tool calling ability is not flawless, so attempts to search sometimes fail. It can have trouble understanding situations where a website blocks the model, or the website is otherwise not readable. Occasionally, these can throw the model into a thinking loop when thinking is on.
Also, Qwen3.5-9B's tool call capabilities are not a good fit for agentic software such as OpenClaw. I could not get it to work with an acceptable level of reliability, and based on the feedback others have posted on Reddit, my experience was typical.
Writing
Writing is a strength of Qwen3.5-9B.
On the plus side, the model is good at writing clear, precise copy. I'd say the average e-mail written by Qwen3.5-9B is easier to read and stays on topic better than 95% of emails I've ever received. I also don't think larger models are meaningfully better at writing overall. They tend to succeed and fail in broadly similar ways.

Screenshot: Matthew Smith
Qwen3.5-9B's big issue is one it shares with a lot of models: the tone is artificial. That's expressed in its choice of words and phrases. It has all the usual tells including the cringe-worthy tendency to start emails with the line "I hope this email finds you well."
Qwen3.5-9B is great for any communication that is functional, impersonal, or not that important. But if you're trying to organize a family trip or circling back with a recruiter about a resume you sent over, then I'd recommend a more personal touch.
Coding
LLMs have improved in coding more quickly than any other domain, and Qwen3.5-9B is no exception. I threw it at several of my usual tests. I tasked it to create a portfolio website, create a to-do list app in JavaScript, and write a web scraper that attempts to download PDFs in Python.
Qwen3.5-9B's one-shot results range from good to virtually perfect. It was particularly adept at basic JavaScript apps, such as the todo list. The website was also usable, though a few extra passes were required to fix minor formatting issues. The PDF scraper was not a problem.
There are definite limits, though. I also asked Qwen3.5-9B to re-create my vibe-coded tabletop role-playing game combat tracker (the version that exists online was mostly handled by Claude and Gemini).
Qwen3.5-9B could put together a basic ordered list where combatants can be added and removed, but that was about the best it could manage. More complex tasks, like coding an attack planner for combatants that can handle multi attacks against a target AC and roll the dice in the app, seemed to elude its understanding.
On the whole, Qwen3.5-9B's capabilities remind me of my early days vibe coding with Claude 3.5. It can frequently one-shot simple tasks, but requires a lot of iteration for anything more complicated.
Vision
The vision capabilities of Qwen3.5-9B are excellent. Quite a few small models offer vision, but Qwen3.5 was extremely good. It gave highly accurate, detailed descriptions of several portrait photos I provided it, and also accurately retrieved text from restaurant menus, receipts, and app screenshots. I didn't see a single mistake or error after serving it a couple dozen images.
What about the smaller models?
Look — don't expect a miracle. Very small models have a long history of issues in particular areas, like tool calling. Calling a tool requires a model to have a solid understanding of how that tool accepts a tool call, and small models often get off track. The tool call fails. The model doesn't always understand why. It can then fall into a loop or even hallucinate a response as if the tool call succeeded.
Web search is a great example of where this breaks. I found it worked well enough with Qwen3.5-9B, but with the smaller models it was hopeless. My attempts failed more often than not.
On the other hand, the small models can prove decent at writing and basic coding. I actually like Qwen3.5-800M for simple emails. It can bang them out at speed (nearly 100 tokens per second even on a MacBook Air M4) and tends to be even more concise than the larger models. It was able to code a simple JavaScript to-do list app and one-column website, though it required a few follow-ups to fix straightforward bugs such as a non-functional delete button.
Qwen3.5 feels a lot like OpenAI's GPT-OSS
As mentioned earlier, Qwen3.5-9B posts benchmark results that beat OpenAI's larger GPT-OSS models in some benchmarks. However, the contests were often close, and that feeling of similarity extends to real-world use.
OpenAI models have a vibe. They tend towards a highly structured response with bullet points and tables. They are friendly and direct but often lack wit. They're quick to praise the user. I think Qwen 3.5 has all of these traits.
Whether that's good or bad depends on your perspective on OpenAI's models generally. I find this style tends to work well for high-level summarization but can become annoying when trying to dive deeper into a topic.
Small Qwen3.5 models perform well on a variety of hardware
I benchmarked the Qwen3.5 models in LM Studio on three different computers.
- Apple MacBook Air M4 with 24GB of RAM
- Asus TUF Gaming A14 with AMD Ryzen AI Max+ 392 and 32GB of RAM
- Custom desktop with AMD Ryzen 5950X, AMD Radeon 7800 XT 16GB, 32GB of system RAM
I used the 4-bit quantized models in all cases except for Qwen3.5-2B and Qwen3.5-800M, where I used the 8-bit quantization. The MacBook ran the MLX versions of these models, while the others were GGUF. Context length was set to 32,000 tokens.
The test prompt was:
Please create a basic javascript to-do list app. It should be a checklist. The user should be able to add and remove items to the checklist.

Screenshot: Matthew Smith
This test shows a clean and unsurprising breakdown between the three tested systems. The Apple MacBook Air M4 was the least capable, the Asus TUF Gaming A14 was in the middle, and the AMD Radeon 7800 XT took the win. All of this is what I expected.
Still, this test does highlight that performance remains an issue for low and mid-range laptops. Qwen3.5-9B barely exceeded 26 tokens per second even on the Asus TUF Gaming A14. That's usable, to be sure, but you're really best off if you can load Qwen3.5-9B on a dedicated GPU.
Conclusion
The Qwen3.5 family is no joke and the small models will be enticing for fans of local LLMs. Qwen3.5-9B is the highlight, as it's somewhat capable of basic tool calling (including web search) and proves helpful in common tasks like writing, summarization, and basic coding.
Of course, it still has limitations. Larger models hosted in the cloud are more intelligent and generally quicker to reply, though Qwen3.5-9B can be fast on capable hardware. However, the overall capability of Qwen3.5-9B feels a lot like what I remember of GPT-3.5 and Claude 3.5. Those models weren't perfect but they felt capable at the time.
We've also reached a point where, in simple tasks, that gap between a local LLM and a frontier model is negligible. Is GPT-5.4-mini better than Qwen3.5-9B for writing an email or summarizing a document that's shorter than 10 or 20 pages? Perhaps some benchmark says so, but in practice I think the difference is trivial. So, if that's your use case for an LLM, Qwen3.5-9B might be good enough to be your go-to model.
More from MC News
- Run AI Locally: The Best LLMs for 8GB, 16GB, 32GB Memory and Beyond
- Quantization Explained: Why the Same LLM Gives Better Results on High-End Hardware
- Why VRAM and Memory Bandwidth are Key for Powering Local AI
- Why AMD's 3D V-Cache is Having a Moment
- The Micro Center Monitor Buying Guide: Part 1
- Keyboard 101: Intro to Computer Keyboards
- Fix It Yourself: Talking to iFixit on Why Repairable Tech Matters
- Hands-on with NVIDIA DGX Spark: Everything You Need to Know
Matthew S. Smith is a prolific tech journalist, critic, product reviewer, and influencer from Portland, Oregon. Over 16 years covering tech he has reviewed thousands of PC laptops, desktops, monitors, and other consumer gadgets. Matthew also hosts Computer Gaming Yesterday, a YouTube channel dedicated to retro PC gaming, and covers the latest artificial intelligence research for IEEE Spectrum.
