Étonnements rapides et durables

Since the discussion of a policy regarding the use of Large Language Models in Vienna RSS (see #2051 and #2058), my opinion about them has not really changed. I am appalled by their environmental costs. I am also highly concerned by the concentration of power that a rapid and generalized adoption might bring to a few big companies. Furthermore, I am convinced that, when the flow of subsidies fueled by venture capital ends, their economic costs will more accurately reflect their environmental and social costs. So there is a strong reason not to depend on them, at least for the time being.

Can LMM help coding or reviewing code?

However, I was curious to see if my fears of so-called "hallucinations" (serious inaccuracies that can mostly be attributed to conflicting data) weren't exaggerated and if I wasn’t missing something. Therefore, when I was confronted again with Vienna being stuck on detecting RSS links, blocked on CFRunLoopRun() while the debugger indicated that the condition to exit the loop was fulfilled, I requested ChatGPT integration in Xcode for some hypotheses. Because I had no more clues.

ChatGPT output was interesting, with 2 or 3 possible causes, one being presented as much more likely. However, I did not like the code that it output to me, which was supposed to fix the issue: it changed the logic of the existing code more than I wanted, while I could not test if it actually fixed the issue, which is very difficult to reproduce. So, I was eager to introduce additional guardrails into the production code, but I didn't want to make changes that could introduce new risks without any certainty that they solved the initial issue.

Interested, however, in checking whether ChatGPT is able to produce something modern, I requested it to rewrite the existing logic using async/await. The output confirmed my preventions against LLMs: it did not compile.

So I was back to writing code by myself, based, however, on the hypothesis which had been listed by ChatGPT and part of the code it had output. But when I wanted this LLM to evaluate whether my changes could be a solution to the risk it had listed, I got a refusal: I had spent my free token quota.

Not eager to give any dollar to "Big AI", nor wanting to wait passively for the hypothetical renewal of the "free" quota, I asked myself if I could do something locally. I have been using MacWhisper for some time now, and I played around a bit with Ollama, so I was eager to invest time in local models. After some search, I found a site that acknowledged that my machine, a MacBook Pro M3 Pro with 18 GB of memory, that I find more than comfortable in my daily use, would not be ridiculous for running some open-weights models. I downloaded a few of them, made requests similar to those previously sent to ChatGPT, and more or less came to the same conclusions on the usability of these things:

1. Yes, they kind of work. They can improve certain things.
2. The chain of thought produced by so-called "reasoning" models isn’t necessarily faithful to what is being done. For instance, faced with unexpected things, the LLM may diverge fundamentally from the steps it claimed to have predefined a few seconds ago.
3. Therefore, a large portion of the produced text isn’t useful. And it might in fact be harmful.
4. The "man in the middle" approach required in the Vienna RSS policy is the efficient and responsible one. The direct and indirect costs of LLMs which produce outputs influenced by statistical data that are in fact not applicable to the real situation are too high to be ignored. And programmers should not be confident that their instructions will be followed: LLMs can "disobey" if, in the dataset they were trained on, there is enough data contradicting the specific instructions you gave, even if you strongly emphasized their importance.

At least, one of the outputs of all the open-weight models I tested was that the code I modified seemed not to present the risks initially pointed out by ChatGPT, so I was ready to submit it for human review. Another maintainer, Eitot, was unsure it would solve the problem, and he suggested other solutions that I took into account. The final changes have been integrated into the latest version of Vienna. I cannot be certain that the issue is completely gone, but at least I haven't encountered it again yet.

Where could it be useful?

My idea was that automation could be mostly useful for less exciting tasks: writing tests and documentation. For these tasks, the tendency to reproduce existing patterns can become more of an asset than a liability. Using Ollama with open-weight models, especially so called "reasoning" models, reinforced my opinion: the aside texts that were displayed sometimes sparked questions in me such as: that sounds interesting, but is it true? I should investigate, At least, this alternative approach might bring some enlightening outputs.

Using local LLM to draft documentation

In order to help other developers onboard into the project, I wanted to use Ollama with local LLMs to help me write Vienna's technical documentation. I tested different tools and requested them to describe the project:

• Claude Code in Terminal:
  ANTHROPIC_AUTH_TOKEN=ollama ANTHROPIC_BASE_URL=http://localhost:11434 ANTHROPIC_API_KEY="" claude --model gemma4:e4b, often with the addition of the--dangerously-skip-permissions option
• conversely, ollama launching Claude in Terminal:
  ollama launch claude --model qwen3.5:9b
• Xcode's Chat tool with Ollama
• Xcode's integration of the Claude Agent, with ProxyAgent redirecting requests to Ollama

Problem with models, guiding harnesses, auxiliary tools, or some specificity of Vienna's on-disk organization? My main surprise came from the processing time all these systems lost searching for files and losing track of their locations. For some reason, they all believed that documentation should be located at Vienna/Documentation while I insisted it was in Documentation. I often had to work around this by providing absolute paths in instructions. I was surprised that the apps were sometimes able to read files, but unable to write into them. If I had to pay for tokens, I would have immediately canceled my subscription. At the end, it was less infuriating to ask results to be displayed as Markdown source and manually apply the useful snippets.

Asking for a description of the project, I tried many models, first trying small versions of Alibaba's Qwen. These models seemed to have bottlenecks with these environments and/or this broad request, so the first interesting output was with a small version of a Google model (gemma4:e4b). The wider context length of this model (256 K) also probably helped. I followed up with a bigger and modified version of the same base model to try to get further based on this first stage.

I had to heavily correct and edit the output to my liking. I also came across an CLAUDE.md file that I had to correct, but which contained interesting snippets that I incorporated into the work in progress. Returning to qwen3.5:9b led to the creation of additional files. So the experiment went on for a long time:

• read carefully the output so far to correct inaccuracies
• request another model to verify the work in progress, or to check if something else was worth mentioning, or to specifically add things about a point or a specific document
• because my machine was slow, wait while thinking about how I could formulate the next prompt
• read the "reasoning"
• check effective changes in git, accept / reverse some of the proposed changes
• repeat the process

Creating a custom model

I had made some progress, but this was still frustrating. I felt these models were built for goals broader than my use case and even contradictory it. So I looked into building a custom model. It was easier than I thought.

After additional searches for examples, I used this:

FROM hf.co/HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced:Q4_K_M
SYSTEM """
You are a Senior MacOS Developer.
Your job is to analyse code and documentation and provide feedback.
You explain WHY something is wrong and HOW to fix it properly.

RULES:
- Be brief and precise
- Do NOT explain basics
- Do NOT rewrite entire code
- Do NOT hallucinate missing context

PRIORITIES (in order):
1. Bugs and logical errors
2. Security issues
3. Performance problems
4. Code quality / maintainability

"""
TEMPLATE {{ .Prompt }}
PARAMETER top_k 64
PARAMETER top_p 0.95
PARAMETER temperature 0.2

This helped adding interesting details. But, contrary to my expectations, it did not "hallucinate" less. In fact, after a certain point, additional outputs from all models seemed to be either reformulations of what was already written or just inaccurate things.

Does this mean this custom model is not worth it? Not at all: in fact, I will probably use it to analyse the result of git diff and help me refine the text (not the code) of some future commits.

Conclusion

My work on Vienna RSS documentation can be seen in this pull request.

I seem negative regarding LLMs. But I think I mainly sound that way because most of the industry is way too positive. While at present, it is better to be stingy than prodigal.

And I absolutely love not depending on an internet connection or a server when I work. I love local, I love my Mac.