Ai on Dexome

Running Local LLMs on an Intel Arc A310

Wed, 10 Jun 2026 00:00:00 +0530

I use an Intel Arc A310 with 4 GB VRAM for Frigate, Immich and media transcoding. I wanted to use the same GPU for a private LibreChat setup, with Ollama as the model server and Netdata exposed through MCP.

Small models worked for normal chat. Netdata tool calling did not work reliably. The first problem was Ollama’s default 4096-token context, which removed the MCP tool definitions from the prompt. After increasing the context, the remaining problem was the capability of models which could fit on this GPU.

My setup

Ollama 0.12.11 and later include an experimental Vulkan backend that can use Intel Arc. I passed only /dev/dri/renderD128, joined the render and video groups, and kept the API on private Podman networks. LibreChat reached Ollama by container DNS; Ollama used a separate egress bridge to pull models.

flowchart TB
	USER[Browser] --> CHAT[LibreChat]
	CHAT -->|private bridge| O[Ollama and Vulkan]
	CHAT -->|13 tool schemas| MCP[Netdata MCP]
	O --> ARC[Intel Arc A310, 4 GB]
	VIDEO[Frigate, Immich, Plex, Jellyfin] --> ARC
	O -->|model pulls only| NET[Internet bridge]

The model API stays private while one render node is shared with the host's media and vision workloads.

The Vulkan backend worked for small local models. The first issue appeared before the model generated any response.

MCP tools were removed by context truncation

LibreChat sent the system message, Netdata instructions, 13 MCP tool schemas, and the user turn. The resulting first prompt was about 11,511 tokens. Ollama’s default context was 4,096.

The log made the failure explicit:

`1`	`truncating input prompt limit=4096 prompt=11511 keep=4 new=4095`

The tool definitions were near the discarded end. A model that never receives the schemas cannot issue a tool call, regardless of its instruction following. It answered in prose because prose was the only action left.

flowchart TB
	P[11.5k-token prompt] --> CUT{4,096-token context}
	CUT --> KEEP[Small retained slice]
	CUT -. discarded .-> TOOLS[Netdata instructions and 13 tool schemas]
	KEEP --> MODEL[Model sees no callable tools]
	MODEL --> TEXT[Plain-text answer]

At the default 4k context, truncation removed the MCP schemas before the model evaluated the request.

Raising OLLAMA_CONTEXT_LENGTH made structured tool attempts appear. That proved truncation was causal. It did not yet produce a usable system.

Larger context used more VRAM

At 16,384 tokens, the KV cache occupied about 1.8 GB. Only 18 of 29 model layers fit on the GPU; the other 11 spilled to CPU. Processing the first 11.5k-token prompt took 131 seconds, during which nothing streamed. LibreChat aborted before the first token.

Turning off the 6 KB server-instructions block helped less than expected. The prompt still measured 10,189 tokens because the 13 tool schemas themselves were the dominant cost, and LibreChat could not expose only a subset of MCP tools. An 8,192-token context still truncated them.

The practical setting was 12,288: enough for the tool prompt plus answer headroom, with fewer layers displaced than at 16k. The first turn remained slow, roughly 80 to 100 seconds for a 3B model. Follow-up turns were fast because Ollama cached the prompt prefix; one 67-token follow-up returned in 0.8 seconds.

flowchart TB
	C4[4k context] -->|small KV cache| FAST[More GPU residency]
	C4 -->|but| TRUNC[Tool schemas truncated]
	TRUNC -. next test .-> C12[12k context]
	C12 -->|schemas fit| VISIBLE[Tools visible]
	C12 -->|larger KV cache| SPLIT[Some layers spill to CPU]
	SPLIT -. next test .-> C16[16k context]
	C16 -->|about 1.8 GB KV| SLOW[18 of 29 layers on GPU, 131s prompt evaluation]

A larger context fixed schema visibility but enlarged the KV cache and forced model layers onto the CPU.

The model files fitted, but the model, KV cache and the other GPU workloads did not always fit at the same time.

Models I tested

With logs confirming no truncation, I tested the models against simple Netdata questions.

Model	GPU placement	First prompt	Result
Qwen2.5 1.5B	29/29 layers	about 55s	Printed a tool call as JSON text; LibreChat could not execute it
Llama 3.2 3B	20-21/29 layers	105-131s	Emitted a real call with schema-invalid arguments
Qwen2.5 7B	13/29 layers	about 218s	Too slow and returned an unusable result

The 3B model even failed the no-argument list_raised_alerts schema. This was not limited to one complicated metrics query. It could choose a tool and emit a tool-call shape, but not produce arguments the MCP client accepted reliably.

LibreChat’s “Ran tool” pill was not proof of success. It appeared when dispatch began; the execution log later showed Received tool input did not match expected schema. Without checking that log, I would have mistaken an attempted call followed by hallucinated prose for real monitoring data.

Increasing the timeout only waited longer for the same result. It did not fix the invalid tool arguments or make the 7B model fit better.

Testing the Intel SYCL backend

Intel’s old IPEX-LLM path looked attractive because it promised optimized SYCL inference. By the time I evaluated it, the repository was archived, its bundled Ollama was old, the images used rolling tags, and the project was flagged with known security issues. I rejected it.

The maintained high-throughput option is upstream llama.cpp’s Intel SYCL image. I staged it beside Ollama rather than replacing the working service. It needed both renderD128 and this host’s actual card node, card1; assuming card0 prevented container creation.

llama-server introduced its own constraints:

the default four parallel slots divided the usable context per request;
--parallel 1 was necessary for the large single-user prompt;
--jinja was required for structured tool calls;
oversized prompts hard-failed unless context shifting was configured;
one server process loaded one GGUF model, unlike Ollama’s model manager;
the first SYCL request paid a JIT compilation cost.

I kept Ollama plus Vulkan as the default. SYCL can improve throughput, but it does not make a 3B model format better arguments, nor does it make a 7B model fit in 4 GB.

What works well on the A310

The A310 is useful for private chat, summarization and small experiments. Llama 3.2 3B was the best local default from my tests. It fitted well enough and could emit a real tool-call structure. Qwen2.5 1.5B was faster but printed the tool call as text.

It was not reliable for the 13 Netdata MCP tools. Their schemas required a large context and the models which remained usable on 4 GB VRAM could not consistently produce valid tool arguments.

The debugging order I use now is:

Confirm the accelerator backend actually loaded; do not infer GPU use from container access to /dev/dri.
Read the prompt token count and truncation log.
Measure KV cache size and GPU layer placement at the chosen context.
Separate prompt-evaluation latency from generation speed.
Verify tool execution success in logs, not in UI decoration.
Check whether the remaining problem is model capability rather than another timeout or context setting.

I kept Ollama with Vulkan as the default because it supports model management and normal local chat worked. llama.cpp with SYCL is available for comparison, but a faster backend does not make the small model better at producing schema-valid tool calls.

Keeping AI Agent Notes in My Homelab Repository

Fri, 05 Jun 2026 00:00:00 +0530

I use AI coding agents regularly while working on my homelab repository. The chat history is useful during a task, but a new session does not automatically know what was proved in an older one.

This became a problem for operational details. For example, one session found the safe way to evaluate my Nix flake without copying 20 GiB of ignored model files. Another found the exact SSH and TTY sequence required for privileged diagnostics. I did not want to investigate those details again.

I now store these notes as Markdown under docs/agent-memory/. They are part of the repository, so I can review and update them along with the configuration.

What I wanted from these notes

The system needed to be:

durable: survive chat sessions, machines, and editor profiles;
reviewable: change through the same review process as code;
routable: let an agent find one relevant note without reading hundreds;
historical: preserve old decisions without presenting them as current;
safe: record where secrets live, never their values;
cheap: make adding a lesson easier than rediscovering it.

I did not want one large file which every agent had to read at the start of a task. I also wanted the notes to remain visible outside one editor or agent.

One Markdown file for each topic

Every memory is a dated Markdown file under docs/agent-memory/:

1
2
3

2026-06-12-traefik-macvlan-asymmetric-return-route.md
2026-07-23-copyparty-rename-strips-group-write-arrs-import-denied.md
2026-08-10-netdata-2.10.3-collector-fixes.md

The ISO date gives a useful filesystem order. The descriptive suffix makes the file discoverable with ordinary text search. One topic per file lets a later note supersede one conclusion without invalidating an unrelated section of a large document.

New notes carry queryable frontmatter:

---
type: incident
title: Podman network drift recreate needs rm -f
description: Stopped containers remain associated with their networks.
timestamp: 2026-06-12
tags: [podman, networking]
resource: hosts/heavymetal/containers.nix
status: resolved
---

I normally write the body in this order:

What was the symptom?
What evidence separated it from similar failures?
What was the root cause?
What exact change fixed it?
How was the fix verified?
How can it be undone or superseded?

I include a failed attempt only when it is likely to be tried again. Normal terminal exploration and command typos do not need to be stored.

Using a small index to find the correct note

Opening every note at the start of every request would replace forgetting with context overload. The repository instead has a hand-curated index.md, grouped by service or topic. Each entry is one complete problem-to-outcome sentence:

1
2
3

- [2026-06-12-podman-network-drift-recreate-needs-force.md](...)
  — Recreating a drifted Podman network requires rm -f because stopped
  containers remain associated with it. (_resolved_)

The agent’s startup rule is simple: read the index, select the few summaries that match the current task, then open only those notes.

flowchart TB
	Q[New engineering request] --> I[Read compact routing index]
	I --> S{Select matching summaries}
	S --> N1[Open relevant incident note]
	S --> N2[Open relevant reference note]
	N1 --> W[Work with prior evidence]
	N2 --> W
	ALL[Hundreds of unrelated notes] -. not loaded .-> W

The routing index keeps startup context small while preserving access to detailed operational evidence.

I write these summaries by hand because a filename or heading usually does not say which fix actually worked. A script checks that every note is included in the index, but it does not generate the summary.

Not every note needs a dated file

Not every fact belongs in the dated-note catalog. I separate three classes:

Memory class	Example	Lifetime
Incident or migration	Why macvlan replies used the wrong route	Historical, may be superseded
Living reference	Port allocation inventory	Updated in place
Durable service guidance	Stable operating constraint	Promoted into reference documentation

A port inventory would become noisy as a sequence of dated files. A completed incident should not be silently rewritten whenever understanding changes. For example, I update the port allocation inventory in place. An incident stays as a dated file because a later note may supersede it without removing the old evidence.

Marking an old note as superseded

Deleting an old note destroys the path that explains why a decision existed. Leaving it unmarked lets an agent follow obsolete instructions. The compromise is bidirectional supersession.

The old note says:

`1`	`> Superseded by: [2026-08-17-new-design.md](...) — reason.`

The new note says:

`1`	`> Supersedes: [2026-06-05-old-design.md](...).`

Both frontmatter records and index entries carry the same state. The old evidence remains available, but every retrieval path points toward current guidance.

flowchart TB
	OLD[Old note: status superseded] -->|superseded by| NEW[New note: current guidance]
	NEW -->|supersedes| OLD
	INDEX[Routing index] -->|current entry| NEW
	INDEX -. historical entry marked superseded .-> OLD

Supersession preserves the audit trail while making the current instruction unambiguous.

I used this when a temporary design was retired and when a shell test showed that umask 0002 worked but the real container supervisor reset it. The old test was still useful, but the new note needed to be the current instruction.

Telling the agent when to read the notes

Notes do not help if the agent does not know when to read them. Repository instructions establish a retrieval protocol:

At the start of a request, read the routing index.
Open only relevant notes.
Consult living inventories before changing shared resources such as ports.
Record a newly proven operational lesson in the repository.
Update the index in the same change.

Without these repository instructions, the notes would only be documentation which an agent might or might not find.

flowchart TB
	TASK[Task begins] --> ROUTE[Index and instruction routing]
	ROUTE --> PRIOR[Relevant prior knowledge]
	PRIOR --> ACT[Implement and validate]
	ACT --> LESSON{New durable lesson?}
	LESSON -->|yes| NOTE[Write dated note and index summary]
	NOTE --> CHECK[Run coverage and consistency checks]
	CHECK --> TASK
	LESSON -->|no| DONE[Finish]

Work produces evidence, evidence becomes a note, and future instructions route the next task back through it.

Checking the index

The repository includes a small dependency-free checker. It compares dated files on disk with links in the index and reports:

notes missing from the index;
index links whose files no longer exist;
notes without frontmatter.

It exits nonzero for missing coverage or broken links. It does not generate summaries, decide which topic heading is best, or resolve contradictory advice. Those are semantic tasks.

A periodic manual pass checks what code cannot reliably infer:

stale pending or parked statuses;
one-directional supersession links;
newer notes that contradict older guidance;
duplicate incidents without a relationship;
ports or service references that drifted;
lessons mature enough to move into stable documentation.

The script checks the structure. I still review the summaries and decide whether one note replaces another.

Do not store secrets or duplicate the code

Repository memory must never contain secret values. A note may say that an MQTT password comes from a named SOPS secret and which service consumes it. It should not contain the password, a token copied from a log, or an unredacted credential example.

I also avoid storing temporary chat details, large summaries of the repository and facts already clear from the code. I add a note when it preserves a useful check, an operational problem or a decision which would take time to reconstruct.

Current workflow

I do not use a vector database or an embedding pipeline for this. Markdown, links, frontmatter, repository instructions and a small checker are enough for my repository.

The workflow is:

capture only proven lessons;
compress each lesson into a routable sentence;
load details only when relevant;
preserve history through supersession;
keep every memory visible to the people responsible for the system.

The agent still starts a new chat without the old conversation. It first reads the index, opens the notes related to the current task and continues with the checks and fixes already recorded there.