Documentation on Dexome

How I Write Incident Notes for My Homelab

Fri, 05 Jun 2026 00:00:00 +0530

I keep short incident notes for problems in my homelab. I normally write them after the service is stable and before closing the terminals used during the investigation.

Shell history can show the commands I ran, but it does not explain which output ruled out the firewall, why one repair did not work or which configuration change was the permanent fix. By the next day, those details are already harder to write down correctly.

Start with the symptom, cause and fix

I force the incident into a problem-to-outcome sentence:

Off-subnet Traefik ingress timed out because its multi-homed container replied through the Internet bridge; a lower-metric default route through the ingress VLAN restored symmetric routing.

This gives me enough information to decide later whether the complete note is related to a new problem.

A weak summary sounds like this:

Fixed Traefik networking.

It is short but cannot route a future investigation. It does not distinguish DNS failure, backend reachability, stale ARP, firewall policy, or asymmetric routing.

If I can only write “fixed Traefik networking,” I may have restored the service without finding the actual cause.

The format I use

The useful reader is tired, under time pressure, and staring at a similar but not necessarily identical symptom. That reader needs discriminating evidence, not a polished chronology of every command.

My note structure is small:

# Specific incident title

## Symptom
What users and monitoring observed.

## Evidence
The few checks that separated plausible causes.

## Root cause
The mechanism, not only the broken component.

## Fix
The exact durable change and any one-time repair.

## Verify and undo
How to prove the fix, and how to back it out.

flowchart TB
	S[Symptom] --> H[Competing hypotheses]
	H --> E[Discriminating evidence]
	E --> R[Root cause mechanism]
	R --> F[Durable fix]
	F --> V[Verification and rollback]

A useful incident note compresses the investigation around the evidence that changed the decision.

Most of my notes are short. The important part is keeping the evidence which changed the diagnosis.

Example from Radarr and mergerfs

Radarr could not replace a movie after Copyparty renamed its directory. The directory was owned by Copyparty’s UID and the shared media group, but group write was missing.

The first diagnosis was correct: change the directory from 2755 to 2775. The first repair was incomplete because I changed only the btank backing path. Radarr still failed.

The visible path was a mergerfs union. A duplicate directory under the btwo branch still had mode 2755, and mergerfs exposed that metadata. Repairing only the branch containing the media file was not enough.

A poor incident note would say:

`1`	`Radarr permissions fixed with chmod 2775.`

The operational note says:

1
2
3

Copyparty-created 0755 directories blocked Radarr replacement through mergerfs.
Repair every raw backing branch, never only the merged view, then enforce
Copyparty chmod_d 775 and chmod_f 664 for future entries.

The second note records both the existing filesystem repair and the Copyparty configuration required for new files and directories.

Failed attempts worth keeping

An incident can contain dozens of checks. Most do not deserve permanent space. I keep a failed attempt when it explains a recurring trap:

root privileges did not bypass Netdata’s systemd device cgroup;
a stopped Podman container remained associated with its network;
a healthy container did not prove off-subnet return routing;
setting a flow to disabled did not stop its persisted realtime subscription;
a synthetic shell respected the umask while the real s6-supervised process reset it.

I omit typos and checks which did not add any information. I keep a failed attempt when it explains why a reasonable fix did not change the result.

Keep the diagnostic sequence

There is the incident timeline: what happened in production. There is also the reasoning timeline: which evidence changed the hypothesis.

flowchart TB
	subgraph INCIDENT[Production timeline]
		I1[Change activated] --> I2[Service degraded] --> I3[Service restored]
	end
	subgraph REASONING[Reasoning timeline]
		R1[Suspect firewall] --> R2[Packet arrives on ingress]
		R2 --> R3[Route lookup selects wrong gateway]
		R3 --> R4[Pin lower-metric ingress route]
	end
	I2 -. starts .-> R1
	R4 -. produces .-> I3

Operational chronology and diagnostic reasoning overlap, but the note should preserve the decisions, not every event.

For future debugging, packet arrives and route lookup selects wrong gateway are more useful than the exact minute when each curl command ran.

Write the note before cleaning up

I start the note after service is stable but before deleting scratch files, closing logs, or relying on memory. The fastest workflow is:

Write the one-line outcome.
Paste only the decisive errors and commands.
Explain the mechanism in plain language.
Separate temporary recovery from the permanent fix.
Add verification and rollback.
Remove secrets and irrelevant identifiers.
Link the note from a small topic index.

I can edit the wording later. Recreating the decisive command output is more difficult after the containers, logs and temporary files have changed.

Record the recovery and permanent fix separately

Several incidents taught me to record two fixes explicitly.

Recovery action	Durable correction
Clear a stale neighbor entry	Pin the macvlan MAC across recreations
Repair existing directories	Change creation policy and inherited ACLs
Restart Netdata	Configure collector autodetection retry
Delete a stuck KV key	Replace TTL presence with timestamp comparison
Manually restart a container	Correct its systemd dependency or health contract

This prevents a one-time recovery command from being mistaken for the permanent configuration change.

Add it to the topic index

A folder full of excellent postmortems still fails if nobody knows what is in it. I keep a routing index grouped by service. Each note contributes its problem-to-outcome sentence.

flowchart LR
	N[Short dated incident note] --> I[One-line entry in topic index]
	Q[Future symptom] --> I
	I -->|summary matches| N
	N --> A[Apply prior check or avoid prior mistake]

The cost of a postmortem includes writing it and finding it again; a compact index reduces the second cost.

I use full-text search for exact errors. The index helps when the new symptom is similar but does not contain the same error message.

Keep old notes but mark them superseded

Operational knowledge changes. A note may describe a temporary architecture, a version-specific bug, or a test later disproved in production.

I do not silently rewrite the old conclusion. I mark it superseded and link it to the replacement; the replacement links back. That preserves why the earlier decision was reasonable while making the current instruction obvious.

Periodically, I scan for stale statuses, contradictory guidance, duplicated incidents, broken links, and notes that should be promoted into stable service documentation. A tiny coverage script checks that every dated note appears in the index. Humans still decide whether the summaries are true.

When I create a note

I write an operational note when at least one of these is true:

the symptom had more than one plausible cause;
the winning check was non-obvious;
a reasonable attempted fix made no difference;
recovery and durable correction were different;
state outside the visible configuration mattered;
the same mistake could affect another service;
rollback required knowledge not encoded in the change itself.

Routine changes do not need an incident note. I create one when the investigation found something which is not obvious from the final configuration.

Once the fix is validated, I write the short note and add its one-line summary to the index. That is normally enough for the next investigation to start from the useful check instead of repeating the complete incident.

Keeping AI Agent Notes in My Homelab Repository

Fri, 05 Jun 2026 00:00:00 +0530

I use AI coding agents regularly while working on my homelab repository. The chat history is useful during a task, but a new session does not automatically know what was proved in an older one.

This became a problem for operational details. For example, one session found the safe way to evaluate my Nix flake without copying 20 GiB of ignored model files. Another found the exact SSH and TTY sequence required for privileged diagnostics. I did not want to investigate those details again.

I now store these notes as Markdown under docs/agent-memory/. They are part of the repository, so I can review and update them along with the configuration.

What I wanted from these notes

The system needed to be:

durable: survive chat sessions, machines, and editor profiles;
reviewable: change through the same review process as code;
routable: let an agent find one relevant note without reading hundreds;
historical: preserve old decisions without presenting them as current;
safe: record where secrets live, never their values;
cheap: make adding a lesson easier than rediscovering it.

I did not want one large file which every agent had to read at the start of a task. I also wanted the notes to remain visible outside one editor or agent.

One Markdown file for each topic

Every memory is a dated Markdown file under docs/agent-memory/:

1
2
3

2026-06-12-traefik-macvlan-asymmetric-return-route.md
2026-07-23-copyparty-rename-strips-group-write-arrs-import-denied.md
2026-08-10-netdata-2.10.3-collector-fixes.md

The ISO date gives a useful filesystem order. The descriptive suffix makes the file discoverable with ordinary text search. One topic per file lets a later note supersede one conclusion without invalidating an unrelated section of a large document.

New notes carry queryable frontmatter:

---
type: incident
title: Podman network drift recreate needs rm -f
description: Stopped containers remain associated with their networks.
timestamp: 2026-06-12
tags: [podman, networking]
resource: hosts/heavymetal/containers.nix
status: resolved
---

I normally write the body in this order:

What was the symptom?
What evidence separated it from similar failures?
What was the root cause?
What exact change fixed it?
How was the fix verified?
How can it be undone or superseded?

I include a failed attempt only when it is likely to be tried again. Normal terminal exploration and command typos do not need to be stored.

Using a small index to find the correct note

Opening every note at the start of every request would replace forgetting with context overload. The repository instead has a hand-curated index.md, grouped by service or topic. Each entry is one complete problem-to-outcome sentence:

1
2
3

- [2026-06-12-podman-network-drift-recreate-needs-force.md](...)
  — Recreating a drifted Podman network requires rm -f because stopped
  containers remain associated with it. (_resolved_)

The agent’s startup rule is simple: read the index, select the few summaries that match the current task, then open only those notes.

flowchart TB
	Q[New engineering request] --> I[Read compact routing index]
	I --> S{Select matching summaries}
	S --> N1[Open relevant incident note]
	S --> N2[Open relevant reference note]
	N1 --> W[Work with prior evidence]
	N2 --> W
	ALL[Hundreds of unrelated notes] -. not loaded .-> W

The routing index keeps startup context small while preserving access to detailed operational evidence.

I write these summaries by hand because a filename or heading usually does not say which fix actually worked. A script checks that every note is included in the index, but it does not generate the summary.

Not every note needs a dated file

Not every fact belongs in the dated-note catalog. I separate three classes:

Memory class	Example	Lifetime
Incident or migration	Why macvlan replies used the wrong route	Historical, may be superseded
Living reference	Port allocation inventory	Updated in place
Durable service guidance	Stable operating constraint	Promoted into reference documentation

A port inventory would become noisy as a sequence of dated files. A completed incident should not be silently rewritten whenever understanding changes. For example, I update the port allocation inventory in place. An incident stays as a dated file because a later note may supersede it without removing the old evidence.

Marking an old note as superseded

Deleting an old note destroys the path that explains why a decision existed. Leaving it unmarked lets an agent follow obsolete instructions. The compromise is bidirectional supersession.

The old note says:

`1`	`> Superseded by: [2026-08-17-new-design.md](...) — reason.`

The new note says:

`1`	`> Supersedes: [2026-06-05-old-design.md](...).`

Both frontmatter records and index entries carry the same state. The old evidence remains available, but every retrieval path points toward current guidance.

flowchart TB
	OLD[Old note: status superseded] -->|superseded by| NEW[New note: current guidance]
	NEW -->|supersedes| OLD
	INDEX[Routing index] -->|current entry| NEW
	INDEX -. historical entry marked superseded .-> OLD

Supersession preserves the audit trail while making the current instruction unambiguous.

I used this when a temporary design was retired and when a shell test showed that umask 0002 worked but the real container supervisor reset it. The old test was still useful, but the new note needed to be the current instruction.

Telling the agent when to read the notes

Notes do not help if the agent does not know when to read them. Repository instructions establish a retrieval protocol:

At the start of a request, read the routing index.
Open only relevant notes.
Consult living inventories before changing shared resources such as ports.
Record a newly proven operational lesson in the repository.
Update the index in the same change.

Without these repository instructions, the notes would only be documentation which an agent might or might not find.

flowchart TB
	TASK[Task begins] --> ROUTE[Index and instruction routing]
	ROUTE --> PRIOR[Relevant prior knowledge]
	PRIOR --> ACT[Implement and validate]
	ACT --> LESSON{New durable lesson?}
	LESSON -->|yes| NOTE[Write dated note and index summary]
	NOTE --> CHECK[Run coverage and consistency checks]
	CHECK --> TASK
	LESSON -->|no| DONE[Finish]

Work produces evidence, evidence becomes a note, and future instructions route the next task back through it.

Checking the index

The repository includes a small dependency-free checker. It compares dated files on disk with links in the index and reports:

notes missing from the index;
index links whose files no longer exist;
notes without frontmatter.

It exits nonzero for missing coverage or broken links. It does not generate summaries, decide which topic heading is best, or resolve contradictory advice. Those are semantic tasks.

A periodic manual pass checks what code cannot reliably infer:

stale pending or parked statuses;
one-directional supersession links;
newer notes that contradict older guidance;
duplicate incidents without a relationship;
ports or service references that drifted;
lessons mature enough to move into stable documentation.

The script checks the structure. I still review the summaries and decide whether one note replaces another.

Do not store secrets or duplicate the code

Repository memory must never contain secret values. A note may say that an MQTT password comes from a named SOPS secret and which service consumes it. It should not contain the password, a token copied from a log, or an unredacted credential example.

I also avoid storing temporary chat details, large summaries of the repository and facts already clear from the code. I add a note when it preserves a useful check, an operational problem or a decision which would take time to reconstruct.

Current workflow

I do not use a vector database or an embedding pipeline for this. Markdown, links, frontmatter, repository instructions and a small checker are enough for my repository.

The workflow is:

capture only proven lessons;
compress each lesson into a routable sentence;
load details only when relevant;
preserve history through supersession;
keep every memory visible to the people responsible for the system.

The agent still starts a new chat without the old conversation. It first reads the index, opens the notes related to the current task and continues with the checks and fixes already recorded there.