FINAL DECISIONS — new.apertia.cz server installment

Host 21.davidstrejc.cz (94.130.65.137, Hetzner, 2× Micron 3500 NVMe RAID1, 64 GB RAM) · audited 2026-08-04 · read-only audit, every claim re-verified against repo (file:line) and live host. This page is the decision record; nothing was changed on the server except publishing this page.

Verdict: No PHP memory leak. The box suffers from four compounding, fixable problems: a scheduler retry storm (process churn), JIT enabled for every PHP process on the host (segfaults + ~3.6 GB residency), thermal throttling (hardware), and a shared-search query shape that defeats every index built for it. Fix order is D1 → D2/D3 → D4/D5 → D7. No hardware purchase, no RAM upgrade, no PostgreSQL tuning knobs.

Corrections to the original audit (verified 2026-08-04)

Audit said	Verified reality
RAID `md3` mismatch_cnt=2560 (implied maybe benign)	md3 is the root filesystem (920 GB `/`, holds PostgreSQL). Swap is separate `md1`. The benign swap-mismatch explanation does not apply — D5 is mandatory.
"JIT wrapper host-only" (drift)	`/usr/local/bin/php8.5-octane` is a no-op passthrough (`exec /usr/bin/php8.5 "$@"`). JIT comes from system php.ini: `opcache.jit=1205`, 256 M buffer, `opcache.enable_cli=On` — every PHP process on the box (all vhosts, all CLI spawns) maps opcache 768 M + JIT 256 M = exactly the 1 GiB `/dev/zero` region every segfault dies in.
"Several PHP 8.5 segfaults Aug 1–3"	166 segfaults since Aug 1 (kernel journal), all at the same offset in the 1 GiB opcache+JIT mapping, mostly short-lived CLI processes — i.e. the scheduler storm is feeding the crash mill.
Candidate commits `d57ff2a21a`, `85276da811`	Exist, author Adam Kalíšek 2026-08-03, branch `codex/accounting-remediation-implementation-20260803` — NOT merged to main. They add exactly the missing occurrence/attempt state (+migration, typed `ScheduledJobExecutionException`, runtime service, tests).
Outbox 4.8 GB (retention gap)	Nightly purge already exists (`erp:outbox:purge --days=1 --vacuum`). Live: heap 334 MB, indexes+toast 4.4 GB — this is index bloat from daily delete churn, not missing retention. ~460 k processed rows/day.
Slow search → "repair query shape"	Sharper: trgm + `f_unaccent` expression indexes already exist, but the runtime emits `f_unaccent(COALESCE(col::text,''))` which matches neither index expression, and ORs `search_vector @@` with the full ILIKE fallback — the planner can never index-serve it (the repo's own 2026-07-16 migration says so). Fix layer: `FilterTextSearch` compiler, not new indexes.
Worker RSS 426–456 MiB (PSS unconfirmed)	PSS confirmed via `smaps_rollup`: ~390–400 MB per Octane worker. RSS ≈ PSS — the footprint is real, not shared-page double counting.

Decisions

P0D1 — Kill the active storm today: fix the recipient data.

Populate invoiceEmail (or email) on Account 1535 via normal UI/API. Pure data fix, zero code.
Resolver behavior is correct by policy (no Contact fallback, InvoiceRecipientResolver.php:13-24) — do not change it.
Verify: scheduler log shows the 08:00 job succeed once; PHP spawn count collapses next day.

P0D2 — Scheduler gets occurrence/attempt state via Adam's branch, review + rebase, no blind cherry-pick.

Root cause on main: last_run written only on success (ErpScheduledJobsRunCommand.php:196); failure leaves the occurrence eligible for the whole 1440-min grace (config/jobs.php:33) with the runner firing every minute → 1 424 retries of one dead occurrence. No retry/attempt columns exist on main at all.
Adopt d57ff2a21a + 85276da811 (branch codex/accounting-remediation-implementation-20260803): persist attempt state separately from last success, typed permanent-vs-transient failure, terminal permanent failure per occurrence, bounded backoff for transient. Review against main, rebase, land through normal gates. Owner of the review: David; author notified (closing the loop).
Also in that slice: delete the dead job_queue table mechanism (has attempts columns, zero consumers — doctrine point 4).
Verify: force one permanent failure in staging → exactly 1 attempt per occurrence; journal PHP spawn count per day drops to the low thousands.

P0D3 — JIT OFF host-wide. A/B on standby only to document the delta, not to decide it.

Evidence: 166 segfaults in 4 days inside the opcache+JIT SHM mapping; ~108–117 MB JIT residency × 32 web workers ≈ 3.6 GB; workload is I/O-bound ERP — JIT cannot pay for itself, and for the thousands of short-lived CLI spawns it is pure compile waste.
Change opcache.jit=off / opcache.jit_buffer_size=0 in the committed ini (see D6), CLI and web. Web picks it up on the next blue/green deploy; CLI immediately.
Standby A/B (replay Lead/Account/API traffic, compare p50/p95, PSS, segfault count) runs to record the numbers. Re-enable only if it shows a ≥10% latency win AND zero segfaults — not expected.
Verify: zero dev/zero segfaults over 7 days; worker PSS drop; p95 unchanged.

P0D4 — Thermal: hardware inspection ticket at Hetzner, before any software turbo-capping.

Package 87 °C against high=85 °C at ordinary load, peak 97.85 °C, 274 683 package throttle events on cpu0. This is a cooling defect (contact/dust/fan/airflow), not a tuning problem.
File Hetzner support ticket for fan/heatsink/airflow check in a low-traffic window (the Octane pair + hot standby tolerates the reboot).
D1–D3 reduce heat load but cannot fix cooling. No BIOS PL1/PL2 or no_turbo capping until after the inspection — capping first would just hide the defect and pay for it in latency.

P0D5 — RAID scrub with backup proof first. md3 is the root fs under PostgreSQL — treat it seriously.

Order is law: (1) pgbackrest --stanza=newerp --type=diff backup + verify; (2) re-run read-only scrub (echo check > /sys/block/md3/md/sync_action) in a maintenance window; (3) compare mismatch_cnt.
Stable or shrinking count on a busy fs → note and monitor monthly (checkarray). Growing count → investigate (SMART is clean on both drives, so suspect in-flight-write artifacts vs real divergence) before any repair. Never blind-repair RAID1 — repair copies primary→secondary without knowing which copy is right.
Backup NFS at 88% (70 T/80 T on 7.davidstrejc.cz): prune policy decision belongs to the backup host, scheduled this month.

P1D6 — One committed runtime contract. The host stops being the only place the truth lives.

Commit the primary unit new.apertia.cz-octane.service (with its drop-ins folded in) to deploy/systemd/ next to the standby unit; sync_unit_files_from_repo already enforces committed units on deploy — the primary just was never committed.
Delete the no-op /usr/local/bin/php8.5-octane wrapper; units call /usr/bin/php8.5 directly. Dead mechanism, doctrine point 4.
Move zzz-octane-stability.ini (opcache sizing, JIT=off per D3) into the repo (deploy/php/) and install it from the deploy, same pattern as unit files.
Verify: deploy preflight diff between host and repo is empty; a fresh host could be rebuilt from the repo alone.

P1D7 — Search: fix the compiler to emit what the indexes can serve. No new indexes.

Layer: app/Services/_Core/Filter/FilterTextSearch.php (shared runtime), per repo law — never entity branches.
(a) Emit f_unaccent(col) matching the existing expression indexes ({t}_{c}_unaccent_idx, GIN trgm), instead of f_unaccent(COALESCE(col::text,'')) which matches nothing. (b) Where search_vector is provisioned, use it alone — stop OR-ing with the ILIKE fallback that forces seqscan; ILIKE remains only for tables without fulltext. (c) COUNT then becomes index-served automatically (it already skips the relevance CASE; it pays the WHERE).
Delete production-dead paths kept alive by self-referential tests: applyCombinedSearch, applyFuzzyNameSearch, addFuzzySimilarityScore + their unit tests.
Acceptance numbers required (doctrine point 20): Lead list p50/p95 and COUNT before/after; targets <800 ms page, <500 ms count (from 3.6–4.3 s / 2.6–3.3 s).
Standing ban re-affirmed: never drop emails_bm25_idx (ParadeDB scans don't increment idx_scan; marked load-bearing in the 2026-07-16 migration).

P1D8 — One leads schema truth. Module owns Lead; the tenant-tree copy dies.

Two live declarations confirmed: database/migrations/tenant/2025_12_16_000002_create_crm_activity_tables.php and app/Modules/CRM/migrations/2025_01_01_000002_create_leads_table.php — both run on every CRM tenant, producing verified duplicate index pairs live in prod: status, source, assigned_user_id (each indexed twice under different names).
Repair at the migration layer: drop one index of each duplicate pair; retire the tenant-tree leads DDL (module owns the entity). Also drop the *_custom_fields_idx GIN family that survived the 2026-07-16 purge only because that purge matched the _gin suffix (same dead-index rationale applies).
Ships as its own reviewed migration slice with erp:schema:diff proof, after D7 (don't move two things under the planner at once).

P2D9 — Outbox: reclaim the 4.4 GB of index/toast bloat; retention already works.

REINDEX (CONCURRENTLY) on outbox_messages indexes in a maintenance window, then monitor — daily delete churn (~460 k rows/day) regrows bloat, so add a monthly reindex to the maintenance schedule rather than a one-off.
No change to the purge command or the outbox mechanism.

P2D10 — Hygiene watchlist, re-measured after D1–D3 land.

Swap 11.5 GB used with 25 GB available RAM — expected to shrink after JIT-off; re-measure before touching swappiness.
Root fs 74%, analytics_visits 2.2 GB + analytics_page_metrics 0.7 GB → retention decision in a later slice.
PostgreSQL temp 29 GB cumulative on the tenant DB — driven by the D7 seqscan sorts; re-read pg_stat_database after D7 instead of raising work_mem.

Execution log — 2026-08-04 evening (non-harmful actions executed)

DONEStorm stopped at 18:10 CEST. The failing job is Send invoice payment reminders (dunning), Failed 944× today alone — every minute since midnight, single candidate invoice 202600538 ("The invoice has no valid billing recipient"). BAEST Machinery Holding has no email anywhere: Espo legacy has zero address rows, no contacts with email, `email` and `invoice_email` both NULL — D1's "populate from authoritative source" is impossible until accounting supplies the address. Interim surgical fix executed instead: autoDunningDisabled=true on invoice 202600538 via EntityManager (audited write, reversible). Next run: Success, Candidates: 0, occurrence closed, no further catch-up runs. Re-enable the flag the day accounting fills in BAEST's billing email.

DONED3 first half: JIT off for all new CLI processes. /etc/php/8.5/cli/conf.d/zzz-octane-stability.ini now opcache.jit=disable + jit_buffer_size=0 (SHM segment not allocated at all). The segfault mill (short-lived CLI spawns) is dry as of now; running Octane workers keep JIT until the next blue/green deploy restarts them — that deploy completes D3. Contract copy committed to deploy/php/.

P0NEW finding — the backup lane was dead, now recovering. pgbackrest repo1 (/var/lib/pgbackrest, daily crons) was never stanza-created — every daily repo1 backup has failed forever; repo2 (NFS) broke ~Aug 2 with permission-denied on backup.info (readable again by Aug 4 — NAS-side change, cause unconfirmed). Net: newest base backup was 13 days old; only WAL archiving stayed healthy. Executed: repo2 diff backup started 18:10 (running); repo1 stanza-create + initial full scheduled 03:30; RAID md3 read-only scrub scheduled 01:00 (both transient systemd timers). Morning check: pgbackrest info shows fresh diff + repo1 full; mismatch_cnt compared against 2560. Root-cause the repo1 cron misconfig + NFS perms in the D5 slice.

DONED5 executed (2026-08-05 morning): scrub clean-ish, repo1 resurrected, backup redundancy restored.

RAID md3 scrub completed overnight: mismatch_cnt 2560 → 384, trend down — consistent with in-flight-write artifacts on a busy root fs, not divergence. Monitor monthly (checkarray); no repair.
Zero PHP segfaults in the 13 h since JIT-off (was ~40/day).
Backup forensics: repo1 was deliberately disabled 2026-05-11 — it pointed at /data, which never existed — and its daily 04:00 crons were left behind, failing every day since ("key '1' is not valid"). The decoy /etc/pgbackrest.conf (unused — the real config is /etc/pgbackrest/pgbackrest.conf) was deleted. repo1 re-enabled on the root NVMe (/var/lib/pgbackrest, retention 2 full / 14 diff, bundle+block), async spool moved to /var/spool/pgbackrest to stop it colliding with the repo layout, stanza created, initial full running. The existing 04:00 crons become functional unchanged. WAL archiver verified healthy after every step.
repo2 (NFS) lane also healthy again: fresh diffs 2026-08-04 18:10 (manual) and 2026-08-05 05:30 (cron). NFS caution: the mount is hard,timeo=600s — deep directory crawls (find/ls -R) over it can wedge a session for many minutes; keep monitoring probes shallow.

DONED6 first half: runtime contract committed. Primary Octane unit (base + both drop-ins folded verbatim) + the CLI opcache ini now live in the repo (deploy/systemd/, deploy/php/, commit 2464be282c); sync_unit_files_from_repo installs the unit on the next deploy with zero behavior change. Remaining reviewed slice: extend the deploy installer to deploy/php/, then delete the no-op wrapper and point both units at /usr/bin/php8.5.

DONERoadRunner verdict: nothing wrong with RR itself. RoadRunner CLI 2025.1.14 (current generation); worker recycling active via --max-requests=5000; PSS flat across worker lifetime (no leak); the segfaults live in PHP's opcache+JIT SHM, not RR. No RR upgrade, no .rr.yaml, no supervisor memory cap needed — recycling already bounds drift.

Fleet addendum — 33.davidstrejc.cz (2026-08-05 morning)

P0D11 — 33.davidstrejc.cz (78.46.85.13, Draivix SaaS instance box): CPU storm, not RAM.

Reported as "under load, RAM in shit". Verified: RAM is healthy — 24 GB available, zero OOM kills since Aug 1, memory PSI 0.00. The scary swap number (21 GB) is parked history, 4.2 GB of it one idle trido-engine engine-api process; no active swapping.
Real problem: load 134 on 20 cores, run queue 105, CPU PSI some=91%, 0 idle / 0 iowait. Driver: 12 per-instance Laravel schedulers × per-tenant events (29 sites, every-minute lanes, demo alone has 27 tenant DBs) ≈ 175 concurrent short-lived artisan spawns at all times — each paying full framework boot plus tracing JIT compile with hyper-eager tuning (99-fastest.ini: opcache.jit=tracing, jit_prof_threshold=0.001, jit_hot_loop=64).
No failing-job retry loops found in sampled tenants — this is the scheduler fan-out architecture itself scaling as instances × tenants × cadence. D2 (occurrence state + one coordinator) is therefore a FLEET fix, priority raised — the per-tenant every-minute loop is untenable on multi-instance boxes.
Executed 06:44 (non-harmful): CLI JIT off via zzz-cli-no-jit.ini override (disable + buffer 0), fpm/web copies untouched, zero restarts. Measured effect within 30 min: load-1m 134 → 55, run queue 105 → 2, concurrent PHP 175 → 73. CPU saturation gone; the remaining base load is the fan-out architecture (D2 fleet fix).
Remaining for the D2 slice on this box: consolidate cadence (most every-minute lanes have no business running per-minute on idle demo tenants), stagger the 12 scheduler timers (RandomizedDelaySec), and consider per-instance lane gating via the module settings registry — never per-host hacks.

DONED12 — vyroba.draivix.com "extremely slow" (2026-08-05): root-caused and fixed, PG exonerated.

Verified stack layer by layer: PostgreSQL innocent (tenant DB 122 MB, cache hit 99.97%, sane globals); code identical to the fleet; login TTFB ≈ prod. The slowness was two infrastructure facts on 33.davidstrejc.cz: (a) the box-wide CPU storm (D11 — tracing-JIT CLI spawns; fixed 06:44), and (b) the instance ran Octane with 2 HTTP workers — measured: 8 concurrent users queued to 1.7–6.5 s TTFB while a lone request took 0.2 s. Any real factory-floor concurrency = "extremely slow".
Fixes applied: workers 2→6 + task 1→2 (unit drop-in, PSS ~150–250 MB/worker, ample headroom), CPUWeight=300 on vyroba Octane; all 12 instance schedulers deprioritized to CPUWeight=40, queues 50; hard CPU caps on the non-customer whales (demo.draivix.com scheduler = 27 tenants → CPUQuota=300%, the four *.deverp.cz dev instances → 100%/50%). Stale failed satellite-deploy unit cleared.
Measured after: 8-parallel 0.9–1.7 s (from 1.7–6.5 s) even during the business-hours load wave; serial ~0.9 s under load-138 contention, ~0.2 s on a quiet box.
Follow-ups: mirror the 6+2 sizing into the satellite instance template (it currently defaults every instance to 2+1); trido.draivix.com periodically re-runs Cycle schema introspection (known disease, fixed in mainline via TenantDb cache — trido runs a checkpointed side-tree aa5c14bff, needs its own update slice); the business-hours load wave still peaks ~140 — D2 fleet scheduler consolidation remains the real fix.

Explicit NON-decisions

No RAM upgrade, no server replacement — 64 GB is sufficient once JIT residency and churn are gone.
No work_mem / shared_buffers changes — fundamentals are healthy (cache hit 97.6%, 67/400 connections, no lock waits).
No worker-count changes — 16+16 is load-bearing for the deploy window (standby carries 100% while primary drains).
No cross-host HA this quarter — the pair-on-one-host limitation stays documented and accepted.
No scheduler architecture rewrite beyond D2 — declarations consolidate into module manifests + JobRegistry as part of that slice, one coordinator, nothing new invented.

Execution order & verification gate

#	Decision	Window	Proof required before "done"
1	D1 recipient data	today	08:00 job green in scheduler log; daily PHP spawn count collapses
2	D2 scheduler state (Adam's branch)	this week	staged permanent failure → 1 attempt; spawn count < 3 k/day
3	D3 JIT off + A/B record	this week (next deploy)	0 segfaults in 7 days; PSS delta; p50/p95 unchanged
4	D4 Hetzner cooling ticket	file now, window ASAP	package temp < 80 °C under load; throttle counter static
5	D5 backup + RAID scrub	next maintenance window	pgbackrest verified; mismatch_cnt trend recorded
6	D6 runtime contract committed	with D3's deploy	host↔repo unit/ini diff empty
7	D7 search compiler	next perf slice	Lead p50/p95 + COUNT before/after numbers published
8	D8 leads schema unification	after D7	erp:schema:diff clean; duplicate indexes gone
9	D9 outbox reindex	monthly maintenance	outbox total size < 1 GB post-reindex

Evidence trail: repo file:line citations verified against main @ a19161caa2; live host readings 2026-08-04 ~17:50 CEST (sensors, /proc/mdstat, smaps_rollup, kernel journal, pg_stat). Full audit conversation retained in the ops session. This page: /home/._hostname/public_html/index.html on 21.davidstrejc.cz — noindex, informational only.