fix(bot): Live-Stand CT116 uebernehmen (Grok-4.3, Hermes-Monitoring, Loki-Filter, Antwort-Timeout)
- llm.py: Text-Modell auf x-ai/grok-4.3 - telegram_bot.py: Fortschritts-Logik auf asyncio.wait_for + shield (15s Timeout) - monitor.py: Hermes-Healthchecks (CT151), Alert-Dedup, EXPECTED_STOPPED erweitert - core/loki_client.py: zentrale ERROR_EXCLUDE_RE fuer transiente cloudflared/QUIC-Flaps - Doku: JELLYFIN.md, infrastructure/STATE.md, hardware-ai-local-coding.md, sessions/ Bis 21.06. uncommittete Live-Edits, gesichert in backup/live-state-20260621. Ref #123.
This commit is contained in:
parent
169ea0c71f
commit
1cbc5df1ad
8 changed files with 362 additions and 34 deletions
|
|
@ -190,3 +190,39 @@ Neue Leitlinie:
|
||||||
- **M5 Ultra** nur dann, wenn ein konkretes Modell das rechtfertigt.
|
- **M5 Ultra** nur dann, wenn ein konkretes Modell das rechtfertigt.
|
||||||
|
|
||||||
`homelab.conf` und `MASTER_INDEX.md` werden parallel angepasst.
|
`homelab.conf` und `MASTER_INDEX.md` werden parallel angepasst.
|
||||||
|
|
||||||
|
## Entscheidungsnotiz: Qwen3.6 + BeeLlama fuer opencode-Tests
|
||||||
|
|
||||||
|
Stand: 2026-05-18
|
||||||
|
|
||||||
|
Qwen3.6 ueber BeeLlama ist fuer einfache, kontrollierte `opencode`-Tests ausreichend:
|
||||||
|
|
||||||
|
- Git-Repos klonen
|
||||||
|
- README, `package.json` und Setup-Hinweise lesen
|
||||||
|
- Projektstruktur erkunden
|
||||||
|
- Shell-/Git-Kommandos ausfuehren
|
||||||
|
- Tests finden und starten
|
||||||
|
- kleine, klar begrenzte Codeaenderungen vorbereiten
|
||||||
|
|
||||||
|
Wichtig: BeeLlama macht das Modell schneller und lokal praktikabler, aber nicht automatisch intelligenter. Fuer komplexe Coding-Agent-Aufgaben wie grosse Refactorings, viele Dateien gleichzeitig, schwierige Debug-Schleifen oder produktive Aenderungen bleibt ein spezialisiertes Coding-Modell zuverlaessiger.
|
||||||
|
|
||||||
|
Empfohlener erster Testauftrag:
|
||||||
|
|
||||||
|
```text
|
||||||
|
Klon dieses Repo nach /tmp/opencode-test, lies README und package.json, aber aendere nichts. Sag danach, wie man es startet und welche Tests definiert sind.
|
||||||
|
```
|
||||||
|
|
||||||
|
## Entscheidungsnotiz: zweite RTX 3090
|
||||||
|
|
||||||
|
Eine zweite RTX 3090 verbessert die Qualitaet desselben Modells nicht direkt. Dasselbe Modell mit derselben Quantisierung liefert grundsaetzlich dieselbe Antwortqualitaet.
|
||||||
|
|
||||||
|
Eine zweite GPU hilft indirekt durch:
|
||||||
|
|
||||||
|
- groessere Modelle, die vorher nicht in den VRAM passten
|
||||||
|
- bessere Quantisierung, z.B. Q5/Q6 statt Q4
|
||||||
|
- mehr Kontext/KV-Cache
|
||||||
|
- weniger CPU-Offload
|
||||||
|
- hoehere Tokenrate bei gutem Layer-/Tensor-Splitting
|
||||||
|
- mehr parallele Requests
|
||||||
|
|
||||||
|
Fuer `opencode` waere der echte Qualitaetsgewinn daher nicht die zweite GPU selbst, sondern dass dadurch eventuell ein besseres oder groesseres Coding-Modell lokal betrieben werden kann.
|
||||||
|
|
|
||||||
|
|
@ -140,3 +140,58 @@ curl -s "http://100.77.105.3:8096/Items?api_key=7285b4a8793541648bf156599ae05b43
|
||||||
| Ralf | nein |
|
| Ralf | nein |
|
||||||
| rolf | nein |
|
| rolf | nein |
|
||||||
| wilfrid | nein |
|
| wilfrid | nein |
|
||||||
|
|
||||||
|
## Troubleshooting: Details fehlen / Remote-Zugriff wirkt kaputt
|
||||||
|
|
||||||
|
Stand: 2026-05-16
|
||||||
|
|
||||||
|
Symptom:
|
||||||
|
- Filmübersicht ist erreichbar, aber Detailseiten/Filmdetails werden im Client nicht angezeigt.
|
||||||
|
- Zugriff über Tailscale-IP `100.77.105.3:8096` hängt oder läuft in Timeouts.
|
||||||
|
|
||||||
|
Befund vom 2026-05-16:
|
||||||
|
- Jellyfin selbst war lokal gesund:
|
||||||
|
- `http://192.168.178.149:8096/health` -> `Healthy`
|
||||||
|
- API `/Items/Counts` lieferte `897` Filme.
|
||||||
|
- Beispiel-Film `2 Guns` hatte Beschreibung, Genres, TMDB/IMDB IDs und Bilder.
|
||||||
|
- Medien-Mounts waren aktiv:
|
||||||
|
- `/mnt/nas`
|
||||||
|
- `/srv/media`
|
||||||
|
- Docker-Container `jellyfin` war `healthy`.
|
||||||
|
- Ursache war Tailscale auf dem Jellyfin-Host:
|
||||||
|
- `tailscale ip -4` meldete zuerst `no current Tailscale IPs; state: NeedsLogin`
|
||||||
|
- `tailscale status` meldete `Logged out`.
|
||||||
|
- Nach Re-Login war `100.77.105.3` wieder aktiv und `http://100.77.105.3:8096/health` antwortete wieder `Healthy`.
|
||||||
|
- Danach waren die Filmdetails im Client wieder sichtbar.
|
||||||
|
|
||||||
|
Schnelltest:
|
||||||
|
```bash
|
||||||
|
# Von pve3 lokal im LAN:
|
||||||
|
curl -sS http://192.168.178.149:8096/health
|
||||||
|
|
||||||
|
# Von CT 116 / Hausmeister oder einem Tailscale-Host:
|
||||||
|
curl -sS http://100.77.105.3:8096/health
|
||||||
|
curl -sS "http://100.77.105.3:8096/Items/Counts?api_key=7285b4a8793541648bf156599ae05b43"
|
||||||
|
```
|
||||||
|
|
||||||
|
Fix:
|
||||||
|
1. Auf dem Jellyfin-Host Tailscale-Status prüfen:
|
||||||
|
```bash
|
||||||
|
tailscale ip -4
|
||||||
|
tailscale status --self
|
||||||
|
```
|
||||||
|
2. Wenn `NeedsLogin` oder `Logged out`: Tailscale neu anmelden.
|
||||||
|
3. Danach erneut prüfen:
|
||||||
|
```bash
|
||||||
|
curl -sS http://100.77.105.3:8096/health
|
||||||
|
```
|
||||||
|
4. Im Browser Jellyfin hart neu laden oder neu einloggen, falls der Webclient alte kaputte API-Antworten gecacht hat.
|
||||||
|
|
||||||
|
Wichtig:
|
||||||
|
- Fehlende Details bedeuten nicht automatisch kaputte Jellyfin-Datenbank oder fehlende Metadaten.
|
||||||
|
- Erst lokale API gegen `192.168.178.149:8096` prüfen, dann Tailscale/Cloudflare-Zugriff.
|
||||||
|
- `systemctl is-active jellyfin` kann irreführend sein, weil Jellyfin hier im Docker-Container läuft. Aussagekräftiger ist:
|
||||||
|
```bash
|
||||||
|
docker ps | grep jellyfin
|
||||||
|
curl -sS http://127.0.0.1:8096/health
|
||||||
|
```
|
||||||
|
|
|
||||||
|
|
@ -5,6 +5,28 @@ from datetime import datetime, timezone, timedelta
|
||||||
|
|
||||||
LOKI_URL = "http://100.109.206.43:3100"
|
LOKI_URL = "http://100.109.206.43:3100"
|
||||||
|
|
||||||
|
# Transiente cloudflared/Cloudflare-Edge-Flaps erzeugen oft error="..."
|
||||||
|
# in WRN/ERR-Zeilen, sind aber keine App-Fehler von WordPress/RSS.
|
||||||
|
ERROR_EXCLUDE_RE = (
|
||||||
|
"caller=metrics|query_hash=|executing query|scheduler_processor|"
|
||||||
|
"Aborted connection|systemd-networkd-wait-online|context canceled|"
|
||||||
|
"AH01630: client denied|flag evaluation succeeded|pluginsAutoUpdate|"
|
||||||
|
"control stream encountered a failure while serving|Serve tunnel error|"
|
||||||
|
"failed to serve tunnel connection|Connection terminated|"
|
||||||
|
"failed to dial to edge with quic|Failed to dial a quic connection|"
|
||||||
|
"no recent network activity|failed to accept QUIC stream|"
|
||||||
|
"control stream|accept stream listener|serve tunnel|tunnel connection|"
|
||||||
|
"edge with quic|quic connection|QUIC stream|"
|
||||||
|
"accept stream listener encountered a failure while serving|"
|
||||||
|
"datagram manager encountered a failure while serving|"
|
||||||
|
"datagram manager|datagram handler|"
|
||||||
|
"failed to run the datagram handler|failed to accept incoming stream requests|"
|
||||||
|
"Application error 0x0|connIndex=|"
|
||||||
|
"beellama-tunnel.service: Main process exited|"
|
||||||
|
"beellama-expose.service: Main process exited|"
|
||||||
|
"health\(warnable=mapresponse-timeout\)"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
def _query(endpoint: str, params: dict, base_url: str = None) -> dict:
|
def _query(endpoint: str, params: dict, base_url: str = None) -> dict:
|
||||||
url = f"{base_url or LOKI_URL}{endpoint}"
|
url = f"{base_url or LOKI_URL}{endpoint}"
|
||||||
|
|
@ -49,9 +71,9 @@ def query_logs(query: str, hours: float = 1, limit: int = 100) -> list[dict]:
|
||||||
def get_errors(container: str = None, hours: float = 1, limit: int = 200) -> list[dict]:
|
def get_errors(container: str = None, hours: float = 1, limit: int = 200) -> list[dict]:
|
||||||
"""Get error-level logs, optionally filtered by container hostname."""
|
"""Get error-level logs, optionally filtered by container hostname."""
|
||||||
if container:
|
if container:
|
||||||
q = f'{{host="{container}"}} |~ "(?i)(error|fatal|panic|traceback|exception)" !~ "caller=metrics|query_hash=|executing query|scheduler_processor|Aborted connection|systemd-networkd-wait-online|context canceled|AH01630: client denied|flag evaluation succeeded|pluginsAutoUpdate|terror"'
|
q = f'{{host="{container}"}} |~ "(?i)(error|fatal|panic|traceback|exception)" !~ "{ERROR_EXCLUDE_RE}"'
|
||||||
else:
|
else:
|
||||||
q = '{job=~".+"} |~ "(?i)(error|fatal|panic|traceback|exception)" !~ "caller=metrics|query_hash=|executing query|scheduler_processor|Aborted connection|systemd-networkd-wait-online|context canceled|AH01630: client denied|flag evaluation succeeded|pluginsAutoUpdate|terror"'
|
q = f'{{job=~".+"}} |~ "(?i)(error|fatal|panic|traceback|exception)" !~ "{ERROR_EXCLUDE_RE}"'
|
||||||
return query_logs(q, hours=hours, limit=limit)
|
return query_logs(q, hours=hours, limit=limit)
|
||||||
|
|
||||||
|
|
||||||
|
|
@ -102,7 +124,8 @@ def get_health(container: str, hours: float = 24) -> dict:
|
||||||
|
|
||||||
return {
|
return {
|
||||||
"host": container,
|
"host": container,
|
||||||
"errors_last_{hours}h": error_count,
|
"error_count": error_count,
|
||||||
|
f"errors_last_{hours}h": error_count,
|
||||||
"sending_logs": has_recent,
|
"sending_logs": has_recent,
|
||||||
"status": "healthy" if error_count < 5 and has_recent else
|
"status": "healthy" if error_count < 5 and has_recent else
|
||||||
"warning" if error_count < 20 else "critical",
|
"warning" if error_count < 20 else "critical",
|
||||||
|
|
@ -114,6 +137,7 @@ WATCHED_SERVICES = [
|
||||||
("wordpress-v2", "wordpress"),
|
("wordpress-v2", "wordpress"),
|
||||||
("fuenfvoracht", "fuenfvoracht"),
|
("fuenfvoracht", "fuenfvoracht"),
|
||||||
("homelab-ai-bot", "hausmeister"),
|
("homelab-ai-bot", "hausmeister"),
|
||||||
|
("hermes-mu", "hermes-gateway"),
|
||||||
]
|
]
|
||||||
|
|
||||||
|
|
||||||
|
|
@ -121,7 +145,7 @@ def count_errors(hours: float = 24) -> dict:
|
||||||
"""Zählt Fehler-Log-Einträge über einen Zeitraum via Loki metric query."""
|
"""Zählt Fehler-Log-Einträge über einen Zeitraum via Loki metric query."""
|
||||||
now = datetime.now(timezone.utc)
|
now = datetime.now(timezone.utc)
|
||||||
start = now - timedelta(hours=hours)
|
start = now - timedelta(hours=hours)
|
||||||
q = '{job=~".+"} |~ "(?i)(error|fatal|panic|traceback|exception)" !~ "caller=metrics|query_hash=|executing query|scheduler_processor|Aborted connection|systemd-networkd-wait-online|context canceled|AH01630: client denied|flag evaluation succeeded|pluginsAutoUpdate|terror"'
|
q = f'{{job=~".+"}} |~ "(?i)(error|fatal|panic|traceback|exception)" !~ "{ERROR_EXCLUDE_RE}"'
|
||||||
# Loki instant metric query für Gesamtanzahl
|
# Loki instant metric query für Gesamtanzahl
|
||||||
data = _query("/loki/api/v1/query_range", {
|
data = _query("/loki/api/v1/query_range", {
|
||||||
"query": q,
|
"query": q,
|
||||||
|
|
@ -160,7 +184,7 @@ def check_service_restarts(minutes: int = 35) -> list[dict]:
|
||||||
ERROR_RATE_THRESHOLDS = {
|
ERROR_RATE_THRESHOLDS = {
|
||||||
"rss-manager": 15,
|
"rss-manager": 15,
|
||||||
"wordpress-v2": 10,
|
"wordpress-v2": 10,
|
||||||
"forgejo": 200, # Oeffentlich — Scanner-404er sind normal
|
"forgejo": 200, # Oeffentlich – Scanner-404er sind normal
|
||||||
}
|
}
|
||||||
ERROR_RATE_DEFAULT = 25
|
ERROR_RATE_DEFAULT = 25
|
||||||
|
|
||||||
|
|
@ -171,7 +195,7 @@ def check_error_rate(minutes: int = 30) -> list[dict]:
|
||||||
alerts = []
|
alerts = []
|
||||||
now = datetime.now(timezone.utc)
|
now = datetime.now(timezone.utc)
|
||||||
for host in all_hosts:
|
for host in all_hosts:
|
||||||
q = f'count_over_time({{host="{host}"}} |~ "(?i)error" !~ "caller=metrics|query_hash=|executing query|scheduler_processor|Aborted connection|systemd-networkd-wait-online|context canceled|AH01630: client denied|flag evaluation succeeded|pluginsAutoUpdate|terror" [{minutes}m])'
|
q = f'count_over_time({{host="{host}"}} |~ "(?i)error" !~ "{ERROR_EXCLUDE_RE}" [{minutes}m])'
|
||||||
data = _query("/loki/api/v1/query", {"query": q, "time": _ns(now)})
|
data = _query("/loki/api/v1/query", {"query": q, "time": _ns(now)})
|
||||||
count = sum(
|
count = sum(
|
||||||
int(float(r.get("value", [None, "0"])[1]))
|
int(float(r.get("value", [None, "0"])[1]))
|
||||||
|
|
|
||||||
|
|
@ -27,8 +27,8 @@ OLLAMA_MODELS = set()
|
||||||
|
|
||||||
|
|
||||||
def warmup_ollama():
|
def warmup_ollama():
|
||||||
"""No-Op: Text-Modell laeuft jetzt ueber OpenRouter (Grok 4.1 Fast), kein Ollama-Warmup noetig."""
|
"""No-Op: Text-Modell laeuft ueber OpenRouter (x-ai/grok-4.3), kein Ollama-Warmup noetig."""
|
||||||
log.info('Ollama warmup uebersprungen — Text laeuft ueber OpenRouter (Grok 4.1 Fast)')
|
log.info('Ollama warmup uebersprungen — Text laeuft ueber OpenRouter (x-ai/grok-4.3)')
|
||||||
PASSTHROUGH_TOOLS = {"get_temperaturen", "get_energie", "get_heizung"}
|
PASSTHROUGH_TOOLS = {"get_temperaturen", "get_energie", "get_heizung"}
|
||||||
|
|
||||||
_LOCAL_OVERRIDES = [
|
_LOCAL_OVERRIDES = [
|
||||||
|
|
|
||||||
|
|
@ -4,6 +4,7 @@ import sys
|
||||||
import os
|
import os
|
||||||
import json
|
import json
|
||||||
import hashlib
|
import hashlib
|
||||||
|
import re
|
||||||
import requests
|
import requests
|
||||||
import time
|
import time
|
||||||
from datetime import datetime, timezone
|
from datetime import datetime, timezone
|
||||||
|
|
@ -22,6 +23,8 @@ ALERT_COOLDOWN_SECONDS = {
|
||||||
"memory_expiry": 43200,
|
"memory_expiry": 43200,
|
||||||
"default": 3600,
|
"default": 3600,
|
||||||
"error_rate": 1800,
|
"error_rate": 1800,
|
||||||
|
"hermes": 7200,
|
||||||
|
"backup": 21600,
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
|
|
@ -45,10 +48,18 @@ HTTP_HEALTH_CHECKS = [
|
||||||
{"name": "WordPress (CT 101)", "url": "http://10.10.10.101/robots.txt"},
|
{"name": "WordPress (CT 101)", "url": "http://10.10.10.101/robots.txt"},
|
||||||
{"name": "Matomo (CT 109)", "url": "http://10.10.10.109"},
|
{"name": "Matomo (CT 109)", "url": "http://10.10.10.109"},
|
||||||
{"name": "Grafana (CT 110)", "url": "http://10.10.10.110:3000/api/health"},
|
{"name": "Grafana (CT 110)", "url": "http://10.10.10.110:3000/api/health"},
|
||||||
{"name": "Flugscanner-Agent (pve-pp-1)", "url": "http://100.126.26.46:5010/status",
|
|
||||||
"retries": 5, "timeout": 25, "retry_delay": 6, "host": "pve-pp-1"},
|
|
||||||
]
|
]
|
||||||
|
|
||||||
|
HERMES_HEALTH_CHECKS = [
|
||||||
|
{"name": "Hermes Promtail", "url": "http://100.109.174.120:9080/ready", "timeout": 5},
|
||||||
|
{"name": "Hermes node_exporter", "url": "http://100.109.174.120:9100/metrics", "timeout": 5},
|
||||||
|
{"name": "Hermes Websuche CT121", "url": "http://100.74.196.29:8080/search?q=hausmeister-health&format=json", "timeout": 8},
|
||||||
|
{"name": "Loki", "url": "http://100.109.206.43:3100/ready", "timeout": 5},
|
||||||
|
{"name": "PBS Muldenstein", "url": "https://100.99.139.22:8007", "timeout": 5, "allow_status": {200, 401}},
|
||||||
|
]
|
||||||
|
|
||||||
|
HERMES_LOG_HOST = "hermes-mu"
|
||||||
|
|
||||||
EXPECTED_STOPPED = {
|
EXPECTED_STOPPED = {
|
||||||
(115, "pve-ka-1"), # flugscanner-asia-old (gestoppt, Cluster pve1) — Live CT auf pve-pp-1
|
(115, "pve-ka-1"), # flugscanner-asia-old (gestoppt, Cluster pve1) — Live CT auf pve-pp-1
|
||||||
(115, "pve-ka-3"), # dieselbe CT, zweite API-Sicht (Cluster)
|
(115, "pve-ka-3"), # dieselbe CT, zweite API-Sicht (Cluster)
|
||||||
|
|
@ -57,6 +68,9 @@ EXPECTED_STOPPED = {
|
||||||
(504, "pve-ka-2"), # Shop-Template — stopped
|
(504, "pve-ka-2"), # Shop-Template — stopped
|
||||||
(8000, "pve-ka-2"), # Kunde0-Shop — stopped
|
(8000, "pve-ka-2"), # Kunde0-Shop — stopped
|
||||||
(8010, "pve-ka-2"), # Kunde1-Shop — stopped
|
(8010, "pve-ka-2"), # Kunde1-Shop — stopped
|
||||||
|
(121, "pve-ka-1"), # Schawarma-Shop — gestoppt
|
||||||
|
(121, "pve-ka-2"), # Schawarma-Shop — gestoppt
|
||||||
|
(121, "pve-ka-3"), # Schawarma-Shop — gestoppt
|
||||||
}
|
}
|
||||||
|
|
||||||
# VMIDs, die auf jedem Proxmox-Host in CONFIG ok sind, solange status == stopped
|
# VMIDs, die auf jedem Proxmox-Host in CONFIG ok sind, solange status == stopped
|
||||||
|
|
@ -83,12 +97,60 @@ def _is_host_suppressed(host: str, suppressed_hosts: set) -> bool:
|
||||||
return False
|
return False
|
||||||
|
|
||||||
|
|
||||||
|
def check_hermes() -> list[str]:
|
||||||
|
"""Prueft Hermes (CT151) als kritischen Dienst ueber HTTP und Loki."""
|
||||||
|
alerts = []
|
||||||
|
headers = {"User-Agent": "Mozilla/5.0 (Hausmeister-Bot/1.0 hermes-check)"}
|
||||||
|
for check in HERMES_HEALTH_CHECKS:
|
||||||
|
allow_status = check.get("allow_status") or set(range(200, 400))
|
||||||
|
try:
|
||||||
|
r = requests.get(
|
||||||
|
check["url"],
|
||||||
|
timeout=check.get("timeout", 5),
|
||||||
|
allow_redirects=False,
|
||||||
|
headers=headers,
|
||||||
|
verify=False,
|
||||||
|
)
|
||||||
|
if r.status_code not in allow_status:
|
||||||
|
alerts.append(f"🔴 Hermes: {check['name']} HTTP {r.status_code}")
|
||||||
|
except requests.RequestException as e:
|
||||||
|
alerts.append(f"🔴 Hermes: {check['name']} nicht erreichbar: {str(e)[:80]}")
|
||||||
|
|
||||||
|
health = loki_client.get_health(HERMES_LOG_HOST, hours=1)
|
||||||
|
if health.get("status") == "silent":
|
||||||
|
alerts.append("⚠️ Hermes: keine aktuellen Loki-Logs von hermes-mu")
|
||||||
|
elif health.get("status") == "critical":
|
||||||
|
cnt = health.get("error_count", health.get("errors_last_1h", "?"))
|
||||||
|
alerts.append(f"🔴 Hermes: {cnt} Fehlerlogs in Loki (1h)")
|
||||||
|
|
||||||
|
errors = loki_client.get_errors(container=HERMES_LOG_HOST, hours=0.5, limit=20)
|
||||||
|
for e in errors:
|
||||||
|
if "error" in e:
|
||||||
|
continue
|
||||||
|
line = (e.get("line") or "").lower()
|
||||||
|
if "diskstats_linux.go" in line and "disabling udev device properties" in line:
|
||||||
|
continue
|
||||||
|
if "collector failed" in line and "node_exporter" in line:
|
||||||
|
continue
|
||||||
|
if "beellama-tunnel.service" in line or "beellama-expose.service" in line:
|
||||||
|
if "main process exited" in line:
|
||||||
|
continue
|
||||||
|
if "health(warnable=mapresponse-timeout)" in line:
|
||||||
|
continue
|
||||||
|
if any(term in line for term in ["traceback", "exception", "failed", "timeout", "tool_call", "connection refused"]):
|
||||||
|
alerts.append(f"🔴 Hermes Log: {(e.get('line') or '')[:140]}")
|
||||||
|
break
|
||||||
|
|
||||||
|
return alerts
|
||||||
|
|
||||||
|
|
||||||
def check_all() -> list[str]:
|
def check_all() -> list[str]:
|
||||||
"""Regelbasierter Check (Stufe 1). Gibt Liste von Alarmen zurück."""
|
"""Regelbasierter Check (Stufe 1). Gibt Liste von Alarmen zurück."""
|
||||||
cfg = config.parse_config()
|
cfg = config.parse_config()
|
||||||
suppressed_hosts = config.get_suppressed_hosts(cfg)
|
suppressed_hosts = config.get_suppressed_hosts(cfg)
|
||||||
suppressed_names = config.get_suppressed_container_names(cfg)
|
suppressed_names = config.get_suppressed_container_names(cfg)
|
||||||
alerts = []
|
alerts = []
|
||||||
|
alerts.extend(check_hermes())
|
||||||
|
|
||||||
containers = proxmox_client.get_all_containers(
|
containers = proxmox_client.get_all_containers(
|
||||||
_get_passwords(cfg), _get_tokens(cfg)
|
_get_passwords(cfg), _get_tokens(cfg)
|
||||||
|
|
@ -322,6 +384,19 @@ def _save_alert_state(state: dict):
|
||||||
|
|
||||||
|
|
||||||
def _alert_key(alert_text: str) -> str:
|
def _alert_key(alert_text: str) -> str:
|
||||||
|
# Volatile Metrik-Alarme (error_rate) ueber einen stabilen Schluessel
|
||||||
|
# deduplizieren: nur Host, nicht der schwankende Zaehlwert. Sonst
|
||||||
|
# erzeugt jede neue Zahl einen neuen Hash und der Cooldown greift nie.
|
||||||
|
if "Fehler in 30 Min" in alert_text:
|
||||||
|
m = re.search(r"([\w.\-]+):\s*\d+\s+Fehler in 30 Min", alert_text)
|
||||||
|
if m:
|
||||||
|
return hashlib.md5(("error_rate|" + m.group(1)).encode()).hexdigest()
|
||||||
|
if "Hermes:" in alert_text and "Fehlerlogs in Loki" in alert_text:
|
||||||
|
return hashlib.md5(b"hermes|loki_errors").hexdigest()
|
||||||
|
if alert_text.startswith("🔴 Hermes Log:"):
|
||||||
|
m = re.search(r"([\w.\-]+\.service)", alert_text)
|
||||||
|
unit = m.group(1) if m else "generic"
|
||||||
|
return hashlib.md5(("hermes|log|" + unit).encode()).hexdigest()
|
||||||
return hashlib.md5(alert_text.encode()).hexdigest()
|
return hashlib.md5(alert_text.encode()).hexdigest()
|
||||||
|
|
||||||
|
|
||||||
|
|
@ -340,6 +415,10 @@ def _alert_category(alert_text: str) -> str:
|
||||||
return "http"
|
return "http"
|
||||||
if "Service-Neustart" in alert_text:
|
if "Service-Neustart" in alert_text:
|
||||||
return "restart"
|
return "restart"
|
||||||
|
if "Hermes" in alert_text:
|
||||||
|
return "hermes"
|
||||||
|
if "Backup" in alert_text or "PBS" in alert_text:
|
||||||
|
return "backup"
|
||||||
if "Memory läuft ab" in alert_text:
|
if "Memory läuft ab" in alert_text:
|
||||||
return "memory_expiry"
|
return "memory_expiry"
|
||||||
return "default"
|
return "default"
|
||||||
|
|
|
||||||
|
|
@ -447,15 +447,11 @@ async def _run_freitext_llm_pipeline(
|
||||||
document_mode=document_mode,
|
document_mode=document_mode,
|
||||||
)
|
)
|
||||||
)
|
)
|
||||||
waited = 0
|
|
||||||
while not llm_task.done():
|
|
||||||
await asyncio.sleep(10)
|
|
||||||
waited += 10
|
|
||||||
if not llm_task.done():
|
|
||||||
try:
|
try:
|
||||||
await update.message.reply_text(
|
answer = await asyncio.wait_for(asyncio.shield(llm_task), timeout=15.0)
|
||||||
"⏳ Noch dran (" + str(waited) + "s) — Save.TV/Modell kann etwas brauchen…"
|
except asyncio.TimeoutError:
|
||||||
)
|
try:
|
||||||
|
await update.message.reply_text("⏳ Noch dran — dauert etwas länger…")
|
||||||
except Exception as te:
|
except Exception as te:
|
||||||
log.warning("Fortschritts-Nachricht fehlgeschlagen: %s", te)
|
log.warning("Fortschritts-Nachricht fehlgeschlagen: %s", te)
|
||||||
answer = await llm_task
|
answer = await llm_task
|
||||||
|
|
@ -506,15 +502,11 @@ async def _run_voice_llm_pipeline(
|
||||||
document_mode=document_mode,
|
document_mode=document_mode,
|
||||||
)
|
)
|
||||||
)
|
)
|
||||||
waited = 0
|
|
||||||
while not llm_task.done():
|
|
||||||
await asyncio.sleep(10)
|
|
||||||
waited += 10
|
|
||||||
if not llm_task.done():
|
|
||||||
try:
|
try:
|
||||||
await update.message.reply_text(
|
answer = await asyncio.wait_for(asyncio.shield(llm_task), timeout=15.0)
|
||||||
"⏳ Noch dran (" + str(waited) + "s) — Save.TV/Modell kann etwas brauchen…"
|
except asyncio.TimeoutError:
|
||||||
)
|
try:
|
||||||
|
await update.message.reply_text("⏳ Noch dran — dauert etwas länger…")
|
||||||
except Exception as te:
|
except Exception as te:
|
||||||
log.warning("Fortschritts-Nachricht fehlgeschlagen: %s", te)
|
log.warning("Fortschritts-Nachricht fehlgeschlagen: %s", te)
|
||||||
answer = await llm_task
|
answer = await llm_task
|
||||||
|
|
|
||||||
|
|
@ -48,3 +48,110 @@
|
||||||
|---|---|
|
|---|---|
|
||||||
| @MutterbotAI_bot | Watchdog-Alerts |
|
| @MutterbotAI_bot | Watchdog-Alerts |
|
||||||
| @Orbitalo_Hausmeister_bot | Homelab AI-Bot |
|
| @Orbitalo_Hausmeister_bot | Homelab AI-Bot |
|
||||||
|
|
||||||
|
## Hermes CT151 Monitoring und Backup
|
||||||
|
|
||||||
|
Stand: 2026-05-16
|
||||||
|
|
||||||
|
Hermes (`hermes-mu`, CT151) ist als kritischer Dienst eingestuft und wird separat abgesichert.
|
||||||
|
|
||||||
|
### Proxmox / PBS Backup
|
||||||
|
|
||||||
|
Backup-Job auf `pve3` / `pve-mu-3`:
|
||||||
|
|
||||||
|
| Feld | Wert |
|
||||||
|
|------|------|
|
||||||
|
| Job-ID | `hermes-mu-backup` |
|
||||||
|
| VMID | `151` |
|
||||||
|
| Container | `hermes-mu` |
|
||||||
|
| Node | `pve3` |
|
||||||
|
| Ziel | `pbs-nvme` / Muldenstein PBS (`100.99.139.22:nvme-pool`) |
|
||||||
|
| Modus | `snapshot` |
|
||||||
|
| Kompression | `zstd` |
|
||||||
|
| Zeitplan | täglich `03:20` |
|
||||||
|
| Nachholen | `repeat-missed=1` |
|
||||||
|
| Prune | `keep-daily=7`, `keep-weekly=4`, `keep-monthly=3` |
|
||||||
|
|
||||||
|
Manuelles Safety-Backup wurde am 2026-05-16 erfolgreich erstellt:
|
||||||
|
|
||||||
|
```text
|
||||||
|
ct/151/2026-05-16T10:51:53Z
|
||||||
|
Dauer: 56s
|
||||||
|
Gesichert: ca. 6.15 GiB
|
||||||
|
Komprimiert übertragen: ca. 3.98 GiB
|
||||||
|
Ergebnis: Backup job finished successfully
|
||||||
|
```
|
||||||
|
|
||||||
|
PBS-Auslastung danach:
|
||||||
|
|
||||||
|
```text
|
||||||
|
pbs-nvme active, ca. 5.63% belegt
|
||||||
|
```
|
||||||
|
|
||||||
|
Schnelltest:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Auf pve3 / pve-mu-3
|
||||||
|
pvesh get /cluster/backup/hermes-mu-backup --output-format json
|
||||||
|
pvesm status --storage pbs-nvme
|
||||||
|
```
|
||||||
|
|
||||||
|
### Prometheus / node_exporter
|
||||||
|
|
||||||
|
Auf CT151 ist `prometheus-node-exporter` installiert und aktiv.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
systemctl is-active prometheus-node-exporter
|
||||||
|
curl -sS http://127.0.0.1:9100/metrics | grep -m1 '^node_uname_info'
|
||||||
|
```
|
||||||
|
|
||||||
|
Von CT116 / Hausmeister erreichbar über:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl -sS http://100.109.174.120:9100/metrics
|
||||||
|
```
|
||||||
|
|
||||||
|
### Loki / Promtail
|
||||||
|
|
||||||
|
Auf CT151 läuft `promtail` und schickt Logs mit Host-Label `hermes-mu` an Loki.
|
||||||
|
|
||||||
|
Wichtige Checks:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
systemctl is-active promtail
|
||||||
|
curl -sS http://100.109.174.120:9080/ready
|
||||||
|
```
|
||||||
|
|
||||||
|
### Hermes-Service-Checks
|
||||||
|
|
||||||
|
Aktive Dienste auf CT151:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
systemctl is-active hermes-gateway beellama-tunnel promtail prometheus-node-exporter
|
||||||
|
curl -sS http://127.0.0.1:11435/v1/models
|
||||||
|
```
|
||||||
|
|
||||||
|
Der lokale BeeLlama/OpenAI-kompatible Tunnel lauscht in CT151 auf `127.0.0.1:11435`; er ist deshalb von CT116 nicht direkt per HTTP erreichbar. Der Health-Check erfolgt lokal auf CT151 bzw. indirekt über Hermes/Hausmeister.
|
||||||
|
|
||||||
|
### Hausmeister-Integration
|
||||||
|
|
||||||
|
Hausmeister (`CT116`) enthält in `/opt/homelab-ai-bot/monitor.py` einen `check_hermes()`.
|
||||||
|
|
||||||
|
Der Check prüft:
|
||||||
|
|
||||||
|
- Hermes Promtail: `http://100.109.174.120:9080/ready`
|
||||||
|
- Hermes node_exporter: `http://100.109.174.120:9100/metrics`
|
||||||
|
- Websuche/SearXNG CT121: `http://100.74.196.29:8080/search?...`
|
||||||
|
- Loki: `http://100.109.206.43:3100/ready`
|
||||||
|
- PBS Muldenstein: `https://100.99.139.22:8007`
|
||||||
|
- Loki-Health für Host `hermes-mu`
|
||||||
|
- auffällige Hermes-Logs der letzten 30 Minuten
|
||||||
|
|
||||||
|
Harmloser `node_exporter`-Diskstats-Logspam (`diskstats_linux.go`, `disabling udev device properties`) wird gefiltert.
|
||||||
|
|
||||||
|
Finaler Test am 2026-05-16:
|
||||||
|
|
||||||
|
```text
|
||||||
|
monitor.check_hermes() -> OK
|
||||||
|
hausmeister-bot.service -> active
|
||||||
|
```
|
||||||
|
|
|
||||||
|
|
@ -1093,3 +1093,38 @@ _(reasoning: Der Memory wurde erfolgreich gespeichert. Ich bestätige das dem Us
|
||||||
|
|
||||||
Willst du noch etwas anderes speichern? 🔥
|
Willst du noch etwas anderes speichern? 🔥
|
||||||
|
|
||||||
|
## Entscheidungsnotiz: Qwen3.6 + BeeLlama fuer opencode-Tests
|
||||||
|
|
||||||
|
Stand: 2026-05-18
|
||||||
|
|
||||||
|
Qwen3.6 ueber BeeLlama ist fuer einfache, kontrollierte `opencode`-Tests ausreichend:
|
||||||
|
|
||||||
|
- Git-Repos klonen
|
||||||
|
- README, `package.json` und Setup-Hinweise lesen
|
||||||
|
- Projektstruktur erkunden
|
||||||
|
- Shell-/Git-Kommandos ausfuehren
|
||||||
|
- Tests finden und starten
|
||||||
|
- kleine, klar begrenzte Codeaenderungen vorbereiten
|
||||||
|
|
||||||
|
Wichtig: BeeLlama macht das Modell schneller und lokal praktikabler, aber nicht automatisch intelligenter. Fuer komplexe Coding-Agent-Aufgaben wie grosse Refactorings, viele Dateien gleichzeitig, schwierige Debug-Schleifen oder produktive Aenderungen bleibt ein spezialisiertes Coding-Modell zuverlaessiger.
|
||||||
|
|
||||||
|
Empfohlener erster Testauftrag:
|
||||||
|
|
||||||
|
```text
|
||||||
|
Klon dieses Repo nach /tmp/opencode-test, lies README und package.json, aber aendere nichts. Sag danach, wie man es startet und welche Tests definiert sind.
|
||||||
|
```
|
||||||
|
|
||||||
|
## Entscheidungsnotiz: zweite RTX 3090
|
||||||
|
|
||||||
|
Eine zweite RTX 3090 verbessert die Qualitaet desselben Modells nicht direkt. Dasselbe Modell mit derselben Quantisierung liefert grundsaetzlich dieselbe Antwortqualitaet.
|
||||||
|
|
||||||
|
Eine zweite GPU hilft indirekt durch:
|
||||||
|
|
||||||
|
- groessere Modelle, die vorher nicht in den VRAM passten
|
||||||
|
- bessere Quantisierung, z.B. Q5/Q6 statt Q4
|
||||||
|
- mehr Kontext/KV-Cache
|
||||||
|
- weniger CPU-Offload
|
||||||
|
- hoehere Tokenrate bei gutem Layer-/Tensor-Splitting
|
||||||
|
- mehr parallele Requests
|
||||||
|
|
||||||
|
Fuer `opencode` waere der echte Qualitaetsgewinn daher nicht die zweite GPU selbst, sondern dass dadurch eventuell ein besseres oder groesseres Coding-Modell lokal betrieben werden kann.
|
||||||
|
|
|
||||||
Loading…
Add table
Reference in a new issue