fix(bot): Live-Stand CT116 uebernehmen (Grok-4.3, Hermes-Monitoring, Loki-Filter, Antwort-Timeout)
- llm.py: Text-Modell auf x-ai/grok-4.3 - telegram_bot.py: Fortschritts-Logik auf asyncio.wait_for + shield (15s Timeout) - monitor.py: Hermes-Healthchecks (CT151), Alert-Dedup, EXPECTED_STOPPED erweitert - core/loki_client.py: zentrale ERROR_EXCLUDE_RE fuer transiente cloudflared/QUIC-Flaps - Doku: JELLYFIN.md, infrastructure/STATE.md, hardware-ai-local-coding.md, sessions/ Bis 21.06. uncommittete Live-Edits, gesichert in backup/live-state-20260621. Ref #123.
This commit is contained in:
Homelab Cursor
parent
169ea0c71f
commit
1cbc5df1ad
8 changed files with 362 additions and 34 deletions
|
|
@ -190,3 +190,39 @@ Neue Leitlinie:
|
|||
- **M5 Ultra** nur dann, wenn ein konkretes Modell das rechtfertigt.
|
||||
|
||||
`homelab.conf` und `MASTER_INDEX.md` werden parallel angepasst.
|
||||
|
||||
## Entscheidungsnotiz: Qwen3.6 + BeeLlama fuer opencode-Tests
|
||||
|
||||
Stand: 2026-05-18
|
||||
|
||||
Qwen3.6 ueber BeeLlama ist fuer einfache, kontrollierte `opencode`-Tests ausreichend:
|
||||
|
||||
- Git-Repos klonen
|
||||
- README, `package.json` und Setup-Hinweise lesen
|
||||
- Projektstruktur erkunden
|
||||
- Shell-/Git-Kommandos ausfuehren
|
||||
- Tests finden und starten
|
||||
- kleine, klar begrenzte Codeaenderungen vorbereiten
|
||||
|
||||
Wichtig: BeeLlama macht das Modell schneller und lokal praktikabler, aber nicht automatisch intelligenter. Fuer komplexe Coding-Agent-Aufgaben wie grosse Refactorings, viele Dateien gleichzeitig, schwierige Debug-Schleifen oder produktive Aenderungen bleibt ein spezialisiertes Coding-Modell zuverlaessiger.
|
||||
|
||||
Empfohlener erster Testauftrag:
|
||||
|
||||
```text
|
||||
Klon dieses Repo nach /tmp/opencode-test, lies README und package.json, aber aendere nichts. Sag danach, wie man es startet und welche Tests definiert sind.
|
||||
```
|
||||
|
||||
## Entscheidungsnotiz: zweite RTX 3090
|
||||
|
||||
Eine zweite RTX 3090 verbessert die Qualitaet desselben Modells nicht direkt. Dasselbe Modell mit derselben Quantisierung liefert grundsaetzlich dieselbe Antwortqualitaet.
|
||||
|
||||
Eine zweite GPU hilft indirekt durch:
|
||||
|
||||
- groessere Modelle, die vorher nicht in den VRAM passten
|
||||
- bessere Quantisierung, z.B. Q5/Q6 statt Q4
|
||||
- mehr Kontext/KV-Cache
|
||||
- weniger CPU-Offload
|
||||
- hoehere Tokenrate bei gutem Layer-/Tensor-Splitting
|
||||
- mehr parallele Requests
|
||||
|
||||
Fuer `opencode` waere der echte Qualitaetsgewinn daher nicht die zweite GPU selbst, sondern dass dadurch eventuell ein besseres oder groesseres Coding-Modell lokal betrieben werden kann.
|
||||
|
|
|
|||
|
|
@ -140,3 +140,58 @@ curl -s "http://100.77.105.3:8096/Items?api_key=7285b4a8793541648bf156599ae05b43
|
|||
| Ralf | nein |
|
||||
| rolf | nein |
|
||||
| wilfrid | nein |
|
||||
|
||||
## Troubleshooting: Details fehlen / Remote-Zugriff wirkt kaputt
|
||||
|
||||
Stand: 2026-05-16
|
||||
|
||||
Symptom:
|
||||
- Filmübersicht ist erreichbar, aber Detailseiten/Filmdetails werden im Client nicht angezeigt.
|
||||
- Zugriff über Tailscale-IP `100.77.105.3:8096` hängt oder läuft in Timeouts.
|
||||
|
||||
Befund vom 2026-05-16:
|
||||
- Jellyfin selbst war lokal gesund:
|
||||
- `http://192.168.178.149:8096/health` -> `Healthy`
|
||||
- API `/Items/Counts` lieferte `897` Filme.
|
||||
- Beispiel-Film `2 Guns` hatte Beschreibung, Genres, TMDB/IMDB IDs und Bilder.
|
||||
- Medien-Mounts waren aktiv:
|
||||
- `/mnt/nas`
|
||||
- `/srv/media`
|
||||
- Docker-Container `jellyfin` war `healthy`.
|
||||
- Ursache war Tailscale auf dem Jellyfin-Host:
|
||||
- `tailscale ip -4` meldete zuerst `no current Tailscale IPs; state: NeedsLogin`
|
||||
- `tailscale status` meldete `Logged out`.
|
||||
- Nach Re-Login war `100.77.105.3` wieder aktiv und `http://100.77.105.3:8096/health` antwortete wieder `Healthy`.
|
||||
- Danach waren die Filmdetails im Client wieder sichtbar.
|
||||
|
||||
Schnelltest:
|
||||
```bash
|
||||
# Von pve3 lokal im LAN:
|
||||
curl -sS http://192.168.178.149:8096/health
|
||||
|
||||
# Von CT 116 / Hausmeister oder einem Tailscale-Host:
|
||||
curl -sS http://100.77.105.3:8096/health
|
||||
curl -sS "http://100.77.105.3:8096/Items/Counts?api_key=7285b4a8793541648bf156599ae05b43"
|
||||
```
|
||||
|
||||
Fix:
|
||||
1. Auf dem Jellyfin-Host Tailscale-Status prüfen:
|
||||
```bash
|
||||
tailscale ip -4
|
||||
tailscale status --self
|
||||
```
|
||||
2. Wenn `NeedsLogin` oder `Logged out`: Tailscale neu anmelden.
|
||||
3. Danach erneut prüfen:
|
||||
```bash
|
||||
curl -sS http://100.77.105.3:8096/health
|
||||
```
|
||||
4. Im Browser Jellyfin hart neu laden oder neu einloggen, falls der Webclient alte kaputte API-Antworten gecacht hat.
|
||||
|
||||
Wichtig:
|
||||
- Fehlende Details bedeuten nicht automatisch kaputte Jellyfin-Datenbank oder fehlende Metadaten.
|
||||
- Erst lokale API gegen `192.168.178.149:8096` prüfen, dann Tailscale/Cloudflare-Zugriff.
|
||||
- `systemctl is-active jellyfin` kann irreführend sein, weil Jellyfin hier im Docker-Container läuft. Aussagekräftiger ist:
|
||||
```bash
|
||||
docker ps | grep jellyfin
|
||||
curl -sS http://127.0.0.1:8096/health
|
||||
```
|
||||
|
|
|
|||
|
|
@ -5,6 +5,28 @@ from datetime import datetime, timezone, timedelta
|
|||
|
||||
LOKI_URL = "http://100.109.206.43:3100"
|
||||
|
||||
# Transiente cloudflared/Cloudflare-Edge-Flaps erzeugen oft error="..."
|
||||
# in WRN/ERR-Zeilen, sind aber keine App-Fehler von WordPress/RSS.
|
||||
ERROR_EXCLUDE_RE = (
|
||||
"caller=metrics|query_hash=|executing query|scheduler_processor|"
|
||||
"Aborted connection|systemd-networkd-wait-online|context canceled|"
|
||||
"AH01630: client denied|flag evaluation succeeded|pluginsAutoUpdate|"
|
||||
"control stream encountered a failure while serving|Serve tunnel error|"
|
||||
"failed to serve tunnel connection|Connection terminated|"
|
||||
"failed to dial to edge with quic|Failed to dial a quic connection|"
|
||||
"no recent network activity|failed to accept QUIC stream|"
|
||||
"control stream|accept stream listener|serve tunnel|tunnel connection|"
|
||||
"edge with quic|quic connection|QUIC stream|"
|
||||
"accept stream listener encountered a failure while serving|"
|
||||
"datagram manager encountered a failure while serving|"
|
||||
"datagram manager|datagram handler|"
|
||||
"failed to run the datagram handler|failed to accept incoming stream requests|"
|
||||
"Application error 0x0|connIndex=|"
|
||||
"beellama-tunnel.service: Main process exited|"
|
||||
"beellama-expose.service: Main process exited|"
|
||||
"health\(warnable=mapresponse-timeout\)"
|
||||
)
|
||||
|
||||
|
||||
def _query(endpoint: str, params: dict, base_url: str = None) -> dict:
|
||||
url = f"{base_url or LOKI_URL}{endpoint}"
|
||||
|
|
@ -49,9 +71,9 @@ def query_logs(query: str, hours: float = 1, limit: int = 100) -> list[dict]:
|
|||
def get_errors(container: str = None, hours: float = 1, limit: int = 200) -> list[dict]:
|
||||
"""Get error-level logs, optionally filtered by container hostname."""
|
||||
if container:
|
||||
q = f'{{host="{container}"}} |~ "(?i)(error|fatal|panic|traceback|exception)" !~ "caller=metrics|query_hash=|executing query|scheduler_processor|Aborted connection|systemd-networkd-wait-online|context canceled|AH01630: client denied|flag evaluation succeeded|pluginsAutoUpdate|terror"'
|
||||
q = f'{{host="{container}"}} |~ "(?i)(error|fatal|panic|traceback|exception)" !~ "{ERROR_EXCLUDE_RE}"'
|
||||
else:
|
||||
q = '{job=~".+"} |~ "(?i)(error|fatal|panic|traceback|exception)" !~ "caller=metrics|query_hash=|executing query|scheduler_processor|Aborted connection|systemd-networkd-wait-online|context canceled|AH01630: client denied|flag evaluation succeeded|pluginsAutoUpdate|terror"'
|
||||
q = f'{{job=~".+"}} |~ "(?i)(error|fatal|panic|traceback|exception)" !~ "{ERROR_EXCLUDE_RE}"'
|
||||
return query_logs(q, hours=hours, limit=limit)
|
||||
|
||||
|
||||
|
|
@ -102,7 +124,8 @@ def get_health(container: str, hours: float = 24) -> dict:
|
|||
|
||||
return {
|
||||
"host": container,
|
||||
"errors_last_{hours}h": error_count,
|
||||
"error_count": error_count,
|
||||
f"errors_last_{hours}h": error_count,
|
||||
"sending_logs": has_recent,
|
||||
"status": "healthy" if error_count < 5 and has_recent else
|
||||
"warning" if error_count < 20 else "critical",
|
||||
|
|
@ -114,6 +137,7 @@ WATCHED_SERVICES = [
|
|||
("wordpress-v2", "wordpress"),
|
||||
("fuenfvoracht", "fuenfvoracht"),
|
||||
("homelab-ai-bot", "hausmeister"),
|
||||
("hermes-mu", "hermes-gateway"),
|
||||
]
|
||||
|
||||
|
||||
|
|
@ -121,7 +145,7 @@ def count_errors(hours: float = 24) -> dict:
|
|||
"""Zählt Fehler-Log-Einträge über einen Zeitraum via Loki metric query."""
|
||||
now = datetime.now(timezone.utc)
|
||||
start = now - timedelta(hours=hours)
|
||||
q = '{job=~".+"} |~ "(?i)(error|fatal|panic|traceback|exception)" !~ "caller=metrics|query_hash=|executing query|scheduler_processor|Aborted connection|systemd-networkd-wait-online|context canceled|AH01630: client denied|flag evaluation succeeded|pluginsAutoUpdate|terror"'
|
||||
q = f'{{job=~".+"}} |~ "(?i)(error|fatal|panic|traceback|exception)" !~ "{ERROR_EXCLUDE_RE}"'
|
||||
# Loki instant metric query für Gesamtanzahl
|
||||
data = _query("/loki/api/v1/query_range", {
|
||||
"query": q,
|
||||
|
|
@ -160,7 +184,7 @@ def check_service_restarts(minutes: int = 35) -> list[dict]:
|
|||
ERROR_RATE_THRESHOLDS = {
|
||||
"rss-manager": 15,
|
||||
"wordpress-v2": 10,
|
||||
"forgejo": 200, # Oeffentlich — Scanner-404er sind normal
|
||||
"forgejo": 200, # Oeffentlich – Scanner-404er sind normal
|
||||
}
|
||||
ERROR_RATE_DEFAULT = 25
|
||||
|
||||
|
|
@ -171,7 +195,7 @@ def check_error_rate(minutes: int = 30) -> list[dict]:
|
|||
alerts = []
|
||||
now = datetime.now(timezone.utc)
|
||||
for host in all_hosts:
|
||||
q = f'count_over_time({{host="{host}"}} |~ "(?i)error" !~ "caller=metrics|query_hash=|executing query|scheduler_processor|Aborted connection|systemd-networkd-wait-online|context canceled|AH01630: client denied|flag evaluation succeeded|pluginsAutoUpdate|terror" [{minutes}m])'
|
||||
q = f'count_over_time({{host="{host}"}} |~ "(?i)error" !~ "{ERROR_EXCLUDE_RE}" [{minutes}m])'
|
||||
data = _query("/loki/api/v1/query", {"query": q, "time": _ns(now)})
|
||||
count = sum(
|
||||
int(float(r.get("value", [None, "0"])[1]))
|
||||
|
|
|
|||
|
|
@ -27,8 +27,8 @@ OLLAMA_MODELS = set()
|
|||
|
||||
|
||||
def warmup_ollama():
|
||||
"""No-Op: Text-Modell laeuft jetzt ueber OpenRouter (Grok 4.1 Fast), kein Ollama-Warmup noetig."""
|
||||
log.info('Ollama warmup uebersprungen — Text laeuft ueber OpenRouter (Grok 4.1 Fast)')
|
||||
"""No-Op: Text-Modell laeuft ueber OpenRouter (x-ai/grok-4.3), kein Ollama-Warmup noetig."""
|
||||
log.info('Ollama warmup uebersprungen — Text laeuft ueber OpenRouter (x-ai/grok-4.3)')
|
||||
PASSTHROUGH_TOOLS = {"get_temperaturen", "get_energie", "get_heizung"}
|
||||
|
||||
_LOCAL_OVERRIDES = [
|
||||
|
|
|
|||
|
|
@ -4,6 +4,7 @@ import sys
|
|||
import os
|
||||
import json
|
||||
import hashlib
|
||||
import re
|
||||
import requests
|
||||
import time
|
||||
from datetime import datetime, timezone
|
||||
|
|
@ -22,6 +23,8 @@ ALERT_COOLDOWN_SECONDS = {
|
|||
"memory_expiry": 43200,
|
||||
"default": 3600,
|
||||
"error_rate": 1800,
|
||||
"hermes": 7200,
|
||||
"backup": 21600,
|
||||
}
|
||||
|
||||
|
||||
|
|
@ -45,10 +48,18 @@ HTTP_HEALTH_CHECKS = [
|
|||
{"name": "WordPress (CT 101)", "url": "http://10.10.10.101/robots.txt"},
|
||||
{"name": "Matomo (CT 109)", "url": "http://10.10.10.109"},
|
||||
{"name": "Grafana (CT 110)", "url": "http://10.10.10.110:3000/api/health"},
|
||||
{"name": "Flugscanner-Agent (pve-pp-1)", "url": "http://100.126.26.46:5010/status",
|
||||
"retries": 5, "timeout": 25, "retry_delay": 6, "host": "pve-pp-1"},
|
||||
]
|
||||
|
||||
HERMES_HEALTH_CHECKS = [
|
||||
{"name": "Hermes Promtail", "url": "http://100.109.174.120:9080/ready", "timeout": 5},
|
||||
{"name": "Hermes node_exporter", "url": "http://100.109.174.120:9100/metrics", "timeout": 5},
|
||||
{"name": "Hermes Websuche CT121", "url": "http://100.74.196.29:8080/search?q=hausmeister-health&format=json", "timeout": 8},
|
||||
{"name": "Loki", "url": "http://100.109.206.43:3100/ready", "timeout": 5},
|
||||
{"name": "PBS Muldenstein", "url": "https://100.99.139.22:8007", "timeout": 5, "allow_status": {200, 401}},
|
||||
]
|
||||
|
||||
HERMES_LOG_HOST = "hermes-mu"
|
||||
|
||||
EXPECTED_STOPPED = {
|
||||
(115, "pve-ka-1"), # flugscanner-asia-old (gestoppt, Cluster pve1) — Live CT auf pve-pp-1
|
||||
(115, "pve-ka-3"), # dieselbe CT, zweite API-Sicht (Cluster)
|
||||
|
|
@ -57,6 +68,9 @@ EXPECTED_STOPPED = {
|
|||
(504, "pve-ka-2"), # Shop-Template — stopped
|
||||
(8000, "pve-ka-2"), # Kunde0-Shop — stopped
|
||||
(8010, "pve-ka-2"), # Kunde1-Shop — stopped
|
||||
(121, "pve-ka-1"), # Schawarma-Shop — gestoppt
|
||||
(121, "pve-ka-2"), # Schawarma-Shop — gestoppt
|
||||
(121, "pve-ka-3"), # Schawarma-Shop — gestoppt
|
||||
}
|
||||
|
||||
# VMIDs, die auf jedem Proxmox-Host in CONFIG ok sind, solange status == stopped
|
||||
|
|
@ -83,12 +97,60 @@ def _is_host_suppressed(host: str, suppressed_hosts: set) -> bool:
|
|||
return False
|
||||
|
||||
|
||||
def check_hermes() -> list[str]:
|
||||
"""Prueft Hermes (CT151) als kritischen Dienst ueber HTTP und Loki."""
|
||||
alerts = []
|
||||
headers = {"User-Agent": "Mozilla/5.0 (Hausmeister-Bot/1.0 hermes-check)"}
|
||||
for check in HERMES_HEALTH_CHECKS:
|
||||
allow_status = check.get("allow_status") or set(range(200, 400))
|
||||
try:
|
||||
r = requests.get(
|
||||
check["url"],
|
||||
timeout=check.get("timeout", 5),
|
||||
allow_redirects=False,
|
||||
headers=headers,
|
||||
verify=False,
|
||||
)
|
||||
if r.status_code not in allow_status:
|
||||
alerts.append(f"🔴 Hermes: {check['name']} HTTP {r.status_code}")
|
||||
except requests.RequestException as e:
|
||||
alerts.append(f"🔴 Hermes: {check['name']} nicht erreichbar: {str(e)[:80]}")
|
||||
|
||||
health = loki_client.get_health(HERMES_LOG_HOST, hours=1)
|
||||
if health.get("status") == "silent":
|
||||
alerts.append("⚠️ Hermes: keine aktuellen Loki-Logs von hermes-mu")
|
||||
elif health.get("status") == "critical":
|
||||
cnt = health.get("error_count", health.get("errors_last_1h", "?"))
|
||||
alerts.append(f"🔴 Hermes: {cnt} Fehlerlogs in Loki (1h)")
|
||||
|
||||
errors = loki_client.get_errors(container=HERMES_LOG_HOST, hours=0.5, limit=20)
|
||||
for e in errors:
|
||||
if "error" in e:
|
||||
continue
|
||||
line = (e.get("line") or "").lower()
|
||||
if "diskstats_linux.go" in line and "disabling udev device properties" in line:
|
||||
continue
|
||||
if "collector failed" in line and "node_exporter" in line:
|
||||
continue
|
||||
if "beellama-tunnel.service" in line or "beellama-expose.service" in line:
|
||||
if "main process exited" in line:
|
||||
continue
|
||||
if "health(warnable=mapresponse-timeout)" in line:
|
||||
continue
|
||||
if any(term in line for term in ["traceback", "exception", "failed", "timeout", "tool_call", "connection refused"]):
|
||||
alerts.append(f"🔴 Hermes Log: {(e.get('line') or '')[:140]}")
|
||||
break
|
||||
|
||||
return alerts
|
||||
|
||||
|
||||
def check_all() -> list[str]:
|
||||
"""Regelbasierter Check (Stufe 1). Gibt Liste von Alarmen zurück."""
|
||||
cfg = config.parse_config()
|
||||
suppressed_hosts = config.get_suppressed_hosts(cfg)
|
||||
suppressed_names = config.get_suppressed_container_names(cfg)
|
||||
alerts = []
|
||||
alerts.extend(check_hermes())
|
||||
|
||||
containers = proxmox_client.get_all_containers(
|
||||
_get_passwords(cfg), _get_tokens(cfg)
|
||||
|
|
@ -322,6 +384,19 @@ def _save_alert_state(state: dict):
|
|||
|
||||
|
||||
def _alert_key(alert_text: str) -> str:
|
||||
# Volatile Metrik-Alarme (error_rate) ueber einen stabilen Schluessel
|
||||
# deduplizieren: nur Host, nicht der schwankende Zaehlwert. Sonst
|
||||
# erzeugt jede neue Zahl einen neuen Hash und der Cooldown greift nie.
|
||||
if "Fehler in 30 Min" in alert_text:
|
||||
m = re.search(r"([\w.\-]+):\s*\d+\s+Fehler in 30 Min", alert_text)
|
||||
if m:
|
||||
return hashlib.md5(("error_rate|" + m.group(1)).encode()).hexdigest()
|
||||
if "Hermes:" in alert_text and "Fehlerlogs in Loki" in alert_text:
|
||||
return hashlib.md5(b"hermes|loki_errors").hexdigest()
|
||||
if alert_text.startswith("🔴 Hermes Log:"):
|
||||
m = re.search(r"([\w.\-]+\.service)", alert_text)
|
||||
unit = m.group(1) if m else "generic"
|
||||
return hashlib.md5(("hermes|log|" + unit).encode()).hexdigest()
|
||||
return hashlib.md5(alert_text.encode()).hexdigest()
|
||||
|
||||
|
||||
|
|
@ -340,6 +415,10 @@ def _alert_category(alert_text: str) -> str:
|
|||
return "http"
|
||||
if "Service-Neustart" in alert_text:
|
||||
return "restart"
|
||||
if "Hermes" in alert_text:
|
||||
return "hermes"
|
||||
if "Backup" in alert_text or "PBS" in alert_text:
|
||||
return "backup"
|
||||
if "Memory läuft ab" in alert_text:
|
||||
return "memory_expiry"
|
||||
return "default"
|
||||
|
|
|
|||
|
|
@ -447,18 +447,14 @@ async def _run_freitext_llm_pipeline(
|
|||
document_mode=document_mode,
|
||||
)
|
||||
)
|
||||
waited = 0
|
||||
while not llm_task.done():
|
||||
await asyncio.sleep(10)
|
||||
waited += 10
|
||||
if not llm_task.done():
|
||||
try:
|
||||
await update.message.reply_text(
|
||||
"⏳ Noch dran (" + str(waited) + "s) — Save.TV/Modell kann etwas brauchen…"
|
||||
)
|
||||
except Exception as te:
|
||||
log.warning("Fortschritts-Nachricht fehlgeschlagen: %s", te)
|
||||
answer = await llm_task
|
||||
try:
|
||||
answer = await asyncio.wait_for(asyncio.shield(llm_task), timeout=15.0)
|
||||
except asyncio.TimeoutError:
|
||||
try:
|
||||
await update.message.reply_text("⏳ Noch dran — dauert etwas länger…")
|
||||
except Exception as te:
|
||||
log.warning("Fortschritts-Nachricht fehlgeschlagen: %s", te)
|
||||
answer = await llm_task
|
||||
|
||||
if session_id:
|
||||
memory_client.log_message(session_id, "user", text)
|
||||
|
|
@ -506,18 +502,14 @@ async def _run_voice_llm_pipeline(
|
|||
document_mode=document_mode,
|
||||
)
|
||||
)
|
||||
waited = 0
|
||||
while not llm_task.done():
|
||||
await asyncio.sleep(10)
|
||||
waited += 10
|
||||
if not llm_task.done():
|
||||
try:
|
||||
await update.message.reply_text(
|
||||
"⏳ Noch dran (" + str(waited) + "s) — Save.TV/Modell kann etwas brauchen…"
|
||||
)
|
||||
except Exception as te:
|
||||
log.warning("Fortschritts-Nachricht fehlgeschlagen: %s", te)
|
||||
answer = await llm_task
|
||||
try:
|
||||
answer = await asyncio.wait_for(asyncio.shield(llm_task), timeout=15.0)
|
||||
except asyncio.TimeoutError:
|
||||
try:
|
||||
await update.message.reply_text("⏳ Noch dran — dauert etwas länger…")
|
||||
except Exception as te:
|
||||
log.warning("Fortschritts-Nachricht fehlgeschlagen: %s", te)
|
||||
answer = await llm_task
|
||||
|
||||
if session_id:
|
||||
memory_client.log_message(session_id, "user", text)
|
||||
|
|
|
|||
|
|
@ -48,3 +48,110 @@
|
|||
|---|---|
|
||||
| @MutterbotAI_bot | Watchdog-Alerts |
|
||||
| @Orbitalo_Hausmeister_bot | Homelab AI-Bot |
|
||||
|
||||
## Hermes CT151 Monitoring und Backup
|
||||
|
||||
Stand: 2026-05-16
|
||||
|
||||
Hermes (`hermes-mu`, CT151) ist als kritischer Dienst eingestuft und wird separat abgesichert.
|
||||
|
||||
### Proxmox / PBS Backup
|
||||
|
||||
Backup-Job auf `pve3` / `pve-mu-3`:
|
||||
|
||||
| Feld | Wert |
|
||||
|------|------|
|
||||
| Job-ID | `hermes-mu-backup` |
|
||||
| VMID | `151` |
|
||||
| Container | `hermes-mu` |
|
||||
| Node | `pve3` |
|
||||
| Ziel | `pbs-nvme` / Muldenstein PBS (`100.99.139.22:nvme-pool`) |
|
||||
| Modus | `snapshot` |
|
||||
| Kompression | `zstd` |
|
||||
| Zeitplan | täglich `03:20` |
|
||||
| Nachholen | `repeat-missed=1` |
|
||||
| Prune | `keep-daily=7`, `keep-weekly=4`, `keep-monthly=3` |
|
||||
|
||||
Manuelles Safety-Backup wurde am 2026-05-16 erfolgreich erstellt:
|
||||
|
||||
```text
|
||||
ct/151/2026-05-16T10:51:53Z
|
||||
Dauer: 56s
|
||||
Gesichert: ca. 6.15 GiB
|
||||
Komprimiert übertragen: ca. 3.98 GiB
|
||||
Ergebnis: Backup job finished successfully
|
||||
```
|
||||
|
||||
PBS-Auslastung danach:
|
||||
|
||||
```text
|
||||
pbs-nvme active, ca. 5.63% belegt
|
||||
```
|
||||
|
||||
Schnelltest:
|
||||
|
||||
```bash
|
||||
# Auf pve3 / pve-mu-3
|
||||
pvesh get /cluster/backup/hermes-mu-backup --output-format json
|
||||
pvesm status --storage pbs-nvme
|
||||
```
|
||||
|
||||
### Prometheus / node_exporter
|
||||
|
||||
Auf CT151 ist `prometheus-node-exporter` installiert und aktiv.
|
||||
|
||||
```bash
|
||||
systemctl is-active prometheus-node-exporter
|
||||
curl -sS http://127.0.0.1:9100/metrics | grep -m1 '^node_uname_info'
|
||||
```
|
||||
|
||||
Von CT116 / Hausmeister erreichbar über:
|
||||
|
||||
```bash
|
||||
curl -sS http://100.109.174.120:9100/metrics
|
||||
```
|
||||
|
||||
### Loki / Promtail
|
||||
|
||||
Auf CT151 läuft `promtail` und schickt Logs mit Host-Label `hermes-mu` an Loki.
|
||||
|
||||
Wichtige Checks:
|
||||
|
||||
```bash
|
||||
systemctl is-active promtail
|
||||
curl -sS http://100.109.174.120:9080/ready
|
||||
```
|
||||
|
||||
### Hermes-Service-Checks
|
||||
|
||||
Aktive Dienste auf CT151:
|
||||
|
||||
```bash
|
||||
systemctl is-active hermes-gateway beellama-tunnel promtail prometheus-node-exporter
|
||||
curl -sS http://127.0.0.1:11435/v1/models
|
||||
```
|
||||
|
||||
Der lokale BeeLlama/OpenAI-kompatible Tunnel lauscht in CT151 auf `127.0.0.1:11435`; er ist deshalb von CT116 nicht direkt per HTTP erreichbar. Der Health-Check erfolgt lokal auf CT151 bzw. indirekt über Hermes/Hausmeister.
|
||||
|
||||
### Hausmeister-Integration
|
||||
|
||||
Hausmeister (`CT116`) enthält in `/opt/homelab-ai-bot/monitor.py` einen `check_hermes()`.
|
||||
|
||||
Der Check prüft:
|
||||
|
||||
- Hermes Promtail: `http://100.109.174.120:9080/ready`
|
||||
- Hermes node_exporter: `http://100.109.174.120:9100/metrics`
|
||||
- Websuche/SearXNG CT121: `http://100.74.196.29:8080/search?...`
|
||||
- Loki: `http://100.109.206.43:3100/ready`
|
||||
- PBS Muldenstein: `https://100.99.139.22:8007`
|
||||
- Loki-Health für Host `hermes-mu`
|
||||
- auffällige Hermes-Logs der letzten 30 Minuten
|
||||
|
||||
Harmloser `node_exporter`-Diskstats-Logspam (`diskstats_linux.go`, `disabling udev device properties`) wird gefiltert.
|
||||
|
||||
Finaler Test am 2026-05-16:
|
||||
|
||||
```text
|
||||
monitor.check_hermes() -> OK
|
||||
hausmeister-bot.service -> active
|
||||
```
|
||||
|
|
|
|||
|
|
@ -1093,3 +1093,38 @@ _(reasoning: Der Memory wurde erfolgreich gespeichert. Ich bestätige das dem Us
|
|||
|
||||
Willst du noch etwas anderes speichern? 🔥
|
||||
|
||||
## Entscheidungsnotiz: Qwen3.6 + BeeLlama fuer opencode-Tests
|
||||
|
||||
Stand: 2026-05-18
|
||||
|
||||
Qwen3.6 ueber BeeLlama ist fuer einfache, kontrollierte `opencode`-Tests ausreichend:
|
||||
|
||||
- Git-Repos klonen
|
||||
- README, `package.json` und Setup-Hinweise lesen
|
||||
- Projektstruktur erkunden
|
||||
- Shell-/Git-Kommandos ausfuehren
|
||||
- Tests finden und starten
|
||||
- kleine, klar begrenzte Codeaenderungen vorbereiten
|
||||
|
||||
Wichtig: BeeLlama macht das Modell schneller und lokal praktikabler, aber nicht automatisch intelligenter. Fuer komplexe Coding-Agent-Aufgaben wie grosse Refactorings, viele Dateien gleichzeitig, schwierige Debug-Schleifen oder produktive Aenderungen bleibt ein spezialisiertes Coding-Modell zuverlaessiger.
|
||||
|
||||
Empfohlener erster Testauftrag:
|
||||
|
||||
```text
|
||||
Klon dieses Repo nach /tmp/opencode-test, lies README und package.json, aber aendere nichts. Sag danach, wie man es startet und welche Tests definiert sind.
|
||||
```
|
||||
|
||||
## Entscheidungsnotiz: zweite RTX 3090
|
||||
|
||||
Eine zweite RTX 3090 verbessert die Qualitaet desselben Modells nicht direkt. Dasselbe Modell mit derselben Quantisierung liefert grundsaetzlich dieselbe Antwortqualitaet.
|
||||
|
||||
Eine zweite GPU hilft indirekt durch:
|
||||
|
||||
- groessere Modelle, die vorher nicht in den VRAM passten
|
||||
- bessere Quantisierung, z.B. Q5/Q6 statt Q4
|
||||
- mehr Kontext/KV-Cache
|
||||
- weniger CPU-Offload
|
||||
- hoehere Tokenrate bei gutem Layer-/Tensor-Splitting
|
||||
- mehr parallele Requests
|
||||
|
||||
Fuer `opencode` waere der echte Qualitaetsgewinn daher nicht die zweite GPU selbst, sondern dass dadurch eventuell ein besseres oder groesseres Coding-Modell lokal betrieben werden kann.
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue