Skip to content

fix: run local LLM as llama-server sidecar; fix red CI builds#2

Draft
Aman-Verma-28 wants to merge 1 commit into
mainfrom
fix/llm-sidecar-and-ci
Draft

fix: run local LLM as llama-server sidecar; fix red CI builds#2
Aman-Verma-28 wants to merge 1 commit into
mainfrom
fix/llm-sidecar-and-ci

Conversation

@Aman-Verma-28

Copy link
Copy Markdown
Contributor

Before Submitting This PR

This is a bug-fix follow-up to #1 (the local-LLM/task feature): it makes the release builds and CI checks pass.

Please confirm you have done the following:

  • I have searched existing issues and pull requests (including closed ones) to ensure this isn't a duplicate
  • I have read CONTRIBUTING.md

Human Written Description

TODO: author to fill in.

Related Issues/Discussions

Follow-up to #1.

What this fixes

After #1 merged, every release build (main-build.yml) failed. Root cause: duplicate ggml/gguf symbols at link timewhisper.cpp (via transcribe-rs) and llama.cpp each statically bundle their own copy of ggml, so linking both into one binary collides on Linux/Windows. (macOS linked by luck; my earlier cargo check never linked, so it slipped through.)

Fix: run the LLM out-of-process. Replaced the in-process llama-cpp-2 crate with a bundled llama-server sidecar:

  • On first task generation the app downloads the official prebuilt llama-server for the platform from the pinned llama.cpp release (b9789), extracts it to the app data dir, and runs it on 127.0.0.1.
  • LlmManager manages that process (lazy start, idle shutdown) and talks to it over the OpenAI-compatible API, reusing llm_client with the existing response_format: json_schema support.
  • llama.cpp is no longer linked into the app binary → the ggml symbol conflict is gone.

Validated locally on macOS: the app links cleanly (was the failure), and the llama-server download + extract + exec works (version: 9789).

Also fixes the other red checks from #1:

  • code quality: added the new task i18n keys to all 19 locales (English placeholders; real translations welcome).
  • test: the in-memory history test schema now includes the tasks_json column.

Testing

  • cargo build --bin sitrep links cleanly (macOS); cargo test → 69 pass
  • ✅ CI test-mock path compiles without transcribe-rs; tsc/eslint clean; check:translations passes 19/19
  • llama-server b9789 download/extract/run verified on macOS arm64
  • ⏳ Not verified here: Linux/Windows release builds (should now link since llama.cpp isn't linked in — relying on this PR's CI), the per-platform llama-server asset mapping, and end-to-end task generation against a downloaded model.

Known follow-ups / out of scope

  • macOS release signing fails on missing Apple Developer Certificate secrets — pre-existing (every prior main-build failed there), to be fixed in repo settings.
  • First-run downloads llama-server silently with no progress UI yet.

AI Assistance

  • No AI was used in this PR
  • AI was used (please describe below)

If AI was used:

  • Tools used: Claude Code (Opus 4.8)
  • How extensively: Diagnosed the CI failures and implemented the sidecar rework. Human review needed for the description, cross-platform binary sourcing, and end-to-end testing.

🤖 Generated with Claude Code

The release build failed to link on Linux/Windows with duplicate ggml/gguf
symbols: whisper.cpp and llama.cpp each statically bundle their own copy of
ggml, so linking both into one binary collides (cargo check hid this — it
doesn't link).

Switch the local LLM from the in-process `llama-cpp-2` crate to a bundled
`llama-server` sidecar: the app downloads the official prebuilt llama.cpp
server for the platform on first use, runs it on 127.0.0.1, and talks to it
over the OpenAI-compatible API (reusing llm_client). llama.cpp is no longer
linked in, so the symbol conflict is gone.

Also fixes the other red CI checks:
- code quality: add the new task i18n keys to all 19 locales
- test: history test schema now includes the tasks_json column

Note: macOS release signing fails on missing Apple secrets — pre-existing,
out of scope here.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@Aman-Verma-28
Aman-Verma-28 marked this pull request as draft June 25, 2026 15:41
@Aman-Verma-28

Copy link
Copy Markdown
Contributor Author

Keeping this open as a reference only — not for merge. Main now builds on Linux + Windows-ARM with the in-process llama-cpp-2 (CPU-only) approach, which is the chosen direction. This PR documents an alternative llama-server sidecar architecture (removes in-process llama.cpp entirely, sidesteps the duplicate ggml/gguf link conflict) in case the in-process build ever causes runtime issues. It conflicts with main's Meeting-mode work and should not be merged as-is.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant