fix: run local LLM as llama-server sidecar; fix red CI builds#2
Draft
Aman-Verma-28 wants to merge 1 commit into
Draft
fix: run local LLM as llama-server sidecar; fix red CI builds#2Aman-Verma-28 wants to merge 1 commit into
Aman-Verma-28 wants to merge 1 commit into
Conversation
The release build failed to link on Linux/Windows with duplicate ggml/gguf symbols: whisper.cpp and llama.cpp each statically bundle their own copy of ggml, so linking both into one binary collides (cargo check hid this — it doesn't link). Switch the local LLM from the in-process `llama-cpp-2` crate to a bundled `llama-server` sidecar: the app downloads the official prebuilt llama.cpp server for the platform on first use, runs it on 127.0.0.1, and talks to it over the OpenAI-compatible API (reusing llm_client). llama.cpp is no longer linked in, so the symbol conflict is gone. Also fixes the other red CI checks: - code quality: add the new task i18n keys to all 19 locales - test: history test schema now includes the tasks_json column Note: macOS release signing fails on missing Apple secrets — pre-existing, out of scope here. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Aman-Verma-28
marked this pull request as draft
June 25, 2026 15:41
Contributor
Author
|
Keeping this open as a reference only — not for merge. Main now builds on Linux + Windows-ARM with the in-process |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Before Submitting This PR
Please confirm you have done the following:
Human Written Description
TODO: author to fill in.
Related Issues/Discussions
Follow-up to #1.
What this fixes
After #1 merged, every release build (
main-build.yml) failed. Root cause: duplicateggml/ggufsymbols at link time —whisper.cpp(via transcribe-rs) andllama.cppeach statically bundle their own copy of ggml, so linking both into one binary collides on Linux/Windows. (macOS linked by luck; my earliercargo checknever linked, so it slipped through.)Fix: run the LLM out-of-process. Replaced the in-process
llama-cpp-2crate with a bundledllama-serversidecar:llama-serverfor the platform from the pinned llama.cpp release (b9789), extracts it to the app data dir, and runs it on127.0.0.1.LlmManagermanages that process (lazy start, idle shutdown) and talks to it over the OpenAI-compatible API, reusingllm_clientwith the existingresponse_format: json_schemasupport.Validated locally on macOS: the app links cleanly (was the failure), and the
llama-serverdownload + extract + exec works (version: 9789).Also fixes the other red checks from #1:
tasks_jsoncolumn.Testing
cargo build --bin sitreplinks cleanly (macOS);cargo test→ 69 passtsc/eslintclean;check:translationspasses 19/19llama-serverb9789 download/extract/run verified on macOS arm64llama-serverasset mapping, and end-to-end task generation against a downloaded model.Known follow-ups / out of scope
main-buildfailed there), to be fixed in repo settings.llama-serversilently with no progress UI yet.AI Assistance
If AI was used:
🤖 Generated with Claude Code