GitHub - nikitakarpei/my-yacy-node: A lightweight, composable reimplementation of a YaCy peer as small Go services

Source: https://github.com/nikitakarpei/my-yacy-node

Latest commit

!Image 5: nikitakarpei!Image 6: claude

nikitakarpei

and

claude

Make crawler's frontier durable (#47 )

Open commit details success

Aug 28, 2026

3152e30·Aug 28, 2026

History

483 Commits

Open commit details

[](https://github.com/nikitakarpei/my-yacy-node/commits/main/)483 Commits

Folders and files

| Name | Name | Last commit message | Last commit date | | --- | --- | --- | --- |

| [.github/workflows](https://github.com/nikitakarpei/my-yacy-node/tree/main/.github/workflows "This path skips through empty directories") | [.github/workflows](https://github.com/nikitakarpei/my-yacy-node/tree/main/.github/workflows "This path skips through empty directories") | [Let users to supply index from web archives (](https://github.com/nikitakarpei/my-yacy-node/commit/d7c794deb66e6ff6741b79d4519a4c645c89b5fe "Let users to supply index from web archives (#43) Closes #41 Closes #42 --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>")#43[)](https://github.com/nikitakarpei/my-yacy-node/commit/d7c794deb66e6ff6741b79d4519a4c645c89b5fe "Let users to supply index from web archives (#43) Closes #41 Closes #42 --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>") | Aug 27, 2026 |

| [doc](https://github.com/nikitakarpei/my-yacy-node/tree/main/doc "doc") | [doc](https://github.com/nikitakarpei/my-yacy-node/tree/main/doc "doc") | [Make crawler's frontier durable (](https://github.com/nikitakarpei/my-yacy-node/commit/3152e30bd4fcbecca37c61e0ce3639607bfa9df9 "Make crawler's frontier durable (#47) Closes #44 Big refactor which moves frontier's state in NATS. Before it was stored in memory and that forced to limit on amount of URLs in frontier. The bigger was frontier the bigger was cost of crash mid-crawling. --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>")#47[)](https://github.com/nikitakarpei/my-yacy-node/commit/3152e30bd4fcbecca37c61e0ce3639607bfa9df9 "Make crawler's frontier durable (#47) Closes #44 Big refactor which moves frontier's state in NATS. Before it was stored in memory and that forced to limit on amount of URLs in frontier. The bigger was frontier the bigger was cost of crash mid-crawling. --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>") | Aug 28, 2026 |

| [libraries](https://github.com/nikitakarpei/my-yacy-node/tree/main/libraries "libraries") | [libraries](https://github.com/nikitakarpei/my-yacy-node/tree/main/libraries "libraries") | [Make crawler's frontier durable (](https://github.com/nikitakarpei/my-yacy-node/commit/3152e30bd4fcbecca37c61e0ce3639607bfa9df9 "Make crawler's frontier durable (#47) Closes #44 Big refactor which moves frontier's state in NATS. Before it was stored in memory and that forced to limit on amount of URLs in frontier. The bigger was frontier the bigger was cost of crash mid-crawling. --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>")#47[)](https://github.com/nikitakarpei/my-yacy-node/commit/3152e30bd4fcbecca37c61e0ce3639607bfa9df9 "Make crawler's frontier durable (#47) Closes #44 Big refactor which moves frontier's state in NATS. Before it was stored in memory and that forced to limit on amount of URLs in frontier. The bigger was frontier the bigger was cost of crash mid-crawling. --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>") | Aug 28, 2026 |

| [plugins/searxng](https://github.com/nikitakarpei/my-yacy-node/tree/main/plugins/searxng "This path skips through empty directories") | [plugins/searxng](https://github.com/nikitakarpei/my-yacy-node/tree/main/plugins/searxng "This path skips through empty directories") | [Let users to supply index from web archives (](https://github.com/nikitakarpei/my-yacy-node/commit/d7c794deb66e6ff6741b79d4519a4c645c89b5fe "Let users to supply index from web archives (#43) Closes #41 Closes #42 --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>")#43[)](https://github.com/nikitakarpei/my-yacy-node/commit/d7c794deb66e6ff6741b79d4519a4c645c89b5fe "Let users to supply index from web archives (#43) Closes #41 Closes #42 --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>") | Aug 27, 2026 |

| [services](https://github.com/nikitakarpei/my-yacy-node/tree/main/services "services") | [services](https://github.com/nikitakarpei/my-yacy-node/tree/main/services "services") | [Make crawler's frontier durable (](https://github.com/nikitakarpei/my-yacy-node/commit/3152e30bd4fcbecca37c61e0ce3639607bfa9df9 "Make crawler's frontier durable (#47) Closes #44 Big refactor which moves frontier's state in NATS. Before it was stored in memory and that forced to limit on amount of URLs in frontier. The bigger was frontier the bigger was cost of crash mid-crawling. --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>")#47[)](https://github.com/nikitakarpei/my-yacy-node/commit/3152e30bd4fcbecca37c61e0ce3639607bfa9df9 "Make crawler's frontier durable (#47) Closes #44 Big refactor which moves frontier's state in NATS. Before it was stored in memory and that forced to limit on amount of URLs in frontier. The bigger was frontier the bigger was cost of crash mid-crawling. --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>") | Aug 28, 2026 |

| [test](https://github.com/nikitakarpei/my-yacy-node/tree/main/test "test") | [test](https://github.com/nikitakarpei/my-yacy-node/tree/main/test "test") | [Let users to supply index from web archives (](https://github.com/nikitakarpei/my-yacy-node/commit/d7c794deb66e6ff6741b79d4519a4c645c89b5fe "Let users to supply index from web archives (#43) Closes #41 Closes #42 --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>")#43[)](https://github.com/nikitakarpei/my-yacy-node/commit/d7c794deb66e6ff6741b79d4519a4c645c89b5fe "Let users to supply index from web archives (#43) Closes #41 Closes #42 --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>") | Aug 27, 2026 |

| [tools](https://github.com/nikitakarpei/my-yacy-node/tree/main/tools "tools") | [tools](https://github.com/nikitakarpei/my-yacy-node/tree/main/tools "tools") | [Prepare](https://github.com/nikitakarpei/my-yacy-node/commit/6c0b2b6c03a666bee70844f4dff365196104b9fc "Prepare `corpusmarkdown` for mcp server introduction (#40) The service now stores redirects it observes during scraping. API was extended and observer interface reworked Co-authored-by: Claude Opus 5 <noreply@anthropic.com>")`corpusmarkdown`[for mcp server introduction (](https://github.com/nikitakarpei/my-yacy-node/commit/6c0b2b6c03a666bee70844f4dff365196104b9fc "Prepare `corpusmarkdown` for mcp server introduction (#40) The service now stores redirects it observes during scraping. API was extended and observer interface reworked Co-authored-by: Claude Opus 5 <noreply@anthropic.com>")#40[)](https://github.com/nikitakarpei/my-yacy-node/commit/6c0b2b6c03a666bee70844f4dff365196104b9fc "Prepare `corpusmarkdown` for mcp server introduction (#40) The service now stores redirects it observes during scraping. API was extended and observer interface reworked Co-authored-by: Claude Opus 5 <noreply@anthropic.com>") | Aug 25, 2026 |

| [.dockerignore](https://github.com/nikitakarpei/my-yacy-node/blob/main/.dockerignore ".dockerignore") | [.dockerignore](https://github.com/nikitakarpei/my-yacy-node/blob/main/.dockerignore ".dockerignore") | [Extract vault storage port from boltvault and add driver conformance …](https://github.com/nikitakarpei/my-yacy-node/commit/ce861eb81e5dda437f358562fdddfa817a5f474b "Extract vault storage port from boltvault and add driver conformance suite Split the storage abstraction into a port and interchangeable drivers: - vault: the port — domain types plus the Engine/EngineTxn/EngineBucket driver interfaces and the wrapper that enforces registration, transaction mode, codec translation, quota gating, and lifecycle. - boltvault, memvault: drivers behind that port. - vaulttest.RunConformance: the storage-contract suite every driver runs against itself (durable round trips, scan ordering, transaction atomicity, bucket isolation, byte accounting). Engine-independent guards (registration rules, write-in-View, codec error surfacing, unset-quota short-circuit, closed-vault and cancelled-context rejection) are tested once in the vault package against an in-test double engine, not duplicated per driver. The port depends on no concrete driver. Exclude the vaulttest support package from the coverage denominator and let COVER_EXCLUDE list multiple package patterns. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>") | Jun 26, 2026 |

| [.editorconfig](https://github.com/nikitakarpei/my-yacy-node/blob/main/.editorconfig ".editorconfig") | [.editorconfig](https://github.com/nikitakarpei/my-yacy-node/blob/main/.editorconfig ".editorconfig") | [Scaffold module layout, layer boundaries, and quality gate](https://github.com/nikitakarpei/my-yacy-node/commit/5ff8874d21b94e757c3a6c8f817dd4393a42e645 "Scaffold module layout, layer boundaries, and quality gate Lay the foundation before any feature work: two modules joined by a go.work workspace (the node and the standalone yacywire library), three internal layers (api/core/infrastructure) plus the cmd composition root, and a single `make verify` gate running formatting, vet, lint, architecture-boundary, coverage, and build checks across both modules with pinned tool versions. - yacywire holds the YaCy wire types, validation, and codecs; every layer including core builds on it. Its isolation is structural (separate module). - go-arch-lint enforces the dependency rules; golangci-lint v2 hosts the gofumpt/gci/golines formatters and the linters. - ADRs 0002-0004 record the rationale; AGENTS.md gains the file-naming, dependency, version-pinning, and testing standing rules. No feature code, ports, or storage adapters yet — those clone this structure. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>") | Jun 19, 2026 |

| [.env.example](https://github.com/nikitakarpei/my-yacy-node/blob/main/.env.example ".env.example") | [.env.example](https://github.com/nikitakarpei/my-yacy-node/blob/main/.env.example ".env.example") | [Add Docker Compose setup for freeworld network](https://github.com/nikitakarpei/my-yacy-node/commit/53eeeb8e6db1157a62e30ab14e0f71b763a284d5 "Add Docker Compose setup for freeworld network Bake in the public seedlists and template the per-peer required settings in .env.example. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>") | Jun 19, 2026 |

| [.gitignore](https://github.com/nikitakarpei/my-yacy-node/blob/main/.gitignore ".gitignore") | [.gitignore](https://github.com/nikitakarpei/my-yacy-node/blob/main/.gitignore ".gitignore") | [Draw each module's architecture on demand](https://github.com/nikitakarpei/my-yacy-node/commit/ce3a23b79c6e7f869a34d4a9624eab0c4140bc93 "Draw each module's architecture on demand Add make arch-diagram, writing one go-arch-lint SVG per module to a gitignored arch-diagrams/. It stays out of verify: .go-arch-lint.yml is the source of truth for the architecture, and a committed diagram would restate it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>") | Aug 13, 2026 |

| [.golangci.yml](https://github.com/nikitakarpei/my-yacy-node/blob/main/.golangci.yml ".golangci.yml") | [.golangci.yml](https://github.com/nikitakarpei/my-yacy-node/blob/main/.golangci.yml ".golangci.yml") | [Refactor node composition root (](https://github.com/nikitakarpei/my-yacy-node/commit/f3c7146a75957063ea1c663f475fd3067c361b05 "Refactor node composition root (#19) Part of #4 Attempt to reduce cognitive load of yacynode's composition root by moving configs and some middlewares out of it in separate packages and dropping indirection which was created because of `funlen` and `argument-limit` lint rules which are not useful for composition roots. --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>")#19[)](https://github.com/nikitakarpei/my-yacy-node/commit/f3c7146a75957063ea1c663f475fd3067c361b05 "Refactor node composition root (#19) Part of #4 Attempt to reduce cognitive load of yacynode's composition root by moving configs and some middlewares out of it in separate packages and dropping indirection which was created because of `funlen` and `argument-limit` lint rules which are not useful for composition roots. --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>") | Aug 16, 2026 |

| [AGENTS.md](https://github.com/nikitakarpei/my-yacy-node/blob/main/AGENTS.md "AGENTS.md") | [AGENTS.md](https://github.com/nikitakarpei/my-yacy-node/blob/main/AGENTS.md "AGENTS.md") | [Refactor white-box tests to black-box tests (](https://github.com/nikitakarpei/my-yacy-node/commit/e20b28090dc512b0f1cfbaa008755d2c5b23dc78 "Refactor white-box tests to black-box tests (#3) Tests should verify observable behavior and public contracts, not internals. This PR turns on the lint rule which prohibits white box tests and convert existing ones in black box tests. Also arch lint config from now on would cover only production code to keep rendered dependency graphs more readable by dropping tests-related noise. --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>")#3[)](https://github.com/nikitakarpei/my-yacy-node/commit/e20b28090dc512b0f1cfbaa008755d2c5b23dc78 "Refactor white-box tests to black-box tests (#3) Tests should verify observable behavior and public contracts, not internals. This PR turns on the lint rule which prohibits white box tests and convert existing ones in black box tests. Also arch lint config from now on would cover only production code to keep rendered dependency graphs more readable by dropping tests-related noise. --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>") | Aug 15, 2026 |

| [CLAUDE.md](https://github.com/nikitakarpei/my-yacy-node/blob/main/CLAUDE.md "CLAUDE.md") | [CLAUDE.md](https://github.com/nikitakarpei/my-yacy-node/blob/main/CLAUDE.md "CLAUDE.md") | [Initial agentic rules](https://github.com/nikitakarpei/my-yacy-node/commit/4a5b732063db983e210a422d9c726a6d222179e3 "Initial agentic rules") | Jun 19, 2026 |

| [LICENSE](https://github.com/nikitakarpei/my-yacy-node/blob/main/LICENSE "LICENSE") | [LICENSE](https://github.com/nikitakarpei/my-yacy-node/blob/main/LICENSE "LICENSE") | [Add AGPL-3.0 license](https://github.com/nikitakarpei/my-yacy-node/commit/4f2aab2f9a827316ea7fd1a3ea521acd3b4b79bf "Add AGPL-3.0 license Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015padqYAiypNFmPdV9Lcxwz") | Jun 19, 2026 |

| [Makefile](https://github.com/nikitakarpei/my-yacy-node/blob/main/Makefile "Makefile") | [Makefile](https://github.com/nikitakarpei/my-yacy-node/blob/main/Makefile "Makefile") | [Let users to supply index from web archives (](https://github.com/nikitakarpei/my-yacy-node/commit/d7c794deb66e6ff6741b79d4519a4c645c89b5fe "Let users to supply index from web archives (#43) Closes #41 Closes #42 --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>")#43[)](https://github.com/nikitakarpei/my-yacy-node/commit/d7c794deb66e6ff6741b79d4519a4c645c89b5fe "Let users to supply index from web archives (#43) Closes #41 Closes #42 --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>") | Aug 27, 2026 |

| [README.md](https://github.com/nikitakarpei/my-yacy-node/blob/main/README.md "README.md") | [README.md](https://github.com/nikitakarpei/my-yacy-node/blob/main/README.md "README.md") | [Let users to supply index from web archives (](https://github.com/nikitakarpei/my-yacy-node/commit/d7c794deb66e6ff6741b79d4519a4c645c89b5fe "Let users to supply index from web archives (#43) Closes #41 Closes #42 --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>")#43[)](https://github.com/nikitakarpei/my-yacy-node/commit/d7c794deb66e6ff6741b79d4519a4c645c89b5fe "Let users to supply index from web archives (#43) Closes #41 Closes #42 --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>") | Aug 27, 2026 |

| [go.mod](https://github.com/nikitakarpei/my-yacy-node/blob/main/go.mod "go.mod") | [go.mod](https://github.com/nikitakarpei/my-yacy-node/blob/main/go.mod "go.mod") | [Run lint and arch tools from pinned release binaries](https://github.com/nikitakarpei/my-yacy-node/commit/d6f4d86ce48bb9a49f928f151f6391337ddd86f2 "Run lint and arch tools from pinned release binaries The root module existed only to host golangci-lint and go-arch-lint as go tool directives, dragging ~250 indirect deps and a 97KB go.sum into a module nothing imports. Running them via go tool recompiled the tools against the workspace graph. Fetch the two tools as checksum-verified official release binaries into .toolchain/bin via tools/install, pinned in tools/tools.lock, and invoke them by path. Strip the root go.mod to a bare arch-lint anchor, drop its go.sum, and remove it from the workspace and the Dockerfiles. Point CI at yacynode/go.mod and cache the tool binaries. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>") | Jun 26, 2026 |

| [mado.toml](https://github.com/nikitakarpei/my-yacy-node/blob/main/mado.toml "mado.toml") | [mado.toml](https://github.com/nikitakarpei/my-yacy-node/blob/main/mado.toml "mado.toml") | [Gate markdown docs with a pinned mado linter](https://github.com/nikitakarpei/my-yacy-node/commit/05aa38baa899da2193a0e37d4aa58a66427b49e5 "Gate markdown docs with a pinned mado linter Adds make lint-md (git-tracked *.md through mado, checksum-pinned like golangci-lint/ruff) into the verify chain, and fixes the findings it surfaced: a missing top-level README heading, non-ordered list numbering, untagged shell fences, and bare URLs in the glossary. MD013 stays off until existing docs are rewrapped. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>") | Aug 8, 2026 | | View all files |

Repository files navigation

- README

- AGPL-3.0 license

More items

What this is

[](https://github.com/nikitakarpei/my-yacy-node#what-this-is)

![Image 7: ci](https://github.com/nikitakarpei/my-yacy-node/actions/workflows/ci.yml)![Image 8: go](https://github.com/nikitakarpei/my-yacy-node/blob/main/services/yacynode/go.mod)![Image 9: license](https://github.com/nikitakarpei/my-yacy-node/blob/main/LICENSE)![Image 10: images](https://github.com/nikitakarpei?tab=packages)

A lightweight, composable reimplementation of a YaCy peer, built as a set of small Go services. Each service runs independently, and a deployment wires together only what it needs. The services can also join the wider YaCy network over DHT.

The implementation follows its own design rather than mirroring the Java version: some features are still missing, while others take a different approach.

Motivation

[](https://github.com/nikitakarpei/my-yacy-node#motivation)

A standard YaCy node can exceed the memory available on small machines such as a Raspberry Pi, especially as the corpus grows. This project started with that constraint and became an experiment in a different way of building a YaCy peer.

Build your own search stack

[](https://github.com/nikitakarpei/my-yacy-node#build-your-own-search-stack)

The core idea is runtime composability: independent pieces can be combined into different deployments.

`doc/build-your-own-search-stack/` explores this through a series of working deployments, starting with a single peer on a spare machine and gradually building toward a complete self-hosted search engine.