Self-hosted knowledge management system with: - RAG API (FastAPI + pgvector) - Markdown vault (Obsidian/Logseq compatible) - Autonomous AI agents (ingestion, tagging, linking, summarization, maintenance) - Web UI (Next.js) - Docker Compose deployment - Ollama integration for local LLM inference Built by Copilot CLI, reviewed by Clawd.main
commit
626b04aa4e
@ -0,0 +1,61 @@
|
||||
# =============================================================================
|
||||
# AI Second Brain — Environment Configuration
|
||||
# Copy this file to .env and adjust values for your setup.
|
||||
# =============================================================================
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Database
|
||||
# ---------------------------------------------------------------------------
|
||||
POSTGRES_PASSWORD=brain
|
||||
POSTGRES_PORT=5432
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Ollama (local LLM)
|
||||
# ---------------------------------------------------------------------------
|
||||
OLLAMA_PORT=11434
|
||||
EMBEDDING_MODEL=nomic-embed-text
|
||||
CHAT_MODEL=mistral
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# RAG API
|
||||
# ---------------------------------------------------------------------------
|
||||
API_PORT=8000
|
||||
LOG_LEVEL=INFO
|
||||
|
||||
# Retrieval defaults
|
||||
SEARCH_TOP_K=10
|
||||
SEARCH_THRESHOLD=0.65
|
||||
RERANK_ENABLED=false
|
||||
|
||||
# Embedding provider: ollama | sentence_transformers
|
||||
EMBEDDING_PROVIDER=ollama
|
||||
EMBEDDING_DIMENSIONS=768
|
||||
|
||||
# CORS — comma-separated origins allowed to access the API
|
||||
CORS_ORIGINS=http://localhost:3000
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Web UI
|
||||
# ---------------------------------------------------------------------------
|
||||
UI_PORT=3000
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Ingestion Worker
|
||||
# ---------------------------------------------------------------------------
|
||||
VAULT_PATH=/vault
|
||||
CHUNK_SIZE=700
|
||||
CHUNK_OVERLAP=70
|
||||
POLL_INTERVAL=30
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# AI Agents
|
||||
# ---------------------------------------------------------------------------
|
||||
INGESTION_POLL=15
|
||||
LINKING_POLL=60
|
||||
TAGGING_POLL=120
|
||||
SUMMARIZATION_POLL=300
|
||||
MAINTENANCE_POLL=3600
|
||||
|
||||
# Enable auto-tagging and summarization by agents
|
||||
AUTO_TAG=true
|
||||
AUTO_SUMMARIZE=true
|
||||
@ -0,0 +1,37 @@
|
||||
# Environment
|
||||
.env
|
||||
.env.local
|
||||
*.env
|
||||
|
||||
# Python
|
||||
__pycache__/
|
||||
*.py[cod]
|
||||
*$py.class
|
||||
.venv/
|
||||
venv/
|
||||
*.egg-info/
|
||||
|
||||
# Node
|
||||
node_modules/
|
||||
.next/
|
||||
out/
|
||||
build/
|
||||
dist/
|
||||
|
||||
# Data
|
||||
*.db
|
||||
*.sqlite
|
||||
*.log
|
||||
|
||||
# IDE
|
||||
.vscode/
|
||||
.idea/
|
||||
|
||||
# Docker volumes (local)
|
||||
postgres_data/
|
||||
redis_data/
|
||||
ollama_data/
|
||||
|
||||
# OS
|
||||
.DS_Store
|
||||
Thumbs.db
|
||||
@ -0,0 +1,41 @@
|
||||
# AI Second Brain
|
||||
|
||||
A fully self-hosted, offline-capable knowledge management system with AI-powered retrieval, autonomous agents, and a Markdown-first philosophy.
|
||||
|
||||
## Quick Start
|
||||
|
||||
```bash
|
||||
cp .env.example .env
|
||||
# edit .env as needed
|
||||
docker compose up -d
|
||||
# wait for ollama to pull models
|
||||
docker compose exec ollama ollama pull nomic-embed-text
|
||||
docker compose exec ollama ollama pull mistral
|
||||
# open the UI
|
||||
open http://localhost:3000
|
||||
```
|
||||
|
||||
## Architecture
|
||||
|
||||
See [docs/architecture.md](docs/architecture.md) for the full design.
|
||||
|
||||
## Components
|
||||
|
||||
| Service | Port | Description |
|
||||
|--------------------|-------|--------------------------------|
|
||||
| Web UI | 3000 | Next.js knowledge interface |
|
||||
| RAG API | 8000 | FastAPI retrieval service |
|
||||
| Ollama | 11434 | Local LLM inference |
|
||||
| PostgreSQL | 5432 | Vector + relational store |
|
||||
| Redis | 6379 | Job queue |
|
||||
|
||||
## Documentation
|
||||
|
||||
- [Architecture](docs/architecture.md)
|
||||
- [Setup Guide](docs/setup.md)
|
||||
- [API Reference](docs/api.md)
|
||||
- [Agents Guide](docs/agents.md)
|
||||
|
||||
## License
|
||||
|
||||
MIT
|
||||
@ -0,0 +1,197 @@
|
||||
services:
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# PostgreSQL with pgvector
|
||||
# ---------------------------------------------------------------------------
|
||||
postgres:
|
||||
image: pgvector/pgvector:pg16
|
||||
container_name: second-brain-postgres
|
||||
restart: unless-stopped
|
||||
environment:
|
||||
POSTGRES_DB: second_brain
|
||||
POSTGRES_USER: brain
|
||||
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:-brain}
|
||||
volumes:
|
||||
- postgres_data:/var/lib/postgresql/data
|
||||
- ./infra/database/schema.sql:/docker-entrypoint-initdb.d/01_schema.sql:ro
|
||||
ports:
|
||||
- "${POSTGRES_PORT:-5432}:5432"
|
||||
networks:
|
||||
- brain-net
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "pg_isready -U brain -d second_brain"]
|
||||
interval: 10s
|
||||
timeout: 5s
|
||||
retries: 5
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Redis (job queue)
|
||||
# ---------------------------------------------------------------------------
|
||||
redis:
|
||||
image: redis:7-alpine
|
||||
container_name: second-brain-redis
|
||||
restart: unless-stopped
|
||||
volumes:
|
||||
- redis_data:/data
|
||||
networks:
|
||||
- brain-net
|
||||
healthcheck:
|
||||
test: ["CMD", "redis-cli", "ping"]
|
||||
interval: 10s
|
||||
timeout: 5s
|
||||
retries: 5
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Ollama (local LLM inference)
|
||||
# ---------------------------------------------------------------------------
|
||||
ollama:
|
||||
image: ollama/ollama:latest
|
||||
container_name: second-brain-ollama
|
||||
restart: unless-stopped
|
||||
volumes:
|
||||
- ollama_data:/root/.ollama
|
||||
ports:
|
||||
- "${OLLAMA_PORT:-11434}:11434"
|
||||
networks:
|
||||
- brain-net
|
||||
deploy:
|
||||
resources:
|
||||
reservations:
|
||||
devices:
|
||||
- driver: nvidia
|
||||
count: all
|
||||
capabilities: [gpu]
|
||||
healthcheck:
|
||||
test: ["CMD", "curl", "-f", "http://localhost:11434/api/tags"]
|
||||
interval: 30s
|
||||
timeout: 10s
|
||||
retries: 5
|
||||
start_period: 60s
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Ollama model bootstrap (pulls required models on first start)
|
||||
# ---------------------------------------------------------------------------
|
||||
ollama-bootstrap:
|
||||
image: ollama/ollama:latest
|
||||
container_name: second-brain-ollama-bootstrap
|
||||
depends_on:
|
||||
ollama:
|
||||
condition: service_healthy
|
||||
volumes:
|
||||
- ollama_data:/root/.ollama
|
||||
networks:
|
||||
- brain-net
|
||||
entrypoint: ["/bin/sh", "-c"]
|
||||
command:
|
||||
- |
|
||||
OLLAMA_HOST=ollama:11434 ollama pull ${EMBEDDING_MODEL:-nomic-embed-text}
|
||||
OLLAMA_HOST=ollama:11434 ollama pull ${CHAT_MODEL:-mistral}
|
||||
restart: "no"
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# RAG API (FastAPI)
|
||||
# ---------------------------------------------------------------------------
|
||||
rag-api:
|
||||
build:
|
||||
context: ./services/rag-api
|
||||
dockerfile: Dockerfile
|
||||
container_name: second-brain-rag-api
|
||||
restart: unless-stopped
|
||||
env_file:
|
||||
- .env
|
||||
environment:
|
||||
DATABASE_URL: postgresql://brain:${POSTGRES_PASSWORD:-brain}@postgres:5432/second_brain
|
||||
OLLAMA_URL: http://ollama:11434
|
||||
depends_on:
|
||||
postgres:
|
||||
condition: service_healthy
|
||||
ollama:
|
||||
condition: service_healthy
|
||||
ports:
|
||||
- "${API_PORT:-8000}:8000"
|
||||
networks:
|
||||
- brain-net
|
||||
healthcheck:
|
||||
test: ["CMD", "curl", "-f", "http://localhost:8000/api/v1/health"]
|
||||
interval: 15s
|
||||
timeout: 5s
|
||||
retries: 5
|
||||
start_period: 30s
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Ingestion Worker
|
||||
# ---------------------------------------------------------------------------
|
||||
ingestion-worker:
|
||||
build:
|
||||
context: ./services/ingestion-worker
|
||||
dockerfile: Dockerfile
|
||||
container_name: second-brain-ingestion
|
||||
restart: unless-stopped
|
||||
env_file:
|
||||
- .env
|
||||
environment:
|
||||
DATABASE_URL: postgresql://brain:${POSTGRES_PASSWORD:-brain}@postgres:5432/second_brain
|
||||
OLLAMA_URL: http://ollama:11434
|
||||
VAULT_PATH: /vault
|
||||
volumes:
|
||||
- ./vault:/vault:ro
|
||||
depends_on:
|
||||
postgres:
|
||||
condition: service_healthy
|
||||
ollama:
|
||||
condition: service_healthy
|
||||
networks:
|
||||
- brain-net
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# AI Agents
|
||||
# ---------------------------------------------------------------------------
|
||||
agents:
|
||||
build:
|
||||
context: ./services/agents
|
||||
dockerfile: Dockerfile
|
||||
container_name: second-brain-agents
|
||||
restart: unless-stopped
|
||||
env_file:
|
||||
- .env
|
||||
environment:
|
||||
DATABASE_URL: postgresql://brain:${POSTGRES_PASSWORD:-brain}@postgres:5432/second_brain
|
||||
OLLAMA_URL: http://ollama:11434
|
||||
VAULT_PATH: /vault
|
||||
volumes:
|
||||
- ./vault:/vault:ro
|
||||
depends_on:
|
||||
postgres:
|
||||
condition: service_healthy
|
||||
rag-api:
|
||||
condition: service_healthy
|
||||
networks:
|
||||
- brain-net
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Web UI (Next.js)
|
||||
# ---------------------------------------------------------------------------
|
||||
web-ui:
|
||||
build:
|
||||
context: ./services/web-ui
|
||||
dockerfile: Dockerfile
|
||||
container_name: second-brain-ui
|
||||
restart: unless-stopped
|
||||
environment:
|
||||
NEXT_PUBLIC_API_URL: http://localhost:${API_PORT:-8000}
|
||||
depends_on:
|
||||
rag-api:
|
||||
condition: service_healthy
|
||||
ports:
|
||||
- "${UI_PORT:-3000}:3000"
|
||||
networks:
|
||||
- brain-net
|
||||
|
||||
volumes:
|
||||
postgres_data:
|
||||
redis_data:
|
||||
ollama_data:
|
||||
|
||||
networks:
|
||||
brain-net:
|
||||
driver: bridge
|
||||
@ -0,0 +1,178 @@
|
||||
# API Reference
|
||||
|
||||
Base URL: `http://localhost:8000/api/v1`
|
||||
|
||||
Interactive docs: `http://localhost:8000/docs` (Swagger UI)
|
||||
|
||||
---
|
||||
|
||||
## Authentication
|
||||
|
||||
No authentication is required by default (local-only deployment). Add a reverse proxy with auth for production.
|
||||
|
||||
---
|
||||
|
||||
## Endpoints
|
||||
|
||||
### POST `/search`
|
||||
|
||||
Hybrid vector + full-text search across the knowledge base.
|
||||
|
||||
**Request:**
|
||||
```json
|
||||
{
|
||||
"query": "machine learning transformers",
|
||||
"limit": 10,
|
||||
"threshold": 0.65,
|
||||
"tags": ["ml", "ai"],
|
||||
"hybrid": true
|
||||
}
|
||||
```
|
||||
|
||||
**Response:**
|
||||
```json
|
||||
{
|
||||
"results": [
|
||||
{
|
||||
"document_id": "uuid",
|
||||
"chunk_id": "uuid",
|
||||
"title": "Introduction to Transformers",
|
||||
"path": "resources/ml/transformers.md",
|
||||
"content": "...chunk text...",
|
||||
"score": 0.923,
|
||||
"tags": ["ml", "transformers"],
|
||||
"highlight": "...bolded match..."
|
||||
}
|
||||
],
|
||||
"total": 5,
|
||||
"query_time_ms": 18.4
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### POST `/chat`
|
||||
|
||||
RAG chat with streaming Server-Sent Events response.
|
||||
|
||||
**Request:**
|
||||
```json
|
||||
{
|
||||
"message": "What do I know about neural networks?",
|
||||
"context_limit": 5,
|
||||
"stream": true
|
||||
}
|
||||
```
|
||||
|
||||
**Response (SSE stream):**
|
||||
```
|
||||
data: {"type":"sources","sources":[{"title":"Neural Nets","path":"...","score":0.91}]}
|
||||
|
||||
data: {"type":"token","token":"Neural"}
|
||||
|
||||
data: {"type":"token","token":" networks"}
|
||||
|
||||
data: {"type":"done"}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### GET `/document/{id}`
|
||||
|
||||
Get a document by UUID.
|
||||
|
||||
**Response:** Full document object including content, frontmatter, tags.
|
||||
|
||||
---
|
||||
|
||||
### GET `/document/path/{path}`
|
||||
|
||||
Get a document by its vault-relative path (e.g., `resources/ml/intro.md`).
|
||||
|
||||
---
|
||||
|
||||
### GET `/document/{id}/related`
|
||||
|
||||
Get related documents ordered by semantic similarity.
|
||||
|
||||
**Query params:** `limit` (default: 5)
|
||||
|
||||
---
|
||||
|
||||
### POST `/index`
|
||||
|
||||
Queue a specific file for indexing.
|
||||
|
||||
**Request:**
|
||||
```json
|
||||
{ "path": "notes/new-note.md" }
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### POST `/index/reindex`
|
||||
|
||||
Queue a full vault re-index.
|
||||
|
||||
**Request:**
|
||||
```json
|
||||
{ "force": false }
|
||||
```
|
||||
Set `force: true` to reindex even unchanged files.
|
||||
|
||||
---
|
||||
|
||||
### GET `/tags`
|
||||
|
||||
List all tags with document counts.
|
||||
|
||||
**Response:**
|
||||
```json
|
||||
[
|
||||
{"tag": "machine-learning", "count": 42},
|
||||
{"tag": "python", "count": 38}
|
||||
]
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### GET `/graph`
|
||||
|
||||
Get the knowledge graph (nodes = documents, edges = links).
|
||||
|
||||
**Query params:** `limit` (default: 200)
|
||||
|
||||
---
|
||||
|
||||
### GET `/stats`
|
||||
|
||||
System statistics.
|
||||
|
||||
**Response:**
|
||||
```json
|
||||
{
|
||||
"total_documents": 1234,
|
||||
"total_chunks": 8765,
|
||||
"total_relations": 3210,
|
||||
"total_tags": 87,
|
||||
"last_indexed": "2026-03-05T19:00:00Z",
|
||||
"embedding_model": "nomic-embed-text",
|
||||
"chat_model": "mistral"
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### GET `/health`
|
||||
|
||||
Health check.
|
||||
|
||||
**Response:**
|
||||
```json
|
||||
{
|
||||
"status": "ok",
|
||||
"database": "ok",
|
||||
"ollama": "ok",
|
||||
"version": "1.0.0"
|
||||
}
|
||||
```
|
||||
@ -0,0 +1,20 @@
|
||||
#!/bin/bash
|
||||
# Applies database migrations in order.
|
||||
# Usage: ./migrate.sh [up|down]
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
DB_URL="${DATABASE_URL:-postgresql://brain:brain@localhost:5432/second_brain}"
|
||||
MIGRATIONS_DIR="$(dirname "$0")/migrations"
|
||||
|
||||
ACTION="${1:-up}"
|
||||
|
||||
if [ "$ACTION" = "up" ]; then
|
||||
echo "Applying schema..."
|
||||
psql "$DB_URL" -f "$(dirname "$0")/schema.sql"
|
||||
echo "Schema applied."
|
||||
elif [ "$ACTION" = "down" ]; then
|
||||
echo "Dropping schema..."
|
||||
psql "$DB_URL" -c "DROP SCHEMA public CASCADE; CREATE SCHEMA public;"
|
||||
echo "Schema dropped."
|
||||
fi
|
||||
@ -0,0 +1,195 @@
|
||||
-- AI Second Brain — PostgreSQL Schema
|
||||
-- Requires: PostgreSQL 14+ with pgvector extension
|
||||
|
||||
-- Enable extensions
|
||||
CREATE EXTENSION IF NOT EXISTS "uuid-ossp";
|
||||
CREATE EXTENSION IF NOT EXISTS vector;
|
||||
CREATE EXTENSION IF NOT EXISTS pg_trgm; -- for fuzzy text search
|
||||
|
||||
-- ---------------------------------------------------------------------------
|
||||
-- DOCUMENTS
|
||||
-- Represents a single Markdown file in the vault.
|
||||
-- ---------------------------------------------------------------------------
|
||||
CREATE TABLE IF NOT EXISTS documents (
|
||||
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
path TEXT NOT NULL UNIQUE, -- relative path within vault
|
||||
title TEXT,
|
||||
content TEXT NOT NULL, -- full raw markdown
|
||||
content_hash TEXT NOT NULL, -- SHA-256 for change detection
|
||||
frontmatter JSONB NOT NULL DEFAULT '{}',
|
||||
tags TEXT[] NOT NULL DEFAULT '{}',
|
||||
aliases TEXT[] NOT NULL DEFAULT '{}',
|
||||
word_count INTEGER,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
||||
updated_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
||||
indexed_at TIMESTAMPTZ,
|
||||
fts_vector TSVECTOR -- auto-maintained below
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_documents_path ON documents (path);
|
||||
CREATE INDEX IF NOT EXISTS idx_documents_tags ON documents USING GIN (tags);
|
||||
CREATE INDEX IF NOT EXISTS idx_documents_aliases ON documents USING GIN (aliases);
|
||||
CREATE INDEX IF NOT EXISTS idx_documents_fts ON documents USING GIN (fts_vector);
|
||||
CREATE INDEX IF NOT EXISTS idx_documents_frontmatter ON documents USING GIN (frontmatter);
|
||||
CREATE INDEX IF NOT EXISTS idx_documents_updated ON documents (updated_at DESC);
|
||||
|
||||
-- Auto-update fts_vector on insert/update
|
||||
CREATE OR REPLACE FUNCTION documents_fts_trigger()
|
||||
RETURNS TRIGGER AS $$
|
||||
BEGIN
|
||||
NEW.fts_vector :=
|
||||
setweight(to_tsvector('english', coalesce(NEW.title, '')), 'A') ||
|
||||
setweight(to_tsvector('english', coalesce(array_to_string(NEW.tags, ' '), '')), 'B') ||
|
||||
setweight(to_tsvector('english', coalesce(NEW.content, '')), 'C');
|
||||
RETURN NEW;
|
||||
END;
|
||||
$$ LANGUAGE plpgsql;
|
||||
|
||||
DROP TRIGGER IF EXISTS trig_documents_fts ON documents;
|
||||
CREATE TRIGGER trig_documents_fts
|
||||
BEFORE INSERT OR UPDATE ON documents
|
||||
FOR EACH ROW EXECUTE FUNCTION documents_fts_trigger();
|
||||
|
||||
-- Auto-update updated_at timestamp
|
||||
CREATE OR REPLACE FUNCTION set_updated_at()
|
||||
RETURNS TRIGGER AS $$
|
||||
BEGIN
|
||||
NEW.updated_at = now();
|
||||
RETURN NEW;
|
||||
END;
|
||||
$$ LANGUAGE plpgsql;
|
||||
|
||||
DROP TRIGGER IF EXISTS trig_documents_updated_at ON documents;
|
||||
CREATE TRIGGER trig_documents_updated_at
|
||||
BEFORE UPDATE ON documents
|
||||
FOR EACH ROW EXECUTE FUNCTION set_updated_at();
|
||||
|
||||
-- ---------------------------------------------------------------------------
|
||||
-- CHUNKS
|
||||
-- Sliding-window text chunks from documents, each with an embedding vector.
|
||||
-- ---------------------------------------------------------------------------
|
||||
CREATE TABLE IF NOT EXISTS chunks (
|
||||
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
document_id UUID NOT NULL REFERENCES documents (id) ON DELETE CASCADE,
|
||||
chunk_index INTEGER NOT NULL,
|
||||
content TEXT NOT NULL,
|
||||
token_count INTEGER,
|
||||
embedding VECTOR(768), -- nomic-embed-text dimension
|
||||
metadata JSONB NOT NULL DEFAULT '{}',-- heading path, page, etc.
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
||||
UNIQUE (document_id, chunk_index)
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_chunks_document_id ON chunks (document_id);
|
||||
|
||||
-- HNSW index — fast approximate nearest-neighbour search
|
||||
-- Requires pgvector >= 0.5.0. Falls back to IVFFlat if unavailable.
|
||||
CREATE INDEX IF NOT EXISTS idx_chunks_embedding_hnsw
|
||||
ON chunks USING hnsw (embedding vector_cosine_ops)
|
||||
WITH (m = 16, ef_construction = 64);
|
||||
|
||||
-- ---------------------------------------------------------------------------
|
||||
-- ENTITIES
|
||||
-- Named entities extracted from documents (optional NER layer).
|
||||
-- ---------------------------------------------------------------------------
|
||||
CREATE TABLE IF NOT EXISTS entities (
|
||||
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
document_id UUID NOT NULL REFERENCES documents (id) ON DELETE CASCADE,
|
||||
name TEXT NOT NULL,
|
||||
entity_type TEXT NOT NULL, -- PERSON, ORG, CONCEPT, PLACE, etc.
|
||||
context TEXT, -- surrounding sentence
|
||||
confidence FLOAT,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_entities_document_id ON entities (document_id);
|
||||
CREATE INDEX IF NOT EXISTS idx_entities_name ON entities (name);
|
||||
CREATE INDEX IF NOT EXISTS idx_entities_type ON entities (entity_type);
|
||||
CREATE INDEX IF NOT EXISTS idx_entities_name_trgm ON entities USING GIN (name gin_trgm_ops);
|
||||
|
||||
-- ---------------------------------------------------------------------------
|
||||
-- RELATIONS
|
||||
-- WikiLink / explicit relations between documents.
|
||||
-- ---------------------------------------------------------------------------
|
||||
CREATE TABLE IF NOT EXISTS relations (
|
||||
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
source_doc_id UUID NOT NULL REFERENCES documents (id) ON DELETE CASCADE,
|
||||
target_path TEXT NOT NULL, -- raw link target (may be unresolved)
|
||||
target_doc_id UUID REFERENCES documents (id) ON DELETE SET NULL,
|
||||
relation_type TEXT NOT NULL DEFAULT 'wikilink', -- wikilink | tag | explicit | ai-inferred
|
||||
label TEXT, -- optional human label for the edge
|
||||
context TEXT, -- surrounding text of the link
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_relations_source ON relations (source_doc_id);
|
||||
CREATE INDEX IF NOT EXISTS idx_relations_target_id ON relations (target_doc_id);
|
||||
CREATE INDEX IF NOT EXISTS idx_relations_target_path ON relations (target_path);
|
||||
CREATE INDEX IF NOT EXISTS idx_relations_type ON relations (relation_type);
|
||||
|
||||
-- ---------------------------------------------------------------------------
|
||||
-- AGENT JOBS
|
||||
-- Persistent job queue consumed by AI agents.
|
||||
-- ---------------------------------------------------------------------------
|
||||
CREATE TABLE IF NOT EXISTS agent_jobs (
|
||||
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
agent_type TEXT NOT NULL, -- ingestion | linking | tagging | summarization | maintenance
|
||||
status TEXT NOT NULL DEFAULT 'pending', -- pending | running | done | failed | cancelled
|
||||
priority INTEGER NOT NULL DEFAULT 5, -- 1 (highest) .. 10 (lowest)
|
||||
payload JSONB NOT NULL DEFAULT '{}',
|
||||
result JSONB,
|
||||
error TEXT,
|
||||
retry_count INTEGER NOT NULL DEFAULT 0,
|
||||
max_retries INTEGER NOT NULL DEFAULT 3,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
||||
started_at TIMESTAMPTZ,
|
||||
completed_at TIMESTAMPTZ,
|
||||
scheduled_for TIMESTAMPTZ NOT NULL DEFAULT now()
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_agent_jobs_status ON agent_jobs (status);
|
||||
CREATE INDEX IF NOT EXISTS idx_agent_jobs_type ON agent_jobs (agent_type);
|
||||
CREATE INDEX IF NOT EXISTS idx_agent_jobs_scheduled ON agent_jobs (scheduled_for ASC)
|
||||
WHERE status = 'pending';
|
||||
|
||||
-- ---------------------------------------------------------------------------
|
||||
-- AGENT LOGS
|
||||
-- Structured log entries written by agents.
|
||||
-- ---------------------------------------------------------------------------
|
||||
CREATE TABLE IF NOT EXISTS agent_logs (
|
||||
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
job_id UUID REFERENCES agent_jobs (id) ON DELETE SET NULL,
|
||||
agent_type TEXT NOT NULL,
|
||||
level TEXT NOT NULL DEFAULT 'info', -- debug | info | warning | error
|
||||
message TEXT NOT NULL,
|
||||
metadata JSONB NOT NULL DEFAULT '{}',
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_agent_logs_job_id ON agent_logs (job_id);
|
||||
CREATE INDEX IF NOT EXISTS idx_agent_logs_created ON agent_logs (created_at DESC);
|
||||
CREATE INDEX IF NOT EXISTS idx_agent_logs_level ON agent_logs (level);
|
||||
|
||||
-- ---------------------------------------------------------------------------
|
||||
-- SYSTEM CONFIG
|
||||
-- Runtime key-value configuration, editable by agents and admins.
|
||||
-- ---------------------------------------------------------------------------
|
||||
CREATE TABLE IF NOT EXISTS system_config (
|
||||
key TEXT PRIMARY KEY,
|
||||
value JSONB NOT NULL,
|
||||
description TEXT,
|
||||
updated_at TIMESTAMPTZ NOT NULL DEFAULT now()
|
||||
);
|
||||
|
||||
-- Seed default configuration
|
||||
INSERT INTO system_config (key, value, description) VALUES
|
||||
('embedding_model', '"nomic-embed-text"', 'Ollama model for embeddings'),
|
||||
('chat_model', '"mistral"', 'Ollama model for chat/generation'),
|
||||
('chunk_size', '700', 'Target tokens per chunk'),
|
||||
('chunk_overlap', '70', 'Overlap tokens between chunks'),
|
||||
('search_top_k', '10', 'Default number of search results'),
|
||||
('search_threshold', '0.65', 'Minimum cosine similarity score'),
|
||||
('rerank_enabled', 'false', 'Enable cross-encoder reranking'),
|
||||
('auto_tag', 'true', 'Auto-tag documents via LLM'),
|
||||
('auto_summarize', 'true', 'Auto-summarize long documents')
|
||||
ON CONFLICT (key) DO NOTHING;
|
||||
@ -0,0 +1,29 @@
|
||||
#!/usr/bin/env bash
|
||||
# scripts/health.sh — Check health of all services.
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
API_URL="${API_URL:-http://localhost:8000}"
|
||||
OLLAMA_URL="${OLLAMA_URL:-http://localhost:11434}"
|
||||
|
||||
check() {
|
||||
local name="$1"
|
||||
local url="$2"
|
||||
if curl -sf "$url" > /dev/null 2>&1; then
|
||||
echo " ✓ $name"
|
||||
else
|
||||
echo " ✗ $name (unreachable: $url)"
|
||||
fi
|
||||
}
|
||||
|
||||
echo "🩺 Second Brain — Health Check"
|
||||
echo ""
|
||||
check "RAG API" "$API_URL/api/v1/health"
|
||||
check "Ollama" "$OLLAMA_URL/api/tags"
|
||||
|
||||
echo ""
|
||||
echo "Detailed API health:"
|
||||
curl -sf "$API_URL/api/v1/health" | python3 -m json.tool 2>/dev/null || echo "(API unavailable)"
|
||||
echo ""
|
||||
echo "Stats:"
|
||||
curl -sf "$API_URL/api/v1/stats" | python3 -m json.tool 2>/dev/null || echo "(API unavailable)"
|
||||
@ -0,0 +1,16 @@
|
||||
#!/usr/bin/env bash
|
||||
# scripts/reindex.sh — Trigger a full vault reindex via the API.
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
API_URL="${API_URL:-http://localhost:8000}"
|
||||
FORCE="${1:-false}"
|
||||
|
||||
echo "🔄 Triggering vault reindex (force=$FORCE)..."
|
||||
|
||||
RESPONSE=$(curl -sf -X POST "$API_URL/api/v1/index/reindex" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d "{\"force\": $FORCE}")
|
||||
|
||||
echo "$RESPONSE" | python3 -m json.tool 2>/dev/null || echo "$RESPONSE"
|
||||
echo "✓ Reindex job queued."
|
||||
@ -0,0 +1,51 @@
|
||||
#!/usr/bin/env bash
|
||||
# scripts/start.sh — Bootstrap and start the Second Brain stack.
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
|
||||
ROOT="$SCRIPT_DIR/.."
|
||||
|
||||
cd "$ROOT"
|
||||
|
||||
echo "🧠 AI Second Brain — startup"
|
||||
|
||||
# Ensure .env exists
|
||||
if [ ! -f .env ]; then
|
||||
echo " → Creating .env from .env.example"
|
||||
cp .env.example .env
|
||||
echo " ⚠️ Edit .env before production use (set POSTGRES_PASSWORD etc.)"
|
||||
fi
|
||||
|
||||
# Ensure vault directory exists
|
||||
mkdir -p vault
|
||||
|
||||
echo " → Starting Docker services..."
|
||||
docker compose up -d --build
|
||||
|
||||
echo " → Waiting for services to be healthy..."
|
||||
sleep 5
|
||||
|
||||
# Poll health endpoint
|
||||
MAX_ATTEMPTS=30
|
||||
ATTEMPT=0
|
||||
until curl -sf http://localhost:8000/api/v1/health > /dev/null 2>&1; do
|
||||
ATTEMPT=$((ATTEMPT + 1))
|
||||
if [ "$ATTEMPT" -ge "$MAX_ATTEMPTS" ]; then
|
||||
echo " ✗ API did not become healthy after ${MAX_ATTEMPTS} attempts."
|
||||
echo " Check logs with: docker compose logs rag-api"
|
||||
exit 1
|
||||
fi
|
||||
echo " ... waiting for API (${ATTEMPT}/${MAX_ATTEMPTS})"
|
||||
sleep 5
|
||||
done
|
||||
|
||||
echo ""
|
||||
echo " ✓ Second Brain is running!"
|
||||
echo ""
|
||||
echo " 🌐 Web UI: http://localhost:$(grep UI_PORT .env | cut -d= -f2 || echo 3000)"
|
||||
echo " 🔌 RAG API: http://localhost:$(grep API_PORT .env | cut -d= -f2 || echo 8000)"
|
||||
echo " 📖 API Docs: http://localhost:$(grep API_PORT .env | cut -d= -f2 || echo 8000)/docs"
|
||||
echo " 🤖 Ollama: http://localhost:$(grep OLLAMA_PORT .env | cut -d= -f2 || echo 11434)"
|
||||
echo ""
|
||||
echo " Run 'docker compose logs -f' to follow logs."
|
||||
@ -0,0 +1,24 @@
|
||||
FROM python:3.12-slim
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
RUN apt-get update && apt-get install -y --no-install-recommends \
|
||||
build-essential libpq-dev \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
# Install ingestion worker deps first (agents depend on ingestion modules)
|
||||
COPY ../ingestion-worker/requirements.txt /tmp/ingestion-requirements.txt
|
||||
RUN pip install --no-cache-dir -r /tmp/ingestion-requirements.txt
|
||||
|
||||
COPY requirements.txt .
|
||||
RUN pip install --no-cache-dir -r requirements.txt
|
||||
|
||||
# Copy ingestion worker source (agents reuse parser, chunker, embedder, indexer, pipeline)
|
||||
COPY ../ingestion-worker /app/ingestion-worker
|
||||
|
||||
COPY . .
|
||||
|
||||
ENV PYTHONUNBUFFERED=1
|
||||
ENV PYTHONPATH=/app:/app/ingestion-worker
|
||||
|
||||
CMD ["python", "main.py"]
|
||||
@ -0,0 +1,190 @@
|
||||
"""
|
||||
base_agent.py — Abstract base class for all AI agents.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import logging
|
||||
import time
|
||||
import traceback
|
||||
from abc import ABC, abstractmethod
|
||||
from typing import Any, Optional
|
||||
|
||||
import asyncpg
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class BaseAgent(ABC):
|
||||
"""
|
||||
All agents inherit from this class.
|
||||
|
||||
Responsibilities:
|
||||
- Poll agent_jobs table for work
|
||||
- Claim jobs atomically
|
||||
- Execute with exponential-backoff retries
|
||||
- Log results / errors to agent_logs
|
||||
"""
|
||||
|
||||
agent_type: str # Must be set by subclass
|
||||
|
||||
def __init__(self, pool: asyncpg.Pool, settings: Any) -> None:
|
||||
self.pool = pool
|
||||
self.settings = settings
|
||||
self._log = logging.getLogger(f'agent.{self.agent_type}')
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Public interface
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
async def run_forever(self, poll_interval: int = 10) -> None:
|
||||
"""Poll for jobs indefinitely."""
|
||||
self._log.info('Agent started (poll_interval=%ds)', poll_interval)
|
||||
while True:
|
||||
try:
|
||||
job = await self._claim_job()
|
||||
if job:
|
||||
await self._execute(job)
|
||||
else:
|
||||
await asyncio.sleep(poll_interval)
|
||||
except asyncio.CancelledError:
|
||||
self._log.info('Agent shutting down')
|
||||
return
|
||||
except Exception as exc:
|
||||
self._log.error('Unexpected error in agent loop: %s', exc, exc_info=True)
|
||||
await asyncio.sleep(poll_interval)
|
||||
|
||||
async def enqueue(self, payload: dict, priority: int = 5, delay_seconds: int = 0) -> str:
|
||||
"""Create a new job for this agent."""
|
||||
import uuid
|
||||
from datetime import datetime, timezone, timedelta
|
||||
job_id = str(uuid.uuid4())
|
||||
scheduled = datetime.now(timezone.utc)
|
||||
if delay_seconds:
|
||||
scheduled += timedelta(seconds=delay_seconds)
|
||||
|
||||
async with self.pool.acquire() as conn:
|
||||
await conn.execute(
|
||||
"""
|
||||
INSERT INTO agent_jobs (id, agent_type, priority, payload, scheduled_for)
|
||||
VALUES ($1::uuid, $2, $3, $4::jsonb, $5)
|
||||
""",
|
||||
job_id, self.agent_type, priority, json.dumps(payload), scheduled,
|
||||
)
|
||||
return job_id
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Abstract
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
@abstractmethod
|
||||
async def process(self, job_id: str, payload: dict) -> dict:
|
||||
"""Process a single job. Return result dict."""
|
||||
...
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Internal helpers
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
async def _claim_job(self) -> Optional[asyncpg.Record]:
|
||||
"""Atomically claim the next pending job for this agent type."""
|
||||
async with self.pool.acquire() as conn:
|
||||
row = await conn.fetchrow(
|
||||
"""
|
||||
UPDATE agent_jobs
|
||||
SET status = 'running', started_at = now()
|
||||
WHERE id = (
|
||||
SELECT id FROM agent_jobs
|
||||
WHERE agent_type = $1
|
||||
AND status = 'pending'
|
||||
AND scheduled_for <= now()
|
||||
AND retry_count < max_retries
|
||||
ORDER BY priority ASC, scheduled_for ASC
|
||||
LIMIT 1
|
||||
FOR UPDATE SKIP LOCKED
|
||||
)
|
||||
RETURNING *
|
||||
""",
|
||||
self.agent_type,
|
||||
)
|
||||
return row
|
||||
|
||||
async def _execute(self, job: asyncpg.Record) -> None:
|
||||
job_id = str(job['id'])
|
||||
payload = dict(job['payload'] or {})
|
||||
self._log.info('Processing job %s', job_id)
|
||||
start = time.monotonic()
|
||||
|
||||
try:
|
||||
result = await self.process(job_id, payload)
|
||||
elapsed = time.monotonic() - start
|
||||
async with self.pool.acquire() as conn:
|
||||
await conn.execute(
|
||||
"""
|
||||
UPDATE agent_jobs
|
||||
SET status = 'done', result = $2::jsonb, completed_at = now()
|
||||
WHERE id = $1::uuid
|
||||
""",
|
||||
job_id, json.dumps(result or {}),
|
||||
)
|
||||
await self._log_event(job_id, 'info', f'Job done in {elapsed:.2f}s', result or {})
|
||||
|
||||
except Exception as exc:
|
||||
elapsed = time.monotonic() - start
|
||||
err_msg = str(exc)
|
||||
self._log.error('Job %s failed: %s', job_id, err_msg, exc_info=True)
|
||||
|
||||
async with self.pool.acquire() as conn:
|
||||
row = await conn.fetchrow(
|
||||
'SELECT retry_count, max_retries FROM agent_jobs WHERE id = $1::uuid', job_id
|
||||
)
|
||||
retries = (row['retry_count'] or 0) + 1
|
||||
max_retries = row['max_retries'] or 3
|
||||
|
||||
if retries < max_retries:
|
||||
# Re-queue with exponential backoff
|
||||
backoff = 2 ** retries
|
||||
await conn.execute(
|
||||
"""
|
||||
UPDATE agent_jobs
|
||||
SET status = 'pending',
|
||||
retry_count = $2,
|
||||
error = $3,
|
||||
scheduled_for = now() + ($4 || ' seconds')::interval
|
||||
WHERE id = $1::uuid
|
||||
""",
|
||||
job_id, retries, err_msg, str(backoff),
|
||||
)
|
||||
await self._log_event(job_id, 'warning',
|
||||
f'Retry {retries}/{max_retries} in {backoff}s', {})
|
||||
else:
|
||||
await conn.execute(
|
||||
"""
|
||||
UPDATE agent_jobs
|
||||
SET status = 'failed', error = $2, completed_at = now()
|
||||
WHERE id = $1::uuid
|
||||
""",
|
||||
job_id, err_msg,
|
||||
)
|
||||
await self._log_event(job_id, 'error', f'Job permanently failed: {err_msg}', {})
|
||||
|
||||
async def _log_event(
|
||||
self,
|
||||
job_id: Optional[str],
|
||||
level: str,
|
||||
message: str,
|
||||
metadata: dict,
|
||||
) -> None:
|
||||
try:
|
||||
async with self.pool.acquire() as conn:
|
||||
await conn.execute(
|
||||
"""
|
||||
INSERT INTO agent_logs (job_id, agent_type, level, message, metadata)
|
||||
VALUES ($1::uuid, $2, $3, $4, $5::jsonb)
|
||||
""",
|
||||
job_id, self.agent_type, level, message, json.dumps(metadata),
|
||||
)
|
||||
except Exception as log_exc:
|
||||
self._log.warning('Failed to write agent log: %s', log_exc)
|
||||
@ -0,0 +1,46 @@
|
||||
"""
|
||||
ingestion/agent.py — Ingestion Agent: indexes new/changed files from the vault.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).parent.parent.parent / 'ingestion-worker'))
|
||||
|
||||
import asyncpg
|
||||
|
||||
from base_agent import BaseAgent
|
||||
from pipeline import ingest_file
|
||||
from settings import Settings as IngestionSettings
|
||||
|
||||
|
||||
class IngestionAgent(BaseAgent):
|
||||
agent_type = 'ingestion'
|
||||
|
||||
async def process(self, job_id: str, payload: dict) -> dict:
|
||||
settings = IngestionSettings()
|
||||
vault_root = Path(settings.vault_path)
|
||||
|
||||
if payload.get('reindex_all'):
|
||||
md_files = list(vault_root.rglob('*.md'))
|
||||
indexed = 0
|
||||
skipped = 0
|
||||
for fp in md_files:
|
||||
async with self.pool.acquire() as conn:
|
||||
result = await ingest_file(fp, settings, conn)
|
||||
if result:
|
||||
indexed += 1
|
||||
else:
|
||||
skipped += 1
|
||||
return {'indexed': indexed, 'skipped': skipped, 'total': len(md_files)}
|
||||
|
||||
elif payload.get('path'):
|
||||
file_path = vault_root / payload['path']
|
||||
async with self.pool.acquire() as conn:
|
||||
result = await ingest_file(file_path, settings, conn)
|
||||
return {'indexed': 1 if result else 0, 'path': payload['path']}
|
||||
|
||||
return {'message': 'No action specified'}
|
||||
@ -0,0 +1,83 @@
|
||||
"""
|
||||
linking/agent.py — Knowledge Linking Agent: infers and creates AI-powered document links.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
|
||||
import asyncpg
|
||||
import httpx
|
||||
|
||||
from base_agent import BaseAgent
|
||||
|
||||
logger = logging.getLogger('agent.linking')
|
||||
|
||||
|
||||
class LinkingAgent(BaseAgent):
|
||||
agent_type = 'linking'
|
||||
|
||||
async def process(self, job_id: str, payload: dict) -> dict:
|
||||
"""
|
||||
For each document without AI-inferred links:
|
||||
1. Find top-5 semantically similar documents (vector search).
|
||||
2. Insert 'ai-inferred' relations.
|
||||
"""
|
||||
async with self.pool.acquire() as conn:
|
||||
# Documents that have chunks but no ai-inferred relations
|
||||
docs = await conn.fetch(
|
||||
"""
|
||||
SELECT DISTINCT d.id::text, d.title, d.path
|
||||
FROM documents d
|
||||
JOIN chunks c ON c.document_id = d.id
|
||||
WHERE NOT EXISTS (
|
||||
SELECT 1 FROM relations r
|
||||
WHERE r.source_doc_id = d.id AND r.relation_type = 'ai-inferred'
|
||||
)
|
||||
LIMIT 50
|
||||
"""
|
||||
)
|
||||
|
||||
linked = 0
|
||||
for doc in docs:
|
||||
doc_id = doc['id']
|
||||
|
||||
# Find similar docs via average chunk embedding
|
||||
similar = await conn.fetch(
|
||||
"""
|
||||
WITH doc_avg AS (
|
||||
SELECT AVG(embedding) AS avg_emb
|
||||
FROM chunks WHERE document_id = $1::uuid
|
||||
)
|
||||
SELECT d2.id::text AS target_id, d2.path AS target_path,
|
||||
1 - (AVG(c2.embedding) <=> (SELECT avg_emb FROM doc_avg)) AS score
|
||||
FROM chunks c2
|
||||
JOIN documents d2 ON d2.id = c2.document_id
|
||||
WHERE c2.document_id != $1::uuid
|
||||
GROUP BY d2.id, d2.path
|
||||
HAVING 1 - (AVG(c2.embedding) <=> (SELECT avg_emb FROM doc_avg)) > 0.75
|
||||
ORDER BY score DESC
|
||||
LIMIT 5
|
||||
""",
|
||||
doc_id,
|
||||
)
|
||||
|
||||
if not similar:
|
||||
continue
|
||||
|
||||
records = [
|
||||
(doc_id, row['target_path'], row['target_id'], 'ai-inferred')
|
||||
for row in similar
|
||||
]
|
||||
await conn.executemany(
|
||||
"""
|
||||
INSERT INTO relations (source_doc_id, target_path, target_doc_id, relation_type)
|
||||
VALUES ($1::uuid, $2, $3::uuid, $4)
|
||||
ON CONFLICT DO NOTHING
|
||||
""",
|
||||
records,
|
||||
)
|
||||
linked += len(similar)
|
||||
|
||||
return {'documents_processed': len(docs), 'links_created': linked}
|
||||
@ -0,0 +1,92 @@
|
||||
"""
|
||||
main.py — Agent worker entry point. Runs all agents concurrently.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
import asyncpg
|
||||
from pydantic_settings import BaseSettings, SettingsConfigDict
|
||||
|
||||
|
||||
class AgentSettings(BaseSettings):
|
||||
model_config = SettingsConfigDict(env_file='.env', extra='ignore')
|
||||
database_url: str = 'postgresql://brain:brain@postgres:5432/second_brain'
|
||||
ollama_url: str = 'http://ollama:11434'
|
||||
chat_model: str = 'mistral'
|
||||
log_level: str = 'INFO'
|
||||
ingestion_poll: int = 15
|
||||
linking_poll: int = 30
|
||||
tagging_poll: int = 60
|
||||
summarization_poll: int = 120
|
||||
maintenance_poll: int = 3600
|
||||
|
||||
|
||||
def setup_logging(level: str) -> None:
|
||||
logging.basicConfig(
|
||||
level=getattr(logging, level.upper(), logging.INFO),
|
||||
format='%(asctime)s [%(levelname)s] %(name)s: %(message)s',
|
||||
datefmt='%Y-%m-%dT%H:%M:%S',
|
||||
stream=sys.stdout,
|
||||
)
|
||||
|
||||
|
||||
async def main() -> None:
|
||||
settings = AgentSettings()
|
||||
setup_logging(settings.log_level)
|
||||
logger = logging.getLogger('agents')
|
||||
logger.info('Starting agent workers...')
|
||||
|
||||
# Add parent dirs to path for cross-service imports
|
||||
sys.path.insert(0, str(Path(__file__).parent))
|
||||
sys.path.insert(0, str(Path(__file__).parent.parent / 'ingestion-worker'))
|
||||
|
||||
pool = await asyncpg.create_pool(settings.database_url, min_size=2, max_size=10)
|
||||
|
||||
# Import agents after path setup
|
||||
from ingestion.agent import IngestionAgent
|
||||
from linking.agent import LinkingAgent
|
||||
from tagging.agent import TaggingAgent
|
||||
from summarization.agent import SummarizationAgent
|
||||
from maintenance.agent import MaintenanceAgent
|
||||
|
||||
agents_tasks = [
|
||||
asyncio.create_task(
|
||||
IngestionAgent(pool, settings).run_forever(settings.ingestion_poll)
|
||||
),
|
||||
asyncio.create_task(
|
||||
LinkingAgent(pool, settings).run_forever(settings.linking_poll)
|
||||
),
|
||||
asyncio.create_task(
|
||||
TaggingAgent(pool, settings).run_forever(settings.tagging_poll)
|
||||
),
|
||||
asyncio.create_task(
|
||||
SummarizationAgent(pool, settings).run_forever(settings.summarization_poll)
|
||||
),
|
||||
asyncio.create_task(
|
||||
MaintenanceAgent(pool, settings).run_forever(settings.maintenance_poll)
|
||||
),
|
||||
]
|
||||
|
||||
logger.info('All agents running.')
|
||||
|
||||
try:
|
||||
await asyncio.gather(*agents_tasks)
|
||||
except asyncio.CancelledError:
|
||||
pass
|
||||
finally:
|
||||
for task in agents_tasks:
|
||||
task.cancel()
|
||||
await pool.close()
|
||||
logger.info('Agent workers stopped.')
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
try:
|
||||
asyncio.run(main())
|
||||
except KeyboardInterrupt:
|
||||
pass
|
||||
@ -0,0 +1,78 @@
|
||||
"""
|
||||
maintenance/agent.py — Maintenance Agent: detects broken links, orphaned documents, stale content.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from datetime import datetime, timezone, timedelta
|
||||
|
||||
from base_agent import BaseAgent
|
||||
|
||||
logger = logging.getLogger('agent.maintenance')
|
||||
|
||||
|
||||
class MaintenanceAgent(BaseAgent):
|
||||
agent_type = 'maintenance'
|
||||
|
||||
async def process(self, job_id: str, payload: dict) -> dict:
|
||||
report = {}
|
||||
|
||||
async with self.pool.acquire() as conn:
|
||||
# 1. Broken WikiLinks (target_doc_id is NULL but target_path exists)
|
||||
broken_links = await conn.fetchval(
|
||||
"""
|
||||
SELECT COUNT(*) FROM relations
|
||||
WHERE relation_type = 'wikilink' AND target_doc_id IS NULL
|
||||
"""
|
||||
)
|
||||
report['broken_wikilinks'] = broken_links
|
||||
|
||||
# 2. Orphaned documents (no incoming links and no outgoing links)
|
||||
orphans = await conn.fetch(
|
||||
"""
|
||||
SELECT d.id::text, d.title, d.path
|
||||
FROM documents d
|
||||
WHERE NOT EXISTS (
|
||||
SELECT 1 FROM relations r WHERE r.target_doc_id = d.id
|
||||
)
|
||||
AND NOT EXISTS (
|
||||
SELECT 1 FROM relations r WHERE r.source_doc_id = d.id
|
||||
)
|
||||
LIMIT 20
|
||||
"""
|
||||
)
|
||||
report['orphaned_documents'] = len(orphans)
|
||||
report['orphan_paths'] = [r['path'] for r in orphans]
|
||||
|
||||
# 3. Documents not re-indexed in >7 days
|
||||
stale_cutoff = datetime.now(timezone.utc) - timedelta(days=7)
|
||||
stale_count = await conn.fetchval(
|
||||
'SELECT COUNT(*) FROM documents WHERE indexed_at < $1 OR indexed_at IS NULL',
|
||||
stale_cutoff,
|
||||
)
|
||||
report['stale_documents'] = stale_count
|
||||
|
||||
# 4. Documents with chunks but no embeddings
|
||||
missing_embeddings = await conn.fetchval(
|
||||
'SELECT COUNT(*) FROM chunks WHERE embedding IS NULL'
|
||||
)
|
||||
report['chunks_missing_embeddings'] = missing_embeddings
|
||||
|
||||
# 5. Resolve previously broken WikiLinks that now have matching docs
|
||||
resolved = await conn.execute(
|
||||
"""
|
||||
UPDATE relations r
|
||||
SET target_doc_id = d.id
|
||||
FROM documents d
|
||||
WHERE r.target_doc_id IS NULL
|
||||
AND r.relation_type = 'wikilink'
|
||||
AND (d.path LIKE '%' || r.target_path || '%'
|
||||
OR d.title = r.target_path
|
||||
OR r.target_path = ANY(d.aliases))
|
||||
"""
|
||||
)
|
||||
report['wikilinks_resolved'] = int(resolved.split()[-1])
|
||||
|
||||
logger.info('Maintenance report: %s', report)
|
||||
return report
|
||||
@ -0,0 +1,4 @@
|
||||
asyncpg>=0.29.0
|
||||
pydantic-settings>=2.2.0
|
||||
httpx>=0.27.0
|
||||
pgvector>=0.2.5
|
||||
@ -0,0 +1,80 @@
|
||||
"""
|
||||
summarization/agent.py — Summarization Agent: generates summaries for long documents.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import re
|
||||
|
||||
import httpx
|
||||
|
||||
from base_agent import BaseAgent
|
||||
|
||||
logger = logging.getLogger('agent.summarization')
|
||||
|
||||
SUMMARY_PROMPT = """You are a knowledge management assistant.
|
||||
Write a concise 2-4 sentence summary of the following document.
|
||||
The summary should capture the main ideas and be useful for quick reference.
|
||||
Respond with only the summary, no preamble.
|
||||
|
||||
Title: {title}
|
||||
|
||||
Content:
|
||||
{content}
|
||||
|
||||
Summary:"""
|
||||
|
||||
|
||||
class SummarizationAgent(BaseAgent):
|
||||
agent_type = 'summarization'
|
||||
|
||||
async def process(self, job_id: str, payload: dict) -> dict:
|
||||
ollama_url = self.settings.ollama_url
|
||||
model = self.settings.chat_model
|
||||
|
||||
async with self.pool.acquire() as conn:
|
||||
# Long documents that don't have a summary in frontmatter
|
||||
docs = await conn.fetch(
|
||||
"""
|
||||
SELECT id::text, title, content, frontmatter
|
||||
FROM documents
|
||||
WHERE word_count > 500
|
||||
AND (frontmatter->>'summary' IS NULL OR frontmatter->>'summary' = '')
|
||||
LIMIT 10
|
||||
"""
|
||||
)
|
||||
|
||||
summarized = 0
|
||||
for doc in docs:
|
||||
doc_id = doc['id']
|
||||
title = doc['title'] or ''
|
||||
content = (doc['content'] or '')[:4000]
|
||||
|
||||
try:
|
||||
summary = await self._generate_summary(title, content, ollama_url, model)
|
||||
if summary:
|
||||
fm = dict(doc['frontmatter'] or {})
|
||||
fm['summary'] = summary
|
||||
await conn.execute(
|
||||
"UPDATE documents SET frontmatter = $2::jsonb WHERE id = $1::uuid",
|
||||
doc_id, __import__('json').dumps(fm),
|
||||
)
|
||||
summarized += 1
|
||||
logger.debug('Summarized: %s', title)
|
||||
except Exception as exc:
|
||||
logger.warning('Failed to summarize %s: %s', doc_id, exc)
|
||||
|
||||
return {'documents_summarized': summarized}
|
||||
|
||||
async def _generate_summary(
|
||||
self, title: str, content: str, ollama_url: str, model: str
|
||||
) -> str:
|
||||
prompt = SUMMARY_PROMPT.format(title=title, content=content)
|
||||
async with httpx.AsyncClient(timeout=60.0) as client:
|
||||
resp = await client.post(
|
||||
f'{ollama_url.rstrip("/")}/api/generate',
|
||||
json={'model': model, 'prompt': prompt, 'stream': False},
|
||||
)
|
||||
resp.raise_for_status()
|
||||
return resp.json().get('response', '').strip()
|
||||
@ -0,0 +1,87 @@
|
||||
"""
|
||||
tagging/agent.py — Tagging Agent: auto-tags documents using the LLM.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
import re
|
||||
|
||||
import httpx
|
||||
|
||||
from base_agent import BaseAgent
|
||||
|
||||
logger = logging.getLogger('agent.tagging')
|
||||
|
||||
TAG_PROMPT = """You are a knowledge management assistant.
|
||||
Given the following document, suggest 3-7 relevant tags.
|
||||
Tags should be lowercase, hyphen-separated, single-concept keywords.
|
||||
Respond ONLY with a JSON array of strings. Example: ["machine-learning", "python", "transformers"]
|
||||
|
||||
Document title: {title}
|
||||
|
||||
Document content (excerpt):
|
||||
{excerpt}
|
||||
|
||||
Tags:"""
|
||||
|
||||
|
||||
class TaggingAgent(BaseAgent):
|
||||
agent_type = 'tagging'
|
||||
|
||||
async def process(self, job_id: str, payload: dict) -> dict:
|
||||
ollama_url = self.settings.ollama_url
|
||||
model = self.settings.chat_model
|
||||
|
||||
async with self.pool.acquire() as conn:
|
||||
# Documents without tags (or with empty tags array)
|
||||
docs = await conn.fetch(
|
||||
"""
|
||||
SELECT id::text, title, content
|
||||
FROM documents
|
||||
WHERE array_length(tags, 1) IS NULL OR array_length(tags, 1) = 0
|
||||
LIMIT 20
|
||||
"""
|
||||
)
|
||||
|
||||
tagged = 0
|
||||
for doc in docs:
|
||||
doc_id = doc['id']
|
||||
title = doc['title'] or ''
|
||||
excerpt = (doc['content'] or '')[:2000]
|
||||
|
||||
try:
|
||||
suggested_tags = await self._suggest_tags(
|
||||
title, excerpt, ollama_url, model
|
||||
)
|
||||
if suggested_tags:
|
||||
await conn.execute(
|
||||
'UPDATE documents SET tags = $2 WHERE id = $1::uuid',
|
||||
doc_id, suggested_tags,
|
||||
)
|
||||
tagged += 1
|
||||
logger.debug('Tagged %s with %s', title, suggested_tags)
|
||||
except Exception as exc:
|
||||
logger.warning('Failed to tag document %s: %s', doc_id, exc)
|
||||
|
||||
return {'documents_tagged': tagged}
|
||||
|
||||
async def _suggest_tags(
|
||||
self, title: str, excerpt: str, ollama_url: str, model: str
|
||||
) -> list[str]:
|
||||
prompt = TAG_PROMPT.format(title=title, excerpt=excerpt)
|
||||
async with httpx.AsyncClient(timeout=30.0) as client:
|
||||
resp = await client.post(
|
||||
f'{ollama_url.rstrip("/")}/api/generate',
|
||||
json={'model': model, 'prompt': prompt, 'stream': False},
|
||||
)
|
||||
resp.raise_for_status()
|
||||
raw = resp.json().get('response', '').strip()
|
||||
|
||||
# Extract JSON array from response
|
||||
match = re.search(r'\[.*?\]', raw, re.DOTALL)
|
||||
if match:
|
||||
tags = json.loads(match.group())
|
||||
return [str(t).lower().strip() for t in tags if t]
|
||||
return []
|
||||
@ -0,0 +1,17 @@
|
||||
FROM python:3.12-slim
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
# System deps
|
||||
RUN apt-get update && apt-get install -y --no-install-recommends \
|
||||
build-essential libpq-dev curl \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
COPY requirements.txt .
|
||||
RUN pip install --no-cache-dir -r requirements.txt
|
||||
|
||||
COPY . .
|
||||
|
||||
ENV PYTHONUNBUFFERED=1
|
||||
|
||||
CMD ["python", "main.py"]
|
||||
@ -0,0 +1,119 @@
|
||||
"""
|
||||
embedder.py — Embedding generation via Ollama or sentence-transformers fallback.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import time
|
||||
from typing import Any
|
||||
|
||||
import httpx
|
||||
import numpy as np
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# Dimensionality per model
|
||||
_MODEL_DIMS: dict[str, int] = {
|
||||
'nomic-embed-text': 768,
|
||||
'all-minilm-l6-v2': 384,
|
||||
'mxbai-embed-large': 1024,
|
||||
}
|
||||
|
||||
|
||||
class OllamaEmbedder:
|
||||
"""Generate embeddings via the Ollama /api/embed endpoint."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
base_url: str = 'http://ollama:11434',
|
||||
model: str = 'nomic-embed-text',
|
||||
timeout: float = 60.0,
|
||||
batch_size: int = 32,
|
||||
) -> None:
|
||||
self.base_url = base_url.rstrip('/')
|
||||
self.model = model
|
||||
self.timeout = timeout
|
||||
self.batch_size = batch_size
|
||||
self.dimensions = _MODEL_DIMS.get(model, 768)
|
||||
|
||||
def embed_batch(self, texts: list[str]) -> list[list[float]]:
|
||||
"""Embed a list of texts, returning a list of float vectors."""
|
||||
all_embeddings: list[list[float]] = []
|
||||
|
||||
for i in range(0, len(texts), self.batch_size):
|
||||
batch = texts[i : i + self.batch_size]
|
||||
embeddings = self._call_ollama(batch)
|
||||
all_embeddings.extend(embeddings)
|
||||
|
||||
return all_embeddings
|
||||
|
||||
def embed_single(self, text: str) -> list[float]:
|
||||
return self.embed_batch([text])[0]
|
||||
|
||||
def _call_ollama(self, texts: list[str], retries: int = 3) -> list[list[float]]:
|
||||
url = f'{self.base_url}/api/embed'
|
||||
payload: dict[str, Any] = {'model': self.model, 'input': texts}
|
||||
|
||||
for attempt in range(1, retries + 1):
|
||||
try:
|
||||
with httpx.Client(timeout=self.timeout) as client:
|
||||
resp = client.post(url, json=payload)
|
||||
resp.raise_for_status()
|
||||
data = resp.json()
|
||||
return data['embeddings']
|
||||
except (httpx.HTTPError, KeyError) as exc:
|
||||
logger.warning('Ollama embed attempt %d/%d failed: %s', attempt, retries, exc)
|
||||
if attempt < retries:
|
||||
time.sleep(2 ** attempt) # exponential backoff
|
||||
else:
|
||||
raise
|
||||
|
||||
|
||||
class SentenceTransformerEmbedder:
|
||||
"""Local fallback embedder using sentence-transformers."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
model_name: str = 'all-MiniLM-L6-v2',
|
||||
batch_size: int = 32,
|
||||
) -> None:
|
||||
# Lazy import so the module loads even if not installed
|
||||
try:
|
||||
from sentence_transformers import SentenceTransformer # type: ignore
|
||||
except ImportError as exc:
|
||||
raise ImportError(
|
||||
'sentence-transformers is required for the local fallback embedder. '
|
||||
'Install it with: pip install sentence-transformers'
|
||||
) from exc
|
||||
|
||||
logger.info('Loading sentence-transformer model: %s', model_name)
|
||||
self._model = SentenceTransformer(model_name)
|
||||
self.batch_size = batch_size
|
||||
self.dimensions = self._model.get_sentence_embedding_dimension()
|
||||
|
||||
def embed_batch(self, texts: list[str]) -> list[list[float]]:
|
||||
vectors = self._model.encode(
|
||||
texts,
|
||||
batch_size=self.batch_size,
|
||||
show_progress_bar=False,
|
||||
normalize_embeddings=True,
|
||||
)
|
||||
return [v.tolist() for v in vectors]
|
||||
|
||||
def embed_single(self, text: str) -> list[float]:
|
||||
return self.embed_batch([text])[0]
|
||||
|
||||
|
||||
def get_embedder(
|
||||
provider: str = 'ollama',
|
||||
ollama_url: str = 'http://ollama:11434',
|
||||
model: str = 'nomic-embed-text',
|
||||
) -> OllamaEmbedder | SentenceTransformerEmbedder:
|
||||
"""Factory function returning the configured embedder."""
|
||||
if provider == 'ollama':
|
||||
return OllamaEmbedder(base_url=ollama_url, model=model)
|
||||
elif provider == 'sentence_transformers':
|
||||
return SentenceTransformerEmbedder(model_name=model)
|
||||
else:
|
||||
raise ValueError(f'Unknown embedding provider: {provider!r}')
|
||||
@ -0,0 +1,142 @@
|
||||
"""
|
||||
indexer.py — Upserts parsed documents and embeddings into PostgreSQL.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
import logging
|
||||
from typing import Any
|
||||
|
||||
import asyncpg
|
||||
|
||||
from chunker import Chunk
|
||||
from parser import ParsedDocument
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def sha256(text: str) -> str:
|
||||
return hashlib.sha256(text.encode('utf-8')).hexdigest()
|
||||
|
||||
|
||||
async def upsert_document(
|
||||
conn: asyncpg.Connection,
|
||||
doc: ParsedDocument,
|
||||
chunks: list[Chunk],
|
||||
embeddings: list[list[float]],
|
||||
) -> str:
|
||||
"""
|
||||
Upsert a document and its chunks atomically.
|
||||
|
||||
Returns the document UUID.
|
||||
"""
|
||||
content_hash = sha256(doc.content_raw)
|
||||
|
||||
async with conn.transaction():
|
||||
# ---- Upsert document ----
|
||||
row = await conn.fetchrow(
|
||||
"""
|
||||
INSERT INTO documents
|
||||
(path, title, content, content_hash, frontmatter, tags, aliases, word_count, indexed_at)
|
||||
VALUES ($1, $2, $3, $4, $5::jsonb, $6, $7, $8, now())
|
||||
ON CONFLICT (path) DO UPDATE SET
|
||||
title = EXCLUDED.title,
|
||||
content = EXCLUDED.content,
|
||||
content_hash = EXCLUDED.content_hash,
|
||||
frontmatter = EXCLUDED.frontmatter,
|
||||
tags = EXCLUDED.tags,
|
||||
aliases = EXCLUDED.aliases,
|
||||
word_count = EXCLUDED.word_count,
|
||||
indexed_at = now()
|
||||
RETURNING id, (xmax = 0) AS inserted
|
||||
""",
|
||||
doc.path,
|
||||
doc.title,
|
||||
doc.content_raw,
|
||||
content_hash,
|
||||
json.dumps(doc.frontmatter),
|
||||
doc.tags,
|
||||
doc.aliases,
|
||||
doc.word_count,
|
||||
)
|
||||
doc_id: str = str(row['id'])
|
||||
is_new = row['inserted']
|
||||
logger.info('%s document %s (%s)', 'Inserted' if is_new else 'Updated', doc.path, doc_id)
|
||||
|
||||
# ---- Delete stale chunks ----
|
||||
await conn.execute('DELETE FROM chunks WHERE document_id = $1', row['id'])
|
||||
|
||||
# ---- Insert chunks + embeddings ----
|
||||
chunk_records = []
|
||||
for chunk, embedding in zip(chunks, embeddings):
|
||||
chunk_records.append((
|
||||
row['id'],
|
||||
chunk.chunk_index,
|
||||
chunk.content,
|
||||
chunk.token_count,
|
||||
embedding,
|
||||
json.dumps(chunk.metadata),
|
||||
))
|
||||
|
||||
await conn.executemany(
|
||||
"""
|
||||
INSERT INTO chunks (document_id, chunk_index, content, token_count, embedding, metadata)
|
||||
VALUES ($1, $2, $3, $4, $5::vector, $6::jsonb)
|
||||
""",
|
||||
chunk_records,
|
||||
)
|
||||
logger.debug('Upserted %d chunks for document %s', len(chunk_records), doc.path)
|
||||
|
||||
# ---- Upsert relations (WikiLinks) ----
|
||||
await conn.execute(
|
||||
'DELETE FROM relations WHERE source_doc_id = $1 AND relation_type = $2',
|
||||
row['id'],
|
||||
'wikilink',
|
||||
)
|
||||
if doc.wikilinks:
|
||||
relation_records = [
|
||||
(row['id'], link, 'wikilink')
|
||||
for link in doc.wikilinks
|
||||
]
|
||||
await conn.executemany(
|
||||
"""
|
||||
INSERT INTO relations (source_doc_id, target_path, relation_type)
|
||||
VALUES ($1, $2, $3)
|
||||
""",
|
||||
relation_records,
|
||||
)
|
||||
# Resolve targets that already exist in the vault
|
||||
await conn.execute(
|
||||
"""
|
||||
UPDATE relations r
|
||||
SET target_doc_id = d.id
|
||||
FROM documents d
|
||||
WHERE r.source_doc_id = $1
|
||||
AND r.relation_type = 'wikilink'
|
||||
AND (d.path LIKE '%' || r.target_path || '%'
|
||||
OR d.title = r.target_path
|
||||
OR r.target_path = ANY(d.aliases))
|
||||
""",
|
||||
row['id'],
|
||||
)
|
||||
|
||||
return doc_id
|
||||
|
||||
|
||||
async def document_needs_reindex(conn: asyncpg.Connection, path: str, content_hash: str) -> bool:
|
||||
"""Return True if the document is new or its content hash has changed."""
|
||||
row = await conn.fetchrow(
|
||||
'SELECT content_hash FROM documents WHERE path = $1',
|
||||
path,
|
||||
)
|
||||
if row is None:
|
||||
return True
|
||||
return row['content_hash'] != content_hash
|
||||
|
||||
|
||||
async def delete_document(conn: asyncpg.Connection, path: str) -> None:
|
||||
"""Remove a document and its cascaded chunks/relations."""
|
||||
result = await conn.execute('DELETE FROM documents WHERE path = $1', path)
|
||||
logger.info('Deleted document %s (%s)', path, result)
|
||||
@ -0,0 +1,33 @@
|
||||
"""
|
||||
main.py — Ingestion worker entry point.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
import sys
|
||||
|
||||
from settings import Settings
|
||||
from watcher import run_watcher
|
||||
|
||||
|
||||
def setup_logging(level: str) -> None:
|
||||
logging.basicConfig(
|
||||
level=getattr(logging, level.upper(), logging.INFO),
|
||||
format='%(asctime)s [%(levelname)s] %(name)s: %(message)s',
|
||||
datefmt='%Y-%m-%dT%H:%M:%S',
|
||||
stream=sys.stdout,
|
||||
)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
settings = Settings()
|
||||
setup_logging(settings.log_level)
|
||||
logger = logging.getLogger('ingestion-worker')
|
||||
logger.info('Starting ingestion worker (vault=%s)', settings.vault_path)
|
||||
|
||||
try:
|
||||
asyncio.run(run_watcher(settings))
|
||||
except KeyboardInterrupt:
|
||||
logger.info('Ingestion worker stopped.')
|
||||
@ -0,0 +1,134 @@
|
||||
"""
|
||||
parser.py — Markdown vault document parser.
|
||||
|
||||
Extracts:
|
||||
- YAML frontmatter (title, tags, aliases, date, custom fields)
|
||||
- Plain text content (Markdown stripped)
|
||||
- WikiLinks [[target|alias]] and #tags
|
||||
- Word count
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
from dataclasses import dataclass, field
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import frontmatter # python-frontmatter
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Data classes
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
@dataclass
|
||||
class ParsedDocument:
|
||||
path: str
|
||||
title: str
|
||||
content_raw: str # original markdown
|
||||
content_text: str # plain text (markdown stripped)
|
||||
frontmatter: dict[str, Any]
|
||||
tags: list[str]
|
||||
aliases: list[str]
|
||||
wikilinks: list[str] # resolved link targets
|
||||
word_count: int
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Regexes
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
_WIKILINK_RE = re.compile(r'\[\[([^\[\]|]+)(?:\|[^\[\]]+)?\]\]')
|
||||
_INLINE_TAG_RE = re.compile(r'(?<!\w)#([\w/-]+)')
|
||||
_HEADING_RE = re.compile(r'^#{1,6}\s+', re.MULTILINE)
|
||||
_MARKDOWN_LINK_RE = re.compile(r'!?\[([^\]]*)\]\([^\)]*\)')
|
||||
_CODE_BLOCK_RE = re.compile(r'```[\s\S]*?```|`[^`]+`', re.MULTILINE)
|
||||
_HTML_RE = re.compile(r'<[^>]+>')
|
||||
_HORIZONTAL_RULE_RE = re.compile(r'^[-*_]{3,}\s*$', re.MULTILINE)
|
||||
|
||||
|
||||
def _strip_markdown(text: str) -> str:
|
||||
"""Convert Markdown to plain text (lightweight, no external deps)."""
|
||||
# Remove code blocks first (preserve whitespace context)
|
||||
text = _CODE_BLOCK_RE.sub(' ', text)
|
||||
# Remove headings marker characters
|
||||
text = _HEADING_RE.sub('', text)
|
||||
# Replace Markdown links with their label
|
||||
text = _MARKDOWN_LINK_RE.sub(r'\1', text)
|
||||
# Replace WikiLinks with their display text (or target)
|
||||
text = _WIKILINK_RE.sub(lambda m: m.group(1).split('/')[-1], text)
|
||||
# Remove HTML tags
|
||||
text = _HTML_RE.sub(' ', text)
|
||||
# Remove horizontal rules
|
||||
text = _HORIZONTAL_RULE_RE.sub('', text)
|
||||
# Normalise whitespace
|
||||
text = re.sub(r'\n{3,}', '\n\n', text)
|
||||
return text.strip()
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Parser
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def parse_document(file_path: Path, vault_root: Path) -> ParsedDocument:
|
||||
"""
|
||||
Parse a single Markdown file and return a ``ParsedDocument``.
|
||||
|
||||
Args:
|
||||
file_path: Absolute path to the Markdown file.
|
||||
vault_root: Absolute path to the vault root (used to compute relative path).
|
||||
"""
|
||||
raw_text = file_path.read_text(encoding='utf-8', errors='replace')
|
||||
relative_path = str(file_path.relative_to(vault_root))
|
||||
|
||||
# Parse frontmatter + body
|
||||
post = frontmatter.loads(raw_text)
|
||||
fm: dict[str, Any] = dict(post.metadata)
|
||||
body: str = post.content
|
||||
|
||||
# ---- Title ----
|
||||
title: str = fm.get('title', '')
|
||||
if not title:
|
||||
# Fall back to first H1 heading
|
||||
h1 = re.search(r'^#\s+(.+)$', body, re.MULTILINE)
|
||||
if h1:
|
||||
title = h1.group(1).strip()
|
||||
else:
|
||||
title = file_path.stem
|
||||
|
||||
# ---- Tags ----
|
||||
fm_tags: list[str] = _normalise_list(fm.get('tags', []))
|
||||
inline_tags: list[str] = _INLINE_TAG_RE.findall(body)
|
||||
tags = list(dict.fromkeys([t.lower().lstrip('#') for t in fm_tags + inline_tags]))
|
||||
|
||||
# ---- Aliases ----
|
||||
aliases = _normalise_list(fm.get('aliases', []))
|
||||
|
||||
# ---- WikiLinks ----
|
||||
wikilinks = list(dict.fromkeys(_WIKILINK_RE.findall(body)))
|
||||
|
||||
# ---- Plain text ----
|
||||
content_text = _strip_markdown(body)
|
||||
word_count = len(content_text.split())
|
||||
|
||||
return ParsedDocument(
|
||||
path=relative_path,
|
||||
title=title,
|
||||
content_raw=raw_text,
|
||||
content_text=content_text,
|
||||
frontmatter=fm,
|
||||
tags=tags,
|
||||
aliases=aliases,
|
||||
wikilinks=wikilinks,
|
||||
word_count=word_count,
|
||||
)
|
||||
|
||||
|
||||
def _normalise_list(value: Any) -> list[str]:
|
||||
"""Accept str, list[str], or None and return list[str]."""
|
||||
if not value:
|
||||
return []
|
||||
if isinstance(value, str):
|
||||
return [value]
|
||||
return [str(v) for v in value]
|
||||
@ -0,0 +1,91 @@
|
||||
"""
|
||||
pipeline.py — Orchestrates the full ingestion flow for a single file.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import logging
|
||||
from pathlib import Path
|
||||
|
||||
import asyncpg
|
||||
|
||||
from chunker import chunk_document
|
||||
from embedder import get_embedder
|
||||
from indexer import document_needs_reindex, upsert_document
|
||||
from parser import parse_document
|
||||
from settings import Settings
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def _sha256(text: str) -> str:
|
||||
return hashlib.sha256(text.encode('utf-8')).hexdigest()
|
||||
|
||||
|
||||
async def ingest_file(
|
||||
file_path: Path,
|
||||
settings: Settings,
|
||||
conn: asyncpg.Connection,
|
||||
) -> bool:
|
||||
"""
|
||||
Full ingestion pipeline for a single Markdown file.
|
||||
|
||||
Returns True if the file was (re)indexed, False if skipped.
|
||||
"""
|
||||
vault_root = Path(settings.vault_path)
|
||||
|
||||
if not file_path.exists():
|
||||
logger.warning('File not found, skipping: %s', file_path)
|
||||
return False
|
||||
|
||||
if not file_path.suffix.lower() == '.md':
|
||||
return False
|
||||
|
||||
raw_text = file_path.read_text(encoding='utf-8', errors='replace')
|
||||
content_hash = _sha256(raw_text)
|
||||
relative_path = str(file_path.relative_to(vault_root))
|
||||
|
||||
# Idempotency check
|
||||
if not await document_needs_reindex(conn, relative_path, content_hash):
|
||||
logger.debug('Skipping unchanged file: %s', relative_path)
|
||||
return False
|
||||
|
||||
logger.info('Ingesting %s', relative_path)
|
||||
|
||||
# Parse
|
||||
doc = parse_document(file_path, vault_root)
|
||||
|
||||
# Chunk
|
||||
chunks = chunk_document(
|
||||
doc.content_text,
|
||||
target_tokens=settings.chunk_size,
|
||||
overlap_tokens=settings.chunk_overlap,
|
||||
)
|
||||
|
||||
if not chunks:
|
||||
logger.warning('No chunks generated for %s', relative_path)
|
||||
return False
|
||||
|
||||
# Embed
|
||||
embedder = get_embedder(
|
||||
provider=settings.embedding_provider,
|
||||
ollama_url=settings.ollama_url,
|
||||
model=settings.embedding_model,
|
||||
)
|
||||
texts = [c.content for c in chunks]
|
||||
embeddings = embedder.embed_batch(texts)
|
||||
|
||||
# Validate embedding dimension consistency
|
||||
if embeddings and len(embeddings[0]) != embedder.dimensions:
|
||||
logger.error(
|
||||
'Embedding dimension mismatch: expected %d, got %d',
|
||||
embedder.dimensions,
|
||||
len(embeddings[0]),
|
||||
)
|
||||
raise ValueError('Embedding dimension mismatch')
|
||||
|
||||
# Store
|
||||
doc_id = await upsert_document(conn, doc, chunks, embeddings)
|
||||
logger.info('Indexed %s → %s (%d chunks)', relative_path, doc_id, len(chunks))
|
||||
return True
|
||||
@ -0,0 +1,10 @@
|
||||
watchdog>=4.0.0
|
||||
asyncpg>=0.29.0
|
||||
pgvector>=0.2.5
|
||||
pydantic-settings>=2.2.0
|
||||
httpx>=0.27.0
|
||||
python-frontmatter>=1.1.0
|
||||
markdown-it-py>=3.0.0
|
||||
tiktoken>=0.7.0
|
||||
sentence-transformers>=3.0.0
|
||||
numpy>=1.26.0
|
||||
@ -0,0 +1,33 @@
|
||||
"""
|
||||
settings.py — Configuration for the ingestion worker, loaded from environment variables.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from pydantic_settings import BaseSettings, SettingsConfigDict
|
||||
|
||||
|
||||
class Settings(BaseSettings):
|
||||
model_config = SettingsConfigDict(env_file='.env', extra='ignore')
|
||||
|
||||
# Database
|
||||
database_url: str = 'postgresql://brain:brain@postgres:5432/second_brain'
|
||||
|
||||
# Vault
|
||||
vault_path: str = '/vault'
|
||||
|
||||
# Ollama
|
||||
ollama_url: str = 'http://ollama:11434'
|
||||
|
||||
# Embedding
|
||||
embedding_provider: str = 'ollama' # ollama | sentence_transformers
|
||||
embedding_model: str = 'nomic-embed-text'
|
||||
|
||||
# Chunking
|
||||
chunk_size: int = 700
|
||||
chunk_overlap: int = 70
|
||||
|
||||
# Worker behaviour
|
||||
poll_interval: int = 30 # seconds between fallback polls
|
||||
batch_size: int = 20 # files per ingestion batch
|
||||
log_level: str = 'INFO'
|
||||
@ -0,0 +1,118 @@
|
||||
"""
|
||||
watcher.py — File system watcher that triggers ingestion on vault changes.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
import time
|
||||
from pathlib import Path
|
||||
from queue import Queue
|
||||
from threading import Thread
|
||||
|
||||
import asyncpg
|
||||
from watchdog.events import FileSystemEvent, FileSystemEventHandler
|
||||
from watchdog.observers import Observer
|
||||
|
||||
from pipeline import ingest_file
|
||||
from settings import Settings
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class VaultEventHandler(FileSystemEventHandler):
|
||||
"""Enqueues changed/created Markdown file paths."""
|
||||
|
||||
def __init__(self, queue: Queue) -> None:
|
||||
super().__init__()
|
||||
self._queue = queue
|
||||
|
||||
def on_created(self, event: FileSystemEvent) -> None:
|
||||
self._enqueue(event)
|
||||
|
||||
def on_modified(self, event: FileSystemEvent) -> None:
|
||||
self._enqueue(event)
|
||||
|
||||
def on_deleted(self, event: FileSystemEvent) -> None:
|
||||
if not event.is_directory and str(event.src_path).endswith('.md'):
|
||||
self._queue.put(('delete', event.src_path))
|
||||
|
||||
def _enqueue(self, event: FileSystemEvent) -> None:
|
||||
if not event.is_directory and str(event.src_path).endswith('.md'):
|
||||
self._queue.put(('upsert', event.src_path))
|
||||
|
||||
|
||||
async def process_queue(
|
||||
queue: Queue,
|
||||
settings: Settings,
|
||||
pool: asyncpg.Pool,
|
||||
) -> None:
|
||||
"""Drain the event queue and process each file."""
|
||||
pending: set[str] = set()
|
||||
DEBOUNCE_SECONDS = 2.0
|
||||
|
||||
while True:
|
||||
# Collect all queued events (debounce rapid saves)
|
||||
deadline = time.monotonic() + DEBOUNCE_SECONDS
|
||||
while time.monotonic() < deadline:
|
||||
try:
|
||||
action, path = queue.get_nowait()
|
||||
pending.add((action, path))
|
||||
except Exception:
|
||||
await asyncio.sleep(0.1)
|
||||
|
||||
for action, path in list(pending):
|
||||
try:
|
||||
async with pool.acquire() as conn:
|
||||
if action == 'upsert':
|
||||
await ingest_file(Path(path), settings, conn)
|
||||
elif action == 'delete':
|
||||
from indexer import delete_document
|
||||
relative = str(Path(path).relative_to(Path(settings.vault_path)))
|
||||
await delete_document(conn, relative)
|
||||
except Exception as exc:
|
||||
logger.error('Error processing %s %s: %s', action, path, exc, exc_info=True)
|
||||
|
||||
pending.clear()
|
||||
|
||||
|
||||
async def initial_scan(settings: Settings, pool: asyncpg.Pool) -> None:
|
||||
"""Index all Markdown files in the vault at startup."""
|
||||
vault_path = Path(settings.vault_path)
|
||||
md_files = list(vault_path.rglob('*.md'))
|
||||
logger.info('Initial scan: found %d Markdown files', len(md_files))
|
||||
|
||||
for i, file_path in enumerate(md_files):
|
||||
try:
|
||||
async with pool.acquire() as conn:
|
||||
indexed = await ingest_file(file_path, settings, conn)
|
||||
if indexed:
|
||||
logger.info('[%d/%d] Indexed %s', i + 1, len(md_files), file_path.name)
|
||||
except Exception as exc:
|
||||
logger.error('Failed to index %s: %s', file_path, exc, exc_info=True)
|
||||
|
||||
logger.info('Initial scan complete.')
|
||||
|
||||
|
||||
async def run_watcher(settings: Settings) -> None:
|
||||
"""Entry point: start file watcher + initial scan."""
|
||||
pool = await asyncpg.create_pool(settings.database_url, min_size=2, max_size=10)
|
||||
|
||||
await initial_scan(settings, pool)
|
||||
|
||||
event_queue: Queue = Queue()
|
||||
handler = VaultEventHandler(event_queue)
|
||||
observer = Observer()
|
||||
observer.schedule(handler, settings.vault_path, recursive=True)
|
||||
observer.start()
|
||||
logger.info('Watching vault at %s', settings.vault_path)
|
||||
|
||||
try:
|
||||
await process_queue(event_queue, settings, pool)
|
||||
except asyncio.CancelledError:
|
||||
pass
|
||||
finally:
|
||||
observer.stop()
|
||||
observer.join()
|
||||
await pool.close()
|
||||
@ -0,0 +1,18 @@
|
||||
FROM python:3.12-slim
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
RUN apt-get update && apt-get install -y --no-install-recommends \
|
||||
build-essential libpq-dev curl \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
COPY requirements.txt .
|
||||
RUN pip install --no-cache-dir -r requirements.txt
|
||||
|
||||
COPY . .
|
||||
|
||||
ENV PYTHONUNBUFFERED=1
|
||||
|
||||
EXPOSE 8000
|
||||
|
||||
CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"]
|
||||
@ -0,0 +1,35 @@
|
||||
"""
|
||||
database.py — Async PostgreSQL connection pool (asyncpg).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncpg
|
||||
|
||||
from core.settings import Settings
|
||||
|
||||
_pool: asyncpg.Pool | None = None
|
||||
|
||||
|
||||
async def create_pool(settings: Settings) -> asyncpg.Pool:
|
||||
global _pool
|
||||
_pool = await asyncpg.create_pool(
|
||||
settings.database_url,
|
||||
min_size=settings.db_pool_min,
|
||||
max_size=settings.db_pool_max,
|
||||
command_timeout=60,
|
||||
)
|
||||
return _pool
|
||||
|
||||
|
||||
async def get_pool() -> asyncpg.Pool:
|
||||
if _pool is None:
|
||||
raise RuntimeError('Database pool not initialised')
|
||||
return _pool
|
||||
|
||||
|
||||
async def close_pool() -> None:
|
||||
global _pool
|
||||
if _pool:
|
||||
await _pool.close()
|
||||
_pool = None
|
||||
@ -0,0 +1,39 @@
|
||||
"""
|
||||
settings.py — RAG API configuration loaded from environment variables.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from pydantic_settings import BaseSettings, SettingsConfigDict
|
||||
|
||||
|
||||
class Settings(BaseSettings):
|
||||
model_config = SettingsConfigDict(env_file='.env', extra='ignore')
|
||||
|
||||
# App
|
||||
app_title: str = 'Second Brain RAG API'
|
||||
app_version: str = '1.0.0'
|
||||
log_level: str = 'INFO'
|
||||
|
||||
# Database
|
||||
database_url: str = 'postgresql://brain:brain@postgres:5432/second_brain'
|
||||
db_pool_min: int = 2
|
||||
db_pool_max: int = 20
|
||||
|
||||
# Ollama
|
||||
ollama_url: str = 'http://ollama:11434'
|
||||
embedding_model: str = 'nomic-embed-text'
|
||||
chat_model: str = 'mistral'
|
||||
embedding_dimensions: int = 768
|
||||
|
||||
# Search defaults
|
||||
search_top_k: int = 10
|
||||
search_threshold: float = 0.65
|
||||
rerank_enabled: bool = False
|
||||
|
||||
# CORS (comma-separated origins)
|
||||
cors_origins: str = 'http://localhost:3000'
|
||||
|
||||
@property
|
||||
def cors_origins_list(self) -> list[str]:
|
||||
return [o.strip() for o in self.cors_origins.split(',') if o.strip()]
|
||||
@ -0,0 +1,59 @@
|
||||
"""
|
||||
main.py — FastAPI application entry point for the RAG API.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import sys
|
||||
|
||||
from contextlib import asynccontextmanager
|
||||
from fastapi import FastAPI
|
||||
from fastapi.middleware.cors import CORSMiddleware
|
||||
|
||||
from core.database import create_pool, close_pool
|
||||
from core.settings import Settings
|
||||
from routers import search, chat, documents, index, meta
|
||||
|
||||
# Global settings instance (imported by routers via dependency)
|
||||
app_settings = Settings()
|
||||
|
||||
|
||||
def setup_logging(level: str) -> None:
|
||||
logging.basicConfig(
|
||||
level=getattr(logging, level.upper(), logging.INFO),
|
||||
format='%(asctime)s [%(levelname)s] %(name)s: %(message)s',
|
||||
datefmt='%Y-%m-%dT%H:%M:%S',
|
||||
stream=sys.stdout,
|
||||
)
|
||||
|
||||
|
||||
@asynccontextmanager
|
||||
async def lifespan(app: FastAPI):
|
||||
setup_logging(app_settings.log_level)
|
||||
logging.getLogger('rag-api').info('Starting RAG API v%s', app_settings.app_version)
|
||||
await create_pool(app_settings)
|
||||
yield
|
||||
await close_pool()
|
||||
|
||||
|
||||
app = FastAPI(
|
||||
title=app_settings.app_title,
|
||||
version=app_settings.app_version,
|
||||
lifespan=lifespan,
|
||||
)
|
||||
|
||||
app.add_middleware(
|
||||
CORSMiddleware,
|
||||
allow_origins=app_settings.cors_origins_list,
|
||||
allow_credentials=True,
|
||||
allow_methods=['*'],
|
||||
allow_headers=['*'],
|
||||
)
|
||||
|
||||
# Register routers
|
||||
app.include_router(search.router, prefix='/api/v1')
|
||||
app.include_router(chat.router, prefix='/api/v1')
|
||||
app.include_router(documents.router, prefix='/api/v1')
|
||||
app.include_router(index.router, prefix='/api/v1')
|
||||
app.include_router(meta.router, prefix='/api/v1')
|
||||
@ -0,0 +1,31 @@
|
||||
"""
|
||||
models/requests.py — Pydantic request schemas for the RAG API.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Optional
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
|
||||
class SearchRequest(BaseModel):
|
||||
query: str = Field(..., min_length=1, max_length=2000)
|
||||
limit: int = Field(default=10, ge=1, le=50)
|
||||
threshold: float = Field(default=0.65, ge=0.0, le=1.0)
|
||||
tags: Optional[list[str]] = None
|
||||
hybrid: bool = True
|
||||
|
||||
|
||||
class ChatRequest(BaseModel):
|
||||
message: str = Field(..., min_length=1, max_length=4000)
|
||||
conversation_id: Optional[str] = None
|
||||
context_limit: int = Field(default=5, ge=1, le=20)
|
||||
stream: bool = True
|
||||
|
||||
|
||||
class IndexRequest(BaseModel):
|
||||
path: str = Field(..., description='Relative path of file within the vault')
|
||||
|
||||
|
||||
class ReindexRequest(BaseModel):
|
||||
force: bool = False # If True, reindex even unchanged files
|
||||
@ -0,0 +1,96 @@
|
||||
"""
|
||||
models/responses.py — Pydantic response schemas for the RAG API.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import datetime
|
||||
from typing import Any, Optional
|
||||
from pydantic import BaseModel
|
||||
|
||||
|
||||
class ChunkResult(BaseModel):
|
||||
document_id: str
|
||||
chunk_id: str
|
||||
title: str
|
||||
path: str
|
||||
content: str
|
||||
score: float
|
||||
tags: list[str]
|
||||
highlight: Optional[str] = None
|
||||
|
||||
|
||||
class SearchResponse(BaseModel):
|
||||
results: list[ChunkResult]
|
||||
total: int
|
||||
query_time_ms: float
|
||||
|
||||
|
||||
class DocumentResponse(BaseModel):
|
||||
id: str
|
||||
path: str
|
||||
title: str
|
||||
content: str
|
||||
frontmatter: dict[str, Any]
|
||||
tags: list[str]
|
||||
aliases: list[str]
|
||||
word_count: Optional[int]
|
||||
created_at: datetime
|
||||
updated_at: datetime
|
||||
indexed_at: Optional[datetime]
|
||||
|
||||
|
||||
class RelatedDocument(BaseModel):
|
||||
document_id: str
|
||||
title: str
|
||||
path: str
|
||||
score: float
|
||||
tags: list[str]
|
||||
|
||||
|
||||
class GraphNode(BaseModel):
|
||||
id: str
|
||||
title: str
|
||||
path: str
|
||||
tags: list[str]
|
||||
word_count: Optional[int]
|
||||
|
||||
|
||||
class GraphEdge(BaseModel):
|
||||
source: str
|
||||
target: str
|
||||
relation_type: str
|
||||
label: Optional[str]
|
||||
|
||||
|
||||
class GraphResponse(BaseModel):
|
||||
nodes: list[GraphNode]
|
||||
edges: list[GraphEdge]
|
||||
|
||||
|
||||
class TagCount(BaseModel):
|
||||
tag: str
|
||||
count: int
|
||||
|
||||
|
||||
class StatsResponse(BaseModel):
|
||||
total_documents: int
|
||||
total_chunks: int
|
||||
total_relations: int
|
||||
total_tags: int
|
||||
last_indexed: Optional[datetime]
|
||||
embedding_model: str
|
||||
chat_model: str
|
||||
|
||||
|
||||
class HealthResponse(BaseModel):
|
||||
status: str
|
||||
database: str
|
||||
ollama: str
|
||||
version: str
|
||||
|
||||
|
||||
class JobResponse(BaseModel):
|
||||
job_id: str
|
||||
status: str
|
||||
message: str
|
||||
@ -0,0 +1,9 @@
|
||||
fastapi>=0.111.0
|
||||
uvicorn[standard]>=0.29.0
|
||||
asyncpg>=0.29.0
|
||||
pgvector>=0.2.5
|
||||
pydantic>=2.7.0
|
||||
pydantic-settings>=2.2.0
|
||||
httpx>=0.27.0
|
||||
python-multipart>=0.0.9
|
||||
sse-starlette>=2.1.0
|
||||
@ -0,0 +1,52 @@
|
||||
"""
|
||||
routers/chat.py — /chat endpoint with SSE streaming.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from fastapi import APIRouter, Depends
|
||||
from fastapi.responses import StreamingResponse
|
||||
|
||||
from core.database import get_pool
|
||||
from core.settings import Settings
|
||||
from models.requests import ChatRequest
|
||||
from services.chat import stream_chat
|
||||
from services.embedder import EmbedService
|
||||
from services.retriever import hybrid_search
|
||||
|
||||
router = APIRouter(prefix='/chat', tags=['chat'])
|
||||
|
||||
|
||||
def _get_settings() -> Settings:
|
||||
from main import app_settings
|
||||
return app_settings
|
||||
|
||||
|
||||
@router.post('')
|
||||
async def chat(req: ChatRequest, settings: Settings = Depends(_get_settings)):
|
||||
pool = await get_pool()
|
||||
embedder = EmbedService(settings.ollama_url, settings.embedding_model)
|
||||
embedding = await embedder.embed(req.message)
|
||||
|
||||
async with pool.acquire() as conn:
|
||||
context_chunks, _ = await hybrid_search(
|
||||
conn=conn,
|
||||
query=req.message,
|
||||
embedding=embedding,
|
||||
limit=req.context_limit,
|
||||
threshold=settings.search_threshold,
|
||||
)
|
||||
|
||||
return StreamingResponse(
|
||||
stream_chat(
|
||||
message=req.message,
|
||||
context_chunks=context_chunks,
|
||||
ollama_url=settings.ollama_url,
|
||||
model=settings.chat_model,
|
||||
),
|
||||
media_type='text/event-stream',
|
||||
headers={
|
||||
'Cache-Control': 'no-cache',
|
||||
'X-Accel-Buffering': 'no',
|
||||
},
|
||||
)
|
||||
@ -0,0 +1,67 @@
|
||||
"""
|
||||
routers/documents.py — Document CRUD and graph endpoints.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Optional
|
||||
from fastapi import APIRouter, HTTPException, Depends
|
||||
import asyncpg
|
||||
|
||||
from core.database import get_pool
|
||||
from core.settings import Settings
|
||||
from models.responses import DocumentResponse, GraphResponse, GraphNode, GraphEdge, RelatedDocument, TagCount
|
||||
from services.retriever import get_related
|
||||
|
||||
router = APIRouter(prefix='/document', tags=['documents'])
|
||||
|
||||
|
||||
def _get_settings() -> Settings:
|
||||
from main import app_settings
|
||||
return app_settings
|
||||
|
||||
|
||||
@router.get('/{document_id}', response_model=DocumentResponse)
|
||||
async def get_document(document_id: str):
|
||||
pool = await get_pool()
|
||||
async with pool.acquire() as conn:
|
||||
row = await conn.fetchrow(
|
||||
'SELECT * FROM documents WHERE id = $1::uuid', document_id
|
||||
)
|
||||
if not row:
|
||||
raise HTTPException(status_code=404, detail='Document not found')
|
||||
return _row_to_doc(row)
|
||||
|
||||
|
||||
@router.get('/path/{path:path}', response_model=DocumentResponse)
|
||||
async def get_document_by_path(path: str):
|
||||
pool = await get_pool()
|
||||
async with pool.acquire() as conn:
|
||||
row = await conn.fetchrow('SELECT * FROM documents WHERE path = $1', path)
|
||||
if not row:
|
||||
raise HTTPException(status_code=404, detail='Document not found')
|
||||
return _row_to_doc(row)
|
||||
|
||||
|
||||
@router.get('/{document_id}/related', response_model=list[RelatedDocument])
|
||||
async def related_documents(document_id: str, limit: int = 5):
|
||||
pool = await get_pool()
|
||||
async with pool.acquire() as conn:
|
||||
related = await get_related(conn, document_id, limit=limit)
|
||||
return [RelatedDocument(**r) for r in related]
|
||||
|
||||
|
||||
def _row_to_doc(row: asyncpg.Record) -> DocumentResponse:
|
||||
return DocumentResponse(
|
||||
id=str(row['id']),
|
||||
path=row['path'],
|
||||
title=row['title'] or '',
|
||||
content=row['content'],
|
||||
frontmatter=dict(row['frontmatter'] or {}),
|
||||
tags=list(row['tags'] or []),
|
||||
aliases=list(row['aliases'] or []),
|
||||
word_count=row['word_count'],
|
||||
created_at=row['created_at'],
|
||||
updated_at=row['updated_at'],
|
||||
indexed_at=row['indexed_at'],
|
||||
)
|
||||
@ -0,0 +1,49 @@
|
||||
"""
|
||||
routers/index.py — /index and /reindex endpoints.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import uuid
|
||||
from fastapi import APIRouter, BackgroundTasks, Depends
|
||||
|
||||
from core.database import get_pool
|
||||
from core.settings import Settings
|
||||
from models.requests import IndexRequest, ReindexRequest
|
||||
from models.responses import JobResponse
|
||||
|
||||
router = APIRouter(prefix='/index', tags=['indexing'])
|
||||
|
||||
|
||||
def _get_settings() -> Settings:
|
||||
from main import app_settings
|
||||
return app_settings
|
||||
|
||||
|
||||
async def _enqueue_job(agent_type: str, payload: dict, pool) -> str:
|
||||
job_id = str(uuid.uuid4())
|
||||
async with pool.acquire() as conn:
|
||||
await conn.execute(
|
||||
"""
|
||||
INSERT INTO agent_jobs (id, agent_type, payload)
|
||||
VALUES ($1::uuid, $2, $3::jsonb)
|
||||
""",
|
||||
job_id,
|
||||
agent_type,
|
||||
__import__('json').dumps(payload),
|
||||
)
|
||||
return job_id
|
||||
|
||||
|
||||
@router.post('', response_model=JobResponse)
|
||||
async def index_file(req: IndexRequest, settings: Settings = Depends(_get_settings)):
|
||||
pool = await get_pool()
|
||||
job_id = await _enqueue_job('ingestion', {'path': req.path, 'force': True}, pool)
|
||||
return JobResponse(job_id=job_id, status='pending', message=f'Indexing {req.path}')
|
||||
|
||||
|
||||
@router.post('/reindex', response_model=JobResponse)
|
||||
async def reindex_vault(req: ReindexRequest, settings: Settings = Depends(_get_settings)):
|
||||
pool = await get_pool()
|
||||
job_id = await _enqueue_job('ingestion', {'reindex_all': True, 'force': req.force}, pool)
|
||||
return JobResponse(job_id=job_id, status='pending', message='Full vault reindex queued')
|
||||
@ -0,0 +1,129 @@
|
||||
"""
|
||||
routers/meta.py — /health, /stats, /tags, /graph endpoints.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from fastapi import APIRouter
|
||||
import httpx
|
||||
|
||||
from core.database import get_pool
|
||||
from core.settings import Settings
|
||||
from models.responses import HealthResponse, StatsResponse, TagCount, GraphResponse, GraphNode, GraphEdge
|
||||
|
||||
router = APIRouter(tags=['meta'])
|
||||
|
||||
|
||||
def _get_settings() -> Settings:
|
||||
from main import app_settings
|
||||
return app_settings
|
||||
|
||||
|
||||
@router.get('/health', response_model=HealthResponse)
|
||||
async def health():
|
||||
settings = _get_settings()
|
||||
db_status = 'ok'
|
||||
ollama_status = 'ok'
|
||||
|
||||
try:
|
||||
pool = await get_pool()
|
||||
async with pool.acquire() as conn:
|
||||
await conn.fetchval('SELECT 1')
|
||||
except Exception:
|
||||
db_status = 'error'
|
||||
|
||||
try:
|
||||
async with httpx.AsyncClient(timeout=5.0) as client:
|
||||
resp = await client.get(f'{settings.ollama_url}/api/tags')
|
||||
if resp.status_code != 200:
|
||||
ollama_status = 'error'
|
||||
except Exception:
|
||||
ollama_status = 'unavailable'
|
||||
|
||||
overall = 'ok' if db_status == 'ok' else 'degraded'
|
||||
return HealthResponse(
|
||||
status=overall,
|
||||
database=db_status,
|
||||
ollama=ollama_status,
|
||||
version=settings.app_version,
|
||||
)
|
||||
|
||||
|
||||
@router.get('/stats', response_model=StatsResponse)
|
||||
async def stats():
|
||||
settings = _get_settings()
|
||||
pool = await get_pool()
|
||||
async with pool.acquire() as conn:
|
||||
docs = await conn.fetchval('SELECT COUNT(*) FROM documents')
|
||||
chunks = await conn.fetchval('SELECT COUNT(*) FROM chunks')
|
||||
relations = await conn.fetchval('SELECT COUNT(*) FROM relations')
|
||||
tags_count = await conn.fetchval(
|
||||
"SELECT COUNT(DISTINCT tag) FROM documents, unnest(tags) AS tag"
|
||||
)
|
||||
last_indexed = await conn.fetchval(
|
||||
'SELECT MAX(indexed_at) FROM documents'
|
||||
)
|
||||
return StatsResponse(
|
||||
total_documents=docs or 0,
|
||||
total_chunks=chunks or 0,
|
||||
total_relations=relations or 0,
|
||||
total_tags=tags_count or 0,
|
||||
last_indexed=last_indexed,
|
||||
embedding_model=settings.embedding_model,
|
||||
chat_model=settings.chat_model,
|
||||
)
|
||||
|
||||
|
||||
@router.get('/tags', response_model=list[TagCount])
|
||||
async def list_tags():
|
||||
pool = await get_pool()
|
||||
async with pool.acquire() as conn:
|
||||
rows = await conn.fetch(
|
||||
"""
|
||||
SELECT tag, COUNT(*) AS count
|
||||
FROM documents, unnest(tags) AS tag
|
||||
GROUP BY tag
|
||||
ORDER BY count DESC, tag
|
||||
"""
|
||||
)
|
||||
return [TagCount(tag=row['tag'], count=row['count']) for row in rows]
|
||||
|
||||
|
||||
@router.get('/graph', response_model=GraphResponse)
|
||||
async def knowledge_graph(limit: int = 200):
|
||||
pool = await get_pool()
|
||||
async with pool.acquire() as conn:
|
||||
doc_rows = await conn.fetch(
|
||||
'SELECT id, title, path, tags, word_count FROM documents LIMIT $1',
|
||||
limit,
|
||||
)
|
||||
rel_rows = await conn.fetch(
|
||||
"""
|
||||
SELECT r.source_doc_id::text, r.target_doc_id::text, r.relation_type, r.label
|
||||
FROM relations r
|
||||
WHERE r.target_doc_id IS NOT NULL
|
||||
LIMIT $1
|
||||
""",
|
||||
limit * 3,
|
||||
)
|
||||
|
||||
nodes = [
|
||||
GraphNode(
|
||||
id=str(row['id']),
|
||||
title=row['title'] or '',
|
||||
path=row['path'],
|
||||
tags=list(row['tags'] or []),
|
||||
word_count=row['word_count'],
|
||||
)
|
||||
for row in doc_rows
|
||||
]
|
||||
edges = [
|
||||
GraphEdge(
|
||||
source=row['source_doc_id'],
|
||||
target=row['target_doc_id'],
|
||||
relation_type=row['relation_type'],
|
||||
label=row['label'],
|
||||
)
|
||||
for row in rel_rows
|
||||
]
|
||||
return GraphResponse(nodes=nodes, edges=edges)
|
||||
@ -0,0 +1,43 @@
|
||||
"""
|
||||
routers/search.py — /search endpoint.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import time
|
||||
|
||||
from fastapi import APIRouter, Depends
|
||||
from fastapi.responses import JSONResponse
|
||||
|
||||
from core.database import get_pool
|
||||
from models.requests import SearchRequest
|
||||
from models.responses import SearchResponse
|
||||
from services.embedder import EmbedService
|
||||
from services.retriever import hybrid_search
|
||||
from core.settings import Settings
|
||||
|
||||
router = APIRouter(prefix='/search', tags=['search'])
|
||||
|
||||
|
||||
def _get_settings() -> Settings:
|
||||
from main import app_settings
|
||||
return app_settings
|
||||
|
||||
|
||||
@router.post('', response_model=SearchResponse)
|
||||
async def search(req: SearchRequest, settings: Settings = Depends(_get_settings)):
|
||||
pool = await get_pool()
|
||||
embedder = EmbedService(settings.ollama_url, settings.embedding_model)
|
||||
embedding = await embedder.embed(req.query)
|
||||
|
||||
async with pool.acquire() as conn:
|
||||
results, elapsed = await hybrid_search(
|
||||
conn=conn,
|
||||
query=req.query,
|
||||
embedding=embedding,
|
||||
limit=req.limit,
|
||||
threshold=req.threshold,
|
||||
tags=req.tags,
|
||||
)
|
||||
|
||||
return SearchResponse(results=results, total=len(results), query_time_ms=elapsed)
|
||||
@ -0,0 +1,87 @@
|
||||
"""
|
||||
services/chat.py — RAG chat: retrieves context, streams LLM response.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
from typing import AsyncIterator
|
||||
|
||||
import httpx
|
||||
|
||||
from models.responses import ChunkResult
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
SYSTEM_PROMPT = """You are a knowledgeable assistant with access to the user's personal knowledge base (Second Brain).
|
||||
Answer questions based on the provided context documents.
|
||||
Always cite which documents you drew information from using the format [Document Title].
|
||||
If the context doesn't contain enough information, say so honestly rather than fabricating answers.
|
||||
Be concise and precise."""
|
||||
|
||||
|
||||
async def stream_chat(
|
||||
message: str,
|
||||
context_chunks: list[ChunkResult],
|
||||
ollama_url: str,
|
||||
model: str,
|
||||
) -> AsyncIterator[str]:
|
||||
"""
|
||||
Stream a chat response via Ollama using the retrieved context.
|
||||
|
||||
Yields Server-Sent Events (SSE) formatted strings.
|
||||
"""
|
||||
# Build context block
|
||||
context_parts = []
|
||||
for i, chunk in enumerate(context_chunks, 1):
|
||||
context_parts.append(
|
||||
f'[{i}] **{chunk.title}** (path: {chunk.path})\n{chunk.content}'
|
||||
)
|
||||
context_text = '\n\n---\n\n'.join(context_parts)
|
||||
|
||||
prompt = f"""Context from knowledge base:
|
||||
|
||||
{context_text}
|
||||
|
||||
---
|
||||
|
||||
User question: {message}
|
||||
|
||||
Answer based on the above context:"""
|
||||
|
||||
url = f'{ollama_url.rstrip("/")}/api/chat'
|
||||
payload = {
|
||||
'model': model,
|
||||
'stream': True,
|
||||
'messages': [
|
||||
{'role': 'system', 'content': SYSTEM_PROMPT},
|
||||
{'role': 'user', 'content': prompt},
|
||||
],
|
||||
}
|
||||
|
||||
# Yield sources first
|
||||
sources = [
|
||||
{'title': c.title, 'path': c.path, 'score': c.score}
|
||||
for c in context_chunks
|
||||
]
|
||||
yield f'data: {json.dumps({"type": "sources", "sources": sources})}\n\n'
|
||||
|
||||
# Stream tokens
|
||||
async with httpx.AsyncClient(timeout=120.0) as client:
|
||||
async with client.stream('POST', url, json=payload) as resp:
|
||||
resp.raise_for_status()
|
||||
async for line in resp.aiter_lines():
|
||||
if not line.strip():
|
||||
continue
|
||||
try:
|
||||
chunk_data = json.loads(line)
|
||||
token = chunk_data.get('message', {}).get('content', '')
|
||||
if token:
|
||||
yield f'data: {json.dumps({"type": "token", "token": token})}\n\n'
|
||||
if chunk_data.get('done', False):
|
||||
break
|
||||
except json.JSONDecodeError:
|
||||
continue
|
||||
|
||||
yield f'data: {json.dumps({"type": "done"})}\n\n'
|
||||
@ -0,0 +1,31 @@
|
||||
"""
|
||||
services/embedder.py — Thin async wrapper around Ollama embeddings for the API.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import time
|
||||
|
||||
import httpx
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class EmbedService:
|
||||
def __init__(self, ollama_url: str, model: str, timeout: float = 30.0) -> None:
|
||||
self._url = f'{ollama_url.rstrip("/")}/api/embed'
|
||||
self._model = model
|
||||
self._timeout = timeout
|
||||
|
||||
async def embed(self, text: str) -> list[float]:
|
||||
return (await self.embed_batch([text]))[0]
|
||||
|
||||
async def embed_batch(self, texts: list[str]) -> list[list[float]]:
|
||||
async with httpx.AsyncClient(timeout=self._timeout) as client:
|
||||
resp = await client.post(
|
||||
self._url,
|
||||
json={'model': self._model, 'input': texts},
|
||||
)
|
||||
resp.raise_for_status()
|
||||
return resp.json()['embeddings']
|
||||
@ -0,0 +1,160 @@
|
||||
"""
|
||||
services/retriever.py — Hybrid vector + full-text search against PostgreSQL.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import time
|
||||
from typing import Optional
|
||||
|
||||
import asyncpg
|
||||
|
||||
from models.responses import ChunkResult
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
async def hybrid_search(
|
||||
conn: asyncpg.Connection,
|
||||
query: str,
|
||||
embedding: list[float],
|
||||
limit: int = 10,
|
||||
threshold: float = 0.65,
|
||||
tags: Optional[list[str]] = None,
|
||||
) -> tuple[list[ChunkResult], float]:
|
||||
"""
|
||||
Hybrid search: vector similarity + full-text search, merged by RRF.
|
||||
|
||||
Returns (results, query_time_ms).
|
||||
"""
|
||||
start = time.monotonic()
|
||||
|
||||
tag_filter = ''
|
||||
params: list = [embedding, query, limit * 2, threshold]
|
||||
|
||||
if tags:
|
||||
tag_filter = 'AND d.tags && $5'
|
||||
params.append(tags)
|
||||
|
||||
# Combined RRF (Reciprocal Rank Fusion) of vector and FTS results
|
||||
sql = f"""
|
||||
WITH vector_results AS (
|
||||
SELECT
|
||||
c.id AS chunk_id,
|
||||
c.document_id,
|
||||
c.content,
|
||||
c.chunk_index,
|
||||
1 - (c.embedding <=> $1::vector) AS vector_score,
|
||||
ROW_NUMBER() OVER (ORDER BY c.embedding <=> $1::vector) AS vector_rank
|
||||
FROM chunks c
|
||||
JOIN documents d ON d.id = c.document_id
|
||||
WHERE 1 - (c.embedding <=> $1::vector) >= $4
|
||||
{tag_filter}
|
||||
ORDER BY c.embedding <=> $1::vector
|
||||
LIMIT $3
|
||||
),
|
||||
fts_results AS (
|
||||
SELECT
|
||||
c.id AS chunk_id,
|
||||
c.document_id,
|
||||
c.content,
|
||||
c.chunk_index,
|
||||
ts_rank_cd(d.fts_vector, plainto_tsquery('english', $2)) AS fts_score,
|
||||
ROW_NUMBER() OVER (
|
||||
ORDER BY ts_rank_cd(d.fts_vector, plainto_tsquery('english', $2)) DESC
|
||||
) AS fts_rank
|
||||
FROM chunks c
|
||||
JOIN documents d ON d.id = c.document_id
|
||||
WHERE d.fts_vector @@ plainto_tsquery('english', $2)
|
||||
{tag_filter}
|
||||
ORDER BY fts_score DESC
|
||||
LIMIT $3
|
||||
),
|
||||
merged AS (
|
||||
SELECT
|
||||
COALESCE(v.chunk_id, f.chunk_id) AS chunk_id,
|
||||
COALESCE(v.document_id, f.document_id) AS document_id,
|
||||
COALESCE(v.content, f.content) AS content,
|
||||
(COALESCE(1.0 / (60 + v.vector_rank), 0) +
|
||||
COALESCE(1.0 / (60 + f.fts_rank), 0)) AS rrf_score,
|
||||
COALESCE(v.vector_score, 0) AS vector_score
|
||||
FROM vector_results v
|
||||
FULL OUTER JOIN fts_results f ON v.chunk_id = f.chunk_id
|
||||
)
|
||||
SELECT
|
||||
m.chunk_id::text,
|
||||
m.document_id::text,
|
||||
m.content,
|
||||
m.rrf_score,
|
||||
m.vector_score,
|
||||
d.title,
|
||||
d.path,
|
||||
d.tags,
|
||||
ts_headline('english', m.content, plainto_tsquery('english', $2),
|
||||
'MaxWords=20, MinWords=10, ShortWord=3') AS highlight
|
||||
FROM merged m
|
||||
JOIN documents d ON d.id = m.document_id
|
||||
ORDER BY m.rrf_score DESC
|
||||
LIMIT $3
|
||||
"""
|
||||
|
||||
rows = await conn.fetch(sql, *params)
|
||||
elapsed_ms = (time.monotonic() - start) * 1000
|
||||
|
||||
results = [
|
||||
ChunkResult(
|
||||
chunk_id=str(row['chunk_id']),
|
||||
document_id=str(row['document_id']),
|
||||
title=row['title'] or '',
|
||||
path=row['path'],
|
||||
content=row['content'],
|
||||
score=round(float(row['rrf_score']), 4),
|
||||
tags=list(row['tags'] or []),
|
||||
highlight=row['highlight'],
|
||||
)
|
||||
for row in rows
|
||||
]
|
||||
|
||||
return results[:limit], round(elapsed_ms, 2)
|
||||
|
||||
|
||||
async def get_related(
|
||||
conn: asyncpg.Connection,
|
||||
document_id: str,
|
||||
limit: int = 5,
|
||||
) -> list[dict]:
|
||||
"""Find documents related to the given document via average chunk embedding."""
|
||||
rows = await conn.fetch(
|
||||
"""
|
||||
WITH doc_embedding AS (
|
||||
SELECT AVG(embedding) AS avg_emb
|
||||
FROM chunks
|
||||
WHERE document_id = $1::uuid
|
||||
)
|
||||
SELECT
|
||||
d.id::text,
|
||||
d.title,
|
||||
d.path,
|
||||
d.tags,
|
||||
1 - (AVG(c.embedding) <=> (SELECT avg_emb FROM doc_embedding)) AS score
|
||||
FROM chunks c
|
||||
JOIN documents d ON d.id = c.document_id
|
||||
WHERE c.document_id != $1::uuid
|
||||
GROUP BY d.id, d.title, d.path, d.tags
|
||||
ORDER BY score DESC
|
||||
LIMIT $2
|
||||
""",
|
||||
document_id,
|
||||
limit,
|
||||
)
|
||||
return [
|
||||
{
|
||||
'document_id': row['id'],
|
||||
'title': row['title'] or '',
|
||||
'path': row['path'],
|
||||
'tags': list(row['tags'] or []),
|
||||
'score': round(float(row['score']), 4),
|
||||
}
|
||||
for row in rows
|
||||
]
|
||||
@ -0,0 +1,31 @@
|
||||
FROM node:20-alpine AS base
|
||||
|
||||
FROM base AS deps
|
||||
WORKDIR /app
|
||||
COPY package.json ./
|
||||
RUN npm install --frozen-lockfile || npm install
|
||||
|
||||
FROM base AS builder
|
||||
WORKDIR /app
|
||||
COPY --from=deps /app/node_modules ./node_modules
|
||||
COPY . .
|
||||
ENV NEXT_TELEMETRY_DISABLED=1
|
||||
RUN npm run build
|
||||
|
||||
FROM base AS runner
|
||||
WORKDIR /app
|
||||
ENV NODE_ENV=production
|
||||
ENV NEXT_TELEMETRY_DISABLED=1
|
||||
|
||||
RUN addgroup --system --gid 1001 nodejs && \
|
||||
adduser --system --uid 1001 nextjs
|
||||
|
||||
COPY --from=builder /app/public ./public
|
||||
COPY --from=builder --chown=nextjs:nodejs /app/.next/standalone ./
|
||||
COPY --from=builder --chown=nextjs:nodejs /app/.next/static ./.next/static
|
||||
|
||||
USER nextjs
|
||||
EXPOSE 3000
|
||||
ENV PORT=3000
|
||||
|
||||
CMD ["node", "server.js"]
|
||||
@ -0,0 +1,136 @@
|
||||
'use client';
|
||||
|
||||
import { streamChat } from '@/lib/api';
|
||||
import { useState, useRef, useEffect, useCallback } from 'react';
|
||||
import { Send, Loader2, Bot, User, BookOpen } from 'lucide-react';
|
||||
|
||||
interface Message {
|
||||
role: 'user' | 'assistant';
|
||||
content: string;
|
||||
sources?: { title: string; path: string; score: number }[];
|
||||
}
|
||||
|
||||
export default function ChatPage() {
|
||||
const [messages, setMessages] = useState<Message[]>([]);
|
||||
const [input, setInput] = useState('');
|
||||
const [streaming, setStreaming] = useState(false);
|
||||
const cancelRef = useRef<(() => void) | null>(null);
|
||||
const bottomRef = useRef<HTMLDivElement>(null);
|
||||
|
||||
useEffect(() => {
|
||||
bottomRef.current?.scrollIntoView({ behavior: 'smooth' });
|
||||
}, [messages]);
|
||||
|
||||
const sendMessage = useCallback(async () => {
|
||||
const text = input.trim();
|
||||
if (!text || streaming) return;
|
||||
setInput('');
|
||||
|
||||
const userMsg: Message = { role: 'user', content: text };
|
||||
setMessages((prev) => [...prev, userMsg]);
|
||||
setStreaming(true);
|
||||
|
||||
const assistantMsg: Message = { role: 'assistant', content: '', sources: [] };
|
||||
setMessages((prev) => [...prev, assistantMsg]);
|
||||
|
||||
cancelRef.current = streamChat(
|
||||
text,
|
||||
5,
|
||||
(token) => {
|
||||
setMessages((prev) => {
|
||||
const next = [...prev];
|
||||
const last = next[next.length - 1];
|
||||
next[next.length - 1] = { ...last, content: last.content + token };
|
||||
return next;
|
||||
});
|
||||
},
|
||||
(sources) => {
|
||||
setMessages((prev) => {
|
||||
const next = [...prev];
|
||||
next[next.length - 1] = { ...next[next.length - 1], sources };
|
||||
return next;
|
||||
});
|
||||
},
|
||||
() => setStreaming(false),
|
||||
);
|
||||
}, [input, streaming]);
|
||||
|
||||
const handleKeyDown = (e: React.KeyboardEvent) => {
|
||||
if (e.key === 'Enter' && !e.shiftKey) {
|
||||
e.preventDefault();
|
||||
sendMessage();
|
||||
}
|
||||
};
|
||||
|
||||
return (
|
||||
<div className="max-w-3xl mx-auto flex flex-col h-[calc(100vh-5rem)]">
|
||||
<h1 className="text-2xl font-bold text-slate-900 mb-4">AI Chat</h1>
|
||||
|
||||
<div className="flex-1 overflow-y-auto space-y-4 pr-1 mb-4">
|
||||
{messages.length === 0 && (
|
||||
<div className="text-center text-slate-400 py-16">
|
||||
<Bot size={48} className="mx-auto mb-3 opacity-30" />
|
||||
<p>Ask anything about your knowledge base</p>
|
||||
</div>
|
||||
)}
|
||||
|
||||
{messages.map((msg, i) => (
|
||||
<div key={i} className={`flex gap-3 ${msg.role === 'user' ? 'justify-end' : 'justify-start'}`}>
|
||||
{msg.role === 'assistant' && (
|
||||
<div className="w-8 h-8 rounded-full bg-brain-600 flex items-center justify-center shrink-0 mt-1">
|
||||
<Bot size={16} className="text-white" />
|
||||
</div>
|
||||
)}
|
||||
<div className={`max-w-[80%] ${msg.role === 'user' ? 'order-1' : ''}`}>
|
||||
<div className={`rounded-2xl px-4 py-3 text-sm leading-relaxed whitespace-pre-wrap
|
||||
${msg.role === 'user'
|
||||
? 'bg-brain-600 text-white rounded-tr-none'
|
||||
: 'bg-white border border-slate-200 text-slate-800 rounded-tl-none'
|
||||
}`}>
|
||||
{msg.content}
|
||||
{msg.role === 'assistant' && streaming && i === messages.length - 1 && (
|
||||
<span className="inline-block w-1.5 h-4 bg-brain-500 animate-pulse ml-0.5 rounded" />
|
||||
)}
|
||||
</div>
|
||||
{msg.sources && msg.sources.length > 0 && (
|
||||
<div className="mt-2 flex flex-wrap gap-1">
|
||||
{msg.sources.map((src, si) => (
|
||||
<span key={si} className="text-xs bg-slate-100 text-slate-600 px-2 py-0.5 rounded-full flex items-center gap-1">
|
||||
<BookOpen size={10} />
|
||||
{src.title}
|
||||
</span>
|
||||
))}
|
||||
</div>
|
||||
)}
|
||||
</div>
|
||||
{msg.role === 'user' && (
|
||||
<div className="w-8 h-8 rounded-full bg-slate-200 flex items-center justify-center shrink-0 mt-1">
|
||||
<User size={16} className="text-slate-600" />
|
||||
</div>
|
||||
)}
|
||||
</div>
|
||||
))}
|
||||
<div ref={bottomRef} />
|
||||
</div>
|
||||
|
||||
<div className="flex gap-2">
|
||||
<textarea
|
||||
value={input}
|
||||
onChange={(e) => setInput(e.target.value)}
|
||||
onKeyDown={handleKeyDown}
|
||||
disabled={streaming}
|
||||
placeholder="Ask your second brain... (Enter to send, Shift+Enter for newline)"
|
||||
rows={2}
|
||||
className="flex-1 px-4 py-2.5 border border-slate-300 rounded-xl focus:outline-none focus:ring-2 focus:ring-brain-500 resize-none text-sm disabled:opacity-60"
|
||||
/>
|
||||
<button
|
||||
onClick={sendMessage}
|
||||
disabled={streaming || !input.trim()}
|
||||
className="px-4 bg-brain-600 text-white rounded-xl hover:bg-brain-700 disabled:opacity-50 disabled:cursor-not-allowed flex items-center"
|
||||
>
|
||||
{streaming ? <Loader2 size={20} className="animate-spin" /> : <Send size={20} />}
|
||||
</button>
|
||||
</div>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
@ -0,0 +1,87 @@
|
||||
'use client';
|
||||
|
||||
import { getDocument, Document } from '@/lib/api';
|
||||
import { useEffect, useState } from 'react';
|
||||
import { useParams } from 'next/navigation';
|
||||
import { FileText, Tag, ArrowLeft, Loader2, Calendar } from 'lucide-react';
|
||||
import Link from 'next/link';
|
||||
|
||||
export default function DocumentPage() {
|
||||
const { id } = useParams<{ id: string }>();
|
||||
const [doc, setDoc] = useState<Document | null>(null);
|
||||
const [loading, setLoading] = useState(true);
|
||||
const [error, setError] = useState('');
|
||||
|
||||
useEffect(() => {
|
||||
getDocument(id)
|
||||
.then(setDoc)
|
||||
.catch(() => setError('Document not found'))
|
||||
.finally(() => setLoading(false));
|
||||
}, [id]);
|
||||
|
||||
if (loading) return (
|
||||
<div className="flex justify-center items-center h-40">
|
||||
<Loader2 className="animate-spin text-brain-500" size={32} />
|
||||
</div>
|
||||
);
|
||||
|
||||
if (error || !doc) return (
|
||||
<div className="text-red-600 p-4">{error || 'Document not found'}</div>
|
||||
);
|
||||
|
||||
return (
|
||||
<div className="max-w-3xl mx-auto">
|
||||
<Link href="/search" className="inline-flex items-center gap-1 text-sm text-slate-500 hover:text-brain-600 mb-4">
|
||||
<ArrowLeft size={14} /> Back to Search
|
||||
</Link>
|
||||
|
||||
<div className="bg-white rounded-xl border border-slate-200 p-8">
|
||||
<div className="flex items-start gap-3 mb-4">
|
||||
<FileText className="text-brain-500 mt-1 shrink-0" size={24} />
|
||||
<div>
|
||||
<h1 className="text-2xl font-bold text-slate-900">{doc.title}</h1>
|
||||
<p className="text-sm text-slate-400 mt-1">{doc.path}</p>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div className="flex flex-wrap gap-4 text-sm text-slate-500 mb-6 pb-6 border-b border-slate-100">
|
||||
{doc.word_count && (
|
||||
<span>{doc.word_count.toLocaleString()} words</span>
|
||||
)}
|
||||
{doc.indexed_at && (
|
||||
<span className="flex items-center gap-1">
|
||||
<Calendar size={13} />
|
||||
Indexed {new Date(doc.indexed_at).toLocaleDateString()}
|
||||
</span>
|
||||
)}
|
||||
</div>
|
||||
|
||||
{doc.tags.length > 0 && (
|
||||
<div className="flex flex-wrap gap-2 mb-6">
|
||||
{doc.tags.map((tag) => (
|
||||
<Link
|
||||
key={tag}
|
||||
href={`/tags?tag=${encodeURIComponent(tag)}`}
|
||||
className="text-xs bg-brain-50 text-brain-700 px-2.5 py-1 rounded-full flex items-center gap-1 hover:bg-brain-100"
|
||||
>
|
||||
<Tag size={10} />
|
||||
{tag}
|
||||
</Link>
|
||||
))}
|
||||
</div>
|
||||
)}
|
||||
|
||||
{doc.frontmatter?.summary && (
|
||||
<div className="mb-6 p-4 bg-slate-50 rounded-lg border border-slate-100">
|
||||
<p className="text-sm font-medium text-slate-600 mb-1">Summary</p>
|
||||
<p className="text-sm text-slate-700">{doc.frontmatter.summary as string}</p>
|
||||
</div>
|
||||
)}
|
||||
|
||||
<div className="prose max-w-none text-slate-700 whitespace-pre-wrap font-mono text-sm leading-relaxed bg-slate-50 rounded-lg p-5 overflow-x-auto">
|
||||
{doc.content}
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
@ -0,0 +1,26 @@
|
||||
@tailwind base;
|
||||
@tailwind components;
|
||||
@tailwind utilities;
|
||||
|
||||
:root {
|
||||
--foreground: #0f172a;
|
||||
--background: #f8fafc;
|
||||
}
|
||||
|
||||
body {
|
||||
color: var(--foreground);
|
||||
background: var(--background);
|
||||
font-family: system-ui, -apple-system, sans-serif;
|
||||
}
|
||||
|
||||
/* Markdown content rendering */
|
||||
.prose h1 { @apply text-2xl font-bold mb-3 mt-6; }
|
||||
.prose h2 { @apply text-xl font-semibold mb-2 mt-5; }
|
||||
.prose h3 { @apply text-lg font-semibold mb-2 mt-4; }
|
||||
.prose p { @apply mb-3 leading-relaxed; }
|
||||
.prose ul { @apply list-disc list-inside mb-3; }
|
||||
.prose ol { @apply list-decimal list-inside mb-3; }
|
||||
.prose code { @apply bg-slate-100 px-1 rounded text-sm font-mono; }
|
||||
.prose pre { @apply bg-slate-900 text-slate-100 p-4 rounded-lg overflow-x-auto mb-3; }
|
||||
.prose blockquote { @apply border-l-4 border-brain-500 pl-4 italic text-slate-600 mb-3; }
|
||||
.prose a { @apply text-brain-600 underline hover:text-brain-700; }
|
||||
@ -0,0 +1,21 @@
|
||||
import type { Metadata } from 'next';
|
||||
import './globals.css';
|
||||
import Sidebar from '@/components/layout/Sidebar';
|
||||
|
||||
export const metadata: Metadata = {
|
||||
title: 'Second Brain',
|
||||
description: 'Your AI-powered personal knowledge base',
|
||||
};
|
||||
|
||||
export default function RootLayout({ children }: { children: React.ReactNode }) {
|
||||
return (
|
||||
<html lang="en">
|
||||
<body className="flex h-screen overflow-hidden bg-slate-50">
|
||||
<Sidebar />
|
||||
<main className="flex-1 overflow-y-auto p-6">
|
||||
{children}
|
||||
</main>
|
||||
</body>
|
||||
</html>
|
||||
);
|
||||
}
|
||||
@ -0,0 +1,67 @@
|
||||
'use client';
|
||||
|
||||
import { getStats } from '@/lib/api';
|
||||
import { useEffect, useState } from 'react';
|
||||
import { Brain, FileText, Layers, Tag, Link } from 'lucide-react';
|
||||
|
||||
interface Stats {
|
||||
total_documents: number;
|
||||
total_chunks: number;
|
||||
total_relations: number;
|
||||
total_tags: number;
|
||||
last_indexed: string | null;
|
||||
embedding_model: string;
|
||||
chat_model: string;
|
||||
}
|
||||
|
||||
export default function HomePage() {
|
||||
const [stats, setStats] = useState<Stats | null>(null);
|
||||
|
||||
useEffect(() => {
|
||||
getStats().then(setStats).catch(console.error);
|
||||
}, []);
|
||||
|
||||
const cards = stats ? [
|
||||
{ label: 'Documents', value: stats.total_documents, icon: FileText, color: 'text-blue-600' },
|
||||
{ label: 'Chunks', value: stats.total_chunks, icon: Layers, color: 'text-purple-600' },
|
||||
{ label: 'Links', value: stats.total_relations, icon: Link, color: 'text-green-600' },
|
||||
{ label: 'Tags', value: stats.total_tags, icon: Tag, color: 'text-orange-600' },
|
||||
] : [];
|
||||
|
||||
return (
|
||||
<div className="max-w-4xl mx-auto">
|
||||
<div className="flex items-center gap-3 mb-8">
|
||||
<Brain className="text-brain-600" size={36} />
|
||||
<div>
|
||||
<h1 className="text-3xl font-bold text-slate-900">Second Brain</h1>
|
||||
<p className="text-slate-500">Your AI-powered knowledge base</p>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
{stats && (
|
||||
<>
|
||||
<div className="grid grid-cols-2 md:grid-cols-4 gap-4 mb-8">
|
||||
{cards.map((card) => (
|
||||
<div key={card.label} className="bg-white rounded-xl shadow-sm border border-slate-200 p-5">
|
||||
<card.icon className={`${card.color} mb-2`} size={24} />
|
||||
<p className="text-2xl font-bold text-slate-900">{card.value.toLocaleString()}</p>
|
||||
<p className="text-sm text-slate-500">{card.label}</p>
|
||||
</div>
|
||||
))}
|
||||
</div>
|
||||
<div className="bg-white rounded-xl border border-slate-200 p-5 text-sm text-slate-600">
|
||||
<p><span className="font-medium">Embedding model:</span> {stats.embedding_model}</p>
|
||||
<p><span className="font-medium">Chat model:</span> {stats.chat_model}</p>
|
||||
{stats.last_indexed && (
|
||||
<p><span className="font-medium">Last indexed:</span> {new Date(stats.last_indexed).toLocaleString()}</p>
|
||||
)}
|
||||
</div>
|
||||
</>
|
||||
)}
|
||||
|
||||
{!stats && (
|
||||
<div className="text-slate-400 animate-pulse">Loading stats...</div>
|
||||
)}
|
||||
</div>
|
||||
);
|
||||
}
|
||||
@ -0,0 +1,108 @@
|
||||
'use client';
|
||||
|
||||
import { search, SearchResult } from '@/lib/api';
|
||||
import { useState, useCallback } from 'react';
|
||||
import { Search, Loader2, FileText, Tag } from 'lucide-react';
|
||||
import Link from 'next/link';
|
||||
|
||||
export default function SearchPage() {
|
||||
const [query, setQuery] = useState('');
|
||||
const [results, setResults] = useState<SearchResult[]>([]);
|
||||
const [loading, setLoading] = useState(false);
|
||||
const [queryTime, setQueryTime] = useState<number | null>(null);
|
||||
const [error, setError] = useState('');
|
||||
|
||||
const handleSearch = useCallback(async (e: React.FormEvent) => {
|
||||
e.preventDefault();
|
||||
if (!query.trim()) return;
|
||||
setLoading(true);
|
||||
setError('');
|
||||
try {
|
||||
const res = await search(query.trim());
|
||||
setResults(res.results);
|
||||
setQueryTime(res.query_time_ms);
|
||||
} catch (err) {
|
||||
setError('Search failed. Is the API running?');
|
||||
} finally {
|
||||
setLoading(false);
|
||||
}
|
||||
}, [query]);
|
||||
|
||||
return (
|
||||
<div className="max-w-3xl mx-auto">
|
||||
<h1 className="text-2xl font-bold text-slate-900 mb-6">Search</h1>
|
||||
|
||||
<form onSubmit={handleSearch} className="flex gap-2 mb-6">
|
||||
<div className="flex-1 relative">
|
||||
<Search className="absolute left-3 top-1/2 -translate-y-1/2 text-slate-400" size={18} />
|
||||
<input
|
||||
type="text"
|
||||
value={query}
|
||||
onChange={(e) => setQuery(e.target.value)}
|
||||
placeholder="Search your knowledge base..."
|
||||
className="w-full pl-10 pr-4 py-2.5 border border-slate-300 rounded-lg focus:outline-none focus:ring-2 focus:ring-brain-500 focus:border-transparent"
|
||||
/>
|
||||
</div>
|
||||
<button
|
||||
type="submit"
|
||||
disabled={loading || !query.trim()}
|
||||
className="px-5 py-2.5 bg-brain-600 text-white rounded-lg font-medium hover:bg-brain-700 disabled:opacity-50 disabled:cursor-not-allowed flex items-center gap-2"
|
||||
>
|
||||
{loading && <Loader2 size={16} className="animate-spin" />}
|
||||
Search
|
||||
</button>
|
||||
</form>
|
||||
|
||||
{error && (
|
||||
<div className="mb-4 p-3 bg-red-50 text-red-700 rounded-lg border border-red-200">{error}</div>
|
||||
)}
|
||||
|
||||
{queryTime !== null && results.length > 0 && (
|
||||
<p className="text-sm text-slate-400 mb-4">
|
||||
{results.length} results in {queryTime}ms
|
||||
</p>
|
||||
)}
|
||||
|
||||
<div className="space-y-4">
|
||||
{results.map((result) => (
|
||||
<Link
|
||||
key={result.chunk_id}
|
||||
href={`/documents/${result.document_id}`}
|
||||
className="block bg-white border border-slate-200 rounded-xl p-5 hover:border-brain-400 hover:shadow-sm transition-all"
|
||||
>
|
||||
<div className="flex items-start justify-between gap-3 mb-2">
|
||||
<div className="flex items-center gap-2">
|
||||
<FileText size={16} className="text-brain-500 shrink-0 mt-0.5" />
|
||||
<h3 className="font-semibold text-slate-900">{result.title}</h3>
|
||||
</div>
|
||||
<span className="text-xs text-slate-400 shrink-0 bg-slate-100 px-2 py-0.5 rounded-full">
|
||||
{(result.score * 100).toFixed(0)}%
|
||||
</span>
|
||||
</div>
|
||||
|
||||
{result.highlight && (
|
||||
<p
|
||||
className="text-sm text-slate-600 mb-3 line-clamp-3"
|
||||
dangerouslySetInnerHTML={{ __html: result.highlight }}
|
||||
/>
|
||||
)}
|
||||
|
||||
<div className="flex items-center gap-2 flex-wrap">
|
||||
<span className="text-xs text-slate-400">{result.path}</span>
|
||||
{result.tags.slice(0, 4).map((tag) => (
|
||||
<span key={tag} className="text-xs bg-brain-50 text-brain-700 px-2 py-0.5 rounded-full flex items-center gap-1">
|
||||
<Tag size={10} />
|
||||
{tag}
|
||||
</span>
|
||||
))}
|
||||
</div>
|
||||
</Link>
|
||||
))}
|
||||
|
||||
{results.length === 0 && queryTime !== null && !loading && (
|
||||
<p className="text-center text-slate-400 py-12">No results found for "{query}"</p>
|
||||
)}
|
||||
</div>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
@ -0,0 +1,60 @@
|
||||
'use client';
|
||||
|
||||
import { getTags, TagCount } from '@/lib/api';
|
||||
import { useEffect, useState } from 'react';
|
||||
import { Tag, Search } from 'lucide-react';
|
||||
import Link from 'next/link';
|
||||
|
||||
export default function TagsPage() {
|
||||
const [tags, setTags] = useState<TagCount[]>([]);
|
||||
const [filter, setFilter] = useState('');
|
||||
|
||||
useEffect(() => {
|
||||
getTags().then(setTags).catch(console.error);
|
||||
}, []);
|
||||
|
||||
const filtered = filter
|
||||
? tags.filter((t) => t.tag.includes(filter.toLowerCase()))
|
||||
: tags;
|
||||
|
||||
const maxCount = tags[0]?.count ?? 1;
|
||||
|
||||
return (
|
||||
<div className="max-w-3xl mx-auto">
|
||||
<h1 className="text-2xl font-bold text-slate-900 mb-6">Tags</h1>
|
||||
|
||||
<div className="relative mb-6">
|
||||
<Search className="absolute left-3 top-1/2 -translate-y-1/2 text-slate-400" size={16} />
|
||||
<input
|
||||
type="text"
|
||||
value={filter}
|
||||
onChange={(e) => setFilter(e.target.value)}
|
||||
placeholder="Filter tags..."
|
||||
className="w-full pl-9 pr-4 py-2 border border-slate-300 rounded-lg text-sm focus:outline-none focus:ring-2 focus:ring-brain-500"
|
||||
/>
|
||||
</div>
|
||||
|
||||
<div className="flex flex-wrap gap-2">
|
||||
{filtered.map(({ tag, count }) => {
|
||||
const size = 0.75 + (count / maxCount) * 0.75;
|
||||
return (
|
||||
<Link
|
||||
key={tag}
|
||||
href={`/search?q=${encodeURIComponent(tag)}`}
|
||||
className="inline-flex items-center gap-1.5 bg-white border border-slate-200 rounded-full px-3 py-1.5 hover:border-brain-400 hover:bg-brain-50 transition-colors"
|
||||
style={{ fontSize: `${size}rem` }}
|
||||
>
|
||||
<Tag size={12} className="text-brain-500" />
|
||||
<span className="text-slate-700">{tag}</span>
|
||||
<span className="text-xs text-slate-400 bg-slate-100 px-1.5 rounded-full">{count}</span>
|
||||
</Link>
|
||||
);
|
||||
})}
|
||||
</div>
|
||||
|
||||
{filtered.length === 0 && (
|
||||
<p className="text-slate-400 text-center py-12">No tags found</p>
|
||||
)}
|
||||
</div>
|
||||
);
|
||||
}
|
||||
@ -0,0 +1,49 @@
|
||||
'use client';
|
||||
|
||||
import Link from 'next/link';
|
||||
import { usePathname } from 'next/navigation';
|
||||
import { Brain, Search, MessageSquare, Tag, GitGraph, Home } from 'lucide-react';
|
||||
import clsx from 'clsx';
|
||||
|
||||
const NAV_ITEMS = [
|
||||
{ href: '/', label: 'Home', icon: Home },
|
||||
{ href: '/search', label: 'Search', icon: Search },
|
||||
{ href: '/chat', label: 'Chat', icon: MessageSquare },
|
||||
{ href: '/tags', label: 'Tags', icon: Tag },
|
||||
{ href: '/graph', label: 'Graph', icon: GitGraph },
|
||||
];
|
||||
|
||||
export default function Sidebar() {
|
||||
const pathname = usePathname();
|
||||
|
||||
return (
|
||||
<aside className="w-56 shrink-0 bg-slate-900 text-slate-300 flex flex-col h-screen">
|
||||
<div className="flex items-center gap-2.5 px-5 py-5 border-b border-slate-700">
|
||||
<Brain size={22} className="text-brain-400" />
|
||||
<span className="font-bold text-white text-lg">Second Brain</span>
|
||||
</div>
|
||||
|
||||
<nav className="flex-1 py-4 space-y-1 px-2">
|
||||
{NAV_ITEMS.map(({ href, label, icon: Icon }) => (
|
||||
<Link
|
||||
key={href}
|
||||
href={href}
|
||||
className={clsx(
|
||||
'flex items-center gap-3 px-3 py-2 rounded-lg text-sm font-medium transition-colors',
|
||||
pathname === href
|
||||
? 'bg-brain-700 text-white'
|
||||
: 'hover:bg-slate-800 hover:text-white',
|
||||
)}
|
||||
>
|
||||
<Icon size={16} />
|
||||
{label}
|
||||
</Link>
|
||||
))}
|
||||
</nav>
|
||||
|
||||
<div className="px-4 py-4 border-t border-slate-700 text-xs text-slate-500">
|
||||
AI Second Brain v1.0.0
|
||||
</div>
|
||||
</aside>
|
||||
);
|
||||
}
|
||||
@ -0,0 +1,138 @@
|
||||
/**
|
||||
* lib/api.ts — API client for the RAG backend.
|
||||
*/
|
||||
|
||||
const API_BASE = process.env.NEXT_PUBLIC_API_URL || 'http://localhost:8000';
|
||||
|
||||
export interface SearchResult {
|
||||
document_id: string;
|
||||
chunk_id: string;
|
||||
title: string;
|
||||
path: string;
|
||||
content: string;
|
||||
score: number;
|
||||
tags: string[];
|
||||
highlight?: string;
|
||||
}
|
||||
|
||||
export interface SearchResponse {
|
||||
results: SearchResult[];
|
||||
total: number;
|
||||
query_time_ms: number;
|
||||
}
|
||||
|
||||
export interface Document {
|
||||
id: string;
|
||||
path: string;
|
||||
title: string;
|
||||
content: string;
|
||||
frontmatter: Record<string, unknown>;
|
||||
tags: string[];
|
||||
aliases: string[];
|
||||
word_count: number | null;
|
||||
created_at: string;
|
||||
updated_at: string;
|
||||
indexed_at: string | null;
|
||||
}
|
||||
|
||||
export interface StatsResponse {
|
||||
total_documents: number;
|
||||
total_chunks: number;
|
||||
total_relations: number;
|
||||
total_tags: number;
|
||||
last_indexed: string | null;
|
||||
embedding_model: string;
|
||||
chat_model: string;
|
||||
}
|
||||
|
||||
export interface TagCount {
|
||||
tag: string;
|
||||
count: number;
|
||||
}
|
||||
|
||||
export interface GraphData {
|
||||
nodes: { id: string; title: string; path: string; tags: string[]; word_count: number | null }[];
|
||||
edges: { source: string; target: string; relation_type: string; label: string | null }[];
|
||||
}
|
||||
|
||||
export async function search(query: string, tags?: string[], limit = 10): Promise<SearchResponse> {
|
||||
const res = await fetch(`${API_BASE}/api/v1/search`, {
|
||||
method: 'POST',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify({ query, tags, limit, hybrid: true }),
|
||||
});
|
||||
if (!res.ok) throw new Error(`Search failed: ${res.status}`);
|
||||
return res.json();
|
||||
}
|
||||
|
||||
export async function getDocument(id: string): Promise<Document> {
|
||||
const res = await fetch(`${API_BASE}/api/v1/document/${id}`);
|
||||
if (!res.ok) throw new Error(`Document not found: ${res.status}`);
|
||||
return res.json();
|
||||
}
|
||||
|
||||
export async function getStats(): Promise<StatsResponse> {
|
||||
const res = await fetch(`${API_BASE}/api/v1/stats`);
|
||||
if (!res.ok) throw new Error('Stats fetch failed');
|
||||
return res.json();
|
||||
}
|
||||
|
||||
export async function getTags(): Promise<TagCount[]> {
|
||||
const res = await fetch(`${API_BASE}/api/v1/tags`);
|
||||
if (!res.ok) throw new Error('Tags fetch failed');
|
||||
return res.json();
|
||||
}
|
||||
|
||||
export async function getGraph(limit = 150): Promise<GraphData> {
|
||||
const res = await fetch(`${API_BASE}/api/v1/graph?limit=${limit}`);
|
||||
if (!res.ok) throw new Error('Graph fetch failed');
|
||||
return res.json();
|
||||
}
|
||||
|
||||
export function streamChat(
|
||||
message: string,
|
||||
contextLimit = 5,
|
||||
onToken: (token: string) => void,
|
||||
onSources: (sources: { title: string; path: string; score: number }[]) => void,
|
||||
onDone: () => void,
|
||||
): () => void {
|
||||
const controller = new AbortController();
|
||||
|
||||
(async () => {
|
||||
try {
|
||||
const res = await fetch(`${API_BASE}/api/v1/chat`, {
|
||||
method: 'POST',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify({ message, context_limit: contextLimit, stream: true }),
|
||||
signal: controller.signal,
|
||||
});
|
||||
|
||||
if (!res.ok || !res.body) return;
|
||||
const reader = res.body.getReader();
|
||||
const decoder = new TextDecoder();
|
||||
let buffer = '';
|
||||
|
||||
while (true) {
|
||||
const { done, value } = await reader.read();
|
||||
if (done) break;
|
||||
buffer += decoder.decode(value, { stream: true });
|
||||
const lines = buffer.split('\n');
|
||||
buffer = lines.pop() ?? '';
|
||||
|
||||
for (const line of lines) {
|
||||
if (!line.startsWith('data: ')) continue;
|
||||
try {
|
||||
const data = JSON.parse(line.slice(6));
|
||||
if (data.type === 'token') onToken(data.token);
|
||||
else if (data.type === 'sources') onSources(data.sources);
|
||||
else if (data.type === 'done') onDone();
|
||||
} catch {}
|
||||
}
|
||||
}
|
||||
} catch (err: unknown) {
|
||||
if ((err as Error)?.name !== 'AbortError') console.error('Stream error:', err);
|
||||
}
|
||||
})();
|
||||
|
||||
return () => controller.abort();
|
||||
}
|
||||
@ -0,0 +1,9 @@
|
||||
/** @type {import('next').NextConfig} */
|
||||
const nextConfig = {
|
||||
output: 'standalone',
|
||||
env: {
|
||||
NEXT_PUBLIC_API_URL: process.env.NEXT_PUBLIC_API_URL || 'http://localhost:8000',
|
||||
},
|
||||
};
|
||||
|
||||
module.exports = nextConfig;
|
||||
@ -0,0 +1,30 @@
|
||||
{
|
||||
"name": "second-brain-ui",
|
||||
"version": "1.0.0",
|
||||
"private": true,
|
||||
"scripts": {
|
||||
"dev": "next dev",
|
||||
"build": "next build",
|
||||
"start": "next start",
|
||||
"lint": "next lint"
|
||||
},
|
||||
"dependencies": {
|
||||
"next": "14.2.3",
|
||||
"react": "^18.3.1",
|
||||
"react-dom": "^18.3.1",
|
||||
"lucide-react": "^0.378.0",
|
||||
"clsx": "^2.1.1",
|
||||
"swr": "^2.2.5"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@types/node": "^20",
|
||||
"@types/react": "^18",
|
||||
"@types/react-dom": "^18",
|
||||
"typescript": "^5",
|
||||
"tailwindcss": "^3.4.3",
|
||||
"postcss": "^8",
|
||||
"autoprefixer": "^10",
|
||||
"eslint": "^8",
|
||||
"eslint-config-next": "14.2.3"
|
||||
}
|
||||
}
|
||||
@ -0,0 +1,6 @@
|
||||
module.exports = {
|
||||
plugins: {
|
||||
tailwindcss: {},
|
||||
autoprefixer: {},
|
||||
},
|
||||
};
|
||||
@ -0,0 +1,22 @@
|
||||
/** @type {import('tailwindcss').Config} */
|
||||
module.exports = {
|
||||
content: [
|
||||
'./app/**/*.{js,ts,jsx,tsx,mdx}',
|
||||
'./components/**/*.{js,ts,jsx,tsx,mdx}',
|
||||
],
|
||||
theme: {
|
||||
extend: {
|
||||
colors: {
|
||||
brain: {
|
||||
50: '#f0f4ff',
|
||||
100: '#dde6ff',
|
||||
500: '#6366f1',
|
||||
600: '#4f46e5',
|
||||
700: '#4338ca',
|
||||
900: '#1e1b4b',
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
plugins: [],
|
||||
};
|
||||
@ -0,0 +1,21 @@
|
||||
{
|
||||
"compilerOptions": {
|
||||
"target": "ES2017",
|
||||
"lib": ["dom", "dom.iterable", "esnext"],
|
||||
"allowJs": true,
|
||||
"skipLibCheck": true,
|
||||
"strict": true,
|
||||
"noEmit": true,
|
||||
"esModuleInterop": true,
|
||||
"module": "esnext",
|
||||
"moduleResolution": "bundler",
|
||||
"resolveJsonModule": true,
|
||||
"isolatedModules": true,
|
||||
"jsx": "preserve",
|
||||
"incremental": true,
|
||||
"plugins": [{ "name": "next" }],
|
||||
"paths": { "@/*": ["./*"] }
|
||||
},
|
||||
"include": ["next-env.d.ts", "**/*.ts", "**/*.tsx", ".next/types/**/*.ts"],
|
||||
"exclude": ["node_modules"]
|
||||
}
|
||||
@ -0,0 +1,52 @@
|
||||
---
|
||||
title: Welcome to Your Second Brain
|
||||
tags: [getting-started, meta, second-brain]
|
||||
aliases: [home, start-here]
|
||||
date: 2026-03-05
|
||||
---
|
||||
|
||||
# Welcome to Your Second Brain
|
||||
|
||||
This is your AI-powered personal knowledge management system.
|
||||
|
||||
## What is a Second Brain?
|
||||
|
||||
A **Second Brain** is an external, digital system for capturing, organising, and sharing ideas, insights, and information. The concept was popularised by [[Building a Second Brain]] by Tiago Forte.
|
||||
|
||||
By externalising your thinking, you:
|
||||
- Free up mental RAM for creative work
|
||||
- Build a personal knowledge base that compounds over time
|
||||
- Make connections between ideas you might otherwise miss
|
||||
|
||||
## How This System Works
|
||||
|
||||
Your notes live in this Markdown vault — fully compatible with [[Obsidian]] and [[Logseq]].
|
||||
|
||||
The AI layer:
|
||||
1. **Ingests** every Markdown file automatically
|
||||
2. **Embeds** content into vector space for semantic search
|
||||
3. **Links** related documents autonomously
|
||||
4. **Tags** untagged documents using an LLM
|
||||
5. **Summarises** long documents for quick reference
|
||||
|
||||
## Getting Started
|
||||
|
||||
1. Add Markdown files to this vault
|
||||
2. Use `[[WikiLinks]]` to connect ideas
|
||||
3. Add YAML frontmatter for structured metadata
|
||||
4. Search your knowledge at `http://localhost:3000/search`
|
||||
5. Chat with your notes at `http://localhost:3000/chat`
|
||||
|
||||
## Folder Structure
|
||||
|
||||
- `daily/` — Daily notes and journals
|
||||
- `projects/` — Active projects
|
||||
- `resources/` — Reference material and research
|
||||
- `areas/` — Ongoing areas of responsibility
|
||||
- `templates/` — Note templates
|
||||
|
||||
## Tips
|
||||
|
||||
- Use `#tags` inline or in frontmatter
|
||||
- `[[WikiLinks]]` create automatic knowledge graph edges
|
||||
- The AI agents run in the background — check the graph view after a few minutes
|
||||
@ -0,0 +1,20 @@
|
||||
---
|
||||
title: Daily Note Template
|
||||
tags: [template, daily]
|
||||
date: {{date}}
|
||||
---
|
||||
|
||||
# {{date}}
|
||||
|
||||
## Focus
|
||||
|
||||
-
|
||||
|
||||
## Notes
|
||||
|
||||
## Links
|
||||
|
||||
-
|
||||
|
||||
## Reflections
|
||||
|
||||
Loading…
Reference in new issue