feat: Convert to monorepo with Express.js worker service and document processing

[RafiulM]

Summary

This PR converts the repository to a monorepo structure with Turborepo and adds a comprehensive Express.js worker service for document processing with AI-powered embeddings and vector search.

Key Features

• Monorepo Structure: Converted to Turborepo with pnpm workspace
• Express.js Worker Service: Background document processing with BullMQ
• AI Document Processing: PDF and Markdown parsing with OpenAI embeddings
• Vector Search: pgvector-powered semantic similarity search
• File Upload Interface: Drag-and-drop upload with real-time progress tracking
• Document Management: Complete CRUD operations for processed documents

Changes

• Repository Structure:

  • apps/web/ - Next.js application (moved from root)
  • apps/worker/ - New Express.js worker service
  • packages/db/ - Shared Drizzle ORM schema and migrations
  • packages/types/ - Shared TypeScript types
  • packages/tsconfig/ - Shared TypeScript configurations

• New Services:

  • BullMQ queue system with Redis for background job processing
  • Document processing pipeline with PDF/Markdown parsing
  • Vector embeddings using OpenAI's text-embedding-3-small
  • Semantic similarity search with pgvector
  • REST API for file upload, job status, and document management

• Frontend Components:

  • File upload component with drag-and-drop and progress tracking
  • Document search interface with real-time results
  • Job status polling and user feedback
  • Document management page with upload and search tabs

• Infrastructure:

  • Docker Compose with Redis and pgvector-enabled PostgreSQL
  • Environment configuration for all services
  • Proper error handling and logging throughout

Test Plan

• Set up environment variables (OpenAI API key, Redis, PostgreSQL)
• Start services with docker compose up -d
• Install dependencies with pnpm install
• Start development servers with pnpm dev
• Test file upload functionality
• Verify document processing completes successfully
• Test semantic search functionality
• Verify job status polling and real-time updates
• Test document management operations (view, delete)

Technical Details

• Uses Turborepo for efficient monorepo builds
• BullMQ for reliable queue processing with Redis
• pgvector for high-performance vector similarity search
• OpenAI embeddings for text understanding
• React Dropzone for intuitive file uploads
• Real-time progress tracking with WebSocket-style polling

🤖 Generated with Claude Code