cena .

Overview

Cena is an edge-native visual copilot that combines real-time object detection, hand tracking, and a lightweight vision-language model. It delivers a continuously refreshed natural-language description of what the camera sees without relying on a remote vector database.

Architecture & Vision

The system is organized in three loosely-coupled layers that run at different frame-rates:

1. Geometric Layer (~30 FPS) - Captures webcam frames and feeds them simultaneously to YOLO-World (1200-class object detection) and MediaPipe Hands (21-point hand landmarks).
2. Semantic Layer (~150 ms per inference, every 10 s) - Moondream-2 processes the latest frame and produces a concise natural-language description.
3. Agent Layer - Merges geometric and semantic data, writing two markdown logs: LiveContext.md (short-term window) and PersistentContext.md (append-only audit trail).

Status & Limitations

This repository is an experimental proof‑of‑concept built by a dedicated developer.

Hardware Environment: The system has been actively tested locally on an Apple Mac with an M4 chip (24 GB RAM). Continuous vision-language inference acts as the primary thermal hotspot on this setup.

Software Stability: The core multi‑threaded asynchronous architecture is highly stable, optimized, and does not crash. The pipeline successfully isolates heavy AI models across different execution threads without memory leaks or deadlocks.

Guidance Limitations: While the code runs reliably, the system is not yet ready for autonomous, real‑time manual‑task guidance or safety‑critical applications. Due to current edge VLM latency (≈0.1 FPS), the semantic understanding layer is too slow to provide immediate, split-second corrections while you perform manual tasks. It is designed to track state evolution, not to act as an instantaneous live instructor.

Production Status: The architecture, performance characteristics, and software logic will continuously evolve through ongoing optimization passes (such as conditional VLM inference based on motion cues). While the system is designed to naturally scale as edge‑AI hardware and 1B‑parameter vision models mature, its roadmap relies heavily on active software‑level refinement, meaning performance metrics are subject to change regardless of external model updates.

Memory Management

The dual-file markdown approach stores external context as plain text, giving you:

• Zero indexing latency - writing to a file is orders of magnitude faster than generating embeddings.
• Token-efficient retrieval - downstream agents can grep or stream-read the markdown directly.

Use LiveContext.md for the current snapshot and PersistentContext.md for a durable history.

Installation & Requirements

1. Python 3.10+ (tested on 3.11).
2. Clone the repository - run git clone https://github.com/harciufff/cena.git && cd cena.
3. Virtual environment - python -m venv venv && source venv/bin/activate.
4. Dependencies - pip install -r requirements.txt.
5. Environment variables (see .env.example):
   • HF_TOKEN - HuggingFace access token. Required to securely download the Moondream-2 model weights from the HuggingFace Hub. To get one: Log into HuggingFace → Settings → Access Tokens → Generate a new “Read” token.
   • MOONDREAM_API_KEY - Required by the Moondream library to validate the runtime and suppress initialization warnings (inference remains 100% local on device). To get one: Register for a free account on the Moondream Console and copy your free key from the dashboard.
   • OBSIDIAN_VAULT_PATH - absolute path to the folder where LiveContext.md and PersistentContext.md are stored.
6. Model files - YOLO-World CoreML package and MediaPipe hand-landmarker task will be downloaded automatically on first run (requires internet).
7. Run the application - launch the project with python3 main.py.