Shamratha G

Problem Statement

• Baseline webcam blink counters rely on Eye Aspect Ratio (EAR) alone — brittle under glasses, poor lighting, and head tilts.
• Counting blinks isn't enough: real eye-strain assessment needs blinks-per-minute, PERCLOS fatigue, screen time, and proximity fused into one signal.
• Break reminders are useless if they only appear when the app has focus.
• The webcam is local hardware, so the ML engine must run on-device while the dashboard stays a modern web app.

System Overview

• MediaPipe tracks 468 facial landmarks in real time; the Eye Aspect Ratio provides the geometric blink signal, with per-user calibration of the blink threshold.
• A MobileNetV2 CNN trained on the ~85k-image MRL Eye Dataset (96.57% test accuracy, ONNX runtime on CPU) is fused with EAR — a blink registers only when geometry and model agree, eliminating false positives from glasses and head tilts.
• Strain engine combines blinks-per-minute, PERCLOS eye-closure, continuous screen time, and screen proximity into a Low/Medium/High strain index with human-readable reasons.
• 20-20-20 break alerts fire an audible beep and desktop notification that reach the user even when the app is in the background.
• FastAPI backend streams live metrics and annotated video over WebSocket, logs sessions to SQLite, and serves the built dashboard; the camera never leaves the local machine.
• React + TypeScript dashboard (hosted on Netlify, talking to the local backend) shows a live strain gauge, blink-rate chart, and session history.

What I Built

• A MobileNetV2 eye-state CNN trained by transfer learning on the MRL Eye Dataset (~84,898 images), reaching 96.57% test accuracy, exported to ONNX for real-time CPU inference.
• EAR + CNN fusion: a blink registers only when the geometry and the model agree, eliminating the false positives EAR alone produces from head tilts and glasses glare.
• MediaPipe 468-point face-landmark tracking with per-user blink-threshold calibration.
• A composite strain scorer combining blinks-per-minute, PERCLOS, continuous screen time, and screen proximity into a Low/Medium/High index with human-readable reasons.
• 20-20-20 break alerts via audible beep and desktop notifications that reach the user even when the app is in the background.
• FastAPI backend streaming live metrics and annotated video over WebSocket, with SQLite session logging.
• React + TypeScript dashboard (Vite, Recharts) with a live strain gauge, blink-rate chart, and session history — hosted on Netlify, talking to the local backend.
• A framework-agnostic CV/ML engine that runs identically in a standalone OpenCV window and inside the FastAPI service.

Key Decisions & Tradeoffs

• Fused learned and geometric signals instead of trusting either alone — the CNN handles the conditions where EAR breaks down.
• ONNX + onnxruntime on CPU so the pipeline runs in real time without a GPU.
• The backend runs on the user's machine and streams annotated video to the browser — the webpage never touches the camera, and the backend answers Chrome's private-network CORS preflight so the Netlify-hosted UI can reach localhost.
• Per-user calibration rather than a universal blink threshold.
• Reproducible training via a Colab notebook: dataset → split → train → evaluate → ONNX export.

Why It Matters

What I'd Improve Next

• Distinguish drowsiness from eye strain using temporal modelling over blink and closure patterns.
• Add low-light environment detection, since poor lighting itself worsens strain.
• Support multi-user profiles with per-user baselines, plus cloud sync for cross-device history.
• Add posture detection (slouching and screen distance over time) as an additional ergonomic signal.
• Integrate with OS do-not-disturb/focus modes and build a mobile companion app.