trying to make gaze detection swift app today

goal: scroll pages w my eyes
maybe cursor too later? idk

what sa good way to click w ur face

planting causal loops; harvesting reflexivity

Have you used AVP? Best gaze tracking I’ve ever used.

I just want us to have fun | Cofounder @ Resolve (YC) - spatial tech for construction. FC tinkering: @livecaster @harmonybot | Ask me about /f1 /vr /orange-dao

cursor's tldr: 

Let me check what Apple APIs we're using for gaze tracking in the project:
Read file: Sources/GazeCore/GazeTracker.swift

Based on the code, here are the Apple APIs we're using for gaze tracking:

## 1. **AVFoundation** (Camera Capture)
- `AVCaptureSession` - Managing the camera capture session
- `AVCaptureDevice` - Accessing the front-facing camera
- `AVCaptureVideoDataOutput` - Getting video frames from the camera
- Using HD resolution: `.hd1280x720`

## 2. **Vision Framework** (Eye Detection)
- `VNDetectFaceLandmarksRequest` - Detecting face landmarks including eyes
- Using **Revision 3** for best accuracy
- Extracting:
  - `landmarks.leftPupil` 
  - `landmarks.rightPupil`
- Getting normalized points (0-1 coordinates)

## 3. **CoreGraphics**
- `CGPoint` for pupil coordinates
- Coordinate transformations

## Key Points:

**We're NOT using:**
- ARKit eye tracking (requires iPhone/iPad with TrueDepth camera)
- No dedicated eye tracking API (Apple doesn't provide one for Mac)

**Instead, we're using:**
- Vision framework's face landmark detection
- Extracting pupil positions from face landmarks
- This is a computer vision approach, not hardware eye tracking

**Performance targets:**
- 60 FPS camera capture
- < 5% landmark dropout rate
- Processing on background queue (`userInitiated` QoS)

This approach uses the built-in camera and Vision framework to detect facial landmarks, then extracts the pupil positions as a proxy for gaze direction. It's not true eye tracking but rather pupil position detection that gets mapped to screen coordinates through calibration.

i need mardkwon support @dwrgoblok pwease

this what my quant said:

### Core Pipeline (Swift + System Frameworks) ⚡
The performance-critical pipeline requires:

1. **AVCaptureSession** (60fps camera access)
   ```swift
   // Direct hardware access, not exposed to SwiftUI
   let session = AVCaptureSession()
   session.sessionPreset = .hd1280x720
   ```

2. **Vision Framework** (ML inference)
   ```swift
   // Runs on separate queue for real-time performance
   visionQueue.async {
       let handler = VNImageRequestHandler(cvPixelBuffer: buffer)
       try handler.perform([faceLandmarksRequest])
   }
   ```

3. **CGEvent System** (Mouse/Scroll injection)
   ```swift
   // Requires kIOHIDPostEvent privileges
   CGEvent(mouseEventSource: src, 
          mouseType: .mouseMoved,
          mouseCursorPosition: point)?.post(tap: .cghidEventTap)
   ```

4. **Real-time Threading**
   - Camera capture: Dedicated AVCaptureSession queue
   - Vision processing: Background DispatchQueue
   - Kalman filtering: High-priority serial queue
   - UI updates: Main thread via @Published properties

### Why This Split Matters

**Latency Budget** (50ms total):
- Camera capture: 16ms (60fps)
- Vision inference: 20-25ms 
- Coordinate transformation: 2-3ms
- Event posting: 5-7ms
- UI update: 2-3ms

SwiftUI's declarative nature and automatic redraws would add 10-15ms overhead to the critical path, pushing total latency to ~65ms - too high for comfortable pointer control.

no i dont have that, but i think i can re-use their API for basic gaze detection and then I need to do my own layer on top

in AVP itself the gaze tracking is much more potent ofc than just a mac or iphone tho