trying to make gaze detection swift app today

goal: scroll pages w my eyes
maybe cursor too later? idk

what sa good way to click w ur face

saw this for webapps the other day, want it on my mac now

blog: https://blog.andykhau.com/blog/eyesite

check it out: https://eyesite.andykhau.com/

github:

saw this for webapps the other day, want it on my mac now

blog: https://blog.andykhau.com/blog/eyesite

check it out: https://eyesite.andykhau.com/

github: https://github.com/akchro/eyesite

yeah i saw that i need to try it,
on the other hand the acccesibility menus are so hard to find its easier to vibe code your own solution

jk lmao

Doesn’t iOS/mac already have accessibility modes like this?

/scribble inminiapp scribbling and nft minting

I think so, for now not even getting the cursor to follow my eyes properly lol

I just want us to have fun | Cofounder @ Resolve (YC) - spatial tech for construction. FC tinkering: @livecaster @harmonybot | Ask me about /f1 /vr /orange-dao

this what my quant said:

### Core Pipeline (Swift + System Frameworks) ⚡
The performance-critical pipeline requires:

1. **AVCaptureSession** (60fps camera access)
   ```swift
   // Direct hardware access, not exposed to SwiftUI
   let session = AVCaptureSession()
   session.sessionPreset = .hd1280x720
   ```

2. **Vision Framework** (ML inference)
   ```swift
   // Runs on separate queue for real-time performance
   visionQueue.async {
       let handler = VNImageRequestHandler(cvPixelBuffer: buffer)
       try handler.perform([faceLandmarksRequest])
   }
   ```

3. **CGEvent System** (Mouse/Scroll injection)
   ```swift
   // Requires kIOHIDPostEvent privileges
   CGEvent(mouseEventSource: src, 
          mouseType: .mouseMoved,
          mouseCursorPosition: point)?.post(tap: .cghidEventTap)
   ```

4. **Real-time Threading**
   - Camera capture: Dedicated AVCaptureSession queue
   - Vision processing: Background DispatchQueue
   - Kalman filtering: High-priority serial queue
   - UI updates: Main thread via @Published properties

### Why This Split Matters

**Latency Budget** (50ms total):
- Camera capture: 16ms (60fps)
- Vision inference: 20-25ms 
- Coordinate transformation: 2-3ms
- Event posting: 5-7ms
- UI update: 2-3ms

SwiftUI's declarative nature and automatic redraws would add 10-15ms overhead to the critical path, pushing total latency to ~65ms - too high for comfortable pointer control.

no i dont have that, but i think i can re-use their API for basic gaze detection and then I need to do my own layer on top

in AVP itself the gaze tracking is much more potent ofc than just a mac or iphone tho

Have you used AVP? Best gaze tracking I’ve ever used.

right wink right click
left wink left click
close eyes for peace and tranquility