Making eyesite

I wanted Apple Vision Pros, but I don’t have $3,500 in my back pocket. So I made Apple Vision Pros at home.

I was interested in making a project that combined computer vision with web design—a website that users could physically interact with. This inspired me to make Eyesite, because who needs a mouse when you have your eyes?

Eye tracking

Luckily, there is already a Javascript library for eye tracking called WebGazer.js. We can achieve decent eye tracking through calibration:

1. Make the user look at a point and click. This maps the current gaze to a point on the screen.
2. Feed the gaze/coordinate mapping into WebGazer to calibrate.
3. Repeat 9x times on the corners, sides, and center to get good mapping data.

I found that it was best to get 5 mappings per point for better eye tracking accuracy.

Calibration in debug mode. The top right shows how WebGazer tracks your eyes and face. The red dot is where it thinks I’m looking.

Website Interaction

Now that we have eye tracking, we can make some cool things with it! I decided to use the user’s gaze as a mouse and have them click with spacebar—kind of like how Apple Vision Pros have you look and pinch. Although I had the main functionality, it was far from finished. There were many considerations with making the experience as smooth and immersive as possible.

The “Invisible” Mouse

Initially, the user could see “where” they were looking at through a red dot.

Main page in debug mode.

This created some problems. First, the red dot was distracting, and users would unconsciously look at it instead of my buttons. Second, the red dot revealed how inaccurate the eye tracking was, which ruined the immersion.

Ultimately, I decided to remove the “eye cursor” and also make the user’s mouse invisible. It made you really feel like you were controlling the website with your eyes rather than moving a mouse around. You can turn on debug mode to see your eye cursor and mouse.

User feedback

Since we don’t have a mouse, we need some way for the user to know they are looking at something. To do this… we track the user’s gaze (how surprising). We hid the eye cursor, but we still have the x and y coordinates of the user’s gaze. Each button component has checks to see if that gaze is within its borders. When the component detects the user is looking at it, it responds with a slight glow and pop.

Large UI

Admittedly, the eye tracking is not the best. You can really see how jittery it is with debug mode on. So I decided to make the UI huge. I also added a screen size restriction so the site is only usable on displays that meet a minimum size threshold (Sorry mobile users! It wouldn’t work on your phone anyway).

The large size of the button accounts for the jitteriness of the eye tracking.

Conclusion

Those were a few details about Eyesite. If you are interested, you can see the source code. Small warning: this project was just a small demo and isn’t a shining example of clean code or best practices.

This was a really fun project to make, and super cool to use too. If you want to make your own computer vision project or improve this one, I encourage you to do so! You can find the project at https://github.com/akchro/eyesite.