What is the EyeHarp
The EyeHarp is a gaze-controlled music interface. It aims to allow people with physical disabilities learn and play music. It is open source and free to download and use. It forms part of the master and PhD thesis of Zacharias Vamvakousis, researcher in the Music and Machine Learning Lab, Pompeu Fabra University, Barcelona, Spain. It is implemented on top of the openFremeWorks open source C++ toolkit. It is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License. We believe that accessibility technology should be accessible to everyone. Its cost shouldn’t be an obstacle for the ones that need it.
Here is a video, from Al Jazeera Media Network, showing Joel, a 9-year old boy with cerebral pasly, learning music using the EyeHarp:
What do I need to make it work?
First of all you need an Eye-tracker: a device that knows where you are looking at. Alternatively the EyeHarp can be controlled through any other device that can control the mouse pointer (e.g. a mouse, a head tracking device or a touch-pad). You then need to download and execute the EyeHarp application. It is free and open-source!
How does it sound?
The EyeHarp comes with a simple analog synthesizer that offers some basic sonification. For better sound quality it can be used as a midi controller. Here is a demo video of a performance with the EyeHarp:
Ok, how can I install it on my computer?
- Download and install Visual C++ Redistributable for Visual Studio 2015 x86 (vc_redist.x86.exe). Make sure you select the x86 version!
- Go to https://github.com/zackbam/TheEyeHarp and press on the green button “Clone or download” and download the zip file.
- Unzip it and run the EyeHarpVS.exe in the bin folder. That’t all! You have the EyeHarp running! If you want create a shortcut of the EyeHarpVS.exe in your desktop in order to have easy access.
How can I control the EyeHarp with my eyes?
In order to do that, you need an eye-tracker. An eye tracker is a special camera that along with the dedicated software can detect where you are looking at every moment. It is normally placed under the screen. The EyeHarp has been tested with two commercial eye-trackers: the tobii 4c, and the eyetribe. More tobii devices might works as well with the EyeHarp (any device that communicates through the EyeX server). Unfortunately the eyetribe product has been discontinued, but if you own one you can still use it with the EyeHarp. The tobii 4c is a low cost 159€ eye-tracker with superb performance (subjective personal experience).
If I own a different eye-tracker, can I still control the EyeHarp with the Eyes?
The EyeHarp can be controlled by any device that moves the mouse pointer. If you own a different eye-tracker, you can get control of the mouse pointer using the software provided your eye-tracker, and then you can control the EyeHarp. Of course you have to let the EyeHarp know that you are using mouse input instead of gaze input. This can be configured along with more settings in a configuration file named “eyeharp.txt”. The options available in this file are described in detail later.
If I do not have an eye-tracker, can I still use the EyeHarp?
Yes, it can be controlled by using any pointing device such as a mouse, a trackball, a head-tracker e.t.c. Nevertheless in order to control it with the eyes, it has to be used along with an Eye-tracker.
How do I play music?
The EyeHarp consists of two layers: the step sequencer layer, and the melody layer. In the step sequencer layer the user can construct arpeggios, and in the the melody layer control the chords of the arpeggios and play melodies at the same time. The user can switch between the two layers by looking on the “layer” button at the bottom left of the screen.
The EyeHarp is a diatonic instrument, meaning that the user sets the scale before or while playing. Each button can button can be mapped to different notes. By default the EyeHarp is tuned in C Major scale. But this can be changed through the configuration txt file and through the setup menu.
Melody Layer Step-Sequencer Layer
Step Sequencer Layer
In this layer the user can construct arpeggios. The step sequencer consists of a square grid of notes-buttons, by default 8×8, but the dimension of the grid can be configured in both the configuration file (“eyeharp.txt”) and in the setup menu of the inteface (explained later).
A button in the EyeHarp interface is a circle with a small “focus point” in the center. A click on button is performed by focusing on the focus point of the button for 700 ms. This method is known as dwell-time. Alternatively, the EyeHarp can be configured to performs clicks with left mouse clicks (configured in the configuration file EyeHarp.txt). Buttons in the step sequencer are selected and unselected by “clicking” on them. Another way to delete the last inserted note, is to look at the “delete” button on the top left of the screen. Next to it, there is the “ClearAll” button. When pressed all in the step sequencer notes are deleted.
In a step sequencer, when the EyeHarp is toned for example in C major, the buttons in the first row correspond to the note ‘c’, in the second to ‘d’, in the 3rd to ‘f’ e.t.c. The notes selected in the first column will sound first. If the tempo is set to 60 beats per minute, after 1 second, the notes selected in the second column will sound, followed by the notes in the third e.t.c. When the final column is reached, the arpeggio starts over.
The step sequencer can be configured to be either monophonic (one note per column can be active) or polyphonic (multiple notes per column can be active).
The melody layer consists of a “pie-like” keyboard. At the center of the Pie there is a resting area, in which no button is placed. At the perimeter of the Pie, the notes are placed. Each “slice” of the pie corresponds to a different note or chord. When the user looks at the perimeter of each “slice”, the corresponding note or chord is triggered. This outer area of the slice can be referred as “selection area” of each note. The first note is placed at the position “9 o’clock” of an analog watch and corresponds to the note ‘g-3’. Next to it at around “8 o’clock” is placed the next note, ‘a-3’. Following the counter-clockwise direction, all notes are placed around the pie. The number of slices in the pie can be configured both at the configuration txt file and inside the setup menu of the EyeHarp (explained later). When the “show chords” function is active, the last 7 slices of the interface are used for changing the chords of the arpeggio created in the step-sequencer. When the “show semitones” function is active, between the slices appear cyclic buttons that correspond to the semitones between the notes of the scale.
The black area inside the Pie is an area in which the user can look freely without triggering any event. Let’s call this “neutral area”. Some numbers appear inside that area. The purpose of these numbers is to allow the user think and look which will be the next note to play, without triggering the note while thinking about it. Number 1 corresponds to the note ‘c’ (when we are in C scale), 2 to ‘d’, 3 to ‘e’ e.t.c. The Latin numbers describe the chords. ‘I’ is C, ‘II’ is D e.t.c. Whether the chords are minor or major depends on the set scale.
Each slice of the Pie has 4 focus points, each of them further away from the center of the pie. The further a focus-point is, the louder the note will sound when the user stares at it.
In order to release a note, the user has to either play the next note, or look at the “release area”. The release area is the bright green area just outside the Pie.
In order to play the same note twice the user can either look at the selection area of the note, then in the neutral area and then back to the selection area of the note. This procedure though required too many eye movements. For this reason, a cyclic “repear button” appears in the center of the pie, when the gaze of the user is in the selection area. The same consecutive note can be played again and again just by looking at the selection area of a note, then back to the center of the pie ans so on. Instead, when looking at the area where the number are placed, the “replay button” disappears. and then the user can freely look inside the neutral area without triggering any event.
Optionally, through the configuration file EyeHarp.txt (explained later), an additional release area can be placed between the selection and the Neutral area, as shown in the following image. This feature seems to be suitable for users that use head-tracking instead of eye-tracking. Nevertheless, some users that use eye-tracking also prefer having this feature active.
Watch this youtube video of an older version of the EyeHarp ON A COMPUTER WITH VIDEO ANNOTATIONS ENABLED, in which some explanatory annotations appear during a performance.
How can I select a different sound?
The EyeHarp comes with a built-in basic additive synthesizer. It offers three basic sounds in the melody layer: two theremin-like sounds (theremin and sinusoidal) and one vibraphone-like sound. It can also be used as a midi controller. In that case any instrument can be produced. A midi controller does not produce any sound. Instead it sends control messages to another external synthesizer. Whether EyeHarp works as a midi controller or as a synthesizer, is set in the configuration file or through the setup menu.
The Setup Menu
When the “Setup” button is pressed in the right of the screen, a number of settings appear.
Settings in the melody Layer
On the top left corner the user can select between three sounds: Theremin, Sinusoidal and Vibraphone. By default the Vibraphone is selected. In these cases, the sound will be produced through the EyeHarp’s built-in synthesizer. For a better sound quality, the “toMidi” option can be selected. In that case, the EyeHarp will look for virtual midi ports. The LoopBe1 and the loopMIDI free midi ports are supported. Then a Virtual Instrument Host Application can be used to load sounds and trigger them with the EyeHarp. Later in this manual will be explained how to set up Reaper digital audio workstation (DAW) (free to try) with free Sforzando Aria free sounds. The midi channel each layer sends to can be configured by a repeat control on the left of the Pie.
On the left of the screen, just under the instrument selection section, there are two repeat buttons with up and down commands. By focusing on each for about 700 ms the user can increase/decrease the number of “slices” of the Pie. More slices means more available notes, less slices means a system less prone to errors, as the size of the selection area of the notes will increase.
Still on the left, there are two vertical repeat buttons with up and down commands. They serve to set the octave of the melody layer.
On the bottom/left of the screen appears a switch named “ShowScale”. When active, the user will be able to tune the EyeHarp to different scales: After closing the setup menu, a number of scales to select from will appear on the left of the screen. Alternatively, by adjusting a parameter in the configuration file, the list of preset scales can be replaced by a menu that allows tuning each note of the scale with semitone steps.
On the top right of the screen there are two buttons. On the left there is a button for quitting the application. Quitting actually consists of two steps. Once pressing the quit button, the same button will change position, and it will have to pressed again. This prevents from accidentally exiting the program. On the right of the quit button there is a “Record” button. This allows recording a sequence of chords. This allows to then play a melody on top. Changing the chords and simultaneously playing a melody is also possible, but sometimes it is more convenient to record first the chords and then play a melody or improvise on top. When pressing this button, at the start of the arpeggio, it will turn red. Once it is red, the recording has started. When the loop of chords we wish to record has finished, the record button should be pressed again. Once pressed, the chords will start being played back in a loop. Of course an arpeggio should have been constructed earlier, in the step sequencer layer.
Finally on the bottom right of the screen there is a switch that allows us to switch between the step sequencer and the melody layer.
Settings in the Step Sequencer Layer
On the left of the screen, two repeat buttons named “meter” are placed for increasing-decreasing the size of the grid in the step sequencer. The size of the grid affects the time signature of the arpeggio. For a example a size of 7×7 would mean a rhythm of 7/8. Recommended sizes are 6, 8, 12 and 16.
On the bottom left, there are two more repeat buttons for adjusting the tempo of the step sequencer. On the top right of the screen appear two switches. When the on the right, named “FocusPoints” is active, additional focuspoints will appear on the buttons of the step sequencer. If the accuracy of the eye tracker is not good, these additional focus point can enhance the user select notes more accurately. On its left there is a switch called “Magnify”. When this function is active, a magnification lens appears, a region of the step sequencer is magnified, and then the notes of the magnified region can be selected or unselected. This function is useful when a big grid size is desired, something that would lead into small buttons’s size.
Setting up the EyeHarp with Sforzando Aria free Sounds and Reaper DAW
The EyeHarp in this case will serve as a midi controller. If you have experience with a different Digital Audio Workstation than Reaper, or if you own purchased sounds (p.ex. any Kontakt library), then most probably you already know how to set-up the EyeHarp as a midi controller. All we need to know is that the EyeHarp outputs to midi channels 1 (melody) and 2 (step Sequencer). You might also need to install LoopBe1 virtual midi port. LoopMIDI is also supported.
- Download and install loopMidi virtual midi port. Download the free version at the bottom of the page. Once you install and run, make use you create a port by pressing the ‘+’ symbol.
- Download and install sforzando :: Free SFZ Player.
- Download and install ARIA Engine Free Sounds from here.
- Download and install the trial version of Reaper. The trial version never expires, but consider buying the software if you like it.
- Run Reaper. Normally the first time you run reaper, it prompts you to set the audio device. Select WASAPI and leave all options as they are.
- Go to Options –> Preferences.
- On the left click on plug-ins –> vst
- Click on Add
- Add the directory C:\Program Files\Plogue\sforzando\VST
- Press Re-scan. Press the Ok button.
- Go to Track –> Insert virtual Instrument on new track.
- Select vsti: sforzando
- Know you should select an instrument. I like the piano sound. Go to Instrument –> Free Sounds –> Garritan –> Jazz Piano Lite. If you like some reverb, set the “send” value to ~75%.
- Enable the LoopMidi port in Reaper. Go to Options -> Preferences -> Midi Devices. Then right click on LoopMidi and click on enable.
- In the track you inserted click on IN (if it does not appear, increase the width of the option area of the track). Select Input:Midi –> LoopBe Internal Midi –> Channel 1.
- Repeat steps 11 to 14 but select midi channel 2 this time. If you want a different sound for the arpeggio, just select a different sound here.
If everything went fine, you should be able to play the piano with your eyes now!
Where can I learn more about the EyeHarp?
You can read our most recent publication at the international Journal Frontiers in Psychology. This publication describes the interface and the results of a conducted evaluation process. You can also watch this youtube video ON A COMPUTER WITH VIDEO ANNOTATIONS ENABLED, in which some explanatory annotations appear during a performance.
A number of keyboard shortcuts are available for tuning on the fly the interface to the needs of each user.
‘k’: toggle between mouse click and dwell time selection method.
‘f’: toggle fullscreen
‘m’: toggle between mouse pointer and gaze data input
‘n’: show/hide the circle corresponding to the area the user is looking at
‘h’: show/hide keyboard shortcuts
The Configuration file
Inside the “bin” folder, where the EyeHarpVS.exe executable is found, there is configuration file named EyeHarp.txt. Inside this file the EyeHarp can be configured. Each line of this value consists of a parameter name and a value. Some values are boolean (true-false) and some are numerical. These are the start-up values of the application. Most of them can be modified through the setup menu as explained before. Let’s clarify one by one the available parameters:
This means that the number of “slices” in the melody layer will be 15 when starting the application.
This means that the size of the step sequencer grid will be 8.
This means that when starting the application all the “slices” of the pie will be dedicated for playing notes. When set to 1, the last 7 slices will be dedicated for changing the chords of the arpeggio created in the step sequencer.
Startup value of the showScale option described in the Setup menu.
This option can only be set through the configuration file. If set to 1, when the showScale is active, the user can tune the EyeHarp to a number of preset scales as explained in the melody layer setup. If set to 0, the user will be able to manually construct the scale, by tuning each note of the scale to the desired value.
If set to 0, the input of the mouse pointer controls the EyeHarp. If set to 1, the EyeHarp will look for a supported eye-tracker, and the interface will be controlled by the data received from the eye-tracker. The mouse/Eyetracker input can be altered also when running the application, by pressing the ‘m’ key on the keyboard.
This option affects the way buttons are pressed in the EyeHarp. When set to 0, a mouse click will be used for pressing control buttons and the buttons of the step sequencer. When set to 1, a dwell time selection method is used: a button is triggered when a time threshold is reached, in which the gaze/pointer stays on a button. The dwell time in most cases is 700 ms. But in the repeat buttons (e.x. for adjusting the tempo), this value is smaller. Remember that the slices of the melody layer are not buttons. These are always triggered instantly once the gaze/mouse is detected on their selection area.
If you use the internal synthesizer for producing sound, and you experience crackles in the sound, this value should be increased.
This hides/shows, on the top right of the screen, a text a number of keyboard shortcuts available.
0: at startup the sound will be generated through the internal synthesizer
1: Midi messages will be send to LoopBe1 or LoopMIDI virtual midi ports.
Sets whether the EyeHarp will start on full-screen or not.
If set to 1, the step sequencer will be monophonic. This means that one note can be selected per column. If set to 0, multiple note can be selected per column.
In order to ignore noisy data, a fixation detection algorithm is utilized. Fixation is called the time where we look at a specific point. In this case, when the value is set to 4, 4 frames (~133 ms) will be used to determine a fixation. If we want to make the system more responsive, this value can be reduced. Nevertheless, if this value is too small, when blinking there is the possibility of triggering notes accidentally.
Determines whether the circle ,indicating the area that the user is looking at, appears or not.
In the case both LoopBe and LoopMIDI virtual ports are installed, select which one is preferred.
Determines whether the semitones will appear in the Pie.
By default the EyeHarp is tuned in C. Through this setting it can be transposed to any tonality.
This option activates (1) or deactivates (0) an extra release region placed at the perimeter of the neutral area.
Set which midi control message to send from the EyeHarp interface. The value of the message is determined by the distance from the center of the Pie. By default all are set to 0 apart from the cc7 that corresponds to the channel volume.
Set the minimum velocity and cc message value to be sent. Range is 0-100. If the distance corresponds to volume, this determines the minimum volume of the melody layer.
Useful when the EyeHarp is controlled with a head tracker or a mouse. When this value is high, notes will be triggered even when the cursor passes with high speed through the selection area of a note. This might cause triggering notes accidentally. If the value is low, the cursor will have to be relativily still for a note to be triggered. If this value is too low though, the system might mecome less responsive. In other words: high value –> more responsive, low value –> more stable.
Inside the /bin folder the user can create songs for studying. The songs are .sng files. If the loadSong value is set to 1, the program will try to read the file “1.sng”. If set to 2, “2.sng” e.t.c. If set to 0, no study file will be loaded. The format of the .sng files is the following.
At the first line the transpose value is set. 0 corresponds to C scale. 1 to C#, 2 to D, e.t.c.. In the second line, the scale is set. 0 2 4 5 7 9 11 corresponds to the major scale. At the third line, the first note of the melody is written, at the second line the second note, e.t.c.. Each note is written in the form OCTAVE DEGREE FLAT. For example
0 5 _ is the first note of the melody layer.
1 2 b corresponds to the note ‘d’ flat at the first octave.
When a song is loaded the next note to be played will be illuminated in red. This way the user can learn written melodies.