What worked well?
After using the Amazon Echo dot for about a week and having another user test its capabilities theirself for 30 minutes straight, there were definitely things that worked well. What was realized is that Alexa does a great job of answering and doing simple questions and tasks. This could be asking about the weather, playing music, or adding items to a shopping list. We also realized that Alexa does a pretty good job listening for its wake word even when other conversations are happening in the same room. That even if you talk in a very quiet tone across the room Alexa was able to hear the wake word and answer the question. Another thing we found convenient when testing the Alexa was the convenience of having it connect to your phone. This made adding events to the calendar, setting reminders, and creating shopping lists easy. The announcement feature was also fascinating, allowing users to create an announcement on their phone away from the house to be played for the people who were home. An example of this was saying that “I am on my way home” and having it play on the Alexa so everyone that was home knew. Overall, Alexa did a good job at answering or doing almost everything we asked it to do.
We also found that the indicator lights were useful and worked pretty well. When asking a question it could sense where the user was in the room and showed that in the lights on the ring. It also showed when it was thinking and answering a question by blinking between light blue and dark blue. However, when calling someone it seemed like the light indicator did not do the best job at indicating that they had answered the call when calling their Alexa devices. It just went to a rotating yellow ring when calling. When calling someone’s phone, the volume included with the light became a bit clearer.
What was difficult to use?
During our user test over the last week there were some things we noticed that were difficult to use. One thing we found to be a bit difficult was music. Even though it did a good job at playing the music it sometimes took a couple times to get it to play what we wanted. When first trying to play music, Alexa would automatically connect to amazon music even though we have never used Amazon music. While it was easy to connect our Spotify account to the Amazon Alexa app, we found that we still had to specify that it play from Spotify otherwise it would still play from amazon music. We tried searching if you could make your spotify the main music player on the Alexa app so that we would not have to say, “on Spotify”, but even if we found a way to do it we could not figure it out on the app as buttons and direction were unclear.
That is why after user testing we found that the Alexa seems to be pretty user friendly while the app to be a little confusing. Certain things were hard to find on the app and navigation was not super easy as well. Though we did like that the app provided some interesting things to try on the dot that we did not think of trying. We enjoyed going through the ‘things to try’ page and really seeing what Alexa could do.
What were some unexpected behaviors you observed?
Not always hearing you ask a question, do not know if it was a human error or the echo but the user often had to ask Alexa twice during our 30 minute test. Overall, however I think this was more human error. The user often did not give Alexa enough time to pick up that they were talking to it. They seemed to not give enough pause between the wake word and the question, as the listening light turned on but then turned off probably thinking that they were not actually asking it a question. The user also seemed to talk really loudly when talking to the Alexa, even though you could talk normally and the Alexa can still understand. I think that this is because of the idea of talking to voice design. Since it is still relatively new, more than often people have had experiences with voice design where the device did not understand or hear them, because of this I think some users tend to talk load and annunciate more when talking to voice interfaces.
One other thing we found to be a little confusing is that you have to specifically name your timers if you want to know what they are for. An example is if you just tell it to set a timer for 30 minutes, when the timer is done it will just start beeping without telling you why and if you forgot you set a timer this could be confusing. We found that it only tells you if you name the timer. That means you have to say something like “set pasta timer for 10 minutes”, that way when the alarm goes off it actually says “pasta timer” in between alarms. We thought it should have at least told you your timer is up when not naming it rather than just doing an alarm.
Another unexpected behavior was that Alexa often talked for way too long giving more information then needed which user often found redundant. It often gave a long answer rather than the best, short answer. If you wanted a short answer your question had to be more specific. However, we did find out that you can change Alexa’s speech rate so that she says her long answers at a faster rate.
What about the experience was exceptionally flawed?
The only thing that seemed exceptionally flawed was with music. It seems like they are really trying to get you to use only their services by making the main music player Amazon Music. More often than not the Alexa did not hear the user properly or did not play what they asked it to play. Also having to say more than you want, to play a song on another music app such as Spotify or Iheartradio seemed to kind of annoy the user, and not being able to easily switch your main music device to make it more convenient seemed flawed. It feels like you should be able to easily change it but it was not clear how to do it in the app or when searching online. It makes it not as smooth as an experience for the user.
We also noticed that it seemed to have a difficult time doing 2 tasks at once except when playing music. If you asked it a question while on a call or while playing a game, it either ended the previous thing to answer the new question or did not answer at all. It seems like it only wants to do one thing at a time. One thing that we noticed is that if the echo is only connected to one person’s device it will not let you do the same thing at the same time. An example of this was I was at work listening to music on spotify when all of a sudden my music stopped playing. I was trying to figure out why when I realized it said it was now connected to the echo. I switched it back to play on my phone and a little bit later it stopped playing again and said it was playing on the echo. I called home and realized that my sister had asked the echo to play Spotify at home and since it was only connected to my device it was playing it from my phone. It would have been nice to be able to do the same thing in different locations without connecting another device but that did not seem to be the case.
What about the voice user experience could be improved?
What could be improved is the users ability to set up their main music device rather than automatically connecting to Amazon music as not everyone uses Amazon music. It also usually plays the wrong thing when asking it to play a specific song causing the user to repeat themselves in a slower more clear voice. We also realized that there was no way to connect our Google Music account to Alexa, as it was not an option in the Alexa app. That is a flaw that could be improved as the music we have bought over the years is located in Google music. When asking Alexa if it could connect to google play music she responded with saying you could connect to Apple music which is not what we wanted. So overall it seems that the music feature could be improved.
Another thing that could be improved is length of responses. This could be done by giving the best short response and then asking if they would like to know more rather than giving the user a ton of information that they might not want to hear causing them to tell Alexa to stop. This could bring a better experience and interaction to the interface.
Also based on users’ thoughts, they wish that it was easier to do two things at once. That is something that could also be improved even if it may be difficult to improve. However, it does a great job at answering simple questions and doing one task at a time really well.
Usability Test Video
The video below is my 30 minute usability test with my sister. I observed and took down notes as my sister interacted with the echo. I tried not to interfere much with what she was doing allowing her to test out the echo with only what she knows. I was there if she had questions and only clarified something every once in a while. She explores some basic functions that the Alexa can perform so that I could determine ease of use and convenience. I also used it off and on for a whole week before the test, taking notes of interactions throughout the week to get a better understanding of the device.