The simplistic answer to that question is “Yes.” Simplistic because the assistant has to listen for its wake-up word like “Hey, Siri.” It is what happens to all those conversations that should be of concern.
It’s useful to understand that there are two ways of “listening” on voice assistants. The first is to send all conversations “to the cloud” for remote processing which involves speech-recognition, searching databases for the answer to the question, and probably more. The second option is to process them on the device itself, which means that conversation data doesn’t leave the device. It doesn’t take a computer expert to recognize that it’s cheaper to do remote processing than to produce a device that has sophisticated technology embedded.
Assuming that processing to answer the query is the only thing going on at the remote location, it doesn’t seem too bad. Unfortunately, much of the conversational content is being stored for use by the manufacturer. There is one legitimate reason. These devices all use Artificial Intelligence (AI) technology. The technology requires “training” to understand a spoken language with various accents, inflections and other aspects of natural speech. The training can be done by AI, which many people find creepy enough. However, it was the recent revelation that data is often subjected to “human review” that has caused a privacy outcry.
Do the same things happen with Google Assistant, Apple Siri and Microsoft Cortana? Yes, they do. The companies are moving toward privacy options at different speeds and in different ways. Amazon recently introduced two voice commands, “Alexa, delete everything I said today,” and “Alexa, delete what I just said.” Amazon and Google both have introduced settings that allow users to prevent “human review” of their voice data. Siri is the only one that is easy to turn off (and on again), but all allow recording to be disabled to a greater or lesser extent. Here is a recent article that gives step-by-step directions for each and includes Samsung’s Bixby if that is useful. This enables users to take greater control of their data. It may also result in the loss of some functionality, but that is often in add-ons like Alexa Skills that may not be essential to most users.
Like most privacy issues these days, it is a trade-off between the services you want and the amount of privacy you require. What is important is to make these decisions thoughtfully.