In 2024, I tested out Bee’s wristband, the company that eventually got acquired by Amazon. The idea behind the band was that it would keep listening to your conversations throughout the day and provide you with action items and insights. In my experience, a separate band that recorded conversations freaked people out. The insights I got from the band were sometimes useful, and often strange or out of place.
Yesterday, Apple launched a similar feature, but in its own style.
The new iPhone Duo was the star of the show at Apple’s fall event, but there was a section in the Apple Watch presentation that caught my eye the most. In that segment, Apple described its new Audio Intelligence feature, which acts like an always-on meeting notetaker. In my mind, I yelled, “Granola!” Things are a bit more nuanced than that.
Ron Huang, the company’s VP of sensing and connectivity, said that the Apple Watch is a “perfect device to help you stay aware of the sounds around you and keep track of your conversations.”
We will get to the summarizing feature in a second, but Apple also debuted a feature called Live Rewind, which generates a transcription of the past 15 seconds of audio. The demo showed a waitress telling customers about new specials at the restaurant at a fast pace, and one of the customers hits the Digital Crown twice to get a transcript of that. But, like reporter Joanna Stern said, why not just ask her again? There is possibly an accessibility angle, but it would be strange to pull up the last 15 seconds of transcript while missing the current conversation. It is likely that the company keeps 15 seconds of audio/transcript in buffer on rolling basis.
”This is great for when you're trying to recall a book recommendation, a colleague's brilliant idea, or something you didn't hear the first time,” Huang said. If this happens in person, you can ask them again. One potential use case is you listening to a live lecture and missing something, but then again, you might be using something to record it already.
Apple said that this feature doesn’t continue to transcribe in the background, but I assume it has to keep track of the last 15 seconds all the time.
Then there is the Siri Recap feature, which is like any other meeting notetaker. A key difference is that you can’t get a transcript and only get summaries, action items, and takeaways.
There is an argument that this will kill meeting notetakers. I don’t think that is the case in the slightest. First, a lot of people, including journalists like myself, use meeting recorders to obtain the recording or transcript. For AI-powered meeting notetakers, the transcript is a crucial part, as it acts as fodder for prompts and action items that could result in further workflows. Apple doesn’t want to get into any of that. Apple’s new feature could actually push people towards more capable notetakers if needed.
There are some privacy-focused steps that could separate this feature from meeting notetakers. First, there is no speaker recognition. So users won’t be able to access who said what. Second, Apple said it will remove sensitive information like financial details or government-assigned identifiers. On the face of it, this sounds good. But it is hard to define what might be sensitive to someone, and Apple will need to iterate on its software to make this definition more inclusive.
Third, to produce the summary, Apple processes the audio on the device, creates a transcript, sends a condensed version to the private cloud, and produces the final summary.
While all these steps are good for user protection, sensitive information can still leak in. And Apple is aware of that.
“While audio is not recorded and stored, and no speakers are attributed, consider those around you where conversations might be private or sensitive,” the company writes on a support page.
Apple is setting this up nicely for ever-recording AI wearables that might have fewer guardrails. But even with limits, the onus is on the user to tell others that they are recording, which might not be the case every time.
In defense of this, the argument that might come up is that “You can also record all the time on your phone.” Theoretically, yes. But you would need the phone's mic for other stuff, and if you are trying to be discreet about it, indicators on your phone might give you away.
Apple is rumored to release AirPods with camera sensors next year. That adds another dimension to feed to AI. With the pair of an ever-listening Apple Watch and ever-watching AirPods, Apple thinks it can take on new “context-aware” AI gadgets easily without sneaking in a new form factor.
But there will still be a ton of wearables that would want to compete with Apple’s ecosystem and might make privacy trade-offs to promise more features and personalization. We just don’t know how people feel about these ambient devices being around them, and whether it is even a good thing.