4 comments

  • ubutler49 minutes ago
    Just a word of advice, it could go a long way to have a human write the README rather than an LLM.<p>Right now, it’s a little hard to follow what the actual findings are, why they are what they are, and why that’s important.
  • tom_11 minutes ago
    &gt; And where a stack endpoints on a fixed silence threshold, those same speakers get talked over two to two and a half times as often<p>Is the reader expected to understand this sequence of words? Because I&#x27;m struggling a bit.<p>(So I guess the AI agents must end up talking over the people that they&#x27;re talking to? And that&#x27;s bad, because nobody likes it when that happens, and it&#x27;ll make it harder for the agents to steal their money or whatever. But it&#x27;s really not very clear. What&#x27;s the stack? And what would it mean for one to endpoint on something anyway? Dear people produce this kind of shit: please stop.)
    • dpritchett7 minutes ago
      When a whisper-based agent thinks you’ve stopped talking, it will begin its own reply. If you’re younger, you’re more likely to get interrupted by a bot that’s prematurely decided that you’re done speaking.
  • jdanford12 minutes ago
    The README for this project is AI slop
    • Barbing6 minutes ago
      Validating that needs gated clinical corpora
  • kzahiri1 hour ago
    I built this expecting the opposite. A lot of voice AI is being pointed at elderly callers right now, and the assumed risk is that the model mishears them, so I went to measure how bad it is.<p>It isn&#x27;t bad. On 2,760 Common Voice clips, with age brackets matched on accent, gender and speaker so age is the only thing varying, Whisper large-v3 gets 4.67% WER on speakers in their seventies against 6.53% for twenty-somethings. Substitutions, deletions and insertions all fall with age. Deletions in particular don&#x27;t rise, which is what you&#x27;d see if quiet speech were being dropped.<p>Since Whisper&#x27;s decoder is a language model and could plausibly be repairing older speakers&#x27; word choices rather than hearing them better, I re-ran the same clips through wav2vec2, which is pure CTC with no decoder and no LM: 10.30% for the sixties against 14.23% for the twenties. Larger effect, same direction. So it&#x27;s acoustic, not a decoder artifact.<p>Where age does hurt is turn-taking. Voice agents end your turn after a fixed stretch of silence, typically 500-800ms. At 700ms, 8.0% of twenties utterances contain an internal pause long enough to be misread as end-of-turn, versus 19.7% for the sixties. Older speakers take about twice as many pauses inside a single utterance. WER is structurally blind to this: the words that arrive are transcribed correctly while the speaker gets cut off.<p>A maintainer corrected me on that and he was right. Mark Backman at Daily&#x2F;Pipecat pointed out that production stacks don&#x27;t endpoint on a bare VAD threshold anymore. Pipecat&#x27;s default is smart-turn, a semantic model that listens to the waveform. So I measured smart-turn v3 on the same clips: the gap halves to +5.9pp and stops excluding zero. The fixed-threshold result describes a real and common configuration, not every configuration, and the README says so above the fold.<p>Things that mattered methodologically, mostly because they nearly produced wrong answers:<p>- One Common Voice contributor holds 9,792 clips in a single shard, and about seven people account for half the 60+ audio. Uncapped, a &quot;bracket&quot; is a description of one person. Capped at 25 clips each; all intervals bootstrap over speakers, not clips.<p>- The corpus&#x27;s younger contributors skew non-native, so age and accent are entangled. Brackets are matched on the (accent, gender) pair. I also spent a while believing accent explained the whole result, based on a 40-clip pilot; at full sample it moves WER by at most 0.44pp. The wrong claim is corrected in the README rather than deleted.<p>- My first VAD was manufacturing the headline. A relative-energy threshold called breathy trailing-off speech &quot;silence&quot;, and breathiness correlates with age, so the detector&#x27;s error was correlated with the variable under study. WebRTC VAD disagreed on 36% of the eighties clips. Both are now recorded.<p>- fp16 on MPS is verified against fp32&#x2F;CPU rather than assumed, because a quantised Whisper KV cache can take large-v3 from 1.91% WER to 100%.<p>There&#x27;s a fourth result in there that&#x27;s really about a different industry. Several startups now sell daily phone check-ins for older adults claiming to detect cognitive decline from &quot;voice biomarkers&quot;. Validating that needs gated clinical corpora, but the within-speaker noise floor doesn&#x27;t. Pause features vary about 99% within the same speaker, frequently within one sitting. Getting a 10% shift in total pause time above that noise takes roughly 19 days of daily calls per reading, so a &quot;six-week trend&quot; is two or three noisy measurements. Speech rate needs under one call. Pause metrics are the most-cited biomarker and the least usable for per-person drift.<p>Limits, stated because they&#x27;re real: Common Voice&#x27;s older speakers are volunteers who chose to record themselves, so this is healthy aging, not clinical. It&#x27;s read speech, not conversation. And the disordered-speech follow-up is blocked on corpus licensing — TORGO and UASpeech need signed agreements and their HuggingFace mirrors look like unlicensed redistribution, so I didn&#x27;t use them. The one openly-licensed option turned out to be fixed 3-second podcast excerpts where 74% start mid-speech, which makes the pause metric meaningless on it. That failure is written up in METHOD.md instead of being quietly dropped.<p>Everything runs locally, no API key, no spend.