What a brilliant idea! is this all done locally? That's incredible.
Very interesting (not for a dashcam, but for home monitoring).
Where is the Exit to this dystopia?
Does anyone know of an open weights models that can embed video? Would love to experiment locally with this.
Not aware of any that do native video-to-vector embedding the way Gemini Embedding 2 does. There are CLIP-based models (like VideoCLIP) that embed frames individually, but they don't process temporal video. you'd need to average frame embeddings which loses a lot.<p>Would love to see open-weight models with this capability since it would eliminate the API cost and the privacy concern of uploading footage.
That's quite interesting, well done! I haven't thought of this use case for embeddings. It open the door to quite many potential applications!
> No transcription, no frame captioning, no intermediate text.<p>If there is text on the video (like a caption or wtv), will the embedding capture that? Never thought about this before.<p>If the video has audio, does the embedding capture that too?
Yes to both. The embedding is over raw video frames, so anything visible (text, signs, captions) gets captured in the vector. And Gemini Embedding 2 extracts the audio track and embeds it alongside the visual frames. So a query like 'someone yelling' would theoretically match on audio. My dashcam footage doesn't have audio though, so I haven't tested that side yet.
Today I learned that Gemini can now natively embed video..<p>Cool Project, thanks for sharing!
Nice use of native video embedding. How do you handle
cases where Gemini's response confidence is low?
Do you have a fallback or threshold?
as of now, no threshold but that is planned in the future.<p>for example, for now if i search "cybertruck" in my indexed dashcam footage, i don't have any cybertrucks in my footage, so it'll return a clip of the next best match which is a big truck, but not a cybertruck
very cool, anybody have apparent use cases for this?
I think a good use case would be searching for certain products or videos across social media (TikTok and Instagram). especially useful for shopping, maybe
dashcam and home security footage are the 2 main ones i can think of.<p>a bit expensive right now so it's not as practical at scale. but once the embedding model comes out of public preview, and we hopefully get a local equivalent, this will be a lot more practical.
State surveillance
why not skip the text conversion? is it usable at all?