how do you pick good goal conditioning images/do you have to hand pick a dataset of good goal images? seems hard if you don't have full context. really cool though!
Does it have language skills? I always thought it would be interesting to train a model within minecraft as a sortof proxy for 'embodiment'. You could then try asking it about its experiences. "whats your favourite food?", "How does it feel when you hear a spider?", "how low does your food bar need to go before it feels really urgent?" etc
Not yet, but we'll add language as a modality to the larger models! The models are trained end-to-end on video data, so we'll need datasets that mix video and language, e.g. transcripts of game streams. When the models are scaled up to cross-videogames and robots there will definitely be a bunch of language data.
Hey everyone! I'm Alex, one of the founders of Pantograph. We've spent the last six months building a pretty smart Minecraft model, coming soon to a server near you!<p>We trained it on about 500k hours of Minecraft videos, and it learned how to fight creepers, build walls and other structures, and explore to find visual goals.<p>We're considering putting up a public API for larger models like this one, let us know if you'd like to be able to put Pan in your own server :)<p>What's most interesting about the model isn't the performance that it gets in Minecraft, but how general the method is. When we scale it up, it should be able to act in any kind of video game, as well as robots in the real world (which are really just another video game).
Reminds me of Google's AI Dreamer that could mine diamonds in Minecraft (<a href="https://www.scientificamerican.com/article/google-deepmind-taught-itself-to-play-minecraft/" rel="nofollow">https://www.scientificamerican.com/article/google-deepmind-t...</a>) or OpenAI training on video to play Minecraft (<a href="https://openai.com/index/vpt/" rel="nofollow">https://openai.com/index/vpt/</a>)
Very much inspired by those papers! One of the things that's interesting about our model is it's goal-conditioned, so it can do any task at inference time without training on it. We had a lot of fun making eval environments after we trained the model trying to find interesting things it can do, and that was all after we trained the model. More like prompting an LLM.<p>(Versus Dreamer, which needs to be trained on a hand-written reward function for each task that you want to do.)