Time Traveling with AI: A Musical Journey

My name is David and yeah, let's talk to you all about time traveling with AI, a musical journey that we did. So, first of all, my name is David Corsid. I am the founder at stately.ai. I'm also a software engineer. And Stephen, he's gonna talk about himself in the pre-recorded video, but he's a designerdeveloper at CodePen. And together we are animation Amigos, Boolean Buddies, everything. We did a show a while ago called the key Framers. And in the key Framers, we brought imaginative user interfaces to life using just HTML, CSS, and of course, a little bit of JavaScript. And if you go on our codepen.ioteam/keyframers and you know we'll provide some links later you'll see some of the really fun experiments that we did. This one is just using HTML and CSS, but needless to say, we had a lot of fun doing this and so we were excited to get the opportunity to do it again on code TV, which is Jason Langdorf's, project where we basically had four hours to accomplish some sort of challenge.

And so the challenge was create a really imaginative user interface with AI using of course AI and Alolia together. And so we were like, hey, why don't we just do some time traveling? So what do we mean by time traveling? I know it sounds crazy and you know, we're not there yet, but with AI, we could time travel in a different way. So typically when you have a conversation with an agent, it goes in terms. So it's your typical chatbased chatbot interface where the user prompts an agent, the agent responds, the user responds, the agent responds, etc. But we wanted to do something a little bit different, sort of like Minority Reports or any of those other crazy movies where you could actually go back in time where we could say, you know what, let's go back to an earlier point in the conversation and sort of branch off from there.

But we didn't just want this to be your typical, no offense, but boring chatbot interface. We wanted it to adhere to the challenge and be imaginative. So, what were we going to do? We were going to make a DJ music recommendation app in 3D. So, we were going to have this AI powered DJ that's just going through space and time and you could chat with it and say, "I want to listen to this." And then you could listen to a song and then you might go back and say, "actually, I liked this sort of genre better, so let's explore songs from that genre." and the DJ would recommend you songs to listen to that fit your mood or fit your genre or fit the types of lyrics that you were that you were looking for.

So, this was a lot of fun to plan and of course we had only four hours to learn the ins and outs of Alolia and hook it up to 3JS and AI and just make it work. Now, to make this possible, we couldn't have just a normal data structure for a chat interface. we had to implement graphs and I love working with graphs. I created the XState state machine library. So nodes and edges are in my blood right now. And so this is the data structure that we needed to use to enable this time traveling mechanism. We needed nodes, edges and a graph where the edges connected these nodes together and we could set the currents node so that we could essentially say hey now we're going back in time. So we're on this node and we could branch off of here. And this allows us to go arbitrarily at different points in time.

So if we wanted to go in the future after going to the past, we could just set the current node to whatever the future node was. So conceptually, this is what it looks like. Now to work with this and because we had a limited time, I used a library I created called XState store. So you might be familiar with XState which is a state machine library. XState store is a smaller library that's more similar to Zustand Jotai Redux etc where it's still states and event space but it provides a really minimal interface for easily working with the data structure the graph data structure that I mentioned and for integrating with React and as we will see soon with Alolia as well.

So if you want to check that out, here's the docs links daily.aiisaixate-store. Now, so where does AI and Alolia come in? We used a lot of AI for the music recommendation part, but also the classification part. We needed to do one of two things. we needed to either say if the user or actually let me explain first how we used Alolia. So basically we fed it a huge data set I think it's the Spotify million songs data set we fed that into Alolia and we basically what was that data set? It was a combination of song titles, genres and lyrics for those songs. So if the user asked, hey, I want songs about heartbreak or I want songs about computers or robots or whatever, then we could query the Alolia data set really easily and really quickly. However, if the user was feeling something that's not easily queryable, like they would tell us their mood or we want songs like this, or similar songs to what we heard, then it would actually go to, at the time we used 03 mini, but it would go to an LLM and the LLM would actually make those recommendations and output them in a suggestions array.

So, the Algolia setup was quite easy. A lot of the docs were tailored towards hey, you could just plop in this Alolia UI into your Nex.js app or React and have it just working. But because of the unique UI interface that we wanted to build, we couldn't just plot that in. We needed to go straight to the source. We had to go to the bare metal core of just using the Alolia client. And I mean, it was actually pretty easy to use. So, unfortunately, we couldn't take advantage of the Algolia UI, but I'm I was surprised how flexible it was at accommodating our specific UI needs just by using the client itself.

Play from 0:00

And now I'm going to hand it over to Steven Shaw to actually demonstrate what we built and some of the cool UI. So, here we go. Welcome to the future. I'm Steven Shaw, the other half of key framers and I were invited out to film an episode of webdev challenge for code TV. The theme of our episode was the future of UX, especially how AI is going to change the ways that we interact with computers. For our app, we wanted to use 3D to help users visualize the flow of conversation and enable a version of time travel with branching paths that results in a really unique experience. To render out 3D elements, we used a library called React 3 Fiber which integrates JSX and 3JS, a 3D rendering library letting you code out entire 3D scenes with React components. We even built a little DJ robot to kind of give a friendly face to the whole chat aspect of the app.

It's all geometries and materials. It's in the code. It's not a 3D model. But enough stalling. Let's show the actual app so that you can experience the fun. Here he is. Here's our little DJ waiting for us in this gorgeous 3D space. We've got an extra like pixelated filter on top of it to complement the retro futuristic vibe we were going for. So, let's chat with our little robot friend here. Get some electrionic country, please, partner. Using David's clever Alolia integration with the LLM, our DJ gives back some relevant results that we can explore. Old Town Road. That's a nice classic. Let's get in there. Now the cool thing about React 3 fiber is you can actually render out regular React components, HTML, all that kind of stuff within the 3D space. So this is just a regular HTML and CSS styled element. Now we can change things up a little bit.

Let's say maybe a little slower right there. And we can get some different results there to explore. Now, if you look back, you can see our timeline rendered out in this 3D space that we can jump between to get all of those other those other options. And if we select something else from there, we actually branch off and create a new timeline. You can see this real clearly in our kind of timeline debug view. This is what kind of enables that time travel feel. So you can change things up and ask for like this and get some neat results from there. And plenty of personality there from a chatbot with 3D AI all these API integrations. It was a lot to tackle in four hours, especially without real time travel. But I hope our little DJ friend here has taught you a few new tricks that you can use in the future of your app.

Thanks so much. All right. So in general one of the big problems or not really a problem just a unique thing that we had to solve was that given the user prompt we had to route it to whether it was going to be handled by Alolia or handled by an LLM. And the simple test for that was is this prompt searching for some sort of lyrics or something that we could easily query the data set that we put into Alolia for and retrieve results quickly. If so, then just use Alolia directly. Otherwise, skip Alolia and go to an LLM and I really wanted to make it work where we just use Alolia for everything. And the great thing is that I was told that this isn't beta but you can do this. So, if we were to redo this challenge again, we would definitely use Alolia agent Studio. You could find it online at this link, ali-agent studio. And this essentially allows you to just create an agent within Alolia that queries your data set for you.

And whether you're looking for lyrics or different kinds of feels or genres or whatever, Alolia is the one-stop shop for you to just query those results using AI. And that would have been really useful when we were building this about half a year ago.

Play from 7:00

But I am excited that they are working on that. So with that, thank you very much. If you want to see our code, I'm warning you it's hackathon quality, but it is fun to look at. It's at GitHub.comkeyframersdai. And you could also see the episode where we made this at this link. And our website is keyframe.rs. Thank you all very much. Awesome. Thank you so much, Stephen. Appreciate it. My I'm looking for questions here. Just to let folks know, you can drop your questions in the Q&A section. I really do feel like this was a bit of time travel because it's amazing how much sort of the whole agent space has already evolved in 6 months, right? It was kind of fun imagining what was going to be there with sort of these agentic mechanisms. And now the capabilities that are there. Think a lot of the conversation now is sort of moving into you know how do agents engage with other tools right is it I like that you kind of have the state machine you sort of have this external agentic flow there's things like agent studio that allow you to sort of bake search directly into the agents there's MCPs right for tooling have you been playing more with sort of agentic flows since then have there have there been other things that you've built and have you seen things change on your side of course.

Yeah. So I am huge into state machines of course and so I do love the idea of using state machines to guide certain flows. And in fact this app that we built DJI was it sort of was leaning in that direction because like you saw Stephen demonstrate you could either be in the state of playing a song or you could be in the state of talking with the agent. And so those are different states that we have to manage and I think that state machines are great guard rails for agents to you know follow specific paths but still have the flexibility to respond however they wants to. So that's the type of experimentation that I've been doing but also doing things like agent self-improvements. That's something I've been experimenting with and something that could feed well into something like Alolia where Alolia can build on top of its data sets by just you know user preferences and you know additional information as it comes in and things like that. So basically just keeping its data sets always up to date. Yeah. No, that makes sense.

I'm I'm being chasted by my producer. I called you Stephen. I'm sorry, David. That's such a compliment. Thank you. You're you're the less beard, right? Yeah. No, I think that's kind of a big piece of it too, right? Is we've got our classic sort of deterministic systems. We now have these new sort of non-deterministic pieces of our applications and we have to figure out how those two are going to play together and how to use those pieces together. Eileen was asking, I think this is more of a question for me. Does agent Studio require a plan with neural search? Agent Studio does not require neural search. For those don't know, neural search is sort of the semantic vector layer that's baked into Alolia search. Agent studio is actually a bring your own API key model. So you leverage your existing relationship with a LLM provider whether it's like an OpenAI or you know open router or Gemini you bring your keys and we sit on top of that.

We also sit on top of your existing search plan. So there's no neural search component to it. You can use whatever models we support. And I'm going to say thank you to you, David. And because I want to do I do want to talk a little bit more about agent studio before we move on to our next presenter. So thanks so much.

Play from 12:00