Mind-Reading with Adaptive and Intelligent User Interfaces

Hello, yeah, my name is David and I am so excited to be here in Taiwan again. Talking to you about mind-reading with intelligence and adaptive user interfaces. Now we all know that developing user interfaces is actually pretty difficult, just because there's many things that can happen. And there's also many ways that a user can use the user interfaces and applications that you build. For example, let's say that we have this fetch data button, ideally, we click it, we get some data and eventually it comes back with a response, hopefully a positive response. But what happens if this data takes a long time to load? We want some way to show the user that this data is in progress of loading, and so we might show a spinner or something until the data finally comes. However, if you're anything like me, then you'll know that you get impatient. After waiting so long for the data that you just start clicking the button lots of times, hoping that the data will come back faster, it never does.

But you're a developer, you're smart, you think? I know? I will disable the button so that the user can't click it, and then they have to wait for the data to come back. But I'm also smart because I'm a developer and I could open the Dev tools. And I could undeceive a button, and then I could just keep trying to make the request. The point is that everyone is different and we all come from different cultures and we speak different languages. This is my second time in Taiwan, and I still, I could say Shishya and I could point to things at the night market that I want, and, you know, hold some Taiwanese dollars. But everyone is different and we all behave in different ways. So that's why I think that a be testing where we show either one user interface or another user interface to the user randomly. That's ineffective because that's only two possible options that the user might be able to choose from, when, in fact, we would be better building user interfaces that adapt to every single possible user.

And also user flows, which are how users go and navigate through the application, are not always linear. That means that a user might take one path by the different user might. Take a different path through the application, and we have to account for all these different ways that users can use our applications. There is a library called guess JAS, which uses predictive analytics to predictably fetch different parts of your application based on what the user is predicted to do. It was created by I think his name is Sam Ciccone at Google. And what it does is every time a user interacts with an application that has Guest:Yes enabled, it sends analytics to Google analytics and it reads those in Olynyk's back when the user uses the application again. And so it could predict if the users on one page and they keep visiting another page, then it's going to prefetch the other page.

Because that means that when they eventually do click that next page, it's already loaded for them. So it's a better user experience, it's faster for them, and you could generalize this with an entire group of users. So that's just one example of a library that could help with what's called adaptive user interfaces, also known as a Hewitt. Now, an adaptive user interface is a user interface that adapts to the user instead of the device, and if you remember from a long time ago, one example of an adaptive user interface is Clippy. Remember when you would go on Microsoft word or Excel, and Clippy would jump up and say, I think you're typing this, or I think you're doing this, which you want some help? So that's the difference between adaptive and responsive user interfaces. So when we think about a user flow in an application, there's many different ways, like we mentioned, the user could go. For example, if you're in an inbox application, they could see the newest mail, or they could create a new email, and then they could decide to either save that email into address or they could send it.

Many different possibilities. So how could we take advantage of these varied user flows and predictive analytics? Well, we could measure, for instance, how often users are going to go from one path to the other, and then we could predict just based on where the user is in the flow of the application, where we could prefetch resources and where we could adapt the user interface for them. So, for example, you might. Have noticed this? In Gmail? When you click a delete? Well, when you delete an email or when you send an email, it will ask you a previous applications asked you a confirmation. Are you sure you want to delete this? You would click yes or no. But it turns out that if you end up deleting or clicking yes a lot of times, then a better user interface would be. When they click delete, it says delete it.

Because we predict they're going to want that action to be the default action, and we give them an option to undo that action if we want them to. And so this completely eliminates a step and makes the user the user interface a lot more pleasant to work with. So another example that's recent in a Gmail is that this predictive text, where as you're typing, it predicts what you're going to say next. And this is a really cool feature that I use all the time, especially on your mobile app. If you use Gmail on mobile, you could just swipe and it goes to the next one, so that's that's really cool. So, adaptive user interfaces are also good for power users because it could provide shortcuts for the power user to navigates to navigate through the app in the flows that he or she uses most often.

And in the same notion a. An adaptive user interface can help users that might not be familiar with the user interface and might need a little help by providing assistive guidance to the user. So both of them reach the same goals. And the power user can do their workflow faster, and the user who needs help can get that automatically provided assistance. However, this field of developing adapted user interfaces, it's not a new idea, it's pretty recent, around 2012 2015 was the first research about these. But developing adaptive user interfaces is challenging because there's a lot of things to think about. And there's no strict, strict system for what you should adapt and what you should change in response to user behavior. But we could consider the first steps to what we could apply to our development practices in model driven development. And so model driven development is creating a model of our application, and this abstract model describes how our application behaves.

And we could. Use that to better support these analytics that drive these adaptive user interfaces. Now, you might be used to seeing this from a designer. These are user interface diagrams, and they show just how applications are supposed to go from one step to the other, from one screen to the next. They're also known as flowcharts, but more formally, you might recognize these as finite state machines, and so just the review of finite state machine is five parts. So we have first, we have our initial states. So that would be the first thing that your user sees, for example, as you go in an application. And then you have a finite number of states, which are these states that the user can be in at any given time? Now, the key is that you could only be in one of these states at a time. You also have a finite number of events which tell you how the user, or like what can happen in the application.

Whether it's the user initiating the action or some other outside events, like a fetch response or versa, or a failure. And then there's transitions that go between the states, which say, when were in the states and this event happens. This is the next state that we're going to end up in. There's also the notion of final states, and so when you're in a final states, that means that the application, it reached the terminal point and nothing can happen after that. For example, if you're signed out, you would have to sign in again because there's no further action to be done from there. So, traditionally, finite state machines can be built. With switch case statements in which you would first identify what state you're in, in this case searching, you would identify what the events that just occurred was, for example. A resolve which we assumed came from a promise, and from there, you would, you would just say what the next state as a result of that is now.

This is a little bit different than Redux from what you might see in the examples of Redux, in which it's action first. And so we're not doing it based on action first, we're doing it based on what state you're on first. Now switch statements are all good and they're supported, and you know. Almost every single language. But I like action mapping instead, because number one, it's a little bit shorter and easier to read, and number two, we could just determine what the next state is. Based on this simple transition function, for instance, let's say that we're on the forum state over here, so we could have our transition function. Look at that state. Look up the events that just happened, like a submit events and then a. It doesn't map to any events, in this case, it Maps that. Thanks, but if it doesn't matter to any events, then we just go back to the original states. Because we're saying that this dates does not handle that events now.

As mentioned, I created XState a couple years ago, and XState takes that same idea of taking the objects notation and making it into an executable state machine. So in this case, you would have the exact same object structure where you would specify the state. Specify the event that just happened and map that to the next state that occurs based on that events. Now you can use XState with React as well, and so if you're using Reacts, then you might be familiar with things such as useReducer. The useMachine hook provided by the XD Triack's package works in a very similar way. You have the current states, and you have a send function that's returned from the tuple abuse machine, and you pass it in a machine that you created. Based on that, you could just match the states based on what the finite state of your application is. For example, if we're on the questioned state, we might want to render a question screen with various callbacks or various behaviors.

That could happen. For example, the user could click good, click bad, or it could close in. Each of these can send an event back to the machine. Now, x 8 also works with View, Angular, Speltz, whatever other frameworks are out there. And that's because when you create a machine, you could interpret it as a service. And then you could said, you could subscribe to state changes as well as send that service events. Now, the next step from state machines is what's known as statecharts, or hierarchical state machines. This was created by a computer. Scientist by the name of David Harel in 1987. And it was created for the Israeli air force because their aircrafts have very complex interfaces. And it had to be extremely precise to know which dates you're in as you're operating one of these complicated aircraft.

Play from 0:00

Now, statecharts extends the notion of state machines by adding a few things. First of all, they add actions which can be on entering a state, on exiting a state, and also transitioning between states. They also add guards, which makes sure that certain conditions are met before transition can be taken. For example, when we search, we want to make sure that the query is greater than 0. It also has the idea of hierarchy. So you could take these state machines and you can nest them into each other infinitely now nesting. I'll explain in a minute. But when you're in a nested state, you're also in the parent states, and also in that parents parent state, so it's a way of naturally grouping states together. There's also orthogonal D, which is defining regions of states that don't really have too much to do to each other. But you could, you could, express that.

Both of these regions are active at the same time. And there's also history which described that once you leave the states and you want to come back to that state, it remembers which child state you were previously in. So, hierarchical states are great for grouping states, in this case, we have both a success in failure states grouped in here. And we're saying that. The behavior of this is that whenever we search from any of these states, we could search again, so we have that same researching behavior now. Hierarchical states are just a copy of the normal state machine definition in the object, but you can nest them inside any one of the states, so, for example, this search states. I just could have the same structure within here, and then I could define transitions either on the parent states or any of the child states as well. All right, so let's take a practical example. Let's say that I'm building this feedback form, and it has three steps.

It says, how was your experience? You could click good, and it says thanks for your feedback, you could click Bed, and then it takes. You to a forum and you could fill it out, then press submit and then it says, thank you for your feedback. Or you could exit any one of these screens and it just closes the forum. So this is what it looks like. You know, you click good, it says thanks for your feedback. If we rerun, we click bad. We could type something here like, no, that says thanks for your feedback. Or in any states, we could just exit and then it closes, so nothing too difficult. But we could describe this with the state machine as shown here. All right, so this is just a traditional state machine that we saw with a switch case statement. So we switch based on which state we're on, such as question, and then based on whether we're clicking good or bad.

It tells us what the next state is going to be. We use this with reacts, and of course, you could use this with Angular or View or anything. And this is just a traditional state machine. You could also use XState for this as well. So here's our gameplan. I wants to improve this feedback form, but I want to do it in a way that's adaptive to the user. So, first of all, we're going to create that abstract model of the feedback form in the form of a finite state machine. And then we're going to apply transition in AX LYTX, which is just determining how often certain transitions between states are taken. We identify which paths can be adaptive and which could change based on user behavior in real time, and then we use that analysis for adaptation and actually build that into the app. So we need to do, we need to understand a few concepts before we do this. And this gets a little bit into.

Play from 13:00

Graph theory, first of all, is the idea of weighted graphs, so a directed graph is a graph connecting notes together with edges in a certain direction. For example, signing in is connected to success or connected to error. Now we could give each of these edges or arrows, for example. We could say that 90% of the time a sign-in leads to a success and 10% of the time it leads to an error. And then we could show that the weight of the. First set, we mean to success, is a lot more than the second edge. So when we think about our user interfaces this way, we have to stop thinking that each one of our transitions is equal.

We now realize that over time, these weights can change. And the reality is that the user behaves the user interfaces with an app in a way that users are more likely to take certain paths and less likely to take other ones, like the ones up here. So I did this sort of thing when I worked on the via SlideShare team at Microsoft, which if you don't know what via SlideShare is, you could go to Aka..ms/offweb. I. Adding analytics to each of these transitions.

Play from 17:00

We created a weighted graph automatically that detailed what happened in a period of time and how often a user was able to go from one state to another. So we see over here that signing in it most often whence to this sign, and there's this externally signing in. Where it would take you to a separate window and say, do you want to log in with Microsoft or GitHub? But usually, the user credentials were restored over here. And then we could see whether users were more likely to share a session or to join a session, and it looks like those are pretty equal over there. And then we could also visualize things such as errors, we could visualize how long each transition took, and more with that data. So let's get back to our feedback example over here. We could click good or bad, but I wants to make one path adaptive, and it's this bad path over here.

And the reason is because we want them to finish the feedback form, but we don't want them to get annoyed with it. So I'm thinking, maybe when they click bad, that we could take them directly to this things for your feedback screen and give them an opportunity to give feedback instead of annoying them and having them provide why they should give feedback. So let's take a look at what that looks like. So over here. If I clicks good and then it says, thanks for your feedback, and then I reloaded the page. Now I have analytics on that transition, so that by that transition, I mean what state are. We in what then just happened, and how many times did it happen? So let's say that as a user, I click bad and then I see this green and I'm like, you know what, I really don't want to fill this out.

So now we reload and we see that when I'm in the forum screen, which is that. Can you explain why I immediately closed it? So I'm going to keep doing that because I really don't want to fill out this form. Now. We could see when it reaches a certain threshold, that's when I click bad. It takes me directly to thanks for your feedback. And this is a very simple example of a user adaptive interface where it says, OK. The user really doesn't like filling the feedback form out, so instead we should take them to this Thing's for you feedback form. And so this works in the reverse way. So let's say that I'm really having a bad experience by wants to tell them why? So I would fill out the form and I would press submit. Now we see that the the adaptive paths are going to change because we keep providing feedback and we keep taking a different path.

So we see that the user behavior, the user being me, changes, changes over time.

Play from 19:00

And so now, when I click that, it will now take us back directly to that screen. And so this is something that happens, as you see in real time, based on those analytics, instead of it being over time. And Ida rating back on the products. And so this really cuts the developments loop for features such as this. Because it could adapt to individual users rather than just defining one set behavior for all users. So there's many ways that we could work with these, that we can analyze these transitions, one of them is by determining the frequency of transition. For example, let's say we're trying to make an Instagram app, we could see that. Well, let's say that when we log in, we want to determine. Do we show the gallery, the camera or the profile view? The gallery being just photo gallery of everyone's photos, the camera just being your photo.

Because you really like to take photos when you open Instagram or WhatsApp or whatever, you use, WeChat and the. Profile page, which is just your profile. So when we take a look at these transitions, we see that reaching the gallery page is about 30% of people, reaching the camera is about 57%, and reaching the profile page is about 12%. And by reaching, I mean, this is what the user chooses to do. So, we see that once a user logs in the in this case, they immediately want to go to the camera page. So we could adapt that tap and say when to user logs in this particular user. Maybe we should just take them directly to the camera page, because that's what they're most likely to do. There's also analysis of events based on what we wants the user to do. In this case, I'm using ads as an example, but I don't like ads.

You could think of this in other terms, but let's say that. We want to determine whether we wants the ads to be interspersed in the gallery feed or shown at the very top every single time. So if they're interspersed and they scroll and they go through the ads, then maybe we want to assign a score of negative one because that's not desirable behavior and that they're shown at the top and the immediately scroll. We assign a score of negative ten because that's definitely not what we want the user to do. However, if they, if they tap the ad and it's interspersed, we assign positive ten to some sort of analytics score. Or we assign positive ten if it's shown at the top and they decide to tap the ad without scrolling. And so we could use that measurement and determine should we show the ads interspersed or should we show them at the very top?

And based on that, we could, you know, just optimize for what the user would want to see and where they would want to see the ads.

Play from 22:00

So this gets a little bit more complex and just like the first sock this morning. About machine learning decision trees are used a lot in machine learning and in making determinations about choices such as this. For example, on our user on our Instagram example, we could determine, okay, are they a new user? And so maybe based on this data and based on the analytics, we could show the. Ad at the top. Or maybe more information is needed, such as what's their friend counts? And so now we could divide groups of users into smaller groups and use these decision trees to make decisions on what actions to take or which paths to adapt. So there's many different ways that we could also make prediction on how we predict that users will behave. One of them is called the shortest path algorithm, and there's many different algorithms to do that.

But we could use weights to determine what are the shortest paths through the application. And these usually determine what the happy paths are, the paths that the users are most frequently going to take. Now, what I love about state machines, and especially using X8, is that. You could provide those analytics right in the transition function over here, or the transition hook of a service. So now, having analytics to either attract, trace or measure what the like, what the user is doing is only in one call, there's one place you do it, you don't have to copy and paste it everywhere in your app. And so you could track what state that we come from which is, or what state are we going to? Which is the target state? What state did we come from? Which is the source state? Which event just happened? And, of course, you want to sanitize this to prevent user details from leaking. And when did this event occur? So, based on these analytics, we could see that sometimes the shortest path is not what it may seem.

For example, if we're going from state A to C, the shortest path looks like A to B to C. However, based on analytics, we might reveal a different story. That users are more likely to go from state A to D to E to C, and in that case, the shortest path would be that A to D to E to C. Instead of it, be the C, which is less frequently traversed. Now there's this other idea called higher mark, a higher order Markov models, which I recommend you looking up. We're not going to go in a big discussion on what that is, but essentially, this is the idea. Let's say that you're in the state A, and you determine that 75% of the time we transition from. State A to state B, so you might say, let's optimize and make an adaptive user interface for optimizing the transition from A to B. However, let's say that we extend that and say, if the user is coming from state D now, they're much more likely to go to state C.

So this really depends not on a single transition, but multiple transitions as well, and so that's what that higher order means in higher order Markov models. So we could also use deep reinforcement learning to further predict what the user is expected to do. And so we could do this over time, because we have that analytics data, we could, we could have a reinforcement learning. Agents learn over time. And, and, you know, either punish it or reinforce its behavior, that whatever the user does, it eventually matches what's the reinforcement learning agent predicts. And so we do this by providing it the environment, which is that finite state machine model of our application, and when we provided actions. Or sorry one. When the agent decides what it should do next. It's going to continuously use that data in order to predict what is the user going to do next. And so if it predicts correctly based on historical data, it gets a reward and then it approve. It improves itself over time.

So overall, you could see that there is a much brighter future to developing user interfaces. Other than just taking a design, coding it, shipping it and calling it a day, there's much more. We could do so when we create this executive a model. We could track these metrics automatically, and then we could feed it into things like Google analytics or Segment, Mixpanel, KISSMetrics. What's it called in Azure? Whatever I work from Microsoft, when I'm talking about Google this entire time, please don't tell Microsoft that. So you, you take that and you feed it back into adaptive, that these data transitions, and by that you make this adaptive user interface that adapts to your user. So this relates to the dev experience. When we use these abstract models, which are these state machines, we provide determinism. In knowing when we're in the state and this event happens, we know exactly which state it's always going to be in. By providing that structure in that machine object format, we could.

Automatically visualize these results. And so, of course, these are things that are very good for developers. But what's also good is that by having these visualizations, and especially having these metrics provided by analytics. We have communication between our designers and our stakeholders, and we're able to visually show them. This is how users behave through the application. Now we also have testability in which we could take these models, especially these models with analytics, and we could automatically generate tests based on that. I gave a talk at React Rally talking about model-based testing and how we could do this. So please look that up if you're interested in that. And we could also, of course, useMachine learning and reinforcement learning to create simulation of how our users are expected to behave in the app. Based on this abstract model. And of course, we could use analytics and greatly improve the user experience by creating these adaptive user interfaces that adapt to them.

So adaptive user interfaces are great because they creepies. Personalize systems for every user instead of having a one-size-fits-all system for everyone. They prevent information overload because you could design these adaptive user interfaces to hide things that some users might not care about, and to make more things prominence that the user is definitely going to care about. They do provide intuitive guidance for users who might not be used to using the system. And over time, this guidance can seamlessly disappear. So that power. Users can use it more efficiently. And it helps users understand tasks and goals as they're navigating through your application. There's a lot of modeling considerations that I want you to think about, and we're not going to go too deep into today. Such as micro versus macro activity, and that's do we do, we go on the micro level? Or do we adapt the application as a whole?

Things such as prediction versus feedback, local versus global adaptation? And whether we use data which is real user data which you want to be really careful about, or simulation data. Now, using statecharts is useful for so many other things, such as with NASA. The use statecharts to visualize the the auto. Or they use it to visualize the Curiosity Rover. The way that it behaved and the way that it was. Designed, and with that, they were able to create these precise diagrams, automatically generate code, provide complete testing coverage. And also they were able to adapt to people coming in and saying, we need this feature. We need to change this behavior without being scared of breaking anything. Now. There are disadvantages, too, in which learning statecharts is, of course, a learning curve, and it's something you have to teach your entire team. Modeling does require planning ahead, and you can't just jump into the code immediately. And not everything can be modeled perfectly into a state machine or a statechart just yet.

There's also complexity trade-offs, whether you use a finite state machine, which can exponentially grow in complexity over time. Using the traditional bottom-up method that we are used to coding, it just linearly grows in complexity, and statecharts might be too complex for simple systems, but over time it greatly reduces that complexity.

Play from 25:00

And so again, I'm working on XState and there's a whole ton of ton of features and tooling that I'm working on. I just released XState test, I'm going to be soon releasing this X8 analytics package. Some more help you manage analytics with X8 and other things such as view integration, visualization and simulation. And also there's a visualizer, which I feel is one of the most important points of XState. Where you could copy and paste these machine definitions and automatically visualize them and interact with them in real-time. There's a lot of resources online for you to take your user interfaces to the next level and make your code to so much more right. So what he was asking is, is there a problem? When you're like, with these user interfaces, there's the issue of they might be different for every user. And so that's definitely true. And that's one of the considerations on how much should you adapt the user interface. And it's still an open area of research.

I mean, this is really cutting edge material. What I would say is that with these analytics, and with having a system modeled in a very abstract but strict way, is that. You could take the analytics of a specific user. Such as, Oh, they're more likely to do this, this and that, and you could plug it in directly as a customer service. RePRESENTATIVe and say, Okay, I see exactly what they're seeing because I have their same analytics on, you know, what they're used to seeing. So they might be able to help you with things like that. But also, adaptive user interfaces are supposed to help the user, so not even involve a customer service person and just help the user itself. And so, ideally, eliminating about [Music] [Applause].

Play from 34:00