Mind Reading with Adaptive and Intelligent User Interfaces

All right, hello everyone. Sorry, but I got your attention anyway. I guess I'll get started because it's 4/20, it's about high time I got to start it anyway. Sorry, my name is David Khourshid. I lived here in Orlando. I'm very excited to be talking at a shopping mall, it's my dream. And they've been talking to you about. Mind-reading with intelligent and adaptive UI is also known as a you eyes. So just a little bit about me. I work at Microsoft, as you can tell by my computer, and that's everything. That's all you know about me. You could talk to me afterwards and we get to know each other. So the most important concept here is realizing the fact that all of your users are different. We all have different needs, every user uses your app in a different way. And so because of that, I say that easy tests are essentially effective because we're only testing.

Like two personas, where our users could span infinitely many personas, right? And also, user flows are not always linear, so there is no set age that these steps see. But often it branches out, and users could do different things depending on what goals and what tests they have to accomplish. And so this is the important realization. This is why I also think, like those funnel charts that you see in segments, I, elbows aren't always effective either. So Google has released a few of you tools, such as SJS, which really help adapt to individual users. And so gay people. If you haven't heard, guest Janice. It's basically a way of prefetching pages based on where the user has gone before. So it uses Google analytics and it is able to predict on an application that uses which page the user is going to visit. Net prefetch fat data so that it loads quickly for them now, google, at the latest Chrome Dev Summit, also released React adapted hooks, which is really, really interesting because it provides you a lot of hooks.

For measuring. Like, device memory usages, device power, device internet speed, whether you're using 2G 40, whether the user has requested limited data so that you can provide adapted user experiences for them. And so you're going to be seeing a lot more this coming in the future, this idea of. Adapted user interfaces so what do I mean by that good user interfaces? So when we think of responsive user interfaces, we think of user interfaces that respond according to the device dimensions, right? Adaptive user interfaces are different, they respond to the user. So depending on what your behavior is, the user interface might behave a little bit different. So one of the earliest examples is Clippy. So, as you remember, when you're using Microsoft word or Microsoft Excel, clip it with pop-up and contextual. We tell you, Hey, I think you're working on this, would you like some help with this? And so this is an example of an adaptive user interface, so consider that you're making like some sort of inbox application, right?

There's many different paths that you could take. We could view the latest emails, or we could compose a new email. You know, we could say that email as draft when we compose it, or we could send it. By the way, now users might do different behaviors, you know, in this inbox app, and so it's important to measure what users are actually doing in real time. Because we get it that the user interface to say, you know, you probably want to view the latest emails instead of composing a new one, so that's the view we should probably show you first. And also, if you're creating a new email, you might want to send it all the time instead of saving it as draft. And so that's why measuring these data points is important. And also, adaptive user basis can really help limit the number of sets your users need to take to accomplish a certain test. For example, if they. If you have a simple delete user flow where you asked, are you sure yes or no?

I'm sure a lot of us have implemented this. Then we might notice that the user always clicks yes. So in that case, what we want to show instead is just a simple deleted with the option to undo. And you see this in Gmail as well, where you have send and then it says, yeah, you send your message, but you have a pink 10:13 grace period, 10-15 second grace area to undo that. And so one of the more recent advancements in adaptive user interfaces that I use all the time. Is this? I don't know what you call it like? Swipe to autocomplete your email messages, and I use this so often. So whenever you get an email from me, it's probably Google talking to you because my can trafficking emails. So you said, all the time. Also a deaf. User interfaces are really good for power users who know what they want to do and they don't need all of these options or extra help, and they could have a streamlined workflow.

Whereas other users might need adaptive user interfaces to change and actually assist them, so developing these adaptive user interfaces is challenging for many reasons. I mean, we want to see which parts that we want to adapt, do we want to adapt the overall application, or just specific components? Do we want to give users the ability to adjust their preferences and manually adapt interfaces themselves? But besides all of this, the key is creating a model driven development workflow, you know, with your company, with your projects.

Play from 0:00

And what this means is that, instead of just like, what is it like just developing and writing code? Oh, moving past and breaking things. We're actually creating a model, and so we're not doing anything except planning out how we're going to program all of our features at the very beginning. And so creating a model looks a lot like creating these user flows, these user diagrams that you might have seen. Where we say this is going to be one state of our application. And the user could do this which takes into this state, or they could do something else which takes into another state. But this is all based on the mathematical principle called finite state machines, which, if you know me, I talk about this a lot. So I'm going to give you a quick review on what these are. A finite state machine has one initial state. That's a start state where everyone starts, and then there's a finite number of states which you could go to. Like the off state of the projector. I see Viking here doing something, which means something must be working.

It's just overheat. Is that one of the black Friday protectors? That's okay, I will make gestures while this is working. So, finite seed machine, you have one initial state, right? That's where every user. When they go into your application, that's a starting state, then you have a finite number of states, and these are the different possible states that your users could go to. And then clicking, you're not going to see this, but you have a finite number of events that can happen now. These events can be initiated by the user doing something like clicking a button, or they could be events such as a request resolving, or some failure happening, or some internal events. Right, are we back? We're back, awesome, thank you. Any questions? You also have a mapping of state transitions triggered by these events, and a finite number of final states as well. So you can essentially just look at this as states. It bends.

When an event happens, just follow the arrow, it's. It doesn't get simpler than that, so it's just like you could think of it as a flowchart, user flow, you know, diagram. And it also prevents impossible states such as this. So, I mean, we don't want to see this in our application. And state machines ensure that you're only in one state at any given time. And that when an event happens, it's always going to go to its designated next state so you can make finite state machines with switch-case statements. Because this is such a core concept in computer science and programming languages that you're basically matching on what the state is. Such as searching what the events that just occurred was such as a result event, and then you would return what the next state as a result of. But what I'd like to do is use object mapping, and in this case, we would have in a given object, which is the state.

And then another nested object where we map events to the next day. So when we're on the searching state and the result event happens now our in Pisa test date. So I created XState in order to handle this. And many more complex parts of state machines and statecharts, such as hierarchical stage parallel states, clear the defects, delays, interpretation, and a lot more. So, I encourage you to go to XCj org in order to you know to find out more about this. And so XState also encompasses principles from hierarchical state machines, also known as date charts. This. Is not a new idea. This was created by a computer scientist by the name of David Herold in 1987. So that's more than 30 years ago, and these ideas are still used widely today in many other programming languages. Also, XT can be used in view, Angular, spelt. React anywhere, where you can basically subscribe to an object and send data to that object as well.

Play from 7:00

It's a very, very simple paradigm. So let's take this in an example, let's say that we have a feedback form, right? We say, how was your experience? The user can click good or bad, and if the user clicks tab, we take them to a screen that asks. Then tell them why you had that experience. And then it. I wha. Regardless of if you click good or you submit the form, it'll say thank you for your feedback and then you could close the form. So this is how it looks like. You know we have a good or bad, you click good, and then you know if you click bad. You could just write feedback there and so you can see. We can describe this as a simple state machine, but I want to make it a dot, right? So let's let's come up with a game plan for how we can make this adaptive? The first step is to create an abstract model.

And that's the state machine that I'm talking about. And then we want to have analytics on those transitions and measure how often users are going from one state to another and when they're doing so. Then we identified be adapted paths, which are the past, where we could say, you know what if a user for first this more, let's take them there instead. And then we could use the analysis from step two in that automatic adaptation in real time. So there's a few concepts we need to learn in order to do this. First is weighted raps. So a graph is just a collection of vertices and edges, or notes and edges. And so you could think of you to notice one of these dates, like signing in, success or error. And one of the edges as these arrows. But here's the thing these edges can be weighted, which means that one can be traversed more often or however else you want to.

Measure the wage, like in a time it takes to get from one to the other. And so we could have what's called a weighted graph in which one edge has more weights than the other edge. And so this is the important realization in designing your you eyes, because you might think of your you eyes now as a collection of states and transitions. However, the reality is, when your users use it, it looks a lot different. So it looks like there are there are paths that are traversal more by your users than other paths. So I use this technique when I was working with the VIA SlideShare team, and I used eight machines in the. How many of you have used last year before? Actually awesome, this half of the room, I'll know what's wrong with this at live. Sheer is a way for you to collaboratively code between two or more people from anywhere in the world.

And BS code, so it's a really, really cool tool that you should check out anyway. We had a lot of complex UI states, and I wanted to first simplify all the code that prevented, you know, impossible states from happening. And make sure that we. We have a very succinctly way of representing what each state can be UI, which is BS code, would show.

Play from 12:00

But I realized, too, that I could add analytics or I could add telemetry to this. And so as a result, this is a power bi representation taken from the data collected from those automatics, automatically from those seat transitions. And we saw that we've created this. Waited breath and saw whether people were signing in externally more, or were they signing in directly, whether more people were sharing or joining sessions. And we could just analyze the data as is. So let's create our adaptive user interface now that we know about weighted graphs. So I want to have this bad button be an adaptive path, where when we click bad, it are shows in the form. Or it takes them to the Scampi for your feedback, where they could optionally give feedback. And so the reasons that we would want to do this is here. Is that Wow, in two minute, anyway? It's because, let's say that as a user, I click bad and then I see this form.

And I'm like, you know what, I really don't feel like filling this out, so I'll just exit out of it now when I reload the page. What I'm doing right now is I'm entering these transitions. So right now, there are very, very simple measurements. And so let's say that I just keep clicking bad, I keep exiting out like I'm having a real bad experience over here. The idea is that by automatically collecting these transition metrics, what we could do is we could adapt the user interface in real time. So that when it shows again, it will actually take us to this new screen. And then this one will optionally give us the beyond the option to provide feedback manually.

Play from 15:00

And so that's an example of a very simple example of an adaptive UI, so we can analyze these transitions based on how either taken. And let's say that we're meeting. An Instagram application, for instance, we could say that when we first log in, that the user wants to go to the camera. More so, we it measure just the way to the different paths and say, you know what? Open the Cameron provided direct path to show the camera instead of the gallery of the profile. We could also use decision trees in order to help us to, and so these decision trees measures different data points and allow us to make more granular decisions based on these different data points. And this is a technique used in machine learning and reinforcement learning a lot. So this is something that I do encourage you to look up. I'm not going to spend too much time because I have like three minutes left or something anyway.

These analytics one of the best things about using state machines in statecharts in your in your application is that managing state only falls into one single place. Your transition function. And what that means is that we could track our analytics in one single place. How many of you have had to add analytics, just like telemetry, throughout your entire app? Just copying and pasting? I mean, that's a really, really annoying process. So having it in one place makes things a lot simpler. Where you would track what state that you're going to, what state you came from, the events where you. Would also sanitize that, or in a preventing user information from spreading, and also have a time stamp just to see one method that's occurred. And that's all the information you need, and so from that, you could have very detailed data on how your users are using your application. So I thought that future ideas for analysis too.

And I looked up higher-order mark of models which are which are definitely interesting. Because when researching this, I realized a Markov model is basically a. It's a linked chain, right between states. And so, let's say that a user, or users in general, would go from state A to state B more often than going to C. So, while this may be true, another important consideration is which state they came from before. So let's say that coming from state D rather than state II, they were more likely to go to C. So then, in order to prevent faulty predictions, we should use these extended Markov models to predict based on two or more states. Also, I experimented with using deep reinforcement learning, taking those analytics, and actually having some sort of agent predict what the next state is going to be, and so reinforcement learning. Just a quick five-second intro, it's basically you have an agent. An agent takes a potentially random action, and based on whether we think that action was a good or bad action, we reward or we punish that agent.

And eventually, the agent, after many times, learns what the what the correct thing to do is, and it would do the behavior that we expect. And so you can see this agents within the environments which is your model and make it predict what the user will do next. Even for hats and different states that have never been taken before. And so you might be thinking, wait, seriously, I don't have time to really show you with demo. So I'm going to show you this video of a very simple tic-tac-toe game. Now this detect, Oh, I made using React and XState, and because of that I was able to analyze all of the movements that users made. And I put this a few hundred times so we can see that there's a frequency graph on each of the squares that shows what the next ones were played most. Often now, instead of playing them manually, I actually sped it to a reinforcement learning agent right in the browser.

And it was able to play like a human, even for so you can see it. Just please stay some frequency. But when it encounters something that's never been played before, it will still play like a human. And in this case, I mean, stupid move so at one. And so that was just using reinforcement learning in the browser to predict that user behavior. So in long story short, if we have an application, we can model that application. Using this abstract model created, create an executable model out of it. In this case, a model is a state machine. Use metrics tracking, like we've showed and collect that within, you know, say, Google analytics or KISSMetrics mix handle segment. I owe and feed that back into the model to have adaptive transitions and in results, we have this adaptive UI that responds to user behavior in real time. And so this idea of using abstract models really, really benefits the user.

Because for the developer, we had this determinism. We have the ability to visualize our models and communicate between designers and other stakeholders. But with users, we have this analytics in order to improve a user experience for them in real time. Without having to have long def cycles. Or these specialized UI's for different groups of users. And so XState I now have this XP analytics package, which I still have to write a documentation for. That allows you to I capture those metrics directly. And also the next version soon. But you could try that on XState at X if you are using XState. And there is also a really cool visualizer which you could copy/paste your machine definitions into and play around with them. And so, yeah, long story short, lots of resources for you, and the goal of all of this is to make your code do more so.

Thank you, Deb. I don't see the other speaker so many questions. All right, my name is Hugo, I'm a UX designer. My question is, what's the alternative to a B testing is? I said, like, a testing is sort of like your, your pigeonholing, like two differents, like possible views. So instead of that, just realize all right. Let's, let's create a possible way for users to use this user interface, or like a separate variation as that user interface. And so that's the tricky thing about. A depth of UI is just like picking those designs, and, you know, just choosing the right ones for your application. So with the death of user interfaces, you really want to see how your users are using the app. Are they running into errors? Do you see them making the same actions over and over again? Those are points where you could observe and you could say, you know what?

That could use an adaptive transition. As in, for these types of users, they could go here, for other types of users, they could go here, for others, they could go here. But this isn't pasted randomly. A be testing is random, it's like, 50% goes here, 50% goes here. Adaptive transitions are more these users who are more prone to do this, go here or here, or here. So it's more like an A to Z rather than a B testing, but based on the end oolitic, not based on random. You know, chance. All right, well, that's it. Thank you very much.

Play from 17:00