# Mind-Reading with Adaptive and Intelligent User Interfaces

Canonical: https://davidk.dev/talks/mind-reading-with-adaptive-and-intelligent-user-interfaces-nordic-js-2019/
Video: https://www.youtube.com/watch?v=e69vvcqxj6U
Speaker: David Khourshid (@DavidKPiano), creator of XState
Type: Conference talk · Event: Nordic.js 2019 · Year: 2019 · Duration: 24:23 · Views: 2596 (as of 2026-10-19)
Topics: Adaptive & intelligent UIs, Statecharts, Forms & UX, Visual modeling

## Summary
I was here two years ago and I was talking about user interface animations, and now I'm going to be talking to you about the future of user interfaces. Power users, whereas other users might need more help in adaptive user interfaces can recognize those users and be able to provide them assistive steps.

## Key takeaways (quoted from the transcript)
- I was here two years ago and I was talking about user interface animations, and now I'm going to be talking to you about the future of user interfaces. (0:00)
- Power users, whereas other users might need more help in adaptive user interfaces can recognize those users and be able to provide them assistive steps. (5:00)
- We could feed this into this reinforcement loop and improve our analytics over time in order to create adaptive user interfaces that learn from the users based on data that's fed. (19:00)
- I just released a testing library and I'm going to soon be releasing in analytics library to help out with creating all of these analytics and adding adaptive user interfaces to your applications. (23:00)

## Chapters
- 0:00 Introduction
- 11:00 Let's say that the user clicks bad because they're having a…
- 15:00 Instead, we could use weights to help us determine which paths…
- 21:00 Besides providing those analytics…
- 23:00 I just released a testing library and I'm going to soon…

## Transcript
[0:00] [Applause] so I am so glad to be in Sweden again. I was here two years ago and I was talking about user interface animations, and now I'm going to be talking to you about the future of user interfaces. Because, I suppose, right, it's not about just the tools, but it's about the community and what we enable and build for our users. So I want to talk about how we could actually enable our UI's to sort of read the minds of users.

[0:00] And to enable better features and more enhanced workflows for everyone, regardless of capability or where they come from, or anything like that. So just to give you an example of the UI so that we have today, we look at it in a very simple way. For example, we might have a button where you click it fetches data, we wait for the data to come and then you know what eventually comes. However, users, we need to consider the way that they behave with our UI.

[1:00] So, for instance, me, I get a little bit impatient when it takes a long time to load, so I might have a loading spinner. That sort of shows were in the process of getting your data and eventually it comes back now. This might be fine for most use cases, but especially if the user is me, I get really impatient. And so I will click that button many times, thinking that, for some reason, the data will come back faster than ever does.

[1:00] But you're a web developer, or a lot of you are, and so you think I'm smart. I will disable the button and that way I can't click it too many times, but I'm also a web developer and I keep going. The Dev tools under stabled that button and make as many requests as I want. The point is that everyone is different in the way that we behave with user interfaces might not be the same amongst all of us.

[2:00] And that's why, in my view, A B tests are sort of ineffective, because if we don't have a B tests, then we're shoehorning all of our users into using the UI in one specific way, but with a B tests? Now, they're choosing between two ways, each one of which might not be the most optimal way that you. Could present the UI to the user. So, in short, one size doesn't fit all, and we can't look at our user flows as to this linear path of.

[2:00] We assume that they're going to login, go to the dashboard, execute tasks and do something else. In reality, our user interfaces are a lot more complex than that. This was actually inspired by me walking around IKEA. So there are tools that exist for this, such as guest J/s and guest. ES. Uses predictive analytics to prefetch webpages in real time as users use them. And it uses Google analytics to just to measure which page is a user going to visit, depending on which page they're on first.

[3:00] And based on this, it could provide a more streamlined user experience by loading the page ahead of time for the user. So this is an example of what we call an adaptive user interface, also known as a UI. Now, some of you might have heard the term responsive user interfaces, and you might think, all right, adaptive sort of sounds like responsive. But there is a main difference, and that's that responsive user interfaces adapt to the content size, so the size of your device.

[3:00] Or like, if you're using an Apple watch or a phone, or maybe if you're using your computer in portrait or landscape. I don't know if you could turn your laptop around, but it's usually landscape anyway. So it responds to users and not to devices. So think of a responsive design where it's based on how we behave and how we are as an individual. One of the earliest, one of the earliest forms of an adaptive user interface you might remember is Clippy.

[4:00] And we also have things such as Cortana, Alexa and Google home as well. That it's just personalized based on the way that you are, so the way adaptive user interfaces work is. We have this mapping in our application of where users can go and what they can do, and so. Based on this, we collect analytics over time and determine what are the most likely paths for the user. So we could see use cases of these adaptive user interfaces.

[4:00] In things such as this flow, where you have a delete button, you could either confirm or say I don't want it to lead it, but this. Is two steps for the user. If we are able to predict that the user is always going to confirm, or most of the time confirm that. They wanted something deleted, then we could shortcut this instead and say, instead, just delete it and just give them the option to undo.

[5:00] And so it's sort of the best of both worlds. We make it easier for the user, but we also provide them a way to choose the alternative option. One example of an of an adaptive user interface two is using Gmail. I don't know if you've noticed, but it actually predicts what you're going to say next in. I've used this so many times. So if you see me using the same responses in my email, so it's probably because I'm just doing this and I'm letting Gmail complete it for me.

[5:00] And also on texting with Google, it does it the same way. I work from Microsoft, not Google, so I don't know why I'm talking about Google so much. Please don't tell Microsoft. Also, adapt Abuser interfaces make it more streamlined for experienced users to use the app in a way that they find more efficient. So, power users, whereas other users might need more help in adaptive user interfaces can recognize those users and be able to provide them assistive steps.

[6:00] However, developing these adaptive user interfaces is challenging. It's not so easy to know which parts should be to make adaptive, which parts should we hide and show. How much control should we get to the user versus automatically adapting the user in their faces to fit their needs? And I have a solution for this, and it's a solution that has, you know, been around for a few decades. It's this idea called model driven development, and so what is model driven development exactly?

[6:00] It starts with realizing that our applications are really these user flows of. We have certain states that our app can being in certain ways, that users can transition between these states, much like this user flow diagram. More formally, this is known as a finite state machine, which is a mathematical model that model states and transitions together just a summary of a finite state machine. It's five different parts. We have an initial states, and that's a state that you start your app in, or your.

[7:00] System or project or whatever, and it's the very first states that the user is going to see. We also have a finite number of states, and these are all the other possible enumerated states that the app might end up in. We also have a series of events that can happen through the app. That transition between each one of these states, and we could also denote some states as final states, meaning that the flow is done and we can no longer transition from these states.

[7:00] Such as when the user signs out from the application. So there's many different ways to actually create these finite state machines, the traditional one is by using a switch case statement. It's a little bit verbose, but it is the way that you know, if you want to use it between languages, this is going to be your most common way. So, in short, a switch case statement. We could say which state were in, such as the searching state.

[8:00] We could figure out which events just happened, such as the resolved events. And then from there we determine what the next data is going to be, such as the success state. However, I like to use object mapping just because it's a little bit more concise, although it requires a tiny bit more code to actually navigate through this object, so it's sort of the same way as a switch statement. We have our states and an object, and then we have this mapping of states to other states based on events.

[8:00] And then so we just determine which event just happened. Look it up on the object. And if that state doesn't handle that offense, then we just defaults to that same states. I created a library called XState that makes this process a little bit easier, and also deals with hierarchical states, parallel states and other statechart features. As all talk about soon. But it's essentially the same format you pass that object into a machine and then you do the exact same thing.

[9:00] You have a state, it looks at which event occurs, and it tells you what the next state is. You could use X8 with React, where you just pull it in and then you use it as a hook. I know many of you are using reacts, but you might be using. Angular spelts View XState also works with all these things. But I do want to mention that you do not need a library to do anything that we're talking about in this talk.

[9:00] It's just the general idea of using state machines and statecharts to model our user interfaces, so statecharts real quick. It enables a few more things, such as actions guards, which allow you to prevent certain transitions from happening. It allows you to nest states together, to make orthogonal regions, and to provide history states. Which not something I'm going to go into in this talk, but I will provide resources at the end now.

[10:00] Statecharts were invented by a computer scientist by the name of David Harel in 1987 in order to simplify the combinatorial explosion of states that he saw happening with typical finite state machines. So one way that hierarchical states simplify is by grouping similar behavior together, for instance, in this state machine. We know that in the success or failure state, when we search, we go back to the searching states. And so this could be represented by nesting B states right into the search states.

[10:00] This sort of parent states in defining common behavior for both of those states, and then we could define that on the search events, regardless of which child state were in, we transitioned back to searching. So let's take a practical example. Let's say that I have a feedback form where I ask, how was your experience?

[11:00] You could just imagine the sort of popping up in your app. You say it was good or bad, and if it was bad, I want to know about it. So you tell me about it. And then you press submit, and whether you press good or bad, it says, thank you. So, just in short, the way this works. Using the same sort of finite state machine approach, we could say good, we could say bad and fill out the form.

[11:00] No, and yeah. So how can we make this type of user interface adaptive? So what we could do is first create an abstract model, which we did, this finite state machine representation of our app. We could use analytics based on the transitions to determine which paths can be potentially, potentially adaptive, and then we could use analysis to enable this adaptation. In our app in real time. So there's a few tools that we need to make this happen.

[12:00] One is weighted graphs and a weighted graph if you know whether the directed graph is. If you don't have been talked to by the H.E. Joshi, who does B.CS, because she has amazing articles written on all these topics. So a weighted graph is basically a way of saying the user is more likely to go from this state to this other states than this the state right here. So, for example, we see that 90% of the time we transition from signing into success rather than to error.

[12:00] Now, the realization is that our user interfaces all paths are not created equal. The user might prefer using a certain path in a certain flow throughout the app more often than they take the other paths. So I did this in visual Studio live CHER, where we were modeling the user interface inside the VS code itself. And so if we have a state machine. And we were determining, like, how many users joined, how many users for creating sessions, how many users were joining sessions.

[13:00] Then we could automatically feed this data into a D3 diagram and visualize what are the most common paths for users to take? So let's do this in real time with this application. Let's say that, for example, I want to know, oops, I want to know whether we should show the form or not, and so right now we just have an empty form. So let's say that the user clicks bad because they're having a bad experience, and let's say that they see this form and they don't really want to fill it out.

[13:00] What we could do is we could track those analytics and track that from the question state they clicks bad, but from the form states. They just closed it out and so we have this close right here. And then let's say that the user continues to have a bad experience, but is just completely alienating. This, this feedback form and so we could see when we click bad escape, all right. So right now we see that the user has clicked close on the form and didn't really complete the form.

[14:00] So what could we do instead? We could measure that and say that once it reaches. Past a certain threshold, when they click bad, now realize none of the code has changed. It takes you directly to the thank you screen and instead provides you a button to provide feedback. And then you could click it, go back to the form screen. And then we could see that if the user keeps doing that, eventually it will show the form by default again.

[14:00] So that's one example of an adaptive user interface. So there's many ways that we could do this sort of analysis, we could, in lies, based on transitions.

[15:00] Take the sample Instagram app, for instance, you know, when you go onto Instagram, it takes you to this profile screen first, or maybe a gallery, or maybe your camera. What we could do is we could determine which path is most likely for the user to take. In this case, it's the camera. And so we could use those analytics, feed it back directly into the app and determine that we should probably open the camera first.

[15:00] When the user opens Instagram. We could also analyze based on events and sort of gives to what we want the user to do and optimize for that behavior. For example, if we want the user to click on that and please don't use this for ads, I'm just saying this as an example, but let's say that if ads are interspersed, if they scroll. We want to assign a negative score because that's not the behavior we're trying to encourage, but if they do click on an ad, then we assign a positive score.

[16:00] And so we could say that depending on whether that's our intersperse. Or showing on top, which one they respond to more and which ones are more likely to click. We could determine which kind of ads showing we wants to show in theirs first are shown at the top. So we could use decision trees to help us to, because there's a whole ton of parameters that go into making these sort of decisions, and decision trees help with that.

[16:00] Distances and trees are a visual model for while making decisions, it's a statistical model too. So we could say that based on user information, like how new they are or how many friends they are. We could say that we should either show an ad at the top or. We should show less ads because they're more likely to click an ad if it's not too common. And so we could make determinations based on that.

[16:00] So, shortest paths can also help us with prediction, because we can't just look at each path in isolation. Instead, we could use weights to help us determine which paths the user is more likely to take, for instance. Or actually, well, in next states. You could have in the on transition method, you could track how the states are going to transition, and you could determine the target, the source, the event, and the time stamp.

[17:00] And feed all of that information into analytics, whether you're using Google analytics, KISSMetrics, segmented IO, things like that. So based on those weights, we could determine what the shortest paths are, and those paths might not be what you expect. For instance, given these measurements, we see that the shortest path from A to C is X ADEC, and that's ABC, but these shortest paths aren't. The weights for the shortest path needs to be determined in sort of a broader scope.

[18:00] For example, let's say that we're going from state A to state B or C, and we determined that its more common for the user to go from A to B. This might not tell the full story because, depending on which state the user came from, it might be more likely that they might travel to a different states. So these are just things that you know you should research and keep in mind.

[18:00] And we could also use deep learning in reinforcement learning to further assist predicting what the user is going to do. For instance, we could see that the path travelled by the user over here is the actual path, and we could have some sort of reinforcement. Learning. Agents try to model what the user is going to do and assign positive rewards or negative rewards. Based on whether it made the correct prediction or not, and then, over time, it will predict correctly what the user is going to do.

[19:00] So we could now use more simulation rather than getting actual user feedback. And so we could feed this into this reinforcement loop and improve our analytics over time in order to create adaptive user interfaces that learn from the users based on data that's fed. Into Google analytics, Mixpanel, KISSMetrics, Segmented IO, and this could be done in real time as well. So this is all based on the idea that we're using software modeling to help not only develop our apps, but also to improve the user experience in real time.

[19:00] And software modeling has a lot of these benefits, like visualization of these graphs, communication between designers, developers and other stakeholders. It automatically can generate tests for us, it allows for simulation based on those machine learning and artificial intelligence models, and finally, it provides analytics that we could use in real time. So, these adaptive user interface allow us to create personalized systems for users, prevent information overload in simplifying the actions that users do.

[20:00] Provide intuitive guidance for users who might be new to the applications you're creating, and help the users understand the tasks and goals necessary for them to complete. There's a lot of considerations when thinking about adapted user interfaces, for example. Do we want to adapt on the component level or on the app wide level? Do we want to adapt on route, or do we want to adapt based on a specific, smaller user flows?

[20:00] And then there's the idea of prediction versus feedback. Prediction is great because the user interface sort of works magically for the user. But feedback can also be a useful tool because the user doesn't want to be interacting with some magical black box. Instead, they might want to have control over how the app adapts to them.

[21:00] And then there's data versus simulation should we be collecting real data, should we instead be simulating and basing it on previous data? So that we protect the users privacy and things like that? So there's many advantages of using software modeling and statecharts. In the example I'd like to point to most is the Mars Rover by NASA, which, by the way, this was a 100 billion dollar project. Imagine if you had one shot at deploying something to production, and if anything went wrong, you just wasted a hundred billion dollars for your company.

[21:00] I mean, this is something that you really don't want to do using statecharts and software modeling. Besides providing those analytics, it provided visual modeling and precise diagrams for even non-technical people to see what kind of logic and behavior this Rover was going. To have it provided automatic code generation, which is just a safer approach, especially when you have millions of lines of code. It provided comprehensive test coverage, which meant that no matter what happened, they were able to predict any possible situation that this Rover can be in.

[22:00] And also late-breaking requirement. Some changes were automatically adapted in that, like when someone made a change or when someone requested a new feature or some changed behavior. They were able to see which parts of the model were going to change as a result. So, the visualizer III made 4X state consumes. These types of abstract models, which you can make executable by adding into your application, and so these can be visualized directly.

[22:00] Just like this, I'm just short on time, so you could just check that out later. The exit eco system, which I've created for software modeling and front-end applications, but also back in applications in anything that uses JavaScript, is growing.

[23:00] I just released a testing library and I'm going to soon be releasing in analytics library to help out with creating all of these analytics and adding adaptive user interfaces to your applications. So there's a lot of resources, such as the world of statecharts, which is just a great explanation of how statecharts are used in practice. And all of the terms and things to learn. There's a spectrum community to where we talk about this.

[23:00] And of course, I have documentation, because you have to have documentation. So, in short, I want to make your code do more. I want your code that you write to do more than just executing a task or accomplishing some sort of business requirements. But I want your code. To actually understand what the users individual needs are. And be able to adapt user interfaces, automatically generate tests, be able to be visualized in so much more.

[24:00] I have one minute left, but I wish I went overtime because they said the bubble machine would like turn on. So anyway, thank you, Nordic Jas. [Applause].