Write Fewer Tests! From Automation to Autogeneration

Thank you! So by the way, let me just clear something up, my last name is not piano, so I just wanted to make that clear. I'm really excited to be at React Rally. This is the fourth time that I've been here, attended twice and I talked in 2017 about an equally boring topic, which was state machines, but thankfully I might or might not talk about state machines again today. We're talking about the terrible concept known as testing. I think I could speak for everyone or mostly everyone, in saying that testing is terrible, and it's a painful thing that none of us want to do, and apparently it's a controversial topic, too, but at least I'm not talking about CSS so we can just move on to testing and it's that a lot of people have really strong feelings about testing and my theory is that we really have strong feelings about writing tests, not necessarily testing itself.

So I put out a poll and I wanted to know what people disliked most about testing applications that they've written and the majority of people just absolutely hate writing tests and a close second is updating and maintaining those tests, as well, and so what I want to talk about today is how we could fix this first point. And not by making testing more enjoyable to write, but just eliminating it altogether and at the same time we're going to be covering updating, determining why tests fail, and discovering edge cases, as well. So if you take a look at your codebases, you'll see that as requirements grow, your lines of code grow, too, and it is a mathematical fact that the lines of code needed to write tests are so much greater than the lines of code that you write in your application, because if this is not the case, then you are missing tests. So as requirements grow, your tests grow, I wasn't able to fit it on the screen, but they grow exponentially.

And the problem with this, too, is that requirements, you see this curve, it slows down, because they say, all right, they have enough tests to write a lot of code to write, and so you're not able to get features and instead you're spending time writing like all these lines of code, instead of doing all the feature work that you should be doing. So there's two schools of thought for this. You could either write a ton of tests, as many as possible, I call this the write tests frequently technique and it's by Dan so you know it's a good technique, or you could be like Ryan Florence and just write code perfectly the first time, which apparently is what he does.

Or like Michael Jackson who wrote the Expects library, you could write a unit testing library and encompass techniques like red, green, refactor or well, because it's Michael Jackson, it's black and white refactor instead. Funny enough, his repo is broken, but that's just human nature. I'm sorry, I'm sorry. I see some of you walking out now, I totally get it. So here's what we were doing. Before test automation became a thing. We manually tested the code. A lot of you remember the pain of this. We would write the code and then we would look at the business requirements and we would manually execute our application clicking things and typing things, and just making sure that our code worked, or better yet, we would hand it off to QA, who always found the way to break our app, no matter how good our code was. Thankfully, this got a bit easier when we started automating our tests, because instead, we could write these unit and integration tests for our code and hopefully have them write.

But what we're going to talk about today is modeling the code so that we could automatically generate exhaustive tests for every possible thing the user to do in the app. And this isn't the only level. There's something called formal methods where you could formally prove that it's impossible for your code to have bugs. This is actually a thing, and some companies do this. But the highest level of enlightenment which maybe I'll talk about one day is just delete your code, because -- [laughter] then you don't have to write any more tests. So this is a testing trophy and it talks about the importance of tests. We have static tests where, you know, this comes from types, whether you're using TypeScript or if you're use flow for whatever reason, I don't judge you. We see that integration and end to end tests are the most important tests. Ideally this is what our test structure looks like, but in our apps it's less after testing trophy and more of a testing ribbon, because it's more like "you tried." What I find a little ridiculous too, is the fact that the code that we write, we're writing the tests for our own code he so we're sort of validating ourself which doesn't make sense we're patting ourself on the back and I can't think of any other industry where mission-critical safety code that might endanger the lives of others is tested by the exact same people who wrote the code. [laughter] I can't -- I'm sorry, that might be too soon.

I didn't know if that was gonna fly. Sorry. So here's my thinking.

Play from 0:00

Instead of writing all these tests, what if they could be generated instead of being written? And what if these tests can automatically regenerate themselves without us having to rewrite the tests? So this is the idea behind model based testing and just like state machines, this is not a new idea. it's a bit younger than state machines, it's about 20 years old give or take where the first research paper of 1999 was written. And ever since other industries have adopted it in some level. There's open source tools available for this, but just like state machines it's something that we're not too familiar with in the frontend and frankly the native mobile web dev technology industry that we're in. So let's do a puzzle over here, how many of you have seen die hard. So you might remember this puzzle from 3 and 5 gallon jugs, how do you measure 4 gallons exactly?

You could power water into one, you could transfer water from one to the other, it's a nice tricky puzzle. So let's try to figure out a solution. What could we do? We could -- let's fill three up, right? And then we could transfer that to the 5-gallon jug and then we could fill three up again. This is just one of the solutions that I, you know, sort of came up with. And then I was thinking, we could take the 3-gallon jug, put it in the 5, which gives us 1 gallon in the 3, and then we could empty the 5-gallon jug, power the 1-gallon into the 5-gallon and then some of you see it coming, we have three gallons and we power it into the 5 and there we go, we have 4 gallons.

So speaking of water. Water break. What does this look like? It looks like a state machine, right? I mean, I don't know if I'm taking crazy pills or not, but that definitely, you know, looks like a state machine and in fact we could represent this entire thing as a state machine. So if you didn't see my talk two years ago, quick review: You could represent your app or your components or whatever as a state machine in which we have an initial state, multiple states, events that trigger transitions between states, and possibly final states, too. So that was my, you know, last talk summed up in 20 seconds. All right, so this jug game can be represented as this finite state machine. And because of this, the solution is just following the path that leads us to one of the final states. And so this is one of two solutions for this, and you could expand this to similar puzzles of that type. So the important thing is realizing that a lot of things, whether it's business requirements or your component logic, or even your app logic at a higher level, is a state machine.

Even biological process, even you are a state machine.

Play from 6:00

It's a very important realization. So let's talk about actually turning these business requirements into a sort of state machine. You might have seen given, when, then as sort of a format for writing these requirements, such as given a user is on the home page when they click the dashboard link, they should be on the dashboard, so we have a precondition, being on the home page, an action, which is clicking the button, and a post condition, which is being somewhere else. So this could be represented exactly like a state machine and so all we have to do is turn that state machine into executable tests by adding assertions on each of the preconditions and post conditions, and adding some sort of execution mechanism for the action, like basically clicking a button or typing in some text in an input. So here's how we do model-based testing in five easy steps. First step is you create a model of your application, and so this is going to be the most important step and it's useful not just in model-based testing, but in application design in general.

So represent your app as these dots. Call them testing dots, call them Kent C. Dots, whatever you want. All right, I'll stop with the puns, sorry. From there you generate abstract tests and all this is doing is path-finding, so we're finding all the paths from one state, our initial state, to ever which other state in our application. From there you make them real. You associate transitions with action, such as a mouse click, keyboard input, some time delay, or anything else. So we have a precondition, the action happens, and then we test the post-condition. All these need to be within the tests because we need to actually execute them and make them real because our abstract tests do nothing until we add these concreatizations of the tests. So then we execute the tests. The tests will quite possible fail: What this means is one of two things. Either our model is wrong and doesn't describe the application exactly or our application is broken and so it's up to you to figure out which one it is.

But then, if your tests pass, once you fix everything you could seriously profit and what I mean by this is you could literally save money. Because you could change the model. When that happens you don't need to rewrite any of the tests, the model will regenerate the tests for you, because again it's just an algorithm that traces each state in each transition and tries to go to every other state. So let's do a really quick example over here. Let's say I have this very simple feedback app where if you click good, it goes to things for your feedback, if you click bad, it goes to this form where you know, you have to enter why your experience was bad and then when you click submit, it goes back to that thanks for your feedback screen. So this could be done manually, and I'm using the amazing testing library of React's library by Kent C.

Dodds, as well, so we're pulling things from the testing library and we are doing three things: Remember we're doing our precondition which is making sure we're in a certain screen, we're doing an action which is making sure that an action that takes us to our post condition is executed, and then we're actually checking that post condition. But this is a lot. It's like we have two full flows and that's already like 30 lines of code, just to do two flows you know, making sure we go to the think screen and this one is going to the form screen, as well. So instead I'm proposing an alternate approach. What if we represented this as a finite state machine and I'm using my X did library to do it so when I'm in the question state and I click bad, it takes me to the form state and so anyone whether you're technical or not, can understand this state machine and be able to trace through and verify that the state machine matches the business requirements of your app.

So then we get to the fun part, which is generating all of these different paths that take us from one state to the other through our application. In this case I'm generating the shortest paths and you could imagine a big long JSON output that outputs all of these for each state, and so for example, there's two ways to get to the thanks screen, you could click bad and or. Because we're representing it as a graph, we are exhaustively checking all the different ways we could get to each of these states. And so, what we could do now is instead of using this graph library to generate these, I'm going to use XState test which is a brand new library that I literally just released this morning, deploying it on Friday is a bad idea. I don't want any bugs or anything. We could add each of these assertions to each of the states and as you can see, there's one to the question state and that's just verifying, are we really on the question state?

So with React testing library it becomes a nice one-liner where you could use expect, I'm using assert. I'm just making sure we do have the question screen on there. Now all we have do is create a model using this testing library and that model is going to do all the complicated algorithmic work of generating all of our paths for us, but one important step: We have to make sure we could make the events real and we could associate, for example, click good, we associate that just to clicking the good button. And so we could also specify test cases, as well, if we want to do things in the future, such as validation, like not letting them submit if they've entered nothing in. And so using this testing library, what we could do is generate test plans that give us all of the paths for each one of the states that we're trying to go to. And then we could iterate through those paths, do the setup, and then test those paths.

So I'm actually going to show you what that looks like, but of course the final important thing is making sure that we reach every single state, and that's what this that test coverage function is for. So let's do a demo. So I have my feedback app over here, and I'm doing exactly the code that you saw on the screen, because I literally copy and pasted, so it should be the same code, and so let me just show you what this looks like real quick, just to convince that you it is actually a real app. So we have a very simple screen and of course it does exactly as we expect. But this is us manually testing. Using React testing library instead, we can verify over here, and using the testing library it's going to generate those paths and test every single state, and test every single transition, as well, making sure that we could reach all of these states, and just to prove that this does in fact reflect our application business logic, let's say that we forgot to add the submit button on the form.

Now it's going to fail. Don't mind this. This is just barfing all over the place. But look up ahead! [laughter] we had the path and we could tell exactly where it failed. So you could copy and paste this and you instantly have a test reproduction that you could share with anyone who cares to see that reproduction. So when you're on the form state, you could see that submit button, that events failed to execute, and so when we add it back in, the tests will pass again. So [applause] so there's many ways to generate the paths. There's shortest paths, which are literally just the quickest way to get from one state to the other. But just like you might know from driving or anything, the shortest path is not necessarily the shortest distance. In fact, we could represent shortest paths by weight and we could, for instance, determine that users take a different path more often, and so we want to place more importance on those paths for testing and you could use something like Dijkstra's algorithm.

I typed this like five times before I finally got it right, by the way. Or we could do simple paths which are much more exhaustive. Simple paths are all the pacts that it takes to get from one state to another state without repeating cycles. So in this case there's a few of these. So you also have to do is trace a path and you'll see. So this is much more exhaustive and it's going to create many, many more paths. So, you might be thinking, like, all right, you know, I see it in the tests, but is this limited to just, you know, testing within React testing library and the answer is no. If we take that exact same testing model and instead, add, say, Puppeteer to it. Let's see what happens. I hope this works. So instead of doing React testing library, it's going to open up a version and literally go through every single one of those paths, executing them on a real browser and so this is going to be your full experience, and we have all the tests passing.

So... [applause] all right. So you could sort of think of this as time travel debugging, I know it was talked about, but this is a little bit different. Instead of going back in time, now we're seeing all possible future events, and so this is really powerful. [laughter] all right, so what does this have to do with React exactly? Absolutely nothing! And that's the point. You could do this in Vue, Angular, it doesn't matter what you write in our application in, we are describing our application in this abstract model that is implementation agnostic, which means that we could use the same testing model for anything we implement our application in. And so this is a pretty powerful concept. And real quick, I wanted to have fun with how many of these paths are generated, just because over here it's generating 12 paths because we have a fairly simple application, but what if we have an application that's more in depth? So over here, I modeled a tic-tac-toe game.

Play from 9:00

The machine is over here. It's a pretty simple machine. This is what it looks like. Don't know if you can see that, but we have three states playing winner. And so you know, we have that, oh, is this going to tie? Yeah, it's gonna tie, but anyway, you get the idea. Now, because I have this state machine, I thought, can I use this same machine to both power the app and power some -- I don't want to call them tests, because it's going to get a bit crazy, I'll show you. But what I did was I took this machine, and I mapped each one of the events, the player events, to actually clicking a square. So whatever the value is, whether it's the 0, 1, 2, 3, 4, 5, 6, 7, 8, square I'm just finding that square and clicking it in Puppeteer. I narrowed it down, I said I want all the paths where the state matches winner, there's more than five moves, and the winner is X.

So... hopefully... it will do that. And so it's going to launch the app and generate all those paths and then we'll see. if it loads. There we go. So it's going to find every single combination AUDIENCE: Whoo! [applause] all right, let's stop this before it gets out of hand. All right. So... I released XState test for this, and of course no documentation yet, but there are other similar tools for this that have existed in the past.

Play from 21:00

I don't know how up to date they are, but there's Simulato, which has similar ideas, and also graphwalker, and that's for Java, not for JavaScript. I did the recruiter move, I'm sorry, that's for Java, not for JavaScript. [laughter] so I'm going to be honest with you, model based testing is harder, I don't mean in terms of effort, I mean in terms of mental energy, because there is a learning curve in terms of how you model the application and just learning the art of modeling so you could learn your application in these state machines or statecharts, but it's also better, faster and stronger for many different ways, it's better because you're going to cover many more edge cases than a human could even determine manually in their lifetime.

It's also going to be faster, because you no longer need to update tests or write tests. all you have to do is update the model it needs updating and in case you want to support different browsers, different frameworks, you just update the model tests and so these are going to result in stronger tests, because you are covering all of those edge cases, you are tweaking how fine-tuned and how much coverage you want your tests to have. And this is possible with model-based testing because we're not writing tests. It's the entire point. So these XState tools are part of this whole idea of model-driven development and that's why I say that XState is so much more than just another state management tool. How many of you use Gatsby? All right, a fair amount of you, so Gatsby actually uses XState under the hood in order to manage processes like file managing and stuff like that.

Also React training uses it, I use it at Microsoft, because I work at Microsoft and I could do that without them knowing and it's also used at other companies which I'm not allowed to say for whatever reason. And it's also used for this really fun game, some of you might know it. It's called service workees, and this is a really fun game with lots of levels and complex animations which are driven by XState and by these state machines and you get this really great experience which is complex and also helps you learn things like service workers. So XState is evolving. I just released XState test and I'm coming soon with so many more tools for XState. Also, there is XState React, and it looks very similar to the useReducer hook, all you do is you pull this useMachine hook and you put a state machine instead of a reducer and you have the same state and dispatch but they're called current and send and it's a really powerful paradigm that you can use to power your React applications.

So basically I want to make your code model-driven and I want you all to make your code model-driven, because this allows you to do so many more things, it allows you to generate tests, generate documentation from those tests, it allows you to even generate prototypes and visuals, so in short, I want to make your code do more. Thank you, React Rally. [cheers and applause]

Play from 23:00