February 12, 2019 // By Paul Grizzaffi
While writing my previous blog post on automation execution environments, I had these thoughts bouncing around in my head as well. Additionally, I’m cleaning out and rearranging my home office, which reminded me of all the times my mom told me to clean my room. It’s a weird place, my head. Anyway…
What’s running on your desktop right now? Here is a subset of the things running on my desktop: Outlook, Chrome, Firefox, Keep, OneNote, Visual Studio, Bash, Skype, and Teams; again, that’s just a subset and I haven’t even listed any processes running in the background. Additionally, I’m not always running all these applications. Though not everyone is running the same applications as everyone else, very few of us close all other programs when we run automation on our desktops.
At some point, we will receive a new deployment of our application and we need to execute some of our automation against that new deployment. Since more people than just us are interested in the results of that execution, we also need to have an accessible record of the results. I call this execution a run-of-record for that deployment. i.e. the run that matters. It’s the one that lets us know “hey, we think we have a good deploy here” or “hey, we think something is amiss with this deployment”.
Often, especially when we first start our automation endeavor, we make our runs-of-record on our desktops. There are challenges with this approach that may be inconsistent with the intent of our automation run. As part of an automation run, we usually set our app to a known, steady state: we set up processes, data, user credentials, etc. so that we can execute specific scenarios. We tend to execute the scripts against the same sets of data and application configurations, so we approximate the same execution steps on each script execution. I liken this “known, steady state” to manufacturing cleanroom, a room designed such that it does not interfere with the manufacturing process. Cleanrooms are intended to reduce the unpredictability that can be introduced by unwanted particles and airflows.
If we take such care when configuring our application, why shouldn’t we take similar care when setting up our execution environment? For most of the software we test, a user device implicitly becomes part of our application’s “execution ecosystem”; it becomes part of the application because some portion of the application’s code runs on the hardware owned by the user. It’s the same for automation execution; the environment on which we execute our tests implicitly becomes part of the application. The extent of “becomes part of” varies from case to case; for example, for browser-based automation, the browser and the device are quite intertwined with our application software, whereas if we are testing via APIs the amount that the execution environment is part of the application is typically far less.
Since our execution environment becomes part of our application, we want to be similarly deliberate in how we configure this environment as we were when we configured the application itself. It’s often difficult to exercise that level of configuration when we use our desktops for our runs-of-record. If we use dedicated execution hardware for these runs, we have more control over that environment. We have control of the environment’s configuration, including the specifications of the processor, disk, and memory; we also have control over what software is installed and running in that environment.
Now, we can set up our execution environment to be our version of a cleanroom. We can know exactly what is installed and running and ensure that we only run the minimum OS background tasks; we can also ensure no other programs are running that might cause fluctuation in our application software’s behavior. We can now have some level of confidence that the part of our application that runs on the user’s device will run consistently and, therefore, our automation will execute consistently.
Does this reflect the real world? Not really. Our users are more like us than a cleanroom; they have multiple applications running, multiple browser tabs, streaming music, etc. They have what we might call a dirty room, i.e. a room whose cleanliness we can neither know nor control. Running our automation in dirty room environments exposes our application execution to these real-world situations. The downside, however, is this reduces our repeatability.
How do we decide whether to execute in a cleanroom or a dirty room? Like so much in software development, it depends on what we are trying to do.
Are we trying to determine if a deployed version of the application is egregiously broken? If so, a cleanroom may be a better approach. Executing in a cleanroom environment helps to reduce the number of variables on a per execution basis; the more predictable our execution environment, the less likely that an issue is due to a fluctuation caused by something “dirty” in that environment.
On the other hand, are we trying to chase out problems and expose risks? In this case, a dirty room may be a better approach. The more that our execution environment is like a user environment, the more likely we are to find issues that pertain to a user’s environment. Dirty room testing is valuable because it better represents actual user situations.
Do we have to choose between cleanrooms and dirty rooms? Perhaps, but that is a business decision. If it’s sufficiently valuable to use multiple execution environments, then we should consider doing so. Perhaps our cleanroom environment is for traditional automated smoke scripts, giving us a quick, stable automation run to test the minimum viability of a new deployment. Once we are ready for more in-depth testing, perhaps that’s the time to run our scripts, the same ones as before or others created expressly for this purpose, in our dirty room.
Could it be that mom wasn’t always right? I’m sure not telling her that.