When I am interviewing test engineers (even with huge amount of experience) I often hear the following statement:
I have a lot of experience in building test automation frameworks from scratch!"
Sometimes - it’s true. But in many cases the candidate means something different. Test automation solution - in particular.
But what is the difference between test automation framework and test automation solution? Let’s discuss! Because each engineer can have their own view on this topic.
Disclaimer: This is my current understanding of the terms based on experience and multiple book sources.
📖 In the search for an answers
What does “Test Automation Fundamentals” book say?
In the Test Automation Fundamentals book, we can find the following definitions:
A test automation solution (TAS) is a specific instance of a TAA* and consists of the test environment and its corresponding automated testware. The latter includes automated test cases (which may be grouped into test suites), test data, and specific configuration files. A TAS is therefore not a monolithic tool or framework, but rather a combination of tools, components, and testware brought together for the purpose of automating testing processes.
*TAA - test automation architecture
A test automation framework (TAF) can, however, be used to provide a test environment, tools, test libraries, or additional test frameworks that can then be reused for faster automated test creation and execution.
🌐 What about Wiki?
Wikipedia provides similar wording:
A test automation framework provides a programming environment that integrates test logic, test data, and other resources. The framework provides the basis of test automation and simplifies the automation effort. Using a framework can lower the cost of test development and maintenance. If there is change to any test case then only the test case file needs to be updated and the driver script and startup script will remain the same.
❗️Test Automation Framework
Test automation framework - is a library or collection of libraries, you can use as a basis for implementing automated tests.
The key thing in framework is EASE OF REUSABILITY. It should be easy to use a framework not only for similar, but also for completely different projects (even in various technologies - if we are talking about end-to-end level). Framework may or may not have a strict structure for test code.
As an example, we can use the frameworks like JUnit, TestNG, pytest, Jest - for writing tests for Web, API, Mobile and other systems.
In general software engineering we can also find frameworks:
- Spring and Hibernate for Java
- Django, Flask, FastAPI for Python
- React, Vue, Angular for Javascript
Big companies can create their own internal frameworks when they have a need of inified solutions for multiple projects. Such frameworks help to onboard new people very fast.
❗️Test Automation Solution
Test automation solution - is a combination of a framework, tests and lots of additional utilities, test data, CICD scripts, cloud runners, reporting, etc.
The key thing here is that solution is always taylored for a particular product.
When you implement a solution - you inevitably implement (or reuse) some kind of framework. But this “framework” part becomes a framework only when it’s extracted and then reused in multiple projects within the company.
In other cases, when you implement custom automated tests for a particular project - you are not writing a framework. You are solving a problem. You are creating something by putting pieces of puzzle together to form a solution. Even when you are doing it “from scratch”
Modern instruments, like Playwright, already contains many of the components of the framework - just out of the box. So the step for creating a framework is either simplified or absent.
🎓 As a conclusion
When you implement automated tests - you always work in the context of the specific test automation solution with the use of particular automated framework. Whether you will later separate part of the solution into a separate “internal framework” - time will tell.
Of course, if you implemented a complete analogue of pytest or JUnit - you have created a framework, without any doubt!
Remember, that any test automation effort should start with asking question:
“which problem are we going to solve with this solution?”
Are you implementing frameworks or a solutions?