Test automation tools can be expensive, and are usually employed in combination with manual testing. Test automation can be made cost-effective in the long term, especially when used repeatedly in regression testing. A good candidate for test automation is a test case for common flow of an application, as it is required to be executed (regression testing) every time an enhancement is made in the application. Test automation reduces the effort associated with manual testing. Manual effort is needed to develop and maintain automated checks, as well as reviewing test results.

In automated testing the test engineer or software quality assurance person must have software coding ability, since the test cases are written in the form of source code which, when run, produce output according to the assertions that are a part of it. Some test automation tools allow for test authoring to be done by keywords instead of coding, which do not require programming.

One way to generate test cases automatically is model-based testing through use of a model of the system for test case generation, but research continues into a variety of alternative methodologies for doing so.[citation needed] In some cases, the model-based approach enables non-technical users to create automated business test cases in plain English so that no programming of any kind is needed in order to configure them for multiple operating systems, browsers, and smart devices.[2]

What to automate, when to automate, or even whether one really needs automation are crucial decisions which the testing (or development) team must make.[3] A multi-vocal literature review of 52 practitioner and 26 academic sources found that five main factors to consider in test automation decision are: 1) System Under Test (SUT), 2) the types and numbers of tests, 3) test-tool, 4) human and organizational topics, and 5) cross-cutting factors. The most frequent individual factors identified in the study were: need for regression testing, economic factors, and maturity of SUT.[4]

Testing tools can help automate tasks such as product installation, test data creation, GUI interaction, problem detection (consider parsing or polling agents equipped with test oracles), defect logging, etc., without necessarily automating tests in an end-to-end fashion. One must keep satisfying popular requirements when thinking of test automation:

• Platform and OS independence
• Data driven capability (Input Data, Output Data, Metadata)
• Customizable Reporting (DB Data Base Access, Crystal Reports)
• Easy debugging and logging
• Extensible & Customizable (Open APIs to be able to integrate with other tools)
• Version control friendly – minimal binary files
• Extensible & Customizable (Open APIs to be able to integrate with other tools)
• Common Driver (For example, in the Java development ecosystem, that means Ant or Maven and the popular IDEs). This enables tests to integrate with the developers' workflows.
• Support unattended test runs for integration with build processes and batch runs. Continuous integration servers require this.
• Email Notifications like bounce messages
• Support distributed execution environment (distributed test bed)
• Distributed application support (distributed SUT)