Testing of a financial data warehouse system typically involves over 100 separate process steps. To perform tests manually can be time-consuming, error prone and presents a high learning curve to test analysts new to the system. Major upgrades to a system typically involve several weeks of functional testing. Minor releases focus on functional testing of enhancements, and quite often release schedules do not allow for full manual regression testing.
A pilot study was conducted over a period of 3 months to investigate the feasibility of using TTCN-3 and a suitable software package to improve accuracy and efficiency of the functional and regression testing of a financial data warehouse. Additional objectives were to standardize the testing process and to reduce the learning curve of test analysts new to the system.
Following the successful completion of the pilot study, TTworkbench was selected as a suitable software package and over the next 5 months was used to assist functional testing of new releases and to implement regression testing on existing systems. This work is still ongoing, but the benefits have been immediately noticeable and there are plans to develop further test scripts to assist in the automation per-formance testing and shakedowns of new environments using TTCN-3.
Application Scenario
TTCN-3 was used primarily as a control program to start various test scripts. The TTCN-3 test cases consisted of calls to a test adapter which issued windows command lines. These windows command lines called test scripts which used third party and custom built software to perform various tasks, such as:
- Check if the SUT is setup correctly
(software running, databases running etc.)
- FTP test data to specified servers
- Calls to batch scheduling programs to load data, start general ledger processes etc.
- Calls to various UNIX scripts to load data, produce reports etc.
- Calls to various SQL scripts to prepare the database of the SUT
- Calls to various custom developed test tools to extract data at various stages in the test cycle and to compare it to the expected results
The test scripts then communicated success/failure/error messages back to the TTCN-3 test cases. The TTCN-3 control module contained logic to determine the next step of the test and to log the results in a test report.
Many of the test analysts in a financial environment are recruited primarily for their business knowledge, although it is common to have experience with high level languages. The TTCN-3 language proved to be easy to program, and following the pilot study which created the basic building blocks for the test suites, many of our analysts were creating their own test suites within days.
Using a standardized test language ensures that we are not limited to the proprietary language of a specific vendor. TTCN-3 consists of language features such as parameterization, running of parallel test cases, timing ability, logging ability, or the ability to logically order test cases and control decisions on which test cases to run next. This allowed us to run the test suite overnight simultaneously for several source systems, dramatically reducing the regression test cycle time TTCN-3 has proved suitable for gradual implementation of an automated test suite: In the first phase we have used it primarily as a test control module, responsible for running existing test tools and recording responses from these test tools. This means we can re-use the functionality in existing tools.
Software from several TTCN-3 vendors was examined during the pilot phase, and TTworkbench Basic was selected for the following reasons:
It offers an integrated development and test suite management environment that is easy to setup and use and can be installed on a single machine or on a server. Some of the other packages examined consisted of several discrete components (editor, execution manager etc.) which were deemed less user friendly. Other packages involved installing software on both client and server machines, which seemed more suitable for telecoms applications in testing of remote distributed applications.
A strategic decision was made to avoid in-house programming of the test adaptor required (which requires Java or C++ skills) and to focus our test analysts on producing the high level TTCN-3 test cases, a language which was more suitable to their skills.
Testing Technologies provided us with a stable test adapter which is suitable for our needs, and we do not yet anticipate the need to improve it. The level of technical support from Testing Technologies during the pilot phase and since then has always been friendly, helpful and with a quick response time.
We found the Testing Technologies account manager to be helpful in providing a package of development and execution licenses to meet our needs and flexible in ensuring we had no down time in development during the software approval and purchase process.
|
|
"Where most (GUI oriented) test automation tools stop, TTCN-3 starts. This is the case for us in a back office, batch driven architecture with a data warehouse. TTCN-3 enables us to easily automate a multitude of manual steps surrounding the batch process under test. TTCN-3 is also very easy to adopt as long as you have a basic understanding of any programming or scripting language. The fact that it is a standardized language ensures that you are not limited to the proprietary language of a specific vendor. The pro active support during our pilot project made us choose TTworkbench as our test automation tool." |
Bas Wieman
Test Manager, International Bank |
