user manual


Version 0.5

15.07.2009


C1 SetCon GmbH

Kapuzinerstraße 7a

80337 München


Fon +49 89 589399-0

Fax +49 89 589399-25


tape@c1-setcon.de

www.c1-setcon.de




Table of Contents



1. Forewords

2. Introducing TAPE

3. Testing with TAPE

3.1 TAPE Project and its elements

3.2 Understanding the Test Process

3.2.1 Define Environment

3.2.2 Define Requirements

3.2.3 Define Tests

3.2.4 Schedule / Execute Tests

3.2.5 Analyze Tests / Track Tests

4. TAPE at a Glance

4.1 The Project window

4.2 Menu bar

4.3 Perspective Tabs

4.4 Tree Views

4.5 Project Pane

4.6 Workbench

4.7 Console

4.8 Information bar

5. Working with TAPE

5.1 Settings

5.1.1 Action

5.1.2 Projects 16

5.1.3 KPIs – Key Performance Indicators

5.2 Environment

5.2.1 Component Type

5.2.1.1 Create

5.2.2 Artifact Spec

5.2.2.1 Create

5.2.3 Component Configuration

5.2.3.1 Create

5.2.3.2 Configuration Item

5.2.3.3 Create

5.2.4 Node

5.2.4.1 Create

5.2.4.2 Verify

5.2.5 Deployment Unit

5.2.5.1 Create

5.2.6 Deployment Scenarios

5.2.6.1 Create

5.2.7 Deployed Artifact

5.2.7.1 Create

5.2.8 Tape Agent

5.2.9 Configuration Specification

5.3 REQUIREMENTS

5.3.1 Requirement Type

5.3.2 Requirement

5.3.3 Requirement Group

5.4 TESTS

5.4.1 Test Cluster

5.4.2 Test Case

5.4.3 Requirements

5.4.4 Scenario

5.4.5 Execution Plan

5.4.6 Task Group

5.4.7 Task

5.4.8 Configuration

5.4.9 Configuration Item

5.4.10 Rules

5.4.11 Test case Report and Chart Templates

5.5 ANALYSIS

5.5.1 Analysis Templates

5.5.2 Test Case Analysis – Reports

5.5.3 Test Case Analysis

5.5.4 Project Analysis

5.6 TRACKING

5.6.1 Plan and time

5.6.2 Test Efficiency

5.6.3 Requirements



1.Forewords


Welcome to TAPE, an innovative and highly adaptable tool that establishes a new way to approach load testing activities and that gives performance engineers a greater control over their load tests and the test management process for complex performance tests.


TAPE has been created based on C1 SetCon’s extensive knowledge in the field of performance engineering, founded on years of experience in a wide variety of customer projects and technologies.


TAPE is intended as a working set of features in order to help performance engineers during their everyday-work, providing a tool-based methodology for the performance engineering process. With TAPE, the test result is automatically documented and can easily be interpreted, the progress is monitored and as a result, the performance test phases are streamlined and a significant amount of time can be saved.


It is the first tool of its kind and will continue to grow through continuous input from customers, enabling significant increment of its features.


2.Introducing TAPE


TAPE uses the flexibility of 3rd party tools (load test tools, monitoring tools, etc.) by allowing users to define, schedule and analyze load tests in a distributed environment. TAPE also allows tests to be tracked and analyzed. The Requirement module gives the user an easy way of ensuring that the traceability and coverage of requirements have been fulfilled.


With TAPE you can use your own load testing tools in order to produce load on applications. You can also integrate your monitoring tools in order to collect data during the test. It is also possible to integrate and control your own scripts and programs, in order to take some specific actions during a test (e.g. starting/stopping an application server, parsing a log file, etc.). TAPE provides an adaptation layer to existing tools called Plug-in interface, that's why you can call it "Meta-Tool".


Thanks to that integration with other tools, TAPE imports in its own result repository (database) all testing results from tools, which are involved in a load test, enabling automatic correlation of different results and quicker finding of bottlenecks.


Analysis without TAPE:


Analysis with TAPE:



With its internal scheduling capabilities TAPE can help optimizing environment resources among many different load test projects, which are running at the same time, sharing a part of their testing environments with each other. TAPE controls environment usage overlapping and suggest to the tester new scheduling timeslot in order to use the shared environment exclusively.




3 Testing with TAPE



3.1 TAPE Project and its elements

TAPE is organized in Projects. Every Project is a container for test cases, environment definitions, requirements, analysis charts and reports. A TAPE user (tester, performance engineer) has access to a defined set of projects, in order to perform testing activities on it. The access is managed in the TAPE Admin client where TAPE projects are configured, and projects are assigned to users.




A TAPE project incorporates different releases and each release has its own set of test clusters, which are sets of test cases to be tested.


A test case comprises a logical association of the application under test (Deployment Scenario), the Requirements, the configuration elements from Deployment Scenario (Configuration Specification) and the Execution Plan which describes how, with what elements and in what order a test should be executed.





For example, following situations can be described in test cases:


"TC001 Load test with 500 concurrent users and standard scenario"


"TC002 Load test with 700 concurrent users and standard scenario"


Test cases are grouped into test clusters in order to associate test cases with commons goals (e.g. scalability, benchmarking, etc.).



A Deployment Scenario is a central part of a test case and describes which environment the load test has to be executed and with which components that are involved (e.g. application servers, native applications, databases, etc.).


TAPE provides an entity hierarchy in order to describe all possible components of a load testing environment:






3.2 Understanding the Test Process


Testing with TAPE typically consists of five phases: Defining Environment, Defining Requirements, Defining Tests, Scheduling/Execution and Analysis/Tracking.




The environment definition will be carried out in the beginning of a project, since in release-based applications the environment seldom changes.


The main workflow has been supported with 4 main perspectives, which incorporate functionality groups for the whole tool.




3.2.1 Define Environment


The Environment constitutes what will be tested and which components and artifacts that are involved. The objective is to compose a Deployment Scenario, which is incorporate later in a Test Case.


The environment workflow consists of the following steps:



3.2.2 Define Requirements


Analyze your application and determine your requirements. The requirements specification workflow consists of the following steps:




3.2.3 Define Tests


The Test defines the Deployment Scenario that is to be tested. Furthermore the Execution Plan and the runtime Rules for the Test to be executed are described. When completed, TAPE allows the user to validate the correctness of the configuration, before carrying on with scheduling the Test Case.






3.2.4 Schedule / Execute Tests


After a test case has been defined, the user can choose to schedule it by defining the start time and estimated maximal duration. If the test case overlaps an already scheduled test, the user is notified and an alternative start time is automatically proposed. When the Test Run is completed, TAPE imports the log files, which the user can choose to view. The scheduling workflow consists of the following:











3.2.5 Analyze Tests / Track Tests


The Analyze module allows the user to quickly analyze run session results based on configurable graphs that present important information about the performance of your scenario. Using these graphs and reports, you can easily pinpoint and identify the bottlenecks in your application and determine what changes need to be made to your system to improve its performance. The analyze workflow consists of the following:






The track workflow consists of the following steps:






4 TAPE at a Glance


4.1 The Project window



Menu bar

Perspective Tabs

Tree View

Project Pane

Workbench

Console

Information bar




4.2 Menu bar


Tape:

> Log out: Log out from TAPE

> Exit: Exit TAPE


View:

> Refresh Refresh TAPE


Settings:

> Actions Create and manage Actions

> Projects Create and manage Releases

> KPIs Create and manage KPIs


Help:

> Help Help file

> About About TAPE



4.3 Perspective Tabs


The Project-Pane consists of three folders:

The Scheduler-Pane displays a list of all test runs in the upper part of the screen and details, log-messages and log-files in the area beneath.



Define and create charts and reports that display the result of the tests.


Display the number of test cases, executed test runs, requirement coverage and test efficiency.



4.4 Tree Views






4.5 Project Pane


The Project Pane enables you to add, edit, duplicate, remove and manage all of the objects in your test.



4.6 Workbench


The Workbench is the main view for creating and managing objects.



4.7 Console


In the Console TAPE displays information and error messages.



4.8 Information bar


The current User, Project and Release are displayed in the information bar.




5 Working with TAPE


5.1 Settings


Before you start defining your tests, you should preferably begin with defining the settings in TAPE. This can also be done at a later stage, but must be completed before any tests can be executed.



5.1.1 Action


An action is a remotely started script and is a basic component of a load test. It is necessary in order to define the Execution Plan in the Test Clusters module.


In TAPE terminology an “Action” is a tool or script that you integrate into TAPE using the TAPE test runs interface specification.


An Action deployed on a specified unit is called a Task. A Task is the central component for defining an Execution Plan in a test case (what has to be executed, when and where).


The following picture shows an example: HP LoadRunner, Start/Stop Scripts for Tomcat and a Windows WMI Monitor are external programs, which are integrated in TAPE through wrappers (TAPE Plug-in interface), in order to be used in several Execution Plans for several Test cases:




5.1.2 Projects


A user can create a new release of a project by cloning the previous release. The user decides which entities of a release will be cloned into the new release.


5.1.3 KPIs – Key Performance Indicators


At the end of each test, the raw data is compiled into a set of consistent results and key performance indicators (KPIs).



5.2 Environment


The Environment defines the system under test and all the involved components, for example operating system, middleware, application and so on, can be defined by the user. In order to be able to execute tests with TAPE, the whole Environment needs to be complete and correctly defined.





The following objects constitute the main object in the Environment under test.


5.2.1 Component Type


A component type describes the high level type of the artifact spec, for example "IBM WebSphere Application Server" belongs to the component type named "Application Server". Other possibilities are for example "Operating System", "Hardware", "Web Server", etc.


5.2.1.1 Create


Project > Environment > Component Types > Context menu: Create




After the Component Type object has been created, you have the possibility in the context menu to select: Edit, Create (a new one), Duplicate and Delete.







5.2.2 Artifact Spec


The Artifact Spec specifies the Component Type. It could for example be an application- or a web-server, e.g. IBM WebSphere, TomCat, or an application, for example Silk Performer.


5.2.2.1 Create


Project > Environment > Artifact Specs > Context menu: Create




After the Artifact Specs object has been created, you have the possibility in the context menu to select: Edit, Create (a new one), Duplicate and Delete.



To create an Artifact Spec:




5.2.3 Component Configuration

The Component Configuration defines the configuration of a component, like artifact, deployment scenario


5.2.3.1 Create


  1. Project > Environment > Artifact Specs > Context menu: Edit

  2. In the Workbench, right-click in the Configurations table and select Create




After the Artifact Specs object has been created, you have the possibility in the context menu to select: Edit, Create (a new one), Duplicate and Delete.







5.2.3.2 Configuration Item



Copytext bold



5.2.3.3 Create


  1. Project > Environment > Artifact Specs > Component Config: Edit

  2. In the Workbench, right-click in the Config Items table and select Create










5.2.4 Node


The Node defines the logical address of the machine where the test will take place.



5.2.4.1 Create


Project > Environment > Node > context menu: Create






Note: After creating the Node, it is possible to verify the connectivity by selecting ‘Verify’ in the context menu. A simple ping is then automatically performed in order to check the availability of the target machine.


5.2.4.2 Verify


To test if the node is reachable, the connection can be verified.


Project > Environment > Node > context menu: Verify



5.2.5 Deployment Unit


A Deployment Unit refers to a physical machine in your testing environment, where the test will be executed. It consists of the Node, the Artifact Spec and the TAPE Agent.


5.2.5.1 Create


Project > Environment > Deployment Units > context menu: Create





5.2.6 Deployment Scenarios


The objective of building the Environment is to put together the Deployment Scenario. It constitutes the application under load, with all its components, machines and software installations and is later incorporate in a Test Case, where it represents the test infrastructure.


After selecting the Deployment Unit, it will appear as a sub node to the Deployment Scenario.


5.2.6.1 Create


Project > Environment > Deployment Scenarios > context menu: Create




5.2.7 Deployed Artifact


The Deployed Artifact represents an Artifact Spec and the Deployment Unit it has been deployed on.


A ‘Deployed Artifact’ is the component, which has been deployed on the Deployment Unit. For example, the "WAS 3 WEBGUI" is an instance of the Websphere application server, i.e. a Deployed Artifact.


5.2.7.1 Create


Project > Environment > Deployment Scenario > Deployment Unit > context menu: Create


The Deployed Artifact can be created by right-clicking the Deployment Scenario and selecting ‘Add new Deployment Artifact’.







5.2.8 Tape Agent


Project > Environment > Deployment Scenarios > Deployment Scenario > Deployment Unit > context menu: Add new TAPE Agent










5.2.9 Configuration Specification


The Configuration Specification describes the system under test.


Project > Environment > Deployment Scenarios > Deployment Scenario > context menu: Add new Configuration Specification










5.3 REQUIREMENTS


The requirements describes in detail all the necessary attributes of the system under test, and provide the user with the foundation on which the entire testing process is based. It is also possible to define thresholds that need to be achieved in order to fulfill a requirement. After specifying the requirement, you link it to the test case in the Project module, in order to ensure that all requirements have been covered.





5.3.1 Requirement Type


Indicates the type of requirement, for example ‘Memory’, ‘Response time’ and ‘Throughput’ and is useful for classifying related requirements.



Project > Requirements > Requirement Types > context menu: Create










5.3.2 Requirement


A Requirement describes a condition or capability to which a system must conform. It is possible to define the KPI (Key Performance Indicator) and the Value that a specific requirement is being measured against.


Project > Requirements > Requirements > context menu: Create










5.3.3 Requirement Group


It can be useful to group similar requirements into Requirement Groups, for example Memory, Response time and Throughput.


Project > Requirements > Requirement Groups > context menu: Create









5.4 TESTS

A test case is a set of conditions or variables under which a performance tester will determine if a non-functional or technical requirement upon an application is partially or fully satisfied. A test case defines:




5.4.1 Test Cluster


To meet the various goals of a project, the tests in your project can be organized into unique clusters. A Test Cluster can be duplicated, but only deleted if no dependencies exist.



Project > Test Clusters > context menu: Create











5.4.2 Test Case


A Test Case is defined with one deployment scenario, one configuration, zero or more requirements and a mandatory execution plan.



Project > Test Clusters > Test Case > context menu: Add new Test Case









5.4.3 Requirements


The in the Requirement module previously defined requirement is linked to the appropriate test case and appear as a sub node to the Test Case in the tree view.


Project > Test Clusters > Test Case > Requirements







5.4.4 Scenario


The Deployment Scenario is selected for the Test Case.


Project > Test Clusters > Test Case > Scenario





5.4.5 Execution Plan


The Execution Plan defines in which order the Task Groups are started and stopped.



Project > Test Clusters > Test Case > Execution Plan




5.4.6 Task Group


A Task Group is a set of tasks, which have a common goal (e.g. starting monitoring or driving the tests).



Project > Test Clusters > Test Case > Execution Plan > Task Group: context menu: Create






5.4.7 Task


A task is an action associated with a deployment unit and its TAPE Agent



Project > Test Clusters > Test Case > Execution Plan > Task Group > context menu: Create new Task







5.4.8 Configuration


A Configuration groups the Configuration Items



Project > Test Clusters > Test Case > Execution Plan > Task Group > Task > Configuration





5.4.9 Configuration Item

A Configuration Item is a set of key/value pairs organized into categories (e.g. Java Properties).


Project > Test Clusters > Test Case > Execution Plan > Task Group > Task > Configuration > Table: Config Items > context menu > Create




The task uses the following naming convention:


<Configuration Item> @ <TAPEAgent> @ <Configuration>




5.4.10 Rules


The Rules determines in which order the Task Groups are be started and/or stopped.



Project > Test Clusters > Test Case > Execution Plan > Table: Rules





Rules:

Task Group

Status

Next Task Group

Command

Possible states:

[User definable]

Started / Stopped

[User definable]

Start / Stop

Comment:

Select Task Group


Next Task Group

Start or Stop Task Group






5.4.11 Test case Report and Chart Templates


Decide what Report- and Chart templates that are to be used to display the result of the test run.



Project > Test Clusters > Test Case > Analysis






5.5 ANALYSIS


The Analysis module helps you assess the result of the tests. However, before this can be done, an Analysis Template that defines the desired layout must be created.





5.5.1 Analysis Templates



The Aggregated Chart Template allows the user to define and display charts of aggregated values.


Create:

Analysis > Analysis Templates > Chart Templates > Context menu: Create Aggregated Chart Template






Time Series Chart Template


The Time Series Chart Template enables the user to display the test in a line chart, plotted over time.


Create:

Analysis > Analysis Templates > Chart Templates > Context menu: Create Time Series Chart Template








Report Template


The Report Template provides the possibility to create a report in which the result of the tests can be presented with graphs and text.


Create:

Analysis > Analysis Templates > Report Templates > Context menu: Create





Define the contents of the pdf-document


XML (context menu):


Please see the Appendix for an example Report


Test Case Analysis

Toggle between test cases







5.5.2 Test Case Analysis – Reports


Create a report based on selected test runs.


Create:

Analysis > Test Case Analysis > Reports > Context menu: Create



5.5.3 Test Case Analysis

Create a chart based on single test run.


Single Test Run - Charts





Single Test Run - Report


Create a report based on a single test run.







5.5.4 Project Analysis


Charts





Reports





Charts:




Reports





5.6 TRACKING

The TAPE tracking module helps you to assess the progress of your tests. You can for example with the pre-defined queries measure the number of executed tests over time and which requirements that must be tested.



5.6.1 Plan and time


Display

Tracking > Queries > Plan and Time > [select query] > Context menu: Display



Specify the time period for which you want to view data.




Report

Description

Test cases in a time-period

Lists the test cases in grid format, that were created in a specific time period

Test runs in a time-period

Lists the test runs in grid format, that were created in a specific time period

Test cases ready to test

Lists the test cases in grid format, that have not yet been executed




5.6.2 Test Efficiency


Display

Tracking > Queries > Test Efficiency > [select query] > Context menu: Display



Specify the time period for which you want to view data.









      Report

      Description

      Amount of Test runs in a time-period

      Displays the amount of test runs over time in a graph

      Amount of Test cases in a time-period

      Displays the amount of test cases over time in a graph



5.6.3 Requirements


Display

Tracking > Queries > Requirements > [select query] > Context menu: Display







Report

Description

Test case with unfulfilled requirements

Lists the test cases in grid format, that are linked to unfulfilled requirements

Unfulfilled requirements

Lists the requirements in grid format, that have not yet been fulfilled





































TAPE
Testing Automation for Performance Engineering

© C1 SetCon GmbH
Kapuzinerstr. 7a

D-80337 Munich

Germany


http://www.c1setcon.de

Please refer to the About dialog box for the list of licensed external libraries and components.


Copyright © 2008-2009 C1 SetCon GmbH. All other marks are the property of their respective owners.


August 2009 - PDF