
user manual
Version 0.5
15.07.2009
C1 SetCon GmbH
Kapuzinerstraße 7a
80337 München
Fon +49 89 589399-0
Fax +49 89 589399-25
tape@c1-setcon.de
www.c1-setcon.de
Table of Contents
3.1 TAPE Project and its elements
3.2 Understanding the Test Process
3.2.4 Schedule / Execute Tests
3.2.5 Analyze Tests / Track Tests
5.1.3 KPIs – Key Performance Indicators
5.2.9 Configuration Specification
5.4.11 Test case Report and Chart Templates
Welcome to TAPE, an innovative and highly adaptable tool that establishes a new way to approach load testing activities and that gives performance engineers a greater control over their load tests and the test management process for complex performance tests.
TAPE has been created based on C1 SetCon’s extensive knowledge in the field of performance engineering, founded on years of experience in a wide variety of customer projects and technologies.
TAPE is intended as a working set of features in order to help performance engineers during their everyday-work, providing a tool-based methodology for the performance engineering process. With TAPE, the test result is automatically documented and can easily be interpreted, the progress is monitored and as a result, the performance test phases are streamlined and a significant amount of time can be saved.
It is the first tool of its kind and will continue to grow through continuous input from customers, enabling significant increment of its features.
TAPE uses the flexibility of 3rd party tools (load test tools, monitoring tools, etc.) by allowing users to define, schedule and analyze load tests in a distributed environment. TAPE also allows tests to be tracked and analyzed. The Requirement module gives the user an easy way of ensuring that the traceability and coverage of requirements have been fulfilled.
With TAPE you can use your own load testing tools in order to produce load on applications. You can also integrate your monitoring tools in order to collect data during the test. It is also possible to integrate and control your own scripts and programs, in order to take some specific actions during a test (e.g. starting/stopping an application server, parsing a log file, etc.). TAPE provides an adaptation layer to existing tools called Plug-in interface, that's why you can call it "Meta-Tool".

Thanks to that integration with other tools, TAPE imports in its own result repository (database) all testing results from tools, which are involved in a load test, enabling automatic correlation of different results and quicker finding of bottlenecks.
|
Analysis without TAPE:
|
Analysis with TAPE:
|
With its internal scheduling capabilities TAPE can help optimizing environment resources among many different load test projects, which are running at the same time, sharing a part of their testing environments with each other. TAPE controls environment usage overlapping and suggest to the tester new scheduling timeslot in order to use the shared environment exclusively.

TAPE is organized in Projects. Every Project is a container for test cases, environment definitions, requirements, analysis charts and reports. A TAPE user (tester, performance engineer) has access to a defined set of projects, in order to perform testing activities on it. The access is managed in the TAPE Admin client where TAPE projects are configured, and projects are assigned to users.

A TAPE project incorporates different releases and each release has its own set of test clusters, which are sets of test cases to be tested.
A test case comprises a logical association of the application under test (Deployment Scenario), the Requirements, the configuration elements from Deployment Scenario (Configuration Specification) and the Execution Plan which describes how, with what elements and in what order a test should be executed.

For example, following situations can be described in test cases:
"TC001 Load test with 500 concurrent users and standard scenario"
Execution
Plan:
Start all monitors, start LoadRunner with
scenario "standard500vu.lrs" until the test finishes. Then
stop all monitors and collect the final result.
Requirements:
Group "GUI", Requirement:
the
response time for 90% of all GUI clicks has to be less than 3
seconds.
Configuration
Specification:
Tomcat Server - Maximum Number of worker Threads
in Thread pool: 150
Deployment
Scenario:
1 Tomcat Server on Server A, 1 Data base on
Server B, LoadRunner on
Workstation C.
"TC002 Load test with 700 concurrent users and standard scenario"
Execution
Plan:
Start all monitors, start LoadRunner with scenario
"standard700vu.lrs" till the test finishes. Then stop all
monitors and finally collect results.
Requirements:
Group
"GUI", Requirement: the response time for 90% of all GUI
clicks have to be less than 3 seconds.
Configuration
Specification:
Tomcat Server - Maximum Number of worker Threads
in Thread pool: 150
Deployment
Scenario:
1 Tomcat Server on Server A, 1 Database on
Server B, LoadRunner on
Workstation C.
Test cases are grouped into test clusters in order to associate test cases with commons goals (e.g. scalability, benchmarking, etc.).

A Deployment Scenario is a central part of a test case and describes which environment the load test has to be executed and with which components that are involved (e.g. application servers, native applications, databases, etc.).
TAPE provides an entity hierarchy in order to describe all possible components of a load testing environment:
Component
Type
A component type describes the high level type of
artifact spec, for example "IBM WebSphere Application Server"
belongs to the component type named "Application Server".
Other possibilities are "Operating System", "Hardware",
"Web Server", etc.
Artifact
Specification
An Artifact spec describes the type of a
deployed artifact. "WAS 3 WEBGUI" for example belongs to
the artifact spec "IBM WebSphere Application Server".
Deployment
Unit
A deployment unit describes a machine in your testing
environment.
Deployed
Artifact
Describes a component, which has been deployed on a
defined deployment unit. E.g. "WAS 3 WEBGUI" is a deployed
artifact consisting of a WebSphere application server with a special
purpose called "WEBGUI".
Deployment
Scenario
A deployment Scenario describes your application
under load with all components, machines, software installations and
so on.

Testing with TAPE typically consists of five phases: Defining Environment, Defining Requirements, Defining Tests, Scheduling/Execution and Analysis/Tracking.


The environment definition will be carried out in the beginning of a project, since in release-based applications the environment seldom changes.
The main workflow has been supported with 4 main perspectives, which incorporate functionality groups for the whole tool.

The Environment constitutes what will be tested and which components and artifacts that are involved. The objective is to compose a Deployment Scenario, which is incorporate later in a Test Case.
The environment workflow consists of the following steps:

Analyze your application and determine your requirements. The requirements specification workflow consists of the following steps:

The Test defines the Deployment Scenario that is to be tested. Furthermore the Execution Plan and the runtime Rules for the Test to be executed are described. When completed, TAPE allows the user to validate the correctness of the configuration, before carrying on with scheduling the Test Case.

After a test case has been defined, the user can choose to schedule it by defining the start time and estimated maximal duration. If the test case overlaps an already scheduled test, the user is notified and an alternative start time is automatically proposed. When the Test Run is completed, TAPE imports the log files, which the user can choose to view. The scheduling workflow consists of the following:

The Analyze module allows the user to quickly analyze run session results based on configurable graphs that present important information about the performance of your scenario. Using these graphs and reports, you can easily pinpoint and identify the bottlenecks in your application and determine what changes need to be made to your system to improve its performance. The analyze workflow consists of the following:

The track workflow consists of the following steps:

|
Menu bar Perspective Tabs Tree View Project Pane Workbench Console Information bar |
|
Tape:
> Log out: Log out from TAPE
> Exit: Exit TAPE
View:
> Refresh Refresh TAPE
Settings:
> Actions Create and manage Actions
> Projects Create and manage Releases
> KPIs Create and manage KPIs
Help:
> Help Help file
> About About TAPE
Project:
The Project-Pane consists of three folders:
Environment: handles all entities to be configured
Requirement: handles the requirements that defines the focus of the tests
Test Clusters: handles the test cases and the information how they are to be executed.
Scheduler:
The Scheduler-Pane displays a list of all test runs in the upper part of the screen and details, log-messages and log-files in the area beneath.

Analysis
Define and create charts and reports that display the result of the tests.
Tracking
Display the number of test cases, executed test runs, requirement coverage and test efficiency.
Environment

Requirements

Test
Cluster

The Project Pane enables you to add, edit, duplicate, remove and manage all of the objects in your test.
The Workbench is the main view for creating and managing objects.
In the Console TAPE displays information and error messages.
The current User, Project and Release are displayed in the information bar.
Before you start defining your tests, you should preferably begin with defining the settings in TAPE. This can also be done at a later stage, but must be completed before any tests can be executed.
An action is a remotely started script and is a basic component of a load test. It is necessary in order to define the Execution Plan in the Test Clusters module.
In TAPE terminology an “Action” is a tool or script that you integrate into TAPE using the TAPE test runs interface specification.
An Action deployed on a specified unit is called a Task. A Task is the central component for defining an Execution Plan in a test case (what has to be executed, when and where).
The following picture shows an example: HP LoadRunner, Start/Stop Scripts for Tomcat and a Windows WMI Monitor are external programs, which are integrated in TAPE through wrappers (TAPE Plug-in interface), in order to be used in several Execution Plans for several Test cases:

A user can create a new release of a project by cloning the previous release. The user decides which entities of a release will be cloned into the new release.
At the end of each test, the raw data is compiled into a set of consistent results and key performance indicators (KPIs).
The Environment defines the system under test and all the involved components, for example operating system, middleware, application and so on, can be defined by the user. In order to be able to execute tests with TAPE, the whole Environment needs to be complete and correctly defined.

The following objects constitute the main object in the Environment under test.
A component type describes the high level type of the artifact spec, for example "IBM WebSphere Application Server" belongs to the component type named "Application Server". Other possibilities are for example "Operating System", "Hardware", "Web Server", etc.
Project > Environment > Component Types > Context menu: Create

After the Component Type object has been created, you have the possibility in the context menu to select: Edit, Create (a new one), Duplicate and Delete.

Name (mandatory field): The name that TAPE assigns to the object.
Description: In the Description box, add a description of the object.
The Artifact Spec specifies the Component Type. It could for example be an application- or a web-server, e.g. IBM WebSphere, TomCat, or an application, for example Silk Performer.
Project > Environment > Artifact Specs > Context menu: Create

After the Artifact Specs object has been created, you have the possibility in the context menu to select: Edit, Create (a new one), Duplicate and Delete.

To create an Artifact Spec:
Name (mandatory field): The name that TAPE assigns to the object.
Description: In the Description box, add a description of the object.
Component Type (mandatory field): In the Component Type list, select the Component Type you want to insert.
Configuration: Right-click in the “Configurations” table and select ‘Create’ in the context menu. This will take you to ‘Create Component Configuration’ mask (see next step). You also have the possibility in the context menu, to delete or edit previously added Configurations.
The Component Configuration defines the configuration of a component, like artifact, deployment scenario
Project > Environment > Artifact Specs > Context menu: Edit
In the Workbench, right-click in the Configurations table and select Create

After the Artifact Specs object has been created, you have the possibility in the context menu to select: Edit, Create (a new one), Duplicate and Delete.

Name (mandatory field): The name that TAPE assigns to the object.
Description: In the Description box, add a description of the object.
Config Items: Right-click in the “Config Items” table and select ‘Create’ in the context menu. This will take you to ‘Create Configuration Item’ mask (see next step). You also have the possibility in the context menu, to delete or edit previously added Config Items.
Copytext bold
Project > Environment > Artifact Specs > Component Config: Edit
In the Workbench, right-click in the Config Items table and select Create


Name (mandatory field): The name that TAPE assigns to the object.
Content (mandatory field):
Is File:
Is Deployment:
Is User-Definable:
Description: In the Description box, add a description of the object.
The Node defines the logical address of the machine where the test will take place.
Project > Environment > Node > context menu: Create

Name: The name that TAPE assigns to the object is based on the IP-Address and the Hostname.
Description: In the Description box, add a description of the object.
IP-Address (mandatory field): The IP-Address of the target machine is automatically verified
Hostname (mandatory field): The Hostname of the target machine. With the system-commands [ipconfig /all] (Win XP), you can display the host-name (and the IP-address.
Note: After creating the Node, it is possible to verify the connectivity by selecting ‘Verify’ in the context menu. A simple ping is then automatically performed in order to check the availability of the target machine.
To test if the node is reachable, the connection can be verified.
Project > Environment > Node > context menu: Verify
A Deployment Unit refers to a physical machine in your testing environment, where the test will be executed. It consists of the Node, the Artifact Spec and the TAPE Agent.
Project > Environment > Deployment Units > context menu: Create

Name (mandatory field): The name that TAPE assigns to the object.
Description: In the Description box, add a description of the object.
Node (mandatory field): Select Node by clicking the object lookup button
The objective of building the Environment is to put together the Deployment Scenario. It constitutes the application under load, with all its components, machines and software installations and is later incorporate in a Test Case, where it represents the test infrastructure.
After selecting the Deployment Unit, it will appear as a sub node to the Deployment Scenario.
Project > Environment > Deployment Scenarios > context menu: Create

Name (mandatory field): The name that TAPE assigns to the object.
Description: In the Description box, add a description of the object.
Deployment Units: In the two multi-selection tables, available Deployment Units can be added to the Deployment Scenario
Image: Allows the up- and download of for example screenshot, for purpose of documentation.
The Deployed Artifact represents an Artifact Spec and the Deployment Unit it has been deployed on.
A ‘Deployed Artifact’ is the component, which has been deployed on the Deployment Unit. For example, the "WAS 3 WEBGUI" is an instance of the Websphere application server, i.e. a Deployed Artifact.
Project > Environment > Deployment Scenario > Deployment Unit > context menu: Create
The Deployed Artifact can be created by right-clicking the Deployment Scenario and selecting ‘Add new Deployment Artifact’.


Artifact Spec (mandatory field): Select the Artifact to be deployed.
Name: The name automatically gets assigned after selecting the Artifact Spec. This is also the name that is visible in the Environment tree.
Real Name: If the actual name differs from the Artifact Spec, it can be entered here.
Description: In the Description box, add a description of the object.
Project > Environment > Deployment Scenarios > Deployment Scenario > Deployment Unit > context menu: Add new TAPE Agent


Name: The name that TAPE assigns to the object.
Description: In the Description box, add a description of the object.
The Configuration Specification describes the system under test.
Project > Environment > Deployment Scenarios > Deployment Scenario > context menu: Add new Configuration Specification

Name (mandatory field): The name that TAPE assigns to the object.
Description: In the Description box, add a description of the object.
Deployment: Scenario: [Automatically selected]
Deployment Unit: [Automatically selected]
Artifact Spec: [Automatically selected]
Deployed Artifact: [Automatically selected]
Configuration: Select which Configuration you want to include. The selected configuration appears as a sub node to the Configuration Specification.
The requirements describes in detail all the necessary attributes of the system under test, and provide the user with the foundation on which the entire testing process is based. It is also possible to define thresholds that need to be achieved in order to fulfill a requirement. After specifying the requirement, you link it to the test case in the Project module, in order to ensure that all requirements have been covered.

Indicates the type of requirement, for example ‘Memory’, ‘Response time’ and ‘Throughput’ and is useful for classifying related requirements.
Project > Requirements > Requirement Types > context menu: Create


Name (mandatory field): The name that TAPE assigns to the object.
Description: In the Description box, add a description of the object.
A Requirement describes a condition or capability to which a system must conform. It is possible to define the KPI (Key Performance Indicator) and the Value that a specific requirement is being measured against.
Project > Requirements > Requirements > context menu: Create


Name (mandatory field): The name that TAPE assigns to the object.
Description: In the Description box, add a description of the object
Requirement Type (mandatory field):
KPI: Select the KPI (Key Performance Indicator) that this Requirement is being measured against.
Deployment Unit:
Deployment Artifact:
Operator: The operator is a Boolean value that defines the relationship between the ‘KPI’ and the ‘Value’
Value (mandatory field): Define the value which the KPI is being measured against.
Aggregation Function: …
Parameter: …
It can be useful to group similar requirements into Requirement Groups, for example Memory, Response time and Throughput.
Project > Requirements > Requirement Groups > context menu: Create


Name (mandatory field): The name that TAPE assigns to the object.
Description: In the Description box, add a description of the object.
Requirements:
Available: All available Requirements
Selected: Selected Requirements that have been added to the Requirement Group. The selected Requirements are displayed as a sub node in the tree view.
A test case is a set of conditions or variables under which a performance tester will determine if a non-functional or technical requirement upon an application is partially or fully satisfied. A test case defines:
What has to be done (Execution Plan), in order to determine requirement satisfaction (load generation, monitoring, etc.)
which requirements (Requirements) have to be satisfied
the configuration (Configuration specification), that has to be applied by the test
the environment (Deployment scenario), that hosts the test

To meet the various goals of a project, the tests in your project can be organized into unique clusters. A Test Cluster can be duplicated, but only deleted if no dependencies exist.
Project > Test Clusters > context menu: Create


Name (mandatory field): The name that TAPE assigns to the object.
Description: In the description box, add a description of the object
Test cases:
Available: All available Test cases
Selected: Selected Test cases that have been added to the Test Cluster. The selected test cases are displayed as a sub node in the tree view.
A Test Case is defined with one deployment scenario, one configuration, zero or more requirements and a mandatory execution plan.
Project > Test Clusters > Test Case > context menu: Add new Test Case


Name (mandatory field): The name that TAPE assigns to the object.
Description: In the description box, add a description of the object
The in the Requirement module previously defined requirement is linked to the appropriate test case and appear as a sub node to the Test Case in the tree view.
Project > Test Clusters > Test Case > Requirements

The Deployment Scenario is selected for the Test Case.
Project > Test Clusters > Test Case > Scenario

Deployment: Scenario: [Automatically selected]
Configuration Specification: Select the configuration specification
The Execution Plan defines in which order the Task Groups are started and stopped.
Project > Test Clusters > Test Case > Execution Plan

A Task Group is a set of tasks, which have a common goal (e.g. starting monitoring or driving the tests).
Project > Test Clusters > Test Case > Execution Plan > Task Group: context menu: Create

A task is an action associated with a deployment unit and its TAPE Agent
Project > Test Clusters > Test Case > Execution Plan > Task Group > context menu: Create new Task


A Configuration groups the Configuration Items
Project > Test Clusters > Test Case > Execution Plan > Task Group > Task > Configuration
Name (mandatory field): The name that TAPE assigns to the object.
Description: In the description box, add a description of the object
A Configuration Item is a set of key/value pairs organized into categories (e.g. Java Properties).
Project > Test Clusters > Test Case > Execution Plan > Task Group > Task > Configuration > Table: Config Items > context menu > Create

Name (mandatory field): The name that TAPE assigns to the object.
Content: In the Content box…
In File (check-box):
Is User-Definable: …
Description: In the Description box, add a description of the object

The task uses the following naming convention:
<Configuration Item> @ <TAPEAgent> @ <Configuration>
The Rules determines in which order the Task Groups are be started and/or stopped.
Project > Test Clusters > Test Case > Execution Plan > Table: Rules

|
Rules: |
Task Group |
Status |
Next Task Group |
Command |
|
Possible states: |
[User definable] |
Started / Stopped |
[User definable] |
Start / Stop |
|
Comment: |
Select Task Group |
|
Next Task Group |
Start or Stop Task Group |

Decide what Report- and Chart templates that are to be used to display the result of the test run.
Project > Test Clusters > Test Case > Analysis

The Analysis module helps you assess the result of the tests. However, before this can be done, an Analysis Template that defines the desired layout must be created.

The Aggregated Chart Template allows the user to define and display charts of aggregated values.
Create:
Analysis > Analysis Templates > Chart Templates > Context menu: Create Aggregated Chart Template


Name (mandatory field): The name that TAPE assigns to the object.
Title: The title is displayed in the header of the
Add Legend (check-box): If selected, an explanatory list of the symbols appear on a map or chart
X-Axis Label: Enter the name for the X-Axis
Y-Axis Label: Enter the name for the Y-Axis
Chart Type: Select what type of chart is to be displayed
Group by: Select according what the chart should grouped by
KPIs:
|
Component: |
KPI |
Deployment Unit |
Deployed Artifact |
Aggreg. Func. |
Parameter |
|
Possible states: |
[User definable] |
[User definable] |
[User definable] |
SUM, MIN, MAX, AVG, PERCENTILE |
[User definable] |
|
Comment: |
Select which Key Performance Indicator you want to measure |
Select the physical machine in your testing environment |
Select the Artifact that has been deployed |
Select the aggregated value to be computed |
|
Time Series Chart Template
The Time Series Chart Template enables the user to display the test in a line chart, plotted over time.
Create:
Analysis > Analysis Templates > Chart Templates > Context menu: Create Time Series Chart Template


Name (mandatory field): The name that TAPE assigns to the object.
Title: The title is displayed in the header of the
Add Legend (check-box):
Value Axis 1: Enter name for axis 1
Value Axis 2: Enter name for axis 2
KPIs:
|
Component: |
KPI |
Deployment Unit |
Deployed Artifact |
Axis |
|
Possible states: |
[User definable] |
[User definable] |
[User definable] |
1, 2 |
|
Comment: |
Select which Key Performance Indicator you want to measure |
Select the physical machine in your testing environment |
Select the Artifact that has been deployed |
Select against which axis the chart will be plotted |
Report Template
The Report Template provides the possibility to create a report in which the result of the tests can be presented with graphs and text.
Create:
Analysis > Analysis Templates > Report Templates > Context menu: Create

Name (mandatory field): The name that TAPE assigns to the object.
Description: In the Description box, add a description of the object
Define the contents of the pdf-document
XML (context menu):
Add Headline 1: 1st headline of the document
Add Headline 2: 2nd headline of the document
Add Text: Text to describe the test
Add Chart: Define the chart to be displayed
Add Chart Template: Define the chart to be displayed
Move Up: Move item up
Move Down: Move item down
Delete: Delete item
Please see the Appendix for an example Report
Test Case Analysis
Toggle between test cases


Basis template: Select the template that the report will be based on
Name: The name that TAPE assigns to the object.
Selected test run(s) to compare: Available test runs
Available test run(s): Test runs added to the report
Create a report based on selected test runs.
Create:
Analysis > Test Case Analysis > Reports > Context menu: Create

Template:
Name: The name that TAPE assigns to the object.
Create a chart based on single test run.
Single Test Run - Charts

Template: Select the Template you want to use
Name: The name that TAPE assigns to the object
Single Test Run - Report
Create a report based on a single test run.

Template: Select the Template you want to use
Name: The name that TAPE assigns to the object
Charts

Basis template: Select the Template you want to use
Name: The name that TAPE assigns to the object
Selected test run(s) to compare: Available test runs
Selected test run(s):
Reports

Template: Select the Template you want to use
Name: The name that TAPE assigns to the object
Selected test run(s) to compare in chart templates:
Selected test run(s) for chart templates:
Charts:

Basis template: Select the Template you want to use
Name: The name that TAPE assigns to the object
Selected test run(s) to compare:
Selected test run(s):
Reports

Template: Select the Template you want to use
Name: The name that TAPE assigns to the object
Selected test run(s) to compare in chart templates:
Selected test run(s) for chart templates:
The TAPE tracking module helps you to assess the progress of your tests. You can for example with the pre-defined queries measure the number of executed tests over time and which requirements that must be tested.
Display
Tracking > Queries > Plan and Time > [select query] > Context menu: Display
Specify the time period for which you want to view data.


|
Report |
Description |
|
Test cases in a time-period |
Lists the test cases in grid format, that were created in a specific time period |
|
Test runs in a time-period |
Lists the test runs in grid format, that were created in a specific time period |
|
Test cases ready to test |
Lists the test cases in grid format, that have not yet been executed |
Display
Tracking > Queries > Test Efficiency > [select query] > Context menu: Display
Specify the time period for which you want to view data.


|
Report |
Description |
|
Amount of Test runs in a time-period |
Displays the amount of test runs over time in a graph |
|
Amount of Test cases in a time-period |
Displays the amount of test cases over time in a graph |
Display
Tracking > Queries > Requirements > [select query] > Context menu: Display

|
Report |
Description |
|
Test case with unfulfilled requirements |
Lists the test cases in grid format, that are linked to unfulfilled requirements |
|
Unfulfilled requirements |
Lists the requirements in grid format, that have not yet been fulfilled |
|
T ©
C1 SetCon GmbH D-80337 Munich Germany
Please refer to the About dialog box for the list of licensed external libraries and components.
|