Making Your Tests Scalable

In a previous topic we discussed how you can make your tests portable in terms of IP addresses by using aliases. dsTest also provides features that enable you to easily scale your tests - either to achieve the desired load on the DUT or to dynamically adjust your tests based on the capacity of a platform during automated testing.

Using Variables

Along with host and interface aliases, your alias configuration can include Variables. Most of the scalar attributes and elements in your test configurations (e.g., subscriber count, node count, or transaction count) support the use of variables and simple mathematical operations upon those variables. You can determine whether an attribute or element supports the use of variables by its type, which will begin with "variable" - variableU32, for example, provides variable support and can accommodate an unsigned 32-bit integer as the resulting value or as a static value.

You can define a variable's value as the result of an operation on a variable that was previously defined, and if you need to perform a complex, multi-step calculation you can establish variables to store the interim results. If you're going to use a value more than once, establish a variable to hold that value; otherwise, simply define the desired operation as the attribute's value.

When you reference a variable, use the syntax @<variable name>. The at sign (@) notifies dsTest that it is dealing with a variable rather than a text value.

The most common variable usage is setting subscriber count and then using that variable to set the values that should adjust relative to the number of subscribers. In this case it's usually best to define the variable as part of your test configuration. If you want to change the variable's value and run the test again you can use a command to change the value as shown below.

Example

Assume you are testing a DRA that routes Gx transactions between your PCEF node and your PCRF node, and that you'd like to run this test with increasing loads to find the DRA's breaking point. Your test has a SmartEvents state machine that will keep each subscriber's session alive for 30 seconds as it generates approximately 0.5 TPS. The action that launches SmartEvents will run until you stop it and it will keep all subscribers active for the duration of the test - as their sessions end they'll initialize new sessions.

First, we determine the variables required:

1.subCount will store the initial number of subscribers.

2.averageTPS will be calculated as subCount divided by 2 and will represent the target TPS for the test.

3.pcefCount will store the number of PCEF nodes.

We'll use these variables to size the subscriber database, set the Gx transaction count and expected TPS on our nodes, and set the number of PCEF nodes used in the test. The variables will be configured with overwrite set to false which will allow us to run the test, use commands to change subCount and/or pcefCount, and then load the test again while preserving the values we modified.

The pertinent portions of the test configuration:

  <config>

      <aliases>

        <variable name="subCount" overwrite="false">10000</variable>

        <variable name="averageTPS" overwrite="false">@subCount / 2</variable>

        <variable name="pcefCount" overwrite="false">1</variable>

      </aliases>

    <nodes>

        <spr name="spr_1">

            <subscriber_database count="@subCount" name="spr_subs">

              <subscriber_profiles>

                  ...

              </subscriber_profiles>

              <subscriber_group percentage="50" name="bronze_subs">

                ...

              <subscriber_group percentage="50" name="gold_subs">

                ...

              </subscriber_group>

            </subscriber_database>

        </spr>

        <pcef name="pcef_1" count="@pcefCount">

          ...

          <gx count="@subCount / @pcefCount" tps="@averageTPS / @pcefCount">

            ...

          </gx>

          ...

          <subscriber_database name="spr_subs"/>

        </pcef>

        <pcrf name="pcrf_1">

          ...

          <gx count="@subCount" tps="@averageTPS">

            ...

          </gx>

          ...

          <subscriber_database name="spr_subs"/>

        </pcrf>

    </nodes>

  </config>

Note that the Gx transaction count (10,000) and expected TPS (5,000) on the PCEF are configured to divide by pcefCount as subscribers will be spread among those nodes when we increase pcefCount, but the PCRF will be serving all subscribers. Also note that we're using the percentage method to size our subscriber groups and that will enable the group size to adjust when we change subCount.

To change subCount and pcefCount for the next run we issue the following commands:

dsTest> aliases variable:"subCount";overwrite:true::50000

dsTest> aliases variable:"pcefCount";overwrite:true::5

Now we have 50,000 subscribers that will be spread across 5 PCEF nodes. Each PCEF node will accommodate the same level of transactions as in the initial configuration but the PCRF will support 50,000 transactions and 25,000 TPS.

Changing a variable's value while the test is running will have no effect on the current test. dsTest only analyzes variables and resolves variable values during test initialization.

Dynamic Timing

Another option you can implement to design a scalable test is SmartAction and its interaction with SmartEvents. With SmartAction you specify the subscriber group, the target action rate, and the number of subscribers that should be active at any point in time. dsTest will then use that information along with the supporting options in your SmartEvents configuration to adjust the timers in your state machine in an attempt to meet the goal defined.

In order to use this feature, your SmartEvents configuration should include the Average Transaction Duration and Average Protocol Transactions attributes. The value of Average Transaction Duration should be the expected duration, in milliseconds, of a SmartEvents transaction - in other words, the time it takes for one subscriber to progress from idle through the state machine and return to the idle state. Average Protocol Transactions should be provisioned with the number of transactions - in this case request/response message exchanges - generated during a SmartEvents transaction. dsTest will calculate default values for both attributes if they are not provisioned.

To identify the timers that may be modified, set the Adjustable Duration attribute on the Start Timer actions in the event handlers that control your state machine timing. You should not set this flag on timers used to identify timeout error conditions.

You can also use variables here to achieve full scalability. The Target Non-Idle Subscriber Count element in SmartAction provides support for variables.

Example

Using the example test above, we would set the SmartEvents attributes thusly:

<SmartEvents average_duration="30" average_proto_tran="15">
  ...
</SmartEvents>

We would also use two SmartAction commands - one for each subscriber group - to execute the test:

<SmartAction name="bronze_group">
  <rate>100</rate>
  <cycle>-1</cycle>
  <group>bronze_subs</group>
  <target_non_idle>@subCount / 2</target_non_idle>
</SmartAction>

We could take it a step further and define a variable to be used as the target_non_idle value. That would give us finer control over the test's TPS from one run to another without changing the number of subscribers.