SmartReport is designed to provide you with an overview of your test results, including the success or failure of the test, while also providing a statistical analysis of those results as an aid to troubleshooting failed tests. Unlike Chart Reports and Tabular Reports, where you define the composition of the report, SmartReport summarizes all measurements generated during your test by measurement category and produces a summary chart for each status category along with application layer transactions-per-second (TPS), user data packets-per-second (PPS), and mega-bits-per-second (MbPS) where applicable. This type of report can only be generated from dsTest OM databases.
Since SmartReport summarizes most of the measurements generated during your test when it opens, it may take some time for the report to be displayed depending on the size of the test, in terms of either time or objects, and the amount of memory you've allocated to the application. You'll see progress bars at the bottom of the application window as the measurement calculations proceed.
The SmartReport screen is comprised of a collapsible pane containing the Dashboard, which displays your pass/fail criteria and results, and the Summary Charts tab containing SmartCharts that coincide with each criteria block in the Dashboard. You can include SmartReport in your automated testing and produce a Dashboard summary for each test by using the HTML Test Results feature. In this topic we'll explain how to define the criteria for a test, how to work with SmartCharts and drill down into more detailed data, and how you can use statistical analysis to isolate the root cause of a failing test.
Defining the Criteria for a Successful Test
You define the criteria SmartReport uses to determine test status as SmartReport Profiles on the Reports tab in your Preferences (File>Preferences). A Default Profile is provided and you can simply modify it to suit your needs. You can also create a new profile (), copy an existing profile (), or delete a profile () using the buttons in the toolbar above the profile list.
You must give unique names to your profiles and within each profile you can use checkboxes to select the summary charts and test criteria to include.
Application Status includes a chart that displays the aggregate application layer transaction attempts, successes, and failures across all interface applications involved in the test, and you can define Success Criteria as the percent of attempts that must succeed.
Application Action includes a chart that displays actual aggregate actions attempts vs the configured target action rate along with action failures and the typical causes of a failure to achieve the target rate - no subscribers available and delays. In this case you can define the required Target Achieved Criteria, the percent of the targeted actions that must be attempted, in addition to Success Criteria, the percent of actions that were successful.
Application Transactions/Second includes a chart that calculates the aggregate TPS across all node emulators and interface applications involved in the test. The success criteria for TPS includes the Target Achieved Criteria (TPS) and a tolerance range TPS Tolerance expressed as the percent of target the actual TPS can exceed or fall short of the target TPS. In addition, you can optionally define a sustained TPS requirement as either Consecutive Intervals Criteria or Percent of Intervals Criteria.
Transaction Duration includes a chart that calculates aggregate transaction duration across all node emulators and applications involved in the test. Maximum Duration Criteria sets the upper limit, in microseconds, that is allowed in a successful test. This criteria is best suited for tests that either use all stateful applications or all stateless applications with comparable transaction durations.
User Data Rates includes two charts that calculate aggregate user data throughput - User Data Packets/Second and User Data Mb/Second - and for each rate you can define Target Achieved Criteria, Tolerance, and the optional duration criteria, Consecutive Intervals Criteria or Percent of Intervals Criteria.
Transport Status includes a chart that displays aggregate attempts, successes, and failures for all sockets/physical interfaces involved in the test, and you can define Success Criteria as the percent of attempts that must succeed.
Protocol Status includes a chart that displays aggregate attempts, successes, and failures for the base control plane protocols involved in the test (e.g., Diameter, REST, GTP), and you can define Success Criteria as the percent of attempts that must succeed.
dsClient will only display the charts and Dashboard summaries that are applicable to your test. You can define User Data criteria, for example, and it will only be displayed if the test included user data traffic.
Adding a Custom Summary Chart and Test Criteria
If there is a particular OM or set of OMs that you monitor to determine test status, you can define criteria for a Dashboard Summary and/or display them in a Summary Chart. Select the Show Custom Chart/Dashboard checkbox to enable the settings.
The example above is configured to add a custom chart that displays the latency delay measurements for Gx and Rx answer messages. These OMs measure the duration from the time a request was sent until the corresponding response was received by dsTest. The text we entered in Chart/Dashboard Title will appear as the custom chart's title. We then added the OM IDs, which you can find in the application's command section of the schema documentation, for the relevant measurements by entering the ID in the text field provided and then clicking the add () button. You can remove an ID by selecting it and then clicking the delete () button. The resulting SmartChart, shown below, displays the aggregate values for each OM, and you can use the SmartChart menu (more on that in the next section) to drill down to detail values by node type and/or node instance if the chart is displaying summarized values.
To add a Dashboard Summary, select the Show Dashboard checkbox and define the criteria. The selections for Target Calculation will become available depending on how many OM IDs you have selected. Compare 2 OMs performs a subtraction based on the order of the IDs in the list, while Sum Selected OMs and Average Selected OMs can operate on any number of measurements. The None option will test the value of a single OM. As with the standard Dashboard Summaries, you can then define the minimum value required for success in Target Achieved Criteria along with a Target Tolerance range and optionally a duration requirement with either Consecutive Intervals Criteria or Percent of Intervals Criteria.
Working with SmartCharts
SmartCharts include the features found in Chart Reports - report mode, zoom, synchronized time lines, selectable legends, etc. The difference here is that you're working with measurements summarized by category rather than by OM. When SmartReport opens the Summary Charts tab displays the charts specified in the SmartReport Profile you selected and each chart plots the summarized values of the OMs within its category. Whenever more detailed measurements or diagnostic measurements are available a SmartChart Menu will be available as indicated by the context menu icon () in the upper-left-hand corner of the chart. The options on the menu will vary depending on the chart and the objects involved in the test, but in general there are three "paths" that may be explored from a particular chart - the details path, the diagnostics path, and the messages path - with new charts created at each step. As new charts are created they will be placed on tabs depending on the measurement category: Application, User Data, Protocol (i.e., control plane protocol), Transport, or Server.
As you open charts dsClient will consume more memory - particularly when tests span thousands of report intervals. If the application's response time increases, you can reclaim resources and improve performance by closing charts or tabs that you're not using.
The Details Path
Whenever the values plotted on a chart have been summarized, the chart's SmartChart Menu will present a details path. The options on this path will create charts first by node type, if different types of nodes are represented in the chart, and then by node instance, if multiple instances of one type of node are represented in the chart. If there are more then ten instances of a node you'll have options to create charts for a range of node instances. The second option on the details path will be presented if different types of objects, or a large number of objects, associated with the top-level category are present. For example, within the Application category you can choose between interface applications while you can choose between different protocols from a Protocol chart. In any case where multiple options are present an All option that creates all available charts will also be present. At any point you can branch off onto the diagnostics path or the messages path.
The charts you create from the details path will be placed on the applicable category's tab.
The Diagnostics Path
Any non-diagnostics chart offers a Diagnostics sub-menu with three options: <category> Diagnostics, Statistical Analysis, and Server Status. This is the path to take if you're troubleshooting a failed test.
The diagnostics chart for a category summarizes all error measurements in the category and also plots major causes for failures depending on the category - No Transactions Available in the Application category and Congestion in the Protocol category, for example. From a diagnostics chart you can drill down to more detailed values by taking one of the Diagnostic Details options or you can take the details path and break out diagnostic measurements by node or object.
The Server Status chart displays information regarding resource availability and consumption on the dsTest platform, and from it's SmartChart Menu you can create a Server Diagnostics chart. The Server Charts tab hosts all server SmartCharts.
You can learn more about the Statistical Analysis option in the next section.
The Messages Path
The messages path creates charts on the Application tab, beginning with the summary-level Application Messages chart that plots Requests Sent, Requests Received, Responses Sent, Responses Received, Transport Sent, and Transport Received across all applications. If more than one type of application is present you can use the details path to isolate an application or node type. When you have a messages chart that represents one type of application the Message Details option will become available and that will create a chart that plots application-layer messages sent and received by name.
Chart Control Options
In addition to the SmartChart Menu, SmartReport charts include special chart control options:
Spotlight Zoom - This option is available when the time line is zoomed in to a subset of report intervals, and it replicates the current chart on a spotlight tab, indicated by the zoom icon (). The newly created chart, and any charts spawned by it will only report that subset of intervals. This allows you to hone in on an interesting time frame while also conserving memory.
Export to Chart Report - You can export the current SmartChart data to a standard Chart Report using this option. If the current chart plots summarized values you can chose which summary measurements to use in the Chart Report - by node or by application.
Using the Statistical Analysis
When dsTest closes a database it launches a utility that calculates the mean, standard deviation, and rate of change for all measurements in the database. SmartReport is the only report that utilizes this information, and it can be invaluable when troubleshooting a test that's failing due to reasons other than protocol or application errors. You can access this information by selecting Statistical Analysis from the SmartChart Menu of any chart in the Diagnostics path or from the Diagnostics sub-menu of any other SmartChart in the report.
The Statistical Analysis dialog displays all measurements that experienced a value change from one interval to the next that ranked in the top ten rates of change during the test. The element names of the node emulator(s) that generated the OMs are displayed in the Owner column. Rate of change is expressed as an order of magnitude change to standard deviation. In the example above, all measurements that experienced a change between 26 and 37 magnitudes have been included. Each rank is color-coded and a dot is plotted on the time line for every occurrence. Consecutive intervals at the same magnitude of change would result in a line. In this DRA test example, sorting by magnitude made it apparent that Transaction Duration OMs and outgoing message Queue Depth for both client and server nodes is changing rapidly indicating that the DRA was overwhelmed, which proved to be the case.
There are several ways in which you can interact with this dialog. It will remain open until you click the Done button.
The table is sorted by Measurement ID by default, which groups measurements by application or interface. Select Magnitude (descending) to sort by rate-of-change, which can bring together measurements from diverse objects and highlight trends.
As with chart legends, you can select a Magnitude from the list to highlight those occurrences in the plot area. Ctrl-click to restore all magnitude plots.
Click and drag within the plot area to zoom in on a particular time span. Click Zoom Out to restore the original time line.
Use the checkboxes in the Delete column to select uninteresting OMs to remove - measurements that are only registering due to the test ramping up or halting, for example - and then click Update Table to delete those rows.
The checkboxes in the Chart column enable you to create a new chart containing the selected OMs and spanning the time determined by your selection of Include All Intervals in the report or Time Frame Shown in the dialog. Click Create Chart when you're satisfied with your selections and the new chart will be added to the Statistical Analysis Charts tab. If you restricted the chart's time frame you'll create a Spotlight Zoom chart. If the measurement data contains multiple objects the chart will display a SmartChart Menu where you can drill down into lower-level details.