Cloud Based Load Tests with Visual Studio Online
An important part of the application life cycle is the Testing phase which validates and verifies whether your application confirms to the requirements and whether its built accordingly. You can create and run test plans, test suites and test cases in Team Web Access, or you can easily launch Microsoft Test Manager (MTM) to manage and run test cases. When you launch MTM, Team Web Access starts in the context of your team project, so there is no need to explicitly connect to your VSO Team Project.
The main objective of cloud based load testing is to test the capacity of a single server, uncover any serious performance issues, and identify opportunities to improve the capacity or performance of the service.
One of the advantages of using the Visual Studio Online Load Testing Service, is that you can take the same web tests that you have created and used on premises, and execute them on the cloud with no additional effort.
Benefits of cloud based load testing:
- Scale infinitely: Generate hundreds of thousands of connections in minutes. You have a wide range of users to test your application in the cloud.
- Easily create powerful load tests: Quickly create loads by recording steps. Run them or customize them using the VSO. You can also use the functional tests to create loads.
- Test from where you’re users are: Take advantage of the cloud capabilities to run tests from one of many global Azure datacenter locations to minimize latency and simulate users’ real-world conditions.
- Free and flexible pricing : Visual Studio Online charges the users based on the per virtual user minute (VUM). It is a measure of how long your test is and how many users the test simulates. The first 20,000 virtual user minutes are free each month.
- Deep Reporting and Diagnostics: View the apps performance with real time statistics and charts. Using the advanced tools such as Application Insights, you can correlate test results with server diagnostics.
Terms used in a Visual Studio Online (VSO) cloud based testing
- Virtual users – Virtual users create load by accessing your app or web site all at the same time during your test run. That way, you can test performance under more realistic or projected conditions. Virtual users are simulated by test agents.
- Test Agents – Test agents are computing resources, like CPU, memory, and network, that generate load by simulating virtual users. Test agents use agent cores to create virtual users.
- Agent Cores – The maximum number of cores for the load test agents for each run is 100 cores. If your test runs need more cores, you can run 10 load tests at the same time. The minimum number of virtual users per agent core is 1.
Points for consideration when using Cloud based load tests with VSO
- There are time limits when running the load based tests. Depending on where you’re running the test, each test run duration limit varies from 1 hr to 48 hr (for VS IDE, VS Team services or the Azure preview portal).
- The virtual machines that host your agents host only one load test run at a time.
- Even when you are behind a firewall, you can load test apps or sites that are only available to your company, like internal or pre-release apps, staging or preproduction deployments, using Trusted IP.
- Some load test features are not supported when you run the tests in the cloud like Goal-based load patterns, Iteration-based runs, Network mix property, SQL Trace properties in run settings, IP switching.
- You can monitor, debug and check the status of the load tests when they are running in the cloud.
- The downloaded results file is stored in the a local SQL Server Express database. You can change the default location so that all the users can access the same results file in the same database.
- The Azure SQL Server is more ideal for storing large reports as you run more load tests in the cloud.
- Cloud based testing services may have issues which makes the load tests to abort during the run or to get errors. You can reload the tests or get help from the Visual Studio Team Services.