We often visit prospects who want to replace their existing storage arrays due to challenges with performance, cost, scalability, or ease of use. On performance, prospective customers want assurance that any new solution will perform better, speeding up their business applications and making their users happier. Seems simple enough.
Companies frequently test potential storage solutions for different business cases, looking for validation of their choice, but not in ways that demonstrate the true nature of the proposed solution. We thought sharing some considerations for performance testing would be helpful.
We believe that the best way to test storage system performance is to run real workloads on it. Testing systems with real workloads provides infrastructure and application teams the best understanding of how the entire system behaves at sustained and peak loads for multiple applications. While it requires more time and effort to run a test like this, it seems more reasonable to spend a few weeks or even a couple of months to select a system that will be a key part of the IT operation for years to come.
The next best option is to use the real applications with simulated user workloads. Using simulated workloads is a reasonable choice, especially when the application has specific, familiar patterns, like batch jobs. Batch jobs typically have a well-defined completion time, and that makes it easier to compare performance between different environments.
The challenge here – you still need to have all of the required equipment on site or provide real data to the prospective vendor. While the data could be anonymized, that also takes time and money. Furthermore, you need a method to simulate user interaction, typically requiring extensive scripting or specialized simulation software. Companies with huge IT budgets have the teams and equipment dedicated to continuous test development and execution. For many businesses, that level of effort is considered excessive and not affordable.
Nowadays, a very common way to test storage performance is by running synthetic workloads in a controlled environment, often in the vendor’s lab. It is quick and requires limited investment beyond the time to pick which tests to run. The challenge with this approach is modeling the workload and properly interpreting the results to make sure they are aligned with how the system will operate in production.
To address this challenge, we collect performance metrics from customer’s existing arrays and extensively customize the synthetic workload to closely mimic the customer’s real workloads. We believe we have established a tried and true method for gathering and analyzing this data (storage tiers, sub-LUN workload distribution, workloads by IO sizes, cache hits, etc.) based on years of experience in storage array performance testing and tuning. Comprehensive testing, including performance, is a key part of our operational plan and ingrained in our company culture.
Another important note about synthetic workload testing – we often see peak workloads emulated incorrectly. Peak workloads differ significantly from the average or standard workload in terms of cache hits, read percentage, and IO size, just to name a few. Why does this matter? In our experience, simulated peak load tests are too short, just minutes long, with short bursts of write intensive IO that can be kept easily in write cache and then destaged later. Emulating peak workloads for a longer period of time, like tens of minutes or even hours, depending on the size and randomness of the data, is a better way of reflecting real world workloads. Taking this approach should overflow the write cache and force the system to destage to the media. THAT is where peak performance comparisons are won or lost.
What do you do if you want to test an application workload for an application that you don’t own yet? Ask the application vendor for workload profiles or ask a friend. You can also ask INFINIDAT customers that have already implemented that application on an InfiniBox. With over 500 petabytes of storage capacity deployed across almost every possible industry, we’ll find you someone to compare notes with.
Frankly speaking, storage performance testing has never been easy to do, given the time and effort required to plan, perform, and analyze the results. Even so, it is critical that anyone looking to acquire a new storage system take the time to choose an approach that will be the most effective in helping to decide where to put their business critical workloads. Hopefully this post has given you a few things to consider for your next storage performance evaluation.
If you have any questions about understanding your array’s performance, let us know. We’ll be happy to help you figure it out and show you what INFINIDAT can do for you and your business.
About Vasily Kostryukov
Vasily is a corporate technical advisor and performance specialist at INFINIDAT. He has worked in IT for over 20 years, including a nine year tenure at EMC. Over the last 15 years, he’s focused on Unix systems, storage infrastructure, and data management solutions.