All the Pieces Matter
I like puzzles. I’m the type of person who, when presented with a challenge, tries to make sure all the pieces fit. The challenge of high-level customer support is that we don’t usually see the picture on the box; we’re just dealing with a handful of puzzle pieces.
Yes, I’m going to use this puzzle analogy throughout this post so strap in.
First a little background about me. I’m from the cold, east coast (Toronto) and grew up on the Blue Jays, Raptors, and Leafs (sigh). I’ve been at this for a little while now; I worked on my CCNA back when there was no such thing as wireless internet (for those of you doing the math on that – stop). I cut my teeth at IBM working on AS/400, AIX, and then midrange and enterprise Storage. From there I moved on to VMware Storage, supporting hundreds of customers per week.
As with anything at that scale there was plenty of repetition, but every once in a while I’d be talking to a customer and thinking “well that’s interesting…I wonder why they do it that way?” There’s a lot of creativity out there in IT, despite what the rest of the world may think of us, and this creativity brings it’s own set of challenges. For every elegant solution to a problem, there’s also someone’s custom script they wrote which only serves to work around some limitation of the hardware they inherited from the previous admin. Working with those challenges taught me a lot about customers’ pain points and the various ways they deploy and implement the products we sell. As I transitioned over to Coho to help build our support organization it was important to me to keep one core idea at the forefront: Support is a partnership between us and our customers – Support has the puzzle pieces, customers have the picture on the box.
With that in mind, let’s talk about how this partnership helps us help you (That’s the first and last Jerry Maguire reference, I promise):
How Customers are Installing and Deploying
Most of you know the pain of planning a new Storage installation; blocking off a day or two to get everything racked, cabled, and powered up–don’t forget the ever exciting LUN/volume creation. If you add in the Fibre-Channel stuff, it’s enough to make you pull out your hair, and that’s before you’ve even used the storage!
Yep, I’ve been there…
For our initial deployments, we would send one of our SEs on site to rack the system, perform a general configuration task, and then assist the customer with any additional tasks such as VAAI plugin installation. This was going to be the norm, but through some discussion we decided we could and would do better.
Nowadays, on-site SE-led installations are rare (necessary in more secure environments) and the typical installation looks something like this:
- We ship a semi pre-configured system (you specify your desired Management IP address prior to shipment)
- The gear arrives
- You rack the system and power it on
- Cable your ESXi hosts
- Login to vCenter and create your datastore
- USE the storage
- Delivering Value to users
I like this deployment strategy a lot better as you don’t have to pull your hair out to get up and running. In fact, the time from powering on the system, following the installation wizard, through to deploying VMs is typically well under 1 hour.
Hair? Check. Emails asking when the installation will be finished? ….None!
How Customers are Using the Coho Datastream
Data is the lifeblood of an organization with many different sub-industries tackling the different aspects of data management, access, and availability. There’s virtualization aiming squarely at data density and efficiency, networking chiefly looking at data delivery, and storage managing data access and availability – in addition to, of course, physically storing the data.
By providing a dedicated 10G switching infrastructure, we can guarantee data delivery. Having a dedicated 10G link from each ESXi host allows for efficient use of your server resources and allows us to provide the density that is needed to make it all worthwhile. Data access and availability are paramount; areas we tackle by using 2-way synchronous replication and failure domains.
As a hybrid system, we leverage PCIe flash and SATA disks and continually adapt to the workloads being run on our system. The system actually learns (not in a skynet kinda way) and gets better the more you use it. A lot of customers have had questions about this–wondering how they can move data within the system to deliver the best performance possible, and efficient use of the PCIe flash. These conversations are always fun as the relief on the other end of the phone when we say “you don’t need to tune anything; it’s automatic” is palpable.
We’re focused on keeping the working set, the data an application needs to run well at a given point in time, in flash and moving the less useful data to slower media. As you can imagine this is not a fixed set of data, so as the workloads change so does the working set and thus our data placement follows suit. Our Customers recognize the power in this approach and the key factor in deciding the size of their Coho deployment; working set size.
How Customers are Troubleshooting with Coho
Our UI delivers a ton of useful information; IOPS, Throughput, and Latency both system-wide and on a per-workload basis. Customers use this information in-house to diagnose issues and get the right resources involved.
For example, if you have an SQL server which is sluggish, the first inclination is to investigate the storage. So you load up your Coho UI and check on the performance statistics. You don’t notice any latency spikes or dips in throughput so you call up the DBA. Sure enough, the issue is elsewhere.
Let’s face it though, things break, and when they do you want to be empowered enough to know what’s wrong so you can contact the right people. Through our UI, you can easily download the log bundle and email Tech Support (Note: please call us if it’s a serious issue). For performance-related calls, the UI has enough history for you to pinpoint the day (or hour) when things changed, all of which are valuable pieces of information for the puzzle.
The most common issues we’ve seen relate to the initial networking configuration as our system is deployed into an existing 10G infrastructure. We support one layer of switching between the ESXi hosts and our system:
The indirect approach shown above alters that all-important picture on the box and presents it’s own unique challenges. We present a single NFS IP and MAC address which is handled easily if you connect to us with two independent switches (one static LAG per switch – shown by the red oval above), but if you use stacked switches to connect to us the configuration needs to be altered:
That diagram was created as a result of a customer engagement, and it took a day or two before the fact that the intermediate switches were stacked was shared. Once the networking piece was taken care of we were able to get things running smoothly fairly quickly.
How we Learn, Adapt, and Adapt again (and again)
Despite our best efforts, we’re not perfect.
Sometimes what we think is simple and clean is lacking detail. I mentioned earlier the way we automatically, dynamically, move data between the flash and disk in response to changing working set – but what if you want to see just how much of your active I/O is being served from flash vs. disk? We admittedly did a poor job of this at first, but through some great customer feedback we’ve been able to incorporate this into our 2.x release.
Our customers are also innovative–what if you frequently want to re-synchronize data across several VMs and leverage hardware acceleration to do so? After this question was posed to us, we had just the thing.
Maybe you’re big on reporting and the pretty graphs in our UI are nice, but you really want to pull that information out into a format you can work with it easily (csv for example). In response to this, we provided our customers with a sample python script and documentation to use our RESTful web service, interacting with a JSON-encoded namespace of our configuration and control APIs, to make this reporting possible.
Or maybe you don’t care about any of those things. You’re not one for glitz and glamour, pretty UI be damned. You just want to be able to take snapshots and clone VMs with a script and get on with your life. You asked and we delivered; using the same web service and a different API endpoint, we crafted sample python and powershell scripts with the necessary documentation.
We encourage our customers to share their ideas, suggest improvements and features, to partner with us. We’ve improved our UI Alerting and our Software Update functionality stemming from Support interactions with customers. Product Management and Engineering work closely with us to evolve the product; keeping one eye to the future while incorporating the feedback of our customers every step of the way.
Completing the puzzle
As I mentioned earlier, I joined Coho to help build our Support organization. Developing true partnerships with our customers and playing a key role in pushing the company forward. This credo is in our DNA; from the CEO to our Twitter handle – this is who we are.
We would be nothing without our customers, they support us, they push us to be better, they partner with us. Our customers allow us to learn, iterate, and innovate. We’ve grown as fast as we have as a direct result of those who work for us and with us.
I guess what I’m saying is: bring on the puzzle. Let’s share the picture on the box and put the pieces together, together.
3,348 total views, 2 views today
The post Four things Customers Teach You (Supporting Storage) appeared first on Coho Data Blog.
