Yesterday we blasted DCIG for their inept attempt to provide a buyer's guide for all flash arrays. As it turns out, the report is worse than expected. Upon our post the author, Ken Clipperton, took to Twitter to defend the DCIG stance. Since then, the situation has devolved further, with multiple vendors pointing out that the report is wildly inaccurate to the point of being intentionally malicious. While calling fraud is perhaps a bit strong, there does appear to be an affinity between those who pay for and promote the DCIG "awards" vs. those who do not.
Yesterday we blasted DCIG for their inept attempt to provide a buyer's guide for all flash arrays. As it turns out, the report is worse than expected. Upon our post the author, Ken Clipperton, took to Twitter to defend the DCIG stance. Since then, the situation has devolved further, with multiple vendors pointing out that the report is wildly inaccurate to the point of being intentionally malicious. While calling fraud is perhaps a bit strong, there does appear to be an affinity between those who pay for and promote the DCIG "awards" vs. those who do not.
As noted, shortly after our first post Ken weighed in last night:
@storagereview I co-authored the report. Not a paid endorsement! Everyone wants an "A", we rank on a curve. Read the guide! Then let's talk.
— Ken Clipperton (@KenDCIG) October 5, 2015
It took almost no time for us to hear from many on the buyer's guide list however that there were problems with the data. Checkboxes incorrectly placed and some omissions that only required looking at a picture to realize DCIG had made a mistake. Clearly no one fact checked the report or else gems like this would have probably been spotted. The EMC VNXe3200 is purported by DCIG to have controllers that cannot be replaced. Unless they somehow get epoxied into place, the claim is flat out wrong. Of course actually looking at product shots since DCIG can't be bothered to test the systems, might have provided a clue:
@cincystorage @vonSpreacker @EMCXtremIO @storagereview @TestLabNut Looks easily replaceable to me: pic.twitter.com/3R9Wbg2jw5
— Bryan Krausen (@btkrausen) October 6, 2015
Not to worry, our own Kevin O'Brien started noticing many other factual errors in the report and brought them up with Ken.
@TestLabNut We only affirm what we can document. Vendors all had at least two opportunities to update or correct data. Most do. Some don't.
— Ken Clipperton (@KenDCIG) October 6, 2015
But here's the rub, a lot of vendors don't participate because of the stigma associated with awards that are largely paid for. And really, if we're being honest, DCIG not only didn't do the minimum to ensure checkboxes were placed in the right spots, they flat out didn't try. Is it really so hard to look at pictures of an array to determine how many ports it has?
@BMBeeler @KenDCIG @storagereview "4" FC ports on the back of the @XIOstorage ISE 860, may want to recount. pic.twitter.com/FbCGFA67di
— Kevin OBrien (@TestLabNut) October 6, 2015
The problem went further in some cases, where no response or an undetermined feature was categorized the same as that feature missing. The response here was the same, pushing it back on the manufacturer for not correcting DCIG. This attitude would be like saying someone is guilty/not-guilty, you just don't know.
UPDATE: It appears that DCIG charges vendors to update reports, indicating money has to change hands to update "errors" or "omissions"
@TestLabNut Correct. Which is why the legend for the X on the data sheets is "Unsupported / Undetermined".
— Ken Clipperton (@KenDCIG) October 6, 2015
The authors clearly didn't bother to peruse online documentation either, opting instead to lazily sit idle, preferring to deflect blame onto vendors instead of doing any actual digging. It's a pretty damning indictment considering this feedback from Josh Goldstein VP Marketing & Product Management for XtremIO at EMC.
Glaring mistakes? I see a bunch. Fix all the errors on XtremIO and the mistakes I found for other arrays and there’s no way we shouldn’t be at least an “Excellent” rating – and if they looked at anything besides checkbox features and speeds & feeds – like what can you actually accomplish with this product in a mixed-workload enterprise environment running typical virtualization and database applications – then they’d realize why XtremIO is #1 in the market.
Well sure, Josh is somewhat biased of course, it's his goal to move a bunch of XtremIO arrays. The main question here is does he have a point? Josh goes on to highlight some of the errors before surely succumbing to exhaustion:
- We do have replication – in fact there are several options for this including RecoverPoint, RecoverPoint for VMs, and VPLEX
- We do have application aware snapshots
- We do have async replication both periodic and continuous
- We do have sync replication (via VPLEX)
- We of course have “lossless deduplication.” If we had lossy dedupe we wouldn’t be in business.
- Metadata is in NV-RAM (all our RAM is battery backed and non-volatile)
- XtremIO does support IPMI
- XtremIO does support threshold alerts
- XtremIO does support SNMP
- XtremIO does support variable stripe sizes
- I don’t know what VAAI Total Number means, we support every single VAAI primitive
- They list a bunch of things like SIOC and Storage DRS and vRealize Operations which are features of VMware. They have nothing to do with the array. We fully work with all of these. For vRealize Operations we have written the API integration so it can retrieve, view, and analyze array statistics.
- We don’t have ALUA because we are symmetrical N-way active. ALUA is not needed.
- DRAM cache is 2TB, not 128GB
- NV-RAM and DRAM in XtremIO’s architecture are the same thing – all DRAM is battery backed
- CPU cores for 2 controllers is 40, not 32
- Scale-out max controllers is 16, not 8
- Storage networking ports max is 64, not 8 (32 iSCSI and 32 FC)
- Max 10GB ports is 32. Max 1GB ports is N/A as we don’t use 1GB for iSCSI
- FC ports is 32 x 8Gbps
- Controller addition is fully supported
- Storage shelf/node addition is fully supported
- Proactive remediation is fully supported
Depending on how you count, that's nearly 25 mistakes or omissions for one product and one vendor. Perhaps not all of these are obvious, especially if you don't bother to look during research, but in our cross-checking this morning, we easily found most of this data. As a disclaimer, neither EMC or DCIG paid us for the effort, though clearly both benefit from us updating the buyer's guide.
Yesterday I warned buyers about placing any weight on this report. Clearly it was worse than I knew, the report is complete failure and at worst could be viewed as intentionally deceptive given the obviously wrong poor marks in not just the XtremIO case, but obvious mistakes with X-IO, AMI and others. Buyers should now view vendors claiming DCIG seals of approval with extreme prejudice as we now have more visibility into how these awards come to be. Enterprise IT is full of pay to play, this is just the latest and most blatent case of how bad IT marketing can be.
Sign up for the StorageReview newsletter
