Consumer Reports® and ConsumerReports.org® are published by Consumers Union, an expert, independent nonprofit organization whose mission is to work for a fair, just, and safe marketplace for all consumers and to empower consumers to protect themselves. To achieve this mission, we test, inform, and protect. To maintain our independence and impartiality, CU accepts no outside advertising, no free test samples, and has no agenda other than the interests of consumers. CU supports itself through the sale of our information products and services, individual contributions, and a few noncommercial grants. Consumers Union is governed by a board of 18 directors, who are elected by CU members and meet three times a year. CU’s President, James Guest, oversees a staff of more than 450.
If you are thinking about buying a refrigerator, a home theater system, or even a car, you get a Consumer Reports and read their ratings. There you will get an idea about safety, energy efficiency, quality, and cost. There are other similar services. Even the US government has a website that will give you the latest information about automobile fuel efficiencies.
If you’ve ever bought anything from Amazon, you’ve probably seen product ratings by people who have presumably bought them, along with comments. This is particularly useful if you are comparing, say, two books on the same topic or two CDs from a musical artist whose work you have never before purchased.
Let’s say you are considering buying a house and you are comparing two of them. What is your criteria? Assuming they are in the same neighborhood and school district, you look at the architecture, the quality of construction, and the maintainability of the buildings and landscaping. There are other factors, of course. You might not be especially quantitative in your analysis, but you would probably have a home inspector look at the buildings to tell you if there were any problems, especially if you are not an expert. The inspection reports that I have seen are check lists that systematically rank the quality and current state of the house from top to bottom.
We have nothing like this for standards.
What would it possibly mean for something to be a “one star standard” versus something that is a “five star standard”?
We have folks who do standards compliance testing for a business, but this is not evaluating the quality of the standards themselves. I will note that we do have the Web Services Interoperability Organization which looks at existing standards and best practices in using them, and then recommends both profiles for deploying the standards well and future changes to the standards that will improve them.
We have thousands of standards and no clear way to decide which of them are good and which are not. Instead, we more of less go by the organizations that create the standards, whether we are actually required to implement them (say, by law or customer requirements), or if the market leaders use them.
I’m going to tackle the issue of quality and standards organizations in a future entry, but let me say that
- Standards organizations are not all equal in quality, though it doesn’t seem that everyone knows that.
- A given standards organization can produce two standards of wildly divergent quality.
- In my opinion, the key measurement of a standards organization is not the quantity of standards produced but the quality of standards produced.
As a disclaimer, I’m very aware that when IBM is involved in the creation of standard, we probably want people to use that. The same goes for everybody else.
In some cases there may be only one standard for a particular purpose. Do we just accept that or can we apply some set of metrics to it to help the maintainers evolve it into something better?
Let me propose some possible criteria for measuring standards quality. Please comment on these or suggest your own, though see my rules about comments at the close. Note that in any given criterion it may be reasonable to consider a numeric value between 0.0 and 1.0 for ranking how a particular standard does. That is, do not assume answers can only be “yes” or “no.” (You don’t get to abstain …)
- Openness criteria:
- Was the standard developed by an independent community of experts in a way that did not advantage any one software provider’s products or projects?
- Will the standard be actively maintained by an independent community of experts in a way that will not advantage any one software provider’s products or projects?
- Is the standard document freely available to everyone?
- Is the standard itself freely implementable by everyone?
- Can a subset of the standard be used or can the standard be used as a component of another standard?
- Is the standard designed to maximize use of pre-existing high quality standards?
- Is the standard well factored to allow modular implementation and maximum re-use of common components? For example, we have only one way for representing an address or bold text. That is, does the standard have the right granularity?
- Are modern best practices for design used, or are legacy formats employed even though better, equivalent, and more advanced alternatives are available?
- Is the standard designed to maximize its use by reference or inclusion in other high quality standards?
- Does the standard have a well designed extension architecture?
- Does the standard adequately represent the semantics of the information encoded within it?
- Are the methods for describing metadata complete and themselves based on high quality standards?
- Can information represented via the standard be processed efficiently by modern tools?
- Does the standard encode product-specific attributes that others are expected to implement?
- Does the standard have gaps or ambiguities that will lead to divergent interpretations and incompatible implementations?
- Does the standard make reference to required or likely use of other standards or proprietary specifications that may not be generally available or considered to be high quality?
I’m sure there are others, please add your suggestions and I’ll do a future entry that pulls them all together.
One criterion which I did not mention explicitly is “Is the standard elegant?”. I mean this in the way mathematicians use the term about theorems and other results: well designed, minimal, clever, and absolutely obvious once you see it. That is, just right.
Here are the rules about the comments:
- Keep them civil.
- Mention no standards by name. I do not want this to devolve into a “this is why that standard by those people is terrible” discussion.
- Talk about qualities and best practices that we should apply to standards today so that they will be useful into the future.
- Think about and discuss metrics and other aspects of quantitative analysis that can apply to the criteria.
I’m less interested in preserving the status quo of standards creation than setting up a framework for understanding the quality of what we have and improving the quality of what we have yet to build.