A very strange thing happened at the beginning of September, 2007. Microsoft issued a press release following the close of balloting on OOXML that began
Today the International Organization for Standardization (ISO) released the results of the preliminary ballot to participating National Body members for the ISO/IEC DIS 29500 (Ecma 376 Office Open XML file formats) ratification process. The results show that 51 ISO members, representing 74 percent of all qualified votes, stated their support for ratification of Open XML. Along with their votes, the National Bodies also provided invaluable technical comments designed to improve the specification. Many of the remaining ISO members stated that they will support Open XML after their comments are addressed during the final phase of the process, which is expected to close in March 2008.
This was talking about a very successful occurrence, evidently, and there was reason for great joy around the world.
In their recent press release about the OOXML Ballot Resolution Meeting that is taking place in Geneva this week, the ISO spelled out what had happened a bit differently:
Approximately 3 500 comments were received during last year’s ballot. By grouping and by eliminating redundancies, these have been edited by SC 34 experts down to 1 100 comments for processing during the five days of the BRM.
The five-month ballot process which ended on 2 September was open to the IEC and ISO national member bodies from 104 countries, including 41 that are participating members of the joint ISO/IEC technical committee, JTC 1, Information technology.
Approval requires at least 2/3 (i.e. 66.66 %) of the votes cast by national bodies participating in ISO/IEC JTC 1 to be positive; and no more than 1/4 (i.e. 25 %) of the total number of national body votes cast to be negative. Neither of these criteria were achieved in the DIS vote, with 53 % of votes cast by national bodies participating in ISO/IEC JTC 1 being positive and 26 % of national votes cast being negative.
What? They had to achieve passing marks on two different criteria, yet both failed?
What should we expect in terms of statements by ECMA and Microsoft about the result of the BRM? Your guess is as good as mine, and it might be fun to write what you expect and compare it to what is published, if a press release is done.
I think it might be useful to have some real quantitative metrics to use to measure the success of the BRM. In particular, these might be useful to think about if your country sent a delegation to Geneva. Some of these use the same data, but in different ways.
To be clear, I fully believe that the convener and the national standards body delegations gave this their best efforts to try to make the BRM a success. I was not in the meeting, but these metrics are useful especially for those who did not attend.
In the following, I will use the phrase “fully discussed with consensus reached” to mean that when a comment suggested a problem with the OOXML specification, a discussion of all proposed resolutions took place in a full and balanced manner, and then consensus developed around whether a particular resolution should be adopted to modify the specification or leave it as-is.
- How many total comments were there and how many were fully discussed with consensus reached at the meeting?
- What percentage of the total comments were fully discussed with consensus reached this week? Was it closest to 10%, 20%, 30%, 40%, 50%,60%, 70%, 80%, 90%, or 100%?
- Conversely, what percentage of the total comments were voted on without a full discussion and consensus reached? Was it closest to 10%, 20%, 30%, 40%, 50%,60%, 70%, 80%, 90%, or 100%?
- Individual or closely related comments from countries were considered in country alphabetical order, cycling through as many times as necessary. How many full cycles were there?
- How many times during the week did your country get an opportunity to begin a full discussion on a new comment?
- What was the percentage of all your comments that were fully discussed with consensus reached during the week?
- Of the comments that were fully discussed with consensus reached, how many of these were simple editorial corrections? What percentage of all the comments that were fully discussed with consensus reached were these simple editorial corrections?
- How many issues were taken offline for discussion?
- How many of the offline issues were subsequently brought back before the BRM for a full discussion and subsequently reached consensus?
- In Euros and in your local currency, what is the total cost for your delegation to come and work in Geneva for the week?
- In Euros and in your local currency, what is the total cost divided by the number of opportunities you had to begin full discussion on a new comment (that is, cost per national body comment discussion opportunity)?
- How much time was spent discussing procedural issues and what percentage of the total BRM time was this?
- How much time was spent discussing how to vote on unaddressed comments and what percentage of the total BRM time was this?
- How many Microsoft Office-specific features remain in the OOXML specification?
- After this experience, how many comments on a proposed JTC1 Fast Track standard would you say is too many to fully discuss and reach consensus on in a five day BRM?
- How long would the BRM have had to have been to fully discuss and reach consensus on all comments?
If you have attended other BRMs, how would you compare the OOXML BRM metrics with similar metrics for the other BRMs?
I’ll leave it to you to interpret the importance of any of these metrics after we get some numbers to plug into them. Presumably we will get a lot of the numbers to use because this is supposed to be an open and transparent process. At that point you can judge the relative success or failure of the BRM in its mission to review and reach consensus on resolutions to the comments offered by the national standards bodies.
What metrics would you use?
Incidentally, regarding the apparent discrepancy in interpretation of the September ballot numbers mentioned at the beginning of this entry, see the eWeek Microsoft Watch article by Joe Wilcox called “Microsoft FUD Watch: OOXML Edition.” It’s worth a read.