Technical Documentation – Monitoring Throughput and Quality

Technical writing has a definite deliverable, and a measurable throughput (number of documents, or number of pages, per unit time). By taking in raw materials and turning out finished goods, technical writing is no different to any other manufacturing process, and so ought to be just as amenable to the monitoring of its throughput and quality. Measuring the former is relatively straightforward, but measuring the latter is less so. However, even if it is not yet easy to put an absolute value on document quality, it really ought to be possible to put a relative value it, or perhaps on that of a suite of documentation, and to be able to say whether that value is getting better or worse (compared to previous years, for example).

Monitoring Document Throughput

The traditional measure for the throughput of a technical writer (technical author) is the number of document pages signed off per working day. There are some who would argue that a better measure is a count of the number of documents signed off, and there is much to be said for this argument, though it is not pursued further, here.

Another important measure, usually forgotten, is the standard deviation. If an employer is judging between two writers, working on identical documents (if this were possible), the one with the higher mean throughput could well be the one giving the better value for money; but if their throughputs are nearly equal, the one giving a less variable, more stable throughput might be the one to prefer.

Of course, it is never as simple as this, but this is the only point from which such decision-making can start. Once the throughput has been monitored for a suitably long period, over a suitably large number of documents, it is possible to start making the same sort of statistics-based comparisons that are applicable to any industrial process.

Firstly, the actual, measured cost of production of previous documents is the main indication that we have for predicting the likely costs of future documents.

Secondly, the behaviour on recently completed documents can yield interesting lessons as to how to adapt the process in the future. The measured parameters for each of the documents in the month just passed, for example, can be compared against those for all of the previously finished documents. Any documents whose parameters differ from those of the overall population by more than two or three standard deviations can be highlighted for special attention (to analyse what went wrong, or what went abnormally well).

Thirdly, as noted earlier, it is also possible to make comparisons between technical writers who work on the same population of documents, to determine whether any of them displays abnormally high or low throughput, from which lessons can be learned for passing back to the rest of the team. Similarly, comparisons can be made between any of the other actors in the documentation process (document owners, reviewers, etc).

Monitoring Document Quality

The central question to ask when judging a document's quality is that of whether the document does the job for which it was required. This is where we encounter the first difficulty in the monitoring task. Most documents, perhaps wrongly, serve multiple functions, many of them not clearly defined. A document, published on the Internet or distributed by email, is in the hands of its target audience. But what the target audience expects to obtain as a result of reading the document is not necessarily consistent with what the author of the document had intended. Confounding further the process of balancing advantages against disadvantages, the drafting process of the document might already have given many subsidiary benefits, such as serving as a tool for helping the engineers to refine the design of the product itself.

Direct Measures

As far as the document owner (the person commissioning the document) is concerned, the purpose of the document is to increase product sales, or to increase sales margins, or to decrease overheads, or some combination of these. It is unlikely, though, that the contribution of the document towards achieving any of these will be easy to determine. The one possible exception is in the area of reducing overheads, as measured by the reduced number of customer support calls that need to be answered (as measured by the total number of minutes of someone's salary spent on them). This can be compared directly with the cost (in man-days of salary or consultancy) incurred during the writing of the document.

Indirect Measures

Since the direct measures are so elusive, there is no option but to consider using indirect measures. But, even here, there is no straightforward solution.

The main problem is that most of the traditional measures end up measuring more than one thing, and give an ambiguous message. A prime example of this is the measure of how frequently the document has been updated since it was first written. A high rate of modification might be an indication of a poorly written document that has had to be repeatedly corrected, or an indication of an extremely well written document that has been kept constantly supplied with up-to-the-minute information, or else an indication of a document describing a product that has itself had to undergo repeated modifications (giving a similarly ambiguous measure of the quality of that product design).

Another ambiguous measure is that of the number of review cycles that a document undergoes before being signed off, with a very similar set of mixed meanings.

One possibility, therefore, is to collect a large number of these measures, and to look for ways of combining them in such a way that the unwanted meanings (the noise) tend to cancel, while the wanted meanings (the signal) tend to be build together. This will probably require a genetic algorithm (GA), or an artificial neural net (ANN), to be trained on a set of test data, and to be allowed to derive, for itself, the appropriate ways for combining the measures.

Meanwhile, in compiling a list of what measures that could be collected, the list will, because of human nature, inevitably be biased towards including a large number of measures that are easy to collect, and a lesser number of measures that are more difficult. Of these, some of the measures will be better indicators than others, and need to be weighted accordingly by the GA or ANN (to the extent that some of them might even be given a zero weighting).

The measurements can be classified under two main headings: objective and subjective. The former are to be preferred, but the latter cannot be avoided since judging the quality of a document is ultimately a subjective matter.

Objective Measures

The following list contains a number of questions that can be asked of each document as it comes round for creation or revision, each question requiring a numerical reply.

• How many reader complaints/questions about the given document have been received (such as from customer support calls)?
• How often has the document been accessed for reading (on the Internet, for example)? This probably says little about the quality, but it does give some indication of the importance of the document, and sets the number of reader complaints in context.
• How frequently has the given document been revised during its life?
  • With what variance over time?
  • With what variance compared to other documents of the same type?
• How close on the previous revision is the current one?
• How many review cycles did the document go through before being signed off?
• How many days did the revision/creation take, from beginning to end?
• What proportion of that time was spent actively working on the document?

A very similar list of questions could be compiled, for the product that the document describes. Although the responses would not say anything about the quality of the document, they could be used to cancel out some of the ambiguous messages in the above list.

Subjective Measures

One important point to note is that it is the "measure" that is subjective, not the "data" (which is still handled objectively in this analysis).

• What proportion of the changes for each draft was due to error (typing errors, misunderstandings on the part of the technical writer, miscommunications on the part of the document owner)?
• What proportion of the changes for each draft was due to the document being used as a design tool (the product being developed in parallel with the document, with the document helping to highlight improvements that could be made to the product)?
• What was the gravity of each reader complaint/question?
• After getting people (inside and outside the company) to fill in a questionnaire, on a scale of 1 to 5 (but noting that all sorts of sample bias need to be taken into account):
  • How do the document readers rate the quality of the document?
  • How do the document owners rate the document's ability to do its job?