Estimating Instructional Design and Development Time
The following is an article written by Dale Munson for the International Society of Performance Improvement. It provides an excellent set of guidelines and tips for providing better estimates - something very difficult to do. From my personal experience, rarely are learning projects estimated well (or any project for that matter). Statistics have shown for decades that project management milestones are usually missed because of poor definition or lack of proper planning. Dale does a good job and deserves a big pat on the back for this.
Estimating Instructional Design and Development Time:
Debunking Industry-Standard Development Ratios
by Dale Munson
This paper describes a framework for expanding the traditional approach of estimating instructional design and development time. It extends familiar industry standard ratios to include conditions which have a high impact on the hours estimated for course development, such as the high probability of design modifi cations during the early stages of the product lifecycle.
After applying our best guess estimates to how long it should take to develop a course—regardless of the estimating method used—why do we often find out that the development times are higher than anticipated? Why is there little correlation to what we have done in the past to what we think will happen in the future? Why does the depth of topic coverage not always correlate with the development time? These are familiar and perplexing questions to many project managers. To answer these questions, let’s start by examining the estimating process.
The Traditional Approach
Generally, we begin to build the estimate of course development time by using industry standard “rules of thumb.” Then, based on such things as the number of objectives, audience, competencies, etc., we arrive at the approximate amount of time required to complete each major phase of the ADDIE Model. The result is a profile of time required to complete all tasks necessary to develop a course or series of courses. We can visualize this as a histogram, with the hours of effort required for each activity indicated by proportional bars, as shown in Figure 1.
Figure 1. Effort-Hour Profile for Instructional Development
The success of our estimates pivots on the efforthour profile. If this profile does not change over the course of a project, then we complete it within budget. However, if one task takes more hours to complete than we anticipated, the effort-hour profile changes and we may find ourselves exceeding our original estimates. For this reason, we attempt to account for all significant conditions that could affect the effort-hour profile, such as the availability of source documents, the availability of subjectmatter experts (SMEs), the experience level of the ID, the number of audiences, cooperation of the course reviewers, and amount of time dedicated to other tasks, and so on.
In attempting to identify these conditions, we sometimes use historical data compiled for similar tasks on similar projects. With sufficient historical data and a stable product, reliable formulas (ratios) can be developed for estimating effort-hours.
The time needed to design and develop a “simple,” “average,” “or “complex” course is then easier toascertain. Here, a certain course archetype can be assigned a standard definition, which includes both design and development time. This definition and associated effort-hours should remain somewhat stable through many course development projects.
Given evolving products/systems, however, this approach overlooks many conditions that are significant, such as design modifications, conflicting information, and the lack of cooperation from courseware reviewers. Since they are not easily quantifiable, these conditions have been dismissed as “extraneous” or “exceptional” considerations.
The more stable the environment, the fewer exceptions and the better the estimates. But, as our environment changes, we find the need to include a fuller range of significant conditions, as well as to reconsider previously underestimated conditions.
The New Paradigm
The lack of correlation between what was expected and reality arise in a world of new products consisting of hardware and software integrated into larger systems; and, more important, being developed concurrently with training materials.
Not only are new products and our relationship with them more complex, but we now must enter their development stage earlier, often at the inception. This means we no longer develop courseware after the fact, but from the onset of the product/system.
Using the old paradigm, where old assumptions about the methods for training, the audience, and the subject changed radically, so too has the significant conditions that determine efforthours.
This is particularly true for multimedia development. Instructional designers, illustrators, and writers are more than passive recipients of information; they actively participate in the product/system development. Today’s IDs do more than translate notes from an engineer; they are the bridge between those who develop products/solutions and those who use them. And, the ID’s audience is no longer a homogeneous group of experts, but a heterogeneous group of novices and sophisticated users. Consequently, the ID must be prepared to design and develop instructional materials at a variety of levels.
The Field Concept
To handle the complexity of these diverse factors, the traditional formula/ratio approach to building the effort-hour profile must be extended to include those significant conditions which have the greatest impact on the amount of effort required to design and develop a new course. A strategic planning, budgeting, and management tool, specifically designed for this purpose, would enable us to objectively characterize the appropriate training medium, as well as expand the field of conditions (dependencies).
The term field is actually a three-dimensional field. For the purposes of this paper, we will look at the field of significant conditions that determine the estimate of effort-hours. We can hypothesize about the magnitude and direction of these conditions and, using fuzzy algorithms with ranges and probabilities instead of ratios with single values, we can take into account conditions which have been slighted.
The principal shortcoming of the traditional approach is that the least critical factors are the most easily captured in formulas or ratios. Estimating print shop production tasks, for example, lends itself to accurate and predictable time formulas; however, instructional development is not easily quantified and hence is not iterated as precisely or confidently. We cannot easily quantify a number of significant conditions affecting instructional development, but we do know that constant changes can skew the traditional development ratios significantly. Past experience has shown that even the smallest software and hardware changes lead to significant, rippling changes in course design. With this in mind, we must change the way we view the average, industry-standard course development ratios. This will enable us to adequately identify the full range or field of significant conditions.
The Field of Significant Conditions
Three axes define the field of significant conditions. One axis defines the users—from novice to expert—and another defines the user’s needs— from acquiring knowledge to mastering skills. The user’s needs, together with the ID’s purpose in satisfying those needs define an informal “contract” between the user and the course content being developed. The content, which includes the technical complexity of the product as it changes throughout its lifecycle—constitutes the third axis. This last factor is the most crucial in expanding the traditional approach.
Once a course is placed on the grid of user and need, we must factor in the conditions imposed by the complexity of the subject itself, classified from “simple” to “complex.” In addition, we must take into account time and change, which means considering the product’s stage of development and the probability of change, along with its content. This dual attention suggests four types of products/systems (see Figure 2).
Figure 2. Product/System Variables Affecting Course Development
Currently, we are accustomed to making estimates based on assumptions about the products’ stability (the left-hand column in Figure 2), regardless of where it lies on the spectrum from simple to complex. The new paradigm shifts our attention to the likelihood of change, as the instructional development process continues (the right-hand column in Figure 2). Consequently, covering the “stable” product in great depth can require less effort than shallow coverage of the “new” product. Thus, the third axis, which together with the users and their needs, includes four basic types of product/systems. These range from the least probability of change to the greatest—stable/simple, stable/complex, new/simple, and new/complex. This is graphed in Figure 3.
Figure 3. Field of Significant Conditions
As you can see from Figure 4, the relationship of these factors is not linear. As we move upward on the axis, the curve changes rapidly.
Figure 4. Effort Hours vs. Product/System Characteristics
As the probability of change increases from the stable/simple to the new/complex, so does the probability of greater effort hours. Course development for a new/simple product or system will not necessarily take more time to develop than a stable/complex one, but the probability is greater.
These hypotheses must be tested in different contexts, but the important point is that analysis of data will yield a weighting factor, and a range of effort-hours according to the probability of change.
The product’s development stage, its probability of change over time, and its technical content together constitute the most important elements in determining the effort-hour profile. Traditionally, the effect of change over the duration of the product development process has been underestimated.
Our new paradigm demands that we accord it equal weight with product content. This is particularly important since change has such a great impact on the effort devoted to design and development, which constitutes an increasingly large percentage of effort, and hence cost as the complexity of the instructional development process continues.
As a long-time instructional developer, I have improved my skills of predicting how long a project will take, but have, nevertheless, been surprised at the final time tally. This results from not taking into account the entire field of significant conditions and their relative weights. Our previous approach assumed a stable product and environment, but these assumptions are no longer valid.
We need therefore to expand our old ways, extending the industry-standard ratios to include the increased number and complexity of significant conditions that affect our effort-hour estimates.
Dale Munson is a senior technical education specialist for Global Learning Solutions at StorageTek in Louisville, Colorado. He is an accomplished corporate presenter with more than 28 years of experience in designing instructional programs and presenting complex material in a simplified, entertaining manner. Dale is a state-licensed and government-certified instructor who has taught courses worldwide for many Fortune 500 companies, including Fairchild Weston, Loral, Hewlett-Packard, IBM, and Dell. For more than 15 years, he designed and presented courses for several top-level agencies within the Intelligence Community.
Dale holds a Bachelor of Arts degree in Information Technology and a Masters degree in Information and Learning Technologies from the University of Colorado at Denver. He is a Certified Performance Technologist and the VP of Programs for the Front Range Chapter of the International Society for Performance Improvement (ISPI).