72 APPROACHES TO REUSABILITY $4.2 benefit from the experience of their elders and peers.How can the general idea contribute to reuse?Design patterns should not encourage a throwback to the "all that counts is design reuse"attitude mentioned earlier.A pattern that is only a book pattern,however elegant and general,is a pedagogical tool,not a reuse tool;after all,computing science students have for three decades been learning from their textbooks about relational query optim ization,Gouraud shading,AVL trees,Hoare's Quicksort and Dijkstra's shortest path algorithm without anyone claiming that these techniques were breakthroughs in reusability.In a sense,the patterns developed in the past few years are only incremental additions to the software professional's bag of standard tricks.In this view the new contribution is the patterns themselves,not the idea of pattern. As most people who have looked carefully at the pattern work have recognized,such See“Programs a view is too limited.There seems to be in the very notion of pattern a truly new with holes'”,page contribution,even if it has not been fully understood yet.To go beyond their mere 506. pedagogical value,patterns must go further.A successful pattern cannot just be a book description:it must be a software component,or a set of components.This goal may seem remote at first because many of the patterns are so general and abstract as to seem impossible to capture in actual software modules;but here the object-oriented method provides a radical contribution.Unlike earlier approaches,it will enable us to build reusable modules that still have replaceable,not completely frozen elements:modules that serve as general schemes (patterns is indeed the appropriate word)and can be adapted to various specific situations.This is the notion of behavior class (a more picturesque term is programs with holes);it is based on O-O techniques that we will study in later chapters, in particular the notion of deferred class.Combine this with the idea of groups of components intended to work together-often known as frameworks or more simply as libraries-and you get a remarkable way of reconciling reusability with adaptability. These techniques hold,for the pattern movement,the promise of exerting,beyond the new-bag-of-important-tricks effect,an in-depth influence on reusability practices. Reusability through the source code Personnel,design and specification forms of reuse,useful as they may be,ignore a key goal of reusability.If we are to come up with the software equivalent of the reusable parts of older engineering disciplines,what we need to reuse is the actual stuff of which our products are made:executable software.None of the targets ofreuse seen so far-people, designs,specifications-can qualify as the off-the-shelf components ready to be included in a new software product under development. If what we need to reuse is software,in what form should we reuse it?The most natural answer is to use the software in its original form:source text.This approach has worked very well in some cases.Much of the Unix culture,for example,originally spread in universities and laboratories thanks to the on-line availability of the source code, enabling users to study,imitate and extend the system.This is also true of the Lisp world. See also“Formats The economic and psychological impediments to source code dissemination limit for reusable compo- the effect that this form ofreuse can have in more traditional industrial environments.But ent distribution"”, a more serious limitation comes from two technical obstacles: page 79 below
72 APPROACHES TO REUSABILITY §4.2 benefit from the experience of their elders and peers. How can the general idea contribute to reuse? Design patterns should not encourage a throwback to the “all that counts is design reuse” attitude mentioned earlier. A pattern that is only a book pattern, however elegant and general, is a pedagogical tool, not a reuse tool; after all, computing science students have for three decades been learning from their textbooks about relational query optimization, Gouraud shading, AVL trees, Hoare’s Quicksort and Dijkstra’s shortest path algorithm without anyone claiming that these techniques were breakthroughs in reusability. In a sense, the patterns developed in the past few years are only incremental additions to the software professional’s bag of standard tricks. In this view the new contribution is the patterns themselves, not the idea of pattern. As most people who have looked carefully at the pattern work have recognized, such a view is too limited. There seems to be in the very notion of pattern a truly new contribution, even if it has not been fully understood yet. To go beyond their mere pedagogical value, patterns must go further. A successful pattern cannot just be a book description: it must be a software component, or a set of components. This goal may seem remote at first because many of the patterns are so general and abstract as to seem impossible to capture in actual software modules; but here the object-oriented method provides a radical contribution. Unlike earlier approaches, it will enable us to build reusable modules that still have replaceable, not completely frozen elements: modules that serve as general schemes (patterns is indeed the appropriate word) and can be adapted to various specific situations. This is the notion of behavior class (a more picturesque term is programs with holes); it is based on O-O techniques that we will study in later chapters, in particular the notion of deferred class. Combine this with the idea of groups of components intended to work together — often known as frameworks or more simply as libraries — and you get a remarkable way of reconciling reusability with adaptability. These techniques hold, for the pattern movement, the promise of exerting, beyond the new-bag-of-important-tricks effect, an in-depth influence on reusability practices. Reusability through the source code Personnel, design and specification forms of reuse, useful as they may be, ignore a key goal of reusability. If we are to come up with the software equivalent of the reusable parts of older engineering disciplines, what we need to reuse is the actual stuff of which our products are made: executable software. None of the targets of reuse seen so far — people, designs, specifications — can qualify as the off-the-shelf components ready to be included in a new software product under development. If what we need to reuse is software, in what form should we reuse it? The most natural answer is to use the software in its original form: source text. This approach has worked very well in some cases. Much of the Unix culture, for example, originally spread in universities and laboratories thanks to the on-line availability of the source code, enabling users to study, imitate and extend the system. This is also true of the Lisp world. The economic and psychological impediments to source code dissemination limit the effect that this form of reuse can have in more traditional industrial environments. But a more serious limitation comes from two technical obstacles: See “Programs with holes”, page 506. See also “Formats for reusable component distribution”, page 79 below
$4.2 WHAT SHOULD WE REUSE? 73 Identifying reusable software with reusable source removes information hiding.Yet no large-scale reuse is possible without a systematic effort to protect reusers from having to know the myriad details of reused elements. Developers of software distributed in source form may be tempted to violate modularity rules.Some parts may depend on others in a non-obvious way,violating the careful limitations which the discussion of modularity in the previous chapter imposed on inter-module communication.This often makes it difficult to reuse some elements of a complex system without having to reuse everything else. A satisfactory form of reuse must remove these obstacles by supporting abstraction and providing a finer grain of reuse. Reuse of abstracted modules All the preceding approaches,although of limited applicability,highlight important aspects of the reusability problem: Personnel reusability is necessary if not sufficient.The best reusable components are useless without well-trained developers,who have acquired sufficient experience to recognize a situation in which existing components may provide help. Design reusability emphasizes the need for reusable components to be of sufficiently high conceptual level and generality-not just ready-made solutions to special problems.The classes which we will encounter in object technology may be viewed as design modules as well as implementation modules. Source code reusability serves as a reminder that software is in the end defined by program texts.A successful reusability policy must produce reusable program elements. The discussion of source code reusability also helps narrow down our search for the proper units of reuse.A basic reusable component should be a software element.(From there we can of course go to collections of software elements.)That element should be a module of reasonable size,satisfying the modularity requirements ofthe previous chapter; in particular,its relations to other software,if any,should be severely limited to facilitate independent reuse.The information describing the module's capabilities,and serving as primary documentation for reusers or prospective reusers,should be abstract:rather than describing all the details of the module (as with source code),it should,in accordance with the principle of Information Hiding,highlight the properties relevant to clients. The term abstracted module will serve as a name for such units of reuse,consisting of directly usable software,available to the outside world through a description which contains only a subset of each unit's properties. The rest of part B of this book is devoted to devising the precise form of such abstracted modules;part C will then explore their properties. More on distribu- The emphasis on abstraction,and the rejection of source code as the vehicle for reuse,do tion formats below. not necessarily prohibit distributing modules in source form.The contradiction is only apparent:what is at stake in the present discussion is not how we will deliver modules to their reusers,but what they will use as the primary source of information about them.It may be acceptable for a module to be distributed in source form but reused on the basis of an abstract interface description
§4.2 WHAT SHOULD WE REUSE? 73 • Identifying reusable software with reusable source removes information hiding. Yet no large-scale reuse is possible without a systematic effort to protect reusers from having to know the myriad details of reused elements. • Developers of software distributed in source form may be tempted to violate modularity rules. Some parts may depend on others in a non-obvious way, violating the careful limitations which the discussion of modularity in the previous chapter imposed on inter-module communication. This often makes it difficult to reuse some elements of a complex system without having to reuse everything else. A satisfactory form of reuse must remove these obstacles by supporting abstraction and providing a finer grain of reuse. Reuse of abstracted modules All the preceding approaches, although of limited applicability, highlight important aspects of the reusability problem: • Personnel reusability is necessary if not sufficient. The best reusable components are useless without well-trained developers, who have acquired sufficient experience to recognize a situation in which existing components may provide help. • Design reusability emphasizes the need for reusable components to be of sufficiently high conceptual level and generality — not just ready-made solutions to special problems. The classes which we will encounter in object technology may be viewed as design modules as well as implementation modules. • Source code reusability serves as a reminder that software is in the end defined by program texts. A successful reusability policy must produce reusable program elements. The discussion of source code reusability also helps narrow down our search for the proper units of reuse. A basic reusable component should be a software element. (From there we can of course go to collections of software elements.) That element should be a module of reasonable size, satisfying the modularity requirements of the previous chapter; in particular, its relations to other software, if any, should be severely limited to facilitate independent reuse. The information describing the module’s capabilities, and serving as primary documentation for reusers or prospective reusers, should be abstract: rather than describing all the details of the module (as with source code), it should, in accordance with the principle of Information Hiding, highlight the properties relevant to clients. The term abstracted module will serve as a name for such units of reuse, consisting of directly usable software, available to the outside world through a description which contains only a subset of each unit’s properties. The rest of part B of this book is devoted to devising the precise form of such abstracted modules; part C will then explore their properties. The emphasis on abstraction, and the rejection of source code as the vehicle for reuse, do not necessarily prohibit distributing modules in source form. The contradiction is only apparent: what is at stake in the present discussion is not how we will deliver modules to their reusers, but what they will use as the primary source of information about them. It may be acceptable for a module to be distributed in source form but reused on the basis of an abstract interface description. More on distribution formats below
74 APPROACHES TO REUSABILITY $4.3 4.3 REPETITION IN SOFTWARE DEVELOPMENT To progress in our search for the ideal abstracted module,we should take a closer look at the nature of software construction,to understand what in software is most subject to reuse. Anyone who observes software development cannot but be impressed by its repetitive nature.Over and again,programmers weave a number ofbasic patterns:sorting, searching,reading,writing,comparing,traversing,allocating,synchronizing... Experienced developers know this feeling of deja vu,so characteristic of their trade. A good way to assess this situation (assuming you develop software,or direct people who do)is to answer the following question: How many times over the past six months did you,or people working for you, write some program fragment for table searching? Table searching is defined here as the problem of finding out whether a certain element x appears in a table t of similar elements.The problem has many variants,depending on the element types,the data structure representation for the choice of searching algorithm. Chances are you or your colleagues will indeed have tackled this problem one or more times.But what is truly remarkable is that-if you are like others in the profession -the program fragment handling the search operation will have been written at the lowest reasonable level of abstraction:by writing code in some programming language, rather than calling existing routines. To an observer from outside our field,however,table searching would seem an See bibliographic obvious target for widely available reusable components.It is one of the most researched references on areas of computing science,the subject of hundreds of articles,and many books starting page 99. with volume 3 of Knuth's famous treatise.The undergraduate curriculum ofall computing science departments covers the most important algorithms and data structures.Certainly not a mysterious topic.In addition: It is hardly possible,as noted,to write a useful software system which does not include one or (usually)several cases of table searching.The investment needed to produce reusable modules is not hard to justify. As will be seen in more detail below,most searching algorithms follow a common pattern,providing what would seem to be an ideal basis for a reusable solution. 4.4 NON-TECHNICAL OBSTACLES Why then is reuse not more common? Most of the serious impediments to reuse are technical;removing them will be the subject of the following sections of this chapter(and of much of the rest of this book).But of course there are also some organizational,economical and political obstacles
74 APPROACHES TO REUSABILITY §4.3 4.3 REPETITION IN SOFTWARE DEVELOPMENT To progress in our search for the ideal abstracted module, we should take a closer look at the nature of software construction, to understand what in software is most subject to reuse. Anyone who observes software development cannot but be impressed by its repetitive nature. Over and again, programmers weave a number of basic patterns: sorting, searching, reading, writing, comparing, traversing, allocating, synchronizing… Experienced developers know this feeling of déjà vu, so characteristic of their trade. A good way to assess this situation (assuming you develop software, or direct people who do) is to answer the following question: Table searching is defined here as the problem of finding out whether a certain element x appears in a table t of similar elements. The problem has many variants, depending on the element types, the data structure representation for t, the choice of searching algorithm. Chances are you or your colleagues will indeed have tackled this problem one or more times. But what is truly remarkable is that — if you are like others in the profession — the program fragment handling the search operation will have been written at the lowest reasonable level of abstraction: by writing code in some programming language, rather than calling existing routines. To an observer from outside our field, however, table searching would seem an obvious target for widely available reusable components. It is one of the most researched areas of computing science, the subject of hundreds of articles, and many books starting with volume 3 of Knuth’s famous treatise. The undergraduate curriculum of all computing science departments covers the most important algorithms and data structures. Certainly not a mysterious topic. In addition: • It is hardly possible, as noted, to write a useful software system which does not include one or (usually) several cases of table searching. The investment needed to produce reusable modules is not hard to justify. • As will be seen in more detail below, most searching algorithms follow a common pattern, providing what would seem to be an ideal basis for a reusable solution. 4.4 NON-TECHNICAL OBSTACLES Why then is reuse not more common? Most of the serious impediments to reuse are technical; removing them will be the subject of the following sections of this chapter (and of much of the rest of this book). But of course there are also some organizational, economical and political obstacles. How many times over the past six months did you, or people working for you, write some program fragment for table searching? See bibliographic references on page 99
$4.4 NON-TECHNICAL OBSTACLES 75 The NIH syndrome An often quoted psychological obstacle to reuse is the famous Not Invented Here("NIH") syndrome.Software developers,it is said,are individualists,who prefer to redo everything by themselves rather than rely on someone else's work. This contention (commonly heard in managerial circles)is not borne out by experience.Software developers do not like useless work more than anyone else.When a good,well-publicized and easily accessible reusable solution is available,it gets reused. Consider the typical case of lexical and syntactic analysis.Using parser generators such as the Lex-Yacc combination,it is much easier to produce a parser for a command language or a simple programming language than ifyou must program it from scratch.The result is clear:where such tools are available,competent software developers routinely reuse them.Writing your own tailor-made parser still makes sense in some cases,since the tools mentioned have their limitations.But the developers'reaction is usually to go by default to one of these tools;it is when you want to use a solution not based on the reusable mechanisms that you have to argue for it.This may in fact cause a new syndrome,the reverse of NIH,which we may call HIN(Habit Inhibiting Novelty):a useful but limited reusable solution,so entrenched that it narrows the developers'outlook and stifles innovation,becomes counter-productive.Try to convince some Unix developers to use a parser generator other than Yacc,and you may encounter HIN first-hand. Something which may externally look like NIH does exist,but often it is simply the developers'understandably cautious reaction to new and unknown components.They may fear that bugs or other problems will be more difficult to correct than with a solution over which they have full control.Often such fears are justified by unfortunate earlier attempts at reusing components,especially if they followed from a management mandate to reuse at all costs,not accompanied by proper quality checks.If the new components are of good quality and provide a real service,fears will soon disappear. What this means for the producer of reusable components is that quality is even more important here than for more ordinary forms of software.If the cost of a non-reusable,one- of-a-kind solution is N,the cost R of a solution relying on reusable components is never zero:there is a learning cost,at least the first time;developers may have to bend their software to accommodate the components;and they must write some interfacing software, however small,to call them.So even if the reusability savings r=- and other benefits of reuse are potentially great,you must also convince the candidate reusers that the reusable solution's quality is good enough to justify relinquishing control. See[M1995] This explains why it is a mistake to target a company's reusability policy to the potential reusers (the consumers,that is to say the application developers).Instead you should put the heat on the producers,including people in charge of acquiring external components. to ensure the quality and usefulness of their offering.Preaching reuse to application
§4.4 NON-TECHNICAL OBSTACLES 75 The NIH syndrome An often quoted psychological obstacle to reuse is the famous Not Invented Here (“NIH”) syndrome. Software developers, it is said, are individualists, who prefer to redo everything by themselves rather than rely on someone else’s work. This contention (commonly heard in managerial circles) is not borne out by experience. Software developers do not like useless work more than anyone else. When a good, well-publicized and easily accessible reusable solution is available, it gets reused. Consider the typical case of lexical and syntactic analysis. Using parser generators such as the Lex-Yacc combination, it is much easier to produce a parser for a command language or a simple programming language than if you must program it from scratch. The result is clear: where such tools are available, competent software developers routinely reuse them. Writing your own tailor-made parser still makes sense in some cases, since the tools mentioned have their limitations. But the developers’ reaction is usually to go by default to one of these tools; it is when you want to use a solution not based on the reusable mechanisms that you have to argue for it. This may in fact cause a new syndrome, the reverse of NIH, which we may call HIN (Habit Inhibiting Novelty): a useful but limited reusable solution, so entrenched that it narrows the developers’ outlook and stifles innovation, becomes counter-productive. Try to convince some Unix developers to use a parser generator other than Yacc, and you may encounter HIN first-hand. Something which may externally look like NIH does exist, but often it is simply the developers’ understandably cautious reaction to new and unknown components. They may fear that bugs or other problems will be more difficult to correct than with a solution over which they have full control. Often such fears are justified by unfortunate earlier attempts at reusing components, especially if they followed from a management mandate to reuse at all costs, not accompanied by proper quality checks. If the new components are of good quality and provide a real service, fears will soon disappear. What this means for the producer of reusable components is that quality is even more important here than for more ordinary forms of software. If the cost of a non-reusable, oneof-a-kind solution is N, the cost R of a solution relying on reusable components is never zero: there is a learning cost, at least the first time; developers may have to bend their software to accommodate the components; and they must write some interfacing software, however small, to call them. So even if the reusability savings and other benefits of reuse are potentially great, you must also convince the candidate reusers that the reusable solution’s quality is good enough to justify relinquishing control. This explains why it is a mistake to target a company’s reusability policy to the potential reusers (the consumers, that is to say the application developers). Instead you should put the heat on the producers, including people in charge of acquiring external components, to ensure the quality and usefulness of their offering. Preaching reuse to application r R N = --- See [M 1995]
76 APPROACHES TO REUSABILITY $4.4 developers,as some companies do by way of reusability policy,is futile:because application developers are ultimately judged by how effectively they produce their applications,they should and will reuse not because youtell them to but because you have done a good enough job with the reusable components(developed or acquired)that it will be profitable for their applications to rely on these components. The economics of procurement A potential obstacle to reuse comes from the procurement policy of many large “GENERALIZA. corporations and government organizations,which tends to impede reusability efforts by T1ON”28.5,page 928 focusing on short-term costs.US regulations,for example,make it hard for a govemment agency to pay a contractor for work that was not explicitly commissioned(normally as part of a Request For Proposals).Such rules come from a legitimate concern to protect taxpayers or shareholders,but can also discourage software builders from applying the crucial effort of generalization to transform good software into reusable components. On closer examination this obstacle does not look so insurmountable.As the concern for reusability spreads,there is nothing to prevent the commissioning agency from including in the RFP itself the requirement that the solution must be general-purpose and reusable,and the description of how candidate solutions will be evaluated against these criteria.Then the software developers can devote the proper attention to the generalization task and be paid for it. Software companies and their strategies Even if customers play their part in removing obstacles to reuse,a potential problem remains on the side of the contractors themselves.For a software company,there is a constant temptation to provide solutions that are purposely not reusable,for fear of not getting the next job from the customer-because if the result of the current job is too widely applicable the customer may not need a next job! I once heard a remarkably candid expose of this view after giving a talk on reuse and object technology.A high-level executive from a major software house came to tell me that,although intellectually he admired the ideas,he would never implement them in his own company,because that would be killing the goose that laid the golden egg:more than 90%of the company's business derived from renting manpower-providing analysts and programmers on assignment to customers-and the management's objective was to bring the figure to 100%.With such an outlook on software engineering,one is not likely to greet with enthusiasm the prospect of widely available libraries of reusable components. The comment was notable for its frankness,but it triggered the obvious retort:if it is at all possible to build reusable components to replace some of the expensive services of a software house's consultants,sooner or later someone will build them.At that time a company that has refused to take this route,and is left with nothing to sell but its consultants'services,may feel sorry for having kept its head buried in the sand
76 APPROACHES TO REUSABILITY §4.4 developers, as some companies do by way of reusability policy, is futile: because application developers are ultimately judged by how effectively they produce their applications, they should and will reuse not because you tell them to but because you have done a good enough job with the reusable components (developed or acquired) that it will be profitable for their applications to rely on these components. The economics of procurement A potential obstacle to reuse comes from the procurement policy of many large corporations and government organizations, which tends to impede reusability efforts by focusing on short-term costs. US regulations, for example, make it hard for a government agency to pay a contractor for work that was not explicitly commissioned (normally as part of a Request For Proposals). Such rules come from a legitimate concern to protect taxpayers or shareholders, but can also discourage software builders from applying the crucial effort of generalization to transform good software into reusable components. On closer examination this obstacle does not look so insurmountable. As the concern for reusability spreads, there is nothing to prevent the commissioning agency from including in the RFP itself the requirement that the solution must be general-purpose and reusable, and the description of how candidate solutions will be evaluated against these criteria. Then the software developers can devote the proper attention to the generalization task and be paid for it. Software companies and their strategies Even if customers play their part in removing obstacles to reuse, a potential problem remains on the side of the contractors themselves. For a software company, there is a constant temptation to provide solutions that are purposely not reusable, for fear of not getting the next job from the customer — because if the result of the current job is too widely applicable the customer may not need a next job! I once heard a remarkably candid exposé of this view after giving a talk on reuse and object technology. A high-level executive from a major software house came to tell me that, although intellectually he admired the ideas, he would never implement them in his own company, because that would be killing the goose that laid the golden egg: more than 90% of the company’s business derived from renting manpower — providing analysts and programmers on assignment to customers — and the management’s objective was to bring the figure to 100%. With such an outlook on software engineering, one is not likely to greet with enthusiasm the prospect of widely available libraries of reusable components. The comment was notable for its frankness, but it triggered the obvious retort: if it is at all possible to build reusable components to replace some of the expensive services of a software house’s consultants, sooner or later someone will build them. At that time a company that has refused to take this route, and is left with nothing to sell but its consultants’ services, may feel sorry for having kept its head buried in the sand. “GENERALIZATION”, 28.5, page 928