The National Library of Medicine and Quality Information Systems, 1978
Page  1

2/24/78 em this can be put in grad four as soon as we can deliver his price me the natioilal library of medicine and quality information systems a discussion of the quality of information systems cannot be divorced from a consideration of the quality of the literature itself any information system no matter how accessible responsive and comprehensive is only as good as the information it contains in this matter professor ziman's discussion of the importance of the traditional scientific literature is much to the point and i look forward to dr relman's remarks on the journal as a quality filter in medicine and to dr garfield's discussion of the citation index as a quality filter in science and technology broadly while the generality that the system is only as good as the information it contains seems self-evident it is not necessarily true that the system cannot allow the user to apply his own quality control searching a large bibliographic file may allow the selection of higher quality material than the average for the total file the national library of medicine has attempted in the development of its services to build quality control into its input process but it has also gone to great pains to develop mechanisms so that users can also apply quality control in selecting bibliographic citations pertinent to their interests as you may know nlm developed the first nationwide online bibliographic retrieval service medlile in 1971 the demonstration service prior to medline provided access to only the top 100 journals

Page  2 2 in clinical medicine the original medline provided access to only the most important 1200 journals medline now includes 3000 journals but it is still possible to limit a search to the more important journals we have also since 1975 added author abstracts where available and also have introduced a capability text words in titles or abstracts users can if they wish limit a search to only the most recent material or to review articles thus the search system allows users to exercise control over the volume and quality of the material they retrieve in 1969 prior to the on-line service the average search retrieved 180 citations in 1978 80 of all searches were completed on-line and result in only 15 to 20 citations the remainder 20 average about 100 citations printed off-line md mailed to the requester thus quality a control has been greatly improved at the output end of the system this does not however obviate nlm's concern for quality input in recent years a number of people have examined the problem of identifying key articles books and journals in science in work supported by nlm pao 1 has developed a quality filter technique based on the bibliographies of review articles that successfully identified key information sources for certain areas of medical education boyce and primov2 demonstrated that this method is effective in selecting papers that will be highly cited in future work and thus seems well suited for scientists as well as educators they point out also that it provides the librarian who is not usually an

Page  3 3 expert in the subject with a tool for compiling a specialized bibliography in which all entries have been filtered for quality goffman and warren have shown 3 that for a given literature a minimal nucleus can be identified that may be more than the most productive journal alone they also reported that within a given body of literature the ratio of articles to authors approximates a constant and that this constant governs the subgrouping of periodicals into a minimal nucleus and succeeding zones to demonstrate this they successfully applied the technique of bradford analysis to the bibliographies of ten medical scientists they found in all cases that the average paper-to author ratio approximated the so-called bradford multiplier or the goffman-warren constant this technique may be used on a broader scale for the selection of journals for large library collections i should not allow false modesty to prevent me from adding that their work also was supported by an nlm grant the science citation index has served a most useful role as a tool to measure the relative importance of individual papers and authors although some care must be used in relating frequency of citation to quality ofascience reported citation index analysis is not only valuable as a a quality filter of literature but may also be used to identify clustering of scientific journals within disciplines 4 and for studying the interdis ciplinary relationships of scientific journals 5 importantly goffman and morris have show that bradford's law applies to a portion of a literature as well as to its totality 6 these may be useful indices for selecting and acquiring literature needed to serve specialized user groups

Page  4 4 i must confess at this point that as ingenious and valuable as these several techniques may be we at the national library of medicine have not yet used them for operational decisions this is not because we are indifferent to these techniques but because we have developed other methods that are adequate for our responsi bilities as a national institution our statutory function requires that we must be omnivorous in acquiring biomedical literature even as a repository however our appetite must be restrained by the realities of money manpower and space for the core subjects of medicine we attempt to have a comprehensive collection acquiring all scientific and scholarly literature no matter where published or in what language nor do we attempt to judge the quality of this literature before acquisition there are other areas however outside of the core subjects of biomedicine in which we attempt to exercise considerable judgment in acquiring the literature we maintain a research collection on certain related topics such as medical sociology and demography for basic subjects such as general chemistry and mathematics we have built reference collections as a guide to building these collections the library follows a scope and coverage manual developed by a consensus of experts this defines the subjects of some pertinence to medicine scope and the extent to which material depending on its degree of pertinence should be collected coverage because they constitute the essence of how we define the collection these policies are periodically

Page  5 5 reviewed by the jational library of medicine's board of regents selecting quality journals in volume serials constitute by far the largest portion of the literature entering the nlm collection over 23,000 titles are currently received of this number index medicus our primary bibliographic publication cites the articles from some 2500 titles without a process for quality control to winnow those 23000 periodicals index medicus might be 9,000 pages per month instead of 1,200 the annual cumulated index medicus would be enormous and the on-line computerized reference retrieval system medline would be completely unmanageable in addition an organization like nlm that operates an indexing service must be cost conscious the cost elements include among others journal acquisition indexing keyboarding printing and other publication costs for the computerized system there are additional costs related to computer storage but in selecting journals the foregoing cost elements are of less concern than the costs to the person who is using index medicus or medline each citation that is presented to a user requires him to decide and to act flrst he decides whether the title suggests that the article is potentially useful to him if so he may expend effort to obtain the article from his library and to scan it additional effort may be required to obtain the article through interlibrary loan and if in a foreign language it may require translation an indexing service does the user a disservice whenever it results in

Page  6 6 his being presented with citations that ultimately prove useless or the docma are unavailable to him primarily for this reason the national library of medicine constantly tries to keep the pool of most useful literature in index medicus from becoming diluted with literature that is likely to be of little use even so we index a quarter of a million articles each year in index medicus and medline the library endeavors to include citations to articles published in those periodicals that are judged to be of greatest potential use to the international biomedical community because our users are an extremely varied group the literature coverage is broad it includes both research and didactic literature relating not only to the science and practice of medicine and public health but also to bioengineering legal medicine bioethics medical economics and other related fields in deciding which journals should be included in index medjcus the library has guidance from a group of nine outside experts these consultants include distinguished health scientists educators editors and librarians as needed we also solicit the suggestions and recommendations of many other subject specialists the group of consultants however reviews all judgments and recommendations to assure a uniform and balanced perspective the consultants review both newly acquired serials and serials that nlm has been asked to reconsider by editors publishers or by others over a period of several years they also re-evaluate the journals that are already being indexed in index medicus to see whether

Page  7 7 their quality continues to merit inclusion there are a great many elements that go into each consultant's rating important among these are evidence of adequate peer review of the manuscripts conformity with reasonable standards of language and style and regularity or continuity of publication because we cover literature of many subjects from many lands in many languages no single rigid set of standards is equally applicable to all our consultants therefore are free to use their individual judgment in determining how much weight should be given in arriving at a rating for a particular journal a final statistical ranking is made of all titles renewed it must be emphasized that there are no formal criteria which if satisfied automatically qualify a journal for indexing each journal competes against every other journal that is presently in dexed or that is seeking to be covered by index medicus the journals that receive most favorable ratings tend to be those that report well-documented original observations and critical reviews these are considered most vital to the needs of health workers around the world less likely to be indexed are journals largely devoted to parochial interests and journals having many articles restating accepted facts or repeating clinical experience in another setting we feel confident that the first thousand journals contain the articles most frequently cited by others the next thousand contain articles less a frequently cited the remaining 500 titles are those most difficult to evaluate since they contain articles rarely cited nevertheless they are requested often through interlibrary loan it is for this reason that we do not depend on citation frequency counts on selection the frequency

Page  8 8 of borrowing may be an equally important indicator of journal value in this context 300 serial titles satisfy 70 of the interlibrary loans requested selecting quality another area in which nlm is vitally concerned about quality is that of medical audiovisuals in recent years audiovisuals have become important in the ducation of physicians dentists and other health professionals they are used extensively at all levels—for undergraduate graduate and continuing professional education the national medical audiovisual center a component of the national library of medicine has been working closely in recent years with the association of american medical colleges to identify and evaluate audiovisual learning packages other professional organizations such as the american association of dental schools and the american nurses association have also helped evaluate these materials it soon became evident that there were great variations in the nature and scope of audiovisual materials acquired on the basis of content and educational purpose the audiovisual materials were divided into two categories one group represented audiovisual documentation of scheduled educational eventsly important indicator of journal value in this context 300 serial titles satisfy 70 of the interlibrary loans requested selecting quality another area in which nlm is vitally concerned about quality is that of medical audiovisuals in recent years audiovisuals have become important in the ducation of physicians dentists and other health professionals they are used extensively at all levels—for undergraduate graduate and continuing professional education the national medical audiovisual center a component of the national library of medicine has been working closely in recent years with the association of american medical colleges to identify and evaluate audiovisual learning packages other professional organizations such as the american association of dental schools and the american nurses association have also helped evaluate these materials it soon became evident that there were great variations in the nature and scope of audiovisual materials acquired on the basis of content and educational purpose the audiovisual materials were divided into two categories one group represented audiovisual documentation of scheduled educational events such as lectly important indicator of journal value in this context 300 serial titles satisfy 70 of the interlibrary loans requested selecting quality another area in which nlm is vitally concerned about quality is that of medical audiovisuals in recent years audiovisuals have become important in the ducation of physicians dentists and other health professionals they are used extensively at all levels—for undergraduate graduate and continuing professional education the national medical audiovisual center a component of the national library of medicine has been working closely in recent years with the association of american medical colleges to identify and evaluate audiovisual learning packages other professional organizations such as the american association of dental schools and the american nurses association have also helped evaluate these materials it soon became evident that there were great variations in the nature and scope of audiovisual materials acquired on the basis of content and educational purpose the audiovisual materials were divided into two categories one group represented audiovisual documentation of scheduled educational events such as lectures grand rounds seminars society meetings etc these materials are entered into our system after a technical screening and cataloging the second category of materials consists of audiovisual pro ductions that have a specific educational purpose these are

Page  9 8a insert p8 five years ago mr davis mccarn of my staff became interested in the use of citation analysis to aid in the selection of journals for index medicus nml with the institute for scientific information to analyze citation frequencies of journals contained in index medicus using a file developed by isi journal citation reports this file represented a consolidation of citation information from the last quarter of 1969 thus it was somewhat dated even at the time the study was done but consolidation of citation information was such a major undertaking it was done infrequently by isi at the time of the study index medicus contained 2140 journals of which 729 were also covered by science citation index there were 908 journals in im not covered by sci but cited by the 729 jm journals there were 503 journals in index medicus not cited at all in the last quarter of 1969 by the 729 journals common to im and sci of these 29 were new leaving 475 journals uncited which might reasonably have been cited this is quite consistent with my prior remarks on the number of journals at the low end of the spectrum the study and its results were reviewed by our journal selection experts and they concluded that it was not very useful however a review of the 475 uncited journals indicates that in 1978 221 or 46 of them are no longer indexed for index medicus thus it appears that there may be a reasonable agreement between the results of the expert selection process of nu7 and citation analysis for the ident ification of marginal journals at the other end of the spectrum there were very few highly cited journals not already covered by index medicus the vast majority

Page  10 of these were in chemistry and physics areas nlm has intentionally left to other services nlm will continue to study the use of citation analysis and library landing statistics as a source of data to improve the journal selection process

Page  11 9 frequently accompanied by instructional guides and self-evaluation materials wlth the help of the aamc and other professional groups these materials receive a thorough evaluation by teams of experts in scientific content instructional design and production techniques in scientific content the review panels look at appropriateness for the intended audience validity of data and information and logical organization in instructional design the reviewers are concerned with clear goals and objectives in an audiovisual production whether there is a good way to test attainment of these goals and the appropriateness of the av methods selected production techniques are assessed by audiovisual experts who rate the visual and sound quality of the 6000 audiovisual productions that have received this scrutiny so far about two-thirds haved by instructional guides and self-evaluation materials wlth the help of the aamc and other professional groups these materials receive a thorough evaluation by teams of experts in scientific content instructional design and production techniques in scientific content the review panels look at appropriateness for the intended audience validity of data and information and logical organization in instructional design the reviewers are concerned with clear goals and objectives in an audiovisual production whether there is a good way to test attainment of these goals and the appropriateness of the av methods selected production techniques are assessed by audiovisual experts who rate the visual and sound quality of the 6000 audiovisual productions that have received this scrutiny so far about two-thirds have been rated as being of acceptable quality a new on-line data base of citations to these reviewed audiovisual productions was created it is known as avline avline represents an unprecedented effort by the national library of medicine to create an information system based on rigorous quality control of material entered into the system avline is available for on-line searching at over 800 institutions in the united states that are members of nlms nationwide on-line network it is used by health science professionals faculty and students to retrieve citations and abstracts that aid in selecting and evaluating audiovisual instructional materials

Page  12 10 the file now containing over 5000 items may be searched in a variety of ways including indexing terms such as fellcal subject heading title year of production and type of audiovisual medium a typical citation includes the name of the author title indication of the intended audience an abstract or summary of the content professional appraisal rating whether the material is for sale or rent and procurement source we expect that the avline data base will eventually include about 10000 items medlars evaluation we have watched with great interest as usage of our on-line bibliographic system continues to grow medline and the other data bases have found increasing acceptance as tools of health research education and practice last year almost one million on-line searches were performed on nlm's data bases at institutions in the u s several hundred thousand were performed by institutions overseas while we are heartened by the acceptance of our information services we are challenged by the problems engendered by this strong demand the quality of our on-line systems receives constant review in terms of content as well as accessibility and responsiveness performance analysis has been a concern of the library since the beginning of its computer-based operations in 1964 the medlars system was not yet two years old when we planned a major evaluation of its bibliographic searching capability at that time of course searching was performed by centralized processing of written requests in a batch mode

Page  13 11 the principal objectives of the study were to determine the requirements of our users to obtain an objective measure of system performance in relation to these requirements to disclose factors adversely affecting performance and to determine ways of satisfying user needs more efficiently or more economically or both 7 the study which took two years to complete was of considerable interest to the information science community at large since it represented the first large-scale evaluation of a major operating computerized information system the evaluation considered six factors that affected the quality of the medlars search service coverage in terms of the proportion of the useful literature in the system on a particular topic recall or its ability to retrieve relevant references precision or ability to suppress non-relevant references from a search response time format of the search results and the amount of effort the user must expend to evoke a satisfactory response from the system the study revealed the need for improving communication between the requester and the system modifying the depth to which some journal articles are indexed and expanding the medical subject headings vocabulary the specific recommendations and the philosophy they embodied were of inestimable help to the library in planning medlars ii and the eventual development of medline and other on-line data bases relating this evaluation to our on-line system as it exists today it is clear that the findings were indeed pertinent communication between the user and the system is now almost instantaneous depth of

Page  14 12 13 indexing or the number of headings assigned to each article has been increased four-fold the third general area for improvement identified by the evaluation was vocabulary development slnce 1966 our subject headings have doubled from 7000 to 14000 and we have greatly increased the number of subjeadings that can be affixed to main headings for greater specificity we now have 75 subheadings available thus several hundred thousand concerts can be cordinated are linned with this vecr experimental data base for technology transfer i would like to turn now to an experimental project designed to assist in the translation of the results of medical research into a form usable by the health care practitioner this project is being carried out by the lister hill national center for blomedical communications the research and development component of nlm libraries by tradition collect what others have published evaluating the quality of journals and audiovisuals as i described earlier does no violence to this precept for the materials are entered into the system as the authors produce them or they are not put in at all

Page  15 13 what we are doing in th is new project however is to take an existing body of medical literature in this case the literature on e viral hepatitis 8 organize it and make it available for on-line minicomputer storage and retrieval i wish to emphasize that we are not talking here about bibliographic references or abstracts but access to reviewed and evaluated data and information synthesized by a a consensus of experts as a comprehensive bank of information this system plans to 1 contain substantive answers to questions posed by practitioners 2 provide answers that are current 3 be immediately and reliably responsive to inquiries and 4 provide data supporting the answers as well as citations to primary publications for more detailed study if desired the contents of the hepatitis data base are presently undergoing validation and updating by consensus of a specially selected group of eleven hepatologists a computer conferencing network linking the geographically dispersed experts with one another and with the staff of the nlm is serving as the principal medium of communication now and during successive monthly updates of the data base once this initial updating is complete the data base will be made available on an experimental basis to practitioners in primary secondary and tertiary health care settings access may be direct via computer terminal or through a trained intermediary using a toll-free dial access telephone number users may also receive computer-generated printed material in hard copy or on microfiche

Page  16 14 either in response to specific queries or as a complete authoritative document on current knowledge pertaining to the disease the infor mation bank will also be made available to professional societies and to other producers of health related information products and services ranging from scholarly monographs to multimedia instructional packages the concept of quality in an information system is admittedly open to varying definitions and interpretations in an organization with information programs as diverse as those of the national library of medicine quality measures range from the journal selection committee's evaluation of how well the articles in a particular journal are peer reviewed to how well flesh tones are reproduced in an audio visual to computer response time in seconds to an on-line query and now to the use of expert consensus for answering questions this places a heavy responsibility on nlm management to be aware of objective techniques of measuring the quality of its products and services the so-called test of the market-place may no longer be adequate because we have learned that in an affluent society people are willing to pay for poor or even harmful products thus we are most grateful that the rocketfeller foundation has invited us to participate in this important conference

Page  17 references 1 pao m l a quality filtering system for medical literature j med educ 50 353-359 1975 2 boyce b and primov k pao's selection method for quality papers and the subsequent use of medical literature j hep educ 52 1001-1002 1977 3 goffman w warren k s dispersion of papers among journals based on a mathematical analysis of two diverse literatures nature 221 1205-1207 1959 4 carpenter m p and narin f clustering of scientific journals jasis 24 425-436 1973 5 narin f and carpenter m and berlt n c interrelationships of scientific journals jasis 23 323-331 1972 6 goffman w and morris t g bradford's law applied to the maintenance of library collections in saraceric tefko introduction to information science new york r r bowker co 1970 p 202 7 lancaster f.w evaluating the performance of a large computerized information system jama vol 107:114-120 january 6 1969 8 the contents of chapter 18 of viral hepatitis by james w mosley m d and john galambos m d pages 500-593 of schiff's diseases of the liver and the contents of the national academy of science's 1975 symposium on viral hepatitis published in the american journal of medical sciences