A peer-reviewed electronic record. Copyright is retained by the initially or sole author, who have grants right of 1st publication for the Practical Analysis, Research , Evaluation. Permission is awarded to distribute this article for not for profit, educational purposes if it is copied in the entirety plus the journal is definitely credited.
Quantity 12, Number 10, Aug 2007 ISSN 1531-7714 The Delphi Technique: Making Impression Of Consensus Chia-Chien Hsu, The Ohio State College or university , Brian A. Sandford, Oklahoma State University The Delphi strategy is a trusted and accepted method for gathering data by respondents in their domain of experience.
The technique is designed as a group, be it natural or processed communication process which should achieve a convergence of opinion on a particular real-world issue. The Delphi process has become used in various fields of study such as program planning, needs examination, policy willpower, and resource utilization to develop a full range of alternatives, check out or expose underlying assumptions, as well as correlate judgments on a topic comprising a wide range of disciplines. The Delphi technique is suitable as a way of consensus-building simply using a series of forms delivered employing multiple iterations to collect data from a panel of selected subjects.
Subject collection, time frames to get conducting and completing a report, the possibility of low response costs, and unintentionally guiding responses from the respondent group are areas which needs to be considered when making and employing a Delphi study. The Delphi technique, mainly manufactured by Dalkey and Helmer (1963) at the Rand Corporation in the year 1950s, is a traditionally used and recognized method for reaching convergence of opinion relating to real-world know-how solicited by experts inside certain subject areas.
Predicated on the explanation that, “two heads are better than one, or, n brain are better than one (Dalkey, 72, p. 15), the Delphi technique is designed as a group conversation process that aims at executing detailed exams and discussions of a particular issue when it comes to goal setting, policy investigation, or perhaps predicting the occurrence of future incidents (Ulschak, 1983, Turoff , Hiltz, 1996, Ludwig, 1997). Common online surveys try to determine “what is, whereas the Delphi technique attempts to address “what could/should be (Miller, 2006).
In the books, Delphi continues to be applied in a variety of fields including program organizing, needs analysis, policy perseverance, and source utilization. Delbecq, Van para Ven, and Gustafson (1975) specifically show that the Delphi technique can be used for achieving the following targets: 1 . To determine or create a range of feasible program alternatives, 2 . To learn or expose underlying presumptions or data leading to different judgments, 3. To seek out details which may generate a opinion on the part of the respondent group, 4.
To correlate knowledgeable judgments on a topic comprising a wide range of procedures, and, your five. To educate the respondent group as to the varied and interrelated aspects of the subject (p. 11). CHARACTERISTICS WITH THE DELPHI STRATEGY The Delphi technique is suitable as a means and method for consensus-building by using a number of questionnaires to get data from a panel of selected subjects (Dalkey , Helmer, 1963, Dalkey, 1969, Linstone , Turoff, 1975, Lindeman, 1981, Martino, 1983, Small , Jamieson, 2001).
Delphi, in contrast to various other data gathering and examination techniques, employs multiple iterations designed to Functional Assessment, Research , Evaluation, Vol 12, No 12 Hsu , Sandford, Delphi Technique produce a consensus of opinion concerning a specific theme. Ludwig (1994) indicates: Iterations refer to the feedback method. The process was viewed as several rounds, in each rounded every player worked through a questionnaire which was returned for the researcher who have collected, modified, and went back to every participator a statement of the position in the whole group and the participant’s own placement.
A summation of feedback made every single participant aware about the range of opinions plus the reasons actual those thoughts (p. 55). More specifically, the feedback procedure allows and encourages the chosen Delphi participants to reflect on their first judgments regarding the information supplied in earlier iterations. Thus, in a Delphi study, the results of previous iterations regarding particular statements and/or items can transform or be modified by simply individual panel members in later iterations based on their ability to review and assess the comments and feedback provided by the other Delphi panelists.
Other notable characteristics inherent with using the Delphi approach are the capability to provide anonymity to participants, a manipulated feedback method, and the suitability of a selection of statistical evaluation techniques to interpret the data (Dalkey, 1972, Ludlow, 1975, Douglas, 1983). These types of characteristics are created to offset the shortcomings of conventional method of pooling opinions obtained from a bunch interaction (i. e., impacts of dominant individuals, noises, and group pressure pertaining to conformity) (Dalkey, 1972).
One of many characteristics and advantages of the Delphi method is subject anonymity which can reduce the effects of dominant persons which often is a concern whenever using group-based processes used to accumulate and synthesize information (Dalkey, 1972). In addition , the issue of privacy is caused by geographic dispersion in the subjects plus the use of electric communication including e-mail to solicit and exchange information.
As such, certain downsides linked to group mechanics such as treatment or coercion to adjust or adopt a certain viewpoint can be minimized (Helmer , Rescher, 1959, Oh, mid 1970s, Adams, 2001). Controlled opinions in the Delphi process is designed to reduce the effect of noise. Based upon Dalkey (1972), noise is that communication which usually occurs in a group process which the two distorts the info and handles group and/or individual interests rather than concentrating on problem solving.
Because of this, the information created from this sort of communication generally consists of prejudice not related to the uses of the analyze. Basically, the controlled feedback process consists of a well organized overview of the before iteration deliberately distributed for the subjects that allows each participator an opportunity to make additional ideas and more carefully clarify 2 the information manufactured by previous iterations.
Through the operation of multiple iterations, topics are expected to become more problem-solving oriented, to offer their opinions more insightfully, and to lessen the effects of noise. Finally, to be able to use record analysis techniques is a practice which further reduces the potential for group pressure for conformity (Dalkey, 1972). More specifically, statistical analysis can easily ensure that opinions generated simply by each subject matter of a Delphi study are well represented inside the final iteration because, “at the end from the exercise there may still be a significant pass on in person opinions Dalkey, 1972, p. 21). That is, each subject would have simply no pressure, both real or perceived, to conform to one more participant’s reactions that may result from obedience to social norms, customs, organizational culture, or standing within a profession. The various tools of record analysis allow for an objective and impartial research and summarization of the accumulated data. THE DELPHI METHOD Theoretically, the Delphi procedure can be continuously iterated right up until consensus is determined to have been achieved.
However , Cyphert and Gant (1971), Brooks (1979), Ludwig (1994, 1997), and Custer, Scarcella, and Stewart (1999) speak about that three iterations tend to be sufficient to gather the required information and also to reach a consensus generally. The following debate, however , delivers guidelines for as much as four iterations in order to assist those who choose the Delphi process as a data collection technique launched determined that additional iterations beyond 3 are required or useful. Round you: In the initially round, the Delphi method traditionally starts with an open-ended customer survey.
The open-ended questionnaire is the foundation of soliciting specific information regarding a content material area in the Delphi subject matter (Custer, Scarcella, , Stewart, 1999). Following receiving subjects’ responses, detectives need to convert the accumulated information right into a well-structured questionnaire. This customer survey is used as the study instrument for the second rounded of data collection. It should be noted that it must be both a suitable and one common modification with the Delphi procedure format to utilize a structured customer survey in Rounded 1 that is based upon an extensive review of the literature.
Kerlinger (1973) mentioned that the make use of a customized Delphi procedure is appropriate if perhaps basic information concerning the concentrate on issue exists and functional. Round a couple of: In the second round, each Delphi participant receives another questionnaire which is asked to examine the items summarized by the researchers based on the data provided inside the first rounded. Accordingly, Delphi panelists can be required to level or “rank-order Practical Examination, Research , Evaluation, Vol 12, Simply no 10 Hsu , Sandford, Delphi Approach items to set up preliminary focus among things.
As a result of rounded two, regions of disagreement and agreement will be identified (Ludwig, 1994, g. 54-55). In some instances, Delphi panelists are asked to state the explanation concerning rating priorities among items (Jacobs, 1996). Through this round, opinion begins forming and the genuine outcomes could be presented among the list of participants’ responses (Jacobs, 1996). 3 Rescher (1959), Klee (1972), and Oh (1974) concur that choosing individuals who are simply educated concerning the concentrate on issue is usually not enough nor suggested.
Considering the necessity of selecting one of the most qualified individuals, Delbecq, Vehicle de Ven, and Gustafson (1975) especially state that 3 groups of people are well competent to be themes of a Delphi study. The authors advise: “(1) the top management decision makers that will utilize the outcomes of the Delphi study, (2) the specialist staff members along with their support team, and (3) the respondents for the Delphi set of questions whose judgments are staying sought (p. 85). Delphi subjects needs to be highly trained and competent in the specialized part of knowledge associated with the target issue.
Investigators need to closely analyze and seriously consider the skills of Delphi subjects. Also (1974) indicates that deciding on appropriate themes is generally based upon the common sense and discretion of the primary investigators. Jones and Twiss (1978) suggest that the principal researchers of a Delphi study ought to identify and choose the most appropriate persons through a candidate selection process. Ludwig (1994) as well states that, “solicitation of nominations of well-known and revered individuals through the members inside the target sets of experts was recommended (p. 2). Generally, the pool of selecting possible Delphi subjects will probably use positional leaders (Kaplan, 1971, Ludwig, 1994), to follow along with a review of writers of journals in the literary works (Meyer, 1992, Miller, 2001), and/or to create contacts with those who have firsthand relationships having a particular issue (Jones, 75, Anderson , Schneider, 1993). The latter quite simply consists of individuals who are primary stakeholders with various interests related to the prospective issue or perhaps research efforts.
Concerning the appropriate number of subject matter to entail in a Delphi study, Delbecq, Van para Ven, and Gustafson (1975) recommend that experts should utilize minimally adequate number of themes and should strive to verify the results through follow-up explorations. Ludwig (1994) notes that the number of professionals used in a Delphi study is “generally determined by the number required to amount to a representative pooling of judgments and the information processing capability of the research team (p. 52). However , what constitutes an optimal number of subjects in a Delphi research never gets to a opinion in the materials.
Delbecq, Van de Ven, and Gustafson (1975) claim that ten to fifteen subjects could be enough if the backdrop of the Delphi subjects is usually homogeneous. As opposed, if different reference groups are involved in a Delphi study, more themes are expected to Round three or more: In the third round, every single Delphi panelist receives a questionnaire that includes the items and ratings described by the researchers in the previous rounded and are asked to modify his/her judgments or “to specify the reasons for leftover outside the consensus (Pfeiffer, late 1960s, p. 52). This round gives Delphi panelists an opportunity to make even more clarifications of both the details and their judgments of the comparative importance of the products. However , when compared to previous circular, only a small increase in the degree of consensus can be expected (Weaver, year 1971, Dalkey , Rourke, 1972, Anglin, 1991, Jacobs, 1996). Round 4: In the 4th and often final round, the list of leftover items, their particular ratings, fraction opinions, and items attaining consensus will be distributed for the panelists.
This kind of round provides a final opportunity for participants to revise their very own judgments. It must be remembered the number of Delphi iterations will depend on largely around the degree of consensus sought by investigators and will vary from three to five (Delbecq, Van de Ven, Gustafson, 1975, Ludwig, 1994). Subject Assortment Regarding the selection of subjects for a Delphi analyze, choosing the ideal subjects is the most important step in the complete process since it directly pertains to the quality of the results produced (Judd, 72, Taylor , Judd, 1989, Jacobs, 1996).
Since the Delphi technique is targeted on eliciting experienced opinions on the short period of the time, the selection of Delphi subjects is usually dependent upon the disciplinary regions of expertise needed by the specific issue. Relating to any collection standards of selecting Delphi subjects, there exists, in fact , not any exact qualifying criterion currently listed in the books concerning the choice of Delphi members. That is, “throughout the Delphi literature, the meaning of [Delphi subjects] has remained ambiguous (Kaplan, 1971, s. 24).
Regarding the criteria accustomed to guide the number of Delphi themes, individuals are regarded as eligible to be invited to participate in a Delphi examine if they have somewhat related backgrounds and experiences concerning the target issue, are capable of surrounding helpful inputs, and are willing to revise their initial or previous decision for the purpose of getting or attaining consensus (Pill, 1971, Oh yea, 1974). Helmer and Practical Assessment, Exploration , Evaluation, Vol 12, No 15 Hsu , Sandford, Delphi Technique become needed.
Witkin and Altschuld (1995) note that the estimated size of a Delphi panel is generally underneath 50, although more have been completely employed. Ludwig (1997) files that, “the majority of Delphi studies possess used between 15 and 20 respondents (p. 2). In sum, the size of Delphi subjects is usually variable (Delbecq, Van de Ven, , Gustafson, 1975). If the sample size of a Delphi research is too small , and these subjects may not be regarded as having offered a representative pooling of decision regarding the concentrate on issue.
In case the sample dimensions are too large, the drawbacks natural within the Delphi technique such as potentially low response prices and the requirement of large obstructs of time by the respondents and the researcher(s) could be the result. Period Requirements Executing a Delphi study can be time-consuming. Especially, when the tool of a Delphi study includes a large number of transactions, subjects should dedicate significant blocks of the time to total the questionnaires.
Delbecq, Van de Ven, and Gustafson (1975), Ulschak (1983), and Ludwig, (1994) recommend that minimal 45 times for the administration of a Delphi analyze is necessary. With regard to the time managing between iterations, Delbecq ain al. (1975) note that supplying two weeks to get Delphi subjects to respond to each round is inspired. Ludwig (1994) indicates, “a drawback to Delphi was that the questionnaire approach may slow the process significantly as a lot of days or perhaps weeks may pass between rounds” (p. 54).
Specifically, since expanding the instrument, collecting the data, and giving the customer survey are interconnected between iterations, ensuring Delphi subjects react to the detectives on time does in many ways possibly promote or perhaps prohibit the capability of the detectives in studying the data, making a new tool based upon the prior responses, and distributing future questionnaires in a timely fashion. These are challenging aspects of doing a Delphi study and do require proper planning and management. The use and prevalence of electronic digital technologies (i. e. email, teleconferencing, and so forth ) may possibly facilitate those who find themselves interested in using the Delphi approach. Witkin and Altschuld (1995) note that electronic digital technology provides an opportunity for visitors to more easily utilize the Delphi process by using advantages of, “(1) the safe-keeping, processing, and speed of transmission capabilities of computers, (2) the upkeep of respondent anonymity, and, (3) the potential for rapid feedback (p. 204). Data Evaluation Regarding info analysis, decision rules has to be established to put together and coordinate the decision and observations provided by Delphi subjects.
Yet , the kind and type 4 of requirements to use to both specify and determine consensus in a Delphi examine is be subject to interpretation. Basically, consensus over a topic could be decided if a certain percentage of the votes comes within a approved range (Miller, 2006). One particular criterion recommends that general opinion is attained by having 80 percent of subjects’ votes show up within two categories on a seven-point range (Ulschak, 1983). Green (1982) suggests that for least 70 percent of Delphi subjects have to rate 3 or higher over a four point Likert-type scale and the typical has to be for 3. a few or higher. Hartgummischeibe, Skutsch, and Schofer (1975) reveal the use of percentage measures can be inadequate. That they suggest that an even more reliable alternate is to gauge the stability of subjects’ responses in successive iterations. Inside the Delphi method, data evaluation can entail both qualitative and quantitative data. Detectives need to handle qualitative info if vintage Delphi studies, which use open-ended questions to get subjects’ views, are carried out in the preliminary iteration.
Future iterations should be identify and hopefully achieve the desired degree of consensus and also any improvements of judgments among panelists. The major statistics used in Delphi studies will be measures of central tendency (means, median, and mode) and level of dispersion (standard deviation and inter-quartile range) in order to present information concerning the collective decision of participants (Hasson, Keeney, , McKenna, 2000). Generally, the uses of typical and setting are favored. However , sometimes, as described by Murray and Jarman (1987), the mean is also workable.
Witkin (1984) queries the appropriateness of using the mean to measure the subjects’ responses in the event scales used in Delphi studies are not delineated at the same intervals. In the literature, the application of median credit score, based on Likert-type scale, is definitely strongly popular (Hill , Fowles, 75, Eckman, 1983, Jacobs, 1996). As Jacobs (1996) says, “considering the anticipated opinion of view and the skewed expectation of responses because they were created, the median would innately appear ideal to indicate the resultant convergence of opinion (p. 57).
The usage of mode is additionally suitable the moment reporting data in the Delphi process. Ludwig (1994) specifically addressed that “the Delphi process tends to create affluence, and though this was usually into a single level, there was the potential of polarization or clustering with the results around two or more details. In these instances, the imply or median could be misleading (p. 57). CONSIDERING DELPHI SHORTCOMINGS AND WEAKNESSES Potential of Low Response Costs Due to the multiple feedback techniques inherent and integral to the concept and use of the Delphi process, the
Sensible Assessment, Exploration , Evaluation, Vol 12, No 15 Hsu , Sandford, Delphi Technique potential exists for low response rates and striving to maintain robust reviews can be a concern. “In the Delphi technique, [poor response rate] is magnified fourfold because a maximum of four online surveys may be delivered to the same panelists (Witkin , Altschuld, 95, p. 196). If a particular portion of those men discontinue their responses during various phases of the Delphi process, the quality of information attained could be reduced or at least seriously scrutinized.
As a result, Ludwig (1994) specifically details subject motivation as the real key to the powerful implementation of your Delphi analyze and researchers need to enjoy an active position in this area to aid ensure while high a reply rate as it can be. Consumption of Large Blocks of your energy The Delphi technique may also be time-consuming and laborious. As opposed to other data collection techniques such as the cell phone survey as well as the face-to-face operations, which can be simultaneously conducted by a group of people and is completed in a period of time if the sample size is small , the Delphi technique is terative and sequential. As a result, the necessity of taking huge block of the time to consecutively, sequentially complete a Delphi process is definitely inescapable. Ludwig (1994) implies that, “a drawback to Delphi was that the questionnaire approach may slower the process tremendously as many days or weeks might pass between rounds (p. 54). Optimally speaking, the iteration characteristics of the Delphi process give you the opportunities to get investigators and subjects to improve the accuracy and reliability of the outcomes.
In contrast, the same characteristic likewise increases the work load of researchers and the amount of time needed to successfully complete the data collection procedure (Cunliffe, 2002). Potential of Molding Thoughts The version characteristics from the Delphi approach can potentially enable investigators to mold opinions (Altschuld, 2003). An test, conducted by Scheibe, Skutsch, and Schofer (1975), indicated that Delphi subjects will rate their particular responses in a different way after getting a distorted responses.
Dalkey and Helmer (1963) also noted that, “some ‘leading’ by experimenters inevitably resulted through the selection of the info supplied (p. 467). Furthermore, Cyphert and Gant (1971) illustrated that a statement within their study was rated substandard. However , Delphi subjects scored the statement above average following receiving fake feedback. Therefore , Cyphert and Gant (1971) concluded that the Delphi technique could, “be used to mold opinion along with collect [data] (p. 273).
Indeed, “subtle pressure to conform with group ratings was one of the major drawbacks inside the Delphi strategy (Witkin , Altschuld, 95, p. 188). Delphi detectives need to be aware, exercise extreme care, and put into practice the proper shields in dealing with this issue. 5 Potential of Determining General Statements vs . Certain Topic Related Information An assumption regarding Delphi members is that they will be equivalent in knowledge and experience (Altschuld , Jones, 1991). However , this assumption might not be validated.
More specifically, the help of Delphi panelists could be erratically distributed, especially in the field of high technology (Marchant, 1988, Altschuld , Jones, 1991). “Some panelists may possibly have much more in-depth understanding of certain topics, whereas various other panelists are definitely more knowledgeable about diverse topics (Altschuld , Thomas, 1991, g. 187). Consequently , subjects who have less specific knowledge of selected topics are unable to specify the most crucial statements which were identified by simply those subject matter who own in-depth know-how concerning the goal issue.
The outcomes of a Delphi study could be the results of identifying several general transactions rather than a great in-depth exposition of the topic (Altschuld , Thomas, 1991). SUMMARY The Delphi technique provides these involved or interested in engaging in research, analysis, fact-finding, concern exploration, or perhaps discovering what is actually well-known or not known about a specific topic a flexible and adjustable tool to collect and examine the needed data. Subject matter selection plus the time frames for conducting and completing a Delphi analyze are two areas which should be considered properly prior to starting the study.
The additional precautions relating to low response rates, accidentally guiding opinions, and surveying panelists of their limited understanding of the topic rather than soliciting their very own expert decision should also always be built into the style and setup of the analyze. The Delphi technique has and definitely will continue to be an essential data collection methodology which has a wide variety of applications and uses for people who need to gather details from individuals who are immersed and imbedded inside the topic appealing and can give real-time and real-world understanding. REFERENCES Adams, S. M. (2001).
Projecting the next decade in safety management: A Delphi technique analyze. Professional Protection, 46 (10), 26-29. Altschuld, J. W. (2003). Delphi technique. Spiel, Applied analysis design. The Ohio Condition University. Altschuld, J. T., , Jones, P. M. (1991). Concerns in the using a revised scree test out for Delphi survey data. Evaluation Assessment, 15 (2), 179-188. Functional Assessment, Analysis , Evaluation, Vol 12, No 12 Hsu , Sandford, Delphi Technique Anderson, D. They would., , Schneider, I. At the. (1993). Using the Delphi process to identify significant recreation research-based innovations.
Log of Recreation area and Excitement Administration, 10 (1), 25-36. Anglin, G. L. (1991). Instructional technology past, present and future. Englewood, COMPANY: Libraries Unrestricted Inc. Creeks, K. Watts. (1979). Delphi technique: Increasing applications. North Central Affiliation Quarterly, 54 (3), 377-385. Cunliffe, S i9000. (2002). Forecasting risks inside the tourism market using the Delphi technique. Tourism, 50 (1), 31-41. Custer, R. D., Scarcella, T. A., , Stewart, W. R. (1999). The customized Delphi technique: A rotational modification. Diary of Vocational and Technological Education, 15 (2), 1-10.
Cyphert, N. R., , Gant, W. L. (1971). The Delphi technique: An instance study. Phi Delta Kappan, 52, 272-273. Dalkey, N. C. (1969). An trial and error study of group thoughts and opinions. Futures, 1 (5), 408-426. Dalkey, D. C. (1972). The Delphi method: A great experimental study of group opinion. In N. C. Dalkey, D. L. Rourke, R. Lewis, , M. Snyder (Eds. ). Studies in the quality of life: Delphi and decision-making (pp. 13-54). Lexington, MA: Lexington Books. Dalkey, N. C., , Helmer, O. (1963). An fresh application of the Delphi solution to the use of specialists. Management Science, 9 (3), 458-467.
Dalkey, N. C., , Rourke, D. M. (1972). Trial and error assessment of Delphi methods with group value decision. In N. C. Dalkey, D. D. Rourke, R. Lewis, , D. Snyder (Eds. ). Studies inside the quality of life: Delphi and decision-making (pp. 55-83). Lexington, MUM: Lexington Books. Delbecq, A. L., Truck de Ven, A. L., , Gustafson, D. L. (1975). Group techniques for plan planning. Glenview, IL: Jeff, Foresman, and Co. Douglas, D. C. (1983). A comparative study of the efficiency of decision making processes which utilize the Delphi and leaderless group strategies.
Unpublished petulante dissertation, The Ohio State University, Columbus. Eckman, C. A. (1983). Development of musical instrument to evaluate intercollegiate athletic trainers: A revised Delphi study. Unpublished doctoral dissertation, Western Virginia School, Morgantown. Green, P. J. (1982, March). The content of a college-level outdoor leadership course. Paper shown at the Meeting of the Northwest District Association for the American 6th Alliance to get Health, Physical Education, Excitement, and Move, Spokane, WA. Hasson, N., Keeney, T., , McKenna, H. (2000). Research guidelines for the Delphi study technique.
Diary of Advanced Nursing, 32 (4), 1008-1015. Helmer, To., , Rescher, N. (1959). On the epistemology of the controuvé science. Administration Science, six, 25-53. Slope, K. Queen., , Fowles, J. (1975). The methodological worth of the Delphi forecasting technique. Scientific Forecasting and Social Change, 7, 179-192. Jacobs, J. M. (1996). Essential examination criteria pertaining to physical education teacher education programs: A Delphi examine. Unpublished important dissertation, Western world Virginia University, Morgantown. Williams, C. G. (1975). A Delphi evaluation of agreement between businesses. In They would. A. Linstone, , M.
Turoff (Eds. ). The Delphi method: Techniques and applications (pp. 160-167). Studying, MA: Addison-Wesley Publishing Company. Jones, They would., , Twiss, B. C. (1978). Predicting technology pertaining to planning decision. London, UK: Macmillan Press Ltd. Judd, R. C. (1972). Usage of Delphi methods in higher education. Technological Predicting and Interpersonal Change, some (2), 173-186. Kaplan, L. M. (1971). The use of the Delphi method in organizational interaction: A case study. Unpublished master’s thesis, The Ohio Point out University, Columbus. Kerlinger, Farreneheit. N. (1973). Foundations of behavioral research.
New York: Holt, Rinehart, and Winston, Inc. Klee, A. J. (1972). The utilization of expert thoughts and opinions in decision-making. AICHE Log, 18 (6), 1107-1115. Lindeman, C. A. (1981). Goals within the medical care system: A Delphi survey. Kansas City, MO: American Nurses’ Association. Linstone, H. A., , Turoff, M. (1975). Introduction. In H. A. Linstone, , M. Turoff (Eds. ). The Delphi method: Techniques and applications (pp. 3-12). Reading, MOTHER: Addison-Wesley Publishing Company. Ludlow, J. (1975). Delphi inquiries and understanding utilization. In H. A. Linstone, , M. Turoff (Eds. ).
The Delphi method: Approaches and applications (pp. 102-123). Reading, MUM: Addison-Wesley Publishing Company. Ludwig, B. G. (1994). Internationalizing Extension: A great exploration of you will evident within a state school Extension system that achieves internationalization. Unpublished doctoral texte, The Kansas State School, Columbus. Useful Assessment, Analysis , Evaluation, Vol 12, No 12 Hsu , Sandford, Delphi Technique Ludwig, B. (1997). Predicting the near future: Have you considered using the Delphi methodology? Journal of Extension, 35 (5), 1-4. Retrieved November 6, 2005 from http://www. oe. org/joe/1997october/tt2. html Marchant, E. Watts. (1988). Methodological problems associated with the use of the Delphi strategy: Some feedback. Fire Technology, 24 (1), 59-62. Martino, J. S. (1983). Technological forecasting to get decision making. Nyc: North-Holland. Meyer, J. L. (1992). Rethinking the view of colleges in whose roots are usually in agriculture. Davis, CA: School of Cal. Miller, G. (2001). The introduction of indicators to get sustainable travel: Results of a Delphi review of tourism researchers. Tourism Management, twenty two, 351-362. Miller, L. Electronic. (2006, October).
Determining what could/should always be: The Delphi technique and its particular application. Newspaper presented with the meeting from the 2006 total annual meeting from the Mid-Western Educational Research Association, Columbus, Kansas. Murray, T. F., , Jarman, W. O. (1987). Predicting future trends in adult fitness using the Delphi approach. Research Quarterly pertaining to Exercise and Sport, fifty eight (2), 124-131. Oh, E. H. (1974). Forecasting through hierarchical Delphi. Unpublished tragique dissertation, The Ohio State University, Columbus. Pill, M. (1971). The Delphi technique: Substance, circumstance, a analyze and a great annotated bibliography.
Socio-Economic Planning Science, 5, 57-71. six Scheibe, M., Skutsch, M., , Schofer, J. (1975). Experiments in Delphi technique. In They would. A. Linstone, , M. Turoff (Eds. ). The Delphi approach: Techniques and applications (pp. 262-287). Reading, MA: Addison-Wesley Publishing Organization. Taylor, 3rd there’s r. E., , Judd, T. L. (1989). Delphi approach applied to tourism. In H. Witt, , L. Moutinho, (Eds. ). Tourism advertising management guide. New York: Prentice Hall. Turoff, M., , Hiltz, S. R. (1996). Computer structured Delphi method. In M. Adler, , E. Ziglio (Eds. ).
Gazing into the oracle: The Delphi method and its software to cultural policy and public health (pp. 56-88). Greater london, UK: Jessica Kingsley Marketers. Ulschak, Farreneheit. L. (1983). Human resource expansion: The theory and practice of need analysis. Reston, VETERANS ADMINISTRATION: Reston Publishing Company, Incorporation. Weaver, W. T. (1971). The Delphi forecasting method. Phi Delta Kappan, 52 (5), 267-273. Witkin, W. R. (1984). Assessing needs in educational and cultural programs. Bay area, CA: Jossey-Bass Publishers. Witkin, B. R., , Altschuld, J. Watts. (1995). Planning and doing needs analysis: A practical guideline.
Thousand Oak trees, CA: Sage Publications, Inc. Young, T. J., , Jamieson, M. M. (2001). Delivery technique of the Delphi: A comparison of two methods. Journal of Park and Recreation Operations, 19 (1), 42-58. Citation Hsu, Chia-Chien , Sandford, Brian A. (2007). The Delphi Technique: Making Feeling of General opinion. Practical Evaluation Research , Evaluation, 12(10). Available online: http://pareonline. net/getvn. asp? v=12, n=10 Editors Note: Another conventional paper on the Delphi Technique that appeared in Practical Examination Research , Evaluation is: Yousuf, Muhammad Imran (2007).
Using Experts’ Opinions through Delphi Strategy. Practical Analysis Research , Evaluation, 12(4). Available online: http://pareonline. net/getvn. asp? v=12, n=4. Authors Chia-Chien Hsu Post-doctoral Studies The Ohio Express University 393 Schrock Road Worthington, OH 43085 Functional Assessment, Exploration , Evaluation, Vol 12, No 12 Hsu , Sandford, Delphi Technique Tel: (614) 885-0763 E-mail: hsu. 127 [at] osu. edu Brian A. Sandford Assistant Professor 214 Willard Lounge Occupational Education Oklahoma Point out University Stillwater, Oklahoma 74074 405-744-3461 brian. sandford [at] okstate. edu 8
