The future of practical skills in undergraduate medical education – an explorative Delphi-Study

 Katja Anne Dannenberg ^1,2
Fabian Stroben ¹
Therese Schröder ³
Anke Thomas ³
Wolf E. Hautz <sup>4

1</sup> Charité – Universitätsmedizin Berlin, Lernzentrum (Skills Lab), Berlin, Germany
² Charité – Universitätsmedizin Berlin, Department of Emergency Medicine at Campus Benjamin Franklin, Berlin, Germany
³ Charité – Universitätsmedizin Berlin, Gynecology and Obstetrics Clinic, Berlin, Germany
⁴ Inselspital Bern, University Emergency Center, Bern, Switzerland

Abstract

Background: 64% of young medical professionals in Germany do not feel adequately prepared for the practical requirements of the medical profession. The goal of “outcome-orientated training” is to structure medical curricula based on the skills needed when entering the workforce after completing undergraduate medical education, and thus to bridge the gap between the skills graduates have attained and those necessary for a career in the medical profession. Outcome frameworks (OFs) are used for this purpose. In preparation for developing the National Competence-Based Catalogue of Learning Objectives for Medicine (NKLM) – the German OF – the “Consensus Statement of Practical Skills in Undergraduate Medical Education” (which structures the teaching and acquisition of practical skills in Germany and which strongly influenced the “Clinical-Practical Skills” chapter of the NKLM) was published in 2011.

It is not uncommon for at least a decade to elapse between the definition and implementation of an OF and the students’ graduation, which can further increase the gap between necessary and acquired skills. Thus, the purpose of this paper is to posit theses for future development in healthcare and to apply these theses to a current OF.

Methodology: Partially structured interviews with experts were used to generate theses pertaining to general, future development in healthcare. These theses were assessed by physician experts based on the likelihood of implementation by the year 2025. The 288 learning goals of the consensus statement were assessed for their relevance for medical education in the interim.

Results: 11 theses were generated for the development of medicine, and these theses were assessed and discussed by 738 experts. These theses include the increase in diseases associated with old age, the increasing significance of interprofessional cooperation, and the growing prevalence of telemedicine applications. Of the 288 learning goals of the consensus statement, 231 of the goals were assessed as relevant, and 57 were deemed irrelevant for the short-term future.

Discussion: The theses on the future of healthcare, which were generated in this study and which were validated by numerous experts, provide indications of future developments of overall requirements for medical school graduates. For example, when applied to the content of the “Clinical-Practical Skills” NKLM chapter, they largely validate the future relevance of developing practical skills while also providing indications for their further development as applied to the consensus statement.

---

---

Introduction

On the one hand, the significance of obtaining practical skills during undergraduate medical studies has increased significantly in recent years [1], [2]. On the other hand, 64.7% of young medical professionals in Germany state that they do not feel adequately prepared for the practical requirements of the medical profession [3], a figure which is startlingly high, even compared to international data [4], [5]. Possible causes identified by the graduate survey in Cologne (Stosch C et al., unpublished) and a national survey (partially published in [6]), were both the narrow scope of practical training and the inadequate or lacking integration of this training in curricula and examinations.

In order to bridge the gap between education and training, medical curricula are increasingly oriented toward national framework curricula, known as “outcome frameworks” (OF) [7], [8], which – generally speaking – describe the skills and knowledge which should be obtained during a training period in a competence-oriented fashion. Various outcome definitions exist internationally [9], [10], [11]. The Tuning Project [12] in Europe is an attempt to synchronize the many national OFs currently in existence. The German “National Competence-Based Catalogue of Learning Objectives for Medicine” (NKLM) was developed by the medical faculty association in cooperation with the Society for Medical Education (GMA) [13] and was initially published in June of 2015 after a six-year period of development [14]. In preparation for developing the NKLM, the “Consensus Statement of Practical Skills in Undergraduate Medical Education” was developed by the committee for practical skills of the GMA in 2011 [15]. This consensus statement “can and should have a formative effect on faculties to adjust their curricula in accordance with guidelines” [15] and strongly influenced the “Clinical-Practical Skills” chapter of the NKLM. The recommendations of the consensus statement have been implemented and validated within at least one faculty department [16]. In addition, the statement serves to assist the simulator network – a merger of the DACH region Skillslabs – to structure its simulator database [17].

There are, however, notable differences in content and structure between different OFs [18], [19], which raises the question of which OF should reasonably be referenced for teaching proficiency. In addition, developing medical curricula is generally a lengthy process: the six stages of the Kern cycle as a widely taught model of curriculum development [20], for example, require a considerable period of time between the initial definition of requirements, implementation, evaluation, and adaptation. Furthermore, an average of 6.4 years [21] elapse between beginning undergraduate medical education and beginning to practice medicine [21]. This contrasts starkly with rapid developments in medicine and the use of new technologies which have become ubiquitous. Consequently, there is a risk that the contents of curricula developed based on current OFs are no longer up-to-date when the medical professionals educated accordingly enter the medical profession.

1. Object of the Study

The object of this study is to examine the “Consensus Statement of Practical Skills in Undergraduate Medical Education,” and thus an important part of the NKLM, for medium-term sustainability. The results should, on the one hand, serve to provide details for the further development of the NKLM; and on the other hand, help enhance the future stability of OF and curricula by means of overarching trends in healthcare which must yet be identified. The applied explorative Delphi method, as well as its results, can also serve to further develop local and national curricula.

2. The explorative Delphi method

Originally developed in the 1950s as a technique for exploring technical developments in a military context [22], this method had been continually developed in the intervening decades [23] and is now considered an established method for analyzing uncertain developments and identifying strategic treatment options [22], [24], [25]. In principle, the Delphi method serves to collect group opinions and to focus group communications [24], as well as to qualitatively and quantitatively assess uncertain facts [24]. Although widely varied definitions of the Delphi method exist [24], certain common basic principles can be identified: anonymity of experts, multiple repetitions of the survey, statistical summary of group opinion, and controlled feedback [22]. The use of the Delphi method has been tested in various contexts, though here it is predominantly of interest to sufficiently documented applications in medical education research, such as for developing guidelines [26], [27], [28], [29].

Methods

The project was structured into preparatory and working phases. During preparations, literary research followed by partially structured stakeholder interviews was used to develop theses about developments in healthcare. These theses were then assessed by means of an expert survey. In addition, the same expert cohort assessed the 288 learning goals of the “Consensus Statement of Practical Skills in Undergraduate Medical Education” within the framework of a 2-level, explorative Delphi survey. The course of the study is depicted in Figure 1 [Fig. 1].

1. Preparatory Phase: Theses on healthcare development

Guidelines for partially structured interview with various healthcare practitioners were developed by means of selective literature research. The topics discussed in the interview included the following:

• The future development of healthcare
• Potential changes to care and to the disease spectrum
• Changes in medical technology and telemedicine
• Interdisciplinarity and cooperation with other occupational groups
• Future changes to undergraduate medical education
• Medical occupations in Germany and abroad
• Medical skills needed in the future

During the preparatory phase, 9 interviews were conducted with experts in the fields of public health, medical technology and pedagogy, clinical and outpatient, practical skills, and students of human medicine (cf. Table 1 [Tab. 1] for details). Interview partners were chosen by means of a “purposive sampling strategy” [30] with the goal of obtaining as broad a spectrum of perspectives on these topics as possible. All interviewed experts were prepared for the interview. Interviews lasted an average of 22 minutes.

The interviewees’ answers were then grouped by topic using a qualitative text analysis and summarized as theses by means of inductive categorization per Mayring [31] by an interdisciplinary research group comprised of two students of futures studies, including one nurse and one student of human medicine with paramedic training, one practicing physician, and one computer scientist. The goal of the Mayring analysis is to systematically process the written communications at hand, and to identify similarities and differences [32]. The principles of categorization were a) category selectivity, and b) a high level of abstraction of the same.

2. Working Phase: Expert interviews on future theses generated, and on consensus statement learning goals

After generating theses, their probability of occurrence was assessed within the framework of an expert interview. The individual learning goals of the “Practical Skills in Undergraduate Medical Education” consensus statement were then assessed by the same experts within the framework of a two-level Delphi study. Physicians in all German medical university hospitals whose email addresses were available on the internet, as well as established physicians, were contacted via email to request their participation in the study. In addition to these 8,000 physicians contacted, others were approached at conferences (e.g., the Skills-Lab Symposium 2012) to request their participation in the study.

Each participant then assessed the probability of occurrence of these theses on the future of medical care (generated during the preparatory phase) using a 4-level Likert scale (1 – very likely to 4 – very unlikely). Afterward, each participant was then assigned randomly to a group of ten in order to assess the future relevance of the consensus statement learning goals. This statement defines 288 learning goals assigned to one of 16 organ systems. There is a statement for three different training stages (“clinical traineeship, practical year, advanced training”) based on a three-tiered scale for each learning goal, to what extent this should be mastered (“seen demonstrated, performed under supervision, performed repeatedly”), and the survey further distinguishes between core and elective goals [15]. Each group was asked to assess a portion of the consensus statement learning goals (ca. 30/group) vis-à-vis their relevance for general medical training up to completing undergraduate medical education in the year 2025 using a 4-level Likert scale (1 – highly relevant to 4 – not at all relevant). The learning goal assessments were depicted based on the degree of mastery required by the advanced training stage, as stipulated in the consensus statement.

After round 1 assessments and individual review of the results by the research group, the round 2 learning goals to be assessed were determined by consensus. Selection criteria included a wide distribution of assessment and the estimated significance of each learning goal as assessed in round 1. The learning goals were then re-evaluated by physicians participating in round 2, who were provided with the result of round 1 evaluations. Registered participants were assigned randomly to two groups for this purpose. Each group re-evaluated circa 50 learning goals.

3. Data evaluation

The online interview was conducted using LimeSurvey (http://www.limesurvey.org). Data evaluation was conducted with Microsoft Excel (Microsoft Corporation, Redmond, WA, USA) and IBM SPSS Statistics 21.0 (SPSS Inc., Chicago, IL, USA).

Results

1. Participants

738 experts registered online for round 1 of the study, and 594 complete data sets (19.5% dropout rate) were usable. 314 experts were registered for round 2, and 188 complete data sets (40.1% dropout rate) were usable. Since the learning goals were assessed within the context of the organ system assigned to them, we only took complete data sets into consideration.

Participants were only asked to assess theses about the future during round 1. Partially-completed questionnaires were taken into consideration here, as the theses about the future can logically be interpreted on an individual basis. For this purpose, 651 expert opinions (11.8% dropout rate) were available.

A large portion of experts in round 1 had more than one year of work experience (96.0%), and 137 experts had been practicing for more than 15 years. The overwhelming majority of physicians worked on an inpatient basis in maximum-care hospitals (87.9%). Study participants represented a total of 26 disciplines. The number of experts in round 2 was smaller, though the characteristics of their working environments were similar. In comparison to German Medical Association (BÄK) statistics from 2014, the proportion of inpatient physicians is large (88.6% in round 1 and 79.3% in round 2, compared to 51.0% by BÄK figures), and the exact distribution of specialties also differed somewhat. Overall, 57.8% (round 1) and 53.7% (round 2) of experts work in disciplines such as surgery, internal medicine, or anesthesia, in addition to general medicine. This is consistent with BÄK data, which lists this figure at 48.8% [http://www.bundesaerztekammer.de/fileadmin/user_upload/downloads/pdf-Ordner/Statistik2014/Stat14AbbTab.pdf]. An exact analysis of the participant cohort in comparison to BÄK figures from 2014 is depicted in Table 2 [Tab. 2].

2. Theses on the future of healthcare

A total of 11 theses on the future of health care were derived from the partially structured interviews (cf. Table 3 [Tab. 3]).

The theses on the future of healthcare trends generated in the preparatory phase are listed below. It was assumed that these theses were listed in accordance with expert assessment (cf. Table 3 [Tab. 3] and Figure 2 [Fig. 2]):

Aspects of managing dementia are becoming significantly more important in physician communication. Increasingly balanced patient-physician relationships are emphasizing non-authoritarian forms of discussion and reasoning.

Increasing mechanization also poses a barrier to entry in the medical field: the relevance of purely manual skills is decreasing, yet IT technologies still cannot determine medical history or diagnoses, requiring the skills of doctors. Diagnostics and patient monitoring will lead to less physical contact, and patients prefer internet and smartphones for this purpose.

The physician remains the personal point of contact in established practice concepts. Duties once performed purely by physicians are, however, increasingly being delegated or substituted. Mobile treatment concepts of primary care are not catching on.

Business economic considerations are slowly moving into focus: while business economic and organizational aspects are included in training, patients’ financial concerns also play a role in the type of care and treatment.

3. Assessment of learning goals

288 learning goals were assessed by experts in the 2 rounds of the study. The average of all expert assessments was used to determine whether a learning goals was deemed relevant or irrelevant. In round 1 of the Delphi study, 240 learning goals were assessed as relevant or highly relevant (average<2.5) and 57 were assessed as somewhat relevant or irrelevant (average>2.5), while 1 learning goal was assessed as neither (average=2.5).

After reviewing the results of round 1, 103 learning goals were selected for assessment in round 2 based on the distribution of round 1 results; of these, 71 were assessed as relevant, 31 were assessed as irrelevant, and 1 was assessed as neutral in round 1. In round 2, experts assessed 62 learning goals as relevant and 41 as irrelevant. In comparing the two rounds, 13 learning goals (12.6%) were rated less relevant and 4 (3.9%) were rated as more relevant in round 2. A total of 231 learning goals were considered relevant and 57 learning goals were considered irrelevant. Figure 1 [Fig. 1] depicts an overview of these results.

A more extensive review based on organ systems revealed that a large portion of learning goals for the sensory organ system (65.0%) were assessed as irrelevant for the future. Likewise, the future relevance of numerous learning goals associated with the skin, urogenital, and GI tract organs (≥30.0% each) was called into question.

31 of the 55 (56.4%) elective learning goals were assessed as irrelevant for the future – on the other hand, more than half (54.4%) of non-relevant learning goals are elective. Only 8% of the learning goals and skills which should be mastered upon completing advanced training were assessed as irrelevant for the future, yet 42% of skills were assessed at the lowest level of skill (“seen demonstrated”). The exact results are depicted in Table 4 [Tab. 4].

The online appendix [Att. 1] of this study shows an overview of all consensus statement learning goals and their assessments in the two rounds of the Delphi study.

Discussion

This paper attempts, on the one hand, to anticipate future global requirements for medical school graduates; and on the other, to concretely examine the future relevance of practical medical skills as an example of a limited scope of competency in undergraduate medical studies. For the latter, the learning goals of the consensus statement of practical skills in undergraduate medical studies was assessed by means of an explorative Delphi study as preliminary work to the NKLM “clinical-practical skills” chapter.

The panel of experts in the Delphi study possesses many years of work experience and represents nearly all medical specialties. The majority of experts work in maximum-care hospitals, including university hospitals in which undergraduate training is primarily conducted in Germany. Though the experts’ more intensive knowledge has influenced the contents and requirements of the study, outpatient physicians (who are needed primarily for contributing opinions on these theses about the future of medical care) are underrepresented. We can only speculate on the reasons for which so few outpatient physicians participated in the study.

At the beginning of this paper, 11 theses for relevant topics for future medical education were identified. We deliberately chose not to provide an OF as a basis for the interviews, as national OFs differ substantially in structure [18] as well as content [19].

An American group has already published a similar approach [33] which did not, however, take any further validation steps for its theses in comparison to our study. Below, we will discuss a few theses of this study in the context of their assessment by participating experts and derive implications for medical education:

The current demographic shift has caused an increase in diseases associated with advanced age, such as mild cognitive impairment and dementia [34], [35]. The learning goals catalog lists two learning goals which could be attributed to dementia illnesses. Determining the medical history of elderly patients and performing simple test procedures such as geriatric assessments or falling risk tests were both assessed as relevant for the future. However, emphasizing geriatric test procedures was listed in the catalog as an elective goal, though it should be a core learning goals according to the experts in this study.

The state of medical care, especially in rural areas with their own specific requirements [36], is in need of improvement [37]. Experience obtained in these places during voluntary training during undergraduate medical studies seemed to have a positive effect on students’ learning and career choices [38] and could improve primary care. The mandatory primary care physician clinical traineeship [39] was recently introduced, and a GMA position paper emphasizes the significance of primary care during undergraduate medical studies [40].

In addition to these structural changes, working on an (interprofessional) team and using telemedicine or E-health applications will become more important in the future: delegating work to non-medical personnel increases the effectiveness of primary care [41], [42]. At the same time, learning goals which do not apply solely to physicians, such as applying plaster casts or demonstrating functional taping, have also been assessed as relevant for the future. Beginning to promote interprofessional cooperation during undergraduate medical studies, such as combined courses with trainees or students of other medical care professions, could be one method of implementing the interpersonal aspect of care more intensively during medical education.

In addition, electronic support system (health information technology) resources that are currently available could be used more effectively [42], [43]. Some positive effects of this technology, such as increased activity for COPD patients [44] or improved control of chronic asthma symptoms [45], have already been demonstrated. Integrating this growing field into training and education seems crucial, and could take place by means of telemedicine modules [46]. A dedicated learning goals catalog for e-health and telemedicine has already been published and can be consulted for future developments [47].

In consideration of this knowledge and the high relevance of soft skills and communication ability in this Delphi study, telephone- or internet-based physician-patient interaction could also grow more significant [48]. In order to do this field justice, training for communication skills (e.g., via telephone) should be intensified [49], as has been implemented in individual cases [50], [51]. Older patients in Germany see telemedicine methods more critically, however, and miss personal contact with and direct feedback from their physician [52].

Prioritization of core and elective learning goals in the OF original publication [15] (which were, in part, determined by means of the Delphi methodology and our results) mutually validate each other. More than 90% of the consensus statement learning goals assessed as needing to be mastered and nearly all border area learning goals were assessed by our participants as especially relevant for the future of the medical profession. On the other hand, more than 50% of skills listed in the consensus statement as elective were also assessed as less relevant by our study.

The experts assessed practical learning goals overwhelmingly as relevant for the future, primarily in the large categories of communication skills, soft skills, interinstitutional skills, cardiovascular, and emergency, but also in narrower disciplines such as mental health and the endocrine system. The portion of learning goals not relevant for the future is largest for the sensory organs and for the urogenital and GI tract systems. This could be related to the choice of experts, but could also be due to the fact that the learning goals were phrased in a very specific manner, and thus there are a great many of them. In other catalog categories, more learning goals tended to be summarized as one, which made it difficult for experts to provide a differentiated assessment. In addition, it cannot be determined whether rejection of a learning goal was due to a general lack of future relevance, or because experts considered the learning goal relevant for the future, but believed that the learning goal should be a part of specialty training rather than general medical training.

In a further step, the detailed results of this study (cf. online appendix) could be used, just like other validation studies [16], to re-assess the individual learning goals of the consensus statement and the NKLM, which would contribute to a review of the consensus statement and the NKLM.

Preparing future physicians to practice medicine should be done, in parallel, on as many levels as possible. Numerous OFs, unlike this study, currently name the “self-directed learning” method resulting from “self-assessment” [53] as a significant method of improving the results of undergraduate medical studies [54]. At the same time, there are significant doubts as to the accuracy of self-assessments [53], [55], [56]. Thus, “lifelong learning” based on self-assessment cannot be the only methods of anticipating and addressing future developments.

This should be done at the level of the OF. In addition to instructions on effective, self-directed, and lifelong learning, optimizing current OFs could make significant contributions. Anticipating future developments in conjunction with current research results could, on the one hand, provide important incentives for new content, and on the other hand, explorative Delphi studies could examine current learning goals and OFs with respect to their sustainability and possibly identify deficiencies. In concrete terms, the results of the Delphi study could serve to justify specific revisions to and implementation of the NKLM in different departments. Broad trend-setting decisions on possible future trends in undergraduate medical education can be derived from the assessment of these theses on the future.

1. Limitations

Cognitive bias must be accepted as a significant limitation of any expert survey. This is of particular importance for the application of the explorative Delphi method for assessing issues that are uncertain, per se, as they take place in the future, since the line between rational assessment and the experts’ personal desires or fears could be blurred [57]. The structure of the survey could also have influenced expert opinions. After first assessing the likelihood of implementing theses in the future, experts were then asked to assess the future relevance of learning goals. This could have led to a bias. Though experts were asked to base their assessments on general education leading up to the medical exam, it cannot be determined whether this was actually done, and to what extent experts assessed general education as opposed to specialty training.

The study population of this paper is comprised primarily of inpatient physicians at maximum-care facilities. Consequently, a bias against outpatient treatment methods cannot be ruled out. In addition, colleagues practicing general medicine were underrepresented (just 2.9% of experts), while anesthesiology and intensive care – two highly specialized subjects – were overrepresented, which could explain the strong emphasis on technological trends in the theses generated. One possible explanation for the large number of anesthesiology practitioners represented in the study could be their disproportionate involvement in conveying practical skills. A follow-up survey with primarily outpatient physician seems reasonable.

Conclusions

The explorative Delphi method provides an adequate opportunity to allow a current outcome framework to be assessed by experts on the basis of its future relevance. In addition, future trends can be anticipated by means of generating and assessing theses. It is important to continually review and adapt current OF and curricula to future developments in order to provide optimal preparation for medical studies graduates for their future daily professional lives.

Acknowledgements

We would like to thank Sascha Dannenberg for his help and support with technical questions about futures studies, and for his assistance in carrying out the study. In addition, we would like to thank Nicole Ambacher and Daniel Knapp for their support in data collection. Finally, we would like to sincerely thank all of the experts who took part in this study.

Data

Data for this article are available from the Dryad Digital Repository: http://dx.doi.org/10.5061/dryad.q4sc8 [58]

Competing interests

The authors declare that they have no competing interests.

---

References

[1] Nicky Hudson J, Tonkin AL. Clinical skills education: outcomes of relationships between junior medical students, senior peers and simulated patients. Med Educ. 2008;42(9):901-908. DOI: 10.1111/j.1365-2923.2008.03107.x
[2] Yudkowsky R, Otaki J, Lowenstein T, Riddle J, Nishigori H, Bordage G. A hypothesis-driven physical examination learning and assessment procedure for medical students: initial validity evidence. Med Educ. 2009;43(8):729-740. DOI: 10.1111/j.1365-2923.2009.03379.x
[3] Ochsmann EB, Zier U, Drexler H, Schmid K. Well prepared for work? Junior doctors' self-assessment after medical education. BMC Med Educ. 2011;11:99. DOI: 10.1186/1472-6920-11-99
[4] Cave J, Woolf K, Jones A, Dacre J. Easing the transition from student to doctor: how can medical schools help prepare their graduates for starting work? Med Teach. 2009;31(5):403-408. DOI: 10.1080/01421590802348127
[5] Goldacre MJ, Taylor K, Lambert TW. Views of junior doctors about whether their medical school prepared them well for work: questionnaire surveys. BMC Med Educ. 2010;10:78. DOI: 10.1186/1472-6920-10-78
[6] Damanakis A. Ein kleiner Statusreport zu Skills Labs in der D-A-CH-Region. Jahrestagung der Gesellschaft für Medizinische Ausbildung (GMA). München, 05.-08.10.2011. Düsseldorf: German Medical Science GMS Publishing House; 2011. Doc11gma133. DOI: 10.3205/11gma133
[7] Harris P, Snell L, Talbot M, Harden RM. Competency-based medical education: implications for undergraduate programs. Med Teach. 2010;32(8):646-650. DOI: 10.3109/0142159X.2010.500703
[8] Busing N, Slade S, Rosenfield J, Gold I, Maskill S. In the spirit of Flexner: working toward a collective vision for the future of medical education in Canada. Acad Med. 2010 Feb;85(2):340-348. DOI: 10.1097/ACM.0b013e3181c8880d
[9] Bloch R, Burgi H. The Swiss catalogue of learning objectives. Med Teach. 2002 Mar;24(2):144-150. DOI: 10.1080/01421590220120759
[10] Frank JR, Danoff D. The CanMEDS initiative: implementing an outcomes-based framework of physician competencies. Med Teach. 2007;29(7):642-647. DOI: 10.1080/01421590701746983
[11] Metz JC. 'Blueprint 1994': common objectives of medical education in The Netherlands. Neth J Med. 1999;55(4):165-167. DOI: 10.1016/S0300-2977(99)00067-4
[12] Cumming A, Ross M. The Tuning Project for Medicine-learning outcomes for undergraduate medical education in Europe. Med Teach. 2007;29(7):636-641. DOI: 10.1080/01421590701721721
[13] Hahn EG, Fischer MR. Nationaler Kompetenzbasierter Lernzielkatalog Medizin (NKLM) für Deutschland: Zusammenarbeit der Gesellschaft für Medizinische Ausbildung (GMA) und des Medizinischen Fakultätentages (MFT). GMS Z Med Ausbild. 2009;26(3):Doc35. DOI: 10.3205/zma000627
[14] Fischer MR, Bauer D, Mohn K, NKLM-Projektgruppe. Finally finished! National Competence Based Catalogues of Learning Objectives for Undergraduate Medical Education (NKLM) and Dental Education (NKLZ) ready for trial. GMS Z Med Ausbild. 2015;32(3):Doc35. DOI: 10.3205/zma000977
[15] Schnabel KP, Boldt PD, Breuer G, Fichtner A, Karsten G, Kujumdshiev S, Schmidts M, Stosch C. A consensus statement on practical skills in medical school - a position paper by the GMA Committee on Practical Skills. GMS Z Med Ausbild. 2011;28(4):Doc58. DOI: 10.3205/zma000770
[16] Blaum WE, Dannenberg KA, Friedrich T, Jarczewski A, Reinsch AK, Ahlers O. The practial use of the consensus statement on practical skills in medical school-a validation study. GMS Z Med Ausbild. 2012;29(4):Doc58. DOI: 10.3205/zma000828
[17] Damanakis A, Blaum WE, Stosch C, Lauener H, Richter S, Schnabel KP. Simulator Network project report: a tool for improvement of teaching materials and targeted resource usage in Skills Labs. GMS Z Med Ausbild. 2013;30(1):Doc4. DOI: 10.3205/zma000847
[18] Hautz SC, Hautz WE, Feufel MA, Spies CD. Comparability of outcome frameworks in medical education: Implications for framework development. Med Teach. 2015;20:1-9. DOI: 10.3109/0142159x.2015.1012490
[19] Hautz SC, Hautz WE, Keller N, Feufel MA, Spies C. The scholar role in the National Competence Based Catalogues of Learning Objectives for Undergraduate Medical Education (NKLM) compared to other international frameworks. GMS Ger Med Sci. 2015;13:Doc20. DOI: 10.3205/000224
[20] Kern DE. Curriculum development for medical education: a six-step approach. Baltimore u.a.: Johns Hopkins Univ. Press; 1998.
[21] Wijnen-Meijer M, Burdick W, Alofs L, Burgers C, ten Cate O. Stages and transitions in medical education around the world: clarifying structures and terminology. Med Teach. 2013;35(4):301-307. DOI: 10.3109/0142159X.2012.746449
[22] Rowe G, Wright G. The Delphi technique as a forecasting tool: issues and analysis. I J Forecast. 1999;15(4):353-375. DOI: 10.1016/S0169-2070(99)00018-7
[23] Grunwald A. Modes of orientation provided by futures studies: making sense of diversity and divergence. Eur J Futures Res. 2013;2(1).
[24] Häder M. Delphi-Befragungen: ein Arbeitsbuch. 2. Aufl. ed. Wiesbaden: VS; 2009. DOI: 10.1007/978-3-531-91926-3
[25] Gönder D. KiTA 2030: Eine Delphi-Befragung zur Zukunft von Kindertageseinrichtungen in Deutschland. iF-Schriftenreihe Sozialwissenschaftliche Zukunftsforschung 04/2014. Berlin: Freie Universität Berlin, Institut Futur; 2014.
[26] Williams PL, Webb C. The Delphi technique: a methodological discussion. J Adv Nurs. 1994;19(1):180-186. DOI: 10.1111/j.1365-2648.1994.tb01066.x
[27] Houwink EJ, Henneman L, Westerneng M, van Luijk SJ, Cornel MC, Dinant JG, Vleuten Cv. Prioritization of future genetics education for general practitioners: a Delphi study. Genet Med. 2012;14(3):323-329. DOI: 10.1038/gim.2011.15
[28] De Villiers MR, de Villiers PJ, Kent AP. The Delphi technique in health sciences education research. Med Teach. 2005;27(7):639-643. DOI: 10.1080/13611260500069947
[29] Hasson F, Keeney S, McKenna H. Research guidelines for the Delphi survey technique. J Adv Nurs. 2000;32(4):1008-1015.
[30] Tavakol M, Sandars J. Quantitative and qualitative methods in medical education research: AMEE Guide No 90: Part II. Med Teach. 2014;36(10):838-848. DOI: 10.3109/0142159X.2014.915297
[31] Mayring, P. Qualitative Inhaltsanalyse. Grundlagen und Techniken. 11., aktualisierte und überarbeitete Auflage, Weinheim, Basel: Beltz Verlag; 2010.
[32] Flick U, von Kardorff E, Steinke I. Qualitative Forschung. Ein Handbuch. Orig.-Ausg., 8. Reinbek bei Hamburg: Rowohlt-Taschenbuch-Verl; 2010.
[33] Pershing S, Fuchs VR. Restructuring Medical Education to Meet Current and Future Health Care Needs. Acad Med. 2013;88(12):1798-1801. DOI: 10.1097/ACM.0000000000000020
[34] Pater C. Mild cognitive impairment (MCI) - the novel trend of targeting Alzheimer's disease in its early stages - methodological considerations. Curr Alzheimer Res. 2011;8(7):798-807. DOI: 10.2174/156720511797633250
[35] Langa KM, Levine DA. The diagnosis and management of mild cognitive impairment: a clinical review. JAMA. 201417;312(23):2551-2561.
[36] Weeks WB, Wallace AE. Rural-urban differences in primary care physicians' practice patterns, characteristics, and incomes. J Rural Health. 2008;24(2):161-170. DOI: 10.1111/j.1748-0361.2008.00153.x
[37] Rosenblatt RA, Andrilla CH, Curtin T, Hart LG. Shortages of medical personnel at community health centers: implications for planned expansion. JAMA. 2006;295(9):1042-1049. DOI: 10.1001/jama.295.9.1042
[38] Barrett FA, Lipsky MS, Lutfiyya MN. The impact of rural training experiences on medical students: a critical review. Acad Med. 2011;86(2):259-263. DOI: 10.1097/ACM.0b013e3182046387
[39] Bundesministerium für Gesundheit. Erste Verordnung zur Änderung der Approbationsordnung für Ärzte vom 17. Juli 2012. Bundesgesetzbl. 2012;I(34);1539. Zugänglich unter/available from: http://www.bgbl.de/xaver/bgbl/start.xav?start=%2F%2F*[%40attr_id%3D%27bgbl112s1539.pdf%27]#__bgbl__%2F%2F*[%40attr_id%3D%27bgbl112s1539.pdf%27]__1436264296102
[40] Huenges B, Gulich M, Böhme K, Fehr F, Streitlein-Böhme I, Rüttermann V, Baum E, Niebling WB, Rusche H. Recommendations for undergraduate training in the primary care sector-position paper of the GMA-Primary Care Committee. GMS Z Med Ausbild. 2014;31(3):Doc35. DOI: 10.3205/zma000927
[41] Altschuler J, Margolius D, Bodenheimer T, Grumbach K. Estimating a reasonable patient panel size for primary care physicians with team-based task delegation. Ann Fam Med. 2012;10(5):396-400. DOI: 10.1370/afm.1400
[42] Chen PG, Mehrotra A, Auerbach DI. Do we really need more physicians? Responses to predicted primary care physician shortages. Med Care. 2014;52(2):95-96. DOI: 10.1097/MLR.0000000000000046
[43] Urban E, Ose D, Joos S, Szecsenyi J, Miksch A. Technical support and delegation to practice staff - status quo and (possible) future perspectives for primary health care in Germany. BMC Med Inform Decis Mak. 2012;12:81. DOI: 10.1186/1472-6947-12-81
[44] Lundell S, Holmner A, Rehn B, Nyberg A, Wadell K. Telehealthcare in COPD: a systematic review and meta-analysis on physical outcomes and dyspnea. Respir Med. 2015;109(1):11-26. DOI: 10.1016/j.rmed.2014.10.008
[45] Fathima M, Peiris D, Naik-Panvelkar P, Saini B, Armour CL. Effectiveness of computerized clinical decision support systems for asthma and chronic obstructive pulmonary disease in primary care: a systematic review. BMC Pulm Med. 2014;14:189. DOI: 10.1186/1471-2466-14-189
[46] Brockes MC, Wirth F, Schmidt-Weitmann S, Battegay E, Gerke W. Implementierung des Moduls Klinische Telemedizin/E-Health" in das Medizinstudium der Universität Zürich. GMS Z Med Ausbild. 2010;27(1):Doc14. DOI: 10.3205/zma000651
[47] Rohrig R, Stausberg J, Dugas M, GMDS project group Medical Informatics Education in Medicine". Development of national competency-based learning objectives "Medical Informatics" for undergraduate medical education. Methods Inf Med. 2013;52(3):184-188. DOI: 10.3414/ME13-04-0001
[48] Car J, Sheikh A. Telephone consultations. BMJ. 2003;326(7396):966-969. DOI: 10.1136/bmj.326.7396.966
[49] Car J, Freeman GK, Partridge MR, Sheikh A. Improving quality and safety of telephone based delivery of care: teaching telephone consultation skills. Qual Saf Health Care. 2004;13(1):2-3. DOI: 10.1136/qshc.2003.009241
[50] Fisher J, Martin R, Tate D. Hands on + hands free: simulated on-call interaction. Clin Teach. 2014;11(6):425-428. DOI: 10.1111/tct.12180
[51] Dickinson M, Pimblett M, Hanson J, Davis M. Reflecting reality: pager simulations in undergraduate education. Clin Teach. 2014;11(6):421-424. DOI: 10.1111/tct.12185
[52] Terschuren C, Mensing M, Mekel OC. Is telemonitoring an option against shortage of physicians in rural regions? Attitude towards telemedical devices in the North Rhine-Westphalian health survey, Germany. BMC Health Serv Res. 2012;12:95. DOI: 10.1186/1472-6963-12-95
[53] Eva KW, Regehr G. Self-assessment in the health professions: a reformulation and research agenda. Acad Med. 2005;80(10 Suppl):S46-54. DOI: 10.1097/00001888-200510001-00015
[54] Hautz SC, Hautz WE, Feufel MA, Spies CD. What makes a doctor a scholar: a systematic review and content analysis of outcome frameworks. BMC Med Educ. 2015. Under Review.
[55] Davis DA, Mazmanian PE, Fordis M, Van Harrison R, Thorpe KE, Perrier L. Accuracy of physician self-assessment compared with observed measures of competence: a systematic review. JAMA. 2006;296(9):1094-1102. DOI: 10.1001/jama.296.9.1094
[56] Li ST, Paterniti DA, Co JP, West DC. Successful self-directed lifelong learning in medicine: a conceptual model derived from qualitative analysis of a national survey of pediatric residents. Acad Med. 2010;85(7):1229-1236. DOI: 10.1097/ACM.0b013e3181e1931c
[57] Gerhold L. Standards und Gütekriterien der Zukunftsforschung: ein Handbuch für Wissenschaft und Praxis. Wiesbaden: Springer VS; 2015. DOI: 10.1007/978-3-658-07363-3
[58] Dannenberg KA, Stroben F, Schröder T, Thomas A, Hautz WE. Data from: The future of practical skills in undergraduate medical education - an explorative Delphi-Study. Dryad Digital Repository. 2016. DOI: 10.5061/dryad.q4sc8

---

Attachments

Attachment 1	Online attachment – only in german (Anhang.pdf, application/pdf, 177.82 KBytes)