Background
The Ministry of Education of Thailand has launched the Qualifications Framework for Higher Education, or TQF: Hed, to ensure the quality of graduates from higher education institutions (Maneerat et al., 2015; Tamronglak, 2020). The Royal Thai Air Force Nursing College (RTAFNC) was established to produce nursing personnel to provide health services to civil servants’ families of the Air Force and the public (Royal Thai Air Force Nursing College, 2018). Currently, the RTAFNC is an associate institution of Mahidol University, focusing on producing registered nurses with knowledge that can facilitate internal changes and be transferred to nursing practice skills (Dhabdhimsri et al., 2019; Khonhkhoontot et al., 2020; Pungsawang & Booncharoenpanich, 2017). According to the TQF, the expected characteristics of nurses are reflected in the form of work behavior in various roles and situations based on knowledge, skills, abilities, and personal characteristics, consisting of six aspects of learning, such as ethical and moral, cognitive, or nursing skills (Tamronglak, 2020).
“Moral and ethical” refers to behaviors that graduates demonstrate to manage moral, ethical, and professional problems by using discretion on values. This aspect includes consideration of others’ fundamental values consisting of moral and ethical behavior. “Knowledge characteristics” refer to behaviors that graduates demonstrate in knowing about nursing and related sciences extensively and systematically. “Cognitive skills” or intellectual attributes refer to behaviors that graduates illustrate using knowledge and analytical thinking, including problem-solving. “Interpersonal skills and responsibility” or social skills include communicating with others through verbal and nonverbal communication, teamwork, and maintaining a positive attitude. “Numerical analysis skills and communication” refer to behaviors that demonstrate ability in numerical analysis, mathematical, and statistical techniques, including verbal and non-verbal communication and using information technology. Finally, “nursing practice skills” refer to behaviors individuals express in nursing practice and the ability to execute tasks (Thai Ministry of Education, 2019).
The Royal Thai Air Force Nursing College is primarily responsible for providing education in the nursing science curriculum by producing quality graduates according to specified goals (Royal Thai Air Force Nursing College, 2018). There must be a practical and continuous assessment of the quality of graduates annually. In operating the TQF curriculum, one instrument was used to evaluate the graduates’ learning outcomes. The 43-item instrument was developed for the first version of the curriculum (Pungsawang & Booncharoenpanich, 2017). This instrument has been used to measure graduate outcomes in six domains since 2014 (Tamronglak, 2020). However, some items were complex sentences or redundant with others, confusing stakeholders or graduate users in responding to the instrument. For example, in ethical and moral development, one item asked, “Do your job with honesty and integrity, do not harm others and take care of yourself, and determine to pursue a career to take care of yourself and your family on the right path,” which is complicated and uses terms like honesty, integrity, and pursuing a career on the right path simultaneously.
Additionally, to our knowledge, the construct validity has never been confirmed. Thus, the instrument needs modification, and its psychometric properties must be examined for construct validity and reliability (Jinruang et al., 2018; Thai Ministry of Education, 2019). The study results will benefit the nurse educators in charge of the curriculum in applying the instrument as a guide in developing future outcome-based education curricula for nursing. A sound TQF: Hed instrument can have significant social value in education development, aligning better with educational objectives and curriculum standards and ensuring the relevance of the instrument while contributing to the improvement of educational programs.
Methods
Study Design
This study followed guidelines outlined by DeVellis (2017), Waltz et al. (2017), and Nunnally and Bernstein (1994), comprising two phases: instrument modification and psychometric testing.
Samples/Participants
The target population consisted of graduate users/stakeholders who are registered nurses working in The Royal Thai Air Force Hospital. Research samples were obtained using a stratified random sampling method. Selection criteria included registered nurses working with graduates for at least one year. The researchers calculated the sample size based on the number of indicators, using a ratio of 1:10. The TQF: HEd instrument contained 28 indicators, so 280 plus 28 (10 percent), totaling 308, was considered (Hair et al., 2014). The stratified sampling was conducted in proportion to each group’s population size. The population included 630 nurses: 40 head nurses, 158 charge nurses, and 432 nursing staff members. Subsequently, the research samples comprised 20 head nurses, 77 charge nurses, and 211 nursing staff members, respectively. The researchers then employed a simple random sampling method from the name list to recruit participants. Eventually, all 308 participants provided research data, resulting in a response rate of 100%.
Instrument Modification
The TQE: Hed was developed by Pungsawang and Booncharoenpanich (2017). This instrument was derived from the original tool used to measure Air Force Student Nurses’ learning outcomes. The initial version comprised 43 items on a five-point Likert scale (1 = strongly disagree to 5 = strongly agree) used to assess learning outcomes, demonstrating a Cronbach alpha reliability 0.97. Subsequently, the instrument underwent modifications following a series of steps based on questionnaire principles to enhance its efficacy (Mason, 2005; Sperber, 2004; Stewart et al., 2012).
The CVI was evaluated by a panel of five experts with extensive experience in graduate evaluation and the TQF nursing curriculum for over ten years. Among them, three experts specialize in curriculum development and evaluation, while two are responsible for program development and student characteristic assessment, according to the Royal Thai Air Force Personnel (Waltz et al., 2017). As a result, thirteen items were eliminated due to redundancy and lack of relevance to the concept definition: two from ethical and moral development factors, four from knowledge, two from cognitive skills, two from social skills, two from analytical and communication skills, and one from nursing practice skills, respectively. Additionally, eight items were adjusted to enhance precision (see Table 1).
| Original Version | Modified Version | Method |
|---|---|---|
| Moral and Ethical Aspects | ||
| 1. Do her/his work even in difficult situations | Do her/his work with strong commitment, even in difficult situations | Modified |
| 2. Volunteer to do her/his work for society | Deleted (no relevance) | |
| 3. Do her/his job honestly and don’t take advantage of others, including copying other people’s work | Do her/his job with honesty and commitment to confidentiality | Modified |
| 4. Respect the rights of patients and families | Remain | |
| 5. Abide by rules and regulations and professional ethics | Remain | |
| 6. Conduct oneself in good morals and follow the rules and regulations of society | Deleted (redundant with item no.5) | |
| Knowledge | ||
| 1. Have knowledge and understanding of nursing science and other fields to be applied in planning nursing care for patients as holistic care | Remain | |
| 2. Have knowledge and understanding of nursing informatics | Recognize the importance of using nursing data for improving practice | Modified |
| 3. Have knowledge and understanding of nursing science and other fields to be used to promote health at the individual, family, and community levels | Have knowledge and understanding to promote well-being at the individual, family, and community levels | Modified |
| 4. Understand culture and global changes that impact the healthcare situation | Remain | |
| 5. Understand how to classify and categorize data in nursing practice and nursing administration | Remain | |
| 6. Apply knowledge in planning nursing care for specific patients | Deleted | |
| 7. Making a good decision in applying knowledge in practice | Deleted | |
| 8. Being able to learn by doing | Deleted | |
| 9. Being able to do Long-life learning | Deleted | |
| Intellectual Skills | ||
| 1. Analyze information thoroughly before deciding | Remain | |
| 2. Connect knowledge in various interdisciplinary fields and use it creatively | Remain | |
| 3. Present analytical results and creative, reasonable, and practical guidelines. | Propose creative, realistic, and practical nursing guidelines | Modified |
| 4. Solve problems using the scientific process consistent with changing health situations and contexts | Remain | |
| 5. Realize self-strength and weakness | Realize self-potential and weaknesses to proceed with self-development | Modified |
| 6. Ready to learn and adapt to every situation | Remain | |
| 7. Respect other’s rights and comment | Deleted | |
| 8. Being part of protection and solving problems of society | Deleted | |
| Interpersonal Skills and Responsibility | ||
| 1. Be able to work as a team in the role of leader and member of the nursing team | Remain | |
| 2. Demonstrate leadership for good change in the organization in a variety of situations | Demonstrate leadership for good change in the organization | Modified |
| 3. Have responsibility for the profession and society | Remain | |
| 4. Have responsibility for continuous self-development, professional development, organization, and society | Deleted | |
| 5. Find various options to solve problems and obstacles | Deleted | |
| 6. Working by considering goals, taking action, and integrity | Working by considering goals, taking action, and being responsible for results | Modified |
| Numerical Analysis Skills: Communication and Use of Information Technology | ||
| 1. Apply knowledge of mathematics or statistics in nursing practice appropriately | Remain | |
| 2. Read, analyze, interpret, and convey information appropriately | Remain | |
| 3. Use digital media and information for searching and presentation appropriately | Remain | |
| 4. Being able to communicate in Thai | Communicate effectively in Thai, including listening, reading, speaking, and writing | Modified |
| 5. Use statistics to reference information | Use statistics to reference information | Remain and deleted |
| 6. Being able to use information technology to search for information | Deleted (redundancy) | |
| 7. Being able to use technology to search for information and self-improvement | Use technology to search for information to proceed with self-improvement | Modified and deleted |
| 8. Use information technology, present, and communicate effectively | Deleted (redundancy) | |
| Nursing Practical Skills | ||
| 1. Understand how to use the science and art of nursing, including related sciences, to care for individuals, families, and communities | Practice by applying the science and art of nursing, including related sciences, for individuals, families, and communities with holistic care | Modified |
| 2. Practice in health promotion, prevention, direct care, and rehabilitation correctly and appropriately | Remain | |
| 3. Provide health advice and appropriately follow up on the patients’ and families’ service results | Remain | |
| 4. Nursing practice according to nursing standards with a volunteer spirit and a human heart | Remain | |
| 5. Practice nursing with consideration for individuals and cultural diversity | Remain | |
| 6. Nursing practice by considering the rules and professional ethics | Deleted (redundancy) | |
After reviewing the items with ten registered nurse participants, two were edited to clarify their meaning. Thirty items were subjected to item analysis to identify item correlation. Two items were deleted due to inter-item correlation exceeding 0.8: items 5 and 7 in the numerical analysis skills communication and use of information technology aspect. Consequently, the researchers retained 28 items for inclusion in the construct validity examination.
Data Analysis
Confirmatory factor analysis (CFA) was used to validate the construct and was analyzed using Mplus software (Kelloway, 2015). Skewness and kurtosis, Bartlett’s test of sphericity, and KMO results met the normality criteria. The CFA model had congruence with goodness-of-fit (Hair et al., 2014; Kelloway, 2015). Content validity and reliability were assessed. The Scale-Content Validity Index (S-CVI) should be at least 0.8. All items met the Item-Content Validity Index (I-CVI) minimum criterion, ranging from 0.80 to 1.00. Additionally, Cronbach’s alpha coefficient for the six factors should be at least 0.8, indicating consistency of items in measuring the same construct (Nunnally & Bernstein, 1994).
Ethical Consideration
This research was approved by the review board of Bhumibol Adulyadej Hospital under number 86/65. The research was structured to safeguard the rights of the participants by providing the research objectives and procedures and enabling participants to make autonomous decisions. The research information was communicated, and participants were asked to consent to participate. Once participants understood and expressed willingness to partake in the research, they signed consent forms. Participants had the option to withdraw from the research without providing reasons. However, declining to participate would not affect the participants, and confidentiality of research results would be maintained.
Results
Characteristics of the Participants
The study involved 308 participants, who are registered nurses working with graduates, with an average experience of 9.26 years (± 8.14). This substantial level of experience indicates that the participants have a wealth of knowledge and expertise in nursing practice, making them well-suited to provide valuable insights into the assessment of graduate nurses' quality. Moreover, nearly all participants held a bachelor’s degree, comprising 93.6 percent, while 5.80 percent had a master’s degree, and 0.6 percent held a doctoral degree, respectively. Overall, the diverse educational backgrounds and extensive professional experience of the participants lend credibility to the study's findings and reinforce the significance of the results in assessing the quality of graduate nurses among Royal Thai Air Force nurses.
Content Validity
A panel of five experts assessed the CVI. For the modified version of the instrument, the I-CVI ranged from 0.80 to 1.00, indicating a strong level of agreement among the experts regarding the relevance of individual items to the measured construct. Additionally, the Scale-Content Validity Index (S-CVI) was calculated to be 0.88, suggesting a high level of overall content validity for the instrument as a whole.
Construct Validity
The 28 items underwent analysis using CFA. The results revealed that Bartlett’s test of sphericity was significant (χ2 = 7571.104, p <0.001), and the KMO was 0.96 (Hair et al., 2014). Factor loadings ranged from 0.684 to 0.878. Initially, the first model yielded unacceptable goodness-of-fit statistics, such as χ2 >0.05, χ2/df >2.858, CFI and TLI <0.95, and SRMR >0.05, respectively. However, after adjusting some error covariances, the modified model demonstrated a good fit. The Chi-square value, along with the model fit indicators and errors, was deemed acceptable (Kelloway, 2015) (refer to Table 2). The standardized-modified model of the final model is illustrated in Figure 1.
| The Goodness of Fit Statistics | Criteria | TQF: HEd model |
|---|---|---|
| χ2 | - | 265.861 |
| df | - | 233 |
| p-value | >0.05 | 0.068 |
| χ2 / df | <2.00 | 1.141 |
| CFI | >0.95 | 0.996 |
| TLI | >0.95 | 0.993 |
| RMSEA | <0.05 | 0.021 |
| SRMR | <0.05 | 0.033 |
Reliability
According to the findings, the TQF: HEd had a scale reliability of 0.93, indicating high internal consistency (Table 3). Additionally, the six factors within the instrument each obtained high reliability scores: 0.90, 0.90, 0.92, 0.85, 0.88, and 0.91, respectively (Nunnally & Bernstein, 1994). These scores suggest strong internal consistency among the items within each factor. Furthermore, the inter-item correlation ranged from 0.333 to 0.689, while the item-total correlation ranged from 0.580 to 0.681. These correlations indicate a moderate to strong association between individual items and the overall scale score, further supporting the reliability and validity of the instrument.
| Factors | Indicators | Cronbach’s α |
|---|---|---|
| Factor 1 (Moral and ethical aspects) | 4 | 0.90 |
| Factor 2 (Knowledge) | 5 | 0.90 |
| Factor 3 (Intellectual skills) | 6 | 0.92 |
| Factor 4 (Interpersonal skills and responsibility) | 4 | 0.85 |
| Factor 5 (Numerical analysis skills: communication and use of information technology) | 4 | 0.88 |
| Factor 6 (Nursing practical skills) | 5 | 0.91 |
Discussion
Summary of the Findings
The study aimed to modify and assess the TQF: HEd instrument for measuring the learning outcomes of Royal Thai Air Force Nurses. The original 43-item version underwent content modification, followed by cross-validation of the resulting instrument’s psychometric properties (Stewart et al., 2012). The findings indicated that the modified TQF: HEd is a reliable and consistent instrument suitable for evaluating the learning outcomes among graduate Royal Thai Air Force Nurses.
The 28-item TQF: HEd demonstrated construct validity with acceptable fit indices in factor analysis (Kelloway, 2015). The structural domain comprised six factors that had significant loadings (p <0.05). These six factors aligned with the concept definition of graduate outcomes of higher education in Thailand, encompassing ethics, knowledge, and nursing practice (Dhabdhimsri et al., 2019; Khonhkhoontot et al., 2020; Kongkar et al., 2021). Furthermore, the results were consistent with a previous study involving the use of the TQF: Hed scale in Thailand, indicating that the measurement model fits empirical data (Prasertsak & Trakoolchokumnuay, 2021).
In terms of scale reliability, the Cronbach’s α coefficient value exceeded 0.80, consistent with the consistency of the existing instrument (0.92-0.96) (Khamchan et al., 2022; Pansuwan & Pakanta, 2023; Wangruangsatid & Kaewsasri, 2022). Moreover, the results for each dimension ranged from 0.85 to 0.91, surpassing the recommended threshold of 0.80 and correlating with a previous study that demonstrated internal consistency of the instrument ranging from 0.83 to 0.94 ((Khamchan et al., 2022; Waltz et al., 2017).
Implications of the Study
The study’s implications are profound for both nursing education and practice. By successfully modifying and validating the TQF: HEd instrument, this research contributes to the advancement of assessment tools tailored specifically for evaluating the learning outcomes of Royal Thai Air Force Nurses. This achievement underlines the instrument’s potential to accurately and consistently measure the educational achievements of graduate nurses within this specific context.
Furthermore, the validation of construct validity, along with the identification of six distinct factors within the structural domain, offers valuable insights into the multidimensional nature of nursing education outcomes. This understanding can inform curriculum development, educational planning, and competency assessment strategies, ensuring that nursing education programs effectively prepare graduates for their roles in the healthcare system. Moreover, the study’s findings reinforce the reliability and validity of the instrument by demonstrating consistency with previous research in similar educational settings. This consistency enhances confidence in the instrument’s ability to yield dependable and meaningful results across different programs and contexts, facilitating its widespread adoption and application in nursing education.
From a practical standpoint, the confirmed scale reliability assures educators and stakeholders of the instrument’s internal consistency and reliability. This assurance is essential for making informed decisions about nursing education curriculum development, program evaluation, and quality improvement initiatives.
Overall, the implications of this study extend beyond the validation of a specific instrument; they highlight the importance of robust assessment tools in driving continuous improvement and ensuring the quality of nursing education and practice. By providing a validated instrument for assessing learning outcomes among graduate nurses, this research contributes to the ongoing enhancement of nursing education programs and, ultimately, the delivery of high-quality nursing care.
Limitations
This study was conducted in a single setting, potentially limiting its generalizability to evaluate nurse graduates across Thailand. To enhance the robustness and applicability of the instrument, it would be advantageous to test it in a larger sample size, involving stakeholders after the students have graduated for more than one year and including participants from diverse settings. This broader scope of testing would provide a more comprehensive understanding of the instrument’s effectiveness and suitability for assessing nursing education outcomes in various contexts.
Conclusion
The TQF: HEd instrument appears to be valid and reliable for measuring the six aspects of learning outcomes among Air Force Student Nurses. Nursing educators can easily and practically utilize the TQF: HEd to assess students’ learning outcomes and adapt the curriculum according to stakeholders’ needs. This process will facilitate graduates in performing their work with high quality. Evaluating students’ learning outcomes post-course completion can aid in revising the current curriculum to enhance teaching and learning methods. Additionally, educators responsible for the curriculum can utilize the results as background information when developing an outcome-based education curriculum for nursing comprising all six elements. As a result, nursing students will emerge as qualified graduates with proficient outcomes, particularly in the digital era.
Declaration of Conflicting Interest
All authors declared no conflict of interest in this study.