Improvement

Course Evaluations – Key 7

During a summer full of conferences, workshops, classes (both on-line and on-site), and numerous online discussion groups, lots of questions have been raised about how to structure course evaluation forms? While there are many good evaluation models available, here are the basic fundamentals that have worked well for the organizations where I have worked. This is part of Key 7 – Evaluations.

First, keep your course evaluations as simple as possible. I like to build a course evaluation using a 1 - 4 point scale. This forces participants to make a choice, to select above or below average. I have used the 1 – 3 and 1 -5 point scales but it allows the borderline evaluators to just be neutral by choosing the middle number. Neutrality isn’t all bad but when you are looking for what to improve, a neutral number doesn’t help much. And for those of you who debate whether using a 1 - 5 or 1 - 10 point scale, the only real difference that I have found are that participants will adjust their choice to reflect similar results of a slide between low, middle, or high scores.

There are 5 key areas where you need accurate feedback and information in order for you to know what, where or how you will want most to improve a course:
1. Content - is the course content useful to the participant?
2. Faculty’s Knowledge of the Subject – while this is often perception, did the faculty or instructor really knowledgeable about his/her subject?
3. Faculty’s Ability to Communicate – was faculty or instructor able to communicate their knowledge to the participants in the audience?
4. Quality of the training aids, handouts, etc. –were they applicable to the course?
5. Will the participant be able to apply the course to their job? Yes / No
Always allow the opportunity for open feedback from the participant as it can capture some amazing information occasionally. Questions about food, room temperature, arrangements, etc. should be left for the open general comments as generally the organization has little control over those issues and if it becomes an issue – it usually will appear in a general comments section. And yes, I am a believer that if you include food it can affect the results of the evaluations – poor food can lower the overall evaluations. Knowing that in advance – don’t provide bad food. And while the evaluators could be all over the scale in their final evaluations in the key areas providing you with detailed information about the course you will find it useful to ask a final general question about “How would you rate the overall program.”

If you have a way of collect the responses electronically, a tool like survey monkey - that could make your life much easier. What generally takes so much time related to surveys after the course is over is the summarizing the results. Anything you can do to minimize the staff time summarizing results is a plus.

My pet peeve is asking respondents questions that you know you are not going to use to improve the courses. Or worse still, requiring evaluation that goes straight to the storage file and then gets lost in a black hole. My final suggestion, keep your course evaluations as simple as possible.

Instructional Method: A Professional Action Plan

Learning contracts do not need to be complex. A number of years ago when I was involved with the AIA/CES Firm Leadership Symposium we needed to a simple method for an after-training-support-by-the-faculty. The process and tool we developed was simple. At the beginning of the workshop we would spend a few minutes explaining that each participant would be expected to identify at least one goal that they would like to accomplish after the workshop was over and they returned to their work environment. The action plan exercise was always planned at the end of the workshop so should that should they choose, the participants could include some of what they learned during the workshop into their action plan.

First we would discuss the purpose of the action plan. We would provide each participant with a Leadership & Learning: Professional Action Plan worksheet. The worksheet was intended as a simple structured outline for developing an individual action plan. Each participant was expected to identify at least one goal or action that they wanted to complete. The Leadership & Learning: Professional Action Plan required that the participants respond to 7 questions:

What is the goal?
What are the strengths related to achieving the goal?
What obstacles are we likely face?
What opportunities would likely be present?
What resources would they need?
What action steps are needed to complete the plan?
What were the related timelines?

First the participants would work to complete their own action plans. Additional time was then set aside to work in pairs - sharing with each other their goal and how they intend to accomplish it. At the end of the exercise participates exchanged POC information and committed to contacting each other after 30 days and again after 60 days.

The group faculty member or facilitator can become as involved after the event as appropriate. For those who did make contact at the 30 day mark, most went on to complete their goals. This process can be accomplished on-site, on-line or as a blended approach. I have since used the action plan approach successfully at the executive, manager and supervisory levels, and in both the private and public sectors. Hope this provides you with enough information. If you would like a free WORD copy of the worksheet just contact me directly at tlowther7@gmail.com.

How to Analyze a Case Study

Photo by Thom Lowther

Most of you who know me well know that I believe passionately in the use of cases studies as a learning tool. Years ago I discovered a excellent tool for analyzing case studies in the "Handbook for Training and Development" published by ASTD. I share below a simplified version of tool. I have used it often - in professional firms, in associations workshops and in college classrooms.

When analyzing a case study, an orderly, step-by-step approach is helpful. It is important to gain an appreciation of the overall situation initially, as well as to learn what information is contained in the case. Therefore, it is suggested that the case study be skimmed first to gain this overall perspective. While or after doing so, jot down the key points and issues that come to mind, as well as your first impression of the problems, issues, and opportunities facing the company. Then read the case in detail, adding to and modifying your initial thoughts. Remember that not everything in the case is vitally important, nor is all the important information necessarily included. The case represents someone's (e.g., management's) description of the company and its situation - it is up to you to probe deeper, sort and shift things out, and acquire additional information. It is your responsibility to analyze and recommend alternatives and approaches to management.

The following guide may be helpful to you in your task:

1. Define the situation. What are the challenges, problems, potential problems, opportunities, and potential opportunities facing the company? Typically, the case will contain various systems you will have to diagnose. To do so, try and isolate the major issues facing the company and their causes. Keep in mind that there are likely to be sub and secondary issues, as well as related and perhaps extraneous issues described in the case. Your task is to assign priorities to the issues, focusing on the critical few.

2. Assemble and analyze the important facts (gleaned from the case) which bear on the situation.

3. Specify important information that is needed but not included in the case. Determine whether or not it is available elsewhere. If available, acquire about it.

4. Make assumptions! For important information that is not available from the case or elsewhere, make logical assumptions as to what it might be. State these assumptions.

5. Draw conclusions Based on your analysis, information, and assumptions.

6. Determine alternatives and their likely outcomes. What are the major alternative actions open to the company, and what is likely to happen if each is adopted? Evaluate each.

7. Make recommendations. Based on your analysis, what do you recommend to management and why? Be prepared to defend your recommendations under critical questioning by the instructor and the class (the types of questions which might be posed by the company's management and other stakeholders).

8. Prepare an implementation plan. How should your recommendation be implemented, by whom? and in what sequence (short-term versus long-term actions). Where will the resources come from?

9. Prepare contingency plans. What do you recommend if your suggestions do not work as anticipated, or if certain external or internal conditions change?

Continuing Professional Education Audit Options for Associations

Certificates: A growing trend

There is a variety of approaches to providing continuing professional education quality assurance (QA) or compliance audits of association’s members. Regardless of whether the intend is to meet internal association education requirements, state mandatory continuing education (MCE) licenses related requirements, or continuing education requirements to maintain a specific skill certification. Below I will outline three approaches to conducting such audits or reviews from the auditing organizations perspective.
1. Professional Member Solely Responsible

The simplest approach for an association is to place full responsibly of compliance on the individual member. The member is responsible for everything relate to compliance. The member’s responsibilities would start from taking and completing the appropriate coursework and obtaining proof of passing the course requirements successfully at an acceptable level and in a timely manner. The responsibility of maintaining accurate records and reporting results to meet related requirements also becomes the full responsibility of the individual. Like taxes, there is generally a compliance time period that all records need to be maintained.

In this model the association only requests documentation from the individual member under extreme situations. Examples may include, but not be limited to a complaint or charge of fraud or incompetence by a client or customer. Another example, the individual member might be charged with a related legal violation or a professional ethics violation. Request of the individuals related continuing professional education documentation may be a required part of their defense. In this model a special review panel should be appointed to review and verify the documentation.

2. Blended Responsibility Model
Another approach would still require that the member be responsible for maintaining all continuing professional education documentation related to their meeting the association’s and/or certification requirements.
This model requires commitment and dedicated resources on the part of the association as they take a more systematic approach. This model requires that a small percentage of the members be audited on a regular pre-determined basis (5 - 20%). The association needs to commit at least a part-time dedicated reviewer that will be responsible to review and verify the documentation. A special audit/review task group should be appointed to establish guidelines and a review process policy. They should also act as a final decision making body for all disputed audits outside of a legal system. The established review process needs to be published and made available to all participants.

3. Association Commitment Model

An extensive association commitment approach should include a blended approach to records maintenance. While the responsibly of compliance falls on the member, course content and delivery should be a role that the association is at least involved with supporting and monitoring. The record keeping in this model becomes a shared approach.

Through an automated system it would be possible to offer a full menu of services. This could includes a selection of courses from pre-approved course content providers or listed options of alternate externally approved methods of obtaining the appropriate skills and knowledge. An automated records system can be monitored by the association. Records for members would include appropriate completed coursework that is maintained and monitored during the compliance time period. This approach also allows the association to provide and ongoing audit and review process towards a 100% compliance rate.

Similar to the Blended Responsibility Model an audit/review standing committee should be appointed to establish guidelines and a review process policy. They should also act as a final decision making body for all disputed audits outside of a legal system.

Depending upon the size of the association and the number of members involved, this model would require full time staff dedicated as reviewers responsible to review and verify the documentation. And depending upon the commitment of using an automated system, the service could be either in-house or contracted out. Appropriate staff to support either effort would be required.

The International Learning Unit (ILU) holds new relevancy in the age of the Internet

Flickr Photo by by epSos.de

As almost everyone who works in the field of continuing education, professional development or organizational development knows - the continuing education unit (CEU) was the model for measurement most frequently used during the 20th century. This time based model worked well in the industrial age. Unfortunately the CEU model hasn’t changed, nor has it kept up to date while our global cultures have changed. So what can improve, update or replace the CEU? The International Learning Unit (ILU) developed in the 21st century to address the changing needs of continuing and professional education in the age of the Internet.

According to Wikipedia the “International Learning Unit (ILU)” is an outcome based measurement of learning designed for lifelong learning activities. The ILU is a competency based approach to measuring learning education courses. The ILU is an alternative measurement and standard to the time based measured courses. The ILU measurement can be used to provide evidence of completion of continuing education requirements mandated by certification bodies, professional societies, or governmental licensing boards.

The ILU was designed for the needs of the 21st century. Instead of recording seat time in class the ILU measures the knowledge and skills of the participants. The ILU adopts well to web based education that has begun to warp, change, and reshape time as it relates to learning. Unfortunately all of the rules and laws are currently written measuring the professionals success in time spent at the event. Does this still make sense in your continuing education or certification program to measure seat time – when there might not even be a seat used? Isn’t it time to adapt the International Learning Unit?

Evaluation and Improvement – The 7th key to quality continuing education for product manufacturers

For the product manufacturer the 7th key towards providing quality continuing education is to evaluate each course upon completion and use the results to continually improve the course. Michael D. Perry, Hon. AIA,is the Vice President Government Sales and STAMP and is currently with Simon Roofing/SR Products. Michael has been a long time advocate of holding the product manufacturer accountable to the highest standards of developing and providing continuing professional education to design professionals. In an effort to improve continuing education quality standards for the design industry Michael was the first to support the AIA Continuing Education System Award for Excellence. He was also responsible for launching the AIA/CES Firm Symposium which assisted firm leaders in the establishment of industry standards for internal professional development programs that relied heavily on support from product manufactures. According to Michael, “professional development and continuing education is all about constant improvement. The only way to measure the impact of the message you are delivering is to conduct an evaluation at the end of a program. This process is essential not only for the content of your message but also for the quality of the presentation. Without good feedback from the course attendees you will never know if the information is beneficial and if your methodology of delivering the information is leaving the audience at the altar.”

David deBear, CTC, CSI, is the National Construction Service Manager and works for Custom Building Products a product manufacturer and a long time registered continuing education provider. Under David’s leadership Custom Building Products was a multi – time winner of the AIA/CES Award for Excellence. When I recently asked David to reflect back on contributing factors to winning the award he share this story with me related to what he called a more technical related course. David stated that he received one evaluation where the participant thought the topic was relevant but that the course was confusing. David indicated that the course had been receiving mixed reviews and not consistently delivering the intended message. This one evaluation was more critical and more specific. In summary the participant stated that they could not follow the story the topic was covering and that it was confusing. With the specifics provided by this particular participant as well as comparing statements from previous evaluations, David realized that the company needed to bring in a curriculum specialist to restructure the story line. The curriculum specialist reorganized the content and to follow the story line so that it was not confusing. The curriculum specialist added a summary of key points so regardless of the knowledge level of the participant, information was received. David said that after the adjustment, participant satisfaction with the course increased dramatically.
A Product manufacturer that offers continuing education in any industry needs to build a system that continually evaluates all of their courses. Catch the evaluations immediately on-site, do not rely on the internet for feedback for on-site courses. Focus on items such as content, instructor delivery and methodology. Use the information you collect to continually improve your program and courses and to build upon your reputation as a product manufacturer that offers reliable quality education.

Evaluation and Improvement – The firm’s 7th key to quality continuing professional education

The question that I like to raise, "what does your firm do with the information collected after having employees evaluate each course upon completion?" Does your firm require course evaluations from each participant before certificates or credit is awarded? Does your firm use a competency based learning approach that ties into performance and bonus pay? Does your firm use a systematic approach to annual or semi-annual review of the overall education program? Does your firm integrate the results of the evaluations into the firm’s business plans? In order to establish a quality education professional education program you should have answered either “yes,” or answered, “We are working on all of these questions.”

It is amazing that firms spent time and effort to provide some type of an evaluation form for their employees and clients at the end of a training session and then do not use the results for improvement of the either the courses, instructors, staff performance, or business improvement. The opportunities for improvement within the firm are great. The collected information can be used to improve future course offerings, content, instructors, and delivery methods. Forward thinking firms can use the results to improve their firm’s product or services. They can also improve their firm’s marketing and promotion by having clients participate in select sessions.

Your firm could build a system that continually evaluates all of the courses and the employees upon completion of the courses. Curriculum could be developed from the results of the evaluations. Faculty or instructors could be developed from a selection process involving high performing employees. Performance improvement could be measured, evaluated and adjusted according to the business needs of the firm. Internal instructors, staff and human resources/training department staff could receive instantaneous feedback on what needs to improve, and maybe even how to improve. Use the information you collect to continually improve your continuing professional education courses and your business. Who knows, it might even help in the firm’s recruitment efforts when emerging professionals discover that the firm is serious about professional development.

Commitment - A key to unlocking the secrets to establishing a quality education program

The first key to unlocking the secrets of building a quality education program within your organization is to gain commitment and support internally. It must start at the top of the organization. But in order for your education program to achieve excellence it will need more than just senior level support. For the program to be truly outstanding it needs senior management’s involvement in creating and sustaining the organization’s educational direction, performance, and focus. This includes the development of a strategic process that ties education into the overall business plan of the organization. The education program will not work effectively or efficiently if it is just an afterthought or an add-on program. Like any business, to achieve quality you need to think strategically which includes a long term investment of time, involvement of people and investing of dollars. To be successful the education program must be integrated within the working foundation of the organization. In order for the strategic process to be maintained and succeed there must be a buy-in, a commitment at all levels of the organization. This means there are no lasting quick fixes.

Leadership Involvement summary includes:
1. Senior leaders set direction and seek future educational opportunities.
2. Leadership addresses performance expectations and long-term commitment.
3. Leadership is involved in the education program’s overall performance.
4. Leadership takes into account the educational needs and expectations of all key personnel.

Demonstration of these elements by the senior management and you are taking your first steps to building a successful program.

Pages