Evaluation

Catalog of Available Courses and Workshops

Below are the titles of seven online courses and/or workshops that are available from Lowther7, LLC Catalog descriptions, learning objectives, and details for each are provided separately following this listing.

Creating Successful Talent Within Your Firm

Available online or by appointment.

Embracing Sustainability in the Workplace

Online only - Instructor-led.

Simple LMS for Firms and Associations

Available online or by appointment.

Cyber Security for Small Businesses

Available online or by appointment.

Overview of Managing Projects

Available online or by appointment.

Developing Online Courses

By appointment only - Instructor-led.

Introduction to Digital & Social Media Marketing

By appointment only - Instructor-led.

Contact us about your workshop questions today; we're happy to help!

Gamification as a Situational Learning Tool

Photo by by azwaldo

The use of games or gamification for learning enhancement is not new in education. During the past few years however, there has been a renewed interest in gamification due largely to the new technologies that has become available. If you Google “gamification” it displays more than 700,000 results. Unfortunately too many people create educational games so that they can demonstration a technology rather than because it is the correct tool to improve or increase knowledge or a competency. Before selecting any delivery tools consider context and learning situation.

Working with several different organizations this past fall I realized the term gamification has very different meanings to different people. So for those of you reading this blog let’s establish a common definition used by Wikipedia. Gamification is the use of game thinking and game mechanics in non-game contexts to engage users in solving problems and increase users' self-contributions. Gamification has been studied and applied in several domains, with some of the main purposes being to engage, teach, entertain, measure[, and to improve the perceived ease of use of information systems.”

In the January 2015 issue of Chief Learning Officer is an interview with Jake Orowitz, Head of Wikipedia Library. In the interview Orowitz explains how Wikipedia uses gamification for situational learning to onboard volunteers, sharing the process related to editing material.

There are several interesting business case studies that use Gamification to enhance learning. For Microsoft the situation was to create a bond between the consulting business’ senior managers and to use the opportunity for content delivery and learning, bringing management up to date on the vision, financial results and strategy for the year. A full gamification solution considering context and situations was designed to motivate participation in the event, measuring engagement with the content presented and creating team spirit within the ad-hoc teams formed during the process. As a part of the process the tools to deliver the content were selected using mobile phones and tablets.

Another situation called for improving a course designed for those learning how to specify building materials for the new LEED MR Credit: Building product disclosure and optimization credit, under the Health Product Declaration (HPD) option. A collaborative team between Expedition21Media.com, Lowther7, LLC, and GreenCE was created to meet the challenge. It was determined by the team that a good way to increase learning and have participants better demonstrate competency was to imbed a mini-game in the course at a point after students learned how to specify building materials. To see the results for yourself play the free version of the LEED Materials Credit mini-game!

For the last three decades the popular workshop, the Accounting Game was offered by Educational Discoveries, Inc. and Professional Training International. The situation called for assisting non-CPA’s to understand basic accounting and balance sheet practices. The one day, on-site workshop used a simple lemonade stand business simulation format.

As I stated at the beginning of this blog post, the use of games or gamification for learning is not new to education. One of my first graduate courses was how to create and use games to promote learning, develop skills, and improve competencies. Kevin Werbach and Dan Hunter have written a book entitled, For the Win: How Game Thinking Can Revolutionize Your Business (Wharton Digital Press, 2012).

Through Wharton – University of Pennsylvania and Coursera, Kevin Werbach, offers the free course, Gamification. It is the application of game elements and digital game design techniques to non-game problems, such as business and social impact challenges. This course teaches the mechanisms of gamification, why it has such tremendous potential, and how to use it effectively.

Building a Design Firm's Professional Development Program -€“ Evaluation and Improvement

Photo of an evaluation form

This final segment of an eight part series covers evaluation, feedback, and continuous improvements. As before, I have assembled requirements from various award programs that appear among several prestigious organizations with consistency. My intent here has been to provide a self-assessment tool that can be used to help improve and more effectively manage a firm's professional education department.

All successful programs include an evaluation and feedback process. A system should be established that will evaluate each course, service or product against (Key 3) measurable short and long-term educational goals using performance projections. Don'€™t collect data just because you can. Collect what you need to help make informed decisions. And if you collect it, don'€™t ignore the information and let it collect dust. Use the information to continually improve your program, build your reputation as a quality organization, and become more profitable.

KEY 7: Evaluation and Improvement

This section provides a list of award winning recommendations for the areas of educational evaluation and program improvement. For each section below there is and established process.

Selection of Information and Data Collection
1. Determine what program evaluation information should be collected, maintained, and reviewed.
2. Addresses the methods used to evaluate the quality of the education program.

Evaluation and Review of Educational Performance
3. Determining comparative data to be used to measure performance.
4. Evaluation of the educational system with identified areas for improvement.
5. Determining how learning activities reach their stated objectives.

Education-Specific Results
6. Evaluate performance results for education services, programs, certification, and licensure compliance.
7. Using information to improve program effectiveness.
8. Keeping current with the changing educational needs of the audience.

Accessibility and Complaint Management
9. Providing access and information to participants who seek assistance or voice complaints about the educational activities.
10. Ensures that complaints are resolved effectively and promptly.

Feedback and Continuous Improvement
11. Insures information is and data shared and reviewed by leadership, and appropriate committees and individuals with the expectations of continual improvement.

Sources:
Using the Baldrige National Quality Award and IACET as models, a special task force created the AIA/CES Award for Excellence for The American Institute of Architects, Continuing Education System. This program was used as a cornerstone for building a national continuing education program that shaped education offered in the Architecture, Engineering, and Construction (A/E/C) industry. Today, other learning and development award programs such as the ASTD-Awards/Best-Awards and the Chief Learning Officer, CLOmedia Awards are also being used to elevate the practice learning and development.

Learning Objectives Simplified: Check out the New Bloom’s Taxonomy Tool

Candle Flame

The tool is simple, easy to understand, and easy to use. If you are the course designer, a trainer, an instructor, or the firm's Learning and Development Coordinator, Manager, Director or the CLO - this tool will make your professional life a little easier. If only this tool had been available during the past 30 years.

I would like to thank the Center for Excellence in Learning and Teaching (CELT) at the University of Iowa for posting on their website the Model of Learning Objectives. This model was created by: Rex Heer, Iowa State University.

Sharing this tool with my professional peers who are working in the A/E/C design industry, this is probably the best gift I can offer for the New Year. Try it for yourself; I think you will like it.

Taxonomy for Learning, Teaching, and Assessing: A revision of Bloom’s Taxonomy of Educational Objectives.

If you have trouble accessing the interactive Flash-based model the content is available in a text-only table.

Course Evaluations – Key 7

During a summer full of conferences, workshops, classes (both on-line and on-site), and numerous online discussion groups, lots of questions have been raised about how to structure course evaluation forms? While there are many good evaluation models available, here are the basic fundamentals that have worked well for the organizations where I have worked. This is part of Key 7 – Evaluations.

First, keep your course evaluations as simple as possible. I like to build a course evaluation using a 1 - 4 point scale. This forces participants to make a choice, to select above or below average. I have used the 1 – 3 and 1 -5 point scales but it allows the borderline evaluators to just be neutral by choosing the middle number. Neutrality isn’t all bad but when you are looking for what to improve, a neutral number doesn’t help much. And for those of you who debate whether using a 1 - 5 or 1 - 10 point scale, the only real difference that I have found are that participants will adjust their choice to reflect similar results of a slide between low, middle, or high scores.

There are 5 key areas where you need accurate feedback and information in order for you to know what, where or how you will want most to improve a course:
1. Content - is the course content useful to the participant?
2. Faculty’s Knowledge of the Subject – while this is often perception, did the faculty or instructor really knowledgeable about his/her subject?
3. Faculty’s Ability to Communicate – was faculty or instructor able to communicate their knowledge to the participants in the audience?
4. Quality of the training aids, handouts, etc. –were they applicable to the course?
5. Will the participant be able to apply the course to their job? Yes / No
Always allow the opportunity for open feedback from the participant as it can capture some amazing information occasionally. Questions about food, room temperature, arrangements, etc. should be left for the open general comments as generally the organization has little control over those issues and if it becomes an issue – it usually will appear in a general comments section. And yes, I am a believer that if you include food it can affect the results of the evaluations – poor food can lower the overall evaluations. Knowing that in advance – don’t provide bad food. And while the evaluators could be all over the scale in their final evaluations in the key areas providing you with detailed information about the course you will find it useful to ask a final general question about “How would you rate the overall program.”

If you have a way of collect the responses electronically, a tool like survey monkey - that could make your life much easier. What generally takes so much time related to surveys after the course is over is the summarizing the results. Anything you can do to minimize the staff time summarizing results is a plus.

My pet peeve is asking respondents questions that you know you are not going to use to improve the courses. Or worse still, requiring evaluation that goes straight to the storage file and then gets lost in a black hole. My final suggestion, keep your course evaluations as simple as possible.

How the CEU and the ILU can work together?

Flickr photo by The Library of Congress

For organizations that deliver continuing education consider using both the Continuing Education Unit (CEU) and the International Learning Unit (ILU). The CEU has been widely used for several decades. While the use and understanding of the CEU has become diluted by the many who do not fully understand the structure and intent, the formal CEU - next to time - is the primary benchmark used to measure continuing education courses and programs. The two, time and the CEU are interlinked but not always interchangeable. Subtle interpretations of how time is counted can affect the number of CEU credits awarded - one example, the 50 minute hour.

The ILU is only been in use since the early 2000’s but like the CEU, the ILU also measures continuing education courses and programs. Both the CEU and ILU records are used to provide evidence of completion of continuing education requirements by agencies and institutions. The CEU and the ILU require that courses taught use their designated standards. For the CEU there is a fee associated, for the ILU registration is required but no fee. Both CEU and ILU require learning objectives and qualified instructors to deliver course material in an appropriate format. And the CEU and ILU both use units of .1 to designate single units of learning. Example both would list 5 units of learning as .5 or 10 units of learning as 1.0.

Employers and faculty are interested in knowing that in a learning situation, substance and retention is more important than time. Where, when or how a person learns is not as important as what they learned and that they learn. The ILU requires an outcome based competency with a minimum result of at least 80% or better. This means that there are results available upon completion of the course. Some CEU providers include some form of test or demonstrated competency as part of their educations courses or programs. For other on-site training, classroom, face-to-face training, and some forms of eLearning a test or a demonstrated competency segment could be included for CEU and ILU credits.

We know that the CEU measures the length of the class, the time from start to finish. As delivery methods for eLearning continue to expand however, the time the participant spends in the process of learning loses importance. By focusing more on comprehension, competencies, outcomes, and retention those providers using the CEU could improve their courses and programs by adopting the additional ILU elements of testing and/or demonstrated competency measures. As education providers adapt to the new technologies and begin to use more e-learning media that are difficult to measure in time, the courses and programs can retain the focus on learning using the ILU standards.

Evaluation and Improvement – The 7th key to quality continuing education for product manufacturers

For the product manufacturer the 7th key towards providing quality continuing education is to evaluate each course upon completion and use the results to continually improve the course. Michael D. Perry, Hon. AIA,is the Vice President Government Sales and STAMP and is currently with Simon Roofing/SR Products. Michael has been a long time advocate of holding the product manufacturer accountable to the highest standards of developing and providing continuing professional education to design professionals. In an effort to improve continuing education quality standards for the design industry Michael was the first to support the AIA Continuing Education System Award for Excellence. He was also responsible for launching the AIA/CES Firm Symposium which assisted firm leaders in the establishment of industry standards for internal professional development programs that relied heavily on support from product manufactures. According to Michael, “professional development and continuing education is all about constant improvement. The only way to measure the impact of the message you are delivering is to conduct an evaluation at the end of a program. This process is essential not only for the content of your message but also for the quality of the presentation. Without good feedback from the course attendees you will never know if the information is beneficial and if your methodology of delivering the information is leaving the audience at the altar.”

David deBear, CTC, CSI, is the National Construction Service Manager and works for Custom Building Products a product manufacturer and a long time registered continuing education provider. Under David’s leadership Custom Building Products was a multi – time winner of the AIA/CES Award for Excellence. When I recently asked David to reflect back on contributing factors to winning the award he share this story with me related to what he called a more technical related course. David stated that he received one evaluation where the participant thought the topic was relevant but that the course was confusing. David indicated that the course had been receiving mixed reviews and not consistently delivering the intended message. This one evaluation was more critical and more specific. In summary the participant stated that they could not follow the story the topic was covering and that it was confusing. With the specifics provided by this particular participant as well as comparing statements from previous evaluations, David realized that the company needed to bring in a curriculum specialist to restructure the story line. The curriculum specialist reorganized the content and to follow the story line so that it was not confusing. The curriculum specialist added a summary of key points so regardless of the knowledge level of the participant, information was received. David said that after the adjustment, participant satisfaction with the course increased dramatically.
A Product manufacturer that offers continuing education in any industry needs to build a system that continually evaluates all of their courses. Catch the evaluations immediately on-site, do not rely on the internet for feedback for on-site courses. Focus on items such as content, instructor delivery and methodology. Use the information you collect to continually improve your program and courses and to build upon your reputation as a product manufacturer that offers reliable quality education.

Evaluation and Improvement – An association’s 7th key to quality continuing education

Does your association have participants evaluate each of courses upon completion? Does your association provide evaluations because it is expected? Does your association require course evaluations from each participant before certificates or credit is awarded? Does your association have a systematic approach to annual or semi-annual review of the overall education program? In order to establish a quality education program you should have answered either yes, or working on it to all of these questions. And, if your answer is yes to any of these questions, what do you do with all of that information once it is collected?

I am amazed at how many associations go through all the time and effort to provide some type of evaluation form for participants at the end of the session then do little to even collect the results. At a minimum, the collected information should be used to improve the course content, format, instruction, delivery, and promotion. It should not be used just to determine if participants liked the food, liked the instructor, and that the temperature of the room was OK.

Your association should build a system that continually evaluates all of the courses upon completion. At a minimum you should share all of the information in a detailed summary with your education advisory committee, your faculty, and your staff. Use the information you collect to continually improve your program and courses and to build upon your reputation as an association that offers reliable quality education.

Closing the Loop: Evaluation and Improvement – the 7th key to quality continuing education

Flickr photo by azwaldo/3256080624

You are almost finished – but not yet. You have completed the assessment and followed through with your planning. You designed the curriculum and developed the course content. You developed your marketing plan and promoted the course(s). You finished up on the delivery of the course(s), so now what? You don’t just want a good program you want an outstanding program, built upon high quality course content and delivery. To do this you included an evaluation and feedback process. You build a system that continually evaluates all of the courses upon completion along with an operational work flow process for each year. Set up a system that will evaluate each course, service or product against (Key 3) measurable short and long-term educational goals with performance projections. Most of all don’t let this valuable information collect dust. I have watched too many organizations evaluate each of their courses because it was expected or required, and then do nothing with the information they collected. Share all of the information, in a summary and detail where appropriate with your education committee members, your faculty, and your staff. Use the information you collect to continually improve program and courses, to build upon your reputation as a quality organization, and to become more profitable.

“Where’s the data?” – The second key to unlocking the secrets of a quality education program

Photo by Dennis Crowley via Flickr

The question I always heard from superiors and peers, “Where’s the data?” I believe that the seeds for needs assessment should be planted with the establishment of the strategic goals. Early on the organization should develop a systematic approach for identifying and analyzing the educational needs that relate to the overall strategic plan. Planning and analysis are simultaneous and should be ongoing.

As defined by Wikipedia, Needs assessment is a process for determining and addressing needs, or "gaps" between current conditions and desired conditions, often used for improvement in individuals, education/training, organizations, or communities. The need can be a desire to improve current performance or to correct a deficiency.

For education and training assessments there are a growing variety of models to from which to choose. Select a model or a blend of models which most closely match your goals, operations, personnel and budget. Methods and techniques for gathering information can vary from formal focus groups, to telephone or mail surveys, to online surveys such as survey monkey. The intent should be to gather timely information to enable those in the organization to make smart decisions based upon relevant and appropriate information.

If the program is intended for internal use of the organization’s staff education then it is important to match the model to the organization’s culture, operational structure, and short and long-term education and professional development needs. The content could relate to technical, conceptual, and/or personnel related needs. It is important to focus on the details of professional staffs’ participation in the needs assessment process. Determine how the needs for the educational program and products/services are identified, how the programs are developed and designed to address those needs.

If the education program is intended for external use, the assessment should relate to the business needs, support the organizations need for delivering training; ensure training delivery design relates to customer’s needs; verifies effective performance; and provides guidance into the evaluation process.

Pages