CAREER CONNECTIONS TO TEACHING WITH TECHNOLOGY (CCTT)
EXTERNAL EVALUATION, YEAR 5
Karen C. Cohen
Karen C. Cohen and Associates
9 Cliff Road
Weston, MA 02493

1. ActiveClassroom
2. NDL Workshops, and
3. ActiveLesson

• Activities before, during, and following the training
• Impact of the training on the teacher participants
• Participants’ evaluation of aspects of the project
• Impact on teacher’s curriculum
• Units developed and/or revised
• Specific types of changes to their teaching resulting from participation
• Implementation and course change specifics
• Sustained Effects and Barriers to Sustained effects
• Impact on Students
• Impact on non-classroom activities
• Impact on Broader Community
• Barriers to implementation
• Time Spent related to the Course
• Demographic Information
• Suggestions and Comments

During Grant Years 3-5 several ActiveClassroom workshops were held for teachers ranging in length from ½ day to several days to introduce teachers to this new software utility developed primarily by a teacher and student from Mainland High School, Volusia County, Florida. In many ways like “Blackboard”, ActiveClassroom allows teachers to organize all of their teaching materials. They can post assignments, tests, class exercises, track attendance and grades, provide parents and students with information about what was going on in each class in general, access to a host of reviewed resources available on the internet, as well as to correspond with the teacher by e-mail. When used with all of its features, it is a marvelous resource. Dramatically positive results of ActiveClassroom on student attitudes, attendance, and increased in enrollment in Physics have been meticulously tracked by Cathy Colwell at Mainland High School, She developed and implemented ActiveClassroom for her Physics students, and this information has previously been reported and is available on the web. ActiveClassroom is being sold through C2T2 Educational Systems, Inc., a not-for-profit organization developed to sustain the viable products produced by the Grant. In 2002, the teacher’s union voted to adopt ActiveClassroom county wide in Volusia County, Florida—the first county to do so.

The Teacher Participant Telephone Survey was administered as a telephone survey to more than half of the participants on lists provided by hub-site managers from the five states in the project. We were provided with a list of about 70 names for ActiveClassroom, teachers who had attended some training and had had time to try to implement what they had learned. Each teacher was called at his/her school and asked to return the call to set and appointment for the lengthy (1 hour-1.5 hour) interview. Teachers were primarily interviewed at home, on their time. 36 interviews were completed. Many of the names provided could not be reached for a variety of reasons: during the past three years some teachers had moved on to other positions, some were on maternity leave, and some had personal tragedies during the time we collected the information.

ActiveClassroom workshop participants were contacted in April-May, 2001 at their school by telephone and asked to set up an appointment for a telephone interview. All responses were recorded by the interviewer, Jane O Coogan. Quantitative findings were keypunched and verified by Tradequotes, Inc. and statistical data analysis of the numerous quantitative findings was performed by Marcus Lieberman. (See Appendix B). Qualitative comments were organized and transcribed by Toni Rosenberg, (See Appendix C). The interim evaluation report is based on the work of these other people. We will use the same methodology on the other two major utilities developed by the Project, NDL Workshops and Active Lesson. Comparisons between and among these three major project outcomes along with cumulative findings will be presented in the final, Year 6 cumulative evaluation report of the CCTT project.

The findings here are presented in a narrative style. Full detail is found in the appendices for each survey question.

Activities before, during, and following the training. Most teachers said they were asked to use lecture notes or other handouts in preparation for the workshop or during the workshops along with other activities, usually some work with computers. During the ActiveClassroom workshops most of the people indicated they identified a unit they wanted to develop or identified other ways they anticipated incorporating project information at their home school. The next two most cited activities were preparing a project or problem to work on during the course and “other” activities, most often related to enhancing their technological skill and learning how to implement ActiveClassroom off site. The teachers said that their course experience most often included introduction to new teaching methods and new technologies.

By the end of the professional development, to be more effective in their teaching, the teachers most often said they needed more work with reading materials and problem sets. Following the workshops, most teachers indicated they either participated in informal group get-togethers or received technical assistance from the project staff.

After the course, most wrote that they communicated with other participants by e-mail. For those who did communicate with other participants, most indicated that their communications were sporadic.

When average scores were computed for numerous possible responses, participants gave the highest ratings to new or improved technological skills, increased motivation or stimulation for teaching excellence and new information about other resources for use in teaching. The lowest ratings were given to new or more in-depth knowledge of issues regarding females and minority students and new contacts with colleagues from other institutions. Only three people said they got no benefit from the program.

Average ratings were calculated for a variety of factors could contribute to what teachers felt had helped them gain from the ActiveClassroom training. Participants rated most highly the interactions with the instructors, content of the sessions, the experience of developing products or materials at the course and discussions on how they would use what they learned in their own classes. The lowest ratings were for preparation prior to the course and follow-up activities.

When teachers were asked “Did you get any benefit out of the program?” 30 responded Yes, 3 responded No, and 2 had no response. Typical comments were:

• "Yes. I created a web-site that I used immediately in my classroom. My students enjoyed it and it was easy to facilitate vocabulary for example—I put words on line and didn’t have to give them out in class”
• “Yes—an enormous amount. [ActiveClassroom] has allowed me to share lesson plans on-line with students’ families, with students who are absent, with students wishing to review materials in preparation for exams. It has allowed me to establish links to educational and scientific web-sites such as the human genome project and other current scientific areas of investigation.”
• “Yes—the benefit is good if you can get it all on the computer—benchmarks, state standards, homework in reading, math, language, science, social studies and spelling—daily update is very hard to do with so any subject areas. I love the idea: get parents on, get kids on, but it’s time consuming.”
• “Definitely. I’ve been a teacher for over 15 years and I’ve been doing ActiveClassroom for 2 years and it’s made me a better teacher, due in part to collaboration with colleagues across the country. It keeps me in tune with new technologies. ‘I’ve seen my students gain interest. They are more motivated. They prefer to work through the computer.”

The three negative comments all related to lack of access to technology and inability to use a home Macintosh computer to interface with the system from home.

Most of the participants indicated they revised one or more existing units. About half the participants developed or revised four (4) or more units. Half the participants developed their units in collaboration with others and half did not. More than half the participants described the units they developed as interdisciplinary. Of those who answered the question, one fourth wrote that their units received formal departmental or program approval.

Teacher comments were again positive. 25 (of the pool of 36) made and described specific changes they made in their curriculum or reaching. Eight did not.

The highest proportion of participants indicated that they had changed their teaching methods through online organization of course materials, introducing new experimental or lab techniques and introducing new content. The change in teaching methods and the new lab techniques were judged of major importance by the most participants. Of those answering, almost 90 percent of the teachers have taught one or more of the units they developed as a result of participating. Of those answering the question, almost forty percent have team-taught the material.

The average number of students who have completed these materials taught by each teacher/participant teacher was 208 and the average percentage of female students was 50.1%. Of those answering the question, almost 90 percent reported the courses or units are still being offered. The multiplier effect of ActiveClassroom throughout the project, therefore, was and will continue to be enormous.

Typical selected comments were:

• “It gave me a new window on how to teach. I became more student-centered instead of teacher-centered. I became more student group work oriented rather than teacher directed. I do the introductory lesson the first day and them I become a facilitator. The downside is that it takes more time and I see that we don’t cover as much curriculum. Content takes longer to cover, the students take their time. It’s longer but better learned.”
• “One of my rare split-level classes …was this past year. Having ActiveClassroom allowed me to conduct two separate levels efficiently in my classroom at the same time. One group looked at ActiveClassroom on the screen for their assignments while at the same time I can actively engage students in the other level in conversational French. Therefore, four of the four language components were met: speaking and listening and reading and writing. My motivation was primarily to get help on …managing… state required standard skills (FCAT) …and this I achieved much more easily through using ActiveClassroom. You go on line with your lesson plan and you call up the skills (FCAT) standards to plant the lesson around. We are required to teach those skills and to shop evidence that they have been taught. By doing this on line I can see what’s been done, as can my administrator and this cuts out an enormous block of time necessary to hand write and then submit all of the required paperwork.”
• “You go online with your lesson plan and you call up the FCAT standards to plan the lesson around. We are required to teach those skills and to show evidence that they have been taught. By doing this online I can see what’s been done as can my administrator and this cuts out an enormous block of time necessary to hand write and then submit all of the required paperwork.”
• “Allowed me to utilize the Internet and to use the Internet effectively.”
• “It made my advanced students more independent. They were responsible for checking the homework calendar and printing out all assignments. For my lower level students, I gave them basic instructions in Internet skills and basic computer skills. I have advanced students and drop-out prevention students.”

All of the numerous possible impact items had an average rating of 4.00 or higher on a scale of 1-5 where 1 = Substantially Worse and 5 = Substantially Better. The highest ratings were given to ability to use advanced technology (4.75) and problem solving skills (4.20). The lowest rating (still 4.00) was given to in-depth knowledge of subject area.

Selected typical verbatim comments on impact(s) on students appear below.

• “Today, my students can walk into class with no knowledge about a particular molecule or metabolic pathway and leave 45 minutes later with a beginning sophistication that would have been rare among college students before now. My students can walk into the classroom with no understanding of a photoreveriible phytochrome, interact with the computer for example with photo and visualize molecules such as photoreversible phytochrome. Whereas the in the past it would not have been subject matter in a high school setting—too boring and too abstract. Now they enjoy looking at molecules…[it is] awesome, basically.”
• “I would say a higher level of motivation and keeping a student on task longer during the class period. For instance, if they are working out of a text book their attention span is 15 minutes. It’s much longer attention span when they click back and forth between resources and labs and worksheets and doing so with others in teams of 2 or 3.
• “The students had a clearer understanding of the key concepts or goals being taught. They also developed more independent learning skills.”

The highest proportion of teachers reported sharing with other colleagues in their schools formally and informally. Most of the teachers reported attending professional meetings, seminars or workshops and participating in further professional development activities. Those who made presentations to local campuses or delivered papers at professional meetings rated their impact the greatest. Those who attended professional meetings and those who participated in further professional development activities rated their impact the least. Impacts ranged from application for and receiving grants for further development related to the project, to increased comfort and knowledge with technology, and becoming part of a peer-community for further communication, collaboration, and development. These findings are all based on the availability and mastery of technology provided by the Grant. (See Appendix C.)

Almost all the participants have shared information or skills they learned with colleagues either in their institution or in any other institution and they most often did that sharing through informal discussions with one or more colleagues.

Most of the participants reported that colleagues with whom they shared knowledge gained from the ActiveClassroom training. They reported that their own experience was expanded even further by their colleagues attending or joining the project.

Over two-thirds of the responding participants report encountering barriers to implementing what they learned from their involvement in this course. These barriers primarily involved time and financial resources to implement in a project that is “winding down.’

Just over two-thirds report that they did intend to develop new materials or units or modify existing materials or units despite these barriers.

• The average number of hours spent during ActiveClassroom training was 16.
• The average number of hours spent in additional development was 31.
• The average number of hours implementing ActiveClassroom was 25.
• The total average number of hours related to ActiveClassroom alone was 58.

• The average number of years that the participants had been at their school was seven years.
• Just over three-quarters of the participants taught at the high school level.
• Only about 1/7 of the participant were male.
• The average age of participants was forty-three.
• Only two participants described themselves as Hispanic or Latino.
• All but one of the participants described themselves as Caucasian.
• All of the participants who responded to the question indicated they were U.S. citizens.
• All but one of the participants wrote that they had no impairment or disability.

The National Digital Library (NDL) workshops were developed and presented by George and Carolyn Breaz. They were the Las Vegas, Nevada hubsite managers. They organized and presented in person and through on-line training, sessions for the other hubsite managers and for large numbers of participants in the project not only in Nevada, but throughout the five states in the project. They focussed both on training teachers, but also on training potential trainers of teachers, especially in their state, Nevada. Thus a large number of librarians, media specialists, and teachers were introduced in one or two day workshops. During the last two years of the project they presented these workshops at project meetings, at national regional meetings, and at national conferences. Finding that small group presentations worked best, since having internet access to the NDL resources were critical to workshops, they provided us with a list, again, of about 70 people who participated in these workshops.

Many could not be located at all since their schools had no telephones, but through using the same telephone survey instrument and process for contacting participants as for the ActiveClassroom project, we reached and completed interviews with 36 participants. The results are summarized in narrative fashion below, both quantitative and qualitative. Full quantitative findings appear in appendix D and full verbatim comments appear in Appendix E.

During the NDL workshops or “courses” participants reported on their primary activities.

During the workshop, all of the people indicated they identified a unit they wanted to develop or identified other ways they anticipated incorporating project information at their home school. All the respondents reported they did “other” activities. All but two indicated they prepared a project or problem to work on during the course.

The teachers said that their course experience most often included incorporating and synthesizing interdisciplinary content and teaching methods. A high proportion of teachers also checked the other areas.

In preparation for the course, were you asked to use any…

Most teachers said they were asked to use lecture notes or other handouts.

By the end of the professional development, to be more effective in your teaching, did you feel you needed more work with…

The most often cited areas that needed more work were reading materials and lecture notes.

In terms of follow-up activities, most teachers indicated they reviewed or site-tested materials or products developed as part of the workshop or participated in one or more informal group get-togethers.

After the course…

After the course, most wrote that they communicated with other participants by e-mail. For those who did communicate with other participants, most indicated that their communications were sporadic.

When average scores were computed for these responses, participants gave the highest ratings to new information about other resources for use in teaching and increased content knowledge. The lowest ratings were given to new or more in-depth knowledge of issues regarding females and minority students and new contacts with colleagues from other institutions. ALL (100%) of the respondents indicated that they did get benefit from the program.

For example, a few selected comments:

• “Yes, I got a lot. Because there is such a vast amount of resources that would not be available to us otherwise except through this site. It’s available to us—the records that would not be available, the music, the art, original written documents are all accessible through this site. It’s really overwhelming—I use it every day in my classroom [emphasis supplied]. You couldn’t get all this information out of history books.”
• “It gave me an excellent perspective on primary sources. It motivated me to share the information with colleagues, It greatly extended my knowledge of the technical aspects of the audio and video capabilities of computers in the classroom.”
• “..I was introduced to new technology. Of especial value the…content of sessions, perspective on teaching and learning, increased motivation for teaching excellence, and new information on resources.”
• “I’m a media specialist. I help teachers to use the internet/print and non-print materials. I find that a lot of teachers do not use primary resources with their students in their research papers. I’ve shown quite a few teachers the Library of Congress site to encourage them and their students to seek primary resources in research projects.”
• “Yes. In fact I used the lesson plans we helped to create at workshop in my lesson plans this year. Through use I did quite a bit of modification. It was a huge teaching [challenge]…we set it up as a group of two students [working together] so I had 12 projects going on times 4 [i.e. in four classes], so I’m changing it to groups of 4 next year. Second thing, I underestimated how well the students could get around on the web and locate the kinds of information required to complete the project. I was amazed. They found sites for me that I kept and subsequently incorporated. I had no idea how much is out there.

Average ratings were calculated for a variety of factors that could contribute to teachers benefiting from the NDL workshops. Participants rated most highly, interactions with the instructors, content of the sessions, the experience of developing products or materials at the course and discussions on how they would use what they learned in their own classes. The lowest ratings for contribution were preparation prior to the course and follow-up activities.

Most of the participants indicated they developed or revised one or more existing units. Just under half the participants developed or revised two or more units. Over half the participants developed their units in collaboration with others. 90% of the teachers responding to this item described the units they developed as interdisciplinary. Of those who answered the question, less than one-fifth wrote that their units received formal departmental or program approval.

• “What I did was that I now have my children, every day, work out of the LOC [Library of Congress]in areas of their interest to learn research techniques. The time allotment is one hour for 8 or 9 year olds. I first taught them what was available in each of the different sites…and they are required to choose one topic of interest in each of the different sites and report on it.
• “We (my colleague and I) are showing teachers how to get their students to use better skills in researching. When we went to the LOC we showed teachers how to do a more in depth research to find it faster and easier.”
• “Basically, most of the teachers did not have experience using the Library of Congress at all. So we basically look....[there] to enhance the lessons that were already there. For example, in a history class they produce a WWII scrapbook and the Library of Congress becomes a resource. Please note that more examples can be given if necessary”
• “Basically we developed a whole new unit. It changed the way we teach Nevada History and Nevada symbols. For example quilt making. We’re integrating more technology use with teaching of social studies. We: art teacher, librarian, music teacher, and me, the computer person.”

Now I’d like to ask you to be more specific about the types of changes you made in units or teaching as a result of participation in the project.

The highest proportion of participants indicated that they had introduced new content, followed by introducing new experimental techniques or lab techniques. The change in content to focus on key issues or “big ideas” and introducing new equipment, materials or computer software that they learned were judged the most important to the changes in the courses they made.

Of those answering, over 80 percent of the teachers have taught one or more of the units they developed as a result of participating. Of those answering the question, sixty percent have taught more than one course or unit developed as a result of participation. Of those who answered the question, over forty percent team-taught the material.

The average number of students who have completed these materials per teacher was 88 and the average percentage of female students was 55.1%. Of those answering the question, two-thirds reported the courses or units are still being offered.

Compare the average level of knowledge and skills of students who completed the courses/units you developed or modified as a result of your participation in the course with the knowledge and skills of students who have completed similar courses/units you taught previously.

On a scale where 1 = ‘a great deal worse’ and 5 = a great deal better, all but one of the items had an average rating of 4.00 or higher (somewhat better). The highest ratings were given to ability to apply new knowledge (4.40), ability to collaborate with others (4.24) and critical thinking skills (4.20). The lowest rating (3.91) was given to communication skills.

• “In one of the collections shows during training—possibly Life in the West—some of the participants recognized people in the photographs ass lifelong residents of Nevada that they knew. Wow—here is a remote rural Nevada family being featured in this rare extensive world wide collection—the Library of Congress. It’s one of the major feature collections, right on the front page—you don’t have to stumble across the information. We are very remote and rural---one room school houses in the school district. So this was really BIG.”
• “They know the difference between primary and secondary sources. They can’t get to primary sources, so this is a primary resource of itself as well as the only resource of information these students have access to as primary. They are restricted to this campus.”
• “My students received knowledge of the subject area that had never been introduced to them before. The project allowed them to ID their multiple intelligence and enhance that multiple intelligence through cooperative learning.”

Most of the teachers reported attending professional meetings, seminars or workshops and participating in further professional development activities. Those who began new communication or continued existing communication with experts in one or more disciplines rated their impact the greatest. Those who attended professional meetings rated their impact the least.

“You get a different angle on ideas for implementation. For instance, when I took Power Point training to become a trainer to provide in-service training on PP this summer, it enabled me to develop a hyperlink to ‘American Memories’. I can demonstrate at the power point in service workshops the scope of sources of the NDL site that my teachers can use in their classes; a double outcome of my summer in service training workshops.”

Almost all the participants have shared information or skills they learned with colleagues either in their institution or in any other institution and they most often did that sharing through informal discussions with one or more colleagues. Most of the participants reported their colleagues attending or joining the project.

Less than half of the responding participants report encountering barriers to implementing what they learned from their involvement in this training.

Just over half report that they did intend to develop new materials or units or modify existing materials or units. Very few of the participants spent time in preparation before the course.

Most of the barriers encountered related to time—enough to implement-- restrictions on the curriculum, and especially lack of technology in the classroom. Teachers also used their own money to pay for ink for printing and had fund-raisers to help cover such costs.

• The average number of hours spent during the course was 27.
• The average number of hours spent in additional development was 16.
• The average number of hours implementing the course/unit was 25.
• Just over forty percent of the respondents reported hours beyond the above areas.
• The estimated total average number of hours spent that were related to the course was 48.

The average number of years that teachers had been at their school was nine years Nine-tenths of the participants taught at the high school level. Slightly more than a fifth of the participants were male. The average age of participants was forty-seven. Only two participants described themselves as Hispanic or Latino. All but three of the participants described themselves as Caucasian. All the participants who responded to the question said they were U.S. citizens. None of the participants reported having an impairment or disability.

The participants had high praise for the instructors. They want, perhaps, about 30 hours of training, and they felt that it should be presented to more and more teachers. The most important negative is probably lack of time to implement and lack of technological resources in many places. Appendix E presents the full set of participant comments.

Copyright © 1997-2003 Career Connection to Teaching with Technology USDOE Technology Innovation Challenge Grant Marshall Ransom, Project Manager All rights reserved.