CAREER CONNECTIONS TO TEACHING WITH TECHNOLOGY (CCTT)
EXTERNAL EVALUATION, YEAR 3
Karen C. Cohen
Karen C. Cohen and Associates
9 Cliff Road
Weston, MA 02493

(1) A Teacher Survey (See Appendix A) which was used with about 80% of the participating teachers in the six hub site schools (N=50). This survey covered their involvement and reactions to the project in detail, and the results provide the project with a good deal of formative feedback as well as external documentation of outcomes and this mid-point in the project (March, 2000).

(2) A Staff Survey (See Appendix B) was used with consortium staff. This survey was developed adapted from the RAND Sourcebook, funded by OERI. [It was used at the beginning of the evaluator’s involvement in the project with all teachers in all hub site schools, and we continue to use it in a variety of contexts for comparisons over time. We were requested, however, by US Department of Education to report only data collected prior to June 8th in this evaluation report.]

• A focus group conducted with all project staff and consultants in January, 2000.
• Participant Observation in all Consortium Meetings.
• On-line review of units and lessons.
• Written reports from each hub-site manager.
• E-mail dialogue and telephone interviews with all hub-site managers.
• Personal and telephone interviews with content specialists and strategic partners.
• Student record data provided by hub sites.

Objective 1: Identify and incorporate into curriculum development the national reading, writing, mathematics, and science achievement standards, benchmarks and accountability measures for students.

Results Year 3:

(a) Student academic gains: Students are using materials developed by teachers participating in the project. However, field testing of materials in classrooms other than that of the authors has only just begun. Teacher reports of student involvement and gains are appearing, however. (See Section 5.0 below). Evaluation Activity: The external Evaluation is establishing a data base including information from student school records to assess such gains in years 4 and 5 where sufficient implementation will have taken place and we can attribute possible gains to use of project materials. This will be done in collaboration with each participating school district/hub site, but common factors across the Consortium will be explored. We will also re-use instruments developed and used by students and teachers in the hub site schools and their districts involved as CCTT materials are implemented in classroom settings in Years 4 and 5 of the Project.

(b) Incorporation of National Standards CCTT has made strong progress incorporating national standards. It has done so both by utilizing content specialists who, in some instances, created and refined the national standards in their own fields. [It should be noted here that some of these sets of standards are, themselves, being rethought and refined.] The project also has started to use content and standards curriculum experts in areas covered by the project who are state-qualified content specialists and are also participating in the on-line curriculum development and review process CCTT has created. This multi-tiered review and collaborative development process is geared toward developing peer and specialist reviewed quality materials. The original proposal suggested using national content specialists who were on the committees that created the standards in each field to review developed units and lessons. This has not materialized successfully, but involving qualified external content specialists who are qualified both in the standards and instruction in each field has replaced this original expectation as a more viable option. Hub site managers and TIES consultants have been active in recruiting these people to add their expertise to the Consortium, and this involvement is strengthening the process.

Evaluation findings for year 3 are supported by interviews with content specialists, surveys with teachers, focus groups with all site managers, review of units, lessons and courses produced in 5 of 6 sites, and participant observation with the entire staff at all full staff project meetings.

Objective 2. Create a career connection to integrated core academic studies that increases the relevance and authenticity of learning.

The project has established an advisory panel and plans for this goal attainment. One hub site reports using career planning activities as part of assessing impact on students. [Collecting evaluation data for this goal is appropriate for Years 4 and 5 of the project. Interviews, participant involvement, and surveys will be used.]

Objective 3. Train teachers and students to access and utilize existing technologies to create original instructional materials, and to collaborate with business partners in co-development of resources. In Year 3 of the project, CCTT has continued to use and develop on-line professional development strategies as well as local workshops in each of the six hub site schools and geographic areas involved in the CCTT Consortium. The project’s teachers, hub site managers, reviewers, and consultants have produced almost 100 units and over 300 lessons. These are in varying stages of consortium and content specialist review. In addition, complete courses have been developed at some sites, and rich web-sites in Physics and Chemistry have been developed by individual teachers and are also available to the consortium. Although implementation has varied at each of the six sites, communication and collaboration across sites has been maintained, utilizing, strengthening, and leveraging the Consortium’s resources. Teachers find peer-reviews very beneficial (See Project and Teacher Impact sections below).

Evaluation findings here for Year 3 are based on:

(1) lengthy and detailed paper-and-pencil surveys returned by nearly all of the teachers at the participating hub site schools who are involved with the project;
(2) participant involvement in the planning of workshops for Years 4 and 5 of the project;
(3) reports of hub site managers;
(4) on-line review of developing and developed materials;
(5) telephone interviews with participants and consultants;
(6) focus groups at selected locations and staff meetings;
(7) Corroborative documentation reports are in process as internal project efforts by consultants from TIES who are heavily involved in the development of the project curriculum development process.

In addition, the project has fostered and supported material and resource development beyond the originally envisioned curriculum units and lessons. These resources include a website replete with resources for full teaching and tracking students’ involvement and achievement for courses in several levels of physics, as well as courses in Graphics Design, CAD, and Criminalistics/Forensics. Evaluation of these additional resources has been formative to this effort to date, but results obtained in collaboration with the teacher-developers indicate strong impacts of these technology-rich resources on student achievement, recruitment, and attendance, among other factors being tracked by the project and the evaluation for years 4 and 5.

Objective 4. Contribute to national educational networks using telecommunications to disseminate products and best practices. The project has made major headway in this area. It has developed and implemented authoring systems and templates for its own use. By becoming the CGLi node (Challenge Grant Learning interchange) housed and supported at no cost to the project and to end-users by Apple Computer Corporation, the project may be resolving issues of authoring systems in state of flux along with incompatible equipment configurations both within sites and across the consortium. Via the CGLi network, teachers can and will develop, publish, and disseminate CCTT materials. CCTT can also disseminate other materials, although it has to resolve a mechanism for doing it. This resource is designed to serve as a national and international locus for other individuals and projects developing similar and or related materials to be approved, publicized, and made available.

Year 3 evaluation of this activity includes ongoing review of systems used and materials developed; feedback from technical specialists associated with and supported by the project as hub site managers or full-time partners--depending on the site; and participant observation by 3 different external participant observers at staff conducted training workshops.

3.1 What were the biggest Challenges in the first half of the project?

• Changing vision and formats/ and presenting those to faculty
• Reforming a common vision
• Setting up contracts and payment procedures
• Overcoming distances and diversities
• Dealing with problems arising from some of the original partnerships
• County Dynamics/District Networks
• Establishing and maintaining deadlines
• Shift from delivering content to becoming curriculum specialists
• Becoming curriculum creators not locators/greater role in instructional design
• Loss of the original grant writer - What was George thinking? New project director
• Establishing a dynamic between technology and pedagogy

3.2 What were the Most Outstanding Successes of the Project to date?

• The group coming together to collaborate and share the decision making process
• Creators of content / Finding direction out of chaos
• Using technology totally to further project’s goals
• Introducing teachers to new technologies and seeing them use them
• Teachers changing from accessing curriculum to creating it
• Gaining access to resources and personnel from our geographically diverse sites
• Dissemination/ presentations/ promoting the grant and DOE
• Moving from a static product to a dynamic support process
• Development of the process for creating standards-based, quality materials

3.3. Anticipated project outcomes (Year 5).

• Product validated by an outside group of teachers and students
• A “boat load” of quality product
• A model product available to more teachers for developing standards-based, technology-infused teaching and learning
• Empowerment of teachers to be innovators and agents of change
• Evidence that technology, money and time are critical for creating change in schools
• User-friendly system for development and distribution of content
• A plan for sustainability

In brief, the CCTT Consortium, as a group, feels that the impact on teachers of involvement with the project and its processes along with its use of its technology-rich materials will fulfill the vision of the original proposal, and that this vision is now structured clearly.

3.4 Infrastructure change(s) The change from DCI to Apple (from an intranet to an Internet-based authoring and potential publishing facility) has been successfully implemented, hopefully to the future benefit of the project.

3.5 The RAVEN simulation algebra tutor The failure of UCF’s Institute for Simulation and Training (IST) to develop a functioning simulation (RAVEN) as an algebra tutorial is being handled on two levels. IST has offered to provide simulation and training resources for on-line professional development and most logical use in physics courses at little or no additional cost to the project. Materials in algebra are being considered that have been developed by others if support is available, or that will be developed by new teachers joining the Consortium as it continues its diffusion, dissemination, and writing workshops. Two participants in the first dissemination workshop have offered to recruit teachers and teachers-in-training at the University of Florida (Gainesville) in mathematics and a second at St. Petersburg High School, also in Florida.

3.6 Sustainability Plans for sustaining and furthering the project and consortium financially, i.e. support and availability for its units, materials, and other resources (including on-line professional development opportunities) are still not concrete. Individual hub sites have leveraged their involvement with CCTT, however, and have received related or co-related funding from public and private groups to augment and further the overall effort to use technology and develop innovative materials and resources to improve teaching and learning for themselves and others.

4.1 The sample 50 teachers responded to a detailed survey about their participation in the project, their units and lessons, project support for what they were doing, and support staff help (See appendix A). These surveys were administered to teachers only in the hub-site schools participating in the project. (Far larger numbers of teachers are involved in developing materials for the project, but we were piloting this survey in an open-ended form and are developing surveys we can easily score in future years based on this year’s input.) The surveys were administered and returned anonymously.

Distribution of Respondents
Hubsite School	Number
Advanced Technologies Academy, Nevada	11
Ft. Leavenworth Unified School District, Kansas	11
Mainland High School, Florida	12
Manual Arts High School, California	15
Omaha North High School, Nebraska	12
Sprayberry High School, Georgia	7
TOTALS	50

Despite their geographical diversity, the teachers participating in the project were remarkably similar:

• They averaged 11 years teaching experience;
• Half (or 25) have BA or BS degrees and half had M.A. or M. Ed. degrees; Most had professional certification far beyond their degree levels;
.
• The year in which they received their highest degree was 1990 (with the range far greater); and
• they had been with the project and average of 1.8 years.

The teachers at Fort Leavenworth taught K-8; all of the other teachers taught at the high school level. (A small number of teachers taught more than one subject.)

The subjects taught were

9 -- Social Studies
8 -- English/Language Arts
6 -- Mathematics
6 -- Chemistry
3 -- Physics
3 -- Science
3 -- Computer Science
3 -- Reading
2 -- Applied Internet Basics
2 -- Biology
2 -- Spanish
1 -- Drama
1 -- Physical Education
1 -- Integrated Science
1 -- Earth Science
1 -- Library Science
1 -- Life Science
1 -- Home Economics

These teachers reporting had developed 125 Units and 370 Lessons for the Project by March, 2000. In addition, some teachers (not included in this sample) had developed complete courses (including lessons and units).

4.2 Challenges in developing Units and Lessons

The most prominent challenges for teachers involved in the project were:

• Narrowing their concepts into units and lessons; refining a topic
• Finding the time to get everything done during the school year, and
• Determining or coordinating with the standards (National and State)

Less frequently mentioned but other frequently mentioned challenges were “Using the template”, “Using CGLi”, etc.

As one teacher effectively stated: “Getting it all together. Learning to create in lessons that followed several objectives/goals such as national/state standards and be creative and fun.”

Other less frequently mentioned problems involved overcoming fear of technology, having technical problems, not getting timely review help or not getting reviews at all, and not having a chance to pre-test what they had done before sharing it with others to use.

4.3 Challenges in Implementing Units and Lessons

For those teachers who had implemented there were three major problems:

• Time—time to teach and time in class
• Technology—availability and reliability, and
• Teaching students Technology skills in addition to materials

4.4 Most Successful Outcomes from Units and Lessons

Teachers reported successful outcomes for their students and for themselves. For their students, they reported:

• a huge increase in student motivation and interest
• successfully using a new medium for students to use to learn (technology)
• students expanding their learning and research with access to the Internet
• greater variety and depth of learning
• collaboration and team-work with small groups of students
• better organization for students—objectives they could understand, meet and master
• self-pacing options for slow and fast (all) learners
• one teacher: “…could see girls figuring it out.”

5.1 Concepts and Understanding

In all sites, some or all of the teachers reported evidence of mastery by student scores on the assessment activities built into the units (with 85-100% mastery) as the strongest indicator along with project reports and products their units involved. In some instances these assessment or mastery items were in units used by teachers in the consortium that were developed by other teachers in the consortium. Several teachers in one site reported student ability to relate what they were learning to career paths was clear evidence of concept mastery and project success. Other teachers reported longer attention span, retention over the year, and impact of visual presentations for “visual and kinesthetic learners”.

5.2 Student Attitudes

Over half of the teachers responding said that their students were excited, engaged, and very much enjoyed the CCTT developed units and lessons. They “loved it” when they saw the projector set up for a power-point presentation. Some teachers felt that students liked the better organization of the lessons—they knew what was coming, what to expect, and were willing to learn. Two teachers reported students using these resources at a distance to keep up or make up for absences from school.

5.3 Student skills

Student increase in technological skills was the overwhelming result of using and implementing CCTT materials. Collaborative or group work was also prominent across all sites. 5.4 Behavior in class Teachers reported that students were almost all engaged, focussed, motivated, interested, on task, and more dependable—fewer discipline problems in general. Some teachers reported frustration that each student did not have his or her own computer, and some frustration with time “off-task” when primary resources (e.g. the Internet) were not available to them.

5.6 Reactions of others Teachers reported strong parent enthusiasm and excitement (at open houses and parent conferences when this was discussed). One parent reported that “I could not get my child to go to the movies, he was so busy working with the computer at home!” In two sites other teachers loved the units and were also using them as well as the new equipment. Teachers also reported varying interest in their schools, but noted peer interest and support, principal support, district support, and invitations to make national presentations.

6.1 Classroom Activities

Most teachers reported (as would be expected), enormous use of computers. They also reported group work and interaction between and among students far more than in their traditional classes. They reported use of CAI tests and tutorials, hands-on activities, all students involved, smoother/more planned classes, and in some instances students leading lessons rather than the teacher. A small minority of teachers reported no change, but even some of them were already using computers as a regular part of their instruction.

6.2 Instructional Materials Used

Beyond universal use of computers and heavy use of the Internet, teachers reported using projectors, powerpoint, texts, and television along with a variety of laboratory materials and utilities, other software utilities, and calculators.

6.3 Your (Teacher’s) role

Teachers overwhelmingly reported being facilitators, guides, mentors, and problem solvers. They “set the stage” and “answered students’ questions” when implementing their units.

6.4 Student Engagement and Participation

Almost 100%

Teachers reported on professional development activities, project staff, project resources, and personal impacts.

7.1 Professional Development Activities

Professional development activities varied quite a bit by site, with each hub site manager organizing and presenting sessions, workshops, or on-line courses for participating teachers. Sessions with speakers were well-received, although the speakers were in some instances presenting too much information at one time. Nearly half of the teachers had NOT attended or participated in professional development activities at the time of responding to the survey.

7.2 Content Specialist Help

Teachers generally felt that the help they received from their hub site managers or peer reviewers was very helpful. Most teachers had not had content/standards specialist help, and a few who had did not find it helpful. One felt it was not required, and there were complaints about the lag time (e.g. waiting from August, 1999 to February, 2000 for reviews).

7.3 The Template

Most teachers found the template easy to use, a good organizer, and as one teacher said “Just seeing it helps me get over the first stumbling block.” Teachers complained that changing the template three times in three years was too much, but those newer to the project found it helpful. A few found technical problems, e.g., saving information, uploading files, and not allowing formatted worksheets to be uploaded, that have to be solved. [The new version of the CGLI website allows uploading of attachments.] A minority felt that it was not good.

7.4 Curriculum Development Process

Teachers felt that time and getting started were the most difficult problems, with finding the time to do the work being the most difficult problem. Some felt that the process was clear, but two felt that they were “just flying solo”. Others felt that it was teacher initiated, good, and one felt that the CCTT review process doesn’t exist. 7.5 Project Staff

Project staff (in this project Hub site Managers) were uniformly helpful and supportive.

7.6 Use of Units and Lessons in the future

95% said they were using their CCTT materials now and would continue doing so. Further, some were using materials other teachers in the project had developed and were planning to use more in the future. The collegiality developed among this virtual community is strengthening CCTT and should expand as the project continues.

7.7 Personal Impact

Beyond finding CCTT as a community where teachers can contribute, receive feedback, develop and share materials (see above), teachers generally reported (in decreasing frequency):

• major person gains in technological literacy and skills
• general growth and development as teachers
• changes in how they deliver instruction
• changes in how they think about their teaching
• learning to manage their time
• overcoming fear of technology

To quote three teachers:

• “I actually can’t believe how excited I am about so much extra work!”
• “I can generate electronic classroom presentations over a weekend that rival what only a Hollywood producer could have done two decades ago.” and
• “It caused me to think through the planning process differently when I know others would be using my unit.”

Copyright © 1997-2003 Career Connection to Teaching with Technology USDOE Technology Innovation Challenge Grant Marshall Ransom, Project Manager All rights reserved.