Work-Integrated Assessment Model: COLLABORATE project

What it is: The Work-Integrated Assessment Model is a model that is applied to the design of assessments where the tasks and conditions are more closely aligned to what the user would experience within employment (work-integrated assessments).

How it works: The model focuses on six areas of assessment which together create the conditions under which students should develop both professional and academic skills during their studies. The six dimensions are Time, Audience, Problem/Data, Collaboration, Structure and Review. The model is designed to be used individually or collaboratively between groups of academic staff which can also include students and employers.

Benefits: The model is a thinking tool to help the user or users reflect on how a current assessment might be marked on the six dimensions of the radar chart, and what changes might be made to move along. For teachers these dimensions give an overall indication of how a designed approach has worked in practice. Likewise students are prompted to reflect on their recent learning: ”How good are we at articulating academic and professional skills?”; “What skills have we been developing through this module?” and “How well can we describe our skills to employers?” The evaluation materials can be used separately as a stand-alone evaluation method for any module. Its physical approach encourages deep reflection and the ability to recognise and articulate skills developed.

Challenges: The full potential of the model will not be realised if the design of assessment tasks is approached as an individual endeavour rather than a collective one. The dimensions identified cut across modules/subjects and the adoption of a common strategy reinforces the capacity of students to reflect on and describe skills they have developed.

Relevance for entrepreneurial teaching: The model was designed and tested in a Higher Education setting but minor tweaks might make the protocol, dimensions and the tools in the evaluation package perfectly valid to ignite CPD conversations and classroom action around EE assessment of entrepreneurial skills in primary and secondary schools.

Applied assessment methods: Formative Assessment, Performance Assessment, Authentic Assessment

Examples from practice: A video of the evaluation materials used in practice can be viewed in the University of Exeter project´s website alongside a short interview with Rachel Wheeler, voicing her thoughts about students’ ability to recognise skills acquired through their studies.

Website https://as.exeter.ac.uk/aspire/aspectsofacademicpractice/assessmentandfeedback/work-integratedassessmentthecollaborateproject/aboutcollaborate/#d.en.485855




Skills Evaluation Tool

What it is: SET (Skills Evaluation Tool) is a potent platform which enables measurement, evaluation and development of entrepreneurial competences. It allows, in a flexible and professional way, the assessment of skills development in entrepreneurial programmes. Students are the real protagonists in evaluation. They reflect upon their own diagnosis and set improvement compromises which are monitored in cooperation with the teacher.

How it works: This platform offered by TKNIKA (Centre for Innovation in Vocational Training in the Basque Country) is interesting for any school as it allows for the adjustment of WHAT, WHO, WHEN AND HOW to evaluate entrepreneurial competences amongst students. It is based on a system of evaluative rubrics shown to users through a pleasant and easy to use interface allowing the visualisation of students’ progress in their competences through different graphs, as well as comparisons with the whole group.

Benefits: It allows the implementation at school-level of a flexible system. It establishes a co-evaluation system involving the student, project peers, and teachers in a quick and visual way. It enables each student to know which competences need improving and to acquire personal compromise, which is then monitored in his/her evolution, through different graphs and reports.

Challenges: It requires Internet connection to be used. In order to conduct measurements and evaluations there is also an App designed for Smartphones. The implementation and usage in the whole school entails preliminary (and hard) work of reflexion and configuration of the assessment model. One of the most important challenges is the choice of appropriate competences and levels of assessment.

Relevance for entrepreneurial teaching: SET offers a technical tool to all schools so that they can evaluate the level of development in entrepreneurial competences. It can be particularly useful in schools adopting active learning methodologies where students become the protagonists of their own learning.

Applied assessment methods: Owing to its power and personalisation capacity it offers a very interesting technical support to be implemented on a small or big scale. The configuration of participation weighting between self-assessment and co-assessment makes it an adaptable tool for the assessment of different competences and complexity levels. Its rubrics are oriented towards assessment of learning outcomes evidenced by student performance and linked to chosen competences and feedback provided by the teacher. SET is easy to use in the classroom thanks to the app designed for smartphones.

Examples from practice: SET has been implemented to facilitate the evaluation of entrepreneurial skills in Basque vocational education schools. It is used by 47 Vocational Education Schools, more than 3,500 students and 1,646 teachers in 142 different courses each year. It has also received good external evaluations and an agreement has recently been signed with the University of Chile in order that it be introduced into several of their programmes.




Self-report Questionnaires

What it is: Self-report questionnaires are the most common approach to assessing entrepreneurial or soft skills among both researchers and practitioners.

How it works: Questionnaires typically ask respondents to integrate numerous observations of thoughts, feelings, or behaviour over a specified period of time ranging from “at this moment” to “in general”. Respondents read the statement, search memories for relevant information, integrate whatever information comes to mind and translate it into one of the response options, finally, adding comments if asked (and motivated) to do so.

Benefits: Self-report questionnaires are cheap, quick, reliable, easy to administer and in many cases, remarkably predictive of objectively measured outcomes. Self-report questionnaires are arguably better suited than any other measure for assessing internal psychological states, like feelings or belongings. People are relatively good at using questionnaires to communicate their true opinions as long as they in fact have answers for the questions asked and feel comfortable reporting accurately on them.

Challenges: Students, particularly those who are younger or lower-achieving, may misunderstand the statement or interpret terms differently (e.g. “rarely” may be what another respondent considers “often”). The fact that questionnaires require recalling past events in search for evidence provides a fertile ground for biases. Students may provide answers that are socially desirable but not accurate. There may be insensivity to short-term changes (consistency bias) and it is quite likely that the more competent students may be harder on themselves (reference bias), not to mention the possibility of “faking”, deliberately inflating or deflating scores.

Relevance for entrepreneurial teaching: Often, self-report questionnaires are used as low-stake diagnostic tools that set the baseline of skill development for individual students as well as their attitudes and intentions towards entrepreneurship. However, literature cautions against taking these diagnoses at face value due to their low reliability. In some cases, they are repeatedly administered over a period of time in order to track progress and support students in the process of skill development. Yet, it is unclear if existing questionnaires are good enough at capturing short-term changes and the quantity and quality of feedback is also an issue. The literature calls into question the validity of self-report questionnaires to measure program effects at the individual level (changes from pre-test to post-test) or comparing schools and/or programmes.

Applied assessment methods: Self-assessment, Formative Assessment, Summative Assessment.

Examples from practice: Entrepreneurial scans, skills audits and profilers are popular among entrepreneurial educators. OctoSkills and the Enterprise Catalyst are a couple of examples of tools based on self-reporting surveys. The Measurement Tool for Entrepreneurship Education is a self-evaluation tool for teachers and educators developed by the University of Laaperanta.

See also: Measurement Matters: Assessing Personal Qualities Other Than Cognitive Ability for Educational Purposes.

 




Empower yourself

What it is: Empower yourself (original in French: Valorise toi) is a self-assessment tool of skills acquired in the framework of volunteering in the French scouts’ movement (‘Scouts et Guides de France’). It translates skills acquired through scouting and volunteering into language appropriate for the job market.

How it works: Empower yourself is a leaflet for young people making applications to their next level education or young job seekers, allowing them to ‘translate’ skills from volunteering into competences that apply to the job market. The simple leaflet breaks this process into four steps:

  1. Scouts choose from a list of statements the experiences or activities they have completed, (e.g. ‘I am aware of my responsibilities in my role as a leader of my unit / group or as the camp chief’), and assess their level of strength for each activity. These are then clustered in a way that translates into competences that help in the job market, e.g. ‘organisational skills’, ‘responsibility’, ‘sense of initiative’, analytical skills’, and so on;
  2. Students are asked to illustrate the skills with personal examples;
  3. They put the skills into context towards their chosen career;
  4. A user manual prepares scouts for their job interview. Empower yourself is available in Czech, Danish, English, French, Polish, Serbian and Spanish.

Benefits: This simple reflection tool translates non-formal learning into employability skills. The method is simple and transferable to a wide range of environments and is low cost. The tool works within a structured and detailed skills framework and clearly identifies skills and experiences and how to translate this into the language of and skills for employment.

Challenges: As the tool is to be used offline (PDF for download only), no user identification or data collection is possible. Neither is an analysis of user skills/ responses included as part of the tool.

Relevance for entrepreneurial teaching: Empower yourself translates civic competences acquired through non-formal learning experiences into skills that are applicable to the job market. The method used to define the skills and translate them is applicable to a range of environments.

Applied assessment methods: Assessment of skills.

Examples from practice: According to case studies, Empower yourself has been used by over 30,000 young people in France. Feedback from employers and users is positive.




Octoskills

What it is: Octoskills is an application for mobile and the web for teachers and practitioners who aim to evaluate students’ entrepreneurial skills. It was developed by the Danish Foundation for Entrepreneurship based on the EU-funded ASTEE project (more below).

How it works: The tool is used in primary, secondary and tertiary education to assess students’ self-efficacy and entrepreneurial attitudes and intentions. It focuses on school engagement, educational motivation, and development of relations with teachers and classmates. Students use the app on their mobile phones to fill in surveys and questionnaires that translate into a spider diagram for teachers on their entrepreneurial skills. The diagrams show the before and after of students’ entrepreneurial skills. Teachers use the app to follow students’ development and learn about their teaching with a view towards continuous improvement. The app is led by the teacher who sets up a class survey and shares a unique code with his/her class. Results are shown by class.

Benefits: Octoskills allows for immediate feedback loops on students’ development and comparisons across class and school level as well as internationally by type. The tool stems from a strong research process and is available in multiple languages. At entire school level, Octoskills translates into a further app, OctoDash, for school development. Octoskills broadly focuses on the entrepreneurial mindset and skills, as well exploring the impact on school engagement and career intentions.

Challenges: The current abilities for students are constrained to filling in the surveys and questionnaires and receiving feedback on their strengths and weaknesses. It could be beneficial to receive support ideas for areas of improvement.

Relevance for entrepreneurial teaching: The tool was developed to support measurement of entrepreneurial skills. It has been designed with teacher support in mind, addressing the previous need for easy-to-use tools that generate insight into teaching effects.

Applied assessment methods: The tool uses e-assessment, ipsative assessment, assessment for learning, assessment of learning, and assessment of skills.

Examples from practice: The tool was developed out of the EU-funded ASTEE project (Assessment Tools and Indicators for Entrepreneurship Education) that aimed to define a tool to assess and evaluate the influence of entrepreneurship education. The ASTEE project was co-funded by the European Commission and took place from December 2012 until June 2014.




Enterprise catalyst

What it is: Enterprise catalyst is an online questionnaire with automatic algorithms to provide a detailed report to the user based on their results. The report is available to six different target audiences including schools, those in further or higher education, teachers and lecturers, employees, employers or business owners, and others (‘none of the groups seem to fit me’).

How it works: Enterprise catalyst provides a range of different user types with an online assessment and coaching tool that gives insight into the user’s entrepreneurial mind set (i.e. their attitudes and behaviours), with connections made to career options and entrepreneurial pathways. Users fill in the questionnaire for free by clicking the relevant box between comparative and contrasting pairs of statements. These statements are repeated in different questions as a control mechanism. There are three sections depending on the user type: describing oneself, awareness of and involvement in entrepreneurial activities, and personal statements. The questionnaire does not use start-up terminology and is firmly focused on supporting the individual and providing them with insight into their own characteristics. In addition, education institutions can use the tool for benchmarking and funders/deliverers may measure the impact of programme investments on a range of outcome criteria.

Benefits: The benefit for the individual user consists of an insightful report with personalised results across four or five sections. The different audiences get the same basic type of analysis; however, the language is tailored for that specific audience, e.g. simpler for younger learners. Younger learners get an additional section on personal learning and thinking skills. The sections are ‘Enterprise fuel’, ‘Enterprise style’, ‘Encouragers, support and obstacles’, ‘Team role’, and ‘Personal learning and thinking skills’.

Challenges: The tool has so far only been used in the UK and is only available in English and Welsh. The presentation is not dynamic and based only on a tick-box approach.

Relevance for entrepreneurial teaching: Enterprise catalyst is squarely focussed on the individual focussing on skills rather than using start-up terminology. The tool works at user and macro level and can act as a detailed learning or discussion tool for job seekers / employment agencies, employees / employers, or students / educators.

Applied assessment methods: The tool uses e-assessment, reflective assessment, and self-assessment.

Examples from practice: Enterprise catalyst has been used at regional level in Wales.




Rubrics

What it is: A rubric is a coherent set of criteria for student’s work that includes descriptions of levels of performance quality on the criteria. The main purpose of rubrics is to assess performances, that is to say the things students would do, make, say or write. The rubric describes a performance, understood either as a process (e.g. a pitch) or as an artefact (e.g. constructed object). The focus is on learning and not on completion of tasks per se.

How it works: The rubric is a 2D grid where criteria are listed in the left hand column and levels of performance in the upmost row. The individual squares in the grid contain descriptors for the level of performance for each criterion. The rubric may or may not contain a mechanism to attach points or grades to different levels of performance.

Benefits: Rubrics help students understand intended learning outcomes and criteria for success. They guide teacher observations and allow more detailed and objective feedback. If designed for repeated use, over time on several tasks (general rubrics), rubrics help coordinate instruction and assessment.

Challenges: Large numbers of criteria and lengthy descriptors for each level of performance may have negative consequences, particularly with younger students. At the other end of the spectrum, the danger lurking is oversimplifying highly complex behaviours to a very tight set of criteria. It is not easy to write good descriptors for each level and each criterion. More often than not, the theoretical basis on which progression across levels of the rubric has been established is often unclear. Rater bias remains problematic particularly when the levels of performance are vague.

Relevance for entrepreneurial teaching: Teachers often rely on rubrics to assess entrepreneurial performances. The extent to which a rubric helps teachers in capturing the willingness, readiness and ability to put into practice entrepreneurial skills is largely dependent on finding the right mix of observable criteria and being clear on the levels of expected performance. Engaging students in the design of the rubric may be a good idea to fine tune the tool but it will never be perfect. While tempting, equipping ourselves with a rubric for each entrepreneurial skill may result in an unbearable assessment workload. Strategies to prevent “death by rubric” entail coordination with other teachers as well as introducing self-assessment and peer-assessment methods. Last but not least, rubrics should not be considered the “end-of-it-all” of entrepreneurial assessment. Rubrics could conceivably play a constructive role in designing assessment tasks that capture the sheer complexity of behaviours elicited but only if combined with other sources of information.

Applied assessment methods: Formative Assessment, Authentic Assessment, Performance Assessment, Self-assessment, Peer-Assessment

Examples from practice: Plenty of ready-made rubrics, templates and rubric generators (e.g. Rubistar) are freely available online.




INNOENT Idea Evaluation Rubric

What it is: This is a tool for evaluation and/or situating the progress of ideas and inventions during the progression of innovation education and entrepreneurship education. It can be used online or as a pen and pencil rubric.

How it works: The rubric is used by participants in workshops or courses in entrepreneurship education. The rubric provides standards for each step of the process, need, solution, product, product development and business plan. The inventor/entrepreneur uses the emerging profile to situate the idea on a scale. Each idea the participant is working has a rubric attached to it and the participant can change the marking on the rubric as many times as he/she finds necessary as the idea progresses. This evaluation is for the participant’s use only, where the teacher/mentor does not have to see it. However, it is often shared with the teacher at the beginning and end of a course.

Benefits: The most common benefit observed is how the individuals use the standards to make up their minds about how far they want to take each idea. They use it to determine where the idea is situated in the progression from a need to a full business, and to decide if they want to take the idea any further. The rubric benefits the participant as they can use it as a guide to suggest the next steps they could, or should, take in the progression of the idea.

Challenges: If students are not familiar with evaluating their own work using the rubric effectively can be a challenge.

Relevance for entrepreneurial teaching: Managing many ideas at the same time is crucial for the serial inventor/entrepreneur. This tool allows for prioritisation of ideas and their progress as well as putting forward suggestions of what could or should be the next step in the progression. The tool increases the entrepreneurial vocabulary of participants as well as allowing them to put self-determination into practise.

Applied assessment methods: This tool can be viewed as ipsative assessment in so far as the participant is constantly looking at furthering the progression of the invention/idea. The tool is inherently self-directed and is digital in nature.

Examples from practice: The tool has been used in teacher education in Iceland where innovation education is a substantial part or a main focus of courses. It has been used in several INNOENT Education courses in Iceland but more in the Middle East and Far East.Participants use the tool to evaluate the eligibility of an invention forproduction, for funding or to access international competitions such as ITEX.




Personal Attributes

What it is: This tool is a sliding scale list for individuals to situate their preferred personal attributes. The tool is based on a rubric published in ‘The Idiot Teacher’ by Gerard Holmes (1952).

How it works: The scale is predetermined by the teachers or leaders of the group. The number of attributes is up to the practitioners and is dependent on the attributes that are related to the present work and goals. Individuals use the scale as a pre and post evaluation of their own progression. For that to happen individuals need to understand the language used. The development of the vocabulary happens in the class discourse with or without the leadership of the teacher.

Benefits: Understanding of language is the foundation needed to consciously work on and with personal attributes. Engaging with concepts such as integrity and deceit, being enterprising and passive in the context of the work allows individuals to form an opinion of the concepts and how much they mean to them. In this way, individuals develop both self-understanding as well as a greater understanding of their ethical stance.

Challenges: Personal attributes are in many ways not central to the way education is evaluated now so the challenges are mostly due to the structure or accepted evaluation structure of education. Also, teachers have a problem with where to situate this kind of evaluation, in relation to the most suitable subject area. Understanding the language used in personal attribute discourse can be challenging for students.

Relevance for entrepreneurial teaching: Understanding one’s strengths and weaknesses is essential to be able to take calculated risks and approach different situations and individuals in appropriate ways. A strong understanding of who one is and personal capabilities is core to entrepreneurial education.

Applied assessment methods: This tool can be used in formative assessment and can be categorised as performance assessment and/or self-assessment.

Examples from practice: This tool has been used in a few locations in Iceland in different contexts and with different ages. The pilot test of this was conducted in one school in Reykjavik where 159 students were asked to use the sliding scale to evaluate their understanding of themselves. The youngest group that took part in this were in year 5 and the oldest group in year 10. Each individual filled out the form and gave it to the researcher as a pre-test and at the end of the course the students filled the form out again and gave comments on their progress. The results from this pilot gave promising results for further development.




Assessment and Teaching of 21st Century Skills (ATC21S)

What it is: ATC21S is a system enabling formative assessment of Collaborative Problem Solving. ATC21S™ has been designed to support the development of social and cognitive skills needed to become a good collaborative problem solver.

How it works: The system consists of four main components:

  1. Empirical progressions representing a typical pathway for Collaborative Problem Solving (CPS) skill acquisition;
  2. An online assessment platform containing prototype assessment tasks. Tasks require students to work in pairs and collaborate in real time. Students responses are recorded in a log file and rated automatically;
  3. A survey completed individually by each student;
  4. Individual and whole-class reports.

Benefits: The system adopts a developmental learning approach to assessment and instruction. It guides a student’s learning forward along a path of increasingly complex knowledge, skills, and abilities. Reports provide data on the emergence of skills mapped against empirical progressions that allow teachers to identify patterns and gaps at individual and/or class level and tailor instruction accordingly to challenge their students to move forward on a developmental continuum. The Project website provides open access to five professional development modules for teachers.

Challenges: None of the designed tasks could sample all the elements of the CPS construct comprehensively. The capacity to capture cognitive skills is greater than the capacity to capture the social skills. The empirical progression needs further investigation. Some other challenges relate to the limitations of online administration and complexity. Potentially a similar approach could be scaled up and adopted to assess application of skills in real-world contexts without compromising the ability to measure them.

Relevance for entrepreneurial teaching: ATC21S conceptualisation of Collaborative Problem Solving as a combination of cognitive and social processes displays a good number of features defining the entrepreneurial key competence (e.g.: goal setting, resource management, tolerance for ambiguity, audience awareness, negotiation, to name a few). The rigorous approach to build an empirical progression and the elaboration of a set of IT-based prototype assessment tasks represents an inspiring example for the design of entrepreneurial teaching and learning activities and assessment tasks.

Applied assessment methods: Formative Assessment, IT-based Assessment, Performance Assessment.
Examples from practice: During 2009-2012, the prototype tasks were trialled by schools in Australia, Singapore, the United States, the Netherlands, Finland and Costa Rica. In 2011, each participating country assessed a minimum of 660 secondary school students.