Evaluating and redesigning teaching learning sequences at the introductory physics level

Jenaro Guisasola, Kristina Zuza, Jaume Ametller, and José Gutierrez-Berraondo Donostia Physics Education Research Group and Applied Physics Department, School of Engineering Gipuzkoa (UPV/EHU), Plaza Europa 1, San Sebastian 20018, Spain Department of Specific Didactics, University of Girona, Girona 17004, Spain Instituto de Máquina Herramienta (IMH) Elgoibar, Elgoibar 20870, Spain (Received 31 May 2017; published 26 December 2017)


I. INTRODUCTION
Although teaching and learning sequences are not the only factors that influence learning in classrooms, they play an important role in learning outcomes.Research on the implementation of teaching and learning sequences (TLSs) has shown that they can be an effective way of feeding research into teaching practices [1][2][3][4][5].The improvement derived from the use of TLSs has been shown in some cases to be significant even for teachers with little specific training of the specific TLS [6,7].In this paper we will use the following definition of TLSs: "A TLS is both an interventional research activity and a product, like a traditional curriculum unit package, which includes well-researched teaching-learning activities empirically adapted to student reasoning.Sometimes teaching guidelines covering expected student reactions are also included" [8].
With this aim, several approaches to designing researchinformed TLSs to teach science topics at different educational levels have been proposed during the last three decades.They are commonly structured as teaching materials presenting didactical transpositions [9][10][11] of science topics (sometimes referred as "school science") accompanied by didactical indications on how to implement them, which might include theoretical insights, empirical research results, and teachers' professional tacit knowledge.However, these proposals often lack details about how theory and research results have been articulated into their design.Furthermore, not all the TLS proposals include an evaluation in terms of learning outcomes and very seldom are these learning outcomes specifically connected to their design.This lack of detailed information on the design and evaluation of the proposed TLSs hinders the possibility of properly evaluating their potential effectiveness and discussing and systematically improving their design.In this paper we put forward a proposal for the design and evaluation of teaching and learning sequences at the university level that could also be used for secondary school science.We will connect our proposal with relevant contributions on the design of teaching sequences, ground it on the design-based research methodology, and discuss how teaching and learning sequences designed according to our proposal relate to learning progressions.
The past thirty years have seen many contributions to the science education literature around different models of designing TLSs connecting theory and research results with teaching materials and proposals.Even though most of these proposals have situated themselves in the social constructivist framework that has dominated the science education field, and have included some common research results, such as the well-established literature on preconceptions, they present significant differences [8].Unfortunately, how these differences are derived from the underpinning theoretical models is not always made sufficiently explicit and the choice of their foci in relation to the empirical results to be included in the TLSs is not always justified.A study of different types of proposals for designing TLSs carried out by Psillos and Kariotoglou [1] shows that there has been some convergence on the design but that there are still important shortcomings to overcome in this field of study.These are, mainly, (i) a more explicit connection between the theoretical and empirical research insights and their effect on the design, (ii) a more robust evaluation procedure, and (iii) a clearer description of the iterative process that lies at the heart of design oriented research.As a result of these shortcomings it is difficult to analyze the proposed TLSs in a way that allows the science education community to build upon them to systematically improve these designs.Our view is that, to use TLSs effectively as a way to bring theoretical and research insights into science teaching, it is paramount to develop TLS design further as a research program and, hence, to explicitly define the methodology of their design and evaluation.
As other authors have pointed out [12,13] presenting all the relevant information about the design and evaluation process of TLSs requires more space than is usually allowed for a paper.Hence we will focus here on two aspects.First, we will present our methodological framework.As we have already mentioned, one of the problems we want to tackle with our proposal is the lack of a common framework to systematically improve science education TLSs.In order to do so we need to adopt a research methodology that tackles both the design process and the evaluation of the learning results.To systematically address this issue would allow the science education community to bring the design of TLS within the framework of a research and development process [8].We will present our design process as an implementation of design-based research (DBR) methodology [14].Even though the benefits of DBR in educational research have been questioned by some authors [3] we can find studies in the literature that report on its benefits in science education research.Juuti and Lavonen [12], reporting on their experience using DBR to design science TLSs, show how DBR provides the basis for a scientific approach to science education research by improving the trustworthiness of the designed TLS effectiveness as well as producing new "educational knowledge" Ref. [12], p. 65.In a recent study Kalle, Lavonen, and Meisalo [15] propose that design-based research projects produce an educational innovation that introduces novel didactical materials in teaching and learning for a wider audience than just the research group.Moreover, they claim that the process of design based research is iterative and offers new educational knowledge to help teachers to teach better and to support students in achieving the learning objectives.Trna and Tronva [16] have written about the use of DBR in the development, implementation, and evaluation of inquiry based science and math education as part of teachers' pedagogical content knowledge.Their study not only supports the view that DBR is an appropriate method in science education design but also that it can provide a methodology to involve teachers in the evaluation of the efficacy of the TLS as part of the teachers' pedagogical content knowledge.These examples, among others, support our proposal of DBR as a methodology to improve TLS and TLS research in science education.
Of all the elements from theory and empirical research that we have included in the design, we think that the elements that are critical to the success of the process, and the ones deserving further research, are the influence of the context of application of the TLS, the definition and use of design tools and their connection to "key ideas" [17], and the choice of evaluation tools and their relationship with the feedback and redesign processes.
Second, we will present our evaluation methodology and its connection with the refinement and redesign of the TLS which is connected with the essential iterative aspect of design-based research [11,15].Some of the TLSs proposals in the literature, though not all, include information on the learning results obtained during their implementations.While these results can be strong advocates for their effectiveness, in terms of improving a targeted aspect of science learning, there are unresolved issues in relation to the evaluation of the TLS that hinder their impact on improving the science teaching-learning sequence and on advancing science education research.The diversity of evaluation methodologies we encounter in the literature points to different views or opinions as to how to decide on the effectiveness of the proposed TLSs, for instance, using pretests and post-tests for students' learning [18] or interviews and the analysis of classroom video recordings to evaluate the teachers' views and implementation of the designed TLS [5].Most of the research literature on TLSs focuses its evaluation on the effectiveness of the designed TLS in a particular context of application [19] which, paired with the lack of common design and evaluation frameworks, makes any generalization of results problematic.Furthermore, the lack of explicit connection between these evaluations and the iterative design process [1] further weakens the generalizability of the results.The issue of generalizability is closely related to the context dependency of the TLSs and, more widely, of the knowledge generated through DBR that we will discuss in the design and procedure section.
Despite its shortcomings, we agree with other authors that the literature on TLS design already constitutes an important body of knowledge that we have drawn upon to make the proposal we will present here.In particular, we are indebted to the work of Refs.[11,[20][21][22][23] and we have also built upon our previous work [24,25].
Taking into account the previous research about designing and evaluating a TLS and DBR, in this study the research question is as follows: to what extent does DBR improve the design, evaluation, and refinement of TLSs?
To answer the research question we will present the design, evaluation, and refinement of a TLS, using the DBR methodology.Our work is placed in the context of introductory courses at the university level.The TLS addresses the general principle of work and mechanical energy, implemented and evaluated in the context of a transformed calculus-based physics course for first-year engineering and science degree students at the University of the Basque Country (UPV-EHU).

II. DBR METHODOLOGY
As has been mentioned, the lack of well-defined, explicit methodologies to guide the design and evaluation of TLSs is a factor hindering the development of a research program in this area.Even though several proposals can be found in the literature [15] proposing ways of connecting theory and practice within a design-oriented perspective, many of them are used only by those who have proposed them or, at best, by a small number of other colleagues.In contrast, we side with other researchers who advocate the use of DBR as the methodology for the design and evaluation of TLSs [12].This methodology is being used by a growing number of researchers in education [26,27] resulting in a community of researchers that discuss it and refine it.This fits with the aforementioned need for a more standardized methodological approach for TLS research.DBR, as a methodology, recognizes the importance of both theory and interventions in order to approach educational situations [14]; hence, it is well suited to the needs we have identified for TLS research in the previous section.Furthermore, DBR has many points in common with other methodologies suggested in the TLS literature, which makes it a candidate one can consider for a common general framework that could be slightly adapted to different specific proposals.
We have followed the definition of the DBR methodology proposed by Easterday, Lewis, and Gerber [14], which identifies six phases (see Fig. 1).In this section we will present how we conceive each of these phases, which type of information and design tools we introduce in each phase as well as some examples from our work on specific TLSs.One must keep in mind that these phases "are not carried out in a linear sequence but rather iteratively" Ref. [14], p. 321, and, hence, we will indicate how different phases impact on other ones during the process.

A. Focus
In this phase, designers establish the audience, topic, and scope of the project.The first step is to define the scientific content to be addressed by the TLS, how it should be approached in relation to the curriculum, and to which students it will be addressed.In sum, this is the stage where we define most of the contextual elements that will constrict the scope of the TLS and the value, in terms of potential for learning, of the chosen topic.

B. Understand
In this phase, designers study existing information on the known difficulties in learning the topic, as defined in the focus phase and the existing solutions available for them.The information that comes into play at this stage for the design of TLS is mainly found in secondary literature but it might require empirical interventions as well.For instance, a basic item in designing a science education TLS is the preconceptions held by students on the addressed topic.If that topic has not been researched before, designers will FIG.1. Different steps of DBR methodology.Adapted from Easterday, Lewis, and Gerber [14].
have to carry out an empirical investigation to establish those preconceptions.
There is a variable number of knowledge insights we can draw upon in this stage depending on the existing research that has been carried out on the scientific topic that constitutes the focus of the TLS.Unfortunately, as mentioned in Sec.I, the lack of a systematic approach to the design and evaluation of the TLS makes it difficult to use solutions (efficient TLS proposals) in the design of other sequences.The two main difficulties are the lack of clarity in the contextual factors involved in those sequences (focus DBR phase) and the differences in the learning objectives being evaluated in different proposals (define DBR phase discussed below).Nevertheless, the literature offers some reliable results that must be taken into account in particular topics.
The understand phase is a key stage in our proposal.Here we coordinate the following elements: Epistemological analysis.-Thefirst design tool we introduce is the "epistemological analysis" of the scientific content to be taught.This analysis uses the internal structure of scientific knowledge to define its construction in educational settings.The result is a justified set of conceptual components that need to be articulated by learners to construct the content identified as the focus of the TLS.
Learning demands.-Oncewe have identified the (components of the) content to be learned we use the "learning demand" [28] design tool, which analyzes ontological and epistemic differences between known students' previous knowledge and the science content to be learned, as defined through the epistemological analysis.These differences will guide the TLS didactical scheme by pointing at the type and degrees of difficulties we can expect students to encounter.

C. Define
At this stage, and in the light of the results from the previous phase, designers need to set the goals and determine how to assess the designed products.In the context of science education TLSs, this is the point where designers establish their learning goals and the indicators to be used to assess their achievement.It is crucial to clearly and explicitly define these learning objectives if we want the results from the evaluation of the resulting TLS to be useful in future designs.While there is some latitude on the definition of these objectives, other than the contextual factors that might limit them, designers should closely relate them to the results of the understand phase.This would naturally create a certain degree of convergence in different TLSs designed around the same focus.

D. Conceive
Designers sketch in this phase a solution to be implemented.This is the stage for the production of the TLS documents.They will typically include teaching materials, based on the analysis of the understand phase, evaluation guidelines in line with the define phase, and accompanying materials for the teachers with information on the intended use of the teaching materials.Not all the aspects in these materials will be directly derived from the previous analysis and hence it is important to inform teachers who will use them about which aspects are grounded on theory and research and which ones are personal choices of the designers.In this way, when the teachers implement it (build DBR phase presented below) they will be able to make informed decisions on which aspects could be modified and which ones should not.This information can be communicated to teachers in ex professo workshops [22], but this is not practical in most scenarios.An alternative is the production of teachers' guides that include information on the design decisions connected to the sources of information selected by the designers and guidelines on the teaching practice when using the activities included in the TLS [24].In both cases the aim is to inform teachers about the intended implementation of the TLS so that it is congruent with its critical points [17].
In this stage we introduce a third design tool (after having introduced the epistemological analysis and the definition of the learning demands): "relevance of science in society."From a nature of science (NOS) perspective it is important to present science not as an isolated endeavor, but as a social enterprise [29].Since science, besides a body of knowledge (facts, definitions, concepts, laws, theories), is a set of processes that are systematized to produce that knowledge [30,31], science education must also integrate this second dimension in the curriculum.As Hodson [32] shows, scientific education should encourage, among other aspects, students developing skills and attitudes for scientific inquiry and problem solving.The Next Generation Science Standards [33] emphasize this perspective in one of the dimensions proposed for the learning of science, under the name of "scientific practices."This perspective enables students to mobilize scientific knowledge to act as informed citizens when making decisions on aspects related to science, for instance, those connected with the development and use of technology.As a design tool, it is used to analyze the meta-scientific connections of a particular topic in order to provide students with a wider perspective.This perspective leads to an increased engagement, which is known to improve students' learning of science.In the approach that we propose, we use activities and problems from science, technology, society, and environment programs [34,35].

E. Build
This is the implementation stage.In the DBR literature this is where the product appears.In the context of the TLS design, the conceive phase produces the material that will inform and guide the teaching and learning process that will take place in the build phase.In other words, the conceive stage produces the intended curriculum and the build stage involves the enacted curriculum.

F. Test
At this stage, the designed proposal is evaluated to assess its efficiency in relation to the objectives that were addressed.Since DBR advocates an iterative process, the evaluation stage involves carrying out tests throughout the design and redesign process.In the case of a TLS, the evaluation and the redesign are also essential characteristics of the TLS research.The design of the TLS must be empirically tested through the evaluation of the proposal and of the achieved learning outcomes, along the following two dimensions [36]: (a) Analysis of the quality of the sequence, which involves (a.1) problems of the sequence related to clarity of the activities to be carried out by students, (a.2) difficulties related to time needed to complete the sequence, and (a.3) difficulties in writing a new sequence with innovative contents.(b) Analysis of learning outcomes, which includes (b. 1) understanding the concepts, theories, and models; and (b.2) acquiring skills related to scientific methodology for understanding the above knowledge.As there are a lot of specific instruments in educational evaluation, the elements that we have chosen and why we use them will be described in the Sec.IV.While the specific evaluation elements are part of the specific decisions that the group of research has to define in the DBR methodology, we put them in Sec.IV.
Another important difficulty in the generation of knowledge, which is relevant to teaching and educational innovation, is that, during implementation, teachers make changes to the designed TLS and its intended use which, in some cases, can affect the original aims and intentions underpinning the TLS.This might result in a teaching whose aims differ from those planned by the designers [37].Furthermore, there is abundant research on teachers' thinking [38,39], showing that they have a positive attitude towards the results of research in didactics, but that they are not willing to change their teaching practice If proposals from that research do not solve, at least partially, some of the teaching problems that they have in their everyday classroom practice.Teachers mention that their teaching practice is strongly influenced by their school colleagues, textbooks, and other didactical materials used in their classes.Considering these results, a didactical tool that might be useful to promote research on informed change is the "teachers' guide."This guide, designed as part of the TLS, must minimally contain a detailed description of the scientific contents and the elements related to the nature of science of the TLS.Furthermore, the guide should contain two other essential elements: a detailed description of the desired learning aims and teaching practices, and a description of the resources teachers will require to implement the innovative TLS [40].

III. DESIGNING THE TLS: MECHANICAL WORK AND ENERGY
This section provides an overview of the design process, showing how the epistemological and cognitive analysis underpins the design choices.First, we will show how general theories on education underpin the design tools that we use to undertake the "fine grain" analysis of the specific contents to be taught in the TLS following the four first phases of DBR.The application of the first four phases of the DBR to the design of the TLS we have presented in the previous section requires the following: knowledge of the curriculum aims according to the target students ("focus" phase); knowledge of the previous research to identify possible learning difficulties and consider feasible teaching strategies ("understand" phase).All of the above must lead to the definition of specific learning aims ("define" phase) and the teaching strategies ("conceive" phase).In our case, we have followed this process to articulate the design of the TLS in the context of a transformed calculus-based physics course for first-year engineering and science degree students at the University of the Basque Country.

A. Defining learning indicators
As we have already mentioned, the TLS we present in this paper addresses the topic of work and mechanical energy for an introductory physics course (focus phase).To design this TLS we have taken into account the relevant learning difficulties reported in the literature (understand phase, see Table II).
The content of the topic is analyzed epistemologically by looking at its historical development, the difficulties that the scientific community had to overcome, and the arguments used to build new concepts and explanatory models [41] (understand phase).This epistemological analysis allows us to put forward a grounded definition of learning indicators (define phase).This is to say, it enables us to justify the choice of those indicators on the basis of disciplinary epistemological evidence while avoiding definitions based on the designers' idiosyncrasies or traditional curricular choices.The notion of "learning indicator" allows us to measure learning with a concrete evaluation tool (questionnaires, reports, etc.) which must be accompanied by their evaluation protocols, hence, the term "indicator" rather than "aim".Moreover, these indicators can be used by designers to sequence the main steps that the teacher will have to consider.
In Tables I, II, and III we provide an example of how to select learning indicators, defining learning demands and activities for introducing the concept of work and the limitations of the kinetic energy theorem, and using the general didactical tools mentioned in the phases understand and define of DBR.The teaching of work has generated a number of discussions in the literature [41][42][43][44].In the history of physics, the concept of "work" had empirical meaning from the beginning.It was constructed by engineers to measure the "labor" work of motor engines and animals.Very soon, however, this initial meaning seems to vanish [45].Many university textbooks over the last decade of the nineteenth century introduce work as a purely theoretical magnitude, without connecting it to the work of motor engines.For example, Sturm [46] and Poncelet [47] defined work with the integral R Tdr, where ds was the arc covered by a mobile under the action of a force P during the instant dt, and T was the tangential component of the force P. The meaning of this work was connected

Elements from the epistemology of physics
Learning indicators -Establishing the elements that will be a part of the system.This makes it possible to define external forces on the system and which types of energy change, or do not change.The setting of the problem leads to the selection of strategies to solve it.
i.1.To define and apply the definition of work as a dot product of two magnitudes.i.2.To define system and to understand work as an energy transfer, which implies a variation of the energy of the system.-Quantifying external work.In particular, recognizing the work done by friction forces as a form of energy transfer.
i.3.To recognize that the kinetic energy theorem is a particular case of the generalized principle of work and energy.
TABLE II.Learning demands for the three learning indicators defined in Table I.

Learning difficulties Learning indicators
Difficulties on defining external work in the generalized principle of energy.They are, partially, due to i.1; i.2 -Difficulties in the scalar product of the definition of work [44].
-The lack of an explicit definition of the system to which it is applied [43].Difficulties on defining the types of energy of the system.The definition of the types of energy depends on the selected system.For instance, the definitions of potential energy and external work [41]. i.2 Misinterpretation of the definition of internal energy of the system.Difficulties to distinguish external work and work done by dissipative forces [41].
i.2; i.3 with the changes of the quantity of motion (mv) or its living force (mv 2 ) of the object.On the other hand, today's common textbook introduces the concept of work with a discussion of a force ⃗ F applied to an object, which then moves through some displacement Δ ⃗r.The mathematical expression of work is represented by W ¼ ⃗ F • Δ ⃗r.However, many textbooks do not discuss explicitly the different components of the definition, such as "displacement of the force" or "acting forces on an object."This vagueness leads to conceptual difficulties in the study of mechanics.In the last decades some studies have shown specific aspects of the definition of work and they define some specific characteristics that the concept has in contemporary physics: -The concept of work depends on the concept of force and a considerable effort must be made to differentiate the scientific concept of work from the everyday meaning of the word (see Aarons and Jewet in Ref. [41]).
-Work is a way to change the energy of a body and what we indicate as work in equations corresponds to the amount of this change [see Mamallinckrodt and Leff in Ref. [41].
-The amount of energy variation is measured by the work equation, which is the scalar product between two vector quantities [see Domenech et al. and Bächtold and Guedj in Ref. [41].
It is necessary to analyze in detail the processes in which the work is involved.As the theoretical framework of physics states, while energy is associated with systems, work is defined in the transformations of the energies involved and, therefore, associating it with a system has no meaning.
Taking into account the above epistemological analysis, we define the learning indicators for the concept of work.Table I provides an example of learning indicators defined for a TLS on mechanical work and energy designed for an introductory course on physics to be used during the first year of science and engineering programs (combination of the understand and define phases).
Table I includes the starting point of the TLS on the definition of work and it includes the cases of the kinetic energy theorem and establishes its limits.The second column defines the physics knowledge that students should acquire as a result of the teaching with the TLS.In the next section we complete the definition of learning indicators with the tool of learning demands that takes into account the difficulties that students have to learn the contemporary meaning of work.

B. Framing a teaching sequence
Research has established that students have their own ideas and reasoning strategies to explain the nature of work and energy (previous ideas used in the socio-constructivist theory of learning).These ideas are difficult to change and tend to persist after instruction.In the case of learning about mechanical work and its relation to the change in energy, the learning of the notion of "system," the use of mathematical operators in the definition of work, and the differentiation among types of energy present serious learning difficulties for most students [41][42][43].As indicated by Leach et al. [13], it is necessary to take into account the gap between students' ideas and learning indicators (the learning demands).The dimension of the gap they need to bridge to achieve meaningful learning, as expressed by the indicators, will determine the strategies to be used in each case (see Table II; another combination of the understand and define phases).
The TLS aims at supporting students in constructing meaning on the relations between work and energy centered on the notion of system.Combining physics epistemology, a socioconstructivist learning approach, studies on metacognition, and the influence of the students' attitudes and interest, the sequence is based on a series of problems the resolution of which allows students to achieve the defined learning indicators.The structure of the TLS is developed during the conceive phase.The sequence includes a series of problems and activities summarized in Table III.The columns of this table refer to (i) the sequence of problems, the resolution of which encompasses the coherent body of knowledge to be taught and learned; (ii) the learning indicators that include the skills to be developed and used, alongside the conceptual knowledge to be learned comprehensively; (iii) didactical strategies; and (iv) activities and tasks.
The design also indicates that activities will take place in small groups where students, through their interaction, will construct their own points of view, followed by a whole group critical revision of the reports written by all groups.The teacher will lead this process (in accordance to socioconstructivist theory) and guide it to construct scientific knowledge.

IV. INSTRUMENTS FOR DATA COLLECTION AND RESEARCH RESULTS: EVALUATION OF IMPLEMENTATION AND REFINEMENT OF THE TLS
As a product-oriented project, one of the essential characteristics of TLS design and evaluation projects is the reelaboration of the teaching sequence based on empirical data obtained during its implementation in an educational intervention.The TLS design must be confronted empirically in the evaluation of the proposal itself and the achieved learning results.DBR methodology identifies the moment for evaluating the achievement of the TLS aims ("test" phase) and the need to incorporate the results of that evaluation in the redesign of the proposal.However, it does not define the research tools to be used in the evaluation.These tools have to be chosen by the researchers according to the specific characteristics of each case.
In accordance with Nieveen [36], our evaluation proposal uses qualitative tools, such as the "teacher's diary," the "students' workbook" and the "external evaluators' report," for the first dimension of analysis of the sequence (analysis of the quality of the sequence).The "teacher's diary" is a classroom diary that teachers implementing the TLS use to communicate with other members of the research team on how they have used the TLS in their lessons.According to Carr and Kemmins [48] this diary allows teachers to improve their understanding of their classroom, practice, and the context in which such practice takes place.This tool is particularly useful to describe "practical problems," understood as conflictive or dialectic situations that take place during specific didactical situations [49].We agree with other authors that the presentation of tangible classroom conflicts is helpful when making decisions leading to the improvement of the TLS implementation.
The "external evaluators' report" evaluation tool is a classroom observation report filled out by a member of the research team, focusing on whether the teacher follows the aims of the TLS activities.Reports are made of some lessons and teachers are not notified beforehand of which lessons will be observed.
Worksheets are a resource for fostering active learning in the classrooms is that can be used in different ways.These are versatile tools and are relatively easy to use [50].To achieve good learning results, it is necessary to take into account the gap between the conceive phase that produces the intended TLS and the build phase that requires students to comprehend the aims of the activities they carry out.Students complete the worksheets in small groups, discussing and posing questions about the worksheet content, using an interactive style which emulates tutorial learning [22].
These tools are used as a source of data and to analyze if the aims of the activities are perceived by the teachers implementing the activities in their lessons as they were intended by the designers, as well as to detect the difficulties encountered in implementing a sequence with innovative contents (Difficulties in writing a new sequence with innovative contents, see a.3 in section "6.test").Furthermore, the obtained data inform us about the problems related to the clarity of the activities to be carried out by students (Problems of the sequence related to clarity of the activities for implementation by students, see a.1.in section 6. test) and the time-related difficulties of implementing the sequence (Difficulties related to time for implementing the sequence, see a.2 in section 6. test).We have chosen qualitative research tools here because our study is exploratory in nature.This is to say, our aim is not to obtain generalizable results on the effectiveness of the designed TLS but to ascertain whether the proposed methodology (DBR) is helpful for overcoming some of the identified difficulties of the design, evaluation, and redesign of TLSs.To that end, it is essential to gather information on the process that will allow us to study how the selected methodology helps us to address these difficulties: the incorporation of theoretical references in the TLS design, the evaluation of the TLS effectiveness and the use of the evaluation results to redesign the TLS.This paper presents a case (a TLS on a topic of the introductory physics curriculum), which could be recognized by teachers as a generic situation.
In the second dimension of the analysis of learning (see section 6. test), we use quantitative research tools such as questionnaires with open-ended questions on the understanding of concepts and theories (pre and postquestionnaires for control and experimental groups), and tests including problems on the learning of laws and the acquisition of science skills (post-test for the experimental group).
On the basis of collected data, we will infer problematic aspects of the activities.Following this analysis, we will define types of students' difficulties (metacognitive difficulties, learning difficulties, related to interpretation and comprehension of information, etc.) and we will proceed to introduce modifications to the activities and their sequencing (see Table IV).The analysis of the data allows us to redesign the TLS according to the two dimensions we intend to evaluate.Particularly, the results of the evaluation can influence aspects of the redesign of TLS such as rewriting parts of the text, analogies used, and the general approach of the TLS; resequencing of activities; redesign of images and figures; restating previous requirements for the sequence and its activities; modifying the format (worksheets, classroom response system), etc.The analysis of the results provides designers with feedback on the validity of the TLS and its theoretical assumptions.This improves the probability of finding an effective design that can be verified afterwards through the final evaluation.
The first column in Table IV shows the instruments that have been used to evaluate the "design quality" dimension.The second column specifies the instruments used to evaluate the "learning achieved" dimensions.The third column shows possible kinds of re-design of the TLS.Some examples of changes made are described below.Table V shows an example of the change made in a question due to difficulties that we have termed students' "metacognitive difficulties" during the TLS implementation.We understand as metacognitive difficulties those related to the students' understanding of the aim of the activity.Students tend to have problems identifying the aim of the activities they engage with even when this aim has been explained to them [51].A description of the students' sample and the context of the TLS implementation can be found in Ref. [52].
Students' standard answers to the activity A.10, gathered from their workbook, during the implementation of version 1 of the activity, showed that they did not reach a clear understanding of the aim of the activity.An example is the following: "Work is equal to the variation of kinetic energy.The kinetic energy of the ball is 1 2 mv 2 ¼ 2.25 J." In these standard answers, students solve the question using the equation, in a correct or incorrect way, but without defining the ball system and without analyzing the forces acting and the process of work and changing energy.These elements were included in the objective of the A.10, but students do not consider them.So we decided to redesigned the activity posing to students a more specific task in a worksheet (see second column of Table V).The corresponding modification of the activity was to make the objective more explicit by adding a worktable.This change leads to students focusing on the objective when answering the question.An example of students' standard answer is the following: "We take as a system the ball, therefore, the forces are those applied on the ball and the space is the distance made by ball with the hand of player.We assume that the path of the ball in the player's hand is horizontal (there is no variation of potential energy).We will not calculate the work as force per space, since we do not have data on the force that the player applies to the ball, nor the space traveled.However, the problem gives us the kinetic energy variation.The ball passes from 0 m=s to 30 m=s.There are no other types of energy and, therefore,W ¼ 1 2 mv 2 ¼ 2.25 J. Let us TABLE V. Students' metacognitive difficulties encountered when implementing the TLS and its redesign (rewriting and rethinking of the approach of the activities).
assume that there is no energy dissipation due to heat loss or ball deformation."It is necessary to emphasize that the great majority of students obtain a correct numerical answer in the A.10 in both versions.However, the way to address and resolve the issue is radically different.In version 2, students should think about the system and the different concepts that appear in the activity.This prepares students for their own problem-solving skills that will be needed to tackle not-sosimple activities.
In the evaluation of the students' conceptual understanding we collect their progressions and difficulties related to the learning indicators defined in Table I.That objective is to evaluate their learning to detect weak points in the design of the activity or in the sequencing of the activities.We will show an example related with the learning indicator i3.This is an indicator of an issue with which students enrolled in a typical module of introductory physics have serious learning difficulties [53].Therefore, we are interested in knowing if the activities included in the TLS have an effect on improving the learning of this objective.To do so, we used several questions in a test completed by the students after the implementation of the TLS.An example is developed in some detail below (see Fig. 2).
Table VI shows the results for Q5 after the implementation of the TLS containing, in the first column, the mean of the pretest results for the four groups (two experimental and two control) during the two years of application of the sequence (no statistic differences between the groups).The pretest and post-test had similar but not identical situations so that students tackled each question once.The second column shows the mean of the post-test results for the control groups for the years 2014-15 and 2015-16, as there are no significant statistical differences between the two years.The third column shows the results of the post-test of the experimental group for version 1 and version 2 of the activity in two consecutive years.We found an increase in correct explanations when using version 1 (17.5%) in relation to traditional teaching, which was encouraging, but we were seeking results for more than the middle of the cohort.
The students' answers to question Q5 showed a fragmentary learning of the relations between work and energy, even though some students give correct answers such as: "No, it depends on the system we analyze.If we throw a ball and we analyze the system "ball", the Earth makes external work, but if we analyze the system "ball-Earth", there is no external work.A fraction of the students (category A.2, TLS version 1) focused on describing particular cases of the relation between work and energy such as work and kinetic energy, work and potential energy, etc., for example, "Mechanical energy is the sum of kinetic energy and potential energy Em  Very few students related work by friction with variation of internal of the system: "When the energy is conservative this is fulfilled but when the energy is not conservative then it is not equal." Moreover, a significant percentage of students (24.0%) following a traditional teaching approach assigned a character of general principle to the relation between work and energy: "Yes, work is always the variation of kinetic energy.In a competition, when the runner changes his speed, this involves work and this work is usually the variation of kinetic energy." This confusion diminished dramatically with version 1 of the TLS (5%).Nevertheless, the number of answers referring to general theoretical arguments based on the general principle of work and energy did not increase significantly (9.5%).Consequently, we decided to change the order of the activities so that the main line of reasoning would lead to considering the general principle from the start and to analyze particular cases farther on.This is to say, reversing the order that is used in the traditional teaching approach and in version 1 of the TLS.
Results obtained after the implementation of version 2 of the TLS (Table VI) show that the modifications that had been introduced resulted in a significant increase in the students' explanation using scientific arguments based on the generalized principle and, therefore, of higher cognitive quality (40.0%).Moreover, taking into account correct explanations based on particular cases, almost three fourths of the students (74.0%, version 2 versus 57.0% in version 1) achieved a satisfactory or almost satisfactory learning of the quantitative analysis of the relations of work and energy in mechanics.
To redesign the TLS is it essential to take into account the evaluation results, to know if the expected results, according to the aims what were set, have been achieved.Moreover, it will be necessary to analyze the different tools that have been used to collect data: the teacher's diary, worksheets and clickers, and pretest and post-test.Triangulation of data collected with these three types of tools is used to make decisions in relation to the redesign.The teacher's diary collects data on the development of the activities over time as well as any incident that might have taken place during the lesson.The first type of information is useful as a guide to make minor adjustments but not to address aspects of the design itself.There are situations in which students show a more than usual need for clarification or for posing questions in relation to the performance of the activity.The data collected through worksheets and clickers during the times when these incidents take place are of particular interest f redesign.If the students' answers to the questions referred to in the teacher's diary have the expected quality and the answers to the post-test related to the aim of that activity are correct in a percentage exceeding 50%, it is usually enough to adjust the proposed activity by changing the text, the graph, or the visual representation included in it.However, if the students' work and their results in the post-test suggest that the students have not achieved the expected results, the activities related to the unfulfilled aim will be included in the redesign of the TLS.

V. DISCUSSION
Proposals of TLS design found in the literature constitute a relevant body of knowledge but despite a movement towards an increasing explicitness on the process followed for TLS design [1,2,3,15] most of these proposals do not yet provide a principled approach to the design with a clear iterative process.The example of design we have presented in this paper is largely based on these existing proposals but we have focused on exemplifying the use of a particular design methodology (DBR) which we suggest could be used as a general methodological framework for different models of TLS while providing a common ground to help research on TLS to develop into a research program.The examples showed in Secs.III and IV suggest that using DBR as the methodology to design, implement, and evaluate a TLS can be useful to improve the design and redesign of the TLS.We have faced the problem of the relation between the evaluation of the TLSs and their iterative design.Through our general design proposal following DBR and with specific examples from our research, we have shown how our designed TLS was evaluated and how this evaluation was used in the iterative design process (Tables V and VI).A particular contribution of our study is the evaluation tools, which cover several aspects of the design that need to be assessed and reworked during this iterative process with a mixed-methods approach (Table IV); this has been suggested as desirable in the literature [12].Our results show learning gains that are connected to the redesigning of the TLS.Our evaluation approach has allowed us to show significant improvements using a combination of qualitative (students' worksheet, teachers' diary, external observation) and quantitative (questionnaires, problem-based test).
This study provides the community of teachers and designers with a viable proposal for a common methodology for the design, implementation, and evaluation of the TLS, which can facilitate establishing comparisons among different TLS materials.We think that the general implementation of a systematic evaluation of TLSs would provide designers with empirically grounded solutions to be used during the design of future TLSs.This process of building upon known solutions to common problems would help to determine which is the most efficient TLS for a given topic in a given context, as well as better determining the difficulties still to be solved.
We have purposefully limited our design proposals to elements with enough backing to be accepted in the community as common design elements.Such elements constitute the design tools to be included in a DBR methodology for materials design, which will not amount, on their own, to a complete TLS.The final result will require making contextual decisions regarding both the use of design tools and the implementation of the resulting TLSs, understood as specific examples of a more generic TLS for a particular topic.Contextual elements include educational settings and implementation decisions, as well as teachers' professional knowledge.The former can be divided into two sets: one subset of elements explicitly referred to in the materials included in the TLSs; the other including elements contingent on contextual circumstances that, at least at present, are not supported by empirical research that would allow us to make a principled choice in the design process.The latter will become teachers' personal choices which might be underpinned by professional tacit knowledge but which cannot be included in a TLS as we understand it.
To sum up, in this paper we have developed a way of using DBR as a methodology for TLS design and have provided a brief worked example following this methodology with a particular focus on its evaluation and it subsequent redesign.We do not suggest this is a final result but we hope it might be a fruitful contribution to shift what is now a significant but disparate area of research into a research program that can constitute a central component of the field of science education.

TABLE I .
Learning indicators to define the work concept and the limitations of the kinetic energy theorem.

TABLE III .
The section about "How can work done on a system of particles and the energy that is transferred be measured?"from the teaching-learning sequence on mechanical work and energy.

TABLE IV .
Instruments for the iterative development of the TLS.

TABLE VI .
mg h.Kinetic energy is the energy a body has because it is moving.Potential energy is the energy that a body has simply because it is at a certain height."Results to the question Q5 after the implementation of the TLS.
and S. Kemmis, Becoming Critical: Education Knowledge Action Research, (Routledge, Taylor & Francis, London, 1986).[49] M. A. Zabalza.Los diarios de clase.Documentos para estudiar cualitativamente los dilemas prácticos de los profesores, The class diaries.Documents to study qualitatively the practical dilemmas of teachers (Barcelona, PPU, S.A, 1991).[50] D. L. Leslie-Pelecky.Interactive worksheets in large introductory physics courses, Phys.Teach.38, 165 (2000).[51] D. F. Treagust, G. Chittleborough, and T. L. Mamiala, Students' understanding of the role of scientific models in learning science, Int.J. Sci.Educ.24, 357 (2002).[52] This research was conducted with university students in first year physics courses in science, and engineering at the University of the Basque Country (UPV/EHU).The questions were administered to students who had already studied the topic of work, and energy in their physics lectures.Moreover, all participating students had taken at least two years of physics in high school.The students in the Basque Country passed the national standard exams in Spain to enter University to study science or engineering.In addition, in the first course the FCI questionnaire was applied in the groups at the beginning of the course [D.Hestenes, M. Wells, and G. Swackhamer, Force concept inventory, Phys.Teach.30, 141 (1992)]; The results showed that the groups in the first year were homogeneous in terms of knowledge about mechanics.The students knew laws and equations from previous courses, but they showed a poor conceptual understanding.The mechanics curriculum was similar to those followed in textbooks such as Tipler and Mosca, Physics for Scientists and Engineers (Macmillan, London, 2007); or Fishbane, Gasiorowicz, and Thornton, Physics: For Scientists and Engineers with Modern Physics (Prentice-Hall, Englewood Cliffs, NJ, 2005).Students spent three week in the topic of "Work and Energy."They received 3.5 h of lectures and 2 h in the laboratory per week, for 14 weeks, on mechanics in the first semester of the first year of their studies.Experimental and control students spent the same time in studying the sequence.The control students followed traditional teaching with lectures, whereas experimental students used the newly designed TLS.[53] J. Gutierrez-Berraondo, K. Zuza, G. Zavala, and J. Guisasola, University students' ideas about the relation between work and enegy in Mechanic at introductory physics courses, Revista Brasileira de Fisica, http://dx.doi.org/10.1590/1806-9126-RBEF-2017-0131, 2017.