Assessing student understanding of physical hydrology

Introduction Conclusions References


Introduction and objectives
Hydrology, as a field, has become increasingly interdisciplinary and technologically complex, and as a consequence there have been calls for examining, evaluating and enhancing hydrology education at the upper division and graduate level (Bourget, 2006;Ngambeki et al., 2012;Merwade and Ruddell, 2012).Hydrology education at this level is primarily intended as preparation for either applied or research careers, although not all students in these courses will become hydrologists.In an early step toward examining the field, Groves and Moody (1992) performed an extensive survey of topics covered in hydrology courses.The results were intended as a resource to the community, rather than an indication of what should be taught.More recently, Wagener et al. (2007) called for the hydrology education community to "analyse, synthesize, and unite hydrology education".These authors surveyed university hydrology educators about current teaching methods and ways that curriculum and instructor preparation could be improved.They found identifying "common principles, core knowledge, and approaches" to be a major challenge within the hydrology community.
The work described here was undertaken as a first step in the effort to articulate this common knowledge base and identify effective instructional strategies to help students develop it.Specifically, our objectives were to (1) articulate, and engage the larger community in a debate about, the overarching goals of physical hydrology, and (2) develop a method of characterizing and assessing advanced (upper division and graduate) students' holistic understanding of physical hydrology.Beyond the immediate study, the longterm goal is to use this method to assess the effectiveness of curriculum modifications.
Prior to assessing any way of teaching hydrology, it is first necessary to provide a yardstick against which the outcomes can be measured.In curriculum development, this approach is often referred to as "backward design" (Wiggins and McTighe, 2005), i.e., clearly articulating the goals and how they will be assessed before designing the curriculum.A further goal is to "identify any misconceptions, preconceptions, and difficulties associated with the specific subject" prior to the curriculum development effort (Ben-Zvi Assarf and Orion, 2005).Following these steps, we proposed a set of overarching goals for instruction in physical hydrology and prototyped an instrument to assess them.
Our instrument employs open-ended responses and evaluation with a rubric.This is in contrast to assessments requiring dichotomous (right/wrong) responses, e.g., the Geoscience Concept Inventory (Libarkin and Anderson, 2005).We feel that such a standardized test is unlikely to be of use to the hydrology community at this time given that clear goals for instruction have not yet been identified and, consequently, student understanding has yet to be characterized.Open-ended responses provide a much richer source of information about student thinking.
After describing the research setting, we discuss the learning goals we have identified, as well as the warrants (justifications) for them.We then describe the development of our assessment instrument and its accompanying rubric.Finally, we report and discuss results from the pilot sample of students, followed by the conclusions we draw from this work.

Setting and participants
The research we report here took place in two sections, one upper division undergraduate and one graduate, of a physical hydrology course, taught by one of the authors (MBC), at a large research university.The course is taught in the Department of Geological Sciences.It is cross-listed as an upper division undergraduate and a graduate course.The requirements for the two courses (homework, exams, projects) were essentially the same; the graduate students were occasionally assigned an extra homework problem.For undergraduate students majoring in Geological Sciences, the prerequisite is Calculus 1 and a lower division (introductory) hydrogeology class; there is no prerequisite for the graduate students.The course is required for (1) a BS in Geology with Hydrogeology Emphasis and (2) a BS in Geosystems Engineering (a hybrid Petroleum Engineering and Hydrogeology program).The course is an elective for the BS in Environmental Science.The graduate students taking the course are typically from the geosciences or engineering, but occasionally students from other areas (geography, biology) take it on a credit/no credit basis.Thus, the undergraduate students are required to have a stronger background in terms of coursework, but the graduate students are expected to have more experience.
Both sections met three hours per week, and there was no laboratory or field component.Although many sources were used, the primary text was Dingman's Physical Hydrology, 2008 version.An outline of the course topics is given in Table A1.Students completed homework sets, typically involving calculations or modeling processes with Excel spreadsheets, as well as synthesis questions requiring students to describe processes or trends identified in the quantitative exercises.In addition, students worked in groups to complete a project requiring analysis of actual hydrological data to answer a question.All students were given the opportunity to consent to participate in the research and fifteen of the undergraduates and ten of the graduate students consented.The participants were roughly equally split between male and female students.

Learning outcomes for physical hydrology
As a starting point, we reviewed the learning goals for the physical hydrology course (identical for the undergraduate and graduate sections) as presented in the syllabus.These were -quantitative process-based understanding of hydrologic processes; -experience with different methods in hydrology; -learning, problem solving, communication skills.
The first two goals clearly define outcomes related directly to hydrology, whereas the third bullet consists of what have been labeled "Twenty First Century Skills"; although they are certainly critical outcomes in all STEM education, they are not unique to hydrology.We chose to focus on the hydrology goals in developing our assessment, although it arguably also demands evidence of learning, problem solving and (given that it was an open-ended assessment requiring students to represent their thoughts in words or illustrations) communication skills.These first two goals might be translated as "knowing what hydrology as a community has come to understand through its collective work" and "knowing how to apply that knowledge to answer questions through hydrological research or solve problems in hydrological practice".As such they represent a logical parsing of the hydrological enterprise, with the distinction of the focus on processes.This focus on process is supported in previous scholarship on hydrology education.For example, Nash et al. (1990) listed "to develop and improve awareness of the totality of interconnected (mainly physical) processes involved in the hydrological cycle" first among the goals of hydrology education (emphasis added).

Key questions and tasks for the assessment tool
We then engaged in the process of "unpacking" these hydrology-specific goals and converting them into measurable, i.e., performance related, objectives.In other words, we considered the question: what should students who have achieved these goals be able to do?We argue that an understanding of hydrological processes requires that students be able to do two things: (1) describe the significant processes in the hydrological cycle in detail (in their own words) and ( 2) describe (again in detail and in their own words) the laws governing these processes, i.e., governing the existence and movement of water in natural systems.We translated these two objectives into equivalent open-ended questions: Q1.In your understanding, what are the most important processes involved in hydrology?Describe them in as much detail as you can.
Q2.What are the relevant physical laws that govern hydrology and how do these laws determine hydrological processes?Describe them in as much detail as you can.
Appropriate responses to these questions might demonstrate familiarity with the scope of collective knowledge of the hydrology community, but still do not demonstrate the ability to apply that understanding to answer questions or make quantitative predictions, as called for by Manduca et al. (2008).Dooge (1986) posits that hydrology involves developing both explanatory theories (i.e., the relationship between the laws and processes represented in the response to Q1 and Q2) and predictive models.To assess students' ability to solve problems and make predictions (simultaneously demonstrating experience with different methods in hydrology) we crafted a third question: Q3.You have been hired as a consultant by ... to (1) assess how urbanization and the current drought have affected a local spring and ( 2) predict what the effects will be in the future if the drought continues.What information would you need to gather?What measurements would you make?What analyses would you perform?Be as specific as you can; feel free to continue on the back of the page.
These three questions made up the instrument.Student and expert responses from the pre-assessment were then evaluated qualitatively to develop a rubric, which was ultimately used to score the responses, as described in the next section.

Rubric development
We used a rubric development process similar to the one used in Covitt et al. (2010).All students in both sections of the physical hydrology course during one semester completed the assessment described above as a pre-test on the first day of class.In addition, one of the authors (MBC) and seven of his graduate research assistants provided expert responses against which to judge the student work.Responses from the experts and those students who had given informed consent were then analyzed using a grounded process (Corbin and Strauss, 1990), meaning that the artifacts were reviewed without prior expectation as to what they should contain, that is, they were not evaluated against any set of standards.We interviewed a subset of students about their responses to the questions to check their interpretation of the questions and our interpretation of the responses, and several modifications were proposed to make the intended interpretation more apparent.
In a first round of "open coding", two of the authors (JAM and AJC) independently reviewed the artifacts and created codes to describe responses.The codes were typically commonly occurring terms (e.g., "water cycle") or characteristic descriptors such as "phase changes" to describe responses that dealt with evaporation or condensation.
The open codes were then compared and a common set of descriptive codes were negotiated.The codes were then grouped by themes to build a model of the phenomenon in question, i.e., to characterize student understanding of physical hydrology.The expert responses were used as an "upper anchor" (the level of understanding at which students are ideally intended to leave the course) for the progression of understanding.Pre-college student conceptions of the water cycle, as reported in the literature, were used as the "lower anchor" (what students could be expected to bring to the course) (NRC, 2007).
The resultant codes were then grouped into four categories: no indication of understanding, some recognition of concepts, basic understanding, and full understanding."Full understanding", while not really requiring the expertise of a working hydrologist, would be expected of an advanced graduate student.The "no understanding" category was reserved for students who did not respond at all, responded with "I do not know" or the equivalent, or gave a completely unrelated response."Some recognition" categorized responses that contained some elements of a correct response, but were incomplete in some significant way."Basic understanding" indicated signs of knowledge not likely to have been obtained other than in a hydrology course.Responses that fell in this category were comprehensive, if skeletal, for example, giving an exhaustive list of processes in the hydrological cycle in response to the first question, but not describing how they related to the presence and movement of water.
A draft rubric was shared with the research team that had provided the expert codes (one of the authors (MBC) and his graduate research assistants) who then used the rubric to code a sample of the artifacts from the first implementation and make suggestions for revisions and clarifications.The resulting rubric was then used by two of the authors (JAM and AJC) and five members of the research team independently to code a subset of the artifacts.The team then met to negotiate a consensus score on each item; further clarifications of the rubric and revision of the assessment were made.
For example, some students appeared to have read the first question as asking them to give the "most important process" rather than "processes" and struggled to decide which element of the hydrological cycle was most important.Likewise, at least one student seemed to interpret the phrase "in your understanding" (intended to make students understand that we were looking for their own formulation, rather than a set, memorized response) to mean "in your opinion" or "in your own research area of interest" and reported the process most relevant to her/his own research.This led to a revision of the first question: Q1.What are the important physical processes involved in hydrology?Describe how they affect hydrologic systems in as much detail as you can.
Finally, a subset of the team (three graduate researchers) independently coded the full set of pre-and post-artifacts (blinded as to whether they were pre-or post-, graduate or undergraduate), resulting in 89 % inter-rater agreement on the pre-tests and 83 % agreement on the post-tests.The full rubric is given in Table B1, as are exemplar responses for each category.

Sample pre-course results
Here we present the responses of students from the pilot implementation to illustrate the use of the rubric and to characterize student thinking prior to and following a physical hydrology course.When asked what were the "most important processes involved in hydrology", most students interpreted Q1 in terms of physical processes affecting the water cycle, the primary organizing framework for hydrology, as intended.
For the first category, students simply cited the water cycle or mentioned the sun, precipitation and/or evaporation, with no mention of surface/groundwater interactions, infiltration, transpiration, etc., displaying a lack of recognition of groundwater and biological interactions common in novice students (Shepardson et al., 2009;Siegel, 2008).
Mention of changes of state was common, which is to be expected given that this topic is emphasized at the precollege level, e.g., in the draft Next Generation Science standards (Achieve, 2012) as well as many state standards.It was also common for students to mention issues related to technology, such as drilling, or water management (Nash et al., 1990).Some respondents interpreted the question to broadly include the activities of hydrologists as opposed to natural processes involved in the hydrological science (leading to a debate about possible revision of Q1).For example one student gave "to estimate the flow of subsurface water as accurately as possible" as an important process.
On the pre-test, one student scored 0, indicating no response, on this question (Fig. 1a).Twenty students scored "1", indicating rudimentary understanding, four students scored "2", indicating a basic understanding, and no student scored "3".This indicates that, as might be expected prior to instruction, most students were solidly positioned at the "lower anchor" of the learning trajectory for physical hydrology, that is, they entered the course with the understanding typically achievable in pre-college education.An engineering-oriented interpretation of what was meant by "processes in hydrology" may have led some students to focus on career aspects rather than the scientific basis for hydrology, an issue that has been raised previously in the literature (Nash et al., 1990).
Student scores on this question improved on the post-test (Fig. 1b).On the 22 post-tests that were blind scored, eleven students demonstrated some recognition of concepts, nine students showed a basic understanding, and two students had a full understanding of the processes linked to hydrology.Half the students had provided evidence of the desired understanding (at the basic or full level); however, half still demonstrated only a rudimentary understanding.It should be noted, however, that as the post-test was ungraded, some students may not have devoted their full efforts to it, so these results cannot be viewed as a definitive indication of student understanding.
Results on Q2 (laws governing hydrology) showed the same pattern (see Fig. 1): on the pre-test, two students scored "0" (no response), 21 students scored "1", indicating rudimentary understanding, two students scored "2", and no student scored "3".On the post-test, again approximately half the students achieved the desired understanding: nine students showed some recognition of concepts, 12 students demonstrated a basic understanding, and one student exhibited full understanding of the laws linked to hydrology.
On Q3, no students scored "0" (i.e., all were able to submit a relevant response), eight students scored "1", 16 students scored "2", and one student scored "3".On the post-test, one student showed some recognition of concepts, 16 students demonstrated a basic understanding, and five students demonstrated a full understanding of the methods linked to physical hydrology.
As can be seen in Fig. 1, students were significantly better at responding to Q3 (the application) as opposed to Q1 and Q2, which were more abstract.Research has shown (contrary to the beliefs of many instructors) that students are often better able to bring resources to bear to solve contextualized problems when they are unable to deal with more abstract tasks (Nathan and Petrosino, 2003;Brown et al., 1989).

Discussion
In the last two decades, there have been a limited number of studies of student understanding of hydrology and how it evolves over the course of schooling.These have been concentrated primarily at the pre-college level (Dove et al., 1999, Shepardson et al., 2005, 2009;Dickerson and Dawkins, 2004;Dickerson et al., 2005;Ben-Zvi Assarf and Orion, 2005;Covitt et al., 2010;Schwartz et al., 2011), with a smaller number looking at introductory college courses (Dickerson et al., 2005;Smith et al., 2006;Sibley et al., 2007;Rappaport, 2009;Fryar et al., 2010;Rathburn and Weinberg, 2011) and a very limited number at the upper division undergraduate/graduate level (Dickerson et al., 2005).
Understanding of the hydrologic cycle (water cycle), especially its physical drivers, proved challenging for the participants in this study, as has been reported in previous studies.Difficulties included incorporating ground water (Dickerson and Dawkins, 2004;Dickerson et al., 2005;Shepardson et al., 2009), surface-ground water interactions (Siegel, 2008), perceptions of the hydrological system as static, i.e., not relating processes and drivers to flow of water (Ben-Zvi Assarf and Orion, 2005), and a non-systems approach, i.e., failure to articulate how the laws and processes related to each other (Kali et al., 2003).Students also frequently either neglected, or exaggerated biological and human interactions with hydrological systems (Dove et al., 1999;Shepardson et al. 2009;Covitt et al., 2010).For example, on the pre-test students rarely mentioned evapotranspiration, in which vegetation plays a critical role, but cited "human processes -such as filtering, preparing water to be tap water, or industrial processes" as the most critical aspects of the hydrological cycle.
It should be noted that student results on conceptual standardized assessments often do not parallel other assessments of course outcomes, most notably grades (Hake, 1998).Even students whose academic success has made them eligible for honors science coursework are not always able to respond successfully to conceptual assessments of subject matter (Thacker et al., 1994).On the one hand, this clearly indicates a limitation of this type of assessment.Students are not able to show the full range of what they know and are only able to perform in a limited time and on only one type of assessment (Hake, 2011).Education researchers, even those who argue the value of standardized assessments, recognize Runoff processes and stream flow 9 10 Watershed and stream network properties 9 11 Flood routing and stream flow analysis 9 12 Groundwater hydraulics 8 13 Groundwater hydraulics 8 the need for multiple forms of assessment for this very reason (Valenzuela, 2002).On the other hand, the results do give an indication of ways in which instruction might be redesigned to target particular difficulties, as recommended by Ben-Zvi Assarf and Orion (2005), and emphasize important themes in a coherent manner.

Conclusions and further work
In our small sample of students, the majority entered the upper division/graduate physical course with only a rudimentary understanding of the processes involved and the laws that govern how those processes relate to the presence and movement of water through natural systems.Understanding was somewhat higher (more students in the "basic" understanding category) in the "methods" dimension, but still not approaching the "full" level.After instruction, as it has been traditionally implemented at our university, student understanding increased as assessed by the rubric we have developed, with the majority of students falling into the "basic" or "full" categories.
It should be noted, of course, that this result has significant limitations, including the small number of participants, who were all from one institution, and the fact that the rubric was still under development.Nevertheless, the fact that we were able to get substantial inter-rater agreement among a group of experts from the discipline of hydrology, who were not themselves responsible for the development of the rubric, is significant.
It may be premature to propose a standardized assessment for hydrology education, given the "lack of current consensus on what constitutes hydrology knowledge and levels of knowledge" (B. Ruddell, personal communication, 2012).We do not propose this instrument, and its accompanying rubric, as a definite means of assessing student understanding of physical hydrology.Rather, we present it as a means of advancing the conversation about what the goals of courses

Methodology
Current water levels in the springs compared to previous years, measuring amount of foreign materials in the water that could be there due to runoff from urbanization of the city.
Historical precipitation and discharge data in the area would be helpful.Population changes in the area would also be important when understanding current conditions.Measurements like gauge height, amount of precipitation and water table height would give data on the area.I would analyze historical data and compare it to the present to notice any trends.If there are common trends throughout history, then there should be a way to make decent predictions.
Collect the meteorological data, e.g., temperature, humidity, wind, precipitation, and land data like soil and vegetation cover properties and hydrological data like stream flow and water table depth.To see the impact of urbanization or drought, we need to isolate each factor from other influences, so we need to collect observational data before/after the urbanization or drought.Also we can use hydrology models to modify the factor to see the individual effect.Finally the models could provide future projections if we assume the drought will continue.
in this subject (and hydrology education in general) should be.We intend for it to serve as a "strawman" proposal for the larger community to critique: if we have not identified critical goals in hydrology education, then what should be added, revised or deleted?Are there questions or tasks that could be used in place of those we have chosen to demonstrate student achievement of the goals?A well-designed pre-and post-assessment can be used to infer whether a given instructional intervention, e.g., a change toward a more student-centered and inductive learning environment (Ngambeki et al., 2012), has caused a change in understanding in a given group of students, but the results are not necessarily generalizable.Further, without broad consensus on the goals of hydrology education, the change might not be one that is meaningful to the larger community, and, without common assessment mechanisms, it will be impossible to calibrate and compare changes that result from different curriculum interventions.
As a next step, we will expand on this work by employing the assessment process vetted here to evaluate a curriculum intervention developed by one the authors (MBC).The curriculum intervention is based on student learning with a data and modeling driven approach (Merwade and Ruddell, 2012) using the COMSOL Multiphysics software (Singha and Loheide, 2011;Li et al., 2009).This intervention will be implemented in a future version of the same hydrology class, and pre-/post-test results will be used to assess and report its effectiveness.