Computational Science & Engineering Research Institute

Solving complex science and engineering problems using high performance computers
In a fluidized diluted granular flow, grains behave like gas molecules. Applying the Bolztmann kinetic-collisional model to a granular flow. S. Dartevelle, Ph.D. student, Geological & Mining Engg. & Sci.	NASA Goddard Space Flight Center Beowulf Cluster. 64 2-processor PC's connected by a 100Mb/sec switch. P. Merkey, Research Ass't Prof., Computer Science Dept.
Evolution of a non-evaporating fuel spray. Colors represent droplet radii in microns. F. Tanner, Assoc. Prof., Mathematical Sciences Dept.	NSF Major Research Instrumentation Sun Enterprise. 12 Sun Ultrasparcs and a small disk farm. S. Seidel, Assoc. Prof., Computer Science Dept.

The discipline of Computation Science & Engineering is the application of computational technologies (computer hardware, software, networks, etc.) to current problems in science and engineering. Typically, these problem are in areas of national interest, including areas of economic interest and security, and areas of state-wide economic interest. Examples of specific problems being studied by the CS&E faculty at MTU are described below.

The CS&E Research Institute was created to:

• provide an academic environment in which researchers (faculty and students) can collaborate on problems and technologies of common interest,
• bring researchers together to develop large scale, long range, interdisciplinary, research programs,
• foster the development of the CS&E Ph.D. program, and
• provide access to medium- and large-scale computational facilities that researchers could not otherwise afford to acquire.

The creation of the CS&ERI was approved by:

• Curt Tompkins, President
  
• Kent Wray, Provost and Sr. Vice President
  
• David Reed, Vice President for Research
  
• Bruce Rafert, Dean of the Graduate School and Distance Learning
  
• Maximilian Seel, Dean of the College and Sciences and Arts
  
• Robert Warrington, Dean of the College of Engineering
  
• Linda Ott, Chair of the Department of Computer Science
  
• Alphonse Baartmans, Chair of the Department of Mathematical Sciences
  
• Ravindra Pandey, Chair of the Department of Physics
  
• Theodore Bornhorst, Chair of the Department of Geological and Mining Engineering and Sciences
  
• Michael Mullins, Chair of the Department of Chemical Engineering
  
• William Predebon, Chair of Mechanical Engineering - Engineering Mechanics
  
• Timothy Schulz, Chair of Electrical and Computer Engineering
  
• Steven Seidel, Associate Professor of Computer Science
  
• Phillip Merkey, Research Assistant Professor of Computer Science
  

A complete description of the CS&E Research Institute can be found in the CS&ERI proposal.

Phillip Merkey is the director of the CS&E Research Institute.

• CS&E problems cannot be solved in a laboratory. CS&E problems cannot be solved by direct ("wet lab") experimentation or by field observation. CS&E problems are too large to "fit" into a laboratory. The only way to solve them is to simulate them inside a computer.
• CS&E problems cannot be solved on your desktop computer. It would take years or centuries to solve CS&E problems on a typical PC. Much much more powerful computers are required. To solve the largest and most interesting problems, more complex computer systems are an absolute necessity.
• CS&E research is interdisciplinary. CS&E research is always a combination of sophisticated computational methods applied to the specific science and engineering problems at hand. CS&E researchers (students and faculty) have been trained in their application areas (the science or engineering problems), they have been trained in the latest computational techniques (i.e., computer program design and implementation), and they understand how to use the most powerful computer systems to solve their problems.
• CS&E research is different than research in computer science. Research in computer science is centered on problems such as how to build better, faster, and cheaper computers, how to design, implement, and maintain software. Some typical research areas in computer science are graphics, computer architecture, programming languages, operating systems, networks, databases, and the theory behind all of these topics. Computer science research is not concerned with physics problems, problems in civil or mechanical engineering, problems in oil field exploration, plate techtonics, or atmospheric chemistry. On the other hand, CS&E research is very much interested in applying the latest computer science advances in graphics, computer architecture, programming languages and networks to problems in physics, civil and mechanical engineering, oil field exploration, and so on. The discipline of Computational Science and Engineering is the application of computer science to problems in the sciences and engineering.

• Important CS&E problems are:
  • Energy efficiency: combustion processes and emissions
  • Simulated crash testing of automobiles
  • Environmental studies: pollution in the ground, water, and air
  • Molecular dynamics: simulating the behavior of atoms and molecules
  • Computational chemistry: simulating chemical processes
  • Bioinformatics: genetic sequencing and recombinant studies
  • Weather forecasting, including hurricane and tornado tracking
  • Aeronautics: airframe design and dynamics
  • National defense and security interests: bio/chemical terrorism, intelligence
  • Nuclear stockpile stewardship

In most cases, computers used in CS&E research are ``parallel" computers made up of tens or hundreds of ordinary computers all connected by a network that allows the computers to ``talk" to each other while they work on solving a problem.

• Parallel computing: It's not the number of processors; it's the network. People working on a project team not only do work, but they cooperate with each other to coordinate their efforts. This requires that team members communicate with one another. The same feature distinguishes parallel computers from ordinary PC's. In the photos below it is important to notice more than just the number of processors (CPUs) that each parallel computer has. In most cases the photo at the right shows the network technology used to connect the computers. In some cases the cost of the network is about the same as the cost of the computers it connects. This is convincing evidence that a fast network is as important as fast computers.
• Parallel programs: A program for a parallel computer must divide the work between those computers. All of the computers work on the problem at the same time. Each computer does its share of work just as members of a project team all contribute to the final result.
• Parallel computers speed up the time to solution and parallel computers solve larger problems: These two advantages go hand-in-hand. If more workers are assigned to a task, then we can expect the task to be completed sooner. Moreover, each computer working on the task comes with its own memory. The more computers there are, the more memory is available, and since more memory is available, more data can be used to represent the problem. This means larger problems can be solved. Problems such as predicting global warming require computers with as much memory as possible.

Don't most researchers at MTU use computers? How is CS&E research different than what these researchers do?

We all use computers for word processing, web browsing and email. None of that begins to tax our computers. When was that last time that your own computer spent more than 20 seconds actually computing something? The only time that most of us usually have to wait for our computers is when large amounts of data (e.g., pictures, movies, web pages) are being moved from one place to another or while a large software package is being ``loaded". Desktop computers today are so powerful that for most day-to-day tasks (except 3-D game playing) they spend most of their time doing nothing at all. Most of the computers on our desktops spend most of their time waiting for us to do something!

In the laboratory, a researcher might use a computer to control the inputs to an experimental device or to automatically collect data during an experiment. Afterwards a computer is used to analyze the data that have been collected and to display the results using commercially available software. Sometimes the researcher writes a programs to analyze the data in a way particularly fitting to the science or engineering problem at hand. From the computer's point of view, none of the work it is required to do is at all taxing. The computer typically spends very little time doing this work; most of its time is spent waiting for the laboratory devices or for the researcher. Even though a computer and its software is an important laboratory tool, the particular computer is not critically important to the work being done. Any reasonably modern PC can usually do these jobs without breaking a sweat.

On the other hand, many CS&E researchers do not work in a traditional laboratory. They do not do experiments or make field observations. All of their research is done inside the computer. Their work is based on mathematical models of physical phenomena. A much of their research effort consists of turning those mathematical models into computer programs. Their computer programs simulate cloud formation, the flow of pollutants through the soil, the pressures on an aircraft wing, or the evolution of black holes. All of these "experiments" are artificial but, if carefully designed models are used, they mimic the real world so accurately and in so much detail that the results reveal what would actually happen in the real world. It is computational scientists who tell us that global warming is (or is not) happening. It is computational scientists who design new drugs without ever using a test tube. Many important problems today cannot be solved by traditional laboratory studies. The largest and most powerful computers are the only and best tools for scientists and engineers to use to solve these problems.

OK, now I understand that CS&E research is focussed on science and engineering problems that can only be solved using computers ...

... but there are so many different units at MTU with "computer" in their names. What are the differences between ... ?

• The CS&E Research Institute: The CS&ERI was created as a home for researchers to collaborate on computational problems and techniques of common interest, to facilitate the development of long range research programs, to support the CS&E Ph.D. prgram, and to provide access to medium- and large-scale computational facilities that would not otherwise be available.
• The CS&E Ph.D. program: The CS&E Ph.D. program is the primary educational component of the CS&E Research Institute. A more detailed description of the CS&E Ph.D. program is given in the next section.
• The Department of Computer Science: The CS Department offers B.S., M.S., and Ph.D. degrees in computer science. It is concerned with all aspects of undergraduate and graduate computer science education. It has established research programs in a number of areas including parallel computing and advanced architectures, compiler technology, graphics, distributed systems, and several others. Some of the CS Department's educational and research activities are directed toward computational science and engineering, but there are many other areas of computing technology that fall within the scope of the CS Department.
• The Computer Science Ph.D. program: The CS Ph.D. program was established in the fall of 2001. It was originally part of the CS&E Ph.D. program. Now that these two programs have separated, the CS Ph.D. program concentrates on graduate education in the CS research areas such as those mentioned above, and the CS&E Ph.D. program concentrates on interdisciplinary graduate education and research in the application of computational techniques to current problems in the sciences and engineering.
• The Computer Engineering component of the Electrical and Computer Engineering Department: The CE "department" is closely allied with the CS Department. Much of their curricula overlap. Some might say that computer engineering is more often concerned with hardware and computer science is more often concerned with software, but this is a gross oversimplification and in many cases simply incorrect. Here is an example of the close association between the disciplines of computer engineering and computer science. Special purpose computers embedded in "appliances" (anything from microwave ovens, cell phones, automobiles, to rockets) might fall closest to the domain of computer engineering. Designing programming languages, compilers, and software for those computers is likely to be the domain of computer science. Further discussion of these distinctions would be out of place here. The main point is that both computer engineering and computer science are distinct from computational science and engineering because CS&E research primarily focusses on problems in the physical world, not problems in the world of computers.
• The Information Technology department in EERC: IT is MTU's phone company, internet provider, and cable TV company. It is a business and service branch of MTU. IT provides internet connectivity across campus and to the outside world, but IT does not provide computational services to MTU researchers.

One of the primary goals of the CS&E Research Institute is to foster graduate research in computational science and engineering. This is being accomplished through the CS&E Ph.D. program.

• The CS&E PhD program is nondepartmental. The CS&E PhD program is one of two active nondepartmental PhD programs at MTU created under the "Ph.D. in Engineering" umbrella Ph.D. program originally established more than 10 years ago. (The other active program in this category is Environmental Engineering.)

• What does "nondepartmental" mean? There is no one department that "owns" the CS&E PhD program. Instead, it is directly responsible to the Graduate School. No other PhD program at MTU has this relationship; all other PhD programs at MTU are responsible to a specific "home" academic department. The Department of Computer Science is responsible for the day-to-day administration of the CS&E PhD program but, in principle, any MTU department may take on this responsibility.

• Any graduate-degree-granting department at MTU may grant a CS&E PhD. This is a tremendously important and distinguishing feature of the CS&E PhD program at MTU. No other PhD program at MTU has this feature. One consequence of this fact is that any MTU department can be a "home" to a CS&E PhD student. The home department provides the student's advisor, financial support, basic computational facilities, and all other support that is usually provided to students in the department's own PhD program. However, the student is subject to the degree requirements of the CS&E PhD program rather than the degree requirements of the PhD program of the home department. This frees the student and his/her advisor to construct a truly interdisciplinary degree program that best fits a computationally-intensive research emphasis.

• Recent history of CS&E PhD program. Two recent events have had a significant impact on the development of the CS&E PhD program.
  • The CS&E PhD program was originally created to house two groups of graduate students: those doing interdisciplinary research in computational science (as applied to problems in the sciences and engineering) and those studying current research in the discipline of computer science (as applied to traditional areas of computer science such as hardware, software, networks, etc.). In the fall of 2001 the Computer Science PhD program was approved by the Board of Control. Since then the CS&E PhD program has been able to focus solely on the mission of producing PhD graduates in the interdisciplinary area of computational science.

  • In the fall of 2001 the CS&E PhD program underwent an external review, as mandated periodically by the Graduate School for all MTU graduate programs. One of the primary recommendations of that review was that a CS&E Research Institute should be created to help foster growth in the computational sciences at MTU. This goal was accomplished in the fall of 2002 with the creation of the CS&E Research Institute.

  Summary: The CS&E PhD program has more flexibility and interdisciplinary breadth than any other PhD program at MTU. It is the only PhD program at MTU that reports directly to the Graduate School, and it is the only PhD degree at MTU that can be awarded by more than one department. A primary goal of the newly created CS&E Research Institute is to foster the development of the CS&E PhD program.

Current CS&E Projects

All of the computer equipment shown here is housed and maintained in the Center for Experimental Computation on the 2nd floor of Fisher Hall.

NASA: Earth and Space Sciences Support for NASA High Performance Computing and Communication Program, P. Merkey (PI), S. Seidel (Co-PI), $758,000.
Technical description: Popular description:
NASA cluster: A cluster built from 64 ordinary PC's.	Cluster interconnection network: The 100Mb/sec Ethernet switch that connects the PC's in this cluster.
The computer: You are looking at the "business ends" of 64 dual-processor PC's. Note that there is only one keyboard and monitor. All of these PC's are connected to form a single, parallel computer. They are connected by an ordinary Ethernet switch (right) that is just like the one that connects your desktop PC to all other PC's in your local network. This "Beowulf" cluster was provided to MTU by NASA Goddard Space Flight Center to support parallel computing projects in the CS&ERI.

Hewlett-Packard: UPC Technology Development Project, S. Seidel (PI) and Co-PIs P. Merkey and C. Wallace, $179,000.
Technical description: Over the past two year Drs. Steven Seidel, Phillip Merkey, and Charles Wallace have received funding from Hewlett-Packard to pursue four projects centered on the development of the Unified Parallel C programming language. UPC is a new language for high performance computing on distributed shared memory architectures. One of this year's projects continues the work begun last year on the MuPC run time system for UPC. New this year are projects to formally specify the UPC memory consistency model, examine programmability and usability aspects of UPC, and produce a specification for UPC collective communication operations. Popular description: Programming parallel computers is a more difficult task than programming ordinary computers because a parallel program has to control many computers at once. Many familiar programming languages used for scientific computing, such as C and Fortran, have been modified so that they can be used to program parallel computers. One of the newest parallel programming languages is UPC, which stands for Unified Parallel C. This language is currently being studied by research groups at UC Berkeley, George Washington University, and MTU. There is current interest in UPC because it appears to have a well designed collection of commands that allows programmers to more easily write parallel programs to solve complex problems. One of the current CSERI projects is to develop a portable version of the UPC programming language that can run on any Linux-based computer system. This version is called MuPC (roughly standing for "Michigan Tech UPC"). One of the reasons for developing MuPC is to help "spread the word" about UPC, so MuPC is freely available to the CS&E research community in the U.S.
The Cray T3E: These three cabinets contain 60 processors and lots of wiring to connect them all.	Inside: This is the inside of one of the cabinets. All of those cables are used to interconnect the processors and to keep them in step with one another.
The computer: In association with this work, the CS&ERI has received a Cray T3E supercomputer. This machine originally cost about $1.5M. It is a few years old now, but current models of this supercomputer maintain their place as the fastest machines in the world. This system was provided to MTU to support UPC-related research. The CSERI has made this system available to researchers at many U.S. institutions, including the UPC research groups at Berkeley and at George Washington University.
NSF: Sandu's CAREER and ITR funding: Sandu, et al., $450,000.
Technical description: Popular description:
The newest CS&E cluster: 20 dual processor 2GHz PC's.	Two interconnection networks: The processors are connected with standard Ethernet (top) and by a very fast Myrinet fiber network (bottom).
The computer: This computer is a brand new "Beowulf" cluster. It is similar to the cluster used for the NASA project (two frames above) but it is built using the latest technology. Each PC is about 1.5" thick and houses two 2 gigahertz processors. There are 16 of these PC's (in groups of 8) above and below the two switches that connect them. These switches are shown on the right. The top switch is a standard ethernet switch like that used for the NASA cluster. The bottom switch uses very fast fiber optic technology (that's why the wires are thinner) to connect the computers. This system was acquired with the NSF Major Research Instrumentation grant described below.
NSF: Computational Facilities for MTU's CS&E Program S. Seidel (PI), and Co-PIs C. Friedrich, J. Jaszczak, A. Mayer, $260,000.
Technical description: This is an equipment acquisition grant to support research and training in computationally intensive projects in several disciplines. Participating researchers come from the departments of physics, mechanical engineering-engineering mechanics, metallurgy and materials engineering, geological engineering and sciences, and computer science. The list of current departments has since grown to include mathematical sciences. The computational system acquired in this project is a primary tool used in their work. Other research projects that use this facility are in the areas of computational fluid dynamics, materials science, groundwater flow, and interprocessor communication. This computational facility is being used to expand the scale of problems that these projects address. Popular description: Over the past 15 years, computing at MTU has evolved from a centrally located, centrally administered "computer center" to a completely distributed, self-administered web of desktop PC's. This trend put computers on our desks, but by doing so it cut up the computational power of these computers into very small pieces. As explained above, these PC's are more than adequate for doing day-to-day tasks, but certain researchers require much greater computational power. For many years, medium- and large-scale computers were not available at MTU because all computing had been "distributed" to the desktop. This NSF grant was one of the first to reverse this trend and make larger computers again available to researchers at MTU This multi-year grant was obtained several years ago. The first piece of equipment acquired was the Sun Enterprise 4500, described below. This computer is freely available to scientists and engineers for computational work that is too large for their desktop machines. During the past several years faculty and students from the College of Engineering and the College of Sciences and Arts have used this computer for a wide range of work including: The study of volcanic processes Student: Sebastiene Dartevelle, Ph.D. student, Geological and Mining Engineering and Sciences Advisor: Bill Rose The study of volcanic processes Student: Song Guo, CS&E Ph.D. student, Geological and Mining Engineering and Sciences Advisors: Gregg Bluth and Bill Rose Groundwater remediation studies Student: Mark Erickson, M.S. student, Geological and Mining Engineering and Sciences Advisor: Alex Mayer Project title needed Student: Hanyi Li, CS&E Ph.D. student, Geological and Mining Engineering and Sciences Advisor: Judith Budd Project title needed Student: Krista Stalsberg-Zarling, CS&E Ph.D. student, Mathematical Sciences Advisor: Franz Tanner Discrete atomic modelling Investigator: Jong Lee, Material Sciences and Engineering Other regular users of cse0.cse.mtu.edu? Student: Advisor:
Sun Enterprise 4500: 12 server-class Sun processors.
The computer: This is 12 processor Sun Enterprise 4500. Unlike the two "Beowulf" clusters above, all connections between processors are contained within the cabinet. This system was acquired with an NSF Major Research Instrumentation grant to support CS&E research at MTU.

Tanner

Collaborator and Sponsor: Helsinki University of Technology (HUT), Finland
Principal Investigators: Franz X. Tanner (MTU) and Martti Larmi (HUT)
Ph.D. Student: Krista Stalsberg-Zarling

Abstract: The simulation of spray combustion requires the modeling of turbulent, reacting multiphase flows, where the gas phase is described with the time-dependent, compressible, Reynolds/Favre-averaged conservation equations for mass, species, momentum and energy, together with the equations for the turbulence model. The liquid phase is governed by a stochastic formulation of the discrete droplet model, where the liquid and gas phases are treated separately. The liquid-gas interaction, as well as the species creation and depletion due to chemical reactions, are coupled with appropriate source terms in the gas conservation equations. One of the main problems in the modeling and simulation of two-phase flows, using the discrete droplet model approach, is the intrinsic dependence of the liquid-gas coupling on the spatial resolution of the gas phase equations. The objective of this research project is the development of a general method which reduces or eliminates such discretization dependencies.

Michalek

Mayer

Gregg/Rose/Guo

This figure shows a numerically simulated volcanic
plume/cloud 115 minutes after eruption. The simulation was done
using ATHAM (Active Tracer High Resolution Atmospheric Model).

Gockenbach?

Jaszczak?

Hansmann?

Perger?

Others?

Students in the CS&E Ph.D. program are members of their "home" department. Their advisor, their funding, their office, etc. all come from their home department. Yet these students are all in the same degree program and must meet the same set of CS&E degree requirements. This results in graduates who understand their own science or engineering research area and who know how to apply the most advanced computational techniques to solve problems in that area. No other Ph.D. program at MTU is designed to meet these goals and no other Ph.D. program at MTU permits this level of interdisciplinary flexibility.

• Song Guo, Geological and Mining Engineering and Sciences
  • Advisors: Gregg Bluth and Bill Rose
• Hanyi Li, Geological and Mining Engineering and Sciences
  • Advisor: Judith Budd
• Krista Stalsberg-Zarling, Mathematical Sciences
  • Advisor: Franz Tanner
• Ping Huo, Computer Science
  • Advisor: Nilufer Onder
• Sakke Karstu, Computer Science
  • Advisor: Linda Ott

See the CS&E web page.

• UPC Technology Development Projects
• Beowulf Cluster Computing
• Granular Volcano Group

During the past year several visiting CS&E researchers have given talks at MTU:

• David Walker, Professor of High Performance Computing, Department of Computer Science, Cardiff University, Wales
• Lori Freitag, Mathematics and Computer Science Division, Argonne National Laboratory
• William Carlson, Center for Computing Sciences, Bowie, Maryland
• George Corliss, Dept. of Electrical and Computer Engineering, Marquette University, Milwaukee, WI
• Brian Wibecan, UPC Development Team, Hewlett-Packard Company, Nashua, NH

• Hewlett-Packard Company has agreed to provide the UPC research group with a 16-node AlphaServer SC cluster in the near future.
• The first CS&E Ph.D. students are expected to graduate in the next academic year.
• The new CILIT Computer Science Hall contains a CS&E lab and additional environmentally controlled computer rooms for specialized CS&E equipment.