CHPC News, March 2000, Volume 11, Number 2
by Julia Harrison, CHPC User Services
The CHPC fileserver, geronimo was replaced by a new IBM S80 fileserver on February 16th, 2000. The now retired geronimo served us well for many years, but over the past several months had ongoing problems with stability. We often lost disks and most recently had problems with a fan which caused it to fail. The primary function of this machine is to provide home directory space to all of the HPC systems.
The new fileserver, named fileserv will have built in redundancy and will provide a single home directory space to the SGI Origin2000 (raptor), the SGI Power-Challenge (inca/maya), and the IBM SP along with many CHPC departmental machines. In the fullness of time, after we remove some technical hurdles, the Beowulf Cluster (icebox) will use the central fileserver as well.
The migration process went fairly smoothly, with only a few problems. Some ownerships of files and links required fixing. Also, if you moved or deleted files between Saturday Feb. 11th and Wednesday Feb. 16th, these files may have also been restored to the previous location before they were moved or deleted, creating duplicate files in those instances. Initial tests indicate the I/O performance of the new fileserver is approximately 16 times that of geronimo.
One interesting note: Moments after the final restore was completed to the new fileserver, geronimo crashed! What timing.
Batch System on IBM SP
The version of PBS (Portable Batch System) we use on the IBM SP was upgraded to version 2.2 on February 16, 2000. At the same time we replaced the scheduler used by PBS which is the more noticeable of the two changes. The new Maui Scheduler was developed at the Maui High Performance Computing Center, and will be implemented on raptor and inca/maya when development is complete for shared memory systems.
The change in scheduler will require a few minor adjustments to your PBS scripts. The changes to your script should be:
- You will no longer specify a queue.
- You should no longer use the norm, spec or p2 node properties.
- Try to closely estimate your walltime for quicker turnaround.
Old way: #PBS -q sp_short #PBS -l walltime=8:00:00,nodes=4:norm New way: #PBS -l walltime=6:00:00,nodes=4
As we continue to work with the new scheduler documentation, helpful hints will be posted to our web page, http://www.chpc.utah.edu.
Service Unit Allocation Changes April 1 2000: icebox enforced, inca/maya not (no joke)
Allocation requests were due March 1st if you wish to be considered for an allocation of time for the Spring 2000 calendar quarter (April-June) and/or subsequent three quarters.
Beginning April 1, 2000 we will begin enforcement of allocations on the Beowulf Cluster (icebox) and will no longer enforce them on the SGI Power-Challenge (inca/maya).
If you have questions or concerns, please contact CHPC User Services by sending email to email@example.com or phoning 801-581-4439.
by Julia Harrison, Assistant Director, User Services, CHPC
November 13-19, 1999 CHPC, along with The Center for Scientific Computing and Imaging (SCI) and the Center for the Simulation of Accidental Fires and Explosions (C-Safe), presented a research exhibit at the SC99 Conference.
The Supercomputing (SC) conference series is an annual event that is organized by a voluntary consortium of persons acting together to advance the science and application of high-performance computing and communications technology.
The three groups shared the research exhibit booth where we displayed posters. A video from the SCI group and a demonstration of the SCIRun Programming Environment for Computational Steering were highlights of the booth along with a live Beowulf cluster.
Three-Dimensional Computational Modeling and Simulation of Platelet Aggregation on Parallel Computers
by Haoyu Yu and Aaron L Fogelson, Department of Mathematics December, 1999
This research is concerned with three-dimensional mathematical modeling and computational simulation of the biological process of platelet aggregation. Platelet aggregates form on an injured blood vessel's wall in response to injury to the vessel. The aggregation process involves physical and chemical interactions among platelets, the suspending fluid, the vessel's wall, and chemical agents secreted by the platelets such as ADP. Platelet aggregation plays an important role in normal blood clotting and is associated with vascular disease. This research enhances understanding of the dynamics of the aggregation process.
In circulating blood, platelets are present in the number of 250,000/mm3 and less than 1% volume concentration. Normally platelets are in the quiescent or non-activated state and circulate in the blood without attaching to each other or the blood vessel wall. However, when the vessel is injured or the endothelial cells are removed, platelets will cling onto the exposed sub-endothelial surface, which is called adhesion, and secrete activating chemicals into the blood plasma. Other platelets activated by the released chemicals will attach to these platelets one to another, which is called cohesion, at the sites of vascular injury to form the aggregate , , .
The activation process triggered by certain chemicals involves shape change, changes in the platelet surface that allow them to cohere with other platelets and release their stored chemicals into the surrounding fluid which in turn activate other platelets. Adenosine diphosphate (ADP) is one of the released chemicals and is believed to be one of the most important chemicals for platelet plug formation . It has been observed that if a sufficiently high concentration of ADP is applied to a suspension of platelets, a sequence of events is seen, which includes platelet shape change from discoid to spherical, the secretion of its stored chemicals, and the interaction of sticky platelet to one another to form an aggregate . The enzyme thrombin is another potent effector of platelet activation. Platelets treated with thrombin release ADP, and at high concentration of thrombin platelet aggregation proceeds.
The fluid dynamics of the blood also has an important influence on aggregation. Blood flow determines the rate of platelet transport to and away from the injured vessel surfaces. It imposes fluid forces on platelets that can affect whether they are able to attach to the wall and whether they later detach from the wall. Blood flow enhances platelet concentration near the injured vessel wall . Aggregates can be limited by the fluid stress that may break the molecular bridges between a new platelet and an aggregated one or the ones within an aggregate. Therefore the aggregation depends on both activating chemicals and fluid-mediated transport of platelets. Fluid forces are important to aggregates' growth. Conversely, the growth of aggregates can remarkably influence the flow to the extent that complete vessel occlusion can occur. Platelet adhesion and aggregation involves strong two-way action between aggregate growth and fluid dynamics.
In order to fully understand platelet aggregation process, a three-dimensional representation and computation is necessary. This research presents the three-dimensional mathematical modeling and computational simulation of the biological process of platelet aggregation.
In this three-dimensional computation, aggregation inside a numerically designed laboratory flow chamber is considered. Two parallel plates are numerically constructed to assemble walls of a chamber through which the blood fluid flows. There are two main types of interactions to be considered for simulating the aggregation process numerically, the mechanical interaction among the fluid, platelets, and elastic vessel wall, and the chemical interaction involving a chemical agent, such as ADP, and the platelets. Modeling the mechanical interaction involves solving the Navier-Stokes equations to numerically simulate a channel flow for a viscous, incompressible fluid with constant den- sity containing immersed elastic objects which represents the platelets and chamber walls. The transport of the secreted chemical ADP from platelets is governed by an advection-diffusion equation.
In summary, the goals for this work include calculating forces on wall-adherent cells under various conditions and calculating the distribution of platelet activating chemical in the immediate environment of aggregates. With the fluid solver that simulates a channel flow containing elastic ball shaped objects, a detail calculation of fluid velocity distribution around the immersed objects and forces on them are studied.
Formulation and computational methods for studying platelet aggregation
It is clear that the best possible way to study platelet aggregation involves in vivo observation of the events under the conditions of interest. Unfortunately, direct observation can be conducted only in very limited circumstances. It is therefore necessary to select various physical models and infer physiological relevance from the results obtained from these models.
The exposure of vessel surfaces under well-defined flow conditions is an in vitro simulation of in vivo flow regimes. This simulation permits the characterization of platelet adhesion and aggregation on the surfaces in terms of basic fluid dynamic parameters. Our mathematical model of a flow chamber which contains vessel walls and platelets is built for a similar purpose. The domain of our interest is a fixed, periodic fluid box containing a section of blood vessel with portions of injury and platelets. The platelets and vessel wall, which are the immersed boundaries , are represented by a system of elastic spring networks with three-dimensional structure. The vessel wall is configured as two parallel plates of elastic networks. Portions of the walls are assumed to be injured and sticky to platelets. The platelets are ball-shaped structures which are constructed with elastic networks. Chemically, platelets are either non-activated or activated. Non-activated platelets have stored ADP and can be activated by encountering an above-threshold concentration of ADP or by touching the sticky portion of the vessel wall. Activated platelets secrete their stored ADP and appear sticky to other activated platelets. Adhesion between a platelet and vessel wall and cohesion between two platelets are modeled by creating elastic links between the two objects.
The blood inside of the walls is presented by a continuous viscous, incompressible fluid with constant density. The space within the chamber and outside the vessel walls is also filled with the same fluid by our assumption. The elastic networks interact with the surrounding fluid dynamically. The elastic springs provide additional point forces to the fluid. In return, the fluid velocity field updates the positions of the networks. The fluid motion is governed by the Navier-Stokes equations under the influence of the forces generated within platelets and walls as well as the adhesion and cohesion forces.
Divide the physical periodic box into a regular rectangular lattice of mesh width h on which the Navier-Stokes equations are solved. The numerical solution of the viscous incompressible Navier-Stokes equations is computed by discretizing the equations to a linear constant-coefficient difference system for unknowns un+1 and pn+1 and solving the difference equations by Fourier methods. All fluid variables, for example, u and p, are defined on this regular lattice of the same mesh width. Thus, Fourier methods can be made very efficient through the use of the FFT (Fast Fourier Transform) algorithm. The background flow in the absence of platelets is in the positive x-direction. Initially, the velocity u has a parabolic profile between two walls. This flow is maintained as the steady background flow (in the absence of platelets) at each grid point between the walls.
Since ADP, one of the chemicals released into the plasma during the adhesion and aggregation process, plays a dominant role in initiating and developing the process, it is incorporated into our model as a chemical stimulant. ADP transportation is governed by the advection-diffusion equation. The advection terms are handled by using the advection algorithms . To handle the diffusion terms requires solution of large sparse linear systems. A parallel multigrid  method is built to solve these equations.
By combining the advection-diffusion equation solver with the fluid solver, serious exploration of the dynamics of aggregation with an aggregate adherent to the vessel wall is conducted. The fluid forces exerted on adhered platelets, the flow around an aggregate, and transport of pro-aggregating chemicals in a micro-domain around the platelet are also studied.
In summary, at each time step, the following is done: A discrete advection-diffusion equation is solved to update the ADP concentration. The elastic spring force f is calculated at each immersed boundary point from the current configuration of the elastic networks. These forces are spread to the rectangular computational grid on which the discrete Navier-Stokes equations are solved. This defines the fluid force density. Discrete versions of the Navier-Stokes equations are solved to update the fluid velocity field at points of the fluid mesh. The mesh velocities are interpolated to each immersed boundary point location in order to update these locations to account for motion at the local fluid velocity. After updating the positions of all immersed boundary points, the local ADP concentration around each unactivated platelet is compared with the activation threshold concentration to determine if the platelet is to be activated. For each platelet which is to be activated, the platelet's state is changed from unactivated to activated, and a new ADP source is created. The networks of adhesive and cohesive links are updated to account for new link formation and breaking of existing links. This action depends on how close platelets are to one other and to the walls. If a platelet is within an adhering zone adjacent to an injured region of the walls, which were predefined, this is the indication that a new adhesion may occur. If it does, an elastic spring is formed between the platelet and the wall, and if the platelet happens to be unactivated before this, then the platelet becomes activated and a new ADP source is created as well. If two activated platelets are close enough, cohesion may happen. If it does, an elastic link is created between the two platelets. Also, each existing adhesive and cohesive link is tested to see if it should be broken, and if so, it is removed.
The computations with the three-dimensional platelet aggregation model are very time-consuming. To speed up the computations, the code is implemented on the University's SGI PowerChallenge and Origin 2000 shared memory parallel computers. The algorithms have been given careful attention make optimal use of these parallel computing capabilities.
Figure 1 shows a sequence of snapshots from a simulation conducted under an initial parabolic flow at 200/sec wall-shear rate. Platelets are introduced at the incoming boundary and are removed when they reach the outgoing boundary. The two channel walls and all the platelets are displayed in the pictures. The time interval between two adjacent pictures is 0.03 second. The isosurfaces of the ADP concentration at 1.0M are also depicted. These pictures clearly show how the aggregation in the middle of the lower wall is developed under the physical and chemical conditions. As discussed before, if balls are located within the adhesion distance from an injured wall, these balls will be linked to the wall by creating adhesion links, and if two balls are within the cohesion distance with each other, there will be cohesion links generated between them. However, whether the balls will stay connected with the wall or with the other balls will completely depend on the dynamic environment around the balls. The existing link forces compete with fluid forces at any instant of time. As a result, new links may be generated between some balls and the wall or between some balls, and some existing links may be subject to breaking if the breaking force on a link is greater than a threshold value. After 0.15 second simu- lation time, there are three more platelets that are adhered with the wall.
Figure 1: Simulation snapshots at every 0.03 second of small platelet aggregates on the bottom wall.
Three-dimensional numerical simulations of the development of platelet aggregates are conducted in this research. In these simulations, a three-dimensional computational flow chamber containing a viscous incompressible fluid, two parallel elastic walls, platelets, and the ADP secreted by the platelets is built. The physical and chemical interactions among the objects are studied. This numerical system is capable of generating platelet aggregates on injured portion of the vessel wall, calculating the ADP concentration distribution in the channel, and estimating the fluid forces exerted on adhered platelets.
Platelets are represented as a set of discrete points connected by elastic springs and initially having a ball-shape. Platelets are observed to deform during the simulations, especially when they are adherent to the wall or coherent with other activated platelets. Adhesion and cohesion links are formed and broken during the development of an aggregate; both processes are important in determining the eventual shape of the aggregate. When platelets enter the adhesion zone around injured portion of the vessel wall, they quickly attach to the vessel. Aggregates with a multilayer of platelets can be seen at the early stages of aggregation. Favorable conditions for aggregation can be found with this simulation model.
The detailed simulation results and the corresponding force analysis can be found in .
This research was carried out on the systems of CHPC. The unconditional support from the director and members of CHPC was very important to the completion and the success of this research. We greatly thank everyone at CHPC.
-  L. Badimon and J.J. Badimon and V.T. Turitto and S. Vallabhajosula and V. Fuster, Platelet Thrombus Formation on Collagen Type I, Circulation, 78 (1988), 1431.
-  V.T. Turitto and H.R. Baumgartner, Platelet-surface interactions, Hemostasis and Thrombosis, 30 (1987), 555-571.
-  H.J. Weiss, Platelet Physiology and Abnormalities of Platelet Function (Part 1), New Engl. J. Med., 293 (1975), 531-541.
-  E.C. Eckstein and D. Bilsker and C. Waters and S. Kipenhan and A. Tilles, Transport of Platelets in Flowing Blood, Blood in Contact with Natural and Artifical Surfaces, Ann. New York Acad. Sci, (1987), 516.
-  Charles S. Peskin and David M. McQueen, A three-dimensional computational method for blood flow in the heart: I. Immersed elastic fibers in a viscous incompressible fluid, Journal of Computational Physics, 81 (1989), 372-405.
-  A.L. FOGELSON, A Mathematical Model and Numerical Method for Studying Platelet Adhesion and Aggregation during Blood Clotting, J. Comput. Phys., 56 (1984), 111.
-  Randall J. LeVeque, High-Resolution Conservative Algorithms for Advection in Incompressible Flow, SIAM Journal of Numerical Analysis, 33 (1996), 627-665.
-  P. WESSELING, `An Introduction to Multigrid Methods', Wiley, New York, 1992.
-  Haoyu Yu, Three-dimensional Computational Modeling and Simulation of Platelet Aggregation on Parallel Computers, Ph.D. Dissertation, December, 1999, the University of Utah.
by Anita Orendt, Staff Scientist CHPC (Molecular Sciences)
We have recently obtained copies of NWChem, a computational chemistry package developed at the Environmental Molecular Science Laboratory (EMSL) at the Pacific Northwest National Lab, to run on the Icebox cluster, the IBM SP, and the SGI Origin 2000. NWChem has broad capabilities, from classical mechanics to quantum mechanics, and it also allows for the combination of both quantum and classical mechanics. It was designed to be very efficient in scaling to large systems using massively parallel processing. Installation is first being completed on the SGI. Later this spring a workshop discussing the use and applications will be held for potential users.
by Julio Facelli, Director CHPC
The University of Utah has been connected to the vBNS backbone since March of 1998; this connectivity has greatly enhanced the ability of our researchers to communicate with other universities, supercomputer centers, and research institutions will be replaced by the end of March 2000. At the end of February, the University will connect to the Abilene backbone (http://www.ucaid.edu/abilene/html/participation.html), this will provide a month of dual connectivity (vBNS and Abilene) giving ample time to migrate gracefully the vBNS traffic to Abilene. The Abilene connection will be at the same bandwidth as the previous vBNS connection (OC3) and due to the existing interconnections between Abilene, vBNS, and other advanced networks, University users should not experience any significant change in network performance.
If you have any questions or comments on our vBNS and/or Abilene connections, please contact Julio Facelli (firstname.lastname@example.org).