Executive Summary
SofTech and Motorola now have formally announced a joint HPF effort. They have been added to the formal implementation list.
Numerous CCI issues were considered, wrapping up the round of considerations for this year.
Jerry Wagener made a report about the status of F95 and what the timing would be for release as an international standard if all reviews and votes are positive. The current version of the draft is available via ftp.
An HPF_SERIAL extrinsic interface was added to the appendix to allow programmers to indicate that a given routine should execute on only one processor, and that distributed data in the arguments should be reassembled prior to the call.
A collection of codes to provide to requirements for HPF2 is being assembled for FTP at hpsl.cs.umd.edu: pub/hpf_bench.
It was decided that the group of "requirements" gathered for HPF2 would be termed the "Scope of Activity". These issues grouped into topical areas will form the subgroups for HPF2 activity. A document containing these, along with a proposal that there be an HPF_kernel, and the benchmark appendix will be created as the primary report from the HPFF94 meetings.
Future meetings: An HPFF BOF is planned for SC94, and an HPFF Implementors Meeting scheduled for January 30, 31, 1995 in Houston. Schedule for HPF2 meetings will be determined at that time.
(end of executive summary)
October 13
The meeting formally began at 9:00, chaired by Ken Kennedy. The first
topic was CCI group 2 report by Mary Zosel. The following subgroup
recommendations were made:
10.1 Are the indexes returned by LOCAL_LINDEX/UINDEX global or local?
10.2 Withdrawn.
10.3 Random numbers: Specific question/issue are needed, suggest bringing
future questions to X3J3. As a side note - parallel random number
generators have dark corners in implementation, such as possibly requiring
a different seed on each processor, even though the language recognizes only
one seed value.
10.4 In MAXLOC and MINLOC, the apparent DIM/MASK incompatibility
is resolved by argument types. X3J3 approves of the HPF form of the call.
Since this question has appeared before, text should be added to the
document explaining the situation
10.5 Question about status of Annex A: Is HPF_LOCAL part of HPF?
10.6 In some of the LOCAL library definitions, arguments are not fully
specified because they are a duplicate of those in the HPF library. This may
cause confusion - a text change would help.
10.7 Are mapping inquiry subroutines pure? Yes, change text page 73, line 7
to read "intrinsic and library functions and subroutines".
10.9 Each local gets its own local block A(1,local-block).
10.10 DO INDEPENDENT with call to subroutine, want to call subr on
subset of the processors - can they use local data that is distributed? Yes, it
it is up to the implementation to do the right thing. Implementation note text
would be useful. For some implementations there may be serialization of
execution.
All recommendations above were accepted without challenge.
3.2 Read-only use of distributed data in PURE routines. This is the same
issue as 10.10.
6.10 Can a NEW variable be a pointer? Second reading. No problems have
been identified. Recommend accept proposed text.
At this point there was discussion or an organizational voting issue. The
formal organizations of some members have changed, even though faces
have not. The proposal was made that all organizations present could vote,
including those who had not been present at the previous two meetings.
Accepted by vote on non-questioned organizations 23 yes, 0 no. 28
organizations are present. The vote on 6.10 was then formally taken.
Vote on proposal 6.10 - 27 yes, 0 no, 1 abstain
8.17 To find the bounds of a global array from a local argument in an
HPF_LOCAL, on second reading four new library routines were proposed:
5.3 This item presents "a very ugly slice that you could pass into LOCAL".
The subgroup recommended that - yes, it is ugly, may want to insert
advisory text "this may result in copies" David Loveman moved, and Shapiro
seconded - local routines cannot accept CYCLIC(N) dummies - argument
that these parameters do not look like F90 arrays, thus different compiler
technology. A
straw poll was taken: is there support, in principle, to hear a formal proposal
tomorrow? Straw poll 17 yes, 8 no, 6 abstain. Further action was postponed
until Friday.
5.8 Additional LOCAL library support requested by this item was addressed
by some changes to the arguments to the routines GLOBAL_TO_LOCAL
and LOCAL_TO_GLOBAL - pages 165+ in the draft document. Second
reading to accept proposed changes: 25 yes, 0 no, 2 abstain. After further
discussion about changes to
LOCAL_BLKCNT & related routines further discussion was postponed.
2.13, 6.16, 6.17, 8.15 All related to new extrinsic proposal discussed in first
reading at the previous meeting. These will be reviewed after lunch.
5.5 Proposed HPF_"scalar" interface. The issues include
sequential dummies, location of execution, and what should we call it, is
there HPF_XXX or just F90_XXX? A presentation will be made later.
9.1 Leave static mapping of allocatable arrays as it is.
9.2 If a template is in a module, there is no way to access it's name
if the USE / ONLY feature of Fortran 90 is used.
9.3 No mapping of assumed-size dummy arguments was discussed, but the
issue was unresolved.
9.4 Why can sequential scalars be mapped? No change.
9.5 Mapping sequential assumed-size arrays: no change.
9.6 Duplicate sequence/nosequence directive: add a constraint against this in
scoping unit.
9.7 Some syntactic restrictions are missing for object-name distributees.
Recommend
Page 26 line 38 strike everything from "in" to the end.
9.8 Typos identified.
9.9 Withdrawn.
10.8 Is a pointer still associated if the target is remapped across a
subroutine call. We need to explicitly include this case so that the pointers
don't become undefined. Guy will discuss this one later.
7.11 Postponed until later.
7.1 Drop constraint about array-valued function results and add in
function name to syntax. Second reading 25 - 0 - 2.
7.3 Circular definitions: Second reading of addition text:
"and there may be no subsequent static mapping directive affecting the
mapping of the array". 25 - 0 - 2
A final vote was taken to approve editorial changes from both CCI1 and
CCI2 reports: 27 - 0 - 0.
Ken announced changes in the implementation status list. SofTech and
Motorola now have formally announced efforts. The current list now is:
At this time, some of the postponed CCI issues were addressed further:
7.11 Proposed new text is: page 24 line 26 (new doc)
No attribute may be specified more than once for a given execution.
Chuck requested that we review the use of italics for distributee in the text.
The new text was approved: 26 - 0 - 1.
7.3 Still don't know exactly where this should go.
10.8 Guy Steele proposed text to address when pointers become undefined.
This was addressing a problem related to a recent X3J3 interpretation for
F90. The problem comes up if within the subroutine where the data has been
remapped, another global pointer might be made to point to the remapped
item and still be alive on exit . Possible text for discussion:
Discussion ensued. There was a suggestion to strike explicitly. After more
discussion, the issue was postponed for more discussion after lunch.
Jerry Wagener gave a report on the status of F95. The working process is that WG5 sets Fxx requirements and X3J3 implements the requirements in the form of a draft standard. The draft is processed as international standard. Individual countries accept it if they want to. The F95 requirements were finalized in Aug. and there were no new ones.
X3J3 will meet again in Jan. 95 (Houston) to complete integration. They
expect to produce draft 95-00 in March 95. Following that the schedule (if
all steps are approved) would be: WG5 meeting, April 95 (Tokyo) to accept
forwarded 007 as an ISO CD (committee draft).
The draft 94-007r3 document is available from
ftp.ncsa.uiuc.edu or ftp.dfrf.nasa.gov in several formats.
After lunch, we resumed discussion of CCI issues postponed earlier.
9.5 / 9.3 about assumed size dummies
10.8 About status of pointers after calls was discussed again. Guy Steele
prepared the following analysis.
then the program is not HPF-conforming unless
(Part 2) If a dummy argument has the TARGET attribute and no explicit
mapping attributes, then INHERIT is assumed - thus falling under case (a)
of rule 1.
This was accepted 24-0-2
(An a comical aside, the comment was made that this fell into the "trust
Guy" category. Guy said he would feel better if it were "trust Guy and Rob",
but then Rob said it would have to be "trust Rob trusting Guy." We hope we
have this right. :-)
Extrinsic kind keywords whose names begin with "HPF" are reserved for
present or future definition by this specification. At present only the
keywords HPF and HPF_LOCAL have defined meaning. A program unit whose
extrinsic kind keyword begins with "HPF" is said to be "of an HPF extrinsic
kind".
Within any module of an HPF extrinsic kind, every module-subprogram must be of
that same extrinsic kind and any module-subprogram whose extrinsic kind
is not given explicitly is assumed to be of that extrinsic kind. Similarly,
within
any main-program or external-subprogram of an HPF extrinsic kind, every
internal-subprogram must be of that same extrinsic kind and any internal-
subprogram whose extrinsic kind is not given explicitly is assumed to be of
that extrinsic kind.
A function-stmt or subroutine-stmt that appears within an interface-block
within a program unit of an HPF extrinsic kind may have an extrinsic prefix
mentioning any extrinsic kind supported by the language processor; but if no extrinsic-prefix appears in such a function-stmt or subroutine-stmt, then it is
assumed to be of the same HPF extrinsic kind as the program unit.
If a module of one HPF extrinsic kind is used from a program unit of another
HPF extrinsic kind, then the effect must be that only names of procedures,
procedure interfaces, and type definitions in the module are made
accessible to the
using program unit; names of data objects may not be used.
Any data defined by
the module must be private, or else the USE statement must have an ONLY
option that specifies only name of procedures, procedure interfaces, and type
definitions.
[Note, however, that the existing prohibition on calling a global HPF routine
from an HPF_LOCAL routine remains as a separate restriction. See section
A.2.1, second paragraph.]
A named COMMON block in any program unit of an HPF kind will be
associated with the COMMON block, if any, of that same name in every
other program unit of that same extrinsic kind; similarly for unnamed
COMMON. But a COMMON block in a program unit of one HPF kind is
distinct from and not associated with any COMMON block in a program
unit of any other HPF kind. (Implementors are advised to
follow a similar rule for all extrinsic kind keywords, not just those starting
with "HPF_".)
Another problem identified: if you want to give distribution directives for the
local, they are HPF-global directives, but we can't give these in a local
module.
Text changes were proposed: in paragraph 4 last two words ... change as the
program unit, to as the host scoping unit.
In paragraph 7, be sure that there is still just a single name space --- can't
have a common for hpf-global with the same name as a common for hpf-
local.---- there are possible ways to do this, but we are not going to go to
extra lengths for a feature like common.
There was a vote about accepting these - realizing that we have identified a
few more issues to resolve about how to describe an hpf-local in hpf-global
terms and how to describe hpf-local function return values. Remainder of
specification approved: 24 - 0 -5
With the video camera gone, the group now registered dissent. The motion
failed 5-14-10.
The first issue discussed was whether the ynh extrinsic kind could be called
from an HPF_LOCAL. David Loveman gave the example of calling the
TIME routine from locals. Saday made an amendment to specify that
HPF_ynh CANNOT be called from within HPF_LOCAL. This passed 11-8-10.
David made an amendment that HPF_ynh could call HPF_GLOBAL. The
motivation being that codes might initialize with a "ynh" extrinsic and then
branch out to the parallel code. This failed 5 - 8 - 14.
There was the question about whether pointers / targets can be dummies
arguments to HPF-ynh. This was left to reconsider overnight . The
procedure was to vote with the restriction and later entertain a proposal to
remove the restriction if appropriate.
The following matrix summarizes the allowed calls.
Many possible names were suggested for "ynh". A preference vote where
each person was allowed a first/second/third choice narrowed the list to
(uniprocess/uni/unilocal), serial, or lpf. Another preference vote narrowed
the choice to "uni*" vs serial. And a strawpoll between these favored serial
17-13. An institutional vote on the same choice again favored serial 15-11. If
was observed that lpf made a good try to be a serial killer.
So, in summary, the new extrinsic interface name is HPF_SERIAL.
Break
I/O benchmarks were discussed briefly. The scalable I/O consortium should
have some codes. Ohio has an out-of-core FFT code.
Joel Saltz proposed the benchmark report be incorporated with the HPF2
requirements document.
There was an extended discussion about what this list means. It is a long
list including items with strong agreement along with items of strong
controversy. In what sense is it binding instruction to HPF2? The items will
be grouped by the topics in Ian's document.
We can't in this one meeting make a decision that anything is "out".
Ken proposed that we get a document that categorizes items into groups that
give us a feel of the subareas where we will have working HPF2 subgroups.
We can group these, and perhaps even prioritize --- but not constrain HPF2.
Ian and Rob will merge.
The group broke for the evening.
Friday Oct. 14, 1994
Meetings started at 8:30.
The first topic was organization of the SC94 HPF BOF (Birds of a Feather).
There will be reports ~15 min each:
- Ken will go over the survey status. He would like detailed information on
announced products and references to other conference demos, activities, etc.
- Ian & Rob will go over the requirements document.
- Paul will go over the benchmarks.
The purpose of this BOF is to update people on what we've done in HPFF94.
We will assume basic familiarity with HPF 1, but not at the Furtney test
level. The January HPF Meeting will be announced at the BOF. We will
need a one-page announcement for the BOF, possibility with the list of
activities/demos on the back.
We will try to have ~40 copies of the 1.1 document, and lots of pages with
instructions of how to get a copy. Also ~150 copies of the requirements
document. Plan for ~2000 copies of the one-page announcement - these can
also be handed out at the CRPC exhibit. Paul volunteered to get copies
made.
Given the example:
So the basic proposal is that the HPF_LOCAL would have the a set of global
mapping directives. The other option is to disallow the inter-module kind
inclusion completely.
Shapiro moved that A module of one extrinsic kind cannot be used in a
module of another extrinsic kind. This was amended to specifically exclude
the case the case of global using local so that an hpf-extrinsic-kind can only
include same kind extrinsic modules and hpf_global can use hpf_serial.
There was a discussion about what kinds of things might be used between
extrinsic. The following "unilocal" diagram was discussed where D stands
for data, P for procedures, and T for types.
Now Guy "moved" the diagram above AND that HPF directives can be used
in HPF_LOCAL code for the dummy arguments and the results.
Piyush M moved that it applies the same for serial: that directives can be
used in serial too. There was no second for this.
SO NOW formally the motion is that 6.16 from yesterday is amended to drop
"as an interface block" and add that global directives in local code for dummy
arguments and function results. This passed: 19 - 1 - 9.
There was a break from 10 to 10:15.
After break, Andy Meltzer led a discussion of a proposal for kernel HPF. He
proposed that we take the kernel HPF requirements and add them to HPF2
requirements - i.e. that there be an HPF_Kernel extrinsics .
The goal is a proper subset allowing for high-performance on many
architectures:
To get greatest benefit it may be necessary to tell compiler that it is
compiling kernel code, e.g. and hpf_kernel extrinsic or a command line
switch.
Rich Shapiro moved that we add this kernel requirement and part of a
document introduction at least, maybe more, to the HPF2 requirements doc.
The straw poll on this was :28 - 0 - 1.
The group was uncomfortable calling this a "requirements" document
because of the breadth of suggestions and lack of agreement about what
really required. It was decided to call this a "scope of activities" document -
thinking of it more as a call for participation. Our requirements are that we
get some coverage for the benchmarks. The scope of activities is to consider
these general language areas plus the hpf kernel. It was decided to add
section about language processor environment. Jim Cowie will do this.
Topics for this section are interacting with tools and interoperability.
The hope is that the document will be concise, an introduction, justifications
kernel, and benchmarks in the appendix.
This general approach was approved 31-0-0.
The answer is "local". This has already been resolved in the draft
document.
HPF_LOCAL can't be implemented on some machines, but if you implement
it this is how it should work. As such it is not a proper part of full HPF.
GLOBAL_LBOUND, GLOBAL_UBOUND, GLOBAL_SHAPE, and
GLOBAL_SIZE - with text essentially all from F90. Boilerplate is needed to
say can only invoke these from highest level HPF_LOCAL routines. Vote -
26 yes, 0 no, 2 abstain.
CCI 1 - Rich Shapiro Report:
... define alignee and distributee more carefully (GLS)
page 31 24-25 strike "in" to the end.
page 31 another constraint ---- 1st constraint from page 26 needs to be here
too.
Break, restarting at 11.
Survey of Vendor HPF Implementation Status
Oct. 94
Announced Products
Applied Parallel Research,
Digital, Intel, KAI, Meiko,
Portland Group
Formally Announced Efforts
ACE, Archipel, Convex, Cray Computer, Fujitsu, Hitachi,
IBM, Lahey, Maspar, Motorola, NAG, NA Software,
nCUBE, NEC, Pacific Sierra Research,
SofTech, Thinking Machines
Interested
ACSET, CRI, HP, SGI, Sun
Since it has been some time since we had contact with ACE, Archipel,
(Lahey), MasPar, and nCUBE Chuck Koelbel agreed to check with these
organizations to make sure that they still have active efforts.
No entity may be given a particular attribute more than once. An entity is
a distributee if it is a given the distribute attribute, it is an alignee if it
is given the align attribute.
some sets of attributes are incompatible, these are answered in the text
where defined ...
If dummy is a TARGET attribute, then it must be {explicitly} mapped
either descriptively or transcriptively (so that no implicit remapping occurs).
Status of F95
Initial F2000 requirements have the following emphasis:
level 1 - high performance ...
level 2 object oriented (task force created)
interpretoperability (task force created)
level 3 - other stuff
Next X3J3 will meet in November. Issues are to resolve status of ENABLE
and clean up integration in 007R3 including FORALL PURE elemental
procedures. The goal is to produce 007R4 in Dec. 94.
Country ballots will be during June-August, with a probably US public
review in June-July. X3J3 would meet ~Aug. to determine the US vote on
the CD. WG5 meeting ~Oct or Nov somewhere in US to
prepare JTC1 ballot. The language can then become an international
standard, if comments are positive - but may also require iteration
rereviews etc.
Break for lunch
reconvened at 1:45
Assumed size dummies (ASDs) are sequence associated by definition because
F90 says they are. An ASD cannot be mapped because its size and shape are
not knowable, so any mappings must be transcriptive - but this is what the
compiler will do anyway, so ONLY transcriptive mappings can be used for
an A.S.D.
So, the recommendation is to prohibit use of an ASD in an explicit mapping.
See the text in cci (maybe). Accepted 25-0-2
(Part 1) If, on invocation of a procedure P,
6.16 (Extrinsic interface issues) Second reading was the next topic. The
proposed last time was reviewed. It is repeated here.
Allow the EXTRINSIC(...) prefix to appear at the beginning of PROGRAM, MODULE,
and BLOCK DATA statements. As with FUNCTION and SUBROUTINE
program units, if the first statement of the program unit has no extrinsic
prefix, then an HPF compiler assumes EXTRINSIC(HPF).
Henry Zongarro asked about the case where an interface block tells you that
it is some other extrinsic-kind ... then that interface block in turn contains
another interface block (one of the dummies is a function). What is the
default assumption of that inner one? (MULL over)
5.3 The next issue was the proposal from David Loveman to limit use of
cyclic(n) in HPF_LOCAL. The following motion was made and discussed:
HPF_SERIAL
#5.5b The next issue considered was the (scalar) Extrinsic proposal that was
outlined at the last meeting. Henry provided a handout with the details of
the proposal. At the last meeting we voted that SCALAR was not the right
name, so for the purpose of the discussion, the proposal was labeled "ynh"
(for "your name here") . This caused a good laugh when Ken admitted that
he had been reading the "ynh" as 1/n+1 - and indeed the hand-written
version did look a bit like that.
callee glb local ynh
caller
glb x x x
local x
ynh x
The proposal was modified to include a call to get the global number of
processors. This proposed extrinsic would go in appendix a.
The vote on the proposal - without knowing the name: 18-2-7.
Benchmark requirements for HPF2
Following the break, Paul Havlak made the benchmark report. He provided
a handout about the status of gathering benchmarks. These are currently
being assembled at an ftp site:
hpsl.cs.umd.edu: pub/hpf_bench
Write-ups are due by next week so that we can have a document for SC94.
The requirements report prepared by Ian Foster and list of potential
requirements prepared by Rob Schreiber was the next topic. The list of
items was considered in strawpolls in a working group the previous
evening. Several items were identified to be added to the list,
including pipelines, persistent arrays, real-time I/O, view
distribute(block) as
distribute(cyclic), and array aliasing.
Guy Steele led a continuation of the discussion of extrinsic problems (6.16 ),
specifically how and where to describe the mapping of the args and the
result. In the case of the result, there is an additional problem of
specification of shape.
Subroutine foo(a,b)
real a(:), B(:)
!hpf$ align b(j,j) with A(j)
This is what happens when the declaration is in an extrinsic HPF_LOCAL
module:
function foo(n) result(r)
real r(2,10)
There are 3 different possible ways that this could be recomposed on the
global with 4 processors: as a 2,40 , as a 8,10 or as a 4,20 and the
distribution of the result variable will tell us.
callee global local serial
caller
global DTP T P TP
local T DTP T
serial T T DTP
The amendment was restated: can only use hpf-kind modules within hpf-
kind modules of same type. This failed. 7-14-8.
Ian led a further report on the requirements document. The issue list from
yesterday was reorganized. These will be an executive summary. To
complete the document, need people to describe the 6 sections. There was
some discussion of regrouping topics, especially if these define the subgroups
for future meetings. Chuck Koelbel with help from U Maryland will take
responsibility for enhanced distributions section. Ian will do the
control/parallelism section. Alok Choudhary the I/O section.. For the
communication optimizations, Rob will take the responsibility and he will
get help from Joel.
The schedule for HPF2 Scope document is:
The schedule for HPF 1.1 language document update:
HPF2 will be the subject of a January meeting. Ken will take the scope
document to make a proposal and get funding set aside as a two year
proposal.
Mary led a brief discussion of the suggestion that the group standardize
some data formats for HPF-related tools to work from. This will go into the
HPF2 document in the interoperability section.
Ken reported on copyright issues. There have been vendor requests about
using the HPF document. David noted that this kind of permission to
"copy" is typically noted on the copyright page of the other document. X3J3
.... would like to use the text too, but ANSI copyrights their document. There
should be a mutual agreement on this - they can use our text and we can use
theirs. Jerry Wagener will take this to ANSI.
The January implementors meeting was discussed. The dates determined
were Jan. 30-31, 1995. The meeting will start Monday morning and finish
Tuesday noon. This will be a conference meeting, with a business meeting at
the end to discuss the plan for HPF2 meetings. Mary will chair a program
committee, with David, Rob, Alok, Chuck, Ian, Robbie, Piyush, Robbie,
Andy J, Joel, Saday, and Chau-Huang volunteering to help. The purpose
will be product and usage reports and initial experience with ideas proposed
for HPF2. There will be a call for formal presentation abstracts.
LPF
The last item of business for the meeting was the final report from LPF
MCVXII, Andy Meltzer.
The meeting was adjourned at ~11:45.
Future meetings
Future meetings: A BOF is planned for SC94, and an HPFF Implementors
Meeting scheduled for January 30, 31, 1995 in Houston. Schedule for HPF2
meetings will be determined at that time.
Attending the October HPFF Meeting:
Robert Babb U. of Denver babb@cs.du.edu
Scott Baden UCSD baden@cs.ucsd.edu
Ralph Brickner LANL rgb@lanl.gov
Alok Choudhary Syracuse U. choudhar@cat.syr.edu
Jim Cowie Cooperating Systems cowie-james@cs.yale.edu
Ian Foster Argonne Lab. foster@mcs.anl.gov
Paul Havlak U. Maryland havlak@cs.umd.edu
Chua-Huang Huang Ohio State U. chh@cis.ohio-state.edu
Andrew Johnson OSF RI andyj@osf.org
Ken Kennedy Rice U./CRPC ken@rice.edu
Chuck Koelbel Rice U. chk@cs.rice.edu
David Loveman Digital Equip. loveman@hpcgrp.enet.dec.com
Piyush Mehrotra ICASE pm@icase.edu
Andy Meltzer Cray Research meltzer@cray.com
Doug Miles PGI miles@pgroup.com
Nicole Nemer-Preece U. of Missouri-Rolla nnemer@cs.umr.edu
E. Nunohiro HITACHI nunohiro@hisoft.soft.hitachi.co.jp
Bruce Olsen Hewlett Packard bruce@apollo.hp.com
Terry Pratt CESDIS/NASA Goddard pratt@cesdis.gsfc.nasa.gov
J. Ramanujam Lousiana St U jxr@max.ee.lsu.edu
P. Sadayappan Ohio State U saday@cis.ohio-state.edu
Joel Saltz U. of Maryland saltz@cs.umd.edu
Rob Schreiber RIACS schreiber@riacs.edu
Yoshiki Seo NEC yoshiki@cs.rice.edu
Richard Shapiro Silicon Graphics rshapiro@boston.sgi.com
Guy Steele Sun Microsystems Guy.Steele@east.sun.com
Roy Touzeau Intel SSD rft@ssd.intel.com
Jeff Vanderlip Pacific-Sierra Research jeff@psrv.com
Paula Vaughan Miss. St. / NSF ERC paula@erc.msstate.edu
Jerry Wagener Amoco jwagener@amoco.com
Henry Zongaro IBM Canada zongaro@vnet.ibm.com
Mary Zosel LLNL zosel@llnl.gov