Semantics of the ON Directive

 

The ON directive restricts the active processor set for a computation to those processors named in its home. The computation controlled is either the following Fortran statement (for a on-directive or the contained block for a block-on-directive. We refer to the controlled computation as the ON-block.

That is, it advises the compiler to use the named processor(s) to perform the ON block. Like the mapping directives ALIGN and DISTRIBUTE, this is advice rather than an absolute commandment; the compiler may override an ON directive. Also like ALIGN and DISTRIBUTE, the ON directive may affect the efficiency of computation, but not the final results.

Advice to implementors.If the compiler may override the user's advice in an ON directive, then the compiler should also offer the user an option to force all directives to be obeyed. Because dummy arguments and local objects are required to be mapped onto active processors, an HPF compiler that fails to heed the programmer's advice with respect to the active processor set may also be required to ignore some of the programmer's advice concerning data mapping. (End of advice to implementors.)

The single-statement ON directive sets the active processor set for the first non-comment statement that follows it. It is said to apply to that statement. If the statement is a compound statement (e.g., a DO loop or an IF-THEN-ELSE construct), then the ON directive also applies to all statements nested therein. Similarly, the ON construct applies the initial ON clause to--i.e., sets the active processor set for--all statements up to the matching END ON directive.

The evaluation of any function referred to in the home expression is not affected by the ON directive; these functions are called on all processors active when control reached the directive. Thus,

!HPF$ ON HOME( P(1: (ACTIVE_NUM_PROCS() - 1)) ) ...

is a reasonable way to idle on one active processor, and is not paradoxically self-referential.

The HOME clause can name a program object, a template, or a processors arrangement. For each of these possibilities, it can specify a single element or multiple elements. This is translated into the processor(s) executing the ON block as follows:

• If the HOME clause names a program object, then every processor owning any part of that object should execute the ON block. For example, if A is an explicitly mapped array, then

  !HPF$ ON HOME ( A(2:4) )
  

  tells the compiler to perform the statement on the processors owning A(2), A(3), and A(4). If A were distributed BLOCK, this might be one processor; if it were distributed CYCLIC, it would be three processors (assuming that many processors were available). Extra copies of elements created by a SHADOW directive (H817) are not taken into consideration by the HOME clause.
• If the HOME clause names a template element or section, then every processor owning any element of the template element or section should execute the ON block. The example above applies here as well, if A is a template rather than an array.
• If the HOME clause names a processors arrangement, then the processor(s) referenced there should execute the ON block. For example, if P is a processors arrangement, then

  !HPF$ ON ( P(2:4) )
  

  will execute the following statement on the three processors P(2), P(3), and P(4).

  In every case, the ON directive specifies the processor(s) that should perform a computation. More formally, it sets the active processors for the statements governed by the ON directive, as described in Section 9.1. That section also describes how some statements (notably ALLOCATE and dynamic remapping directives) require that particular processors be included in the active set. If one of these constructs occurs in the ON block and the active processor set does not contain all the required processors, then the program is not standard-conforming.

  Note that the ON directive only specifies how computation is partitioned among processors; it does not indicate processors that may be involved in data transfer. Also, the ON clause by itself does not guarantee that its body can be executed in parallel with any other operation. However, placing the computation can have a significant effect on data locality. As later examples will show, the combination of ON and INDEPENDENT can also provide control over the load balance of parallel computations.

  Advice to implementors. If the HPF program is compiled into Single-Program-Multiple Data (SPMD) code, then the ON clause can always be implemented (albeit inefficently) by having all processors compare their processor id to an id (or list of ids) generated from the HOME clause. (Similar naive implementations can be constructed in other paradigms as well.) If the ON clause will be executed repeatedly, for example in a DO loop, it is worthwile to invert this process. That is, instead of all processors executing all the HOME clause tests, the compiler should determine the range of loop iterations that will test true on the given processor. (See the "Advice to implementors" in section 9.2.3 for more details.) For example, consider the following complex case:

  DO I = 1, N
      !HPF$ ON HOME( A(MY_FCN(I)) ) BEGIN
          ...
      !HPF$ END ON
  END DO
  

  Here, the generated code can perform an "inspector" (i.e., a skeleton loop that only evaluates the HOME clause of each iteration) to produce a list of iterations assigned to each processor. This list can be produced in parallel, since MY_FCN must be side-effect free (at least, the programmer cannot rely on any side effects). However, distributing the computatio of home to all processors may require unstructured communications patterns, possibly negating the advantage of parallelism. In general, more advanced compilers will be able to efficiently invert more complex HOME clauses. It is recommended that the abilities (and limitations) of a particular compiler be documented clearly for users.

  Note that pprocessors "screened out" by the naive iplementation may still be required to participate in data transfer. If the underlying architecture allows one-sided communication (e.g., shared memoy or GET/PUT), this is not a problem. On message-passing machines, a request-reply protocol may be used. This requires the inactive processors to enter a wait loop until the ON block completes, or requires the inactive processors to enter a wait loop until the ON block completes, or requires the runtime system to handle requests asynchronously. Again, it is recommended that the documnetation tell programmers which cases are likely to efficient and which inefficient on a particular system. (End of advice to implementors.)

  Advice to users. The form of the home in an ON directive can be arbitrarily complex. This is a two-edged sword; it can express very complicated computation partitioning, but the implementation of these partitions may not be efficient. More concretely, it may express a perfectly load-balanced computation, but force the compiler to serialize the computation to implement the HOME clauses. Although the amount of overhead for an ON clause will vary based on the HPF code, the compiler, and the hardware, one can expect that compilers will generate very good code based solely on array mappings or a named processors arrangement, and progressively worse code as the complexity of the home increases. A rough measure of the complexity of an ON directive is the amount of run-time data used to compute it; for example, a constant offset is fairly simple, while a permutation array is very complex. See Section 9.2.3 below for more concrete exaples of this phenomenon.

  It should also be noted that the ON clause does not change the semantics of a program, in the same sense that DISTRIBUTE does not change semantics. In particular, an ON clause by itself does not change sequential code into parallel code, because the code in the ON block can still interact with code outside the ON block. (To put it another way, ON does not spawn processes.) (End of advice to users.)

  It is legal to nest ON directives, if the set of active processors named by the inner ON directive is included in the set of active processors from the outer directive. The syntax of on-construct automatically ensures that it is properly nested inside other compound statements, and that compound statements properly nest inside of it. As with other Fortran compound statements, transfer of control to the interior of an on-construct from outside the block is prohibited: an on-construct may be entered only by executing the (executable) ON directive. Transfers within a block may occur. However, HPF also prohibits transfers of control from the interior of an on-construct to outside the on-construct, except by "falling through" the END ON directive. Note that this is stricter than in ordinary Fortran. If ON clauses are nested, then the innermost home effectively controls execution of the statement(s). A programmer can think of ths as successively restricting the set of processors at each level of ON nesting; clearly, the last restriction must be the strongest. Alternately, the programmer can think of this as a fork-join approach to nested parallelism.

  Rationale The restrictions about control flow into and out of an ON block essentially make it a single-entry single-exit region, thus simplifying the semantics considerably. (End of rationale.)

  If an ON directives includes a NEW clause, the meaning is the same as a NEW clause in an INDEPENDENT directive. The operation of the program would be identical if the NEW variables were allocated anew, and distributed onto the active processors, on every entry to the ON directive's scope, and deallocated on exit from the ON block. That is, the NEW variables are undefined on entry (i.e., assigned before use in the ON block) and undefined on exit (i.e., not used after the ON block, unless first reassigned). In addition, NEW variables cannot be remapped in the ON clause's scope, whether by REALIGN, REDISTRIBUTE, or by argument association (at subroutine calls). If a variable apears in a NEW clause but does not meet these conditions, then the program is not HPF-conforming. NEW variables are not considered by any nested RESIDENT directives, as detailed in Section 9.3.

  The NEW variables are implicitly reallocated and remapped onto the active processors on entry to the ON block. For this reason, there are restrictions on their explicit mappings.

  • An ON block NEW variable may not occur as an alignee.
  • An ON block NEW variable may occur as a distributee only if there is no ONTO clause

  !HPF$ DISTRIBUTE X(BLOCK, *)
  !HPF$ DISTRIBUTE Y ONTO P              ! Nonconforming due to ONTO clause
  !HPF$ ALIGN WITH X :: Z                ! Nonconforming; ALIGN forbidden
  !HPF$ ON (P(1:4), NEW(X, Y, Z), BEGIN
  !HPF$ END ON
  

  Rationale.NEW clauses provide a simple way to create temporary variables. This ability is particularly important when RESIDENT directives come into play, as will be clear below. (End of rationale.)
  Advice to implementors.Because they are not used outside of the ON blocks, NEW variables need not be kept consistent before and after ON clauses. Therefore, no communication outside of the active processor set, determined by the ON directive, is required to implement them. Scalar NEW variables should be replicated over the active processor set, or allocated in memory areas shared by the active processor set. Note that memory must be dynamically allocated if there is a possibility that multiple instances of the ON block could be active concurrently. This is similar to the requirements for implementing NEW variables in INDEPENDENT loops. (End of advice to implementors.)