SWI-Prolog -- Reloading files, active code and threads

4.3.2 Reloading files, active code and threads

Traditionally, Prolog environments allow for reloading files holding currently active code. In particular, the following sequence is a valid use of the development environment:

Trace a goal
Find unexpected behaviour of a predicate
Enter a break using the b command
Fix the sources and reload them using make/0
Exit the break, retry executing the now fixed predicate using the r command

Reloading a previously loaded file is safe, both in the debug scenario above and when the code is being executed by another thread. Executing threads switch atomically to the new definition of modified predicates, while clauses that belong to the old definition are (eventually) reclaimed by garbage_collect_clauses/0.^{62As
of version 7.3.12. Older versions wipe all clauses originating from the
file before loading the new clauses. This causes threads that executes
the code to (typically) die with an undefined predicate
exception.} Below we describe the steps taken for reloading a file to help understanding the limitations of the process.

If a file is being reloaded, a reload context is associated to the file administration. This context includes a table keeping track of predicates and a table keeping track of the module(s) associated with this source.
If a new predicate is found, an entry is added to the context predicate table. Three options are considered:
1. The predicate is new. It is handled the same as if the file was loaded for the first time.
2. The predicate is foreign or thread local. These too are treated as if the file was loaded for the first time.
3. Normal predicates. Here we initialise a pointer to the current clause.
New clauses for‘normal predicates’are considered as follows:
1. If the clause's byte-code is the same as the predicates current clause, discard the clause and advance the current clause pointer.
2. If the clause's byte-code is the same as some clause further into the clause list of the predicate, discard the new clause, mark all intermediate clauses for future deletion, and advance the current clause pointer to the first clause after the matched one.
3. If the clause's byte-code matches no clause, insert it for future activation before the current clause and keep the current clause.
Properties such as dynamic or meta_predicate are in part applied immediately and in part during the fixup process after the file completes loading. Currently, dynamic and thread_local are applied immediately.
New modules are recorded in the reload context. Export declarations (the module's public list and export/1 calls) are both applied and recorded.
When the end-of-file is reached, the following fixup steps are taken
1. For each predicate
  1. The current clause and subsequent clauses are marked for future deletion.
  2. All clauses marked for future deletion or creation are (in)activated by changing their‘erased’or‘created’ generation. Erased clauses are (eventually) reclaimed by the clause garbage collector, see garbage_collect_clauses/0.
  3. Pending predicate property changes are applied.
2. For each module
  1. Exported predicates that are not encountered in the reload context are removed from the export list.

The above generally ensures that changes to the content of source files can typically be activated safely using make/0. Global changes such as operator changes, changes of module names, changes to multi-file predicates, etc. sometimes require a restart. In almost all cases, the need for restart is indicated by permission or syntax errors during the reload or existence errors while running the program.

In some cases the content of a source file refers‘to itself’. This is notably the case if local rules for goal_expansion/2 or term_expansion/2 are defined or goals are executed using directives.^{63Note that initialization/1
directives are executed after loading the file. SWI-Prolog
allows for directives that are executed while loading the file
using :- Goal. or initialization/2}. Up to version 7.5.12 it was typically needed to reload the file twice, once for updating the code that was used for compiling the remainder of the file and once to effectuate this. As of version 7.5.13, conventional transaction semantics apply. This implies that for the thread performing the reload the file's content is first wiped and gradually rebuilt, while other threads see an atomic update from the old file content to the new.^{64This feature was
implemented by Keri Harris.}

4.3.2.1 Errors and warnings during compilation

Errors and warnings reported while compiling a file are reported using print_message/2. Typical errors are syntax errors, errors during macro expansion by term_expansion/2 and goal_expansion/2, compiler errors such as illegal clauses or an attempt to redefine a system predicate and errors caused by executing directives, notably using initialization/1 and initialization/2.

Merely reporting error messages and warnings is typically desirable for interactive usage. Non-interactive applications often require to be notified of such issues, typically using the exit code of the process. We can distinguish two types of errors and warnings: (1) those resulting from loading an invalid program and (2) messages that result from running the program. A typical example is user code that wishes to try something and in case of an error report this and continue.

    ...,
    E = error(_,_),
    catch(do_something, E,
          print_message(error, E)),
    ...

User code may be (and often is) started from directives, while running user code may involve compilation due to autoloading, loading of data files, etc. As a result, it is unclear whether an error message should merely be printed, should result in a non-zero exit status at the end or should immediately terminate the process.

The default behaviour is defined by the Prolog flags on_error and on_warning. It can be fine tuned by defining the hook predicate message_hook/3. The compiler calls print_message/2 using the level silent and the message below if errors or warnings where printed during the execution of load_files/2.

load_file_errors(File, Errors, Warnings): Here, File is the raw file specification handed to load_files/2, i.e., ’myfile.pl’ or library(lists), Errors is the number of errors printed while loading and Warnings is the number of warnings printed while loading. Note that these counts include messages from (initialization) directives.

This allows the user to fine tune the behaviour on errors and, for example, halt the process on a non-zero error count right after loading the file wth errors using the code below.

:- multifile user:message_hook/3.

user:message_hook(load_file_errors(_File, Errors, _Warnings),
                  _Level, _Lines) :-
    Errors > 0,
    halt(1).

4.3.2.2 Compilation of mutually dependent code

Large programs are generally split into multiple files. If file A accesses predicates from file B which accesses predicates from file A, we consider this a mutual or circular dependency. If traditional load predicates (e.g., consult/1) are used to include file B from A and A from B, loading either file results in a loop. This is because consult/1 is mapped to load_files/2 using the option if(true)(.) Such programs are typically loaded using a load file that consults all required (non-module) files. If modules are used, the dependencies are made explicit using use_module/1 statements. The use_module/1 predicate, however, maps to load_files/2 with the option if(not_loaded)(.) A use_module/1 on an already loaded file merely makes the public predicates of the used module available.

Summarizing, mutual dependency of source files is fully supported with no precautions when using modules. Modules can use each other in an arbitrary dependency graph. When using consult/1, predicate dependencies between loaded files can still be arbitrary, but the consult relations between files must be a proper tree.

4.3.2.3 Compilation with multiple threads

This section discusses compiling files for the first time. For reloading, see section 4.3.2.

Multiple threads can compile files concurrently. This requires special precautions only if multiple threads wish to load the same file at the same time. Therefore, load_files/2 checks whether some other thread is already loading the file. If not, it starts loading the file. If a thread detects that another thread is already loading the file the thread blocks until the other thread finishes loading the file. After waiting, and if the file is a module file, it imports the exported predicates and operators from the module.

Note that this schema does not prevent deadlocks under all situations. Consider two mutually dependent (see section 4.3.2.2) module files A and B, where thread 1 starts loading A and thread 2 starts loading B at the same time. Both threads will deadlock when trying to load the used module.

The current implementation does not detect such cases and the involved threads will freeze. This problem can be avoided if a mutually dependent collection of files is always loaded from the same start file.