1/* Part of SWI-Prolog 2 3 Author: Jan Wielemaker 4 E-mail: jan@swi-prolog.org 5 WWW: https://www.swi-prolog.org 6 Copyright (C): 2008-2026, University of Amsterdam 7 VU University Amsterdam 8 SWI-Prolog Solutions b.v. 9 All rights reserved. 10 11 Redistribution and use in source and binary forms, with or without 12 modification, are permitted provided that the following conditions 13 are met: 14 15 1. Redistributions of source code must retain the above copyright 16 notice, this list of conditions and the following disclaimer. 17 18 2. Redistributions in binary form must reproduce the above copyright 19 notice, this list of conditions and the following disclaimer in 20 the documentation and/or other materials provided with the 21 distribution. 22 23 THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS 24 "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT 25 LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS 26 FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE 27 COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, 28 INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, 29 BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; 30 LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER 31 CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT 32 LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN 33 ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE 34 POSSIBILITY OF SUCH DAMAGE. 35*/ 36 37:- module(pure_input, 38 [ phrase_from_file/2, % :Grammar, +File 39 phrase_from_file/3, % :Grammar, +File, +Options 40 phrase_from_stream/2, % :Grammar, +Stream 41 phrase_from_stream/3, % :Grammar, +Stream, +Options 42 stream_to_lazy_codes/2, % :Stream -Codes 43 stream_to_lazy_chars/2, % :Stream -Chars 44 stream_to_lazy_list/2, % :Stream -Codes 45 46 syntax_error//1, % +ErrorTerm 47 % Low level interface 48 lazy_list_location//1, % -Location 49 lazy_list_character_count//1 % -CharacterCount 50 ]). 51:- autoload(library(error), [type_error/2, permission_error/3, must_be/2]). 52:- autoload(library(option), [option/3]). 53 54:- set_prolog_flag(generate_debug_info, false).
77:- predicate_options(phrase_from_stream/3, 3, 78 [ as(oneof([codes,chars])) 79 ]). 80:- predicate_options(phrase_from_file/3, 3, 81 [ pass_to(system:open/4, 4), 82 pass_to(phrase_from_stream/3, 3) 83 ]).
Options are passed to open/4 and phrase_from_stream/3. The latter
notable allows for as(chars) to call a DCG compiled for dealing with
character lists rather than codes.
Below is an example that counts the number of times a string appears in a file. The library dcg/basics provides string//1 matching an arbitrary string and remainder//1 which matches the remainder of the input without parsing.
:- use_module(library(dcg/basics)).
file_contains(File, Pattern) :-
phrase_from_file(match(Pattern), File).
match(Pattern) -->
string(_),
string(Pattern),
remainder(_).
match_count(File, Pattern, Count) :-
aggregate_all(count, file_contains(File, Pattern), Count).
This can be called as (note that the pattern must be a string (code list)):
?- match_count('pure_input.pl', `file`, Count).
125:- meta_predicate 126 phrase_from_file(//, +), 127 phrase_from_file(//, +, +), 128 phrase_from_stream(//, +). 129 130phrase_from_file(Grammar, File) :- 131 phrase_from_file(Grammar, File, []). 132 133phrase_from_file(Grammar, File, Options) :- 134 setup_call_cleanup( 135 open(File, read, In, Options), 136 phrase_from_stream(Grammar, In, Options), 137 close(In)).
codes (default) or chars. See also
stream_to_lazy_codes/2 and stream_to_lazy_chars/2.149phrase_from_stream(Grammar, In) :- 150 phrase_from_stream(Grammar, In, []). 151 152phrase_from_stream(Grammar, In, Options) :- 153 option(as(Type), Options, codes), 154 must_be(oneof([codes,chars]), Type), 155 stream_to_lazy_list(In, List, Type), 156 phrase(Grammar, List).
166syntax_error(Error) -->
167 lazy_list_location(Location),
168 { throw(error(syntax_error(Error), Location))
169 }.185lazy_list_location(Location, Here, Here) :- 186 lazy_list_location(Here, Location). 187 188lazy_list_location(Here, Location) :- 189 '$skip_list'(Skipped, Here, Tail), 190 ( attvar(Tail) 191 -> get_attr(Tail, pure_input, State), 192 State = lazy_input(Stream, PrevPos, Pos, _Type, _), 193 Details = [Line, LinePos, CharNo], 194 ( stream_property(Stream, file_name(File)) 195 -> PosParts = [file, File|Details] 196 ; PosParts = [stream, Stream|Details] 197 ), 198 Location =.. PosParts, 199 ( PrevPos == (-) % nothing is read. 200 -> Line = 1, LinePos = 0, CharNo = 0 201 ; stream_position_data(char_count, Pos, EndRecordCharNo), 202 CharNo is EndRecordCharNo - Skipped, 203 set_stream_position(Stream, PrevPos), 204 stream_position_data(char_count, PrevPos, StartRecordCharNo), 205 Skip is CharNo-StartRecordCharNo, 206 forall(between(1, Skip, _), get_code(Stream, _)), 207 stream_property(Stream, position(ErrorPos)), 208 stream_position_data(line_count, ErrorPos, Line), 209 stream_position_data(line_position, ErrorPos, LinePos) 210 ) 211 ; Tail == [] 212 -> Location = end_of_file-Skipped 213 ; type_error(lazy_list, Here) 214 ).
229lazy_list_character_count(Location, Here, Here) :- 230 lazy_list_character_count(Here, Location). 231 232lazy_list_character_count(Here, CharNo) :- 233 '$skip_list'(Skipped, Here, Tail), 234 ( attvar(Tail) 235 -> get_attr(Tail, pure_input, State), 236 arg(3, State, Pos), 237 stream_position_data(char_count, Pos, EndRecordCharNo), 238 CharNo is EndRecordCharNo - Skipped 239 ; Tail == [] 240 -> CharNo = end_of_file-Skipped 241 ; type_error(lazy_list, Here) 242 ).
262stream_to_lazy_codes(Stream, List) :- 263 stream_to_lazy_list(Stream, List, codes). 264stream_to_lazy_chars(Stream, List) :- 265 stream_to_lazy_list(Stream, List, chars). 266stream_to_lazy_list(Stream, List) :- 267 stream_to_lazy_list(Stream, List, codes). 268 269stream_to_lazy_list(Stream, List, Type) :- 270 ( stream_property(Stream, buffer(false)) 271 -> permission_error(create, lazy_list, Stream) 272 ; true 273 ), 274 stream_to_lazy_list(Stream, -, Type, List). 275 276stream_to_lazy_list(Stream, PrevPos, Type, List) :- 277 stream_property(Stream, position(Pos)), 278 put_attr(List, pure_input, lazy_input(Stream, PrevPos, Pos, Type, _)). 279 280attr_unify_hook(State, Value) :- 281 '$notrace'(attr_unify_hook_ndebug(State, Value)). 282 283attr_unify_hook_ndebug(State, Value) :- 284 State = lazy_input(Stream, _PrevPos, Pos, Type, Read), 285 ( var(Read) 286 -> fill_buffer(Stream), 287 ( Type == codes 288 -> read_pending_codes(Stream, NewList, Tail) 289 ; read_pending_chars(Stream, NewList, Tail) 290 ), 291 ( Tail == [] 292 -> nb_setarg(5, State, []), 293 Value = [] 294 ; stream_to_lazy_list(Stream, Pos, Type, Tail), 295 nb_linkarg(5, State, NewList), 296 Value = NewList 297 ) 298 ; Value = Read 299 )
Pure Input from files and streams
This module is part of
pio.pl, dealing with pure input: processing input streams from the outside world using pure predicates, notably grammar rules (DCG). Using pure predicates makes non-deterministic processing of input much simpler.Pure input uses attributed variables to read input from the external source into a list on demand. The overhead of lazy reading is more than compensated for by using block reads based on read_pending_codes/3.
Ulrich Neumerkel came up with the idea to use coroutining for creating a lazy list. His implementation repositioned the file to deal with re-reading that can be necessary on backtracking. The current implementation uses destructive assignment together with more low-level attribute handling to realise pure input on any (buffered) stream.