1/* Part of SWI-Prolog 2 3 Author: Jan Wielemaker 4 E-mail: jan@swi-prolog.org 5 WWW: https://www.swi-prolog.org 6 Copyright (C): 2008-2026, University of Amsterdam 7 VU University Amsterdam 8 SWI-Prolog Solutions b.v. 9 All rights reserved. 10 11 Redistribution and use in source and binary forms, with or without 12 modification, are permitted provided that the following conditions 13 are met: 14 15 1. Redistributions of source code must retain the above copyright 16 notice, this list of conditions and the following disclaimer. 17 18 2. Redistributions in binary form must reproduce the above copyright 19 notice, this list of conditions and the following disclaimer in 20 the documentation and/or other materials provided with the 21 distribution. 22 23 THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS 24 "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT 25 LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS 26 FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE 27 COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, 28 INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, 29 BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; 30 LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER 31 CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT 32 LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN 33 ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE 34 POSSIBILITY OF SUCH DAMAGE. 35*/ 36 37:- module(pure_input, 38 [ phrase_from_file/2, % :Grammar, +File 39 phrase_from_file/3, % :Grammar, +File, +Options 40 phrase_from_stream/2, % :Grammar, +Stream 41 phrase_from_stream/3, % :Grammar, +Stream, +Options 42 stream_to_lazy_codes/2, % :Stream -Codes 43 stream_to_lazy_chars/2, % :Stream -Chars 44 stream_to_lazy_list/2, % :Stream -Codes 45 46 syntax_error//1, % +ErrorTerm 47 % Low level interface 48 lazy_list_location//1, % -Location 49 lazy_list_character_count//1 % -CharacterCount 50 ]). 51:- autoload(library(error), [type_error/2, permission_error/3, must_be/2]). 52:- autoload(library(option), [option/3]). 53 54:- set_prolog_flag(generate_debug_info, false). 55 56/** <module> Pure Input from files and streams 57 58This module is part of pio.pl, dealing with _pure_ _input_: processing 59input streams from the outside world using pure predicates, notably 60grammar rules (DCG). Using pure predicates makes non-deterministic 61processing of input much simpler. 62 63Pure input uses attributed variables to read input from the external 64source into a list _|on demand|_. The overhead of lazy reading is more 65than compensated for by using block reads based on read_pending_codes/3. 66 67Ulrich Neumerkel came up with the idea to use coroutining for creating a 68_lazy list_. His implementation repositioned the file to deal with 69re-reading that can be necessary on backtracking. The current 70implementation uses destructive assignment together with more low-level 71attribute handling to realise pure input on any (buffered) stream. 72 73@tbd Provide support for alternative input readers, e.g. reading 74 terms, tokens, etc. 75*/ 76 77:- predicate_options(phrase_from_stream/3, 3, 78 [ as(oneof([codes,chars])) 79 ]). 80:- predicate_options(phrase_from_file/3, 3, 81 [ pass_to(system:open/4, 4), 82 pass_to(phrase_from_stream/3, 3) 83 ]). 84 85%! phrase_from_file(:Grammar, +File) is nondet. 86%! phrase_from_file(:Grammar, +File, +Options) is nondet. 87% 88% Process the content of File using the DCG rule Grammar. The space 89% usage of this mechanism depends on the length of the not committed 90% part of Grammar. Committed parts of the temporary list are reclaimed 91% by the garbage collector, while the list is extended on demand due 92% to unification of the attributed tail variable. 93% 94% Options are passed to open/4 and phrase_from_stream/3. The latter 95% notable allows for as(chars) to call a DCG compiled for dealing with 96% character lists rather than codes. 97% 98% Below is an example that counts the number of times a string appears 99% in a file. The library dcg/basics provides string//1 matching an 100% arbitrary string and remainder//1 which matches the remainder of the 101% input without parsing. 102% 103% ``` 104% :- use_module(library(dcg/basics)). 105% 106% file_contains(File, Pattern) :- 107% phrase_from_file(match(Pattern), File). 108% 109% match(Pattern) --> 110% string(_), 111% string(Pattern), 112% remainder(_). 113% 114% match_count(File, Pattern, Count) :- 115% aggregate_all(count, file_contains(File, Pattern), Count). 116% ``` 117% 118% This can be called as (note that the pattern must be a string 119% (code list)): 120% 121% ``` 122% ?- match_count('pure_input.pl', `file`, Count). 123% ``` 124 125:- meta_predicate 126 phrase_from_file(//, +), 127 phrase_from_file(//, +, +), 128 phrase_from_stream(//, +). 129 130phrase_from_file(Grammar, File) :- 131 phrase_from_file(Grammar, File, []). 132 133phrase_from_file(Grammar, File, Options) :- 134 setup_call_cleanup( 135 open(File, read, In, Options), 136 phrase_from_stream(Grammar, In, Options), 137 close(In)). 138 139%! phrase_from_stream(:Grammar, +Stream). 140%! phrase_from_stream(:Grammar, +Stream, +Options). 141% 142% Run Grammer against the character codes on Stream. Stream must 143% be buffered. Options: 144% 145% - as(Type) 146% Where Type is one of `codes` (default) or `chars`. See also 147% stream_to_lazy_codes/2 and stream_to_lazy_chars/2. 148 149phrase_from_stream(Grammar, In) :- 150 phrase_from_stream(Grammar, In, []). 151 152phrase_from_stream(Grammar, In, Options) :- 153 option(as(Type), Options, codes), 154 must_be(oneof([codes,chars]), Type), 155 stream_to_lazy_list(In, List, Type), 156 phrase(Grammar, List). 157 158%! syntax_error(+Error)// 159% 160% Throw the syntax error Error at the current location of the 161% input. This predicate is designed to be called from the handler 162% of phrase_from_file/3. 163% 164% @throws error(syntax_error(Error), Location) 165 166syntax_error(Error) --> 167 lazy_list_location(Location), 168 { throw(error(syntax_error(Error), Location)) 169 }. 170 171%! lazy_list_location(-Location)// is det. 172% 173% Determine current (error) location in a lazy list. True when 174% Location is an (error) location term that represents the current 175% location in the DCG list. 176% 177% @arg Location is a term file(Name, Line, LinePos, CharNo) or 178% stream(Stream, Line, LinePos, CharNo) if no file is 179% associated to the stream RestLazyList. Finally, if the 180% Lazy list is fully materialized (ends in =|[]|=), Location 181% is unified with `end_of_file-CharCount`. 182% @see lazy_list_character_count//1 only provides the character 183% count. 184 185lazy_list_location(Location, Here, Here) :- 186 lazy_list_location(Here, Location). 187 188lazy_list_location(Here, Location) :- 189 '$skip_list'(Skipped, Here, Tail), 190 ( attvar(Tail) 191 -> get_attr(Tail, pure_input, State), 192 State = lazy_input(Stream, PrevPos, Pos, _Type, _), 193 Details = [Line, LinePos, CharNo], 194 ( stream_property(Stream, file_name(File)) 195 -> PosParts = [file, File|Details] 196 ; PosParts = [stream, Stream|Details] 197 ), 198 Location =.. PosParts, 199 ( PrevPos == (-) % nothing is read. 200 -> Line = 1, LinePos = 0, CharNo = 0 201 ; stream_position_data(char_count, Pos, EndRecordCharNo), 202 CharNo is EndRecordCharNo - Skipped, 203 set_stream_position(Stream, PrevPos), 204 stream_position_data(char_count, PrevPos, StartRecordCharNo), 205 Skip is CharNo-StartRecordCharNo, 206 forall(between(1, Skip, _), get_code(Stream, _)), 207 stream_property(Stream, position(ErrorPos)), 208 stream_position_data(line_count, ErrorPos, Line), 209 stream_position_data(line_position, ErrorPos, LinePos) 210 ) 211 ; Tail == [] 212 -> Location = end_of_file-Skipped 213 ; type_error(lazy_list, Here) 214 ). 215 216 217%! lazy_list_character_count(-CharCount)// 218% 219% True when CharCount is the current character count in the Lazy 220% list. The character count is computed by finding the distance to 221% the next frozen tail of the lazy list. CharCount is one of: 222% 223% - An integer 224% - A term end_of_file-Count 225% 226% @see lazy_list_location//1 provides full details of the location 227% for error reporting. 228 229lazy_list_character_count(Location, Here, Here) :- 230 lazy_list_character_count(Here, Location). 231 232lazy_list_character_count(Here, CharNo) :- 233 '$skip_list'(Skipped, Here, Tail), 234 ( attvar(Tail) 235 -> get_attr(Tail, pure_input, State), 236 arg(3, State, Pos), 237 stream_position_data(char_count, Pos, EndRecordCharNo), 238 CharNo is EndRecordCharNo - Skipped 239 ; Tail == [] 240 -> CharNo = end_of_file-Skipped 241 ; type_error(lazy_list, Here) 242 ). 243 244 245%! stream_to_lazy_codes(+Stream, -List) is det. 246%! stream_to_lazy_chars(+Stream, -List) is det. 247%! stream_to_lazy_list(+Stream, -List) is det. 248% 249% Create a lazy list representing the character codes (chars) in 250% Stream. List is a partial list ending in an attributed variable. 251% Unifying this variable reads the next block of data. The block is 252% stored with the attribute value such that there is no need to 253% re-read it. 254% 255% @deprecated stream_to_lazy_list/2 is a backward compatibility alias 256% for stream_to_lazy_codes/2. 257% @compat Unlike the previous version of this predicate this 258% version does not require a repositionable stream. It 259% does require a buffer size of at least the maximum 260% number of bytes of a multi-byte sequence (6). 261 262stream_to_lazy_codes(Stream, List) :- 263 stream_to_lazy_list(Stream, List, codes). 264stream_to_lazy_chars(Stream, List) :- 265 stream_to_lazy_list(Stream, List, chars). 266stream_to_lazy_list(Stream, List) :- 267 stream_to_lazy_list(Stream, List, codes). 268 269stream_to_lazy_list(Stream, List, Type) :- 270 ( stream_property(Stream, buffer(false)) 271 -> permission_error(create, lazy_list, Stream) 272 ; true 273 ), 274 stream_to_lazy_list(Stream, -, Type, List). 275 276stream_to_lazy_list(Stream, PrevPos, Type, List) :- 277 stream_property(Stream, position(Pos)), 278 put_attr(List, pure_input, lazy_input(Stream, PrevPos, Pos, Type, _)). 279 280attr_unify_hook(State, Value) :- 281 '$notrace'(attr_unify_hook_ndebug(State, Value)). 282 283attr_unify_hook_ndebug(State, Value) :- 284 State = lazy_input(Stream, _PrevPos, Pos, Type, Read), 285 ( var(Read) 286 -> fill_buffer(Stream), 287 ( Type == codes 288 -> read_pending_codes(Stream, NewList, Tail) 289 ; read_pending_chars(Stream, NewList, Tail) 290 ), 291 ( Tail == [] 292 -> nb_setarg(5, State, []), 293 Value = [] 294 ; stream_to_lazy_list(Stream, Pos, Type, Tail), 295 nb_linkarg(5, State, NewList), 296 Value = NewList 297 ) 298 ; Value = Read 299 )