Scieneer Common Lisp 1.3.9 online documentation

Fast Streams

Introduction

The Scieneer Common Lisp streams are designed for high performance buffered operation, and also support unbuffered operation. All streams include quickly accessible buffers to speed the basic character and bytes operations. When the input or output buffer has a zero length the stream operates in an effectively unbuffered mode for input or output respectively. Streams that are largely compatible with the popular Gray Streams extension can be easily defined as unbuffered streams. Each stream may be either a character or a binary stream, and when both character and binary operations are necessary on a stream this may be implemented by encapsulating a binary stream with a character conversion stream.

Only simple-string and simple (unsigned-byte 8) vector buffers are supported to avoid dispatch overhead on general stream buffer operations. The simplicity of the stream buffer access supports the inlining of the basic stream character and byte operations for high performance. Many of the basic stream functions may be inline expanded including: read-char, unread-char, read-byte, ext:unread-byte, peek-char, ext:peek-byte, listen, read-char-no-hang, ext:read-byte-no-hang, write-char, write-byte, ext:write-char-no-hang, and ext:write-byte-no-hang.

Since only (unsigned-byte 8) vector buffers are supported for binary streams the only element type supported for fast buffered binary streams is also (unsigned-byte 8). However it is possible to define unbuffered binary streams that use other element types although these are not provided in the Scieneer Common Lisp.

The base stream is a standard-object, and new stream classes may be defined by implementing methods for a number of stream generic functions. For buffered streams, only a small set of stream buffer methods need to be defined - methods for peeking and unreading are not necessary for buffered streams as these operations are implement using the buffers. For unbuffered streams a set of stream character or byte methods need to be defined, which must include peek and unread methods for an input stream.

The design borrows from the Gray Streams proposal, but adds a fast buffer layer. A subset of the Gray Streams character and byte stream generic functions have been complemented by a smaller set of stream buffer generic functions that can significantly increase performance for character and binary '(unsigned-byte 8) streams. Only a subset of the Gray Streams functions are used by the implementation, in particular the following are not defined: stream-read-line, stream-write-string, stream-terpri, stream-fresh-line, stream-advance-to-column, stream-start-line-p, stream-read-sequence, and stream-write-sequence.

Stream extenstions

A few extra base stream functions have been added.

The functions ext:unread-byte and ext:peek-byte have been added and are handy when processing byte streams, as for the respective character stream functions unread-char and peek-char. The function ext:read-byte-no-hang may be used to read input if available on a byte stream, as read-char-no-hang does for character streams.

A function for determining the current line column for a character output stream is provide by ext:line-column. This function accounts for buffered output in contrast to the generic function ext:stream-line-column.

The functions ext:read-chars, ext:write-chars, ext:read-bytes, and ext:write-bytes are needed when writing composite stream methods which read or write to other streams. These parallel the stream buffer generic functions ext:stream-read-chars, ext:stream-write-chars, ext:stream-read-bytes, and ext:stream-write-bytes respectively.

ext:unread-byte byte stream[Function]

Puts the byte back on the front of the input stream.

ext:peek-byte stream &optional peek-type eof-error-p eof-value recursive-p[Function]

Peeks at the next byte in the input stream.

ext:line-column &optional output-stream[Function]

Returns the number of characters on the current line of output of the given output stream, otherwise nil if that information is not available.

ext:read-chars stream string start requested &optional waitp[Function]

Attempt to read the requested number of characters into the string starting at start, returning the number of characters read or -1 if at the end-of-file. On pipes or similar devices, this function returns as soon as any data is available, even if the amount read is less than requested and the end-of-file has not been reached. When waitp is true, wait for at least one character before returning, otherwise return zero if no input is available.

ext:write-chars stream string &optional start end waitp[Function]

Attempt to write the string from the start to the end position to the stream, returning the number of characters consumed. If waitp is true then wait until all characters have been consumed before returning. When waitp is false, consume as many characters as possible without blocking, returning the number of characters consumed which may be zero. The characters may be buffered in the stream output buffer. When less characters are consumed than requested, the function ext:stream-waiting-file-descriptor may be called to find the file descriptor and directions blocked and when ready the function may be called again to continue the operation.

ext:read-bytes stream buffer start requested &optional waitp[Function]

Attempt to read the requested number of bytes into the buffer starting at start, returning the number of bytes read or -1 if at the end-of-file. On pipes or similar devices, this function returns as soon as any data is available, even if the amount read is less than requested and the end-of-file has not been hit. When waitp is true, wait for at least one byte before returning, otherwise return zero if no input is available.

ext:write-bytes stream bytes &optional start end waitp[Function]

Attempt to write the bytes vector from the start to the end position to the stream, returning the number of bytes consumed. If waitp is true then wait until all bytes have been consumed before returning. When waitp is false, consume as many bytes as possible without blocking, returning the number of bytes consumed which may be zero. The bytes may be buffered in the stream output buffer. When less bytes are consumed than requested, the function ext:stream-waiting-file-descriptor may be called to find the file descriptor and directions blocked and when ready the function may be called again to continue the operation.

ext:read-byte-no-hang stream &optional eof-error-p eof-value recursive-p[Function]

Returns the next byte from the stream if one is availible, otherwise nil.


Byte vector input and output streams

Byte vector input and output streams similar to the standard string input and output streams are provided.

ext:make-byte-input-stream bytes &optional start end[Function]

Returns a binary input stream which will supply bytes from the bytes vector between the start and end positionss.

ext:byte-input-stream[Class]

Class precedence list:

ext:byte-input-stream, ext:binary-input-stream, ext:input-stream, ext:binary-stream, stream, standard-object, ext:instance, t.

Byte vector binary input stream.

ext:with-input-from-bytes (var bytes &key :start :end) &parse-body (forms decls)[Macro]

Binds the var to an input stream that returns bytes from the bytes vector and executes the body.

ext:make-byte-output-stream &key size[Function]

Returns a binary output stream which will accumulate all output written to it. The output may be obtained by the function ext:get-output-stream-bytes.

ext:byte-output-stream[Class]

Class precedence list:

ext:byte-output-stream, ext:binary-output-stream, ext:output-stream, ext:binary-stream, stream, standard-object, ext:instance, t.

Byte vector binary output stream.

ext:get-output-stream-bytes stream[Function]

Returns a byte vector of all the bytes written to the stream made by ext:make-byte-output-stream since the last call to this function.

ext:with-output-to-bytes (var) &parse-body (forms decls)[Macro]

Binds var to a byte output stream. The output is incrementally appended to a byte vector which is returned.


Character conversion streams

Character conversion streams are provided for conversion between a byte stream and a character stream. Given an implementation of a binary stream, these composite streams allow character input and output on the stream.

ext:character-conversion-stream[Class]

Class precedence list:

ext:character-conversion-stream, ext:encapsulating-character-stream, ext:encapsulating-stream, ext:character-stream, stream, standard-object, ext:instance, t.

Base class for character conversion streams that present a character stream interface and convert to a byte stream.

ext:character-conversion-input-stream[Class]

Class precedence list:

ext:character-conversion-input-stream, ext:character-conversion-stream, ext:encapsulating-character-input-stream, ext:encapsulating-input-stream, ext:encapsulating-character-stream, ext:encapsulating-stream, ext:character-input-stream, ext:input-stream, ext:character-stream, stream, standard-object, ext:instance, t.

Character conversion input stream.

ext:character-conversion-output-stream[Class]

Class precedence list:

ext:character-conversion-output-stream, ext:character-conversion-stream, ext:encapsulating-character-output-stream, ext:encapsulating-output-stream, ext:encapsulating-character-stream, ext:encapsulating-stream, ext:character-output-stream, ext:output-stream, ext:character-stream, stream, standard-object, ext:instance, t.

Character conversion output stream. This stream keeps track of the line column and optionally performs line buffering.

ext:character-conversion-input-output-stream[Class]

Class precedence list:

ext:character-conversion-input-output-stream, ext:character-conversion-input-stream, ext:character-conversion-output-stream, ext:character-conversion-stream, ext:encapsulating-character-output-stream, ext:encapsulating-output-stream, ext:encapsulating-character-input-stream, ext:encapsulating-input-stream, ext:encapsulating-character-stream, ext:encapsulating-stream, ext:character-input-stream, ext:input-stream, ext:character-output-stream, ext:output-stream, ext:character-stream, stream, standard-object, ext:instance, t.

Character conversion input and output stream.

ext:make-character-conversion-stream stream &key input output external-format close-stream-p buffering in-buffer-size out-buffer-size finalizep pathname input-error-value output-error-value[Function]

Create a character conversion stream that reads and/or writes the given binary stream.

The functions file-position and file-length are fully supported on all character conversion streams when the character encoded width is a fixed size and the encapsulated binary stream also supports these functions, and are also supported when the character conversion stream is a file input stream. However when the character encoded width is variable and the character conversion stream is not a file input stream then file-length is not support and file-position has only limited support for positioning to the start or end of the file. The position and length of character streams are measured in units of characters, and determining the position within a file with a variable character encoding size requires reading the file to count the number of characters. So if the file-length is needed or the file-position is being changed on an output file using a variable width character encoding then the file must also be opened for input. Overwriting a file with a variable character encoding size is permitted but may change the file length or leave the file corrupted.

The default action upon an encoding or decoding error is to signal an appropriate condition, however it is possible to substitute a character or ignore errors if possible. Note that ignoring input or output errors affects the file-length and file-position.

:input
If true, allow input operations. This is the default if neither :input or :output is specified.
:output
If true, allow output operations.
:external-format
The external character format to use. The default value is :default for which the value of ext:*default-external-format* is used.
:close-stream-p
If true then the target stream is closed when this stream is closed.
:buffering
Indicates the kind of output buffer flushing to use which may be one of: :none, :line, or the default :full.
:pathname
When true, the returned stream will be a subclass of the class file-stream. This should generally be true if the stream is a file-stream.
:input-error-value
A character or character code to be substituted upon a decoding error. When nil, decoding errors are ignored and an attempt is made to move on to the next character. When the value is the keyword :error, which is the default, an ext:character-decoding-error condition is signaled upon a decoding error.
:output-error-value
A character or character code to be substituted upon an encoding error. When nil, encoding errors are ignored and the character output is skipped. When the value is the keyword :error, which is the default, an ext:character-encoding-error condition is signaled upon an encoding error.

ext:character-conversion-stream-stream stream[Function]

Returns the target binary stream associated with the given character conversion stream.


Stream classes

All stream classes inherit from the base stream class, and some classes are defined for hanging methods off.

stream[Type]

Class precedence list:

stream, standard-object, ext:instance, t.

Base class for all streams.

ext:input-stream[Class]

Class precedence list:

ext:input-stream, stream, standard-object, ext:instance, t.

A sub-class of stream, for which input-stream-p returns true.

ext:output-stream[Class]

Class precedence list:

ext:output-stream, stream, standard-object, ext:instance, t.

A sub-class of stream, for which output-stream-p returns true.

ext:character-stream[Class]

Class precedence list:

ext:character-stream, stream, standard-object, ext:instance, t.

A subclass of stream, for which stream-element-type returns base-char.

ext:binary-stream[Class]

Class precedence list:

ext:binary-stream, stream, standard-object, ext:instance, t.

A subclass of stream, for which stream-element-type returns '(unsigned-byte 8).

ext:character-input-stream[Class]

Class precedence list:

ext:character-input-stream, ext:input-stream, ext:character-stream, stream, standard-object, ext:instance, t.

A subclass of both class input-stream and class character-stream.

ext:character-output-stream[Class]

Class precedence list:

ext:character-output-stream, ext:output-stream, ext:character-stream, stream, standard-object, ext:instance, t.

A subclass of both class output-stream and class character-stream.

ext:binary-input-stream[Class]

Class precedence list:

ext:binary-input-stream, ext:input-stream, ext:binary-stream, stream, standard-object, ext:instance, t.

A subclass of both class input-stream and class binary-stream.

ext:binary-output-stream[Class]

Class precedence list:

ext:binary-output-stream, ext:output-stream, ext:binary-stream, stream, standard-object, ext:instance, t.

A subclass of both class output-stream and class binary-stream.

ext:encapsulating-stream[Class]

Class precedence list:

ext:encapsulating-stream, stream, standard-object, ext:instance, t.

A mixin class for composite streams that encapsulate another stream. All the stream methods are defined and pass through to the encapsulated stream. The close method only closes the encapsulating stream and not the encapsulated stream. This is a useful mixin class for basic composite streams because it defines all the required methods.


User definable streams

A new stream may be defined by defining a stream class and an appropriate set of methods for the following generic functions.

A few of the functions do not interact with the stream buffers, and can be called without consideration of buffering: open-stream-p, input-stream-p, output-stream-p, stream-element-type, ext:stream-line-length, interactive-stream-p, ext:stream-pathname, ext:stream-timeout, ext:stream-expiration, and ext:stream-file-descriptor.

A buffered character stream must defined methods for: ext:stream-read-chars, and ext:stream-write-chars. A buffered byte stream must define methods for: ext:stream-read-bytes, and ext:stream-write-bytes. These methods are typically only called when the stream buffer is exhausted, thus they need not interact with the stream buffer. When composite stream methods operate on other streams they will typically call functions that respect the buffers on the other streams. For example an ext:stream-read-chars method may call ext:read-chars on a source stream.

An unbuffered character stream must define methods for: ext:stream-read-char, ext:stream-read-char-no-hang, ext:stream-unread-char, ext:stream-peek-char, and ext:stream-write-char. An unbuffered byte stream must define methods for: ext:stream-read-byte, ext:stream-read-byte-no-hang, ext:stream-unread-byte, ext:stream-peek-byte, and ext:stream-write-byte. These methods are called by the implementation for streams with zero length input or output buffers. A stream may use different buffering for input and output.

A file stream may have a pathname which is returned by the Common Lisp function pathname and this function is implemented by ext:stream-pathname. User defined file streams can add a method for this function, and user defined composite streams may define a method to return the underlying stream pathname.

Streams may have a timeout for individual blocking operations, and an expiration timer to bound the time taken for a series of stream operations. User defined streams may define methods for ext:stream-timeout and ext:stream-expiration to implement their own timers, and user defined composite streams may provide a methods to pass the calls to the source and target streams which may implement the timers.

UNIX file and socket based streams have an associated file descriptor which is a number used to refer to the stream at the level of the UNIX interface. For these streams the file descriptor is returned by the function sys:fd-stream-fd, and this is implemented by the generic function ext:stream-file-descriptor. Methods are provided for composite streams to return the file descriptor of the underlying stream, and user defined composite streams should define a method for ext:stream-file-descriptor if it is to work with sys:fd-stream-fd.

open-stream-p stream[Generic Function]

Return true if stream is not closed. A default method is provided for the class stream which returns true if close has not been called on the stream.

input-stream-p stream[Generic Function]

Returns true if the given stream can perform input operations. Default methods are provided for class stream which returns false, and for class input-stream which returns true.

output-stream-p stream[Generic Function]

Returns true if the given stream can perform output operations. Default methods are provided for class stream which returns false and for class output-stream which returns true.

close stream &key abort[Generic Function]

Closes the given stream. No more I/O may be performed, but inquiries may still be made. If :abort is false then an attempt is made to finish-output on the stream and clean up the side effects of having created the stream. A default method is provided for the class stream that clears the flag read by open-steam-p.

ext:stream-clear-input stream[Generic Function]

Clear any unread input, bypassing the stream input buffer. The caller should firstly clear the input buffer if desired. Called by clear-input after clearing the stream input buffer. A default method is provided for the class ext:input-stream which does nothing.

ext:stream-finish-output stream[Generic Function]

Attempt to finish sending any buffered output before returning. Output buffered in the stream output buffer is not sent - the caller should firstly flush the output buffer if desired. Called by finish-output after flushing the stream output buffer. A default method is provided for the class ext:output-stream which does nothing.

ext:stream-force-output stream[Generic Function]

Attempt to force any buffered output to be sent. Output buffered in the stream output buffer is not sent - the caller should firstly flush the output buffer if desired. Called by force-output after flushing the stream output buffer. A default method is provided for the class ext:output-stream which does nothing.

ext:stream-clear-output stream[Generic Function]

Clear the output stream. The stream output buffer is not cleared, which if desired is the responsibility of the caller. Called by clear-output after clearing the stream output buffer. A default method is provided for the class ext:output-stream which does nothing.

stream-element-type stream[Generic Function]

Returns a type specifier for the kind of object returned by the stream. Default methods are provided for class character-stream which returns base-char, and for class binary-stream which returns '(unsigned-byte 8).

ext:stream-line-length output-stream[Generic Function]

Return the output-stream line length otherwise nil if unknown. Called by line-length. A default method for the class character-output-stream returns nil.

ext:stream-line-column output-stream[Generic Function]

Return the number of characters on the current line of output of the given output-stream, otherwise nil if unknown. This function bypasses the output buffer. Called by line-column, after flushing the output-stream output buffer. A default method for the class character-output-stream returns nil.

ext:stream-file-position stream &optional position[Generic Function]

When the position is nil, returns the current position within the stream or nil if the position can not be determined. When a position is given it becomes the new file position, returning t if the position is successfully changed otherwise returning nil. The second argument may also be :start or :end for the start and end of the file, respectively.

The position is measured in units of bytes for binary streams and in units of characters for character streams, and is defined as the number of bytes or characters returned by read-byte or read-char when reading from the start of the file to the current position. The position may not be the same as the position in the underlying file due to non-unity encoding. For example: character encoding streams, compressed binary streams, or end-of-line conversion streams.

For each byte or character written the position change will typically be one but in general may be more or less than one. For example: an indenting output stream can insert characters at the start of each line resulting in a multi-character change in position when a linefeed is written; or a character conversion output stream may ignore and drop characters that can not be encoded. The caller is expected to firstly write and finish any pending buffered output so that the output can be processed and the position updated, otherwise position may not be correct and the output may be written to the wrong location. The caller is also expected to compensate for any buffered input on the stream. A default method is provided for the class stream that does nothing and returns nil.

interactive-stream-p stream[Generic Function]

Return true if stream does I/O on a terminal or other interactive device.

ext:stream-read-char stream[Generic Function]

Reads one character from the stream. It returns either a character or the symbol :eof if the stream is at end-of-file. Every unbuffered character input stream must define a method for this function.

ext:stream-read-char-no-hang stream[Generic Function]

Returns either a character from the stream, nil if no input is currently available, or :eof if the stream is at the end-of-file. The default method calls ext:stream-read-char which should be fine for a file stream, however unbuffered character interactive and socket streams will probably want to defined a method for this function.

ext:stream-unread-char stream character[Generic Function]

Un-does the last call to read-char, as in unread-char. Returns nil. Every unbuffered character input stream must define a method for this function.

ext:stream-peek-char stream[Generic Function]

Used to implement peek-char; this corresponds to a peek-type of nil. It returns either a character or :eof. The default method calls read-char and unread-char.

ext:stream-read-chars stream string start requested waitp[Generic Function]

Attempt to read the requested number of characters from the stream into the given string starting at start, and return the number of characters read or -1 if at the eof-of-file. On pipes or similar devices, this function returns as soon as any data is available, even if the amount read is less than requested and the end-of-file has not been reached. When waitp is true, wait for at least one character before returning, otherwise return zero if no input is available. The stream input buffer is bypassed. Called by read-chars when the stream input buffer is empty. See the documentation for read-chars. Every buffered character input stream must define a method for this function. When less characters are read than requested, the function ext:stream-waiting-file-descriptor may be called to find the file descriptor and directions blocked and when ready the function may be called again to continue the operation.

ext:stream-write-char stream character[Generic Function]

Writes character to the stream and returns the character. Every unbuffered character output stream must have a method defined for this function.

ext:stream-write-chars stream string start end waitp[Generic Function]

Attempt to write the string from the start to the end position to the stream, returning the number of characters written. If waitp is true then wait until all characters have been written before returning. When waitp is false, write as many characters as possible without blocking, returning the number of characters written which may be zero. The write bypasses the stream output buffer. Called by write-chars to write the stream buffer. Every buffered character output stream must define a method for this function. When less characters are written than requested, the function ext:stream-waiting-file-descriptor may be called to find the file descriptor and directions blocked and when ready the function may be called again to continue the operation.

ext:stream-read-byte stream[Generic Function]

Reads one byte from the stream. It returns either an integer or the symbol :eof if the stream is at end-of-file. Every unbuffered binary input stream must define a method for this function.

ext:stream-read-byte-no-hang stream[Generic Function]

Returns either a byte from the stream, nil if no input is currently available, or :eof if the stream is at the end-of-file. The default method calls ext:stream-read-byte which should be fine for a file stream, however unbuffered binary interactive and socket streams will probably want to defined a method for this function.

ext:stream-unread-byte stream byte[Generic Function]

Undoes the last call to read-byte, as in unread-byte. Returns nil. Every unbuffered binary input stream must define a method for this function.

ext:stream-peek-byte stream[Generic Function]

Used to implement peek-byte; this corresponds to a peek-type of nil. It returns either an integer byte or :eof. The default method calls read-byte and unread-byte.

ext:stream-read-bytes stream buffer start requested waitp[Generic Function]

Attempt to read the requested number of bytes from the stream into the given buffer starting at start, and returning the number of bytes read or -1 if at the eof-of-file. On pipes or similar devices, this function returns as soon as any data is available, even if the amount read is less than requested and the end-of-file has not been reached. When waitp is true, wait for at least one byte before returning, otherwise return zero if no input is available. The stream input buffer is bypassed. Called by read-bytes when the stream input buffer is empty. See the documentation for read-bytes. Every buffered binary input stream must define a method for this function. When less bytes are read than requested, the function ext:stream-waiting-file-descriptor may be called to find the file descriptor and directions blocked and when ready the function may be called again to continue the operation.

ext:stream-write-byte stream byte[Generic Function]

Writes byte to the stream and returns the byte. Every unbuffered binary output stream must define a method for this function.

ext:stream-write-bytes stream bytes start end waitp[Generic Function]

Attempt to write the bytes vector from the start to the end position to the stream, returning the number of bytes written. If waitp is true, then wait until all bytes have been written before returning. When waitp is false, write as many bytes as possible without blocking, returning the number of bytes written which may be zero. The write bypasses the stream output buffer. Called by write-bytes to write the stream buffer. Every buffered binary output stream must define a method for this function. When less bytes are written than requested, the function ext:stream-waiting-file-descriptor may be called to find the file descriptor and directions blocked and when ready the function may be called again to continue the operation.

ext:stream-listen stream[Generic Function]

Returns t if any input is waiting otherwise nil. The stream input buffer is bypassed, so the caller should firstly check the input buffer if desired. Called by listen when the input buffer is empty. The default method tries to read a character or byte without waiting, and then unreads the character or byte.

ext:stream-pathname stream[Generic Function]

Return the pathname associated with the stream. The stream pathname is implemented for streams of class file-stream, and for composite streams that may encapsulate file streams. When appropriate, user defined composite streams should provide a method to return the stream-pathname of the source or target stream. A default method is provided for the class stream that returns nil.

(setf (ext:stream-pathname stream) pathname)[Generic Function]

Set the pathname associated with the stream. The stream pathname is implemented for streams of class file-stream, and for composite streams that may encapsulate file streams. When appropriate, user defined composite streams should provide a method to set the stream-pathname of the source or target stream. A default method is provided for the class stream that does nothing.

ext:stream-timeout stream[Generic Function]

Return the current timeout for the stream in seconds, or nil if there is no timeout. The timeout is implemented for file descriptor based streams which includes network socket streams, and SSL streams, and also methods are provided for composite streams to set the source and target stream timeout. When appropriate, user defined composite streams should provide a method to return the source or target stream timeout. A default method is provided for the class stream that returns nil. See also ext:stream-expiration.

(setf (ext:stream-timeout stream) timeout)[Generic Function]

Set the timeout for the stream. The timeout may be either a real number specifying the timeout in seconds, or nil for no timeout. When a timeout is specified, blocking operations on the stream may timeout and generate an ext:io-timeout error. The stream timeout sets a time limit on individual stream operations and can be useful for placing a time limit on idle streams. A time limit can be placed on a series of stream operations by setting the ext:stream-expiration. The effective timeout is reduced to be no greater than the time to stream expiration if any.

The timeout is implemented for file descriptor based streams which includes network socket streams, and SSL streams, and also methods are provided for composite stream to set the source and target stream timeout. When appropriate, user defined composite streams should provide a method to set the source and target stream timeout. A default method is provided for the class stream that does nothing if the new timeout is nil, otherwise it signals an error.

ext:stream-expiration stream[Generic Function]

Return the current time in seconds to expiration of the stream, or nil if there is no expiration time. The returned time is a real number, or nil. The expiration timer is implemented for file descriptor based streams which includes network socket streams, and SSL streams, and also methods are provided for composite stream to return the source or target stream expiration time. When appropriate, user defined composite streams should provide a method to return the source or target stream expiration time. A default method is provided for the class stream that returns nil. See also ext:stream-timeout.

(setf (ext:stream-expiration stream) time)[Generic Function]

Set the time to expiration of the stream. The time is a real number giving the elapsed real time from the current time after which the stream expires. A time of nil disables stream expiration. When the time is exceeded, blocking operations such as read and write generate a sys:io-timeout error. The stream expiration is useful for placing a bound on the time taken for a series of stream operations, whereas the ext:stream-timeout places a bound on individual operations. The timeout on individual operations is effectively reduced to be no greater than then remaining time to expiration. After the stream has expired the expiration time must be reset to successfully continue operations on the stream.

The stream expiration timer is implemented for file descriptor based streams which includes network socket streams, and SSL streams, and also methods are provided for composite stream to set the source and target stream expiration. When appropriate, user defined composite streams should provide a method to return the source or target stream expiration time. A default method is provided for the class stream that does nothing if the new time is nil, otherwise it signals an error.

ext:stream-file-descriptor stream direction[Generic Function]

Return the underlying file descriptor for the stream in the given direction, or nil if not applicable. If a stream has multiple input or output file descriptors then the descriptor blocking the most recent operation on the stream is returned. Implements the function sys:fd-stream-fd. The file descriptor based streams include local file streams, network socket streams, and SSL streams. Methods are provided for composite streams to return the source or target stream descriptors. When appropriate, user defined composite streams should provide a method to return the source or target stream file descriptor. A default method is provided for the class stream that returns nil.


Non-blocking input and output

Non-blocking input is supported by the function read-char-no-hang which may be used to read a character without blocking, and the extension ext:read-byte-no-hang may be used to read a byte without blocking. A character or byte vector sequence may be read without block by using the optional waitp argument to ext:read-chars and ext:read-bytes.

Non-blocking output is supported by the extensions: ext:write-byte-no-hang, ext:write-char-no-hang, and ext:finish-output-no-hang. A character or byte vector sequense be be written without blocking by using the optional waitp argument to ext:write-chars and ext:write-bytes. Non-blocking output is currently only supported by binary file-descriptor streams and SSL streams, and other stream may block.

When a non-block stream operation on a file descriptor based stream does not complete, the function ext:stream-waiting-file-descriptor may be called to find the file descriptor that blocked and the directions that the stream is waiting on to before it can make further progress. The file descriptor and direction can be passed to sys:wait-until-fd-usable before retrying the last operation. This facility allows a single thread to service many streams and allows bidirectional stream tunnels to be implemented, and it is used by the Scieneer Common Lisp HTTP server to read and write many concurrent requests in a single thread. Note that this is an experimental facility - please let Scieneer know if you are making use of it.

ext:write-byte-no-hang integer stream[Function]

Outputs the integer to the binary stream without waiting, returning the integer upon success or nil if the write would block.

ext:stream-write-byte-no-hang stream byte[Generic Function]

Writes byte to the stream if the write will not block and returns the byte. Otherwise return nil without waiting. The default method calls ext:stream-write-byte which should be fine for a file stream, however unbuffered binary output socket streams will probably want to defined a method for this function. When a byte is not written the function ext:stream-waiting-file-descriptor may be called to find the file descriptor and directions blocked and when ready the function may be called again to continue the operation.

ext:write-char-no-hang character &optional stream[Function]

Outputs the character to the stream without waiting, returning the character upon success or nil if the write would block.

ext:stream-write-char-no-hang stream character[Generic Function]

Writes character to the stream if the write will not block and returns the character. Otherwise return nil without waiting. The default method calls ext:stream-write-char which should be fine for a file stream, however unbuffered character output socket streams will probably want to defined a method for this function. When a character is not written the function ext:stream-waiting-file-descriptor may be called to find the file descriptor and directions blocked and when ready the function may be called again to continue the operation.

ext:finish-output-no-hang &optional stream[Function]

Attempt to ensure that all output sent to the stream has reached its destination returning t upon success or nil if the operation would block. Implemented by the generic function ext:stream-finish-output-no-hang. When the write is not completed the function ext:stream-waiting-file-descriptor may be called to find the file descriptor and directions blocked and when ready the function ext:finish-output-no-hang may be called again to continue the operation. See also: ext:flush-output.

ext:stream-finish-output-no-hang stream[Generic Function]

Attempt to finish sending any buffered output returning t upon success or nil if the operation would block. Output buffered in the stream output buffer is not sent - the caller should firstly flush the output buffer if desired. Called by ext:finish-output-no-hang after attempting to flush the stream output buffer. The default method calls ext:stream-finish-output which should be fine for a file stream, however socket output streams will probably want to defined a method for this function. When the write is not completed the function ext:stream-waiting-file-descriptor may be called to find the file descriptor and directions blocked and when ready the function ext:stream-finish-output-no-hang may be called again to continue the operation.

ext:stream-waiting-file-descriptor stream direction[Generic Function]

Return the file descriptor and directions that the stream is waiting on to make further progress in the given direction at the operating system interface. The direction may be either :input, :output, or :io. Two values are returned: the file descriptor number and the wait directions which will be either: nil, :input, :output, or :io.

This function is intended to be called after a non-blocking operation on the stream fails to complete because a read or write to an operating system file descriptor would block. For example, after a call to the functions ext:read-byte-no-hang, ext:read-bytes, ext:write-byte-no-hang, ext:write-bytes, or ext:finish-output-no-hang etc.

The direction should match the direction of the previous operation on the stream, otherwise the result may not be relevant. The :io direction may be implemented to support asynchronous two way communication on a stream and is expected to follow a write and read attempt on the stream. When this function is called with direction :io, the prior read should generally follow a prior write because it is possible that a stream write operation will read further input for the stream.

The returned values are suitable for passing as arguments to the function sys:wait-until-fd-usable to wait before retrying the prior operation.

This function only indicates when progress can be made on the stream at the operating system interface and it is not intended to be used to check if a read or write can complete without blocking and it does not check the lisp stream read or write buffers.

The returned wait direction is not necessarily the same as the requested direction because the stream may have a hidden communication protocol that may require writing to the file descriptor before more user data can be read or reading from the file descriptor before more user data can be written. For example an SSL socket stream.

The returned file descriptor is not necessarily the same for input and output. For example, a two-way-stream may have different file descriptors for input and output. Further, the returned file descriptor may change after further progress is made on the stream.

Note that this is an experimental interface and is not yet thoroughly implemented. Binary file descriptor streams are currently supported, including socket streams, and SSL streams. If you are making use of this interface then please contact Scieneer for more information.


Stream buffers

All streams must have input and out buffers. Where a stream operates in unbuffered mode, or when a stream does not implement input or output, a zero length buffer must be allocated. When a stream is closed, zero length buffers may be installed so that the buffer vectors may be returned to a resource pool. This policy ensuring that these buffers are always allocated and avoids extra checks in fast inline stream functions. The defaults slot definitions for ext:character-stream and ext:binary-stream based streams initialize the buffer slots to zero length vectors.

When writing stream methods it is often necessary to access the stream buffers. The input buffer vector, and the head and tail index are initialized via the initargs :in-buffer, :in-head, and :in-tail respectively. The accessors for these input buffer slots are ext:stream-in-buffer, ext:stream-in-head, and ext:stream-in-tail. The output buffer, and the tail index, are initialized via the initargs :out-buffer, and :out-tail respectively. The accessors for the output buffer slots are ext:stream-out-buffer, and ext:stream-out-tail.

All stream also define a stream open flags which may be initialized using the initarg :open-p and accessed using ext:stream-open-p. When a stream is closed the input and output buffers must not be accessed and may be set to nil.

ext:stream-in-buffer object[Function]

Accessor for the stream input buffer which must be a simple-string for character streams and a '(simple-array (unsigned-byte 8) (*)) for binary streams. All streams must have a stream input buffer. If the stream does not handle input or if the stream operates in unbuffered mode then the buffer length must be zero. The default slot definitions for ext:character-stream and ext:binary-stream initialize the input buffer to an appropriate zero length buffer if not already initialized. When the stream is closed for input it must still have an appropriate input buffer but the vector elements are not accessed so a small shared vector may be used - a stream operating in buffered input mode must have an input buffer of at least one element even when closed because the buffer length is used to dispatch to either unbuffered or buffered stream methods.

ext:stream-in-head object[Function]

Accessor for the head index into the stream input buffer. When the head index is less than the tail index it points to the next element to be read from the stream input buffer.

ext:stream-in-tail object[Function]

ext:stream-out-buffer object[Function]

Accessor for the stream output buffer which must be a simple-string for character streams and a '(simple-array (unsigned-byte 8) (*)) for binary streams. All streams must have a stream output buffer. If the stream does not handle output or if the stream operates in unbuffered mode than the buffer length must be zero. The default slot definitions for ext:character-stream and ext:binary-stream initialize the output buffer to an appropriate zero length buffer if not already initialized. When the stream is closed for output it must still have an appropriate output buffer but the vector elements are not accessed so a small shared vector may be used - a stream operating in buffered output mode must have an output buffer of at least one element even when closed because the buffer length is used to dispatch to either unbuffered or buffered stream methods.

ext:stream-out-tail object[Function]

Accessor for the tail index into the stream output buffer. While the tail is less than the output buffer length, elements may be written directly to the output buffer increasing the tail index. When the buffer is full, the buffer output method ext:stream-write-chars or ext:stream-write-bytes will be called. If the output buffer length is zero then the unbuffered stream ext:stream-write-byte or ext:stream-write-char method will be call to output an element.

ext:stream-open-p object[Function]

Accessor for the stream open-p flag.


External formats

The external-format may be a keyword representing a character set to use when opening character streams, for example :iso-8859-1, or :utf-8.

The external-format used when opening local files, or passed to sys:make-fd-stream, may also be a list specifying a chain of format conversions. Leading character to binary format conversions are ignored when opening a binary stream allowing the same external-format list to be used for opening both binary and character streams.

Conversion between a character and binary stream can be specified using the external format element '(:character :charset <character set>). If the charset is not supplied then it defaults to the value :default for which the value of ext:*default-external-format* is used. See ext:make-character-conversion-stream for more information.

End-of-line character conversion can be specified by the external format element '(:eol :mode <mode>), where the mode is :unix, :dos, :mac, or :any-input, and the default is :unix. See ext:make-eol-conversion-stream for more information.

Character conversion for an XML input file can be specified using the external format element :xml. XML files may also need end-of-line translation and the following external format list may be appropriate: '((:eol :mode :any-input) :xml). See ext:make-xml-character-conversion-stream for more information.

Gzip compression or gunzip expansion streams may be opened simply by using the :gzip format element with the keyword arguments :level, :time, :file-name, :window-bits, :mem-level, and :allow-concatenated-p. See ext:make-gzip-output-stream and ext:make-gunzip-input-stream for more information about these arguments.

The cryptographic library adds support for the :encrypt format element with the cipher and password as keyword arguments.

For example, to open a character file using DOS end-of-line conversion with characters encoded using UTF-8 and with gzip compression and encryption the following external-format may be passed to open: '((:eol :mode :dos) (:character :charset :utf-8) (:gzip :level 9) (:encrypt :cipher :aes256-plain :password "password"))

An experimental interface is provided to allow new formats to be added. This interface may be revised in future so please contact Scieneer if you make use of it.

ext:make-external-format-stream style args stream &key input output element-type in-buffer-size out-buffer-size buffering auto-close pathname[Generic Function]

Make a new stream for the given external-format style with argument list args. The returned stream encapsulates the given stream. When the pathname is supplied a file-stream should be created if applicable.

ext:external-format-element-types format[Generic Function]

Return the element-types applicable to the format. The element types should be either character or '(unsigned-byte 8). Two values must be returned, the first is the element-type after encapsulation, and the second value is the element-type before encapsulation.


XML file character encoding

The character encoding of an XML file is specified by the file byte order mark and the file document declaration. The utility function ext:make-xml-character-conversion-stream reads the source binary stream to determines the character encoding and then returns a suitable character conversion stream. End-of-line termination conversion may be applied to be resulting stream, see: ext:make-eol-conversion-stream.

ext:make-xml-character-conversion-stream stream &key input output close-stream-p buffering in-buffer-size out-buffer-size finalizep pathname[Function]

Create a character conversion stream that reads or writes the given binary stream with the character encoding determined from the XML file byte order mark and document declaration. The stream input buffer is used to read ahead and must be large enough to read the encoding declaration. If there is a byte order mark then it is consumed, but no content is consumed. The declared encoding string, if any, is returned as a second value and may differ from the stream encoding if it was not compatible with the initial encoding used to read the declaration. End-of-line termination conversion may be applied to be resulting stream, see: ext:make-eol-conversion-stream.

:input
If true, allow input operations. This is the default if neither :input or :output is specified.
:output
If true, allow output operations.
:close-stream-p
If true then the target stream is closed when this stream is closed.
:buffering
Indicates the kind of output buffer flushing to use which may be one of: :none, :line, or the default :full.
:pathname
When true, the returned stream will be a subclass of the class file-stream. This should generally be true if the stream is a file-stream.

Reading ahead multiple elements

The utility functions ext:buffer-bytes and ext:buffer-chars allow reading ahead without consuming stream input. The input is stored in the stream input buffer so is limited in size to this buffer.

ext:buffer-chars stream required &optional waitp[Function]

Attempt to read the required number of characters into the stream input buffer. When waitp is true, wait for at least the required number of characters before returning, and return the number of characters available in the buffer upon success or return -1 if the end of the file is encountered before reading the required number of characters. When waitp is false, return zero if no further input is currently available in which case the function ext:stream-waiting-file-descriptor may be called to find the file descriptor and directions blocked and when ready the function may be called again to continue the operation. The number of required characters is limited to the buffer length so a short count will be returned once the buffer is full. Note that the ext:stream-in-head may be modified, so the caller must reload it if necessary.

ext:buffer-bytes stream required &optional waitp[Function]

Attempt to read the required number of byte into the stream input buffer. When waitp is true, wait for at least the required number of bytes before returning, and return the number of bytes available in the buffer upon success or return -1 if the end of the file is encountered before reading the required number of bytes. When waitp is false, return zero if no further input is currently available in which case the function ext:stream-waiting-file-descriptor may be called to find the file descriptor and directions blocked and when ready the function may be called again to continue the operation. The number of required bytes is limited to the buffer length so a short count will be returned once the buffer is full. Note that the ext:stream-in-head may be modified, so the caller must reload it if necessary.


File positions

The file position is measured in units of bytes for binary streams and in units of characters for character streams, and is defined as the number of bytes or characters returned by read-byte or read-char when reading from the start of the file to the current position. The position may not be the same as the position in the underlying file due to non-unity encoding. For example: character encoding streams, compressed binary streams, or end-of-line conversion streams.

For each byte or character written the position change will typically be one but in general may be more or less than one. For example: an indenting output stream can insert characters at the start of each line resulting in a multi-character change in position when a linefeed is written; or a character conversion output stream may ignore and drop characters that can not be encoded.

See: file-position, file-length, and ext:stream-file-position.