Scieneer Common Lisp 1.3.9 online documentation

HTTP Server Support

Introduction

The Scieneer Common Lisp HTTP library adds support for applications needing to use the Hypertext Transfer Protocol (HTTP), such as HTTP servers or client applications. The design has the potential to support a large number of connections and the current implementation supports many thousands of in-flight and persistent and connections.

The server manages the connections, reads requests, and sends the responses. When a new request is received and read an application service function is called with a request object and should return a response object which the server will send. The response body may be a string, a byte vector, a source stream, or a list of vector chunks and source streams. The body may also contain a file descriptor to splice with the request stream to implement the HTTP CONNECT method. The server makes use of a light weight state machine to manage the connections allowing a large number of connections to be managed.

The application service function is called from a pool of dedicated threads. The service function does not read or write to the connection stream so it does not risk blocking while waiting for input from the connection or blocking while waiting to transmit the response and this allows it to quickly execute to completion. This design allows the pool of threads generating the responses to be smaller than the number of connections.

The application can optionally be passed the partially read request after the headers have been read and may choose to either return and continue with the request or return an immediate response. An optional application log function may be called after sending the response, and is passed the request and response objects.

The design contrasts other popular Common Lisp HTTP servers such as CL-HTTP and Portable AllegroServe which dedicate a thread to the management and servicing of each connection. Since threads are much heavier than connections, the number of connections that these servers can manage becomes limited by the number of threads. However these other popular servers are more flexible because the connection stream is passed to the application allowing the application to take complete control. The Scieneer Common Lisp HTTP server design trades off some flexibility to support a much larger number of connections and to use thread resources more efficiently.

Generating the response content into a vector in memory before sending the response has a number of advantages. Responses that access a database can generate content while within a database transaction. Transactions will typically restart upon a synchronization error, and if the connection is being written to an in-memory vector then it can be safely discarded, whereas if the content were being written to the connection stream then the response would be corrupted. Further, error handlers can cleanly send a response knowing that a stream has not been partially written. A high performance server will often pre-load static content into memory in which case the content will be on hand to pass to the server in the response object. Having the response content in memory also simplifies content compression, which can be performed automatically by the Scieneer Common Lisp HTTP server, and also simplifies content caching. Other potential applications are speculative computation of the response, dispatch to a cluster, and redundant computation for error detection and correction.

A drawback of generating content into a vector in memory is the potential for increased latency because of the need to generate the complete response before the transmission of the response commences.

For very large content, that is not practical to load into memory, or for real time streaming content, the response body may also include sources from binary streams. The server also supports rate limiting when sending the response which can useful for large content.

The library provides automatic conversion of the response body from a string into bytes, and provides a function for decoding a request body into a string. By using this support an application can avoid dealing with implementation dependent details of character conversion. The Scieneer Common Lisp supports Unicode characters as does the server.

The library does not attempt to define the higher level layers of a practical web server. It is recognized that many developers have their own framework. A compatibility layer for the Portable AllegroServe web server is provided in the freeware collection that is built on the Scieneer Common Lisp HTTP library and provides an example of higher level support.


The request and response objects

http:request[Class]

Class precedence list:

http:request, standard-object, ext:instance, t.

Base class for a HTTP request.

http:request-client-host object[Generic Function]

Reader for the HTTP request client host IP number of the client that created the request.

http:request-client-port object[Generic Function]

Reader for the HTTP request client port number which is the TCP port number of the client that created the request.

http:request-server-host object[Generic Function]

Reader for the HTTP request server host IP number of the server that received the request.

http:request-server-port object[Generic Function]

Reader for the HTTP request server port number which is the TCP port number of the server that received the request.

http:request-method object[Generic Function]

Reader for the HTTP request method. This is a string and typically one of the standard HTTP methods such as "GET", "POST", etc. If the request method was not read then the method will be nil.

http:request-uri object[Generic Function]

Reader for the HTTP request URI string. If the request URI was not read then the URI will be nil.

http:request-protocol object[Generic Function]

Reader for the HTTP request protocol string. If the request protocol was not sent, as for a HTTP/0.9 request, or if it was not read then the protocol will be nil.

http:request-protocol-major object[Generic Function]

Reader for the HTTP request protocol major number if any.

http:request-protocol-minor object[Generic Function]

Reader for the HTTP request protocol minor number if any.

http:request-stream-timeout object[Generic Function]

The stream timeout in effect while reading the request. The timeout may be modified by the headers service function before reading the body, and defaults to the http:server-stream-timeout.

http:request-stream-expiration object[Generic Function]

The stream expiration in effect while reading the request. The expiration time may be modified by the headers service function before reading the body, and defaults to the http:server-stream-expiration.

http:request-maximum-body-size object[Generic Function]

The maximum allowable size of the request body. The maximum size may be modified by the headers service function before reading the body, and defaults to the http:server-maximum-body-size.

http:request-headers object[Generic Function]

Reader for the HTTP request headers alist. The header fields may span multiple lines and are unfolded. The header field names are case insensitive and are canonicalized to lower case. Apart from unfolding, no other processing is performed on the raw field values. Fields that occur repeatedly are placed in a list of the header raw values. The headers appear in the order they were read.

http:request-body object[Generic Function]

Reader for the HTTP request body. If the request has a body, for example a POST request, then the body is read. The body is a octect byte vector. If the request has no body then return nil. If the content type is text then the body may be converted into a simple-string by the function http:request-body-string.

http:https-server-request[Class]

Class precedence list:

http:https-server-request, http:request, standard-object, ext:instance, t.

Base class for a HTTPS server request.

http:request-cipher-name object[Generic Function]

Reader for a HTTPS request SSL cipher name. This is a string, or nil if not applicable. See ssl:ssl-stream-cipher.

http:request-cipher-version object[Generic Function]

Reader for a HTTPS request SSL cipher version. This is a string, or nil if not applicable. See ssl:ssl-stream-cipher.

http:request-verify-result object[Generic Function]

Reader for a HTTPS request SSL cipher result which will be zero upon success or if a peer certificate was not presented. See ssl:ssl-stream-verify-result.

http:request-x509-name object[Generic Function]

Reader for a HTTPS request SSL peer X509 distinguished name. This is an alist of attribute and value strings. See ssl:ssl-stream-peer-name

http:response[Class]

Class precedence list:

http:response, standard-object, ext:instance, t.

Base class for a HTTP response.

http:response-protocol object[Generic Function]

Reader for the HTTP response protocol string.

http:response-protocol-major object[Generic Function]

Reader for the HTTP response protocol major number.

http:response-protocol-minor object[Generic Function]

Reader for the HTTP response protocol minor number.

http:response-status-code object[Generic Function]

Reader for the HTTP response status code. This is an integer number.

http:response-reason-phrase object[Generic Function]

Reader for the HTTP response status line reason-phrase string.

http:response-stream-timeout object[Generic Function]

The stream timeout to use when sending the response. The default value is the keyword :default, in which case the http:request-stream-timeout is used when sending the response. When nil the timeout is unlimited.

http:response-stream-expiration object[Generic Function]

The stream expiration time to use when sending the response. The default value is the keyword :default, in which case the request stream expiration time is used when sending the response. When nil the time is unlimited.

http:response-stream-rate object[Generic Function]

The target rate at which the response will be transmitted, in bytes per second, or nil for unlimited. The default value is the keyword :default, in which case the server default stream rate is used.

http:response-headers object[Generic Function]

Reader for the HTTP response headers alist.

http:response-body object[Generic Function]

Reader for the HTTP response body. If the body is nil then a body is not sent with the reply. The response body may be a vector or a list of vector chunks with each chunk being either a vector or a list of the vector and an optional start and end position. The body of a sent response will contain only octet byte vectors, however the server accepts strings for the response body which will be converted into byte vectors when sending the response. The body of a HTTP server response may also include stream sources or a file descriptor to splice with the connection stream, see http:make-response.

http:make-response &key protocol protocol-major protocol-minor status-code reason-phrase headers stream-timeout stream-expiration stream-rate body[Function]

Make a HTTP response. The status-code is the integer HTTP response code, 200 for success etc, and the associated reason-phrase is a human readable string describing the status code. The headers is an alist of headers and an associated list of values.

The server protocol string to send with the response may be supplied to override the default in which case the protocol-major and protocol-minor number must also be supplied.

The stream-timeout and the stream-expiration time to use while sending the response may also be supplied. When nil the time is unlimited. The default is the keyword :default in which case the request defaults will be used when sending the response.

The stream-rate is the target rate in bytes per second at which the response will be transmitted. When nil the target rate is unlimited. The default is the keyword :default in which case the server default rate is used.

The body, if any, may be a byte vector or a string, or a list of vector chunks with each chunk being list of a vector and an optional start and end position. A string vector will be converted into bytes before sending the response. The body may also be the list '(:source <stream>) in which case the content is read from the binary stream, or the body may be a list of such source stream lists and vector chunks. When the body is the list '(:splice <file-descriptor>) the connection stream is spliced with the supplied file descriptor and data is passing through in each direction - this is included to support the HTTP CONNECT method.


Server interface

Multiple HTTP servers may be created and each may have multiple ports upon which they listen for new connections. Each server has an associated management thread that listens for new connections and manages the active connections. When a new request is received, the task of processing the request is queued, using the task queue support. Idle persistent connections are managed, keeping them open as long as requested, and closing them if necessary to ensure that the connection limit is not exceeded. New requests are queued in the order received, sharing the load fairly.

http:server[Class]

Class precedence list:

http:server, standard-object, ext:instance, t.

Base HTTP server class.

http:make-server &key service-function service-headers-function log-function enable-keep-alive-p enable-content-compression-p stream-timeout stream-expiration stream-rate idle-timeout linger-timeout maximum-connections maximum-connection-requests maximum-headers-size maximum-body-size task-queue queue-task protocol protocol-major protocol-minor[Function]

Make and a new HTTP server object. Connection listeners may be added to the the server by http:start-http-listener, and http:start-https-listener. When a request is received it is read and packaged into a http:request object, and the service-function called with the server and request as arguments. A service-function must be defined and must return a http:response object which is also immutable.

An optional service-headers-function may be defined which is called after reading the request headers but before reading the request body. The server object and the partial request object are passed as arguments. This function may modify the request object stream timeout and expiration time, and set the maximum number of bytes permitted when reading the body. This function may also return a response which is sent to the client immediately without reading the body.

After preparing and sending a response, the log-function, if any, is called. See http:server-log-function for the function arguments.

When the enable-keep-alive-p argument is true, the connection is kept open if appropriate after sending a response for up to the idle-timeout. A connection is forced to close once the optional maximum-connection-requests limit is reached. The server will close idle connections prematurely if the server maximum-connections limit is reached to give priority to new connection requests, and this strategy makes it practical to keep idle connections alive for a relatively long period without blocking new requests.

If an individual stream operation is not completed within the stream-timeout then the request will be aborted and the stream closed which can protect against communication with the client stopping. If the time elapsed since the start of a request exceeds the stream-expiration time then the request will also be aborted and the stream closed which can protect against slow communication with the client even if data is trickling through. When nil the time is unlimited. The timeout and expiration time may be set per-request by the service-headers-function.

The stream-rate is the default target rate in bytes per second at which responses are sent. If this is not supplied or nil then the default target rate is unlimited. The rate may be controlled for individual responses by providing a rate for a response, see http:make-response.

The optional maximum-headers-size limits the size in bytes of the request headers, and the optional maximum-body-size limits the size in bytes of the request body. These limits may be specified to manage denial-of-service attacks and broken clients. The maximum request body size may also be set per-request by the service-headers-function.

The response content may be automatically compressed, using gzip compression, before sending if enable-content-compression-p is true, but only text/html and text/plain context may be automatically compressed.

The default server protocol is HTTP/1.1, but this may be overridden by specifying the protocol string, the protocol-major and protocol-minor numbers. The only other meaningful values are HTTP/1.0, 1, and 0 respectively.

The lisp system process may have a soft and hard system limit on the number of open file descriptors and when the server is activated, by adding a listener, the maximum-connections limit is compared with the system limits for the process. The soft system limit is increased if necessary and if the hard limit is reached then a warning is be printed and the server maximum-connections is reduced.

When a task-queue is specified, the listeners for this server may use a pool of threads to process each connection. If the task-queue is nil then the listeners will create a new thread to process each connection. The task queue avoids the overhead of creating a new thread for each connection, and the task queue can keep the connection resource usage in bounds. If a task-queue is not specified then a default queue is created with a maximum of 32 threads and a queue length of 300 connections. The queue-task function will be called with the task-queue and a function as arguments.

A default server object is provided and stored in the http:*default-server* variable, but it is not activated until a listener is started.

http:start-http-listener &key server host port backlog reuse-address[Function]

Start a HTTP listener for the given server. The port is the port to listen for connections, and defaults to port 80. A host may also be defined which is useful if the system has multiple interfaces. The backlog defines the maximum length of the queue of pending connections. When reuse-address is true the socket will succeed in binding to the address except if there is already an active listening socket bound to the address. If the server already has a listener on the port then it is firstly removed.

http:start-https-listener &key server host port backlog reuse-address certificate key verify-mode verify-depth ssl-context[Function]

Start a HTTP listener for the given server, using the Secure Socket Layer (SSL) protocol. The port is the port to listen for connections, and defaults to port 443. A host may also be defined which is useful if the system has multiple interfaces. The backlog defines the maximum length of the queue of pending connections. When reuse-address is true the socket will succeed in binding to the address except if there is already an active listening socket bound to the address. If the server already has a listener on the port then it is firstly shutdown. A pathname to the SSL certificate and key, and the SSL verify-mode and verify-depth, and the ssl-context may be specified - see the SSL documentation for more information, in particular ssl:make-ssl-server-stream.

http:remove-listener &key server host port[Function]

Shutdown the HTTP listener on the given host and port. If the server is using a task queue then pending connection tasks may continue to execute until done, even after the listener has shutdown.

http:shutdown-server &optional server[Function]

Shutdown the server, removing all port listners. If the server is using a task queue then it is not shutdown because it may have been shared with other servers which are still running, so pending tasks may continue to execute until done. For the default task queue, see thread:shutdown-task-queue to shutdown the task queue. The server may be later restarted by starting new listeners.

http:*default-server*[Variable]

A default HTTP server object.

http:*compressible-content-types*[Variable]

List of content types that the server will automatically compress. The default types are: text/html and text/plain.

http:server-stream-timeout object[Generic Function]

The HTTP server stream timeout, in seconds, or nil for unlimited. This timeout can be used to place a time limit on individual read and write operations. See ext:stream-timeout and also http:server-stream-expiration. The stream timeout may be modified per request after reading the headers by the http:server-service-headers-function function, see http:request-stream-timeout.

http:server-stream-expiration object[Generic Function]

The HTTP server stream expiration time, in seconds, or nil for unlimited. The stream expiration timer is reset at the start of each request and cleared when complete, and can be used to place a bounds on the total time a request takes to read and write data to the client. See ext:stream-expiration and also http:server-stream-timeout. The stream expiration time may be modified per request after reading the headers by the function http:server-service-headers-function, see http:request-stream-expiration.

http:server-stream-rate object[Generic Function]

The HTTP server default response target transmission rate, in bytes per second, or nil for unlimited. The rate may be specified per response, see http:make-response.

http:server-idle-timeout object[Generic Function]

The HTTP server idle connection timeout, in seconds. When persistent connections are kept alive, they will be closed if idle for more than this timeout.

http:server-linger-timeout object[Generic Function]

The HTTP server lingering-close timeout, in seconds. The server closes the write half of a connection and lingers around waiting for the client to acknowledge and also close the connection. If the client does not response within the lingering-timeout then the connection is closed by the server.

http:server-maximum-connections server[Function]

The HTTP server maximum number of open connections, which includes both active connections and idle persistent connections. The server manages persistent connections, closing them in least recently used order when necessary to allow new connections to be accepted. If the limit is reached and there are no idle connections remaining to close then new connections are closed immediately without processing.

http:server-enable-keep-alive-p object[Generic Function]

The HTTP server flag to enable connection keep-alive. When true the server will keep a connection alive when possbile otherwise the connection is closed at the end of each request.

http:server-enable-content-compression-p object[Generic Function]

The HTTP server flag to enable content compression. When true the server will GZIP compress the content if the MIME content type is text/plain or text/html.

http:server-service-function object[Generic Function]

The HTTP server service function. A service function must be specified and is called to generate a response to requests. This should be a function or a symbol naming a function. This function is called with the arguments: server, and the request.

http:server-service-headers-function object[Generic Function]

The optional HTTP server request headers service function. This service function is called after reading the request headers but before reading the request body. The function may modify the request parameters affecting the reading of the body: the maximum body size, and the stream timeout and expiration time.

If the function returns a http:response object then this response is sent to the client immediately without firstly reading the response. This can be useful when the client has sent an expect: 100-continue' header, or to close the stream to defend against denial-of-service attacks and broken clients.

If the function returns nil then the server continues to read the request body, firsly sending a continue response if the client sent an expect: 100-continue' header. The request will be passed to the server service function when read.

http:server-log-function object[Generic Function]

The HTTP server log function. The log function is optional and if specified will be called after attempting to send a response. This should be a function, a symbol naming a function, or nil. The function is called with four required arguments: the server, request, the generate response, and the sent response. The function is also passed keyword arguments, and should allow other keys for future compatibility. The keywords arguments are:

:content-bytes-written
The number of bytes of content written. If an error occurs and is detected by the server then this may be less than the response body size.
:response-write-complete-p
A flag which is true if the complete response was written. If an error occurs and is detected by the server then this will be false.
:response-write-milliseconds
The number of milliseconds that elapsed while transmitting the response. This does not include the time to read and generate the response.
:internal-errors
A list of error strings. When the server encounters errors processing a request it places them in this list and these may be useful for debugging and diagnostics.

http:server-protocol object[Generic Function]

The server HTTP protocol string. The default is "HTTP/1.1".

http:server-protocol-major object[Generic Function]

The server HTTP protocol version major number. The default is 1. The major number must match the http:server-protocol string.

http:server-protocol-minor object[Generic Function]

The server HTTP protocol version minor number. The default is 1. The minor number must match the http:server-protocol string.

http:server-maximum-connection-requests object[Generic Function]

The HTTP server maximum number of requests per connection before the server will close the connection. When nil there is no limit.

http:server-maximum-headers-size object[Generic Function]

The HTTP server maximum request headers size, in bytes. If the server reads more bytes while reading a request header then the request is aborted and the connection closed. This provides the server with some protection against errant clients.

http:server-maximum-body-size object[Generic Function]

The HTTP server maximum request body size, in bytes. If the server reads more bytes while reading a request body then the request is aborted and the connection closed. This provides the server with some protection against errant clients. The maximum request body size may be modified per request after reading the headers by the http:server-service-headers-function function.

http:server-task-queue object[Generic Function]

Accessor for the server task queue object if any.

http:server-queue-task object[Generic Function]

Accessor for the server function called to queue a new task.


Supporting code

A small range of supporting functions are made available. These are not essential and a developer may choose to use their own.

http:header-values name headers[Function]

Return the list of header values associated with the given header name in the headers alist. The values are returned in the order that they occur in the request. The header name comparison is case insensitive and if it is a symbol then the symbol-name is compared.

http:parse-comma-separated-header-values values[Function]

Parse the comma separated header value. Empty tokens are allowed and are ignored. The tokens are canonicalised to lower case and returned in a list and in the same order that they appear in the headers. The values for a header may be read via http:header-values. If the header values have parameters then http:parse-comma-separated-header-values-with-params may be useful.

http:parse-comma-separated-header-values-with-params values[Function]

Parse the comma separated header values for which each extended token may have associated parameters. Empty tokens are allowed and are ignored. The tokens are canonicalised to lower case and returned in a list and in the same order that they appear in the headers. The parameters in each extended token are parsed and they are returned as a list of the token followed by an alist of attribute and value pairs.

((token-1 (attrib-1a . value-1a) (attrib-1b . value-1b) ...)
(token-2 (...) (...) (...)) ...)

http:request-header-values name request[Function]

Return the list of header values associated with the given header name in the request. The values are returned in the order that they occur in the request. The header name comparison is case insensitive and if it is a symbol then the symbol-name is compared.

http:request-header-value name request[Function]

Return one header value associated with the given header name in the request. Only the first value associated with the header name is returned. The name should have already been canonicalised to lower case.

http:response-header-values name response[Function]

Return the list of header values associated with the given header name in the response. The values are returned in the order that they occur in the response. The name should have already been canonicalised to lower case.

http:response-header-value name response[Function]

Return one header value associated with the given header name in the response. Only the first value associated with the header name is returned. The name should have already been canonicalised to lower case.

http:request-pathname request[Function]

Return the request URI pathname. If the URI is nil then the root path is returned. If the URI does not define a scheme then it defaults to https for an SSL request otherwise to http. If the URI does not define an authority then the host and port default to the host header values if any. The pathname is cached in the variable *request-pathname* when within the request service function to avoid recalculation.

http:request-query &optional request[Function]

Return the request query alist of name/value pairs. There may be multiple query values for a query name. The query pairs are obtained from the URI query string and from a POST method request body - the URI query string values appear first in the result. Values are only read from the POST method body when the content type is application/x-www-form-urlencoded.

The query list is cached in the variable *request-query* when within the request service function to avoid recalculation. See also http:query-value and http:query-values.

http:request-query-values name &optional request[Function]

Return the list of query values associated with the given name in the request. The name should have already been canonicalised to lower case. See also http:request-query.

http:request-query-value name &optional request[Function]

Return the first query value associated with the given name in the request. The name should have already been canonicalised to lower case. See also http:request-query.

http:request-body-string request &key external-format errorp[Function]

Return the request body content as a string if possible, decoding the transfer encoding and content encodings and converting the body into a string. Only gzip transfer and content encoding are currently supported. The request headers may need to be updated to reflect the decoding, and the modified headers are returned as the second argument. If the content type is not text then the external-format argument must be supplied.

If an unrecognized encoding or character set is encountered, or an error occurs decoding the content or translating the characters then if errorp is false then the original body and header are returned otherwise an error is signaled.