Integrated, high performance, and scalable HTTP support

Core infrastructure for building high performance and scalable HTTP services is integrated into the Scieneer CL implementation. The core HTTP server infrastructure offers support for the Secure Socket Layer via an interface to the OpenSSL library when available, plus gzip content compression, the management of thousands of connections, and multi-threaded request processing, making it ideal for building secure scalable web sites.

HTTP client support is integrated into the Scieneer CL, adding innovative extensions to the Common Lisp language pathname and file system functions that allow the use of industry standard URIs as pathnames and that allow a range of Common Lisp file system operations to operate on HTTP and HTTPS based URIs.

HTTP server

The Scieneer CL HTTP library adds support for applications needing to use the Hypertext Transfer Protocol (HTTP), such as HTTP servers or client applications. The design has the potential to support a large number of connections and the current implementation supports many thousands of in-flight and persistent connections.

The server manages the connections, reads requests, and sends the responses. When a new request is received and read an application service function is called with a request object and should return a response object which the server will send. The response body may be a string, a byte vector, a source stream, or a list of vector chunks and source streams. The body may also contain a file descriptor to splice with the request stream to implement the HTTP CONNECT method. The server makes use of a light weight state machine to manage the connections allowing a large number of connections to be managed.

The application service function is called from a pool of dedicated threads. The service function does not read or write to the connection stream so it does not risk blocking while waiting for input from the connection or blocking while waiting to transmit the response and this allows it to quickly execute to completion. This design allows the pool of threads generating the responses to be smaller than the number of connections.

The application can optionally be passed the partially read request after the headers have been read and may choose to either return and continue with the request or return an immediate response. An optional application log function may be called after sending the response, and is passed the request and response objects.

The design contrasts with other popular Common Lisp HTTP servers such as CL-HTTP and Portable AllegroServe which dedicate a thread to the management and servicing of each connection. Since threads are much heavier than connections, the number of connections that these servers can manage becomes limited by the number of threads. However these other popular servers are more flexible because the connection stream is passed to the application allowing the application to take complete control. The Scieneer CL HTTP server design trades off some flexibility to support a much larger number of connections and to use thread resources more efficiently.

Generating the response content into a vector in memory before sending the response has a number of advantages. Responses that access a database can generate content while within a database transaction. Transactions will typically restart upon a synchronization error, and if the connection is being written to an in-memory vector then it can be safely discarded, whereas if the content were being written to the connection stream then the response would be corrupted. Further, error handlers can cleanly send a response knowing that a stream has not been partially written. A high performance server will often pre-load static content into memory in which case the content will be on hand to pass to the server in the response object. Having the response content in memory also simplifies content compression, which can be performed automatically by the Scieneer CL HTTP server, and also simplifies content caching. Other potential applications are speculative computation of the response, dispatch to a cluster, and redundant computation for error detection and correction.

A drawback of generating content into a vector in memory is the potential for increased latency because of the need to generate the complete response before the transmission of the response commences.

For very large content, that is not practical to load into memory, or for real time streaming content, the response body many also include sources from binary streams. The server also supports rate limiting when sending the response which can useful for large content.

The library provides automatic conversion of the response body from a string into bytes, and provides a function for decoding a request body into a string. By using this support an application can avoid dealing with implementation dependent details of character conversion. The Scieneer CL supports Unicode characters as does the server.

The library does not attempt to define the higher level layers of a practical web server. It is recognized that many developers have their own framework. A compatibility layer for the Portable AllegroServe web server is provided in the freeware collection that is built on the Scieneer CL HTTP library and provides an example of higher level support.

The HTTP server manage the total number of open connections, closing idle connections when necessary to keep the total within a specified limit. The Scieneer CL supports the maximum number of open sockets for the operating environment which for most ports is many thousands. The server queues requests in the order received, to fairly handle the load, and the requests are processed in turn by a pool of threads.

HTTP client

The HTTP library adds client support and extends the Common Lisp file system to handle http and https scheme pathnames. There are three levels at which the client may be used.

At the lowest level, the client accepts request objects to be sent and returns response objects. The client can manage a pool of persistent connections which may be reused when sending a request. The client can also manage the total number of open connections and the number of connections to each remote host port. When a new connection is to be processed and the limit of open connections has been reached, the client will try to close the oldest unused connection, otherwise it will wait for another request to complete.

A higher level function can process a request for a given URI, building a suitable request for the client, taking into account optional proxy and SSL parameters, and can following redirect responses and retry the request upon a communication failure.

The client also extends the Common Lisp file system to allow http and https scheme pathnames to be opened for input or output. The opening of an input stream is implemented by a GET method, and the closing of an output stream is implemented by a PUT method. The 'probe-file and 'file-write-date functions are implemented by the HEAD method, and the 'delete-file function is implemented by the DELETE method. The requests are sent to the http:*default-client*, and use it's default parameters such as the proxy and SSL certificate for this client.

Due to the sharing of pooled connections, and the connection limits managed by each client, it may be useful to used multiple clients in an application. Connections are not shared between the clients and the connection limits are managed separately by each client. The client also holds default parameters for the proxy and SSL parameters are it may be useful to use multiple clients with different defaults.