Scieneer Common Lisp 1.3.9 online documentation

HTTP Client Support

Introduction

The HTTP library adds client support and extends the Common Lisp file system to handle http and https scheme pathnames. There are three levels at which the client may be used.

At the lowest level, the client accepts request objects to be send and returns response objects, see http:make-https-client-request and http:process-client-request. The client can manage a pool of persistent connections which may be reused when sending a request. The client can also manage the total number of open connections and the number of connections to each remote host port. When a new connection is to be processed and the limit of open connections has been reached, the client will try to close the oldest unused connection, otherwise it will wait for another request to complete.

The function http:process-uri-request processes a request for a particular URI, building a suitable request for the client, taking into account optional proxy and SSL parameters, and can following redirect responses and retry a request upon a communication failure.

The client also extends the Common Lisp file system to allow http and https scheme pathnames to be opened for input or output. The opening of an input stream is implemented by a GET method, and the closing of an output stream is implemented by a PUT method. The probe-file and file-write-date functions are implemented by the HEAD method, and the delete-file function is implemented by the DELETE method. The requests are sent to the http:*default-client*, and use it's default parameters such as the proxy and SSL certificate for this client.

Due to the sharing of pooled connections, and the connection limits managed by each client, it may be useful to used multiple clients in an application. Connections are not shared between the clients and the connection limits are managed separately by each client. The client also holds default parameters for the proxy and SSL parameters are it may be useful to use multiple clients with different defaults.

The client request and response objects

For basic HTTP requests, the request and response objects are the same as for the server. For a HTTPS client request, additional information may be required and this is included in the class http:https-client-request. The result of a HTTPS request is a http:https-client-response object.

http:make-request &key client-host client-port server-host server-port method uri protocol protocol-major protocol-minor headers body[Function]

Make a HTTP request. HTTP requests are created by the server for each received request, and may be created by a client to send to a server. The server-host and server-port must be supplied when making a client request. The server-host should be the server IP address number or domain name and the server-port is the server port number. The method is the HTTP method string, for example "GET", or "POST", and must be supplied. The uri is the URI string. The headers is an alist of headers and an associated list of values. The body, if any, may be a byte vector or a string - a string will be converted into bytes before sending the request. The request object is immutable, and should not be modified. A client protocol string to send with the request may be supplied to override the default in which case the protocol-major and protocol-minor numbers must also be supplied. The client-host and client-port are typically not used when making a client request but are filled in by the server when receiving a request from a client.

For a proxy request, the server-host and server-port should specify the proxy server host and port to which the client connects. The proxy will likely expect the uri to be absolute in contrast to relative for a direct request.

http:make-https-client-request &key client-host client-port server-host server-port method uri protocol protocol-major protocol-minor headers body certificate key verify-mode verify-depth ciphers ssl-options ssl-context tunnel-p[Function]

Make a HTTPS client request. The arguments are the same as for the function http:make-request with the addition of the SSL parameters: certificate, key, verify-mode, verify-depth, ciphers, ssl-options, and ssl-context For more information see ssl:make-ssl-client-stream.

When tunnel-p is true, a tunneling TCP connection will be requested from the proxy server over which the secure request may be sent. The server-host and server-port specify the proxy to use. The target server host and port will be obtained from the request "Host" header and must be supplied in the headers.

http:https-client-request[Class]

Class precedence list:

http:https-client-request, http:request, standard-object, ext:instance, t.

Base class for a HTTPS client request.

http:request-certificate object[Generic Function]

Reader for a HTTPS client request SSL certificate file. See ssl:make-ssl-client-stream.

http:request-key object[Generic Function]

Reader for a HTTPS client request SSL key file. See ssl:make-ssl-client-stream.

http:request-verify-mode object[Generic Function]

Reader for a HTTPS client request SSL verify mode, either :none, :request, or :require. See ssl:make-ssl-client-stream.

http:request-verify-depth object[Generic Function]

Reader for a HTTPS client request SSL verify depth. See ssl:make-ssl-client-stream.

http:request-ciphers object[Generic Function]

Reader for a HTTPS client request SSL ciphers, a colon separated list of the restricted set of ciphers to use. See ssl:make-ssl-client-stream.

http:request-ssl-options object[Generic Function]

Reader for a HTTPS client request SSL options. See ssl:make-ssl-client-stream.

http:https-client-response[Class]

Class precedence list:

http:https-client-response, http:response, standard-object, ext:instance, t.

Base class for a HTTPS client response.

http:response-cipher-name object[Generic Function]

Reader for a HTTPS client response SSL cipher name. This is a string, or nil if not applicable. See ssl:ssl-stream-cipher.

http:response-cipher-version object[Generic Function]

Reader for a HTTPS client response SSL cipher version. This is a string, or nil if not applicable. See ssl:ssl-stream-cipher.

http:response-verify-result object[Generic Function]

Reader for a HTTPS client response SSL cipher result which will be zero upon success or if a peer certificate was not presented. See ssl:ssl-stream-verify-result.

http:response-x509-name object[Generic Function]

Reader for a HTTPS client response SSL peer X509 distinguished name. This is an alist of attribute and value strings. See ssl:ssl-stream-peer-name


Client interface

http:client[Class]

Class precedence list:

http:client, standard-object, ext:instance, t.

http:make-client &key stream-timeout stream-expiration idle-timeout protocol protocol-major protocol-minor pool-enabled-p maximum-server-connections maximum-connections maximum-connection-requests retries retry-interval accept-gzip-encoding-p proxy-host proxy-port non-proxy-hosts certificate key verify-mode verify-depth ciphers ssl-options ssl-context[Function]

Make and return a HTTP client object. The HTTP client object holds parameters used when processing client requests, and may contain a pool of connections. A default client object is available in http:*default-client*.

The stream-timeout, if not nil, is the time in seconds to wait upon individual stream operations before giving up and can be used to timeout if the server stops communicating. The default stream-time is 300 seconds.

The stream-expiration, if not nil, is the total real time in seconds from the start of each request until the stream expires and can be used to timeout if the server communication is too slow even though data may be trickling through. The default stream-expiration is nil. The stream expiration is reset at the start of each request and cleared when idle.

The idle-timeout, if not nil, is the number of seconds to keep idle connections alive in the connection pool before closing them. The default idle-timeout is 300 seconds.

The client protocol may be the default of HTTP/1.1 or alternatively HTTP/1.0. The protocol string and the protocol-major and protocol-minor numbers must be specified if not using the default.

When pool-enabled-p is true, client requests will reuse connections from a connection pool which can improve performance. The connection pool is enabled by default.

The client keeps track of the number of open connections and is able to limit the number of connections to each host port, and also to limit the total number of connections for the client. Limiting the number of connections to a host is often a nice behavior for a client, helping to avoid overloading the server.

A limit may also be placed on the number of requests issued over a persistent pooled connection by specifying the maximum-connection-requests. A persistent connection will be closed when this maximum is reached and a new connection will be opened if necessary. This could be a nice behavior for a client, periodically freeing the server resources for use by other clients. The default value is 200 requests.

The client contains some other default parameters used by the function http:process-uri-request when preparing a request, such as the proxy and SSL parameters for https requests. These parameters may be overridden by arguments supplied to http:process-uri-request. A range of standard Common Lisp file system operations such as open and probe-file can operate on URIs with a http or https scheme, and these operations use the default client parameters in the http:*default-client* object. Applications may change the parameters used by such operations by changing the default parameters in the http:*default-client* or by creating an alternative client object and binding it to http:*default-client* for the extent of the operation.

When accept-gzip-encoding-p is true, http:process-uri-request will include an Accept-Encoding header specifying gzip compression with requests.

A default proxy server may be specified, and will be used by http:process-uri-request when preparing requests. The proxy-host is the hostname string and the proxy-port is the number of the proxy port which defaults to port 80. Requests to hostnames matching a hostname or partial domain name from the non-proxy-hosts list will not use the proxy, see http:client-non-proxy-hosts.

A default number of retries and a retry-interval may be specified for the client, along with the SSL connection parameters: certificate, key, verify-mode, verify-depth, ciphers, ssl-options, ssl-context. These are all used by http:process-uri-request as defaults when preparing a request.

http:*default-client*[Variable]

A default HTTP client object.

http:process-client-request request &key client retries retry-interval new-connection-p[Function]

Process a client request, returning a response object upon success, otherwise returning nil and an possible error condition. The HTTP client defaults to the value of http:*default-client*.

The maximum number of automatic retries defaults to three, and is disabled when zero. When positive, requests are automatically retried if a response is not received from the server because this likely indicates that the server closed a persistent connection before receiving the request. Requests using the GET, HEAD, and PUT, methods are automatically retried if a stream error occurs while receiving the response.

When new-connection-p is true, a new connection is always opened for the request, even if the client has a matching pooled connections. This option may be particularly useful for non-idempotent methods, such as a POST method, to reduce the risk of hitting an asynchronous close from the server.

http:process-uri-request uri &key method headers body client external-format content-type accept-gzip-encoding-p certificate key verify-mode verify-depth ciphers ssl-options ssl-context proxy-host proxy-port follow-redirects errorp retries retry-interval[Function]

Process a http based URI request, returning a http response object and the final URI. If links are followed then the final URI may differ from the requested uri.

The method is the http request method, and may be a string or a keyword. The default is the http "GET" method.

The headers is an alist of http headers to send with the request. Some of the arguments noted below setup the headers for most common cases.

A body may be sent with the request, as is typically necessary when sending a http POST method request. The body should be either a (unsigned-byte 8) vector, or a string. A string will be converted into bytes before sending the request, using the content-type character set otherwise trying the :iso-8859-1 or :utf-8 character sets.

The request will use connections managed by the given client which defaults to the http:*default-client*, see http:make-client. The client provides default values for many of the arguments, such as the proxy and the SSL parameters.

The external-format is used to setup a Accept-Charset header which requests that the response use this encoding although the server may choose to return a response using a different encoding.

The content-type is used to setup a Content-Type header. A content-type is typically necessary when a http body is sent with the request. If the body is a string then the content type will default to text/plain.

The client can accept and decode gzip encoded responses. When the client accept-gzip-encoding-p flag is true, an Accept-Encoding header is sent to indicate to the server that it may gzip encode the response body. If this argument is not supplied then the client default is used.

For a https URI the arguments certificate, key, verify-mode, verify-depth, ciphers, and ssl-options are used when opening the SSL connection. For more information see ssl:make-ssl-client-stream. If these arguments are supplied they override the default values for the client.

The client may use a http proxy. The default proxy parameters are obtained from the client object, see http:make-client, but may be overridden by the arguments proxy-host, and proxy-port when supplied.

If follow-redirects is true then responses redirecting to other locations will be followed, otherwise the redirect response is returned.

The request may be automatically retried upon a connection error, or upon a redirection response. The maximum number of retries is controlled by the retries argument. If a response code indicating an error is received the then it is returned immediately when errorp is false otherwise an error is signaled.

For alternate request preparation or response handling the more primitive function http:process-client-request may be appropriate.

http:client-pool-enabled-p client[Function]

Return true if the connection pool is enabled. When the connection pool is enabled, connections to the same host and port may be reused between separate client requests.

http:client-maximum-connections client[Function]

Return the maximum number of connections allowed for this client, or nil if there is no limit. When the limit is reached a client request will wait for a prior request to complete before proceeding. The client keeps track of the total number of active connections to enforce this limit. This limit is effective whether the connection pool is enabled or disabled.

http:client-maximum-server-connections client[Function]

Return the maximum number of connections allowed to a remove server port, or nil if there is no limit. When the limit is reached a client request will wait for a prior request to complete before proceeding. The client keeps track of the number of open connections to each server port to enforce this limit. This limit is effective whether the connection pool is enabled or disabled. For a requests via a proxy port, this limit applies to the maximum number of connections to the proxy port.

http:client-maximum-connection-requests client[Function]

Return the maximum number of requests per connection after which the connection will be closed.

http:client-protocol object[Generic Function]

Accessor for the HTTP client protocol string.

http:client-protocol-major object[Generic Function]

Accessor for the HTTP client protocol major number.

http:client-protocol-minor object[Generic Function]

Accessor for the HTTP client protocol minor number.

http:client-stream-timeout object[Generic Function]

Accessor for the HTTP client stream timeout in seconds, or nil if there is no timeout. This is the time to wait upon individual stream read and write operations before giving up. See ext:stream-timeout and also http:client-stream-expiration.

http:client-stream-expiration object[Generic Function]

Accessor for the HTTP client stream expiration time in seconds, or nil if it does not expiration. This is the real time, relative to be start of each request, after which the client stream will expire. See ext:stream-expiration and also http:client-stream-timeout.

http:client-idle-timeout object[Generic Function]

Accessor for the HTTP client idle connection timeout, or nil if there is no timeout. This is the number of second to keep idle connections alive in the connection pool before closing them.

http:client-proxy-host object[Generic Function]

Accessor for the client proxy host string. URI requests using this client will be sent to the specified proxy host unless the request hostname matches one of the non-proxy hostnames or patial domain names, see http:client-non-proxy-hosts.

http:client-proxy-port object[Generic Function]

Accessor for the client proxy host port number. The default port is 80.

http:client-non-proxy-hosts object[Generic Function]

Accessor for the client non-proxy hosts list. If a request hostname matches any of the hostname or partial domain name strings in this non-proxy hosts list then the proxy will not be used for the request. Each element of the list may be either a hostname string, or a partial domain name string preceeded by a period. For example ".scieneer.com" matches all hostnames with this suffix, whereas "www.scieneer.com" matches only a matching hostname. The comparison is case insensitive, and a trailing period is ignored.

http:client-retries object[Generic Function]

Accessor for the client retries parameter. URI requested may be retried upon a connection error, or redirection, up to this limit.

http:client-retry-interval object[Generic Function]

Accessor for the client retry backoff interval in seconds. When a URI request is retried, the client may firstly wait which may allow time for the issue causing the error to clear.

http:client-accept-gzip-encoding-p object[Generic Function]

Accessor for the client flag used to enable the sending of an Accept-Encoding gzip header.

http:client-certificate object[Generic Function]

Accessor for the client SSL certificate which is used for https requests, see ssl:make-ssl-client-stream.

http:client-key object[Generic Function]

Accessor for the client SSL certificate key which is used for https requests, see ssl:make-ssl-client-stream.

http:client-verify-mode object[Generic Function]

Accessor for the client SSL verify mode which is used for https requests, see ssl:make-ssl-client-stream.

http:client-verify-depth object[Generic Function]

Accessor for the client SSL verify depth which is used for https requests, see ssl:make-ssl-client-stream.

http:client-ciphers object[Generic Function]

Accessor for the client SSL ciphers which are used for https requests, see ssl:make-ssl-client-stream.

http:client-ssl-options object[Generic Function]

Accessor for the client SSL options which are used for https requests, see ssl:make-ssl-client-stream.


Supporting code

http:response-body-string response &key external-format errorp[Function]

Return the response body content as a string if possible, decoding the transfer encoding and content encodings and converting the body into a string. Only gzip transfer and content encoding are currently supported. The request headers may need to be updated to reflect the decoding, and the modified headers are returned as the second argument. If the content type is not text then the external-format argument must be supplied.

If an unrecognized encoding or character set is encountered, or an error occurs decoding the content or translating the characters then if errorp is false then the original body and header are returned otherwise an error is signaled.