waterbutler.core.streams package

BaseStream

class waterbutler.core.streams.BaseStream(*args, **kwargs)

A wrapper class around an existing stream that supports teeing to multiple reader and writer objects. Though it inherits from asyncio.StreamReader it does not implement/augment all of its methods. Only read() implements the teeing behavior; readexactly, readline, and readuntil do not.

Classes that inherit from BaseStream must implement a _read() method that reads size bytes from its source and returns it.

size

add_reader(name, reader)

remove_reader(name)

add_writer(name, writer)

remove_writer(name)

feed_eof()

read(size=-1)

Read up to n bytes from the stream.

If n is not provided, or set to -1, read until EOF and return all read bytes. If the EOF was received and the internal buffer is empty, return an empty bytes object.

If n is zero, return empty bytes object immediately.

If n is positive, this function try to read n bytes, and may return less or equal bytes than requested, but at least one byte. If EOF was received before any byte is read, this function returns empty byte object.

Returned value is not limited with limit, configured at stream creation.

If stream was paused, this function will automatically resume it if needed.

ResponseStreamReader

class waterbutler.core.streams.ResponseStreamReader(response, size=None, name=None)

Bases: waterbutler.core.streams.base.BaseStream

partial

content_type

content_range

name

size

add_reader(name, reader)

add_writer(name, writer)

at_eof()

Return True if the buffer is empty and ‘feed_eof’ was called.

exception()

feed_data(data)

feed_eof()

read(size=-1)

Read up to n bytes from the stream.

If n is not provided, or set to -1, read until EOF and return all read bytes. If the EOF was received and the internal buffer is empty, return an empty bytes object.

If n is zero, return empty bytes object immediately.

If n is positive, this function try to read n bytes, and may return less or equal bytes than requested, but at least one byte. If EOF was received before any byte is read, this function returns empty byte object.

Returned value is not limited with limit, configured at stream creation.

If stream was paused, this function will automatically resume it if needed.

readexactly(n)

Read exactly n bytes.

Raise an IncompleteReadError if EOF is reached before n bytes can be read. The IncompleteReadError.partial attribute of the exception will contain the partial read bytes.

if n is zero, return empty bytes object.

Returned value is not limited with limit, configured at stream creation.

If stream was paused, this function will automatically resume it if needed.

readline()

Read chunk of data from the stream until newline (b’ ‘) is found.

On success, return chunk that ends with newline. If only partial line can be read due to EOF, return incomplete line without terminating newline. When EOF was reached while no bytes read, empty bytes object is returned.

If limit is reached, ValueError will be raised. In that case, if newline was found, complete line including newline will be removed from internal buffer. Else, internal buffer will be cleared. Limit is compared against part of the line without newline.

If stream was paused, this function will automatically resume it if needed.

readuntil(separator=b'\n')

Read data from the stream until separator is found.

On success, the data and separator will be removed from the internal buffer (consumed). Returned data will include the separator at the end.

Configured stream limit is used to check result. Limit sets the maximal length of data that can be returned, not counting the separator.

If an EOF occurs and the complete separator is still not found, an IncompleteReadError exception will be raised, and the internal buffer will be reset. The IncompleteReadError.partial attribute may contain the separator partially.

If the data cannot be read because of over limit, a LimitOverrunError exception will be raised, and the data will be left in the internal buffer, so it can be read again.

remove_reader(name)

remove_writer(name)

set_exception(exc)

set_transport(transport)

RequestStreamReader

class waterbutler.core.streams.RequestStreamReader(request, inner)

Bases: waterbutler.core.streams.base.BaseStream

size

add_reader(name, reader)

add_writer(name, writer)

at_eof()

Return True if the buffer is empty and ‘feed_eof’ was called.

exception()

feed_data(data)

feed_eof()

read(size=-1)

Read up to n bytes from the stream.

If n is not provided, or set to -1, read until EOF and return all read bytes. If the EOF was received and the internal buffer is empty, return an empty bytes object.

If n is zero, return empty bytes object immediately.

If n is positive, this function try to read n bytes, and may return less or equal bytes than requested, but at least one byte. If EOF was received before any byte is read, this function returns empty byte object.

Returned value is not limited with limit, configured at stream creation.

If stream was paused, this function will automatically resume it if needed.

readexactly(n)

Read exactly n bytes.

Raise an IncompleteReadError if EOF is reached before n bytes can be read. The IncompleteReadError.partial attribute of the exception will contain the partial read bytes.

if n is zero, return empty bytes object.

Returned value is not limited with limit, configured at stream creation.

If stream was paused, this function will automatically resume it if needed.

readline()

Read chunk of data from the stream until newline (b’ ‘) is found.

On success, return chunk that ends with newline. If only partial line can be read due to EOF, return incomplete line without terminating newline. When EOF was reached while no bytes read, empty bytes object is returned.

If limit is reached, ValueError will be raised. In that case, if newline was found, complete line including newline will be removed from internal buffer. Else, internal buffer will be cleared. Limit is compared against part of the line without newline.

If stream was paused, this function will automatically resume it if needed.

readuntil(separator=b'\n')

Read data from the stream until separator is found.

On success, the data and separator will be removed from the internal buffer (consumed). Returned data will include the separator at the end.

Configured stream limit is used to check result. Limit sets the maximal length of data that can be returned, not counting the separator.

If an EOF occurs and the complete separator is still not found, an IncompleteReadError exception will be raised, and the internal buffer will be reset. The IncompleteReadError.partial attribute may contain the separator partially.

If the data cannot be read because of over limit, a LimitOverrunError exception will be raised, and the data will be left in the internal buffer, so it can be read again.

remove_reader(name)

remove_writer(name)

set_exception(exc)

set_transport(transport)

FileStreamReader

class waterbutler.core.streams.FileStreamReader(file_pointer)

Bases: waterbutler.core.streams.base.BaseStream

size

close()

add_reader(name, reader)

add_writer(name, writer)

at_eof()

Return True if the buffer is empty and ‘feed_eof’ was called.

chunk_reader()

exception()

feed_data(data)

feed_eof()

read(size=-1)

Read up to n bytes from the stream.

If n is not provided, or set to -1, read until EOF and return all read bytes. If the EOF was received and the internal buffer is empty, return an empty bytes object.

If n is zero, return empty bytes object immediately.

If n is positive, this function try to read n bytes, and may return less or equal bytes than requested, but at least one byte. If EOF was received before any byte is read, this function returns empty byte object.

Returned value is not limited with limit, configured at stream creation.

If stream was paused, this function will automatically resume it if needed.

readexactly(n)

Read exactly n bytes.

Raise an IncompleteReadError if EOF is reached before n bytes can be read. The IncompleteReadError.partial attribute of the exception will contain the partial read bytes.

if n is zero, return empty bytes object.

Returned value is not limited with limit, configured at stream creation.

If stream was paused, this function will automatically resume it if needed.

readline()

Read chunk of data from the stream until newline (b’ ‘) is found.

On success, return chunk that ends with newline. If only partial line can be read due to EOF, return incomplete line without terminating newline. When EOF was reached while no bytes read, empty bytes object is returned.

If limit is reached, ValueError will be raised. In that case, if newline was found, complete line including newline will be removed from internal buffer. Else, internal buffer will be cleared. Limit is compared against part of the line without newline.

If stream was paused, this function will automatically resume it if needed.

readuntil(separator=b'\n')

Read data from the stream until separator is found.

On success, the data and separator will be removed from the internal buffer (consumed). Returned data will include the separator at the end.

Configured stream limit is used to check result. Limit sets the maximal length of data that can be returned, not counting the separator.

If an EOF occurs and the complete separator is still not found, an IncompleteReadError exception will be raised, and the internal buffer will be reset. The IncompleteReadError.partial attribute may contain the separator partially.

If the data cannot be read because of over limit, a LimitOverrunError exception will be raised, and the data will be left in the internal buffer, so it can be read again.

remove_reader(name)

remove_writer(name)

set_exception(exc)

set_transport(transport)

HashStreamWriter

class waterbutler.core.streams.HashStreamWriter(hasher)

Bases: object

Stream-like object that hashes and discards its input.

digest

hexdigest

can_write_eof()

write(data)

close()

StringStream

class waterbutler.core.streams.StringStream(data)

Bases: waterbutler.core.streams.base.BaseStream

size

add_reader(name, reader)

add_writer(name, writer)

at_eof()

Return True if the buffer is empty and ‘feed_eof’ was called.

exception()

feed_data(data)

feed_eof()

read(size=-1)

Read up to n bytes from the stream.

If n is not provided, or set to -1, read until EOF and return all read bytes. If the EOF was received and the internal buffer is empty, return an empty bytes object.

If n is zero, return empty bytes object immediately.

If n is positive, this function try to read n bytes, and may return less or equal bytes than requested, but at least one byte. If EOF was received before any byte is read, this function returns empty byte object.

Returned value is not limited with limit, configured at stream creation.

If stream was paused, this function will automatically resume it if needed.

readexactly(n)

Read exactly n bytes.

Raise an IncompleteReadError if EOF is reached before n bytes can be read. The IncompleteReadError.partial attribute of the exception will contain the partial read bytes.

if n is zero, return empty bytes object.

Returned value is not limited with limit, configured at stream creation.

If stream was paused, this function will automatically resume it if needed.

readline()

Read chunk of data from the stream until newline (b’ ‘) is found.

On success, return chunk that ends with newline. If only partial line can be read due to EOF, return incomplete line without terminating newline. When EOF was reached while no bytes read, empty bytes object is returned.

If limit is reached, ValueError will be raised. In that case, if newline was found, complete line including newline will be removed from internal buffer. Else, internal buffer will be cleared. Limit is compared against part of the line without newline.

If stream was paused, this function will automatically resume it if needed.

readuntil(separator=b'\n')

Read data from the stream until separator is found.

On success, the data and separator will be removed from the internal buffer (consumed). Returned data will include the separator at the end.

Configured stream limit is used to check result. Limit sets the maximal length of data that can be returned, not counting the separator.

If an EOF occurs and the complete separator is still not found, an IncompleteReadError exception will be raised, and the internal buffer will be reset. The IncompleteReadError.partial attribute may contain the separator partially.

If the data cannot be read because of over limit, a LimitOverrunError exception will be raised, and the data will be left in the internal buffer, so it can be read again.

remove_reader(name)

remove_writer(name)

set_exception(exc)

set_transport(transport)

MultiStream

class waterbutler.core.streams.MultiStream(*streams)

Bases: asyncio.streams.StreamReader

Concatenate a series of StreamReader objects into a single stream. Reads from the current stream until exhausted, then continues to the next, etc. Used to build streaming form data for Figshare uploads. Originally written by @jmcarp

size

streams

add_streams(*streams)

at_eof()

Return True if the buffer is empty and ‘feed_eof’ was called.

exception()

feed_data(data)

feed_eof()

read(n=-1)

Read up to n bytes from the stream.

If n is not provided, or set to -1, read until EOF and return all read bytes. If the EOF was received and the internal buffer is empty, return an empty bytes object.

If n is zero, return empty bytes object immediately.

If n is positive, this function try to read n bytes, and may return less or equal bytes than requested, but at least one byte. If EOF was received before any byte is read, this function returns empty byte object.

Returned value is not limited with limit, configured at stream creation.

If stream was paused, this function will automatically resume it if needed.

readexactly(n)

Read exactly n bytes.

Raise an IncompleteReadError if EOF is reached before n bytes can be read. The IncompleteReadError.partial attribute of the exception will contain the partial read bytes.

if n is zero, return empty bytes object.

Returned value is not limited with limit, configured at stream creation.

If stream was paused, this function will automatically resume it if needed.

readline()

Read chunk of data from the stream until newline (b’ ‘) is found.

On success, return chunk that ends with newline. If only partial line can be read due to EOF, return incomplete line without terminating newline. When EOF was reached while no bytes read, empty bytes object is returned.

If limit is reached, ValueError will be raised. In that case, if newline was found, complete line including newline will be removed from internal buffer. Else, internal buffer will be cleared. Limit is compared against part of the line without newline.

If stream was paused, this function will automatically resume it if needed.

readuntil(separator=b'\n')

Read data from the stream until separator is found.

On success, the data and separator will be removed from the internal buffer (consumed). Returned data will include the separator at the end.

Configured stream limit is used to check result. Limit sets the maximal length of data that can be returned, not counting the separator.

If an EOF occurs and the complete separator is still not found, an IncompleteReadError exception will be raised, and the internal buffer will be reset. The IncompleteReadError.partial attribute may contain the separator partially.

If the data cannot be read because of over limit, a LimitOverrunError exception will be raised, and the data will be left in the internal buffer, so it can be read again.

set_exception(exc)

set_transport(transport)

FormDataStream

class waterbutler.core.streams.FormDataStream(**fields)

Bases: waterbutler.core.streams.base.MultiStream

A child of MultiSteam used to create stream friendly multipart form data requests. Usage:

>>> stream = FormDataStream(key1='value1', file=FileStream(...))

Or:

>>> stream = FormDataStream()
>>> stream.add_field('key1', 'value1')
>>> stream.add_file('file', FileStream(...), mime='text/plain')

Additional options for files can be passed as a tuple ordered as:

>>> FormDataStream(fieldName=(FileStream(...), 'fileName', 'Mime', 'encoding'))

Auto generates boundaries and properly concatenates them Use FormDataStream.headers to get the proper headers to be included with requests Namely Content-Length, Content-Type

Parameters:fields (dict) – A dict of fieldname: value to create the body of the stream

classmethod make_boundary()

Creates a random-ish boundary for form data separator

classmethod make_header(name, disposition='form-data', additional_headers=None, **extra)

end_boundary

headers

The headers required to make a proper multipart form request Implicitly calls finalize as accessing headers will often indicate sending of the request Meaning nothing else will be added to the stream

finalize()

add_fields(**fields)

add_field(key, value)

add_file(field_name, file_stream, file_name=None, mime='application/octet-stream', disposition='file', transcoding='binary')

add_streams(*streams)

at_eof()

Return True if the buffer is empty and ‘feed_eof’ was called.

exception()

feed_data(data)

feed_eof()

read(n=-1)

Read up to n bytes from the stream.

If n is not provided, or set to -1, read until EOF and return all read bytes. If the EOF was received and the internal buffer is empty, return an empty bytes object.

If n is zero, return empty bytes object immediately.

If n is positive, this function try to read n bytes, and may return less or equal bytes than requested, but at least one byte. If EOF was received before any byte is read, this function returns empty byte object.

Returned value is not limited with limit, configured at stream creation.

If stream was paused, this function will automatically resume it if needed.

readexactly(n)

Read exactly n bytes.

Raise an IncompleteReadError if EOF is reached before n bytes can be read. The IncompleteReadError.partial attribute of the exception will contain the partial read bytes.

if n is zero, return empty bytes object.

Returned value is not limited with limit, configured at stream creation.

If stream was paused, this function will automatically resume it if needed.

readline()

Read chunk of data from the stream until newline (b’ ‘) is found.

On success, return chunk that ends with newline. If only partial line can be read due to EOF, return incomplete line without terminating newline. When EOF was reached while no bytes read, empty bytes object is returned.

If limit is reached, ValueError will be raised. In that case, if newline was found, complete line including newline will be removed from internal buffer. Else, internal buffer will be cleared. Limit is compared against part of the line without newline.

If stream was paused, this function will automatically resume it if needed.

readuntil(separator=b'\n')

Read data from the stream until separator is found.

On success, the data and separator will be removed from the internal buffer (consumed). Returned data will include the separator at the end.

Configured stream limit is used to check result. Limit sets the maximal length of data that can be returned, not counting the separator.

If an EOF occurs and the complete separator is still not found, an IncompleteReadError exception will be raised, and the internal buffer will be reset. The IncompleteReadError.partial attribute may contain the separator partially.

If the data cannot be read because of over limit, a LimitOverrunError exception will be raised, and the data will be left in the internal buffer, so it can be read again.

set_exception(exc)

set_transport(transport)

size

streams

CutoffStream

class waterbutler.core.streams.CutoffStream(stream, cutoff)

A wrapper around an existing stream that terminates after pulling off the specified number of bytes. Useful for segmenting an existing stream into parts suitable for chunked upload interfaces.

This class only subclasses asyncio.StreamReader to take advantage of the isinstance-based stream-reading interface of aiohttp v0.18.2. It implements a read() method with the same signature as StreamReader that does the bookkeeping to know how many bytes to request from the stream attribute.

Parameters:

• stream – a stream object to wrap
• cutoff (int) – number of bytes to read before stopping

size

The lesser of the wrapped stream’s size or the cutoff.

read(n=-1)

Read n bytes from the stream. n is a chunk size, not the full size of the stream. If n is -1, read cutoff bytes. If n is a positive integer, read that many bytes as long as the total number of bytes read so far does not exceed cutoff.