hikari.iterators#

Lazy iterators for data that requires repeated API calls to retrieve.

For consumers of this API, the only class you need to worry about is LazyIterator. Everything else is internal detail only exposed for people who wish to extend this API further!

Module Contents#

class hikari.iterators.All(conditions)[source]#

Bases: Generic[ValueT]

Helper that wraps predicates and invokes them together.

Calling this object will pass the input item to each item, returning True only when all wrapped predicates return True when called with the given item.

For example…

if w(foo) and x(foo) and y(foo) and z(foo):
    ...

is equivalent to

condition = All([w, x, y, z])

if condition(foo):
    ...

This behaves like a lazy wrapper implementation of the all builtin.

Note

Like the rest of the standard library, this is a short-circuiting operation. This means that if a predicate returns False, no predicates after this are invoked, as the result is already known. In this sense, they are invoked in-order.

Warning

You should not generally need to use this outside of extending the iterators API in this library!

Parameters:
conditionstyping.Callable[[ValueT], bool]

The predicates to wrap.

conditions: Collection[Callable[[ValueT], bool]][source]#

Collection of the condition callbacks within this.

class hikari.iterators.AttrComparator(attr_name, expected_value, cast=None)[source]#

Bases: Generic[ValueT]

A comparator that compares the result of a call with something else.

This uses the spel module internally.

Parameters:
attr_namestr

The attribute name. Can be prepended with a . optionally. If the attribute name ends with a (), then the call is invoked rather than treated as a property (useful for methods like str.isupper, for example).

expected_valuetyping.Any

The expected value.

casttyping.Optional[typing.Callable[[ValueT], typing.Any]]

Optional cast to perform on the input value when being called before comparing it to the expected value but after accessing the attribute.

class hikari.iterators.BufferedLazyIterator[source]#

Bases: Generic[ValueT], LazyIterator[ValueT], abc.ABC

A special kind of lazy iterator that is used by internal components.

The purpose of this is to provide an interface to lazily deserialize collections of payloads received from paginated API endpoints such as GET /channels/{channel_id}/messages, which will return a certain number of messages at a time on a low level. This class provides the base interface for handling lazily decoding each item in those responses and returning them in the expected format when iterating across this object.

Implementations are expected to provide a _next_chunk private method which when awaited returns a lazy generator of each deserialized object to later yield. This will be iterated across lazily by this implementation, thus reducing the amount of work needed if only a few objects out of, say, 100, need to be deserialized.

This _next_chunk should return None once the end of all items has been reached.

An example would look like the following:

async def some_http_call(i):
    ...

class SomeEndpointLazyIterator(BufferedLazyIterator[SomeObject]):
    def __init__(self):
        super().__init__()
        self._i = 0

    def _next_chunk(self) -> typing.Optional[typing.Generator[ValueT, None, None]]:
        raw_items = await some_http_call(self._i)
        self._i += 1

        if not raw_items:
            return None

        generator = (SomeObject(raw_item) for raw_item in raw_items)
        return generator
class hikari.iterators.FlatLazyIterator(values)[source]#

Bases: Generic[ValueT], LazyIterator[ValueT]

A lazy iterator that has all items in-memory and ready.

This can be iterated across as a normal iterator, or as an async iterator.

class hikari.iterators.LazyIterator[source]#

Bases: Generic[ValueT], abc.ABC

A set of results that are fetched asynchronously from the API as needed.

This is a typing.AsyncIterable and typing.AsyncIterator with several additional helpful methods provided for convenience.

Examples

You can use this in multiple ways.

As an async iterable:

>>> async for item in paginated_results:
...    process(item)

As an eagerly retrieved set of results (performs all API calls at once, which may be slow for large sets of data):

>>> results = await paginated_results
>>> # ... which is equivalent to this...
>>> results = [item async for item in paginated_results]

As an async iterator (not recommended):

>>> try:
...    while True:
...        process(await paginated_results.__anext__())
... except StopAsyncIteration:
...    pass

Additionally, you can make use of some of the provided helper methods on this class to perform basic operations easily.

Iterating across the items with indexes (like enumerate for normal iterables):

>>> async for i, item in paginated_results.enumerate():
...    print(i, item)
(0, foo)
(1, bar)
(2, baz)

Limiting the number of results you iterate across:

>>> async for item in paginated_results.limit(3):
...    process(item)
awaiting(window_size=10)[source]#

Await each item concurrently in a fixed size window.

Warning

Setting a large window size, or setting it to 0 to await everything is a dangerous thing to do if you are making API calls. Some endpoints will get ratelimited and cause a backup of waiting tasks, others may begin to spam global rate limits instead (the fetch_user endpoint seems to be notorious for doing this).

Note

This call assumes that the iterator contains awaitable values as input. MyPy cannot detect this nicely, so any cast is forced internally. If the item is not awaitable, you will receive a TypeError instead. You have been warned. You cannot escape the ways of the duck type young grasshopper.

Parameters:
window_sizeint

The window size of how many tasks to await at once. You can set this to 0 to await everything at once, but see the below warning.

Returns:
LazyIterator[ValueT]

The new lazy iterator to return.

chunk(chunk_size)[source]#

Return results in chunks of up to chunk_size amount of entries.

Parameters:
chunk_sizeint

The limit for how many results should be returned in each chunk.

Returns:
LazyIterator[typing.Sequence[ValueT]]

LazyIterator that emits each chunked sequence.

async collect(collector)[source]#

Collect the results into a given type and return it.

Parameters:
collector

A function that consumes a sequence of values and returns a collection.

async count()[source]#

Count the number of results.

Returns:
int

Number of results found.

enumerate(*, start=0)[source]#

Enumerate the paginated results lazily.

This behaves as an asyncio-friendly version of enumerate which uses much less memory than collecting all the results first and calling enumerate across them.

Parameters:
startint

Optional int to start at. If omitted, this is 0.

Returns:
LazyIterator[typing.Tuple[int, T]]

A paginated results view that asynchronously yields an increasing counter in a tuple with each result, lazily.

Examples

>>> async for i, item in paginated_results.enumerate():
...    print(i, item)
(0, foo)
(1, bar)
(2, baz)
(3, bork)
(4, qux)

>>> async for i, item in paginated_results.enumerate(start=9):
...    print(i, item)
(9, foo)
(10, bar)
(11, baz)
(12, bork)
(13, qux)

>>> async for i, item in paginated_results.enumerate(start=9).limit(3):
...    print(i, item)
(9, foo)
(10, bar)
(11, baz)
filter(*predicates, **attrs)[source]#

Filter the items by one or more conditions.

Each condition is treated as a predicate, being called with each item that this iterator would return when it is requested.

All conditions must evaluate to True for the item to be returned. If this is not met, then the item is discarded and ignored, the next matching item will be returned instead, if there is one.

Parameters:
*predicatestyping.Union[typing.Callable[[ValueT], bool], typing.Tuple[str, typing.Any]]

Predicates to invoke. These are functions that take a value and return True if it is of interest, or False otherwise. These may instead include 2-tuple objects consisting of a str attribute name (nested attributes are referred to using the . operator), and values to compare for equality. This allows you to specify conditions such as members.filter(("user.bot", True)).

**attrstyping.Any

Alternative to passing 2-tuples. Cannot specify nested attributes using this method.

Returns:
LazyIterator[ValueT]

LazyIterator that only emits values where all conditions are matched.

flat_map(flattener)[source]#

Perform a flat mapping operation.

This will pass each item in the iterator to the given function parameter, expecting a new typing.Iterable or typing.AsyncIterator to be returned as the result. This means you can map to a new LazyIterator, typing.AsyncIterator, typing.Iterable, async generator, or generator.

Remember that typing.Iterator implicitly provides typing.Iterable compatibility.

This is used to provide lazy conversions, and can be used to implement reactive-like pipelines if desired.

All results are combined into one large lazy iterator and yielded lazily.

Parameters:
flattener

A function that returns either an async iterator or iterator of new values. Could be an attribute name instead.

Returns:
LazyIterator[AnotherValueT]

The new lazy iterator to return.

Examples

The following example generates a distinct collection of all mentioned users in the given channel from the past 500 messages.

def iter_mentioned_users(message: hikari.Message) -> typing.Iterable[Snowflake]:
    for match in re.findall(r"<@!?(\d+)>", message.content):
        yield Snowflake(match)

mentioned_users = await (
    channel
    .history()
    .limit(500)
    .map(".content")
    .flat_map(iter_mentioned_users)
    .distinct()
)
async for_each(consumer)[source]#

Forward each value to a given consumer immediately.

async last()[source]#

Return the last element of this iterator only.

Note

This method will consume the whole iterator if run.

Returns:
ValueT

The last result.

Raises:
LookupError

If no result exists.

limit(limit)[source]#

Limit the number of items you receive from this async iterator.

Parameters:
limitint

The number of items to get. This must be greater than zero.

Returns:
LazyIterator[ValueT]

A paginated results view that asynchronously yields a maximum of the given number of items before completing.

Examples

>>> async for item in paginated_results.limit(3):
...     print(item)
map(transformation)[source]#

Map the values to a different value.

Parameters:
transformationtyping.Union[typing.Callable[[ValueT], bool], str]

The function to use to map the attribute. This may alternatively be a string attribute name to replace the input value with. You can provide nested attributes using the . operator.

Returns:
LazyIterator[AnotherValueT]

LazyIterator that maps each value to another value.

async next()[source]#

Return the next element of this iterator only.

Returns:
ValueT

The next result.

Raises:
LookupError

If no more results exist.

reversed()[source]#

Return a lazy iterator of the remainder of this iterator’s values reversed.

Returns:
LazyIterator[ValueT]

The lazy iterator of this iterator’s remaining values reversed.

skip(number)[source]#

Drop the given number of items, then yield anything after.

Parameters:
numberint

The max number of items to drop before any items are yielded.

Returns:
LazyIterator[ValueT]

A paginated results view that asynchronously yields all items AFTER the given number of items are discarded first.

skip_until(*predicates, **attrs)[source]#

Discard items while all conditions are False.

Items after this will be yielded as normal.

Parameters:
*predicatestyping.Union[typing.Callable[[ValueT], bool], typing.Tuple[str, typing.Any]]

Predicates to invoke. These are functions that take a value and return True if it is of interest, or False otherwise. These may instead include 2-tuple objects consisting of a str attribute name (nested attributes are referred to using the . operator), and values to compare for equality. This allows you to specify conditions such as members.skip_until(("user.bot", True)).

**attrstyping.Any

Alternative to passing 2-tuples. Cannot specify nested attributes using this method.

Returns:
LazyIterator[ValueT]

LazyIterator that only emits values once a condition has failed. All items before this are discarded.

skip_while(*predicates, **attrs)[source]#

Discard items while all conditions are True.

Items after this will be yielded as normal.

Parameters:
*predicatestyping.Union[typing.Callable[[ValueT], bool], typing.Tuple[str, typing.Any]]

Predicates to invoke. These are functions that take a value and return True if it is of interest, or False otherwise. These may instead include 2-tuple objects consisting of a str attribute name (nested attributes are referred to using the . operator), and values to compare for equality. This allows you to specify conditions such as members.skip_while(("user.bot", True)).

**attrstyping.Any

Alternative to passing 2-tuples. Cannot specify nested attributes using this method.

Returns:
LazyIterator[ValueT]

LazyIterator that only emits values once a condition has been met. All items before this are discarded.

async sort(*, key=None, reverse=False)[source]#

Collect all results, then sort the collection before returning it.

take_until(*predicates, **attrs)[source]#

Return each item until any conditions pass or the end is reached.

Parameters:
*predicatestyping.Union[typing.Callable[[ValueT], bool], typing.Tuple[str, typing.Any]]

Predicates to invoke. These are functions that take a value and return True if it is of interest, or False otherwise. These may instead include 2-tuple objects consisting of a str attribute name (nested attributes are referred to using the . operator), and values to compare for equality. This allows you to specify conditions such as members.take_until(("user.bot", True)).

**attrstyping.Any

Alternative to passing 2-tuples. Cannot specify nested attributes using this method.

Returns:
LazyIterator[ValueT]

LazyIterator that only emits values until any conditions are matched.

take_while(*predicates, **attrs)[source]#

Return each item until any conditions fail or the end is reached.

Parameters:
*predicatestyping.Union[typing.Callable[[ValueT], bool], typing.Tuple[str, typing.Any]]

Predicates to invoke. These are functions that take a value and return True if it is of interest, or False otherwise. These may instead include 2-tuple objects consisting of a str attribute name (nested attributes are referred to using the . operator), and values to compare for equality. This allows you to specify conditions such as members.take_while(("user.bot", True)).

**attrstyping.Any

Alternative to passing 2-tuples. Cannot specify nested attributes using this method.

Returns:
LazyIterator[ValueT]

LazyIterator that only emits values until any conditions are not matched.

hikari.iterators.AnotherValueT[source]#

Type-hint of the type of a value by a mapped lazy iterator.

hikari.iterators.ValueT[source]#

Type-hint of the type of the value returned by a lazy iterator.