Skip to content

feat: Add iterate methods for paginated collections#771

Open
Pijukatel wants to merge 3 commits intomasterfrom
add-iterate-helpers
Open

feat: Add iterate methods for paginated collections#771
Pijukatel wants to merge 3 commits intomasterfrom
add-iterate-helpers

Conversation

@Pijukatel
Copy link
Copy Markdown
Contributor

@Pijukatel Pijukatel commented Apr 30, 2026 •
edited by vdusek
Loading

Description

Support more convenient iteration through paginated endpoints of collection clients.

  • Added iterate methods to following clients(same for async clients):
    • ActorCollectionClient
    • BuildCollectionClient
    • RunCollectionClient
    • ScheduleCollectionClient
    • TaskCollectionClient
    • WebhookCollectionClient
    • WebhookDispatchCollectionClient
    • DatasetCollectionClient
    • KeyValueStoreCollectionClient
    • RequestQueueCollectionClient
    • StoreCollectionClient
    • ActorEnvVarCollectionClient
    • ActorVersionCollectionClient

Example usage

...
# Sync
datasets_client = ApifyClient(token='...').datasets()

# Same as before
list_page = datasets_client.list(...)

# New functionality
individual_items = [item for item in datasets_client.iterate(...)]

...
# Async
datasets_client = ApifyClientAsync(token='...').datasets()

# Same as before
list_page = await datasets_client.list(...)

# New functionality
individual_items = [item async for item in datasets_client.iterate(...)]

Testing

  • Unit tests
  • Manual API tests

Checklist

  • CI passed

Issues

Closes: #539

@github-actions github-actions Bot added this to the 139th sprint - Tooling team milestone Apr 30, 2026
@github-actions github-actions Bot added the t-tooling Issues with this label are in the ownership of the tooling team. label Apr 30, 2026
@Pijukatel Pijukatel force-pushed the add-iterate-helpers branch from 809e66d to a6d7d51 Compare April 30, 2026 08:49
@github-actions github-actions Bot added the tested Temporary label used only programatically for some analytics. label Apr 30, 2026
@codecov
Copy link
Copy Markdown

codecov Bot commented Apr 30, 2026 •
edited
Loading

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 97.77%. Comparing base (94d5bf4) to head (c24dc1f).
⚠️ Report is 5 commits behind head on master.

Additional details and impacted files 
@@            Coverage Diff             @@
##           master     #771      +/-   ##
==========================================
+ Coverage   97.56%   97.77%   +0.20%     
==========================================
  Files          47       48       +1     
  Lines        4725     4855     +130     
==========================================
+ Hits         4610     4747     +137     
+ Misses        115      108       -7
Flag Coverage Δ
integration 93.69% <59.27%> (-1.84%) ⬇️
unit 86.17% <100.00%> (+4.06%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@Pijukatel Pijukatel force-pushed the add-iterate-helpers branch from a6d7d51 to 9c38df4 Compare April 30, 2026 09:16
This was referenced Apr 30, 2026
Merged
Closed
vdusek pushed a commit that referenced this pull request Apr 30, 2026
@Pijukatel
ci: pass secrets to unit tests reusable workflow (#772)
63b866a 
## Summary

Adds `secrets: inherit` to the `unit_tests` job in
`.github/workflows/_tests.yaml` so `CODECOV_TOKEN` is forwarded to the
reusable workflow at
`apify/workflows/.github/workflows/python_unit_tests.yaml@main`.

## Why

The reusable unit-tests workflow declares `CODECOV_TOKEN` as a
`workflow_call` secret and gates its upload step on `if:
env.CODECOV_TOKEN != ''`. Without `secrets: inherit` (or an explicit
`secrets:` map) on the caller, the secret is empty inside the called
workflow, so the upload step is silently skipped on every commit and the
`unit` flag is never sent to Codecov — Codecov shows it as
carried-forward.

This is why PRs (e.g. #771) only see the `integration` flag in the
Codecov comment and patch coverage numbers reflect only integration
tests. `secrets: inherit` on the grandparent (`on_pull_request.yaml`)
does not auto-propagate through an intermediate caller — it must be set
on each `uses:` invocation.

`apify/apify-sdk-python` already has `secrets: inherit` set on the same
reusable workflow call.
@vdusek vdusek changed the title Add iterate helpers feat: Add iterate methods for paginated collections May 5, 2026
@vdusek vdusek marked this pull request as ready for review May 5, 2026 07:32
@vdusek vdusek force-pushed the add-iterate-helpers branch from fb54c8e to 1259abb Compare May 5, 2026 07:44
@vdusek vdusek requested review from janbuchar and vdusek May 5, 2026 07:44
@vdusek vdusek force-pushed the add-iterate-helpers branch from 4685d7f to e8e4dfc Compare May 6, 2026 11:40
vdusek
vdusek approved these changes May 6, 2026
View reviewed changes
Copy link
Copy Markdown
Contributor

@vdusek vdusek left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I did a few updates, LGTM from my side now. @janbuchar, could you take a look at this as well?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@vdusek vdusek vdusek approved these changes

@janbuchar janbuchar Awaiting requested review from janbuchar

Assignees

@Pijukatel Pijukatel

Labels

t-tooling Issues with this label are in the ownership of the tooling team. tested Temporary label used only programatically for some analytics.

Projects

None yet

Milestone

139th sprint - Tooling team

Development

Successfully merging this pull request may close these issues.

Add better support for pagination

3 participants

@Pijukatel @vdusek @apify-service-account