ApiSkills

Power Query (M) · 12 min

Power Query: fetch all pages

Use List.Generate to follow the relative "next" link until it runs out.

The data endpoints return pagination.next as a relative path. In M, List.Generate loops: it keeps calling the next path until the API returns next = null, collecting each page. Then you expand the combined rows.

let
    Base = "https://api.ifsjaipur.cloud",
    Key  = "YOUR_API_KEY",

    Fetch = (rel as text) =>
        Json.Document(Web.Contents(Base, [
            RelativePath = rel,
            Headers = [ #"x-api-key" = Key ]
        ])),

    Pages = List.Generate(
        () => [ Rel = "playground/data/stocks/candles?shape=flat&page_size=50",
                Resp = Fetch(Rel) ],
        each [Rel] <> null,
        each let Next = [Resp][pagination][next]
             in [ Rel  = if Next = null then null else Text.TrimStart(Next, "/"),
                  Resp = if Next = null then null else Fetch(Text.TrimStart(Next, "/")) ],
        each [Resp]
    ),

    Rows = List.Combine(
        List.Transform(List.RemoveNulls(Pages), each [data])
    ),
    Table = Table.FromRecords(Rows)
in
    Table
  • The relative next starts with "/"; Text.TrimStart removes it so it works as a RelativePath.
  • Each page's rows live in [data]; List.Combine flattens them into one list.
  • Cap the loop in class by adding a page counter to the state record if you want a safety limit.

Tip · This is the exact same idea as the n8n "follow next URL" pagination — different syntax, identical concept. Teaching both side by side makes the pattern click.