Quantcast
Channel: Power Query topics
Viewing all articles
Browse latest Browse all 31015

Getting Data from Elastic Stack with REST with Scroll

$
0
0

Hello,

 

I'm pulling data into Power BI using the Elastic Stack REST API, but I'm running into a problem when trying to pull more than 10k records.

 

In order to do so, I need to use the scroll function: https://www.elastic.co/guide/en/elasticsearch/reference/master/search-request-scroll.html

 

The first call:

 

 

GET /_search?scroll=1m 
{ "query": ... }

Will return the first set of results plus a scroll id: 

 

DXF1ZXJ5QW5kRmV0Y2gBAAAAAAAAAD4...

 

Which is used for subsequent calls:

 

 

GET /_search {  "scroll_id": "DXF1Z..."
}

 

 

Each call returns a set of records plus a new scroll id. The scroll id is then passed onto the next record, and so on. 

 

For performance reasons, I don't want to increase the number of records that can be returned by the REST API.

 

Is it possible to solve this using Power BI?


Thanks,


Ken

 


Viewing all articles
Browse latest Browse all 31015

Trending Articles



<script src="https://jsc.adskeeper.com/r/s/rssing.com.1596347.js" async> </script>