Hi All,
Thank you for taking the time to review this post;
To give some background on the issue with which I am facing; I am working with the INSIGHTLY API - there are 3 versions of the API each with subtle differences - such as the availability of certain fields in v.2.2/2.3 which do not exist in 2.1. Critically, 2.2 & 2.3 have restrictions on the number of rows (500 max) which can be retrieved in making the call. I have managed to get around this and I am able to pull all records from the required views using a method I found and modified which uses the following function;
this is the call made to the API within which I am using a reccursive method to pull 500 rows, the pull another 500 skipping the previous 500. This is explained as clearly as possible within the code and the 4 screenshots below;
Notice above within the URL "skip=" & OffSet & "&top=500"))" in conjunction with the table below. The call pulls 500 records from each offset point.
Notice above here we have intervals amounting to the maximum number of rows the api will allow.
Sets of 500 records are retrieved and upon expanding the columns, the desired result is returned in the below screenshot; to tidy this up I could delete the offset column once expanded but this is not really important at this stage;
The issue I have is that this is not future proof. As I know how many records exist within the table (based on V2.1 of the API which is unlimited) I am able to set my intervals out to 11001 in buckets of 500 and all records are retrieved; note the formula in the second screenshot above;
"= Table.TransformColumnTypes(Table.FromRows({{"0"},{"501"},{"1001"},{"1501"},{"2001"},{"2501"},{"3001"},{"3501"},{"4001"},{"4501"},{"5001"},{"5501"},{"6001"},{"6501"},{"7001"},{"7501"},{"8001"},{"8501"},{"9001"},{"9501"},{"10001"},{"10501"},{"11001"}},{"OffSet"}) ,{{"OffSet", type text}})"
When the records surpass 11501, I will need to manually add a new upper bracket to the list in order to continue retrieving all rows within the table; the solution I am therefore looking to implement is within the function above as pasted in italics - is there a method to have this table populate in buckets of 500 (or other increment) but to an upper limit which can be in someway calculated based on a count of records within another table. For instance if I have a separate table performing a count of all records from API 2.1, and the result of this is 11750 (for example), could the above function be amended to take that max row count and create the table accordingly - i.e. without manually adding a new upper bucket manually?
Just for clarity, the solution I have works perfectly for the time being - using the manually assigned intervals - in that I am able to manually update, retrieve all records etc - the issue is going to come when I have to hand this to a less experienced end user to maintain, automating this piece will be a big win. I am also aware of others who have experienced similar issues both with this particular API and other similar.
Any help would be greatly appreciated,
Thanks.