Using Power Query, can someone take one of the resulting data tables and load it to an external system?

Imagine Power BI Desktop was pulling data in from several totally different data locations.  Could that resulting merged or combined data then be exported to SQL Server?

Basically using PBI Desktop for simple ETL?  I find that a lot of the data work I do in PBI Desktop has other useful applications if we were to dump it into SQL server or even to archive it off.

For example, I have this simple SQL statement:

SELECT a, SUM(b)*a  FROM table  GROUP BY a

How can I implement "SUM(b)*a" in a Table.Group function? For "SUM(b)" Power Query generates "each List.Sum([b])", but how to multiply this by "a" ? Or how to implement SUM(b*a), which cause the same result?

I am attempting to connect to a web data source.

The logon page has three fields:
-Username

-Password

-Organization

I've used the authentication built into Power Query before, and I've used HTTP request header parameters. Unfortunately I can't pass the "Organization" parameter in the header.. This is the error when I try that:
" 'login-organization' header is only supported when connecting anonymously "

Any ideas on a different way to pass all the credential elements? or use the basic authentication and pass the "Organization" one in a different way?

Hi all,

My organisation stores documents in HP TRIM. Excel documents are commonly stored in this environment.

How would one connect Power BI to HP TRIM, and then to the excel document in question stored in HP TRIM?

Thanks

Adrian

Hey, 

I have three tables called Projects, Demand and Date. Project and Demand are connected through ProjectID and Demand and Date are connected through date. Currently I have a report where I can see sum of demand hours for each project for selected days. The problem is that sometimes projects get delayd, meaning for example that no work can be done for a month, and the required resourced demand will move forwards one month. 

I have used what-if parameters before, and managed to create a slicer, which allows the user to choose a project, and adjust the total demand for the project. What I've recently been trying to do is to allow the user to choose a project and for example move the demand forwards one month, like you can see below. 

Current demand

Project Name         January          February           March

Project X                  100h               100h                 50h

Project Y                   50h                 50h                  100h

Project Z                   100h               100h                 0h

Demand after Project Y and Project Z have been delayd by one month

Project Name         January          February           March

Project X                  100h               100h                 50h

Project Y                   0h                  100h                 100h

Project Z                   100h               0h                    100h

How should I start working on this? 

I have tried creating a new measure that changes the date in Demand table using the following:

Hello everyone,

I want to return a column if their corresponding criteria has been met, for example;

I only want to filter names whose sales growth were positive which is computed and not included in the database.

hope you can help me with this one.

Hi

I have an OData feed who work perfectly but not in a particular configuration. I work on Dynamics 365 CRM and can obtain all tables since :

"https://{MY_SITE}.api.crm4.dynamics.com/api/data/v9.1/"

and contacts Attributes with

https://{MY_SITE}.api.crm4.dynamics.com/api/data/v9.0/EntityDefinitions(LogicalName='contact')?$select=LogicalName&$expand=Attributes($select=LogicalName;$filter=AttributeType eq Microsoft.Dynamics.CRM.AttributeTypeCode'Picklist')

but not since

https://{MY_SITE}.api.crm4.dynamics.com/api/data/v9.0/EntityDefinitions(LogicalName='contact')/Attributes/Microsoft.Dynamics.CRM.PicklistAttributeMetadata?$select=LogicalName&$expand=OptionSet,GlobalOptionSet

I've followed all instruction on https://docs.microsoft.com/fr-fr/dynamics365/customer-engagement/developer/webapi/query-metadata-web-api but it don't work.

Someone can say why? I would like to get OptionList attributes but it was not present in table GlobalOptionSetDefinition

The complete code in M in power query :

let
    Source = OData.Feed("https://{MY_SITE}.api.crm4.dynamics.com/api/data/v9.0/EntityDefinitions(LogicalName='contact')?$select=LogicalName&$expand=Attributes($select=LogicalName;$filter=AttributeType eq Microsoft.Dynamics.CRM.AttributeTypeCode'Picklist')", null, [Implementation="2.0", Headers = [Accept = "application/json"]]),
    Attributes = Source[Attributes]
in
    Attributes

Thanks

Hi.  This might be simple but still a newbie here.  Would appreciate any help to push me in the right direction.

What I am trying to do is count the rows the equal to
current tix-1

current tix-2

current tix-3

current tix-4

current tix-5

= Table.AddColumn(#"Added Custom1", "Count", each List.Count(Table.SelectRows(#"Added Custom1", (C) => 
    (
        [Tix]>=C[Tix]-(List.Generate(()=>1,each _ 5, each _ - 1))
        )
        )[Column1]))

Thanks in advance.

I am trying to load the ZIP file in power BI through below query but it throws the error “We didn’t recognizes the format of your first file (). Please filter the list of files so it contains only supported types (Text, .csv, Excel workbooks etc.) and try again”.

I have tried this http://sql10.blogspot.com/2016/06/reading-zip-files-in-powerquery-m.html and https://www.excelandpowerbi.com/?p=155 but did not work either...I want to extract the .csv from ZIP file and load the data (if .csv can then be converted to .xlsx that would be more great)

Please help..

Hello everyone!

I'm having trouble preparing my data using Power Query in Power BI. I have a dataset that has the following columns: CustomerID, Street Name, Zip Code and Location Code. Location Code has some missing values, but I am able to fill in some of those by merging another query (customer info). Still, I am left with some missing values. In Excel I solved the problem by looking up street name and zip code, and returning corresponding location code (location code is based on this pair).

However, I am a literal noob in Power Query M and, even though I have given this a lot of thought, have not been able to solve the problem. Even though the number of missing values is not that significant, I would prefer not losing data in this case.

My data looks something like this:

Obviously, the end result should look like this:

I would appreciate any help and suggestions. I am not even sure if this is possible or practical in the first place.

I am writing a custom connector to interface with an API, and I'm running into some incredibly frustrating issues with the way that Web.Contents is behaving. The connector goes as follows;

1. supply a list of IDs (referencing server stored objects)
2. supply a Username and Password (The authorization is UsernamePassword)
3. The username and password are combined to make a short term auth token, which are sent to a /login endpoint server side that returns a longer lived authtoken that will be used later
4. Using this new authoken that was the result of hitting the /login endpoint, make a dynamic amount of calls (dependant on other information that was returned in the /login request) to an endpoint that generates csv files. Most of these csv files are relatively small (~100 rows, 5 columns) with one large csv at the end (~400,000 rows and ~10 columns).
5. All of these csvs are then processed (promote headers, modify some column types, etc.)  and then placed into a navigation table, which is then shown to the user.

The connector looks something like this:

shared Connector.Contents = (IDs as list) =>
let
       longLivedAuthToken = 
       let
	      shortLivedAuthToken = Extension.GetCredentials()[Username] & Extension.GetCredentials()[Password],
	      response = Binary.Buffer(Web.Contents("/Login", Authorization = shortLivedAuthToken)),//This does not get cached
       in
	      response[longLivedAuthToken],
       BigCSV = 
       let
	      table = Binary.Buffer(Web.Contents("/BigCSVEndpoint", Authorization = longLivedAuthToken)),//Neither does this
	                                                          //                      ^
                                                                  //It looks like this doesn't use the value longLivedAuthToken,
                                                                  //But rather it replaces this with the actual call (think a C macro)
              promoted = Table.PromoteHeaders(table),
       	      named = CustomRenameFunction(promoted, Table.First(promoted)),
       in
	      named,
      manySmallCSVs = 
      let
             manySmallCSVs = List.Transform(IDs, each Binary.Buffer(Web.Contents("/SmallCSVEndpoint", Authorization = longLivedAuthToken, query = _))),//Or this
                                                                                                                         // ^
                                                                                                          //It looks like this doesn't use the value longLivedAuthToken,
                                                                                                          //But rather it replaces this with the actual call (think a C macro)	     
             promoted = List.Transform(manySmallCSVs, each Table.PromoteHeaders(_)),
	     renamed = List.Transform(manySmallCSVs, each (CustomRenameFunction(_, Table.First(_))))
      in
	     renamed
      navtable = 
      let
 	     allCSVs = List.Combine(manySmallCSVs, {BigCSV}),
	     table = Table.GenerateNavigationTableFromList(allCSVs)
      in
	     table
in
	navtable;

My issue is two fold:

Firstly, I can see in the server logs that a log in attempt is made over 7-8 times for each CSV in the import (in general there are 7-10 csvs). This eventually causes the /login endpoint to return a 409 Conflict error (too many concurrent users of the endpoint) which causes the whole import to fail. I have tried buffering the results of the login call using Binary.Buffer() on the Web.Contents call, but it does not seem to work.

Secondly, the first issue also manifests with the CSV downloads. Hitting these endpoints with a browser generally takes ~2 seconds to download the files individually in a single request, but for some bizarre reason importing them in the connector takes minutes, and on the server side there are literally hundreds of requests being made. (It looks like Power BI is downloading them in small chuncks, and making subsequent requests to get the next chunck? This is not programmed behaviour on the API side but most of the request contain the line 'expect 100 continue' which seems to indicate this is what is happening)

I have found that disabling parallel loads solves the issue of the /login attempts returning a 409 status code, but that is just because they all happen in series rather than in parallel, the same number of total calls are still made and the whole import takes a lot longer (scales with the number of csvs).

An interesting thing that I've noted though is that there seem to be Three distinct phases of the import. The first phase occurs immediately after supplying the ID and credentials, and results in ~3 calls to the /login endpoint and a similarly small number of calls to the csv endpoints. This phase ends when the preview window opens with all the csv files in the navigation table unchecked. The names of these tables are derived from specific cells in the tables, so they must have been loaded into memory for the names to have been pulled out (and the endpoints send the csvs in full). At this point I would expect that all the necessary requests have been made, and that no further API calls are necessary (otherwise how could the tables be dynamically named based off of data inside the table?).

Now when I start checking all the boxes for the tables, phase 2 starts, and calls start being made again for what I'm assuming is preview data. Why does this have to happen, shouldn't the data already be stored/cached?

Phase 2 ends and Phase 3 starts when I click the load button. Doing this causes all the endpoints to start being hit again, accounting for a further 4-5 calls per CSV which finally overwhelms the /login endpoint and the load errors out. 

I managed to eek out an error message from Phase 3 that looks somethin like this

Formulas:

section Section1;

shared _csv1 = let
    Source = Connector.Contents("471516961986314240"),
    _csv11 = Source{[Key="_csv1"]}[Data]
in
    _csv11;

shared _csv2 = let
    Source = Connector.Contents("471516961986314240"),
    _csv21 = Source{[Key="_csv2"]}[Data]
in
    _csv21;

shared _csv3 = let
    Source = Connector.Contents("471516961986314240"),
    _csv31 = Source{[Key="_csv3"]}[Data]
in
    _csv31;
...

This error message contains a shared _csvX = let... once for each check box that I checked. What it looks like it's doing internally is calling the connector (which gets all the csv's) once for every csv (which again, makes no sense because the csv's have already been imported). So in total if the import generates 7 output csv's, this explicitly fetches 7*7 csv's, or 49. This relationship of ~n^2 coincides with the number of login attempts that I am seeing. I'm assuming that this is not intended behaviour, and that the Source = Connector.Contents("...") should be cached from the initial import that generated the navtable that I interacted with, or that at least only one more call to it should be made. It's impossible to find out what is and isn't being cached, and the lazy loading evaluation model means that even if I write Binarry.Buffer(), which would result in an in memory copy being kept and subsequent uses of the output being fetched from cache rather than from calling Web.Contents, the call to Binary.Buffer itself seems to be deffered until the contents are actually needed, which means that a call to it can be outside of the initial scope where it would have been useful to cache the information.

I think that this is essentially what is going wrong:

let
    rand = (List.Random(2)),
    x = rand,
    y = rand,
    z = rand
in
    {x{0},y{0},z{0}}

In this example, you would expect that rand gets evaluated into a list containing two random numbers, and x,y,z would then reference that List. The output would then be a list containing the same three numbers. 

This is not the case. The evaluation of List.Random(2) is put off until it is absolutely needed, and the evaluation model generates something like this as the actual output {(List.Random(2)){0}, (List.Random(2)){0}, (List.Random(2)){0}} For a dynamic call like this you end up with three diferent numbers.

If you add a buffer and change the code to 

let
    rand = List.Buffer(List.Random(2)),//Buffer this now
    x = rand,
    y = rand,
    z = rand
in
    {x{0},y{0},z{0}}

Then the output generated is a list containing three identical numbers. 

It seems as if in my connector buffering Web.Contents does not have the same effect, and I'm left with a waterfall of API calls that should not need to be made.

Any help would be greatly appreciated.

Hi everyone, this is my first post so i'm not too sure if i'm in the right page. 

My dashboard currently uses direct query to filter through 2 scenarios (out of a list of a 1000+) and obtain rather granular data that spans into the thousands of rows. It seems that applying a hard filter in power query to limit the query on power query to a list of 2 scenarios, there is a substantial speed up of about 10x. 

Basically i'm trying to calcaulte percentiles of data that is halfhourly as a measure, but it seems that DAX/Powerquery iterates through all scenarios on a halfhourly level to check if it belongs in that scenario, explaining why the query is so slow. 

I'm having no issues with the Zendesk data source.  It's a fantastic help.  I just need to know if it is using the Zendesk Search API and if the upcoming curbs on that data is going to affect this: 

Message from Zendesk attached. 

I have a SharePoint list that has records as follows;

1. For each location, there are multiple records as defined by 2. below.
2. For each month of the year, there is a single record per location.

In each record (location and month) there is a target goal.

In another SharePoint list, there is a single record for

1. Each day of the year
2. Each location.

The sum of the values in the daily records, by location, is stored in the location record by month.  So location A for month 6 has the sum of the records for location A for all the days in month 6.  Addotionally, the monthly records have a goal value for that location. 

I have successfully gotten the daily records to display for a given month as the sum of the daily values on the vertical axis and the location names on the bottom axis.  What I have been unable to sort out is how to display, either as side by side bar or a line chart over the bars what the monthly goals are that correspond to that location.

If instead of a single chart I create two and drag the sales goals and location names to the second one, I get a display that is just like the daily one which is what I want.  But I need to combine them on one chart for comparison.

Here's what's puzzling.  I can drag the month onto the monthly chart and then filter it by month number, and the monthly goals figures appear.  If I do the same thing on the combined chart, month filtering shows blank and no goals appear.  It's as though there were no records.

I thoght it might be in the query types but since it works standalone i have assumed that is not the case.  Could it be the filtering on the combined graph?  but why whould selecting something to include/exlude on that list affect the other?

Hi folks

I'm trying to implement a range of data governance workflows in an organizational context in Power BI desktop and service. I want to programatically capture transforms that occur on data, specifcally aiming for column and ID/code text label transforms and want to ultimately manage these in the CDM folders on azure storage as entity mapping terms. It's to support a controlled taxonomy of terms in essence. 

Even if some parts require some intervention. I can do it all manually but that simply won't be sustainable.

This is a long multipart use case ( a thought dump). I don't expect a full answer, but a simple... your dreaming, yes it's possible, use this approach, or look at this tool it does similar things would be so so appreciated !

Request: I'm hoping for some inspiration on how to automate the extraction of a set of data transformations in the Power BI stack

Goal:  Automatically (as much as possible) capture Power Query function transformation actions such as column or text pattern replacements, that occur on original SQL/csv column names or text strings e.g. ID's or short codes which are changed to user labels for final BI reports. 

Tech requirements: It's in a government agency built on .NET, Azure and centered on Power BI. So tools in those frameworks are key dependencies.

For background.... I use RDF https://www.w3.org/RDF/ ,  SKOS https://www.w3.org/TR/skos-reference/#labels and an enterprise level vocabularby manager to do all this with other clients. But this specific client does not have the option to go down that level of investment.... but they still want data governance ( he bangs head against wall  )

My envisaged workflow is:

1. BI report authors publish a report and/or dataset to Power BI service

2. a scheduled event .... a  dataflow Power Query script or R script or an Azure service or powershell scriot or cron job or something else ! ... runs and performs introspection on a) each table in the loaded data model and/or b) dumps a copy of the M-code for parsing. 

3. a script extracts key metadata about a set of transformations that I define. e.g. I want to capture all replace.text function calls on columns or text patterns so that textOriginal and textFinal are captured (plus some other metadata). Hence at this stage I see that I need to be able to parse the M-query itself, not just get the "final" table headers or cell contents

3. create, append or export this information to a persisted file or dataset. e.g.

        3a) directly create/append data to a new power bi table: or

         3b. if an intermediate step is required, a csv, for example in sharepoint, then run a script to ingest this into a             Power BI dataset/flow

4. export/serialize the final table/data file in the required CDM folder structure ( I woud use python most likely)

5. load the custom entity mapping to the CDM folder ( python)

So what have I acheived so far ....

6. I've experimented with some R sctipts to export a table as csv - works fine but only gets end state column labels or cell content

7. I've set up a bunch of regex functions that extract all the text replace functions and NameOriginal, NameFinal on columns or text strings from the M-Code.

Next steps/ Gaps

8. work out how to export this data to a sharepoint folder

9. how to run "something" automatically to extract the M-Code to somewhere e.g. sharepoint power bi storage folder? 

10. how to run the regex parsing on this file

       10a) I'm assume Powershell is the line here. And could be scheduled ?

Can powershell tasks be automated on Azure ? i guess yest

        10b) it appears that R can now use regex and so I wonder if R scripts can scheduled in Power BI service in anyway? 

11. Key issue..... I haven't found a likely candidate to then run a scheduled/event driven process to create/append this data to a Power BI dataset or report. e.g. M code that can scheduled to create stage table/data

At this point I'm expecting to have a set of scripts run on an azure server, likely python flask to orchestrate things. 

Ideas/Questions:

I've connected to the tabular model of the power bi desktop files a lot.

I know once in the tabular model or .bim files all dax and I think M-code can be obtained. So I could then parse these. But it's all manual so far as I've seen via Daxstudio or excel.

Any ideas on automating this ?

Can the same be acheived to Power BI service ? or Azure gen2 storage ?

Any ideas or feedback most welcome !

thanks in advance

Simon

I have the two following tables (each has hundreds of thousands of observations):

Table 1 (example) 

Trans_ID   Cust_ID  Date_of_trans  Reject_Flag

1                    10              6/28                 0

2                    11              6/27                 1

3                    13              6/26                 0  

4                    10              6/26                 1

...

Table 2 (example)

Cust_ID     Date_of_first_rejection

11                           6/27

10                           6/26

15                           6/13

...

The two tables are connected via the customer ID. In table one the Trans_ID is the unique identifier and in Table 2 the cust_ID is the unique identifier. There are many instances of the same customer in table one since they can have many transactions on many different dates. What I need to do is to create two new columns in table 2, one which counts the number of transactions (Trans_ID) that a customer had before the date of their first rejection and another column that does the same but for transactions after the date of their first hold. 

I have been trying to use calculate and count functions but cant get the right numbers. Is this even possible to do in power bi? 

I have the two following tables (each has hundreds of thousands of observations):

Table 1 (example) 

Trans_ID   Cust_ID  Date_of_trans  Reject_Flag

1                    10              6/28                 0

2                    11              6/27                 1

3                    13              6/26                 0  

4                    10              6/26                 1

...

Table 2 (example)

Cust_ID     Date_of_first_rejection

11                           6/27

10                           6/26

15                           6/13

...

The two tables are connected via the customer ID. In table one the Trans_ID is the unique identifier and in Table 2 the cust_ID is the unique identifier. There are many instances of the same customer in table one since they can have many transactions on many different dates. What I need to do is to create two new columns in table 2, one which counts the number of transactions (Trans_ID) that a customer had before the date of their first rejection and another column that does the same but for transactions after the date of their first hold. 

I have been trying to use calculate and count functions but cant get the right numbers. Is this even possible to do in power bi? 

Data ModelData table - report

I have the data model above with 2 fact tables.  One for Forecast and one for actuals.  They are at different granularities.  Forecasts are for the month - but done every week.  Actuals are added for the month when the month is complete.  I want a data table report as shown above actuals to replace forecast once they are available.  I am not xcertain if I need a different data-model or I need to de-normalize to a single fact table or DAX measures or DAX calculated columns.

Hello everyone, I hope you guys could help me.

Im working on Power BI Desktop, I connect to a SQL instance and data loads into Power Query, everything looks fine.

When I try to save the query got the following message: