Hello,
As per recent update from Azure monitor, from nov 1,2018 onwards, the log data will be pushed into blob in json lines format.
We use powerBI to load and analyse those json data from azure blob.
Till now, we have been using json.Document method to parse the current json format.
But now, the json.Document method doesn't seem to be parsing the json line format. It throws 'We found extra characters at the end of JSON input'.
So, I happened to find from other posts that to solve this, I had to parse it as a text file and then convert it to json.
-------------------------------------------------------------------------------------------------------------
let
FxFixJson = (jsonfilepathname as text) =>
let
Source = AzureStorage.Blobs("https://MYBLOBSTOREACCT.blob.core.windows.net/"),
#"Expanded Data" = Table.ExpandTableColumn(Source, "Data", {"Content", "Name"}, {"Data.Content", "Data.Name"}),
#"Filtered Rows" = Table.SelectRows(#"Expanded Data", each ([Data.Name] = jsonfilepathname)),
#"Data Content" = #"Filtered Rows"{0}[Data.Content],
#"Imported Text" = Table.FromColumns({Lines.FromBinary(#"Data Content",null,null,1252)}),
#"Parsed JSON" = Table.TransformColumns(#"Imported Text",{},Json.Document),
#"Expanded Column1" = Table.ExpandRecordColumn(#"Parsed JSON", "Column1", {"deviceId", "messageId"}, {"deviceId", "messageId"})
in
#"Expanded Column1"
in
FxFixJson
-------------------------------------------------------------------------------------------------------------
My concern is that my current parsing logic should support both current json format and upcoming json lines format.
I dont see anything to distinguish between both the formats (for eg, the extension is same for both the formats).
I am wondering if this has to be a trial and error method.
Like Parse with json.Document and if it fails, then parse it as text and convert to json and it might impact performance.
Is there a cleaner way of parsing of both the json formats? I dont see json lines format is natively supported by powerBI yet.