Professional Documents
Culture Documents
Power Query - Reference Book PDF
Power Query - Reference Book PDF
Table of Contents
Part 1
Power Query Basics
Power Query Editor ........................................................................................................................ 3
Overview ................................................................................................................................................ 3
Updating the Source Location/Name .................................................................................................... 4
Deleting a Query Step ............................................................................................................................ 5
Transform Versus Add Column .............................................................................................................. 6
Split Text by Delimiter ........................................................................................................................... 7
Automatic Data Typing .......................................................................................................................... 7
Rename a Column.................................................................................................................................. 8
Removing (Deleting) a Column .............................................................................................................. 8
Perform Quick Mathematical Operations ............................................................................................. 9
Refreshing Source Data During Query Composition ............................................................................. 9
Quick Insights on Data – Column Profiling, Data Quality & Distribution ......................................... 10
Displaying a Complete List When Filtering .......................................................................................... 10
Displaying Column Profile Information ............................................................................................... 11
Displaying a Complete List for Selected Column ................................................................................. 12
Column Distribution............................................................................................................................. 12
Column Quality .................................................................................................................................... 13
Using Monospaced Fonts .................................................................................................................... 13
Discover the Total Number of Rows .................................................................................................... 14
Formula Bar – Applied Steps & M Code ......................................................................................... 15
Renaming Applied Steps ...................................................................................................................... 15
Other Applied Steps Options ............................................................................................................... 15
Adding Documentation to Applied Steps ............................................................................................ 16
Power Query Formula Bar ................................................................................................................... 16
Advanced Editor................................................................................................................................... 17
Close & Load Destinations ............................................................................................................. 18
Working with Existing Queries............................................................................................................. 18
Creating a Data Connection ................................................................................................................. 19
Changing the Default Load Destination ............................................................................................... 19
“Saving” Your Queries ......................................................................................................................... 20
Microsoft Power Query enables Excel users to import data from a wide range of data sources
Excel into Excel for analytics and visualizations.
Starting with Excel 2016, Power Query capabilities are natively integrated and can be
found under the “Get & Transform” section of the Data tab in the Excel Desktop
ribbon.
Excel 2010 and 2013 users can also leverage Power Query by installing the Microsoft
Power Query for Excel add-in.
Microsoft Power Query enables data analysts and report authors to connect and transform data
Power BI as part of creating Power BI reports using Power BI Desktop.
Microsoft SQL Business Intelligence Developers can create Azure Analysis Services and SQL Server
Server Data Analysis Services tabular models using SQL Server Data Tools for Visual Studio. Within
Tools for this experience, users can leverage Power Query to access and reshape data as part
Visual Studio of defining tabular models.
Common Common Data Service lets you securely store and manage data that's used by
Data Service business applications. Data within Common Data Service is stored within a set of
entities. An entity is a set of records used to store data, similar to how a table stores
data within a database.
Common Data Service includes a base set of standard entities that cover typical
scenarios, but you can also create custom entities specific to your organization and
populate them with data using Power Query. App makers can then use Power Apps
to build rich applications using this data.
Power Query
Basics
Power Query Power Query Basics
Overview
You can open the Power Query editor from the Excel’s Data tab, by going to Data > Get Data > Launch
Power Query Editor.
You can also automatically launch the editor by importing data into Power Query. Simply select the
appropriate option from the Get & Transform Data Group on the Ribbon.
The advantage of the gear icon is that you are presented with a user-friendly way to browse to the new
location, thus avoiding typographic errors.
NOTE: Any time you see a gear icon next to a step, it means that the step has parameters that can be
adjusted.
These adjustments are usually presented in user-friendly dialog boxes, eliminating the need to write
complex formulas and statements.
Many of the most common table transformation tools are in a single, easy to navigate list.
• Transform will replace the original data (column) with the transformed version of the data.
• Add Column will create a new column to hold the results of the transformation.
From here, you can select the delimiter and whether you want to split at every occurrence or just the
first or last occurrence.
File-Level Deactivation
File → Options & Settings → Query Options → Current Workbook → Data Load → Type Detection →
Detect column types and headers for unstructured sources.
Program-Level Deactivation
File → Options & Settings → Query Options → Global → Data Load → Type Detection → Never detect
column types and headers for unstructured sources.
Rename a Column
Columns can be renamed in several ways:
NOTE: If you have many columns to delete and only a few columns to preserve, it may be more efficient
to select the columns you wish to keep and use the Remove Other Columns feature to delete the
unwanted columns.
This is a superior way of removing unwanted columns, as opposed to selecting unwanted columns and
deleting them.
Later, if you change your mind, you can click the gear icon and reselect a column to add it back into
the data set.
PRO TIP: If you perform the same operation multiple times in a row, Power Query will consolidate all like
steps into a single step in the Applied Steps list. This means, if you delete 5 columns then rename 5
columns, you will only see 2 steps in the Applied Steps list.
Conversely, if you were to delete a column, then rename a column and repeat this 4 more times, you will
see 10 steps in the Applied Steps list.
NOTE: Be mindful of the order at which columns are selected. This is important when performing
operations such as division (ex: Total Quantity, versus Quantity Total).
This will read the fill data set and display a complete list of items for that column.
When you select a column, statistical and distribution information is displayed for the first 1,000 rows of
data.
• Count of rows
• Empty cells
• Min and max values
NOTE: System performance may suffer when loading the entire data set. Consider loading all data to
capture some quick insights into the data, but return the captured set to the first 1,000 to improve
performance.
Column Distribution
To view the number of distinct versus unique values in each column, click View (tab) → Data Preview
(group) → Column Distribution.
This provides a miniature column chart at the top of each column to display the relationship between
distinct and unique values.
NOTE: These mini-charts are based on the setting for “first 1,000” or “all records”.
12 | P a g e Copyright © 2020 Leila Gharani
Power Query Power Query Basics
Column Quality
To display the percentage of valid data vs. errors vs. empty cells, click View (tab) → Data Preview
(group) → Column Quality.
The column headings now display percentages of each category for the column.
NOTE: These percentages are based on the setting for “first 1,000” or “all records”.
This may make the text easier to read, especially when working with columns containing large numbers.
This will reduce the entire data set to a single value that represents the total number of rows.
This is often performed to satisfy a casual curiosity. The step is often deleted once the question has
been answered.
Similar to Excel’s Formula Bar, we can see the “behind the scenes” instructions for the selected step.
These instructions are written in the Power Query ‘M Language’ (Data Mashup Language).
This is especially useful for steps that do not have the gear icon . We can see in the Formula Bar what
is occurring and potentially perform a direct-edit on the instruction.
For steps that have more instructions than can be viewed on a single line of the Formula Bar, click the
small down-arrow to the right of the Formula Bar to expand the height of the Formula Bar.
Advanced Editor
Although you can see each step individually in the Formula Bar, it’s difficult to remember what you saw
in one step when comparing to another step.
The entire set of “M” code can be displayed by opening the Advanced Editor.
To open the Advanced Editor, select Home (tab) → Query (group) → Advanced Editor or by selecting
View (tab) → Advanced (group) → Advanced Editor.
This will display the entire instruction set of “M” code for the query.
This will open the Import Data dialog box where you can select a new destination for the query output.
Connection Only queries greatly reduce file size. This is because the data remains at the source and a
copy is not stored locally in the Excel file.
Note: Selecting a custom load but NOT selecting a worksheet or Data Model will establish a Connection
Only query.
Copyright © 2020 Leila Gharani 19 | P a g e
Power Query Power Query Basics
Refresh Options
Each query has a set of refresh options that will allow for a certain degree of automation.
Select Data (tab) → Queries & Connections (group) → Properties to open the Properties dialog box.
• Enable background refresh – this allows you to continue working in Excel as the query is being
refreshed. Disabling this feature is useful when the results of the query are needed for another
operation (like a macro), and you don’t want the following step to run prematurely.
• Refresh every N minutes – this schedules a refresh at a defined interval (in minutes). The
workbook needs to be open for this option to work.
• Refresh data when opening the file – self-explanatory.
Important
Power Query
Tips & Tricks
Power Query Important Power Query Tips & Tricks
5. Rename the query output to something like “SalesReport”. This will become the name of the
output table on the Excel worksheet.
• Refresh
• Edit
• Delete
• Duplicate / Reference
• Merge / Append
These features are also available by way of right-clicking on a query in the Queries & Connections panel.
Useful Table Settings
Two table settings that are worth exploring:
• Table Design (tab) → External Table Data (group) → Properties → Adjust column width.
• Table Design (tab) → External Table Data (group) → Properties → Preserve cell formatting.
• Define a range larger than your data to account for future data addition
• Create a dynamic named range by using formulas in the Name Manager to calculate the
height/width of the data range
NOTE: If you are using a Named Range to define your data, select the Named Range before sending it to
Power Query. This prevents Power Query from selecting what it thinks is your data, which may not
match your Named Range definition. This will also prevent Power Query from upgrading your data into
an Excel Table.
The advantage of the gear icon is that you are presented with a user-friendly way to browse to the new
location, thus avoiding typographic errors.
NOTE: Remember that Power Query is case-sensitive. Ensure you type the names as they appear in the
source material.
3. Update the incorrect table name to the new table name by editing the reference in the Formula
Bar
Unfortunately, we do not have any buttons to lead us to the offending step, nor any gear icon to open a
user-friendly correction window. This is why having the Formula Bar visible is so important.
5. Update the entry in the Formula Bar to display the correct column heading name.
NOTE: Remember that Power Query is case-sensitive. Ensure you type the names as they appear in the
source material.
Make the necessary changes in the editor and click when finished.
IMPORTANT: Remember that Power Query does not posses the Undo feature. Consider creating a
backup of the file/query before editing. This way, if you make a mistake during the direct editing of the
query, you can restore your query to its original state.
Date Dates from 1900 to 9999 are supported. Stored as whole numbers.
If we examine the number more closely, we can see that the change in data type has altered the value.
This precision cannot be restored once the alteration has taken place. It would be necessary to delete
the step that applied the change to regain the original precision.
Increasing Decimal Precision
If you import whole numbers into Power Query and later add data that has fractional precision, you will
lose the fractions.
This is because the original numbers are data typed as Whole Numbers. Future values will be converted
to Whole Numbers, thus losing their fractional precision.
If you believe that you may encounter fractions in your data, it’s a good idea to set the data type to
Decimal as a way of “future-proofing” your query.
Currency-based Values
When values are data typed as Currency in Power Query, the results are displayed in the table with 2-
decimal place precision, but the underlying values are stored and calculated at 4-decimal place
precision.
This is an indication that the cell is truly empty; no spaces or hidden characters
Shortcut Task
F2 Edit the name of a column or query
Arrow keys (L & R) Navigate left or right through columns
CTRL key Select multiple, non-contiguous columns
Shift key Select contiguous columns
CTRL-A Select ALL columns
CTRL-Space Select the entire column of a selected cell
ALT (while opening Excel) Open a second, unrelated instance of Excel
This is a superior way of removing unwanted columns, as opposed to selecting unwanted columns and
deleting them.
Later, if you change your mind, you can click the gear icon and reselect a column to add it back into
the data set.
Detecting Data Types
It’s not uncommon to delete the automatically applied type detection step from the Applied Steps list in
Power Query.
When it comes time to perform data type detection, you can select the column(s) you want to data type
and click Transform (tab) → Any Column (group) → Detect Data Type.
Power Query creates a new query that extracts the complete record for all rows of the results query that
contain errors. Each row is prefaced with a row number indicating the record position in the source
data.
If you click to the right of the error message, Power Query displays a detailed explanation of the error.
In the Replace Errors dialog box, enter the value you wish to replace the errors with, such as 0 (zero).
• Select a query from the Queries & Connections panel, then select Query (tab) → Reuse (group)
→ Duplicate.
• Select a query from the Queries list, then select Home (tab) → Query (group) → Manage →
Duplicate.
• Select a query from the Queries & Connections panel, then select Query (tab) → Reuse (group)
→ Reference.
• Select a query from the Queries list, then select Home (tab) → Query (group) → Manage →
Reference.
If the output query were to be deleted, the dependent queries would no longer have a source with
which to draw data.
To visualize these dependencies in Power Query, select View (tab) → Dependencies (group) → Query
Dependencies.
The following Query Dependencies dialog box will graphically display the queries, the relationship
hierarchy, and their load destinations.
Delete a Query
If you are in Excel:
• From the Queries & Connections panel, right-click a query and select Delete or press the Delete
key on the keyboard.
• Select a query from the Queries & Connections panel, then select Query (tab) → Edit (group) →
Delete.
• Select a query from the Queries list, then select Home (tab) → Query (group) → Manage →
Delete.
NOTE: If you delete a query, the output of the query remains in the workbook.
5. Click Close.
All the queries will have been removed while leaving the query results intact.
46 | P a g e Copyright © 2020 Leila Gharani
Power Query Important Power Query Tips & Tricks
NOTE: The .ODC file is an eXtensible Markup Language (XML) file that can be viewed in any text editor.
5. In the Existing Connections dialog box, select the applicable query from the list.
NOTE: If the query is being transferred to a different computer, you can click “Browse for more…” and
manually locate the .ODC file.
Organizing Queries
If you are working in a workbook with many queries (perhaps dozens), you can organize your queries
into Query Groups based on similar purpose, function, or phase.
Create a Query Group
1. From the Queries & Connections panel, click a query then hold the CTRL key and select one or
more other queries.
2. Right-click one of the selected queries and click Move to Group.
3. You can select one of the existing groups or create a new group by selecting “New Group…” If
you are creating a new group, give the group a name and consider adding in the Description field
some explanation as to the purpose of the queries in this group.
The results can be expanded or collapsed to create a cleaner appearance to the queries list.
Helpful
Power Query
Transformations
Power Query Helpful Power Query Transformations
3. Select the Department/Position column and click Home (tab) -> Transform (group) -> Split
Column -> By Delimiter.
4. In the Split Column by Delimiter dialog box, select Custom and use a forward slash (/) as the
delimiter.
5. Click OK.
NOTES:
• Once the split has been performed, another “Changed Type” step is added. This is to reassess
the newly split data for possible new data types. We could delete this step as it is not creating
anything we didn’t already possess.
• Technically, we could have used a “ / “ (space, forward slash, space) as the custom delimiter.
This would have allowed us to eliminate the upcoming “Trim” step.
Removing the Leading & Trailing Spaces
If you select to the right of one of the Department names, you will be presented with a preview window
at the bottom on the screen.
If you then click to the right of the data in the preview window you will notice that the cursor does not
rest directly next to the last letter; there is a trailing space.
We can also discover that there is a leading space before the Position in the second column.
I prefer using the “space – forward slash – space” approach as it reduces the query by a step.
NOTE: We will deal with the column headings at the end. For now, we can leave them a bit messy.
Select the Unique Code column and click Transform (tab) -> Text Column (group) -> Split Column -> By
Non-Digit to Digit.
This separates the Last Name from the Employee ID/First Name.
Next, select the column with Employee ID and Last Name and click Transform (tab) -> Text Column
(group) -> Split Column -> By Digit to Non-Digit.
Select the column that contains Employee IDs and click Transform (tab) -> Text Column (group) ->
Format -> Add Prefix.
In the Prefix dialog box, enter a Value of “E-“ and click OK.
There is a wealth of options sitting withing the Extract feature in Power Query.
We don’t want to transform what we have into a new set of data; we want to leave the original Full
Name column while adding additional columns for First Name and Last Name.
For this operation, we will use the version of Extract located on the Add Column ribbon.
Extracting First Names
Select the Full Name column and click Add Column (tab) -> From Text (group) -> Extract -> Text Before
Delimiter.
In the Text Before Delimiter dialog box, enter a space in the Delimiter field and click OK.
We are presented with a new column that contains all text up to the first space in the Full Name text.
In the Text After Delimiter dialog box, enter a space in the Delimiter field. Expand the Advanced
Options and set the Scan for the Delimiter option to “From the end of the input” and click OK.
Starting from the end and “looking” backward is necessary because of the existence of middle names in
some of our records.
If we began our search for a space from left-to-right, we would stop before a middle name and extract
the middle and last names.
We are presented with a new column that contains all text after the last space in the Full Name text.
Finishing Touches
• We no longer require the original column of Full Names, so we can select the Full Name column
and press Delete.
• Rename the column headings with more meaningful names.
You can load the results to a table on a new sheet, or an existing sheet.
All three requirements have been satisfied and we didn’t have to write a single formula to get the job
done.
• To replace the existing content, select Transform (tab) → Text Column (group) → Merge
Columns or right-click the selected columns and select Merge Columns.
• To create additional content, select Add Column (tab) → Text Column (group) → Merge
Columns.
In the Merge Columns dialog box, select the separator (delimiter) you wish to use to separate each
column’s data within the result.
In the New Column Name field, type the name you wish to use for the heating of the new column.
Potential Issue
Because we were including middle names in the above example, names that do not contain a middle
name are padded with two spaces between the first and last name.
Click in the desired column and select Home (tab) → Transform (group) → Replace Values. In the
Replace Value dialog box, enter a “space” character in the Value to Find field and a “null” (no quotes) in
the Replace With field.
Replacing Data
Do not forget; Power Query is CASE-SENSITIVE. If you were to replace data by keyword, you need to
ensure that all versions of the desired data match the defined casing in the replacement procedure.
To replace text, select the needed column(s) and click Transform (tab) → Any Column (group) →
Replace Values. In the Replace Value dialog box, enter the text you are searching for in the Value to
Find field and the replacement text in the Replace With field.
If you click “Advanced Options” in the Replace Values dialog box, you can activate features such as:
NOTE: The Advanced Options do not contain an option for case-insensitivity. If you need to replace it in
a case-insensitive fashion, you will need to apply a formatting transformation, before the replacement,
that will standardize the casing of the text.
Standardizing Casing
To standardize the casing of data in a column, select the needed column(s) and click Transform (tab) →
Text Column (group) → Format → then the desired casing style (i.e. “Capitalize Each Word”.)
Simple Sorts
To perform a simple sort (defined as sorting the table by a single column), select the desired column and
click Home (tab) → Sort (group) → Sort Ascending/Sort Descending.
You can also click the Sort/Filter dropdown arrow at the top of the column and select Sort Ascending or
Sort Descending.
Look closely and you can see an “up arrow” indicator in the Sort/Filter button to indicate that the
column is sorted with an ascending option.
Multi-Level Sorts
We have already covered creating a simple sort, so we will use this existing sort and build off it.
The original sort (ascending by Department) will be our Primary sort. Within each Department, we
want to perform a Secondary sort by Position. Perform an ascending sort on the Position column.
Finally, we can sort within each Position by Salary in a descending strategy. This will be our Tertiary
sort. Perform a descending sort by Salary.
When you have more than one level in the sort logic, you receive numeric indicators next to the
Sort/Filter buttons to indicate Primary-Secondary-Tertiary order. The numbers are very small, so you
may need to look closely.
This creates a column of numbers that we can use as markers to denote the original row number
locations.
Once our transformations are complete, we can perform a simple sort on the Index Column to return
the list to its original order. The Index Column can then be removed.
70 | P a g e Copyright © 2020 Leila Gharani
Power Query Helpful Power Query Transformations
NOTE: You can also KEEP only the duplicate rows by selecting Home (tab) → Reduce Rows (group) →
Keep Rows → Keep Duplicates.
REMEMBER: Power Query is case-sensitive. If you have the same word occurring in your column, but in
differing cases, they will be treated as separate, unique items. Consider standardizing your casing before
the Remove Duplicates operation.
In the Round dialog box, in the Decimal Places field, enter the value that denotes the level of decimal
precision and click OK.
NOTE: The following features are only available on the Add Column tab. The options are grayed-out on
the Transform tab. If you do not wish to retain the original columns post-aggregation, you will need to
manually remove the original columns.
If you wish to add the values of two columns to produce a third aggregation column (i.e. add, subtract,
multiply, divide, max, min, median, etc.), Select the two columns and click Transform (tab) → Number
Column (group) → then the appropriate aggregation option.
NOTE: When performing certain mathematical operations (ex: division), the order you select the columns
determines certain argument types, like numerator and denominator.
• Use the Add Column (tab) → From Number (group)→ Statistics → Sum feature instead of the
Add Column (tab) → From Number (group)→ Standard → Add feature.
This behavior is the same when performing other arithmetic operations, like multiplication and
subtraction.
Now you can enter the parameters for the text to be discovered, such as “all rows that contain the word
‘Manager’”.
Notice in the above image we are searching for any row that contains the lowercase and proper-case
versions of the word “Manager”. If your text casing had been standardized before this step, you could
list only a single argument for “Manager” in the corresponding casing style.
The logic fails our needs because Power Query applies the AND operator for Position to the
“Department = Marketing” requirement, not the “Department = Management” requirement.
To get Power Query to apply the Position requirement to both Department requirements, we need to
perform two separate filter operations; one for both departments, then another for the Position.
NOTE: If you are feeling adventurous, you can modify the M-Code in the Formula Bar to logically group
the two Department requirements as a single thought, then apply the Position requirement. This is
accomplished by placing a set of parentheses around the two Department requirements.
…becomes…
Look closely for the opening and closing parentheses in the second version of the instruction.
If we never add a new Position with the word “Assistant” for any position other than what we currently
have, we will be okay. This is because of the way Power Query wrote the filter step.
The query is NOT looking for Positions that contain the word “Assistant”, it is instead looking for four
specific Positions that contain the word “Assistant”.
Instead of checkmark selecting the four Positions based on the search results, we should have created a
different search, something like “Text that Contains… ‘Assistant’ ”.
This will account for future entries that have the word “Assistant” but may also contain words that were
not in the original list.
The Changed Type step is defining a data type for each column. Nothing wrong with that… until later.
We apply some additional transformations:
This is because the Changed Type step is attempting to set the data type for a column named Employee,
which doesn’t exist any longer.
Program-level Deactivation
1. Select File → Options & Settings → Query Options.
2. In the Query Options dialog box, under the Global category – Data Load – Type Detection, select
the option that best suits your needs for most projects.
= Table.SelectColumns(Source,{"Department", "Position"})
Suppose the removal step was constructed with opposing logic. In other words, instead of telling the
query what columns to keep, we tell the query what columns to remove.
= Table.RemoveColumns(Source,{"Employee", "Salary"})
We run the same risk as before where a user renames a column in the source data that no longer
matches the column name in the query step.
Powerful
Power Query
Transformations
Power Query Powerful Power Query Transformations
The purpose of Columns From Example is to use an example (or series of examples) to restructure the
data into a new arrangement based on an established pattern.
When launched, Power Query provides an empty test column by which examples are created.
For our example, suppose we need to extract the first 6 characters from the last name and the first
character from the first name. These need to be concatenated and cased in a Proper Case style (first
letter capitalized).
Although not a requirement, users typically start at the top of the test column and begin creating “the
version they need” via manually typing.
After the user hits ENTER, if the pattern is obvious enough, a faded suggestion list is presented.
The feature isn’t perfect; certain patterns elude the feature’s ability to deduce. It is, however, a great
“first attempt” when restructuring data based on a pattern.
This gives us an appreciation of what we are NOT having to develop when writing queries.
2. Capture the first six characters from the last names (Step 1 results) by selecting Transform (tab)
→ Text Column (group) → Extract → First Characters and set the option to 6.
3. Apply the Proper Case format (the first letter capitalized, remaining letters lowercase) by
selecting Transform (tab) → Text Column (group) → Format → Capitalize Each Word.
4. Capture the first character of the first name by selecting the original data column and clicking
Transform (tab) → Text Column (group) → Extract → First Characters and set the option to 1.
5. Merge the results by selecting the column of six characters of the last name, hold CTRL, and
select the first initial column. Next, select Transform (tab) → Text Column (group) → Merge
Columns. There will be no separator used and the new column name will be “LoginID”.
Both methods achieve the same result. Pick the method you are most comfortable with and which you
have the most faith in succeeding.
In the Add Conditional Column dialog box, give the result a name (such as “Group”), and define each
level in the logic hierarchy.
Since each condition is looking at the same column (“Salary”) and using the same logic (“is greater than
or equal to”), we only need to define the Values and the Output.
NOTE: Define the logic from the highest value to the lowest; in other words, the most restrictive to the
least restrictive.
The Else field will act as a “catch-all” for any value that does not meet any of the above criteria.
NOTE: Be aware that the Group By feature REPLACES your original data with the grouped version of the
data. If you need to retain the original data source, consider making a copy of the original query and
performing the Group By on the copy.
If you select a single column, you will be presented with the Basic view of the Group By feature.
If you select multiple columns, you will be presented with the Advanced view of the Group By feature.
• For each sub-group you need to create, click the Add Grouping button.
• For each additional aggregation, click the Add Aggregation button.
• Calculate the salary deviation for each person using the average salary of their department as a
basis
If we click OK, all the data will be replaced with the aggregated results.
Each of the “Table” hyperlinks is a complete table of information, all stored within a single cell. Imagine
every record that pertains to the Department – Sales has been “zipped” into a single object called
“Table” and stored in a single cell.
If you click a “Table” hyperlink, you will extract all the related records into a new table that replaces the
current table.
If you click to the side of a “Table” hyperlink, you can see a preview of the contents of the nested table.
You can extract all the records for each nested table by clicking the Expand Tables button in the heading
of the column of nested tables.
From here, you can select which columns you wish to extract and attach to the existing table.
Since we already possess Department and we don’t require Position, we will select Employee and
Salary. We will not use the column prefixing feature at this time.
The original column information will be repeated for each expanded row.
Final Touches
• Rename the Subtraction column to Deviation.
• Remove the Salary and Average Salary columns.
• Sort the Deviation column from Largest to Smallest (Descending)
• Send the report to Excel as a Table.
NOTE: Unpivoting is NOT the same as Transposing. When you Transpose, you “flip” the categories
between their row/column positions but you are still left with a cross-tabular structure.
Unpivot Example
We have the following dataset:
• Select the “anchor” column and click Transform (tab) → Any Column (group) → Unpivot Other
Columns.
• Select all columns except the “anchor” column and click Transform (tab) → Any Column (group)
→ Unpivot Columns.
NOTE: These features can also be accessed by right-clicking the heading of the selected column(s).
In this case, you will get the same results with either method.
To unpivot this into a tabular structure, in Power Query, select the “Product / Sales” column and click
Transform (tab) → Any Column (group) → Unpivot Other Columns.
There are two ways to deal with these extra, unwanted rows:
• In the results, filter the “Product / Sales” column to not include “Grand Total”.
• Before the unpivot step, filter the “Product / Sales” column to not include “Grand Total”.
If we perform a typical unpivot operation on the table, we get the results as expected.
The problem occurs when we bring in another month (ex. July)to the data.
When we refresh the query, everything looks correct. Looking at a deeper level, we have a problem.
This is because the step that defined the data types were performed before the unpivot step. The
Changed Type step was hard-coded to look for months between “January” and “June”.
Since “July” did not exist when this step was defined, it will not be accounted for in a Refresh operation.
When we bring this table into Power Query, the empty cells are perceived as NULL cells.
In an unpivot operation, cells containing NULLs are omitted from the unpivot result.
But is you require all data to be present after the unpivot step, you need to replace the NULLs with 0
(zeroes).
Before the Unpivot step, select all the columns that could contain NULLs and click Transform (tab) →
Any Column (group) → Replace Values.
In the Replace Values dialog box, the Value To Find will be null and the Replace With will be 0.
When the next month’s data is received (ex. July), the Replace Values step will not be applied to the new
month’s data.
A Little Bit of “M” Goes a Long Way
It’s times like this that knowing just a small amount of “M” code will reap great benefits.
We need to make dynamic the portion of the Replace Values step that looks for months.
Table.ColumnNames(#”Filtered Rows”)
NOTE; In this example, the reference to “#Filtered Row” is the name of the previous step. Replace this
reference with the name of the previous step in your query.
The most important step is to determine which column to pivot. In other words, which column in the
table should become the new columns and headings in the cross-tabular report.
For our data, we want the Products to be listed down the left side of the report while the Months are to
be listed across the top of the report. The Sales will become the intersecting aggregations.
We need to select the Months before the pivot operation. Selecting Months informs Power Query to
create a unique list of items from the selected columns and create a new column for each item.
In the Pivot Column dialog box, select the field that will serve as the aggregations for the row/column
intersections.
You can also select Advanced Options to get the ability to select the aggregation method (i.e. SUM,
AVERAGE, COUNT, etc.)
The result is a cross-tabular report that is easier to understand than a flat table.
Our objective is to count the number of Available Colors for each Product.
The Problem
The problem is that the report designer has inserted in-cell carriage returns (ALT-Enter) in the cells that
hold colors.
The Solution
In Power Query, perform the following steps:
1. Select the Available Colors column.
2. Select Home(tab) → Transform (group) → Split Column → Split Column by Delimiter.
3. In the Split Column by Delimiter dialog box, observe that Power Query has determined the
process needed to split each color by the embedded in-cell carriage return characters.
4. Click OK.
The result is a cross-tabular report that separates each product’s colors into separate columns.
Next, we will revert this cross-tabular report into a tabular report (table).
5. Right-click the header for Product and select Unpivot Other Columns.
6. Remove the unneeded Attribute column.
Next, use the Group By feature to count the number of colors for each product.
7. Select Transform (tab) → Table (group) → Group By.
8. In the Group By dialog box, select Product as the column to be grouped and Count Rows as the
operation to be performed for each Product.
The columns to create for storing colors are restricted to three columns (the maximum number of colors
for any given product.)
This needs to be made dynamic so it can account for any number of supplied colors.
4. In the Advanced Options section, change the Split Into option from Columns to Rows.
5. Click OK.
The result is as follows.
An advantage of this technique is that it saves the steps required to perform the unpivot operation.
A disadvantage is the presence of NULL values in the Available Colors column.
Next, we will perform the same steps we performed in the previous section to create the grouped list of
products and their color counts.
To account for this possibility, we need to perform a Remove Duplicates operation on the color before
the grouping step.
To make this even better, we will standardize the casing in the event users do not follow the same casing
patterns.
6. Select the Split Column by Delimiter step.
7. Select Transform (tab) → Text Column (group) → Format → Capitalize Each Word.
8. Select .
9. Select both the Product column and Available Colors column.
10. Select Home (tab) → Reduce Rows (group) → Remove Rows → Remove Duplicates.
11. Select .
12. Select the last step in the query (Grouped Rows).
You can use these from the Add Column tab or Transform tab depending on whether you’d like to add a
new column for the transformation or update the values in the existing column.
Selecting a column of dates and clicking Add Column/Transform (tab) → Date & Time (group) → Date -
→ Week → Week of Month gives us exactly what we need without all the stress of writing a
complicated formula.
The Excel formula equivalent to this one-click action would require writing the following formula:
Step 2: Convert the results to display as the difference in hours by selecting the results of Step 1 and
click Duration → Hours.
Feature Use
Age Return the duration between the current local time and the values in the
selected columns
Date Only Extract the Date component from the Date/Time values
Parse Return a Date value parsed from the text
Year Extract the Year component from the Date/Time values
Start of Year Return the first day if the year corresponding to each Date/Time value
End of Year Return the last day if the year corresponding to each Date/Time value
Month Extract the Month component from the Date/Time values
Start of Month Return the first day if the month corresponding to each Date/Time value
End of Month Return the last day if the month corresponding to each Date/Time value
Days in Month Return the number of days in the month corresponding to each Date/Time
value
Name of Month Return the name of the month corresponding to each Date/Time value
Quarter of Year Return the quarter corresponding to each Date/Time value
Start of Quarter Return the start of the quarter corresponding to each Date/Time value
End of Quarter Return the end of the quarter corresponding to each Date/Time value
Week of Year Return the week of the year corresponding to each Date/Time value
Week of Month Return the week of the month corresponding to each Date/Time value
Start of Week Return the start of the week corresponding to each Date/Time value
End of Week Return the end of the week corresponding to each Date/Time value
Day Extract the Day component from the Date/Time values
Day of Week Return the day of the week corresponding to each Date/Time value
Day of Year Return the day of the year corresponding to each Date/Time value
Start of Day Return the start of the day corresponding to each Date/Time value
End of Day Return the end of the day corresponding to each Date/Time value
Name of Day Return the name of the day corresponding to each Date/Time value
Subtract Days Return the number of days between the values in the first and second
selected columns
Combine Date and Time Merge the selected columns into a new column containing both Date and
Time data
Earliest Return the earliest Date value
Latest Return the latest Date value
Feature Use
Time Only Extract the Time component from the Date/Time values
Local Time Return the Date/Time/Timezone values
Parse Return the Time value parsed from the text
Hour Extract the Hour component from the Date/Time values
Start of Hour Return the start of the hour corresponding to each Time value
End of Hour Return the end of the hour corresponding to each Time value
Minute Extract the Minute component from the Date/Time values
Second Extract the Second component from the Date/Time values
Subtract Contains the duration between the values in the first and second selected
columns
Combine Date and Time Merge the selected columns into a new column containing both Date and
Time data
Earliest Return the earliest Time value
Latest Return the latest Time value
Feature Use
Days Return the days component corresponding to each Duration value
Hours Return the hours component corresponding to each Duration value
Minutes Return the minutes component corresponding to each Duration value
Seconds Return the seconds component corresponding to each Duration value
Total Years Return the total number of years in each Duration value
Total Days Return the total number of days in each Duration value
Total Hours Return the total number of hours in each Duration value
Total Minutes Return the total number of minutes in each Duration value
Total Seconds Return the total number of seconds in each Duration value
Subtract Contains the duration between the values in the first and second selected
columns
Multiple Multiplies each value in the selected columns by a specified value
Divide Divides each value in the selected columns by a specified value
Statistics (Sum, Min, Sum, Min, Max, Median, or Average of all the Durations
Max, Median, Average)
Days.Hours:Minutes:Seconds
Because of this, it is common to reformat the duration to something more easily understood, such as
Total Years, Total Days, or Total Hours.
Custom
Columns
Power Query Custom Columns
Within the Custom Column dialog box, you can create unlimited functions to produce virtually any
output needed.
For a complete list of available Power Query functions along with detailed explanations and examples,
visit the following Web resource.
docs.microsoft.com/en-us/powerquery-m/power-query-m-function-reference
The library can also be accessed by clicking the link in the lower-left corner of the Custom Column dialog
box labeled “Learn about Power Query formulas”.
NOTE: Custom Columns can also be used to create more complex query steps that combine several
click-based steps into a single step. This can result in reduced code and increased performance. It does,
however, require a more advanced understanding of the “M” language.
Text to Numbers
To convert text to a number, we can use the Number.From function.
Number.From( [TaxRate] )
Date Function
In a dataset where we have Year and Month as separate columns, formatted with a Whole Number
data type, we wish to create a date.
Since we are missing DAY information, we will fix the days to the 1st for each date.
The function and syntax for this date creation are as follows.
Our example will use the Start Year and Start Month columns while using a fixed value of 1 for the day.
• #duration
• #datetimezone
• #date
• #time
• #dateTime
For a complete list of available Power Query functions along with detailed explanations and examples,
visit the following Web resource.
docs.microsoft.com/en-us/powerquery-m/power-query-m-function-reference
The library can also be accessed by clicking the link in the lower-left corner of the Custom Column dialog
box labeled “Learn about Power Query formulas”.
Power Query
Online Data
Sources
Power Query Power Query Online Data Sources
• Web pages
• Embedded files within a Web page
• SharePoint
• OneDrive for Business
• Outlook email
• OData feeds
The list of connectors is vast and ever-increasing.
Keep in mind is that not every connector is available to all versions of Power Query. Certain connectors,
like “connect to a Web page” are universally available, while other connectors like “SharePoint Online
List” are limited to later or higher-level versions of Office.
The below link will provide a list (current as of Spring 2020) of connectors and their platform availability.
https://support.microsoft.com/en-us/office/where-is-get-transform-power-query-
e9332067-8e49-46fc-97ff-f2e1bfa0cb16
Connecting to a Website
In most cases, the URL used to view the data in a Web browser is the same URL needed for Power Query
to retrieve the data.
We launch the connector by selecting Data (tab) → Get & Transform Data (group) → From Web.
In the From Web dialog box, enter the URL and click OK.
NOTE: Power Query requires that tables on a website be formatted using HTML to be detected and
imported.
138 | P a g e Copyright © 2020 Leila Gharani
Power Query Power Query Online Data Sources
https://www.odata.org
Several sample data models are available for testing purposes. These are similar to data models that
would require complex SQL servers working locally to host and allow access.
The main website provides access to these test data modes by selecting Developers → Reference
Services.
Connecting to ODATA
To connect to an ODATA feed, select Data (tab) → Get & Transform (group) → Get Data → From Other
Sources → From ODATA Feed.
NOTE: The data models available on the ODATA website can be accessed anonymously.
Once the format is selected, click Publish to obtain the URL that grants access to the data.
In most cases, the URL used to view the data in a Web browser is the same URL needed for Power Query
to retrieve the data.
We launch the connector by selecting Data (tab) → Get & Transform Data (group) → From Web.
In the From Web dialog box, enter the Google Sheets URL and click OK.
Because Google Sheets have a similar structure to Excel workbooks, it is necessary to select a sheet (or
sheets) of which to import.
NOTE: When making changes to the source material in Google Sheets, it may take upwards of 5 minutes
for the changes to become available to the query when refreshing the data model in Excel.
To access the Inbox, folders, and emails from your Exchange account, select Data (tab) → Get &
Transform (tab) → Get Data → From Online Services → From Microsoft Exchange Online.
If you are running a local Outlook installation, you can select Data (tab) → Get & Transform (tab) → Get
Data → From Other Sources → From Microsoft Exchange.
In the Microsoft Exchange dialog box, enter your email address and click OK.
At this point, you will need to provide authentication credentials. These can be your local Active
Directory / Exchange credentials or your Microsoft account credentials.
Once connected, we see the major functional areas of the Exchange account. We can click Mail (on the
left) to display the folders and items in the Exchange Inbox. Notice the Headings of the Inbox preview.
• Easily share files with coworkers where the file’s size would be too large for email attachments.
• Allow for simultaneous, shared access to files.
NOTE: The following process requires a valid Microsoft Business account with access to SharePoint or
OneDrive for Business.
The most difficult part of this process is locating the path for use in Power Query. One common
technique is as follows.
1. Open a browser and access your SharePoint drive or OneDrive for Business drive.
2. Locate the file by which you require a connection.
3. Click the “three stacked dots” to the right of the file’s name.
6. Click the icon to the right of Path to copy the link address to memory.
To access the SharePoint file, select Data (tab) → Get & Transform (tab) → Get Data → From Web and
paste the copied link into the URL field and click OK.
NOTE: Because the source pointer is not pointing to your local computer, if you share this file and a user
attempts a refresh of the data, they will be required to enter their SharePoint or OneDrive for Business
credentials.
Combining &
Appending Data
Power Query Combining & Appending Data
Appending Data
When you append data, you “stack” the results. It would be like taking multiple identical tables (same
columns/same headings) and place one table below the other. Appending grows tables in rows.
Things to be aware of when appending results from separate queries:
• You must have the same number of columns. If you don’t, rows from one table that do not have
entries for a column in the other table will contain nulls in the non-matching column.
• The column heading’s names must be the same. If the names are different, Power Query will
treat the columns as separate columns.
• The names must match in casing (formatting). Power Query is case-sensitive. Differences in the
casing will result in separate columns of potentially duplicate information.
• The columns should have matching data types. Matching columns with dissimilar data types will
result in a loss of data type definition. Results will be presented as the “Any” data type.
• The columns do NOT need to be in the same order. Columns will be matched based on the
heading names and formatting.
Once you have created and processed the separate datasets, you have two options when appending the
results into a single table:
• Append Queries
• Append Queries as New
Append Queries
This option will attach the results of one query to the bottom of the results of another query. You are
altering one of the queries’ dataset, but no additional queries are being created.
Append Queries as New
The combined results of the original datasets are stored in a new query’s results.
If you need to append more than two query results, select “Three or more tables” and place the desired
queries in the “Tables to append” list (right).
Merging Data
When you merge data, you find matching records in separate tables and combine their content into a
single table record. This is like performing a VLOOKUP (or any other type of lookup operation) in Excel.
Merging grows tables in columns.
Things to be aware of when merging results from separate queries:
• The tables being merged must have at least one column in common (ex: Part Number).
• The common columns must be of the same data type.
• Records of one table with no matching row in the other table will contain nulls in the merged
columns.
Once you have created and processed the separate datasets, you have two options when merging the
results into a single table:
• Merge Queries
• Merge Queries as New
Merge Queries
This option will attach the results of one query to the side of the results of another query. You are
altering one of the queries’ dataset, but no additional queries are being created.
Merge Queries as New
The combined results of the original datasets are stored in a new query’s results.
Merge Options
Join Kinds
Explained
Power Query Merge Options – Join Kinds Explained
The Merge feature can also be accessed from within Excel by selecting Data (tab) → Get & Transform
(group) → Get Data → Combine Queries → Merge.
REMEMBER: Merge expands your data left-to-right (columns), where Append expands your data
top-to-bottom (rows).
NOTE: Connecting columns (key fields) must be the same data type.
Advantages of Merges
• Merges can occur between different external data sources
• Merging is faster on larger data sets compared to performing a traditional lookup operation
• The user is not required to write any complicated (or simple) formulas
Merge Options
Power Query is equipped with the following merge strategies:
In Power Query, likely due to the design of the Merge Queries interface, the tables are referred to as
“first” table and “second” table.
Right Outer
Keep all records from the second table and matching records from the first table.
Full Outer
Keep all records from both tables.
Left Anti
Keep records that only appear in the first table.
Right Anti
Keep records that only appear in the second table.
Inner
Keep records that only appear in both tables.
To help differentiate the records, perform the merge by matching multiple columns (ex: FirstName &
LastName to matching FirstName & LastName).
This can be accomplished by selecting the first column to match (ex: LastName) in the Merge dialog box,
press and hold the CTRL key, then click the second column to match (ex: FirstName).
You are not limited to two columns for greater specificity; you can select as many columns as needed to
achieve uniqueness.
NOTE: Because the act of expanding the fields of the merged table will likely reorder the rows of the
companion table, you may wish to add an index column before expanding the merged table. This will
allow you to restore the list to its original order. Once restored, you can remove the index column.
Copyright © 2020 Leila Gharani 159 | P a g e
Power Query Merge Options – Join Kinds Explained
When we perform a Merge with a Left Outer join, we can see that there are multiple “hits” against
Employee ID “67”.
When you extract the nested table’s content, you will receive all matching rows.
5. To retrieve the missing information, merge the results of this table with the original TableMaster
table using Employee ID and Latest Date/Start Date as the keys.
After extracting the needed fields from the nested tables, the results are as follows.
• Matching items of each list’s key column must be of the same data type. Power Query is
type-sensitive.
• Matching items of each list’s key column must match in casing (i.e. uppercase, lowercase, proper
case). Power Query is case-sensitive.
• Extraneous spaces before or after the data can cause “invisible” differences when matching
items. Consider running a TRIM operation before performing the Merge operation.
• Non-printable characters in the text (i.e. carriage return or line feed characters) before or after
the data can cause “invisible” differences when matching items. Consider running a CLEAN
operation before performing the Merge operation.
When to use
Power Pivot &
Load to Data
Model
Power Query When to use Power Pivot & Load to Data Model
A Data Model allows you to integrate data from multiple tables, effectively building a
relational data source inside an Excel workbook.
Within Excel, Data Models are used transparently, providing tabular data used in
PivotTables and PivotCharts.
A Data Model is visualized as a collection of tables in a Field List, and most of the time,
you’ll never even know it's there.
The Data Model is used primarily by Power Pivot and Power BI.
• Data can span well beyond the 1 million row limit of an Excel worksheet.
• Pivot tables can be built from elements originating from different tables.
NOTE: Unless you have a compelling reason to see the table in a worksheet, it’s better to create a
connection to the data and only load it into the Data Model. This will save on memory and storage.
The interface is not as user-friendly, but the basic connectivity tools exist.
Understanding
M Formula
Language
Power Query Understanding M Formula Language
When we select a step from the list, we can see the underlying M code in the Formula Bar.
NOTE: If you do not see the Formula Bar in the Power Query Editor, select View (tab) → Layout (group)
→ Formula Bar.
If you want to see the entire code set at once, instead of clicking step-by-step while you try to remember
what you read from another step, you can activate the Advanced Editor by selecting Home (tab) →
Query (group) → Advanced Editor.
NOTE: Consider experimenting with the Display Options in the Advanced Editor. Features like “Display
Line Numbers” are especially helpful.
Expressions
The let Expression
The purpose of the let expression is to return a value. The returned value can be as small as a single
number or letter, or as large as a table with millions of rows and thousands of columns.
Each line in the code is a separate instruction ending with a comma except for the LAST instruction.
Think of the comma as the word “and”. You could say, “Do this AND do this AND do this.” You wouldn’t
put an “AND” at the end because that implies another step should occur.
A1 = B1 + B2
Source = Excel.CurrentWorkbook(){[Name="TEmployee"]}[Content],
NOTE: If you have but a single step query, you can omit the let, in, and named memory location from the
M code. This is not encouraged, but it’s an interesting behavior.
The in Expression
After the final step in the query, the in expression is used to signify the end of the steps and where to
retrieve the final result. Typically, the final result is the memory name used in the final step.
It’s a way of saying, “Get the last result produced and deliver it to the user.”
Comments in M Code
To add a single-line comment, precede the comment with a double forward slash.
To add a multi-line comment, start with the “/*” characters and end with the “*/” characters.
#”Changed Type”
If the step name does not contain spaces, these characters can be omitted.
ChangedType
NOTE: Omitting the spaces tends to make the code cleaner, easier to read, and less prone to syntax
errors.
WARNING: If you manually modify the name of a step in the Advanced Editor, remember to modify any
other steps that reference the renamed step. Otherwise, your code will fail.
“M” Characteristics
Code Variations
When performing manipulations on the date, Power Query will write the code in a way it thinks is most
efficient.
Example:
If we were to filter out a single item, the code is likely to read as “do not include this one item”.
But if we filtered out most items, the code is likely to read as “only keep this one item”.
This can have unintended side-effects when adding additional data. You may inadvertently include the
new data when you didn’t want to, or you may omit the new data when you needed it to be part of the
result.
Step Result Retention
Upon code completion, except for the final step result, the individual step results that were stored in
named memory locations are discarded.
This allows for more efficient use of memory by not retaining step results that are no longer needed.
Function Syntax
The beginning of a function requires the establishment of variables (named places in memory) to accept
input for processing. These must be contained within a set of parentheses, separated by commas, and
followed by “=>” characters.
After the “=>” characters are the instructions for processing the supplied data.
This produced an interface that allows the user to manually type the variables.
When we supply data into the fields and press , we see the result of the function as a separate
query.
If we supply data that cannot be processed by the function (ex: text where it expects a number), we
encounter an error.
This can be mitigated by requiring specific data types for each variable.
NOTE: As with a traditional query, if the function is but a single line, the need for let and in statements is
not required.
The input interface now instructs the user as to the desired data types and alerts the user when invalid
data is supplied.
Multi-Step Functions
If we wish to perform several steps when transforming the data, we must include the let, in, and named
memory address where the final result is stored.
Saving a Function
When executing a Close & Load on a function, the query is always stored as a Connection Only query.
Supply the results column a name and define the fields to be used by the related function arguments.
We see a new step has been added to the query that invokes the custom and applies the function logic
to each row in the table.
The invoked custom function code can be altered to incorporate additional actions (ex: data typing).
NOTE: The function can reside anywhere within the let..in section. The function will not be executed until
it is called by another line of code.
=YEAR( A1 )
We are not required to tell Excel which category the YEAR function resides because all functions reside
on the same list.
With M being the “new kid on the block”, Microsoft took advantage of its youth and has created
different categories for different functions. This allows M to reference the same named function (ex:
FROM), but get different results based on the category from which FROM was retrieved.
In other words, the results you get when using the “From” function will be different depending on
whether you used Date.From, Byte.From, Duration.From, or Number.From.
Clicking a function category will present a list of all functions within the selected category.
Clicking a function will present a detailed explanation of the function, the function arguments, as well as
examples of use.
To locate a specific function or category in the library, a Search box is provided in the top-left of the
page.
= #shared
This tends to be a bit much to deal with, so we can make things easier by converting the list to a table.
This provides us with a table that we can filter based on numerous criteria.
Clicking to the right of a function’s hyperlink will display the full documentation of that function in the
lower panel of the Power Query Editor.
Function Context
When using a function, be mindful of the function’s context (i.e. category). Functions that begin with
Text, List, Record, or Table will view the data differently.
Lists
Think of a list as a table with only one column. What makes the list different is that you can have a
mixture of data types within the same single column.
When working with lists, Power Query provides a new set of tools in a second Transform ribbon under
List Tools. This replaces the functionality of the traditional Transform ribbon. (Note the different icon in
the queries list.)
[Employee Name]
The above list was generated from a column named “Employee Name” from the previous step named
“Changed Type”.
Records
Think of a record as a table with only one row. When you click on a row number in a table, the row is
displayed in the lower panel as a record.
= TEmployee
= TEmployee[Employee Name]
= TEmployee{0}
= TEmployee[Employee Name]{0}
NOTE: When referencing values, the order of Field/Row can also be made as Row/Field. Most users
prefer the Field/Row version.
= TEmployee[Employee Name]{0}
or
= TEmployee{0}[Employee Name]
=TEmployee{5}
NOTE: This only works if there are no duplicates in the requested field. If there are duplicates, an error
will occur.
The above example would return all rows that have “MSFT” as the stock symbol.
Square brackets can also be used during field selection to return a single record.
When defining several fields, each field is enclosed in square brackets while the entire set of fields is
enclosed in a set of square brackets.
Curly brackets can also hold a list of values. These values can be of differing data types.
{ 1..100 }
{ “Department”, 500 }
One way to test column manipulations without risking corrupting your existing query is to start a new
blank query and extract a column from the other query as a list.
You can now practice the manipulations without fear of damaging your existing work.
={ 1 .. 10 }
List of Lists
It sounds strange, but a list can contain other lists.
This is nothing new; you’ve been using “lists of lists” since day one of using Power Query.
Examine the following #”Changed Type” step:
It’s a list of lists. In this case, column names and data types.
=[Company ID = 123]
The name of the column “Company ID” is not enclosed in square brackets because the entire statement
is within square brackets.
To create a record with multiple columns, the syntax is as follows:
= #table()
The above table function requires a list for the column headers and a list of lists for the data.
At its core, the Each keyword tells Power Query to loop through many values, usually rows.
This would be analogous to writing a single formula in Excel and using Fill Series to repeat the formula
down a column for a series of rows.
Consider the following instruction:
We’re using the Table.SelectRows function to extract records where the [Department] is equal to
“Sales”.
The function will process each row using the same criteria.
Shorthand Notation
The Each keyword is shorthand notation for a function and has a single argument. This allows us to
write a custom function within a step without confusing syntax.
Examine the following code:
Underscore Notation/Reference
There will be steps that utilize an underscore character in their code.
= Table.TransformColumns(#"Filtered Rows",
{ {"Employee Name", each Text.BeforeDelimiter( _, " " ), type text } } )
The underscore character is M Code for “element in current column”. Since we have defined the
“Employee Name” column as an argument for the Table.TransformColumns function, we don’t need to
declare it a second time for the Text.BeforeDelimiter function. The underscore denotes elements in the
already defined column.
Here is another example of the use of an underscore character when working with lists:
The underscore relieves us from writing a complicated reference to elements in an already declared
source.
Each time we want to run a query for a different Region, we must either modify the existing query or
(worse yet) write an entirely new query that is mostly the same.
Suppose we could write the step in a way that prompts the user to pick a region each time the query is
refreshed.
becomes
The moment you switch back to the query that calls the parameter; the results have changed.
If you have several parameters that need to be set, you can select Home (tab) → Parameters (group) →
Manage Parameters → Edit Parameters to launch a dialog box where all parameters are displayed in a
convenient interface.
After:
This can be especially useful when passing a table to a function as the table will be read only once.
• table to be repeated
• number of times to repeat
An example of its use would be the following:
We can now run any number of transformations against the large data set to test performance.
Once testing is complete, we can delete this instruction and the remaining transformations will now
execute against the actual data.
After we have used the Table.Repeat function, we see a noticeable increase in table rows.
NOTE: If you are only seeing 1000 as the Count result, you likely need to change the option to view the
entire data set (bottom-left corner).
NOTE: Query Folding does not apply to flat files like Text files or Excel files.
This will display the code necessary to perform the selected step in the source language.
As you progress down the list of query steps, you can view each new transformation’s instructions.
Some steps may contain very little code while other steps can get quite lengthy.
NOTE: Displaying the native query is not always possible for all sources, even when query folding is
occurring. Connecting to an OData feed will capitalize on query folding, but you will not be able to
display it using the View Native Query feature. To see the query folding operations, you would need to
consult the OData server log files.
IF THEN STATEMENT –
LOOKUP OPERATORS TO GET VALUE FROM PREVIOUS ROW
= if [Index] = 0 then null else [Gold Price Close] - #"Added Index"[Gold Price Close]{[Index] - 1 }
When we reference [Gold Price Close], we are looking at a single value for the current row. This changes
as we move from row to row during table processing.
We need to point to the entire column of gold prices, so we reference the previous query step’s results
as a reference table.
We need to locate a specific gold price from this column, so we use the value stored in [Index].
Because Power Query starts counting rows at 0 (zero), we need to make a small adjustment to the value
by deducting 1 from the [Index].
This returned value is subtracted from the [Gold Price Close] of the currently examined row.
= TransformTable{[Before = [App]]}[After]
Although we are retrieving the new app names for listed apps, we are receiving errors for apps that do
not require new names.
Open the Custom Column and modify the formula as follows:
, type text
… becomes …
Working with
Lists & Table
Function
POWER QUERY TEXT FUNCTIONS
• Text.Remove
• Text.Insert
• Text.Replace
Text.PositionOf
The syntax for Text.PositionOf is
Example of Text.PositionOf
The result would be 8 because the word “Force” starts in the 8th character position of the text.
Remember, Power Query starts counting at 0 (zero).
Practical Example
We have the following data:
type logical
The formula takes the [Article Description] and capitalizes the first letter of each word using the
Text.Proper function. We use the same function to capitalize the first letter in the [Color] field.
Using those results, we concatenate a “space-open parentheses” to the [Color] to a “close parentheses”.
The Text.Replace function locates the concatenated result in the [Article Description] and replaces it
with nothing (two double quotes).
We will data type the output as Text responses by adding the proper declaration to the end of the M
Code in the Formula Bar.
type text
More Resources
MORE RESOURCES
In addition to this Reference book, we’ve also put together a Power Query Solution book for you to
handle different scenarios you might come across. This book is also available as a resource inside your
online Power Query course on XelPlus.com.
Let's talk about more learning.
On my site, you’ll find weekly Excel articles on different aspects of Excel: XelPlus.com/tutorials/
There are a lot of great Excel websites out there. For more Excel information, check out my list of
resources here: XelPlus.com/resources
The more you know of Excel the easier it gets for you to implement complex projects.
If you're interested to become more advanced and create great visualizations or effective dashboards,
check out my other courses: https://www.xelplus.com/courses.
I also share weekly tutorial videos on my YouTube channel so make sure you check it out if you're
interested to learn more. Just type in Leila Gharani on YouTube.
I wish you the best of luck on your journey and hope to see you in my other courses.