Our fourth podcast! This month we have some fun and build our own city rankings, take a look at rankings of cities and regions in Canada, and look at data and tools released from Statistics Canada that you may have missed.
Ranking Cities on Business Counts
Before we get into the rankings by the professionals, we decided to do a little exercise of making a simple ranking ourselves to see if we could learn something from it. For this we chose Statistics Canada table 33-10-0269-01: Canadian Business Counts with Employees. That provides the number of businesses that employed at least one person registered in each CMA on a regular basis. It’s most often used to look at trends in business openings and closings across cities in Canada. We’re just going to use it to have some fun and rank cities according to those business counts. First up, let’s look at gas stations:
Saguenay, Quebec and Moncton, New Brunswick get top billing here, dense urban areas (eg: Vancouver and Toronto) have the fewest per capita. Aside from density, seeing a ranking like this you could start to ask questions about how public transit systems might affect gas station numbers. What about car ownership levels? Why does Montreal rank higher than other large cities? If you are wondering why Windsor is highlighted, it’s selected in our tool. We’ll talk about what that means later.
Let’s take a look at another ranking – drinking places (ie: bars and other such establishments):
And restaurants:
Here’s the thing about rankings – often the histograms aren’t provided and you can’t easily see the shapes in the data. So, if you just saw a local news article that said “Drinking on an empty stomach: Saint John ranks third highest in drinking establishments but third last in restaurants”, you might think that the shapes on both the graphs would look about the same, whereas the variation in the distribution of the drinking establishment numbers is much higher. Given that, always consider how the information is being presented when reading about city rankings. A ranking where the distribution of data is uniform can make a ranking arbitrary and misleading.
We decided to have a little more fun with this and see if we could display all the rankings for all 4-digit NAICS industry codes for a city all at once. Here’s what it looks like for Windsor, ON:
Now you can see why Windsor was red earlier – that’s the city we has selected in our tool. Where you see frequent darker colours in a column, those are high ranked industries in the same broad industrial sector. Lighter coloured blocks indicate lower rankings. If they are in the same column, they are in related industries. It’s a good way to zero in on specific areas that are the city’s strengths. Let’s check out some examples of a dark (high ranked) areas for Windsor: column 33 (NAICS code 33XX).
Windsor has a reputation of a manufacturing and industrial town for a reason. You can play around with this heat map and bar chart yourself at this link. Use the drop downs or click on the bars and heat map squares to change city and/or industry. If you do though, make sure you use a device with a larger screen and click the “View in Full Screen” button so that you can view the graphs side by side.
One note on these rankings is that it is a simple ranking on the number of businesses with employees, not the number of employees nor the contribution to GDP. A city with a few very large employers would be underrepresented in these and a cities with a large number of small employers would be over-represented. It’s a good illustration of keeping in mind what measures are being ranked when looking at a ranking.
Ranking the Professional Rankings
Media loves to cover research that rates municipalities, especially when the local municipality is at the top or the bottom of the scoreboard. Community leaders and local media have a tendency to uncritically promote rankings that score their municipality highly and find fault with those that don’t. That’s fair enough, but one thing often missing from the public conversation about rankings is a more objective look at their findings and relative quality. We compared ten different rankings of cities in Canada, scoring them in five categories, each with a possible score of 0, 1, or 2, resulting in a score out of ten that allows us to compare these pieces of research with each other.
Our scoring system is focused on identifying how repeatable, reliable and valid each city ranking is. We arrived at our scores by consensus according to a set of specific criteria for each category. These are our five scoring categories:
Qaulity of Data
Good data is the foundation of any piece of objective research. We looked at the data sources that each ranking identified and checked how closely they matched the stated goal of the research, looked at the overall quality and reputation of the data source, and whether the research being evaluated used appropriate weighting and filtering techniques for their research goal.
Applicability
In this category, we evaluated the applicability of the data used to the question being asked. One way we did that was to consider the intent of the original data collection and research that supplied the underlying data to each piece of city-ranking research.
Transparency
To score this category we looked at how easy it is to verify and repeat the work done by the researchers, and whether they disclose their own potential biases. Does the research identify its data sources clearly enough for a reader to find it on their own? Is the underlying data available for anyone to see? Does the ranking explain its methodology, and does the organization who did the research disclose its funding sources?
Methodology
Diving deeper into methodology, our scores in this category take a look at the scoring and evaluation methods used in each piece of research to determine whether it is appropriate for the data and uses generally accepted procedural standards. For example: does the methodology make adjustments to the data that supplies each component of the scoring system to make sure that a highly-variable subscale doesn’t swamp other components?
Comprehensiveness
This category asks whether the data has scope sufficient to answer the research question. A ranking will score poorly in this category if the data is a subset of what’s needed to answer the question, or if possible confounding factors are ignored.
Our Ranking of the Rankings
These are the ten rankings we looked at, in order from highest to lowest score.
A Snapshot of Canada’s Developer Talent
Organization: Brookfield Institute
Link: http://brookfieldinstitute.ca/wp-content/uploads/BrookfieldInstitute_StackingUp.pdf
| Measure | Score |
|---|---|
| Quality of Data | 1 |
| Applicability | 2 |
| Transparency | 2 |
| Methodology | 2 |
| Comprehensiveness | 2 |
| Overall | 9 |
This report is focused on software developers, which the authors call a “critical component of Canada’s tech talent.” Brookfield partnered with Stack Overflow, a popular website for software developers, to look beyond traditional labour market data. This innovative approach is a theme of Brookfield’s research, which has benefits and drawbacks. It can illuminate the landscape in ways that research based on traditional data doesn’t, but it also can make it difficult to compare to similar research, or build on their work by connecting it to other data sets.
This study was the most transparent ranking that we looked at. The authors publish their data, calculations and source code. They also called out some of the issues with their own data.
We deducted a point for data quality as the underlying survey data is self-selected, the data source is primarily English speaking, and may under-represent some classes of software developer.
The Best and Worst Places to be a Woman in Canada 2019
Organization: Canadian Centre for Policy alternatives
Link: https://www.policyalternatives.ca/publications/reports/best-and-worst-places-be-woman-canada-2019
| Measure | Score |
|---|---|
| Quality of Data | 2 |
| Applicability | 1 |
| Transparency | 1 |
| Methodology | 2 |
| Comprehensiveness | 2 |
| Overall | 8 |
The CCPA’s annual ranking of Canadian Cities on gender equity provides a snapshot of the gaps in men’s and women’s access to economic security, personal security, education, health and positions of leadership in Canada’s largest 26 metropolitan areas. CCPA modeled the study on global measures of gender equality produced by the World Economic Forum, adding some additional measures which are more easily compared in a national study because of the higher degree of uniformity within national datasets over international datasets.
Sources are clearly identified and there is care taken in the methodology to ensure that highly-variable measures do not “swamp” other less-variable measures. This is an illuminating report which meets a high standard of quality. We deducted one point for transparency as source data is not readily available for independent analysis, and one point from applicability because of a temporal issue with the data: elected officials are typically replaced on a four-year cycle in Canada, so this annual ranking has one subscale which is mostly frozen between election cycles.
Regional Labour Market Report Card
Organization: BMO
Link: https://economics.bmo.com/media/pdf/e7058fa4-db39-4d68-9fe7-097908d7100a.pdf
| Measure | Score |
|---|---|
| Quality of Data | 2 |
| Applicability | 2 |
| Transparency | 1 |
| Methodology | 1 |
| Comprehensiveness | 2 |
| Overall | 8 |
This quarterly report from BMO Economics Research is focused on labour market performance in Canadian cities. It’s brief, offers limited textual analysis and is narrowly focused on a few data points. The data is valuable, well-presented and provides a ranking of cities that is clear and easy to understand. Within its limited scope it does a good job of ranking cities relative to each other.
Points were deducted for not identifying data sources, publishing a detailed methodology, or identifying a rationale for the relative weights given to different measures of employment. It would be difficult to replicate this research exactly without a fair bit of work.
Canada’s Best Communities
Organization: Maclean’s Magazine / Moneysense
Link: https://economics.bmo.com/media/pdf/e7058fa4-db39-4d68-9fe7-097908d7100a.pdf
| Measure | Score |
|---|---|
| Quality of Data | 2 |
| Applicability | 2 |
| Transparency | 1 |
| Methodology | 1 |
| Comprehensiveness | 1 |
| Overall | 7 |
This annual study compares Canadian municipalities across ten categories aimed at finding the most livable, best city to live in. Sourcing data from Statistics Canada and Environics Research, Maclean’s clearly invested significant resources to deliver a substantive and valuable product. Their results are based on a weighting of priorities for a typical Canadian family, and they also provide a tool for consumers of the report to re-weight the data according to their own priorities.
We deducted points because Maclean’s does not provide a link to source data, does not provide a clear rationale for the point system provided for subscales, and because some measures seem to double-count variables like current Crime Severity Index (CSI) and the 5-year change in CSI.
Sustainable Cities Index
Organization: Arcadis
Link: https://www.arcadis.com/en/canada/our-perspectives/sustainable-cities-index-2018/canada/
| Measure | Score |
|---|---|
| Quality of Data | 1 |
| Applicability | 1 |
| Transparency | 1 |
| Methodology | 2 |
| Comprehensiveness | 2 |
| Overall | 7 |
To develop an index of overall sustainability that aims at building a greater understanding of the characteristics that enable some cities to outperform their peers, Arcadis measures cities on three dimensions People, Profit, and Planet. The report also clusters cities into archetypes based on seven core needs that a city should aim to fulfill in order for citizens to thrive. This research product is full of insight. People interested in a holistic view of city performance on a global scale will benefit from a careful reading of this work.
We deducted points for transparency because the underlying data is not provided and because it would be very difficult to assemble the data independently. Some of the data is crowdsourced, which puts an upper limit on accuracy and confidence. There is some reliance on data that has already been processed because it is the product of a separate methodology, which raises questions about the value of certain measures that can only be answered by studying the reports they are sourced from. Overall the methodology of this research is sound and the approach to filling the data gaps endemic to global research is sound.
Quality of Life in Canada
Organization: Numbeo
Link: https://www.numbeo.com/quality-of-life/country_result.jsp?country=Canada
| Measure | Score |
|---|---|
| Quality of Data | 1 |
| Applicability | 1 |
| Transparency | 1 |
| Methodology | 2 |
| Comprehensiveness | 2 |
| Overall | 7 |
Unique among the rankings we examined, Numbeo’s website is first and foremost a tool for collecting and disseminating crowdsourced information about communities around the world. We focused on application of its Quality of Life index in Canada, ranking the 14 cities for which sufficient data exists. It is a highly ambitious and interesting project with some challenges arising from the structure of the data itself.
We deducted points because the report is limited in scope, this report uses user-reported perception to rate some measures that have good public data available (safety and crime and health care quality). The methodology is not explained in detail and the explanation that is given is idiosyncratic – chunks of Java source code showing the formula used in developing indices.
School Performance
Organization: Fraser Institute
Link: https://www.fraserinstitute.org/school-performance
| Measure | Score |
|---|---|
| Quality of Data | 1 |
| Applicability | 1 |
| Transparency | 1 |
| Methodology | 1 |
| Comprehensiveness | 1 |
| Overall | 5 |
The Fraser Institute has been ranking Canadian schools for over twenty years. This year’s report includes rankings for elementary and secondary schools in Ontario, Alberta and British Columbia as well as secondary schools in Quebec. Schools are ranked based on their performance against peers using publicly-available data, mostly based on standardized test results. Although there are some issues across the board, FI does a good job creating comparables out of EQAO data. Though this data is dubious for the stated purpose of the ranking, it does provide information that is potentially useful and interesting to the public.
There is significant debate in Canada about standardized testing in schools, but one thing is clear: there is a consensus that standardized test results are not sufficient to compare school quality. In Ontario, the EQAO specifically cautions against using data in this way. Back in 2000, the Canadian Psychological Association provided a position paper on media use of EQAO which is applicable to the Fraser Institute’s work: https://cpa.ca/cpasite/UserFiles/Documents/publications/Joint%20position%20achievement%20test%20results.pdf
Scoring Tech Talent
Organization: CBRE
Link: https://www.cbre.ca/en/research-and-reports/Canada-Scoring-Tech-Talent-2020
| Measure | Score |
|---|---|
| Quality of Data | 1 |
| Applicability | 1 |
| Transparency | 1 |
| Methodology | 0 |
| Comprehensiveness | 1 |
| Overall | 4 |
CBRE analysed 20 of Canada’s leading cities to create a scorecard ranking them for tech talent. They use data from a variety of sources including Statistics Canada. There is valuable information to be gleaned from this report, particularly regarding employment growth and the labour pool, but the inclusion of office space and non-tech labour costs as factors in a ranking of cities for tech talent is a bit of an odd choice.
We deducted points because there are elements with weak applicability to the question being asked, the methodology is not published, not all metrics are provided for all cities, and there are significant questions that can’t be answered about source data. For example, some communities report “N/A” for employment in particular jobs. Our guess is that this is because of sampling issues in the Labour Force Survey data being used, but as methodology and data are not provided that can only be a guess.
Innovation Cities Index 2019: Americas
Organization: 2thinknow
Link: https://www.innovation-cities.com/innovation-cities-index-2019-americas/18843/
| Measure | Score |
|---|---|
| Quality of Data | 0 |
| Applicability | 1 |
| Transparency | 0 |
| Methodology | 1 |
| Comprehensiveness | 1 |
| Overall | 3 |
The data consultancy 2thinknow produces this ranking from their own proprietary data sets. In order to see the data in depth or understand how their indicators are built, users must purchase the full report. 2thinknow provides a list of indicators, but there is no information provided about how these indicator values are derived or weighted, nor is there any information provided about the values for each particular city. If you’re curious how your city ranks on indicators like “Fashion Designers,” “Sports Fanaticism,” or “Architectural Layering” you’ll have to pay to find out.
Top 10 Cities to Settle in Canada as a New Immigrant
Organization: 2thinknow
Link: https://www.immigroup.com/news/top-10-best-cities-settle-canada-new-immigrant-2020
| Measure | Score |
|---|---|
| Quality of Data | 1 |
| Applicability | 1 |
| Transparency | 0 |
| Methodology | 0 |
| Comprehensiveness | 0 |
| Overall | 2 |
This report, from an immigration consultancy in Toronto, provides lots of data about the cities on its list, sourced from Numbeo whose own quality of life index appears a bit higher on our list of rankings. There is no rationale provided for the ranking besides a couple of narrative paragraphs attached to each city. There may be significant value in the opinion of longtime immigration consultants, but it is impossible to reproduce their work or understand how they arrived at their rankings by what is provided here.
Canada’s Best Locations
Organization: Site Selection Magazine
Link: https://siteselection.com/issues/2020/sep/canadas-best-locations-infrastructure-for-innovation-talent-for-tomorrow.cfm#gsc.tab=0
| Measure | Score |
|---|---|
| Quality of Data | 0 |
| Applicability | 1 |
| Transparency | 0 |
| Methodology | 0 |
| Comprehensiveness | 0 |
| Overall | 1 |
This list is a product of Site Selection Magazine, a publication of the Industrial Assets Management Council. They report that they analyze corporate end-user project data on a cumulative and per-capita basis to discover the most competitive province and a list of 20 development groups representing “Canada’s Best Locations.”
Unfortunately, Site Selection does not show their work, or even provide a ranking of the 20 cities chosen for the list. There are no organized data or scores provided to back up their conclusions. It is impossible to verify any of their conclusions or even understand what type of data they are using to draw them. This list may have promotional value for the cities listed but it does not provide much information of value to interested readers.
What you Missed From Statistics Canada
Study: The Employment Consequences of Robots: Firm-level Evidence
Firms that invested in robots from 1996 to 2017 employ more, not fewer, workers. That is the overall conclusion of new studies released today, titled “The Employment Consequences of Robots: Firm-level Evidence” and “The Effect of Robots on Firm Performance and Employment.” The studies found that firms expanded both their high- and low-skilled workforce, although not their middle-skilled workforce after investing in robots. Firms also use fewer managers when robots arrive.
Labour market outcomes for college and university graduates, 2010 to 2016
Today, Statistics Canada is releasing data on labour market outcomes for college and university graduates (graduating classes of 2010 to 2016). This release includes information on the median employment income by educational qualification, field of study, age group, gender, and status of student in Canada—for Canada, the provinces and the territories.
In Ontario – 0.67 men graduate for every women from Undergrad – yet male median earnings after 2 years post graduation were almost 15% higher. 19% higher after 5 years then female earnings.
Statistics Canada has some great data visualizations for this by province, degree type, field of study and income data.
Incomes grow for the top 1% and bottom half of tax filers in 2018, stays flat for the rest
Among tax filers who were not in low income in 2017, 3.9% entered into low income in 2018. For example, statistics on entry and exit from low income indicate that some groups tend to be both more likely to enter low income and less likely to exit low income than others. The tax data allow us to examine low-income entry and exit rates by sex, age, family structure and immigration status
With an entry rate of 9.8%, tax filers from lone-parent families with children under age 17 had an above-average chance of falling into low income. Immigrant filers who had lived in Canada for fewer than six years (7.2%), filers who lived alone (6.6%) and filers aged 18 to 24 (6.1%) also had higher low-income entry rates. While the low-income entry rate for these groups of Canadians varies slightly from year to year, these same groups were more vulnerable to falling into low income in most years.
Incomes grow for the top 1% and bottom 50% tax filers, but stay flat for those from the upper-middle of the distribution
Women represented nearly one-quarter (24.3%) of the 283,015 Canadian tax filers in the top 1% in 2018, up 0.1 percentage points from 2017. Women in the top 1% in 2018 continued to have lower average total income ($429,500) than their male counterparts ($518,600), a gap of 17.2%.
Effective tax rate edges up slightly in 2018 Canadian tax filers spent, on average, 11.8% of their modified total income on federal, provincial/territorial income taxes and employee contributions to Employment Insurance (EI) and the Canada/Quebec Pension Plan (federal payroll taxes) in 2018, up by 0.1 percentage points from 2017.
With the updated Consumer Price Index information coming out from statistics canada they have released a calculator tool that lets you determine your own personal inflation based on your spending patterns. You input certain monthly, annual and one time expenses and it generates a historical inflation pattern for you based on national and provincial CPI trends.
Mean, Median, and Moose 
One thought on “Rankings, Restaurants, and Robots”