Census 2011 and Data Quality

Data Quality

1. Statistics Canada has always worked hard to create qgod quality datasets.  When their budgets were reduced, they have tried to adjust methodologies to preserve as much as possible.  Of course, these adjustments are often made without consultation with the users because the users object to change.  But the users are the ones that understand how data can be used.
2. Statistics Canada will always try to produce good quality data points.  Their commitment to consistent methodologies so that we can analyze vital questions of change is somewhat more questionable because their first goal is to deliver the product that governments mandate.  I am sure that they will never publish bad data but focusing on usefulness is not in their mandate.  In fact, their warranty makes their goals clear.
   

   This data product is provided ‘as-is,’ and Statistics Canada makes no warranty, either express or implied, including but not limited to, warranties of merchantability and fitness for a particular purpose. In no event will Statistics Canada be liable for any direct, special, indirect, consequential or other damages, however caused.

3. One of the key ways in which the quality of the Census 2011 will be evaluated is by looking at changes from the last Census.  Of course, since change is what is interesting to economists, that creates a challenge for analysis. If there are key differences in information about sector employment, labour force education or other key long-form characteristics, we won't really be comfortable whether the difference in the observations is real change, the impact of methodological differences or just plain bad data.
4. The quality of the results depends on the sample size and more importantly on the response rate and the distribution of the responses.  Statistics Canada is planning on a 50% response rate.  The experience in the U.S. that I cite in other posts highlights the problems of response rates.

Letter to G-M about the Resignation of the Chief Statistician

Letter Submitted via Email  to the Globe and Mail on July 22, 2010

Your editorial on Mr. Sheikh’s resignation was very correct.  One of your commentators, Mr. Spector, was not as accurate.  Mr. Sheikh did the right thing at the right time. I do not think it is appropriate to criticize Mr. Sheikh for not resigning earlier. His job was to minimize the damage.  None of us would have expected him to support internally the Tories strategy.  When it was implied that he did, he correctly resigned. For that, he is to be respected and not criticized.

 It is terrifying how ill informed the census discussion has been.  The long form is required to provide a good statistical base for the analysis of policy and business decisions.  For example, most of us use the industrial and occupational data in the Labour Force Survey on a very regular basis (i.e., monthly).  The quality of that data is directly dependent on the long form.  The individual information is not important except in that it supports accurate aggregates available for small local areas such as smaller cities, e.g. Brantford etc.  Accurate data facilitate good decisions for business and government.

The coercive element of the census is the price of citizenship.  The American Community Survey is also mandatory in the US for reasons of efficient accurate cost-effective data.  We do not have compulsory registration, identity cards or other forms of personal tracking in Canada as in many European countries.  As I write this letter, I am in  Germany where the first portion of the automobile license identifies the city of registration and where identity cards are a requirement.  Those restrictions are considered appropriate for persons living here.  A mandatory census is a much better tradeoff for good government.

In terms of privacy, Canadians routinely surrender privacy to their merchants through loyalty cards or even electronic purchases (remember tjx/winners).  Statistics Canada, unlike far too many retailers, has never had a privacy breach and goes to extreme lengths to see that none outside of census operations sees the personal records.  As a heavy data user, I am extremely aware of the costs and complications of their privacy protection policy.  The personal data are not even available to Mr. Clement's ministry.  The privacy concerns are trumped up by the Tories to appeal to the lowest common denominator of voter which is their constituency.

Going ahead with the destruction of the census-based data systems will result in enormous gaps.  We will not be able to compare data appropriately.  None of us will be comfortable comparing observations of a new survey for several cycles.  In other words, we will have enough experience to judge the quality of the proposed National Household Survey in perhaps 20 years.  Since it is a very expensive process to introduce a survey, far more than any savings from dropping the long form, the Tories should stop this nonsense now. If the Tories do not revisit this decision, it can only confirm their status as inappropriate managers of our country.

Census 2016

More Noise from Statistics Canada

1. The government has directed Statistics Canada to look at even more options for changing the Census in subsequent iterations.  Since they have broken the important continuity of the data systems, this probably makes sense.

2. The use of registration data is one of the options discussed for future changes to the Census.  Of course, in a CP story, Jennifer Ditchburn correctly pointed out that we don't have mandatory registration by place of residence and we do not require significant locational and demographic information from all of our residents.  The use of identity cards and mandatory registration does not scare me personally. Most of the people who would object to such things probably provide more information to their retailers or other loyalty card systems.  Statistics Canada has made innovative use of administrative data and should be encouraged to continue to do so.  Additional information can always be used to improve quality and timeliness.

3. My biggest fear with such data is that we have too many jurisdictions who insist on managing such systems themselves.  Trying to sort out 14 jurisdictional games will just increase costs and lower quality.  To many of our health care datasets are forced to exclude Quebec because of issues of integration and control of administrative data systems.

4. Mention is often made of the US use of an post-censal survey.  It should be noted that the U.S, found that this must be mandatory to get an adequate response rate.

JCI Census Notes

1.  The long-form is important for direct and indirect use.
   1. For most economists, the indirect use through its importance in the design, weighting and benchmarking of every household survey, public or private, is the key issue.  The household surveys such as the LFS are critical for analysis of all sorts of short- and medium-term issues.
   2. Direct usage targets location and the attributes of persons and households.  For example, it is a key source of data for commuting patterns and housing characteristics.
2. Businesses use the data to influence the location of facilities such as shopping complexes or major plant investments.
3. Small agencies use the data to plan local services.  For example, the public library branch was closed in a small town called Drumbo near Woodstock.  A group of us started the Drumbo Opportunity Centre (www.drumbo-opportunity.ca) to replace the lost services.  We used the long-form census data to help plan the services that we would emphasize.
4. For JCI, key attributes include very detailed occupation, education, and labour force structure for specific industries and regions
   1. In the last year, JCI used that directly in a detailed study for the Retail Council of Canada and HRSDC.   Such studies are instrumental in work force planning to develop training and retention strategies. 
   2. In April 2009, JCI completed the most recent in a series of studies, for Minister Clement’s ministry, Industry Canada, on knowledge-based activities in detailed industries in Canada that was directly reliant on special tabulations from the long-form data.
5. The data can also be used to guide the development and evaluation of government activities.  For these activities, the local data is particularly important.
   1. In the spring of this year, JCI completed a study on the impact of government program at the local area utilizing census household income measures down to the township level.
   2. Last summer, JCI developed a strategic plan report for the Toronto Central LHIN that used the long-form census data to incorporate some statistics on the socio-economic factors affecting health service demand.
6. Response rates have been shown to fall if a survey is transitioned from mandatory to voluntary.  Achieving good response rates would require a high rate of follow up to the mail survey which is very expensive and very intrusive and difficult to target.  This was the issue experienced with the U.S. trial of making the American Community Survey voluntary.
7. The government has actually planned to increase the number of people they will hand out the survey to, from 20% to 33%. This does not compensate for the drop in the number of respondents because the survey is no longer mandatory. A larger sample cannot compensate for biases in the responses rates from specific groups.  The result will simply be more responses from the groups that are likely to react well to the census.
8. Groups with specific challenges such as those with multiple jobs, language challenges or other issues have been shown least likely to respond to general surveys.  Such groups include everyone from new immigrants, to single mothers, the elderly and the isolated. Yet, those are the people who may place differential demands on services and need to be identified.  For example, public health in Toronto used the census to help them identify populations at risk during the SARS epidemic.
9. Comparisons between survey observations are a critical aspect of any analysis program.  It will be technically impossible to comfortable with comparisons between the first cycle of the planned NHS and the previous censuses.  When we have a few cycles (i.e. 20 years) of NHS, we might be comfortable with its properties but analysis in the intervening period will be very much like working in a fog, a fog of unknown information density.
10. The coercive element of the census is the price of citizenship. The American Community Survey is also mandatory in the US for reasons of efficient accurate cost-effective data. We do not have compulsory registration, identity cards or other forms of personal tracking in Canada as in many European countries which can substitute for some aspects of a census.  Nothing substitutes for the broad analytical data available in the long form.  There are not other sources for accurate and complete information.
11. In terms of privacy, Canadians routinely surrender privacy to their merchants through loyalty cards or even electronic purchases (remember the   tjx/winners privacy breach).  Statistics Canada, unlike far too many retailers, has never had a privacy breach and goes to extreme lengths to see that none outside of census operations sees the personal records.  The individual personal data records are not even available to Mr. Clement's ministry. As a heavy data user, I am extremely aware of the costs and complications of their privacy protection policy.  Their policies work to provide secure aggregate analytical data to business and government and should be preserved.
12. Statistics Canada already makes extensive use of administrative computer files including the various files within the taxation system and other administrative databases.  In that area, they probably lead the world in the efficient exploitation of such resources.  Private business databases are less useful for statistical purposes because of their necessary proprietary nature, their incompleteness and their general inaccuracy.  Naïve suggestions to use records such as credit card data clearly have not worked with those sources.  Analyses of such data while useful are NOT a substitute for clean censuses and surveys.  The literature on business data mining is rich with discussions of how to surmount these issues for private purposes.  However, for public purposes and for good analysis, we require completeness which only the major statistical vehicles can provide. With the census, we are doing more than count people.  We are evaluating attributes that are not available in commercial databases.
13. The response of the Canadian Association for Business Economics is found here.