Download Methods To Foster Transparency And Reproducibility Of Federal Statistics Proceedings Of A Workshop 1St Edition And Medicine Engineering National Academies Of Sciences online ebook texxtbook full chapter pdf

Methods to Foster Transparency and
Reproducibility of Federal Statistics:

Proceedings of a Workshop 1st Edition
And Medicine Engineering National
Academies Of Sciences
Visit to download the full and correct content document:
https://ebookmeta.com/product/methods-to-foster-transparency-and-reproducibility-of-
federal-statistics-proceedings-of-a-workshop-1st-edition-and-medicine-engineering-na
tional-academies-of-sciences/
More products digital (pdf, epub, mobi) instant
download maybe you interests ...
Approaches to the Development of Character Proceedings

of a Workshop 1st Edition And Medicine Engineering
National Academies Of Sciences
https://ebookmeta.com/product/approaches-to-the-development-of-
character-proceedings-of-a-workshop-1st-edition-and-medicine-
engineering-national-academies-of-sciences/
Communicating Clearly about Medicines Proceedings of a

Workshop 1st Edition And Medicine Engineering National
https://ebookmeta.com/product/communicating-clearly-about-
medicines-proceedings-of-a-workshop-1st-edition-and-medicine-
Neuroscience Trials of the Future Proceedings of a

https://ebookmeta.com/product/neuroscience-trials-of-the-future-
proceedings-of-a-workshop-1st-edition-and-medicine-engineering-
national-academies-of-sciences/
Implementation of Lung Cancer Screening Proceedings of

a Workshop 1st Edition And Medicine Engineering
https://ebookmeta.com/product/implementation-of-lung-cancer-
screening-proceedings-of-a-workshop-1st-edition-and-medicine-
Using Technology to Advance Global Health Proceedings
of a Workshop 1st Edition And Medicine Engineering
https://ebookmeta.com/product/using-technology-to-advance-global-
health-proceedings-of-a-workshop-1st-edition-and-medicine-
Sustainable Diets, Food, and Nutrition: Proceedings of

a Workshop 1st Edition And Medicine Engineering
https://ebookmeta.com/product/sustainable-diets-food-and-
nutrition-proceedings-of-a-workshop-1st-edition-and-medicine-
Globalization of Defense Materials and Manufacturing

Proceedings of a Workshop 1st Edition And Medicine
Engineering National Academies Of Sciences
https://ebookmeta.com/product/globalization-of-defense-materials-
and-manufacturing-proceedings-of-a-workshop-1st-edition-and-
medicine-engineering-national-academies-of-sciences/
Adaptability of the US Engineering and Technical

Workforce Proceedings of a Workshop 1st Edition And
Medicine Engineering National Academies Of Sciences
https://ebookmeta.com/product/adaptability-of-the-us-engineering-
and-technical-workforce-proceedings-of-a-workshop-1st-edition-
and-medicine-engineering-national-academies-of-sciences/
Statistical Challenges in Assessing and Fostering the

Reproducibility of Scientific Results Summary of a
https://ebookmeta.com/product/statistical-challenges-in-
assessing-and-fostering-the-reproducibility-of-scientific-
results-summary-of-a-workshop-1st-edition-and-medicine-
Methods to Foster
Transparency and
Reproducibility
of Federal Statistics
PROCEEDINGS OF A WORKSHOP
Michael L. Cohen, Rapporteur
Committee on National Statistics
Division of Behavioral and Social Sciences and Education
THE NATIONAL ACADEMIES PRESS

Washington, DC
www.nap.edu
THE NATIONAL ACADEMIES PRESS 500 Fifth Street, NW
Washington, DC 20001
This activity was supported by contracts between the National Academy of

Sciences and the National Science Foundation (#SES-1024012). This grant
provides support for the work of the Committee on National Statistics from a
consortium of federal agencies. Any opinions, findings, conclusions, or
recommendations expressed in this publication do not necessarily reflect the views
of any organization or agency that provided support for the project.
International Standard Book Number-13: 978-0-309-48629-3

International Standard Book Number-10: 0-309-48629-7
Digital Object Identifier: https://doi.org/10.17226/25305
Epub ISBN: 978-0-309-48632-3
Additional copies of this publication are available for sale from the National
Academies Press, 500 Fifth Street, NW, Keck 360, Washington, DC 20001; (800)
624-6242 or (202) 334-3313; http://www.nap.edu.
Copyright 2019 by the National Academy of Sciences. All rights reserved.
Printed in the United States of America
Suggested citation: National Academies of Sciences, Engineering, and Medicine.

(2019). Methods to Foster Transparency and Reproducibility of Federal Statistics:
Proceedings of a Workshop. Washington, DC: The National Academies Press. doi:
https://doi.org/10.17226/25305.
The National Academy of Sciences was established in 1863 by
an Act of Congress, signed by President Lincoln, as a private,
nongovernmental institution to advise the nation on issues related to
science and technology. Members are elected by their peers for
outstanding contributions to research. Dr. Marcia McNutt is
president.
The National Academy of Engineering was established in 1964

under the charter of the National Academy of Sciences to bring the
practices of engineering to advising the nation. Members are elected
by their peers for extraordinary contributions to engineering. Dr. C.
D. Mote, Jr., is president.
The National Academy of Medicine (formerly the Institute of

Medicine) was established in 1970 under the charter of the National
Academy of Sciences to advise the nation on medical and health
issues. Members are elected by their peers for distinguished
contributions to medicine and health. Dr. Victor J. Dzau is president.
The three Academies work together as the National Academies of

Sciences, Engineering, and Medicine to provide independent,
objective analysis and advice to the nation and conduct other
activities to solve complex problems and inform public policy
decisions. The National Academies also encourage education and
research, recognize outstanding contributions to knowledge, and
increase public understanding in matters of science, engineering,
and medicine.
Learn more about the National Academies of Sciences, Engineering,

and Medicine at www.nationalacademies.org.
Consensus Study Reports published by the National Academies of
Sciences, Engineering, and Medicine document the evidence-based
consensus on the study’s statement of task by an authoring
committee of experts. Reports typically include findings, conclusions,
and recommendations based on information gathered by the
committee and the committee’s deliberations. Each report has been
subjected to a rigorous and independent peer-review process and it
represents the position of the National Academies on the statement
of task.
Proceedings published by the National Academies of Sciences,

Engineering, and Medicine chronicle the presentations and
discussions at a workshop, symposium, or other event convened by
the National Academies. The statements and opinions contained in
proceedings are those of the participants and are not endorsed by
other participants, the planning committee, or the National
Academies.
For information about other products and activities of the National

Academies, please visit www.nationalacademies.org/about/whatwed
o.
STEERING COMMITTEE ON TRANSPARENCY AND
REPRODUCIBILITY OF FEDERAL STATISTICS
WILLIAM EDDY (Chair), Department of Statistics, Carnegie Mellon

University
GEORGE ALTER, Inter-university Consortium for Political and Social
Research
JEREMY FREESE, Department of Sociology, Stanford University
GRAHAM KALTON, Westat
AUDRIS MOCKUS, Min H. Kao Department of Electrical Engineering &
Computer Science, University of Tennessee, Knoxville
SUSAN OFFUTT, U.S. Government Accountability Office (retired)
MICHAEL L. COHEN, Study Director

MICHAEL SIRI, Program Associate
COMMITTEE ON NATIONAL STATISTICS
ROBERT M. GROVES (Chair), Office of the Provost, Department of

Mathematics and Statistics, and Department of Sociology,
Georgetown University
MARY ELLEN BOCK, Department of Statistics, Purdue University
(emeritus)
ANNE C. CASE, Woodrow Wilson School of Public and International
Affairs, Princeton University
MICHAEL CHERNEW, Department of Health Care Policy, Harvard
Medical School
JANET M. CURRIE, Woodrow Wilson School of Public and
International Affairs, Princeton University
DONALD A. DILLMAN, Social & Economic Sciences Research Center,
Washington State University
DIANA FARRELL, JPMorgan Chase Institute, Washington, DC
DANIEL KIFER, Department of Computer Science and Engineering,
The Pennsylvania State University
THOMAS L. MESENBOURG, U.S. Census Bureau (retired)
SARAH M. NUSSER, Office of the Vice President for Research, and
Department of Statistics, Iowa State University
COLM O’MUIRCHEARTAIGH, Harris School of Public Policy, The
University of Chicago
JEROME P. REITER, Department of Statistical Science, Duke
University
JUDITH A. SELTZER, Department of Sociology, University of
California, Los Angeles
C. MATTHEW SNIPP, Department of Sociology, Stanford University
BRIAN A. HARRIS-KOJETIN, Director

CONSTANCE F. CITRO, Senior Scholar
Acknowledgments
We would first like to thank the National Science Foundation

(NSF), particularly Amy Friedlander, deputy director, Office of
Advanced Cyberinfrastructure, and her colleague, Patricia Knezek,
senior adviser, Mathematical and Physical Sciences Directorate (until
January 2018) for their support of this project, and for Amy
Friedlander’s presentation, which kicked off the workshop. In
addition, John Gawalt, director (until April 2018), NSF National
Center for Science and Engineering Statistics, attended the
workshop and provided helpful advice on the importance of this
topic for the agencies in the federal statistical system.
We are so very indebted to the presenters, who prepared
extremely useful presentations and were candid about current
practices at their institutions: John Abowd (U.S. Census Bureau),
David Barraclough (Organisation for Economic Co-operation and
Development), Wesley Basel (U.S. Census Bureau), Peter Brodie (UK
Office for National Statistics), Michaela Denk (International Monetary
Fund), John Eltinge (U.S. Census Bureau), Daniel Gillman (Bureau of
Labor Statistics), Hermann Habermann (Committee on National
Statistics), Brian Harris-Kojetin (Committee on National Statistics),
Sarah Henry (UK Office for National Statistics), H.V. Jagadish
(University of Michigan), Ruth Ann Killion (U.S. Census Bureau), Tom
Louis (Johns Hopkins University), Jennifer Madans (National Center
for Health Statistics), Peter Miller (U.S. Census Bureau), Juan Muñoz
(Instituto Nacional de Estadística y Geografía), Eric Rancourt
(Statistics Canada), Robert Sienkiewicz (U.S. Census Bureau), Sally
Thompson (Bureau of Economic Analysis), and Lars Vilhuber (Cornell
University).
Finally, we are also indebted to John Eltinge (U.S. Census Bureau)
for helpful comments on the structure of the workshop and to
Hermann Habermann, Nancy Kirkendall, and Michael Siri (Committee
on National Statistics) for contributing to the planning meeting of the
steering committee in October 2016 and workshop in June 2017.
Furthermore, we thank Yvonne Wise, Eugenia Grohman, and Kirsten
Sampson Snyder, staff members of the Division of Behavioral and
Social Sciences and Education, for their efforts in editing, review,
and report preparation.
This Proceedings of a Workshop was reviewed in draft form by
individuals chosen for their diverse perspectives and technical
expertise. The purpose of this independent review is to provide
candid and critical comments that will assist the National Academies
of Sciences, Engineering, and Medicine in making each published
proceedings as sound as possible and to ensure that it meets the
institutional standards for quality, objectivity, evidence, and
responsiveness to the charge. The review comments and draft
manuscript remain confidential to protect the integrity of the
process.
We thank the following individuals for their review of this
Proceedings: David Barraclough, Statistics and Data Directorate,
Smart Data Practices and Solutions Division, Organisation for
Economic Co-operation and Development, and Patrick J. Cantwell,
Sampling/Estimation, U.S. Census Bureau.
Although the reviewers listed above provided many constructive
comments and suggestions, they were not asked to endorse the
content of this Proceedings nor did they see the final draft before its
release. The review of this Proceedings was overseen by Colm
O’Muircheartaigh, Harris School of Public Policy, The University of
Chicago. He was responsible for making certain that an independent
examination of this proceedings was carried out in accordance with
standards of the National Academies and that all review comments
were carefully considered. Responsibility for the final content rests
entirely with the rapporteur and the National Academies.
Michael L. Cohen, Study Director
Steering Committee on Transparency and
Reproducibility of Federal Statistics
Contents
1 Introduction
2 Existing Guidelines Related to Transparency
3 Benefits and Costs of Transparency: Views from Three Statistical
Agencies
4 Benefits and Costs of Transparency: Views from the United
Kingdom and Canada
5 Two U.S. Examples: SAIPE and LEHD
6 Operationalizing Transparency
7 Summarizing Day 1
8 Standards for Metadata and Work Processes
9 Possible Next Steps
Appendixes
A Workshop Agenda
B List of Participants
1
Introduction
In 2014, the National Science Foundation (NSF) provided support

to the National Academies of Sciences, Engineering, and Medicine
for a series of Forums on Open Science in response to a
government-wide directive to support increased public access to the
results of research funded by the federal government. The forums
were successful in raising a number of important issues concerning
the advantages of open science and presenting ideas to support
greater openness in scientific enterprises. However, the breadth of
the ideas precluded a focus on any specific topic or discussion about
how to improve public access.
With continuing support from NSF, the Committee on National
Statistics (CNSTAT) organized the Workshop on Transparency and
Reproducibility of Federal Statistics with the following statement of
task:
An ad hoc steering committee will organize and conduct a public workshop on
key aspects of transparency and reproducibility in federal statistics, including
data access, archiving, and documentation, as a follow-on activity to the NSF-
funded Forum on Open Science at the National Academies of Sciences,
Engineering, and Medicine. The workshop will help focus discussion of issues
surrounding the credibility and transparency of federally funded scientific
digital data in a manner that will help not only federal statistical agencies, but
also other federal agencies that fund original data collection for scientific use.
A rapporteur will prepare a proceedings that summarizes the workshop
presentations and discussions.
For the workshop, CNSTAT defined transparency to mean that one

can generally know how something was done and reproducibility to
mean that one could independently recreate the process that led to
the creation of a given statistical data product. The overall goal of
the workshop was to develop some understanding of what principles
and practices are, or would be, supportive of making federal
statistics more understandable and reviewable by both agency staff
and the public.
The workshop was organized around eight broad questions:
1. What official federal guidance or standards currently exist to

provide assistance to the federal statistical system?
2. What guidance or standards are used by foreign national
statistical agencies that the U.S. federal statistical system could
learn from?
3. What are the benefits and costs of greater or lesser degrees of
transparency for a federal statistical agency, and how is this
tradeoff affected by the growing number of nonfederal entities
that are attempting to produce federal statistical system-type
data products?
4. For complicated statistical data products, what should and
should not be documented, given that some details may be of
little interest to outside analysts or the public?
5. What data, metadata, paradata, and assessments of data
quality should be archived so that they can be retrieved by
others?
6. How do these answers change given that there are now various
nonsurvey sources of data, including administrative records and
Internet transaction data sources?
7. How do these answers change for agency staff and for people
outside an agency?
8. What tools currently exist, either domestic or foreign, that can
expedite either the documentation of methods or the archiving
of data sources?
These questions come at a time when various changes are

exerting pressure on the federal statistical system, as detailed in
several recent CNSTAT reports.1 First, survey response rates are
decreasing for all types of survey instruments, so there is greater
interest in the use of alternative sources of information, especially
administrative records. Second, while statistical agencies were once
primarily interested in producing totals, means, percentages, and
cross-tabulations for various geographic or demographic subsets of
the population, statistical agencies now make much greater use of
statistical modeling. Third, private entities are increasingly producing
estimates of quantities of national interest. And fourth, there is
increasing interest in a number of tools that assist in documentation
and archiving of statistical data along with associated metadata and
paradata.
In the workshop’s opening session, Amy Friedlander (National
Science Foundation) described NSF’s work in the area. The agency
has been hosting a series of events concerning data, data
management, public access to data, reproducibility, and questions
that derive from a concern about the integrity of the scientific
research process. She noted that this is by necessity a cross-
disciplinary issue and that the federal statistical agencies could learn
a great deal by including computer scientists in the discussions.
Friedlander further noted that the reliability of the information
provided, which is supported by transparency, is crucially important
because the data products of the federal statistical system are used
for important policy programs. She added that another key aspect of
this issue is the need to be aggressive toward maintaining data
confidentiality and other aspects of data security and data integrity.
In addition, Friedlander pointed out, the reuse of historical methods
necessitates the proper understanding of the context of the data
and, therefore, including metadata in a data archive is important.
Furthermore, she said, documentation of methods gets into the
issue of retention of the form of instrumentation (and mode bias)
and methodology workflow.
This Proceedings has been prepared by the workshop rapporteur
as a factual summary of what occurred at the workshop. The
planning committee’s role was limited to planning and convening the
workshop. The views contained in this Proceedings are those of
individual workshop participants and do not necessarily represent
the views of all workshop participants, the planning committee, or
the National Academies.
__________________
1 See National Academies of Sciences, Medicine, and Engineering. (2017).
Innovations in Federal Statistics: Combining Data Sources While Protecting Privacy.
Robert M. Groves and Brian A. Harris-Kojetin, Editors. Panel on Improving Federal
Statistics for Policy and Social Science Research Using Multiple Data Sources and
State-of-the-Art Estimation Methods, Committee on National Statistics, Division of
Behavioral and Social Sciences and Education. Washington, DC: The National
Academies Press.
Also see National Research Council. (2013). Nonresponse in Social Science
Surveys: A Research Agenda. Roger Tourangeau and Thomas J. Plewes, Editors.
Panel on a Research Agenda for the Future of Social Science Data Collection,
Committee on National Statistics, Division of Behavioral and Social Sciences and
Education. Washington, DC: The National Academies Press.
2
Existing Guidelines Related to

Transparency
PETER MILLER: AAPOR TRANSPARENCY INITIATIVE

The goal of the first session was to consider existing guidelines or
standards regarding transparency in the U.S. federal statistical
system, specifically those on surveys from the American Association
for Public Opinion Research (AAPOR) and the standards for the
federal statistical system issued by the Office of Management and
Budget (OMB). Peter Miller (U.S. Census Bureau), who was AAPOR
president from 2009 to 2010, began with some background on
AAPOR and a summary of its Transparency Initiative.
In 1947, AAPOR was founded as an organization of individual
survey research professionals with the goal of advancing survey
research. From its inception, AAPOR included government,
academic, and commercial members and was intended to serve not
only public opinion researchers but all survey professionals. Miller
explained that standards of methodological disclosure were
discussed for many years before being formally adopted in 1967.
Subsequently, procedures for enforcement of the disclosure
standards were adopted in 1975. Disclosure was the focus of the
standards and the enforcement, rather than procedures or
performance. Following the adoption of the standards, the
association set up a mechanism to receive complaints about failure
by survey organizations to disclose methods and initiated a set of
procedures to investigate such complaints. A standards case could
result in the association publicly censuring an organization for failing
to disclose useful methodological details of a study that it had
released. It is important to mention, Miller said, that the standards
and procedures are “living documents,” having been revised on
multiple occasions since they were adopted.
In its requirements for disclosure, Miller explained, AAPOR
distinguishes between items that can be reported immediately when
a survey is published and those that can be reported within 30 days.
The items that have to be reported immediately include who
sponsored, conducted, and funded the research; the exact wording
of survey questions; the specification of the population of study; the
geographic location of interest; a description of the sample frame;
the sample design; the sample size and error; weighting and cluster
adjustments; and method(s) and dates of data collection. The types
of items that have to be reported within 30 days are the instructions
for interviewers and respondents; relevant stimuli (such as show
cards); the coverage of the sampling frame; methods of panel
recruitment (if the sample is drawn from a pre-recruited panel);
sample design details (e.g., eligibility rules, screening procedures,
and oversampling); use of incentives; weighting details; information
for calculating response rates; and information on efforts to validate
the data.
In the years since the standards were adopted, AAPOR has been
involved in a number of standards cases, Miller said, which put a
burden on the organization because of the time needed to
adjudicate them and because they can become legally contentious.
He discussed two cases that clarified for him the importance of
disclosure and the difficulties faced by those who enforce it. First, in
the late 1990s, Arianna Huffington, a journalist, asked the major
public opinion polling organizations to report on their response rates.
One major national organization complained that doing so would be
difficult. Miller said he was surprised to learn that it—and other
organizations—did not have a routine system for documenting such
information. Educating organizations about embedding transparency
in their normal workflows can be useful. Some years later, AAPOR
had to process a number of standards cases connected to 2008 pre-
election polling, one of which led to censuring an organization for
refusal to disclose methodological information. (The organization
was subsequently cited by Nate Silver from FiveThirtyEight for
falsifying data.)1 The other standards cases were resolved once
organizations were able to pull information together from records.
The experience reinforced belief in the need for education on
routinized transparency and on the importance of public censure for
organizations that refused to adhere to this key norm.
When he became AAPOR president, Miller proposed that, in
addition to maintaining sanctions for failure to disclose, AAPOR
should have a program that incentivized systemic, routine disclosure
practices. Thus, the Transparency Initiative was launched in 2010.
After long discussions and pretests, organizations wishing to join the
initiative were vetted and enrolled beginning in 2014. The initiative
included training modules on disclosure practices, along with
periodic auditing and feedback. AAPOR’s public recognition of the
member organizations includes a seal of approval and media
releases. The original concept envisioned an archive of
methodological disclosures, but due to costs, it remains a future
possibility.
Currently, Miller said, the AAPOR Transparency Initiative has nearly
90 member organizations, large and small, from all sectors of survey
research. AAPOR assumes all of the program expenses, and there
are no membership fees to join. Investment in their own
documentation processes could be useful to organizations. AAPOR
conducts audits every 2 years to assess compliance, and recent
audits have described the need for enhanced and sustained
continuing education for survey organizations since their staff
members turn over, Miller said. He noted that, over time,
membership may change as audits continue. Organizations may drop
out, or AAPOR may decide that they are not fulfilling the initiative’s
requirement and cease to recognize them.2
Federal statistical agencies are not members of the initiative, but
some large contractors of the agencies are. Initially, some
contractors questioned whether the requirements of the
Transparency Initiative would either conflict with or duplicate federal
standards of disclosure. To allay such concerns, in 2015 the Federal
Committee on Statistical Methodology (FCSM) sent a letter to AAPOR
detailing the federal commitment to transparency in a variety of
standing policies,3 noting that AAPOR’s Transparency Initiative
requirements align with the disclosure requirements of statistical
agencies.
John Eltinge (U.S. Census Bureau) added that FCSM is currently
exploring questions of enhancing transparency for reporting on the
integration of multiple data sources, and this workshop is salient to
that discussion.
Miller stressed that the level of disclosure in the Transparency
Initiative is incomplete in the sense of achieving reproducibility or
replication. He said that the AAPOR standards create the opening for
a discussion and further public exploration of methodology, which
might need to become much more comprehensive to fully achieve
that goal.
In summary, Miller presented three points for discussion
concerning transparency and reproducibility: (1) AAPOR’s experience
with the Transparency Initiative to date suggests that public
recognition and education can incentivize documentation and
disclosure of methodological elements; (2) the level of disclosure
required in AAPOR’s Transparency Initiative is one step toward what
may be required for replication or reproducibility; and (3) the
process of implementing an effort like the Transparency Initiative
may inform the process of establishing norms and methods for
replication or reproducibility.
BRIAN HARRIS-KOJETIN: RELEVANT OMB DIRECTIVES

Turning to OMB, Brian Harris-Kojetin (Committee on National
Statistics [CNSTAT]), former senior statistician in OMB’s Statistical
and Science Policy Office, provided details for the agency’s
standards, regulations, and guidance related to transparency. He
began by noting two relevant documents: (1) the Paperwork
Reduction Act (PRA), which requires public comment periods and
documentation for any new survey data collection; and (2) OMB
policy directive No. 2, Standards and Guidelines for Statistical
Surveys, which directs what is essential for survey documentation.
Harris-Kojetin explained that the purpose of the PRA was to improve
the quality and utility of information that is useful to the federal
government and to reduce the resulting paperwork burden on the
public. Given this authority, OMB reviews agency requests to collect
information to ensure that activities have practical utility, reduce
duplication, meet a specific agency need, and minimize burden. He
detailed the five steps of the PRA process: (1) an agency’s
publication in the Federal Register of a notice of the intent to submit
a survey proposal to OMB, which has a 60-day comment period; (2)
revisions by the agency to address public comments from that
notice; (3) the agency’s publication of a notice in the Federal
Register that it is submitting to OMB a request for approval, which
has a 30-day comment period; (4) OMB review and feedback to the
agency; and (5) OMB approval.
To demonstrate compliance with the PRA, Harris-Kojetin explained,
an agency has to provide the following information to OMB:4 (1)
type of submission, (2) title, (3) purpose, (4) number of
respondents, (5) estimate of burden, (6) frequency of collection, and
(7) use of statistical methods. He also provided details about the
necessary two-part supporting statement: part A covers justification
for information collection, and part B covers statistical methods.
In part A, an agency is asked to provide the necessity of
information collection, the use to which the information will be
applied, the use of information technology, efforts to identify
duplication, minimizing burden on small entities, consequences if the
information is not collected, a record of consultation with the public
and public comments, assurances of confidentiality, questions of a
sensitive nature, time and monetary burden estimates, total annual
cost burden, annual cost to the federal government, explanation of
any program changes, and plans for publication and tabulation. In
part B, which covers statistical methods, an agency is asked to
provide the sampling universe and respondent selection, procedures
for collecting information, methods to maximize response, tests of
procedures, and contacts for statistical aspects and data collection.5
In addition to documents related to the PRA, in 2006 OMB issued
Standards and Guidelines for Statistical Surveys,6 which includes 20
standards, Harris-Kojetin said. The standards are supplemented with
guidelines or best practices to help the statistical agencies interpret
and fulfill the goals of the standards. The standards call for details
on seven elements: (1) the development of concepts, methods, and
design, (2) the method of data collection, (3) the method for
processing and editing data, (4) the production of estimates and
projections, (5) data analysis, (6) procedures for review, and (7)
dissemination of information products.
Turning to the issue of survey documentation, Harris-Kojetin
pointed out that the standards call for an agency to provide
documentation that includes information both to understand how to
analyze every survey’s data and to replicate and evaluate each
survey’s results. That documentation must be readily accessible to
users, he said, unless it is necessary for access to be restricted to
protect confidentiality. There are associated guidelines to this
requirement, he noted, which are detailed in Guideline 7.3.1.7 This
information must be made available within 30 days.
OMB’s survey documentation standards also contain three other
detailed guidelines, Harris-Kojetin said. Guideline 7.3.2 concerns
survey replication and evaluation; Guideline 7.3.3 requires a periodic
evaluation report, such as a methodology report, that itemizes all
sources of identified error; and Guideline 7.3.4 calls for the retention
of all survey documentation consistent with appropriate federal
records disposition and archival policy.8
Harris-Kojetin pointed out that OMB’s standards and guidelines are
clearly survey-centric, but not exclusively so. For example, he noted,
Guideline 4.1.6 concerns the development of estimates and
projections, requiring that agencies document the methods and
documents used; ensure objectivity, utility, transparency, and
reproducibility of the estimates and projections; and archive the data
and models.9
In addition, Harris-Kojetin noted that OMB has recently provided
additional guidance regarding administrative records. That guidance
states that, to the extent possible, the previously mentioned
standards also cover the compilation of statistics based on
information collected from administrative records. In addition, he
noted, OMB memo M-14-06 provides guidance on providing and
using administrative data for statistical purposes.10 However, Harris-
Kojetin said that there has not been a systematic effort to develop
the same kind of guidelines and standards and regulations for the
use of administrative records and for the products of complicated
statistical models as there has been for surveys and the usual survey
estimates.
A participant asked Harris-Kojetin whether, when OMB established
these standards, there was a problem when providing the
documentation for electronic survey instruments, given the
complicated skip patterns and other characteristics. Harris-Kojetin
responded that OMB wrestled with that issue, since there is no paper
form for computer-assisted telephone interviewing (CATI) and
computer-assisted personal interviewing (CAPI) instruments. He said
one way to document such an instrument is to provide the
instrument itself, along with the software to run it, but this raises the
issue of what to do about the documentation for edits and
imputations. He said that often there is no codebook or an
annotated code; in some cases, only a few people really know what
is done and its justification, and other people just keep it going.
Jennifer Madans (National Center for Health Statistics [NCHS])
agreed that this is often the case. The documentation is often
incomplete, and one has to have someone explain what is done and
why it is done that way. What is documented is a black box that
varies greatly by survey and agency, she said. She mentioned
another standards group that she served on concerning cognitive
aspects of survey interviewing. The majority of the standards were
about documentation, which is often missing, and how and what to
document. However, there still is no consensus about how
comprehensive the documentation should be and where it should be
posted.
David Barraclough (OECD) noted that it is a national initiative and
asked if there is any kind of effort to promote international
alignment on the use of standards and recommendations with other
statistical offices. Also, he asked, how are the OMB guidelines being
implemented? Since publication in 2006, are there efforts to
continuously renew them?
In response, Miller said that AAPOR is really an international
organization with many foreign members. Therefore, AAPOR does
have reach outside the United States. He noted that it has had
discussions with the European Society for Opinion and Marketing
Research, although AAPOR’s work is more related to procedures and
performance work. He said there is no concrete linkage between
AAPOR and the guidelines of other organizations, though there have
been attempts at harmonization.
In response to the question about the OMB standards, Harris-
Kojetin said that the PRA requires that ongoing data collections be
submitted for renewal every 3 years, so there is definitely follow-up
on certain aspects of the data collection. However, he noted, in his
tenure at OMB he never asked for all 18 items of survey
documentation, since a fair number of things are subsumed in
agencies’ general documentation. He explained that OMB’s focus is
on sources of error, especially the results of nonresponse bias
analyses, and the evaluations that have been carried out concerning
the quality of the estimates. In terms of updating over time, Harris-
Kojetin noted that when the OMB standards were promulgated in
2006, there had not been any revisions for 30 years. He said that
there is no current plan to systematically update them; however,
OMB is aware of some areas that could be looked at in the near
future. One such area is cognitive interviewing.
A participant pointed out that many surveys are one-off surveys,
and so the idea of improving the next repetition is not relevant.
Another participant asked Harris-Kojetin whether the PRA is the
only federal regulation or if there were other contexts outside of
OMB for which there are regulations on similar topics. Harris-Kojetin
responded that OMB may not be the exclusive place for such
regulations.
Miron Straf (Virginia Tech) asked about a situation in which
surveys need to make quick changes, especially pilot surveys, and
the PRA has been criticized for making things difficult under such
circumstances. Harris-Kojetin responded that he did not know of a
single federal agency that would want to quickly change its CAPI or
CATI instrument 1 week before it was going to the field or while in
the field. There are, however, provisions for agencies to make small
changes without having to go through the 60-day public comment
period. There are also generic clearances for doing cognitive testing
and pretesting, he said, as well as what are called nonsubstantive
changes for small modifications and provisions for emergency
procedures.
The next question concerned OMB Memo M-14-06. Harris-Kojetin
said that when dealing with administrative records, each agency is
doing what it believes is in compliance since guidance has not
stabilized. He was then asked whether there are some best practices
that he has seen regarding documentation. He answered that he has
not seen much documentation of administrative records. He asked
whether Madans or Eltinge had seen this at NCHS and the U.S.
Census Bureau, respectively.
Madans responded that several FCSM groups are looking into
administrative data and how to use them, document their use,
assess their quality, and to consider what paradata and metadata
mean in the context of administrative records. There was a big push
to carry out quality evaluation of administrative data, but there is still
no agreement about how to do this. At the U.S. Department of
Health and Human Services, Madans said, staff have been using
administrative data for a long time, for example, from the Medicare
program. In the early days, the collection of Medicare data was not
well documented, but things are much better now. However, Madans
said, there has not been sufficient thought concerning best practices
about the use of these records. As a result, there are some examples
where problems occurred, but there have also been some successes.
The big question, she said, is what the value is of an administrative
records database for a specific analytic problem.
Eltinge described the administrative records quality checklist
developed by the Federal Committee on Statistical Methodology’s
(FCSM’s) Administrative Records Subcommittee, which is available on
the Web.11 At the same time, FCSM is looking at the integration of
multiple data sources.
Referring back to an earlier question about making changes in the
field, John Abowd (U.S. Census Bureau) said that the question was
likely referring to adaptive design, which means that the procedure
in the field changes in ways that are not transparent. He said that
there might be standards for adaptive design, and he hopes that the
U.S. Census Bureau can start to encode some of these in its
procedures. Turning to a broader issue, he said that although
standards have required agencies to store the historical metadata
associated with a survey, the standards that are used to guide how
to do this could be reexamined. In particular, he said, such
information could be linked to permanent digital object identifiers
(DOIs). Otherwise, it becomes difficult to find such historical
documents. The U.S. Census Bureau has become a member of the
Inter-university Consortium for Political and Social Research in order
to obtain their professional DOIs and services for metadata curation.
This is in reaction to the problem exemplified by the current difficulty
in finding the 1983 Survey of Income and Program Participation
metadata, and the fact that various American Community Survey
datasets exist.
The last comment in the session was from Connie Citro (CNSTAT),
noting that documentation and having good metadata allows users
to evaluate the quality of the data and know what they are dealing
with. Therefore, updating, archiving, and curation standards are
extremely important. Currently, she said, no statistical agency
regularly makes this information available to users in a single
location, so the issue is not simply what an agency does internally or
within the statistical system but also getting that information out to
external users.
__________________
1 See Dewan, S. (2009). Polling firm’s reprimand rattles news organizations. The
New York Times, October 2.
2 See https://www.aapor.org/Transparency_Initiative.htm [January 2018].
3 See http://www.aapor.org/AAPOR_Main/media/AM15/Federal-Transparency-Gu
idances_31DEC15.pdf [January 2018].
4 This submission is referred to as the RISC and OIRA Consolidated Information
System core data. Available: https://georgewbush-whitehouse.archives.gov/omb/g
ils/oira-gils.html [January 2018].
5 For details, see https://www.reginfo.gov/public [January 2018].
6 See https://obamawhitehouse.archives.gov/omb/inforeg_statpolicy [January
2018].
7 U.S. Office of Management and Budget. (2006). Standards and Guidelines for
Statistical Surveys, pp. 26–28. Available: https://www.ftc.gov/system/files/attachm
ents/data-quality-act/standards_and_guidelines_for_statistical_surveys_-_omb_-_s
ept_2006.pdf [January 2018].
8 See reference in fn. 5.
9 See reference in fn. 5
10 See https://obamawhitehouse.archives.gov/sites/default/files/omb/memoran
da/2014/m-14-06.pdf [January 2018].
11 See https://nces.ed.gov/FCSM/index.asp [January 2018].
3
Benefits and Costs of Transparency:

Views from Three Statistical Agencies
Susan Offutt (member, steering committee) began the session by

discussing broad issues of the costs and benefits of transparency,
noting that transparency and reproducibility are important to the
federal statistical system. One benefit from being transparent to
external users and researchers is that those users may suggest
improvements to federal statistical products. There are also internal
benefits to agencies of being transparent, including the facilitation of
technology transfer. However, she noted, increasing the degree of
transparency is not costless, since the costs include the time
necessary for documentation and archiving. Therefore, agencies
might consider both costs and benefits in thinking about how much
transparency to strive for.
Offutt added that this consideration is not just for survey data.
While this session will focus on survey data and the associated
statistical products, the use of other types of data sources is
increasing. She noted, for example, that the Panel on Methods for
Integrating Multiple Data Sources to Improve Crop Estimates, of
which she is a member, is studying forecasts of crop yields at the
U.S. Department of Agriculture by looking not only at survey data
but also remote sensing data that look at crop progress. To combine
the data from these different sources, statistical models are used.
The use of models gets complicated, which increases the costs of
transparency, she said, but perhaps there are gains from having an
informal external review that is possible with transparency. Thus,
one needs to understand the tension between what agencies can
afford to spend on transparency and the demands of their users. In
addition, it is important to understand whether there are
technological solutions that can make transparency more cost
efficient.
Offutt introduced the session’s three speakers, all of whom are
high-level government officials with key insights into this tradeoff:
Jennifer Madans (National Center for Health Statistics [NCHS]), Sally
Thompson (Bureau of Economic Analysis [BEA]), and John Eltinge
(U.S. Census Bureau).
JENNIFER MADANS: BENEFITS AND COSTS OF

TRANSPARENCY
Jennifer Madans began by stating that at NCHS there is a long
tradition of maximizing transparency in all of its activities. She said
she does not have a strong interest in a formal cost-benefit analysis
of the value of transparency because there are very clear benefits to
the agency. However, there are costs that grow as the data and
methods in question become more specific and detailed, so proactive
decisions could be made about how much information should be
made publicly available. Even with the efforts NCHS makes to
maximize transparency, every now and then there is a user who
thinks the agency has not gone far enough.
Overall, Madans said, NCHS views transparency as one of the
necessary costs of being a federal statistical agency. The
commitment to transparency sets federal statistical agencies apart
from other data collectors. She acknowledged that the costs to the
agency are substantial, and that as more money is spent on
transparency, there is less money that can be spent on other
activities, such as the scope and amount of data that can be
collected, the scope and number of products that can be released,
and the speed of both collection and release.
Aside from the responsibility of openness as part of the federal
statistical system, Madans said that a major benefit of transparency
to NCHS as an agency is to support agency credibility. This, in turn,
maximizes the utility of the information because it is then trusted.
The approach also protects agency independence because when
people do question a result, the first thing they do is criticize the
methods. NCHS opens the books and explains clearly what was
done.
NCHS has always had a focus on releasing as much data as
possible for use by the public, both through public-use files and with
restricted access through research data centers. The agency’s
publication series is meant to support the data releases and to
highlight key findings. NCHS produces few projections and results of
sophisticated models; these have not been a focus of the agency.
Rather, NCHS’s focus has been on the collection and dissemination of
information as quickly as possible to as many users as possible.
Madans described the major NCHS data collection systems: the
National Health Interview Survey (NHIS), the National Health and
Nutrition Examination Survey (NHANES), the National Survey of
Family Growth, the National Vital Statistics System, and the National
Health Care Surveys. She noted that NHANES presents some unique
confidentiality issues because a health examination is administered
as part of the survey. Clearly, NCHS collects large amounts of survey
data of national interest for which transparency is important.
However, she pointed out that NCHS does not collect the data itself:
it contracts with either the U.S. Census Bureau or various private
companies. As a result, there are some small “black boxes” in the
operation where the contractors know precisely how something is
carried out but NCHS does not know. NCHS is trying to reduce or
eliminate those holes, she said. However, in general, the agency has
a good grasp of its methods and the methods of its contractors.
Madans noted that the National Vital Statistics System operates
under somewhat different circumstances. NCHS does not have a lot
of control over the collection of those data at the source because
vital registration is a state function. For this system, NCHS has what
is now a 100-year collaboration with the 57 jurisdictions that collect
vital records. NCHS supports standardization of what is collected and
how it is collected across the jurisdictions, and documents the
processes used.
Lastly, Madans described NCHS’s set of surveys called the National
Health Care Surveys. Respondents include physicians, hospitals,
emergency departments, outpatient departments, and ambulatory
surgery centers, who provide records of their activities. The data
from the records are not what traditionally are thought of as
administrative records, because NCHS abstracts from the records
and so there is some modification of the original responses. NCHS is
trying to move to electronic health records, but it will have even less
control of what is entered into those records. She noted that when
people look at their own medical records, they are amazed at what is
in there and what is not. It is hard for NCHS to provide
documentation for aspects of these surveys because NCHS does not
collect the original information.
Madans turned back to the broad issue of transparency. NCHS
attempts to standardize and comprehensively document data
collection processes, but that can be challenging. Data collection and
documentation generally go through the following stages: (1)
planning—telling people what NCHS is going to collect; (2)
development—developing and documenting the actual survey
instrument, how it is administered, what the interviewer guidelines
are, etc.; (3) field work—collecting the information and documenting
what happens in the field; (4) data preparation, editing, and
imputation, including edit programs that may have been written a
long time ago that are rarely examined; (5) file release with
documentation; and (6) published reports and papers published by
others. For the last point, the key question is whether one can follow
the references and get back to the documentation that is on NCHS’s
Website. Transparency can be considered separately for each stage.
In her opinion, NCHS gets very good grades for transparency in
planning and this has further improved recently. It also gets good
marks in transparency in development and in published reports, and
excellent marks in transparency in field work and in file release.
However, it could do better in transparency in data preparation,
editing, and imputation.
Madans said her main worry is about issues of data quality of
which the agency could be unaware. There are only certain things
for which NCHS will do an in-depth evaluation, and sometimes those
are done after the data have been collected and so it is not always
clear how to document that stage of the process. An example is the
collection of data regarding vitamin D in the NHANES survey. The
surveys were showing a population-wide decline in vitamin D, but
the change appeared to be due to the tests used. To address this,
NCHS did a crossover study, which was feasible because the
respondents’ blood samples were banked, allowing the agency to
redo all of the tests using different techniques. Madans posed the
question of where this kind of situation should be documented. In
this case, NCHS wrote a report, which is on the agency’s Web page,
but is that sufficient? Also, for survey data, NCHS does not know
what may be a hidden example of mis-response. More broadly,
where should NCHS put its resources? For example, should NCHS go
back to the 1985 NHIS to document the metadata when very few
people in the last 10 years have needed that information? It is
possible, she said, that funds should be moved from documentation
to evaluation.
Madans added that she is aware that this workshop is concerned
about the extent to which federal statistics are reproducible, and she
raised several questions. First, is transparency a goal only in that it
supports reproducibility—that is, is the goal only that someone can
do what was done using the published data and methods? What is
the appropriate starting point for such a discussion? For NCHS, the
documentation of public-use files and analyses are designed so that
analyses based on the public-use files can be reproduced. Is this
sufficient? And is that really of interest to anyone other than to show
that the number that was published is correct? It does not say
anything about the way that the number was calculated or why the
question on the survey was asked the way it was. She raised several
additional questions: What should be documented about how a
public-use file was created? Does an agency go back and let people
recreate a public-use file? What can be useful about analysis-specific
decisions?
A key issue, Madans said, is how an agency deals with missing
data. NCHS has tried to document that process, but sometimes it is
not straightforward. How far back should one go in the file to try to
document how particular decisions were made? Should this apply
only to key indicators or to the content of all NCHS publications?
What about reports and papers not published by NCHS? Is it
sufficient for the agency to maintain the information and make it
available on request or does the information have to be available on
its Website? Does an agency have to make everything known about
a data element available every time it is reported, or is it sufficient
for such information to be available on the agency Website?
One can spend a lot of time worrying about reproducibility,
Madans said, but she questioned whether that is really the primary
benefit of efforts toward transparency. She repeated her idea that
NCHS is being transparent to add to the credibility of the agency.
Although complete transparency should result in reproducibility, she
asked whether transparency in decisions about what data to collect
and how to collect the data should be required even if those
decisions are not directly related to reproducibility. She raised
another series of questions: How much transparency is enough and
in what areas? Is transparency in all aspects of the data collection
process important for replication? What are acceptable ways to
document data collection processes? Must everything be publicly
available even if the user community is small for some data
elements? Is it sufficient to maintain information and make it
available on request? Madans noted that people often take the
NHANES methodology or the NHIS questionnaires and try to use
them in another setting. In such cases, information on how the tools
were implemented to get comparable results can be useful. This
situation raises the question of what an agency might be able to do
for such users in order for one to determine whether the results are
consistent.
Madans said that she has no answers to these more general
questions about tradeoffs. NCHS will maintain a commitment to
transparency so that others can reproduce what it does, but mostly
so that others understand what the agency is doing and so that it
can maintain credibility in the user community.
In closing, Madans acknowledged that there are costs to NCHS’s
approach and the agency has accepted those costs. The costs
associated with transparency affect the scope of NCHS’s data
collection, data releases, and publication programs. The agency
would like to minimize those costs, she said, and it would appreciate
guidance from the statistical and wider community on how to better
define what those boundaries are, what necessitates transparency,
and the extent of transparency.
SALLY THOMPSON: BENEFITS AND COSTS OF

TRANSPARENCY
Thompson began her presentation by noting that the various
agencies in the federal statistical system all operate differently. She
describes some of what is done at BEA, acknowledging that some of
what she is saying will not generalize. One major difference is that
BEA does not directly collect data as much as some of the other
agencies; it spends more time and energy combining information
from various sources. She said that BEA gets a lot of its data from
the U.S. Census Bureau and much of its price data from the Bureau
of Labor Statistics, but it also gets data from the U.S. Departments
of Agriculture, Defense, and the Treasury; the Federal Reserve
Board; the Internal Revenue Service; and the Energy Information
Administration.
Source data are very important to BEA, she said, and they must
be timely and of high quality, which are often in conflict. She quoted
from the agency’s mission statement:1 “The Bureau of Economic
Analysis (BEA) promotes a better understanding of the U.S. economy
by providing the most timely, relevant, and accurate economic
accounts data in an objective and cost-effective manner.”
Thompson listed some of BEA’s best-known data products: the
national accounts, which have components of gross domestic
product (GDP), personal income, and corporate profits; international
accounts, including balance-of-payment accounts, trade in goods
and services, and foreign direct investment; industry accounts,
including input-output accounts, GDP by industry, and travel and
tourism accounts; and regional accounts, including GDP accounts by
state and personal income information by state and local areas. She
said that BEA data products help with economic analyses of
businesses and households and support decisions about monetary
policy, fiscal policy, and state and local planning and allocation of
funds. Thompson continued that there are various mechanisms for
ensuring objective statistics. They include statutory requirements
and various directives and standards. At the bureau level, there are
security and release procedures, the transparency of methods and
sources, and regular review of revisions.
Regarding statutory requirements, the Information Quality Act of
2001 directed OMB to issue government-wide guidelines that
“provide policy and procedural guidance to Federal agencies for
ensuring and maximizing the quality, objectivity, utility, and integrity
of information disseminated by Federal agencies.”2 Thompson
reminded workshop participants that the resulting OMB guidelines
directed federal agencies to establish and issue their own guidelines
toward this end. BEA’s Information Quality Guidelines were
implemented in 2002; they are available on the BEA Website and are
updated regularly.3
Thompson noted that in terms of directives and standards, there
are OMB Statistical Policy Directives, which identify minimum
requirements for federal principal statistical agencies when they
engage in statistical activities. There are also measurement
standards and various classification standards. She noted that OMB
also makes an effort to conform its directives to various international
standards.
For security and release, Thompson detailed BEA’s six procedures:
(1) limited access to sensitive estimates to those with a need to
know; (2) physical and computer security necessary to limit access
to those with a need to know; (3) ensuring no BEA employee uses
(or gives the appearance of using) prerelease information for
personal gain or inadvertently provides prerelease information to the
media or other unauthorized individuals; (4) regular “best practices”
training for staff handling and processing sensitive data; (5)
“lockups” for principal economic indicators; and (6) releasing data
according to an announced schedule, which minimizes the risk of
prerelease access, provides a clear separation of statistical agency
and policy officials’ statements, and ensures simultaneous release to
all users and the public.
Thompson then turned to the transparency of sources and
methods. For source data, the Source Data Improvement and
Evaluation Program was established in the 1980s. Since BEA relies
on others for data collection, this program monitors the needs of all
of BEA’s programs for data provided by federal agencies, and it
documents the data sources underlying BEA’s estimates. For
concepts, methods, and estimation procedures, BEA strives to make
information readily available. For each of the 2,500 separate
accounts it maintains, there may be a separate type of model-based
estimation, imputation method, extrapolation method, or seasonal
adjustment method. She stressed that BEA does a lot of analysis
with the input data, and transparency facilitates an understanding of
the concepts and methods. In addition, she said, for each estimate
there is usually a set of external gross domestic product (GDP)
forecasters who try to forecast GDP; to do so, they have to have
studied all of the methodology documents and know where to get
the same source data for input into their estimation methodology.
Thompson said that she does not know of any other agency that has
such an “industry” that works on its data. When the data and
methods are not transparent, this user group lets them know. When
BEA publishes a number that is different from ones suggested by the
forecasts, the agency gets phone calls and e-mails. She noted,
however, that although BEA documents what data it is using and the
estimation methodology used for its various statistics, the agency
does not publish detailed formulas relating source data to final
estimates.
She noted that BEA publishes regular revision studies in its Survey
of Current Business journal. These studies examine the accuracy of
the estimates over time and test for any systematic overstatement
or understatement.
Thompson then discussed ensuring the integrity of BEA products
in three ways. The first is by addressing measurement challenges.
This involves improving the accuracy of more timely GDP estimates.
BEA collaborates with source-data agencies to accelerate and
improve the data used to estimate GDP. For instance, BEA
collaborated with the U.S. Census Bureau to achieve major
reductions in revisions to GDP estimates because the U.S. Census
Bureau was able to accelerate the publication of trade data,
inventory data, and quarterly services data. BEA targets the
timeliness, coverage, and quality of macroeconomic indicators
leading to improvements in timeliness and accuracy of measures of
economic growth.
There is also evidence of residual seasonality in some of BEA’s
GDP estimates, she explained, for which BEA has a three-stage plan
to resolve. First, the agency is revising existing BEA source data and
aggregations for origins of residual seasonality. Second, it is
modifying the GDP estimation process based on the results of the
first-stage review. And third, it is developing and releasing current-
quarter GDP estimates that are not seasonally adjusted. The results
of this plan will be reflected in the statistics released as part of the
comprehensive update of the National Income and Products
Accounts in July 2018.
BEA is also developing new products as a way of ensuring
integrity, Thompson said. Some of this work includes the creation
and development of various satellite accounts, including research
and development, which have now been incorporated into BEA’s core
accounts; health care spending; arts and culture; travel and tourism;
small businesses; and outdoor recreation.
Thompson explained that another way BEA is working to ensure
integrity is by exploring new data sources. The current push to
exploit “big data” presents challenges in terms of using them for
official economic statistics. Examples include data from Zillow and
from the Athena Health System. Also, BEA is exploring the use of
more credit card data. Thompson ended her presentation by noting
that it is unclear what the relevant transparency issues are for these
data, and she offered several questions: How representative are the
data? Do the concepts match those necessary to measure output,
prices, employment, etc.? Do the data provide consistent time series
and classifications? Is it possible to bridge gaps in coverage? How
timely are the data? How cost-effective are the data? And what
confidentiality issues arise and how limiting are they?
JOHN ELTINGE: COSTS AND BENEFITS OF TRANSPARENCY

John Eltinge opened his presentation with the proposition that
there is a relationship among the goals of transparency,
reproducibility, and replicability with quality/risk/cost profiles and
stakeholder value in the production of statistics. In particular, he
said, this relationship is going to be critical when one considers the
benefits and costs of transparency standards in terms of the
alignment of transparency activities with various elements of those
underlying profiles.
Eltinge said his presentation will focus on three elements to make
this connection: a qualitative description of transparency,
reproducibility, and replicability; some elements of a conceptual
framework; and some specific types of benefits, risks, and costs,
extending some of the ideas put forward by the two previous
speakers.
In general, Eltinge said, one can think about transparency for
federal statistics in terms of three questions: Is enough information
being provided to users so that they can gauge the quality, risk,
cost, and value profiles of the products that an agency produces?
That is, can those users use the information provided to assess the
implications of those features for their particular use? Can one use
assessments from users to give the U.S. Census Bureau some
insights into ways in which it can improve its products and expand
its product lines?
For reproducibility, Eltinge said there is one critical question: Could
an independent analyst who looks at the description of what has
been done for the original production, along with the input data
(though that is not clearly defined), reproduce what was published?
In contrast, for replicability, there is a different question: Could one
get basically the same inferences based on independent measures
from new independent data collected from the same population?
The issue of replicability for the U.S. Census Bureau appears in a
number of cases in which there may be “house effects” when one is
looking at a collection and processing activity that the agency
previously contracted out to one organization that was later
contracted out to another organization. The U.S. Census Bureau
often sees a difference in the products, and it is often attributed to
such house effects. He noted that there may be some very
interesting Bayesian interpretations for each of these components of
transparency, reproducibility, and replicability.
In general, Eltinge said that for all U.S. statistical agencies, their
mission is to produce high-quality information on a sustainable and
cost-effective basis. To do so, agencies could align the operational
definitions of transparency, reproducibility, and replicability with the
mission statement. In addition, he said, agencies would be
transparent in communicating with their stakeholders about the
costs and risks of transparency, reproducibility, and replicability.
Agencies might consider users’ expectations of being transparent on
risk factors and cost structures. This question is especially important
when an agency encounters financial challenges and has to make
decisions as to whether to drop a product series, reduce its sample
size, or take other steps that may end up reducing quality.
In addition, Eltinge said, when one thinks about quality, the
customary criteria, according to Brackstone (1987),4 are accuracy,
relevance, timeliness, comparability, coherence, and accessibility. He
noted that accuracy for survey estimates is addressed by the
assessment of total survey error, which gets into population
coverage, sampling error, incomplete data, specification error, and
measurement error. There are also modeling errors, adjustment
errors, the use of disclosure limitation methods, etc.
Eltinge added that the context of estimates is also important.
Sources of variability would be considered as to whether they are
controlled in the design or included in the model. Also, he said, there
is the distinction between reproducibility and replicability and which
sources of variability one is conditioning on. There are also political
implications as to what factors are shown to influence which
outcomes.
Eltinge then discussed the prospective benefits, costs, and risks of
transparency, reproducibility, and replicability. The benefits include
quality improvements, which derive from a better understanding of
the data through stakeholders’ use, in that transparency is a
precondition for data improvement. This factor is particularly
important when integrating data from multiple sources and
managing a complex supply chain of statistical information. Risks
can be reduced by reducing the likelihood of tunnel vision and
reducing the frequency of undetected blunders. He also noted that
cost increases as a result of quality improvement efforts will result
from the development and implementation of more efficient designs
and models.
Turning to risks, Eltinge noted that there will be contract
management and various institutional dynamics. In addition, there
are the usual risk factors that accompany the development of
standards and requirements. These risks result from formal
compliance being divorced from substance and regulatory capture
and accountability issues. Also, he said, it may be the case that the
underlying science and practice are not yet sufficiently mature to
support a refined standard. The risks specific to transparency are
calcification, Eltinge said, because the additional cost of
transparency increases barriers to innovation, and the perceived loss
of intellectual property rights can discourage cutting-edge
investments. He said there is also the chance for unreasonable
criticism. To mitigate these risks, one can be aware of these
possibilities and identify their impact on stakeholder use and value.
Eltinge then noted some specific costs of transparency. First, there
are direct labor costs, which involve documentation and curation of
code, along with refined datasets. There are also opportunity costs
for one’s most productive employees, he said, and there is a
cognitive and operational burden for users, especially if transparent
information is not well calibrated with the stakeholder information
base.
Eltinge offered four suggestions for agencies with respect to
increasing transparency and reproducibility for statistical agencies.
First, anchor transparency in an agency’s mission statement. Second,
have the goal of practical improvement in long-term stakeholder
value. Third, align efforts with the dominant sources of variability.
And fourth, work out transparency and reproducibility improvements.
He added that it is especially important in terms of big data to
consider, in addition to documentation, conducting sensitivity
analyses to indicate how much various assumptions matter.
__________________
1 See https://www.bea.gov/about [November 2018].
2 See https://obamawhitehouse.archives.gov/omb/fedreg_final_information_qua
lity_guidelines [January 2018].
3 See https://www.bea.gov/about/policies-and-information/information-quality
[January 2018].
4 Brackstone, G.J. (1987). Statistical issues of administrative data: Issues and
challenges. In Proceedings of the Statistical Uses of Administrative Data: An
International Symposium organized by Statistics Canada, November 23–25.
4
Benefits and Costs of Transparency:

Views from the United Kingdom and
Canada
The next session featured two speakers from the UK Office for
National Statistics (ONS), Peter Brodie and Sarah Henry, and one
speaker from Statistics Canada, Eric Rancourt.
PETER BRODIE: FOREIGN VIEW OF BENEFITS AND COSTS

OF TRANSPARENCY
Peter Brodie began his presentation by providing some context,
noting that he was very interested in the presentations that
preceded his because issues regarding transparency are very
different in the United States and the United Kingdom. The United
Kingdom has the fifth largest economy in the world, and the ONS
has the responsibility of producing the information that supports the
public good through better statistics that provide for better
decisions, including economic ones.
Brodie said that there are costs and benefits of that work.
Unfortunately, the benefits are hard to measure, and the burdens on
businesses for providing data and on both businesses and people to
respond to surveys are not easy to assess either. As ONS increases
its use of administrative data, the burden becomes increasingly hard
to measure. What is the burden of collecting administrative data,
and what is its value to decision making for public policies? Brodie
said the office’s challenges and opportunities include the following:
matching and linking data to describe the society and the economy,
exploiting nonsurvey data in the age of the data revolution,
remaining trusted in an era of allegations of “fake news,” and
safeguarding the data while allowing it to be used as a public asset
for the public good.
Brodie explained that the ONS has a unique position: as a result of
its legal framework to collect data, it is in control of a tremendous
public asset. Thus, it has a duty to exploit that public asset for the
public good. He pointed out that ONS’s approach is different from
that in the United States in part because the office is part of the UK
Statistics Authority, which is itself answerable to Parliament, but it is
a body that is independent of government. That is a subtle but
important difference, he stressed.
Brodie explained that the UK Statistics Authority comprises three
entities. One is the Government Statistical Service, a cross-
government network, led by the national statistician. Although ONS
produces most of the statistics for the Authority, there are other
government agencies that produce their own: notably, agricultural
statistics and health statistics are not under ONS’s jurisdiction. The
second entity is ONS; it not only produces the majority of the
nation’s official statistics, but it also has the goal of improving
national statistics. The third entity is the Office for Statistical
Regulation, which ensures that statistics are produced and
disseminated in the public interest. It does so by assessing official
statistics for compliance with the Code of Practice for Official
Statistics, monitoring and reporting publicly on statistical issues, and
challenging the misuse of statistics. Looking toward the future,
Brodie said, he expects ONS’s organization to be quite different than
it currently is. There are three layers expected to change—delivery
and impact, production and flow, and capacity and capability.
Delivery and impact are the most visible parts of the organization;
production and flow are where all of these statistics are produced;
and capability and capacity are the least visible part of the operation,
encompassing how ONS is actually reacting to what is needed
outside of itself.
One of the key drivers for change is how to be much more flexible
in the future, Brodie said, not just to produce the statistical products
ONS has always produced. What are the policy issues coming in the
future? For example, something that is topical in the United Kingdom
at the moment is the measurement of net migration. Brodie said
ONS should be ahead of the game on what it is doing on this issue
and not wait to be asked by policy makers. Another example is in
the measurement of services, which could be more agile. On this
topic, ONS is determined to work more closely with academics in the
future. Internationally, he said, ONS realizes that collaboration with
its international partners will answer the questions facing UK society.
Brodie noted that with Britain’s exit from the European Union, those
international relationships will change. How this will play out remains
to be seen, he said, but it may lead to an increased focus on
relationships with the OECD, the International Monetary Fund, and
the United Nations. Eurostat and other national statistics institutes in
Europe and around the globe will remain key partners and
colleagues, he said.
SARAH HENRY: FOREIGN VIEW OF THE BENEFITS AND

COSTS OF TRANSPARENCY
Sarah Henry began her presentation on the subject of what
opportunities transparency and reproducibility will provide not only
to ONS as a statistics agency but to society and the economy. The
first opportunity, she said, is enhanced knowledge. ONS staff have
spent some time thinking about how to use the information they
have to provide the story that sits behind some of the country’s most
important issues by not just describing the society and the economy,
but understanding them. She said that over the past decade ONS is
increasingly looking at how to describe the economy and society
together; they are not two separate entities.
Henry noted that the United Kingdom is facing some type of
“social recession” that has not been predicted. For many survey
questions in the United Kingdom, the results show a split in society.
There is a huge amount of blogging and tweeting, but the volume is
no indication that people really understand what happened in terms
of Brexit, for example. What the agency can do, she suggested, is to
make sure that data and methods are made available to people who
want to focus on explaining these important issues. Democratizing
the data in a safe and responsible way is an important pillar of
democracy.
Turning to products, Henry said that ONS has a wide range of
statistical data products. Many of them are just descriptive statistics.
However, ONS wants to be more engaging with a wider range of
users, so it is improving its Website and improving how it tweets to
help its users understand and interpret what the agency is
producing. Although many years ago most users may have been
professionally knowledgeable, the vast majority of current users are
lay users of statistics.
Henry explained that another big opportunity for ONS is the data
revolution. There is a huge amount of data produced, some of which
is administrative data that taxpayers have paid to produce and
arguably own as well. How does ONS make that available? In many
cases, data are by-products of companies producing things—and
those data are very important to them—but there are also data that
are products in their own right. She gave an example of the
difference. Mobile phone companies provide services and, as a by-
product, produce very interesting data that can help explain and
describe a wide range of phenomena, such as commuting patterns.
In contrast, a company like Thomson Reuters, which offers
intelligence, technology, and human expertise, produces data that
are very important to explaining the economy and society. ONS
would like to tap into those kinds of data.
One very important achievement in the past few months is the
passing of the Digital Economy Act, Henry said, particularly with
regard to improvements for data sharing. Data sharing touches on a
number of areas, although statistics and research are the ones that
matter most to ONS in the context of this workshop. Another very
important improvement in the new act relates to public service
delivery. The underlying argument, she said, is that public service
agencies could stop operating in silos and gain access to data that
help them improve delivery of services.
Henry said that the new legislation generally gives ONS the right
to go to pretty much any organization and say it needs that
organization’s data (though it is not quite as simple as that). It then
uses those data to improve its data products. She noted that ONS is
now attempting to replace some of its survey data with
administrative or other data and is creating an entirely new set of
other statistics, along with enhancing their statistical outputs.
Another opportunity and challenge for ONS is centered around
trust, Henry said. This is where transparency is of the utmost
importance. ONS has regulations and it also attempts to explain the
statistics, as well as the processes used to produce them, as much
as possible. ONS is not shy about addressing inaccurate
representation when it happens.
Henry noted that the United Kingdom’s national statistician
announced that the practice of prerelease access for some users
would cease as of July 1, 2017. This change had been under debate
for quite a while, and she said it represents the importance of
transparency.
ONS also has a code of practice.1 Henry said that her sense is that
the code captures quite a lot of the things that were mentioned in
the previous talks. It is written in plain English to make it accessible
to as many people as possible, but there are technical notes included
as well. Of the eight principles, she highlighted the one concerning
sound methods and assured quality. Within that principle, she
indicated eight practices:
1. Seek to achieve continuous improvement in statistical

processes.
Another random document with
no related content on Scribd:
5. Stamina a Pistillo diducta; anthera una lente aucta.
6. Stylus, et Stigma lente, aucta.
SPECIFIC CHARACTER.
Heath, with crested tips, within the blossom; the leaves grow by threes; the
branches are thread-shaped, the blossoms small and pitcher-shaped; the
leaves of the cup are longer than the flower, the foot-stalks very long.
DESCRIPTION.
Stem thread-shaped, grows half a foot high, upright, and very much
branched.
The Leaves grow by threes, are lance-shaped, concave on the upper
surface, pressed to the stem, and smooth, having very short foot-stalks.
The Flowers are numerous, and terminate the smaller branches in
bunches; the foot-stalks are very long, flesh coloured, with three coloured
floral leaves.
Empalement. Cup four-leaved, which are of a broad oval form, flesh
coloured, and longer than the blossom.
The Blossom is small, of a pitcher-shape, light flesh colour at the end,
greenish at the base; the segments of the mouth are bent back, and deep
flesh coloured.
Chives. Eight hair-like threads; the tips crested, and within the blossom.
Pointal. Seed-vessel nearly globular; the shaft thread-shaped, partly out
of the blossom. Summit four-cornered.
Native of the Cape of Good Hope.
Flowers from September, till December.
REFERENCE.
1. The Empalement, and Blossom.

2. The Blossom.
3. The Empalement magnified.
4. The Chives and Pointal.
5. The Chives detached from the Pointal; one tip magnified.
6. The Shaft and its Summit magnified.
ERICA campanulata.
CHARACTER SPECIFICUS.
Erica, antheris muticis, inclusis; floribus solitariis, cernuis; calycibus

coloratis; corollis globoso-campanulatis, luteis; foliis quaternis, linearibus,
angustissimis.
DESCRIPTIO.
Caulis fruticosus, pedalis, erectus, filiformis; rami et ramuli filiformes,

virgati, glabri.
Folia quaterna, linearia, angusta, glabra, erecta; petiolis brevissimis,
adpressis.
Flores terminales, subsolitarii, cernui; pedunculi colorati, bracteis tribus,
coloratis, subulatis, instructi.
Calyx. Perianthium tetraphyllum, foliolis ovatis, acuminatis, concavis,
adpressis, coloratis.
Corolla globoso-campanulata, lutea; laciniis semi-ovatis, subreflexis.
Stamina. Filamenta octo, capillaria, incurvata. Antheræ muticæ, inclusæ.
Pistillum. Germen sub-globosum. Stylus columnaris. Stigma
tetragonum.
Habitat ad Caput Bonæ Spei.
Floret a mensi Junii, in Augustum.
REFERENTIA.
1. Calyx, et Corolla.
2. Corolla.
3. Calyx, lente auctus.
4. Stamina, et Pistillum.
5. Anthera una, aucta.
6. Pistillum, lente auctum.
SPECIFIC CHARACTER.
Heath, with beardless tips, within the blossom; flowers grow singly,
hanging down; cups coloured; blossoms globularly bell-shaped, and yellow;
leaves grow by fours, are linear, and very narrow.
DESCRIPTION.
Stem shrubby, grows a foot high, upright, and thread-shaped; the greater
and smaller branches are thread-shaped, twiggy, and smooth.
Leaves grow by fours, linear, narrow, smooth, and upright; very short
foot-stalks, pressed to the branches.
Flowers grow mostly solitary, at the end of the smaller branches,
hanging down; the foot-stalks are coloured, having three awl-shaped,
coloured floral leaves on them.
Empalement. Cup four-leaved, leaflets egg-shaped, pointed, concave,
pressed to the blossom, and coloured.
Blossom, globularly bell-shaped, and yellow; the segments half egg-
shaped, and a little reflexed.
Chives. Eight hair-like threads, curved inwards. Tips beardless, and
within the blossom.
Pointal. Seed-bud nearly globe-shaped. Shaft pillar-shaped. Summit
four-cornered.
Flowers from June, till August.
REFERENCE.

2. The Blossom.
3. The Empalement, magnified.
4. The Chives, and Pointal.
5. A Tip, magnified.
6. The Pointal, magnified.
ERICA capitata.
Erica, antheris muticis, sub-inclusis; corollis globosis, albidis, tomentosis,

tectis calyce magno, hispido, colorato; foliis ternis, pilosis.
DESCRIPTIO.
Caulis erectus, laxus, filiformis, fuseus; rami filiformes, villosi; ramuli

capillares, frequentes, tomentosi.
Folia terna, linearia, obtusa, dorso sulcata, pilis longis hirta; petiolis
brevissimis, adpressis.
Flores in extremis ramulis terminales bini, vel terni, cernuo-patenti;
pedunculi brevissimi, bracteis tribus, minutis, adpressis, instructi.
Calyx. Perianthium tetraphyllum, foliolis ovatis, concavis, incurvatis
maximis, totis densissime hirtis, adpressis, luteo-viridis.
Corolla sub-globosa, alba, lanata, in sinu calycis fere recondita.
Stamina. Filamenta octo capillaria, receptaculo inserta. Antheræ muticæ
sub-inclusæ.
Pistillum. Germen subrotundum, apice pilosum. Stylus filiformis,
exsertus. Stigma tetragonum.
Floret a mensi Julii, in Octobrem.
REFERENTIA.
2. Corolla.
5. Stamina a Pistillo diducta, anthera una lente aucta.
6. Stylus, et Stigma, lente aucta.
SPECIFIC CHARACTER.
Heath, with beardless tips, just within the blossoms, which are globular,
white and downy, being covered with a large, hairy, coloured cup; leaves
grow by threes, and are hairy.
DESCRIPTION.
Stem upright, weak, thread-shaped, and brown; branches thread-shaped,

and hairy; small branches like hairs, numerous, and downy.
Leaves grow by threes, linear, blunt, furrowed at the back, and covered
with long, harsh hairs; foot-stalks very short, and pressed to the branches.
Flowers grow at the extremity of the smaller branches, by twos or
threes, spreading out, and nodding; foot-stalks very short, having three
small floral leaves, which are pressed to the blossom.
Empalement. Cup of four leaves, which are egg-shaped, concave, turned
inwards, very large, quite covered with strong hairs, pressed to the
blossoms, and of a yellow-green.
Blossom nearly globular, white and woolly, almost hid within the cup.
Chives. Eight hair-like threads, fixed into the receptacle. Tips beardless,
and nearly within the blossom.
Pointal. Seed-bud nearly round, and hairy at the end. Shaft thread-
shaped, and without the blossom. Summit four-cornered.
Flowers from the month of July, till October.
REFERENCE.
1. Empalement, and Blossom.

2. A Blossom.
5. The Chives detached from the Pointal, one Tip magnified.
6. The Shaft, and its Summit, magnified.
ERICA cerinthoides.
Erica, antheris muticis, inclusis; corollis tubulato-ventricosis, læte

sanguineis, fasciculatis, hispidis; foliis quaternis, rigidis, ciliatis.
DESCRIPTIO.
Caulis flexibilis, erectus, cinereus, sesquipedalis; rami pauci, erecti, raro

ramulosi.
Folia quaterna, ciliata, oblonga, convexa, subtus sulco exerata, petiolis
brevibus, adpressis.
Flores magni, sessiles, aggesti in capitulum, cernui, pedunculi hispidi,
bracteis tribus foliis similibus instructi.
Calyx. Perianthium tetraphyllum, foliolis lanceolatis, hispidis, foliis
similibus.
Corolla, tubulato-ventricosa, læte sanguinea, hirsuta, ore obsolete
quadrifida; pollicaria.
Stamina. Filamenta octo capillaria. Antheræ muticæ, inclusæ.
Pistillum. Germen cylindricum, hirsutum. Stylus filiformis, sub-
exsertus. Stigma tetragonum.
Floret a mensi Augusti, in Aprilem.
REFERENTIA.
2. Calyx lente auctus.
4. Stamina a Pistillo diducta; anthera una lente aucta.
5. Stylus, et Stigma, lente aucta.
SPECIFIC CHARACTER.
Heath, with beardless tips, within the blossoms, which are of an inflated
tubular shape, of a rich blood colour, hairy, and bundled together; the leaves
grow by fours, are harsh, and lashed.
DESCRIPTION.
Stem grows upright, pliant, ash-coloured, and a foot and a half high; the
branches are few, and upright, seldom branching.
The Leaves grow by fours, are lashed, oblong, rounded on the upper
surface, and deeply furrowed on the under side, with short foot-stalks,
pressed to the stems.
The Flowers are large, growing in close bunches, fixed altogether at the
end of the branches, bending downward; the foot-stalks are hairy, having
three floral leaves, similar to the other leaves.
Empalement. Cup four-leaved, which are lance-shaped, hairy, and like
the other leaves.
Blossom, of an inflated tubular form, hairy, and of a rich red or blood
colour, the mouth slightly cut into four segments; an inch long.
Chives. Eight hair-like threads. Tips beardless, and within the blossom.
Pointal. Seed-vessel cylinder shape, and hairy. Style thread-shaped,
nearly without the blossom. Summit four-cornered.
Flowers from August, till April.
REFERENCE.

2. The Empalement magnified.
5. The Shaft, and its Summit, magnified.
ERICA cernua.
Erica, antheris cristatis, inclusis; floribus umbellatis, cernuis, secundis, sub-

ovatis, pallide-carneis; foliis quaternis.
DESCRIPTIO.
Caulis fruticosus, erectus, pedalis; rami sub-simplices, erecto-patenti.

Folia quaterna, linearia, obtusa, subtus sulcata, sub-scabrida; petiolis
brevissimis, adpressis.
Flores in apice ramorem umbellati, cernui; pedunculi longi, colorati,
bracteis tribus, linearibus, instructi.
Calyx. Perianthium tetraphyllum, foliolis minutis, subulatis, ciliatis,
coloratis.
Corolla sub-ovata, pallide-carnea; laciniis limbi acuminatis, sub-
erectis.
Stamina. Filamenta octo capillaria apice introrsum declinata. Antheræ
cristatæ, inclusæ.
Pistillum. Germen turbinatum, fulcatum. Stylus cylindricus, sub-
inclusus. Stigma obsolete tetragonum.
Floret a mensi Augusti in Decembrem.
REFERENTIA.
3. Stamina a Pistillo diducta, anthera una lente aucta.
4. Germen, Stylus, et Stigma, lente aucta.
SPECIFIC CHARACTER.
Heath, with crested tips, within the blossom; the flowers grow in bunches,
nodding, all pointing one way, nearly egg-shaped, and of a pale flesh colour;
leaves growing by fours.
DESCRIPTION.
Stem shrubby, upright, and grows a foot high; the branches are almost
simple, upright, and spreading.
Leaves grow by fours, linear, blunt, channelled beneath, and roughish;
having very short foot-stalks pressed to the branches.
Flowers grow in bunches, at the end of the branches, nodding; the foot-
stalks are long and coloured, with three linear floral-leaves on them.
Empalement. Cup of four leaves, which are small, awl-shaped, fringed,
and coloured.
Blossom, nearly egg-shaped, of a pale flesh-colour; the segments of the
border tapered, and nearly upright.
Chives. Eight hair-like threads, bent downward on the inner side. Tips
crested, and within the blossom.
Pointal. Seed-bud turban-shaped, and channelled. Shaft cylindrical just
within the blossom. Summit obscurely four-cornered.
Flowers from August, till December.
REFERENCE.

4. The Seed-bud, Shaft, and Summit, magnified.
A SHORT DISSERTATION, &c.
Antecedent to the year 1772, the few species of this, now so numerous
Genus, known in our British gardens, were, the E. vulgaris, E. Tetralix, E.
cineria and E. vagans natives; the E. Dabœcii, from Ireland; the E. arborea
introduced in 1748, from Madeira; the E. herbacea or carnea in 1763, from
Switzerland; the E. mediterranea in 1765, from Minorca, and the E.
scoparia, E. viridi-purpurea, E. australis, E. ciliaris and E. umbellata, from
Portugal, between the years 1768 and 1707. The two other European species
we possess, the E. stricta and E. multiflora, natives of Spain, have been but
twelve years in cultivation with us. Of the African species, found within the
district of the Cape of Good Hope and the adjacent territory, which have
swelled the Genus to so great an extent; and which, but as an echo to the
general voice, may be said to contribute, by the extreme brilliancy of the
flowers of these species, more than any other, to the present splendor of our
green-houses, were unknown, till the above æra, to our English botanists,
but by name. In the year 1771 seeds of two species were received, at the
Hammersmith nursery, from the Cape, both of them vegetated; the first
which flowered, proving the E. tubiflora, of the Sp. Plant. of Linnæus; the
other, from the resemblance it bears to the Spruce Fir, was then named E.
abietina; but since, altered in the Kew catalogue, to E. concinna. Two years
subsequent, 1774, Mr. Francis Masson, botanical collector to His Majesty at
the Cape, laid the foundation for the celebrity of that superb collection at
Kew, which for many years, with unrivalled lustre, far outshone all others,
particularly by the number and variety of this most beautiful tribe of plants:
for which we refer to the second Vol. of the catalogue of that garden; where
the E. curviflora, E. lutea, E. cruenta, E. persoluta, E. baccans, E. marifolia,
E. abietina, E. corifolia, E. paniculata, E. empetrifolia, E. spumosa, E.
capitata, E. conspicua, E. cerinthoides, E. viscaria, E. Plukenetii, E. Petiveri,
and E. petiolata, are all stated to have been of that year’s introduction. From
this period, till within these few years, the accession was so rapid, that it
would be difficult, nay nearly impracticable, to ascertain the precise date
when most of the remaining species were introduced; as many different
collectors were about this time, or shortly after, producing in their
collections new species to which they each claimed the honour of priority of
introduction; the enumeration of these, therefore, in succession would be but

Download Methods To Foster Transparency And Reproducibility Of Federal Statistics Proceedings Of A Workshop 1St Edition And Medicine Engineering National Academies Of Sciences online ebook texxtbook full chapter pdf

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Download Methods To Foster Transparency And Reproducibility Of Federal Statistics Proceedings Of A Workshop 1St Edition And Medicine Engineering National Academies Of Sciences online ebook texxtbook full chapter pdf

Uploaded by

Copyright:

Available Formats

Methods to Foster Transparency and

Reproducibility of Federal Statistics:

Approaches to the Development of Character Proceedings

Communicating Clearly about Medicines Proceedings of a

Neuroscience Trials of the Future Proceedings of a

Implementation of Lung Cancer Screening Proceedings of

Sustainable Diets, Food, and Nutrition: Proceedings of

Globalization of Defense Materials and Manufacturing

Adaptability of the US Engineering and Technical

Statistical Challenges in Assessing and Fostering the

Michael L. Cohen, Rapporteur

Committee on National Statistics

Division of Behavioral and Social Sciences and Education

THE NATIONAL ACADEMIES PRESS

This activity was supported by contracts between the National Academy of

International Standard Book Number-13: 978-0-309-48629-3

Copyright 2019 by the National Academy of Sciences. All rights reserved.

Printed in the United States of America

Suggested citation: National Academies of Sciences, Engineering, and Medicine.

The National Academy of Engineering was established in 1964

The National Academy of Medicine (formerly the Institute of

The three Academies work together as the National Academies of

Learn more about the National Academies of Sciences, Engineering,

Proceedings published by the National Academies of Sciences,

For information about other products and activities of the National

WILLIAM EDDY (Chair), Department of Statistics, Carnegie Mellon

MICHAEL L. COHEN, Study Director

ROBERT M. GROVES (Chair), Office of the Provost, Department of

BRIAN A. HARRIS-KOJETIN, Director

We would first like to thank the National Science Foundation

In 2014, the National Science Foundation (NSF) provided support

For the workshop, CNSTAT defined transparency to mean that one

1. What official federal guidance or standards currently exist to

These questions come at a time when various changes are

Existing Guidelines Related to

PETER MILLER: AAPOR TRANSPARENCY INITIATIVE

BRIAN HARRIS-KOJETIN: RELEVANT OMB DIRECTIVES

Benefits and Costs of Transparency:

Susan Offutt (member, steering committee) began the session by

JENNIFER MADANS: BENEFITS AND COSTS OF

SALLY THOMPSON: BENEFITS AND COSTS OF

JOHN ELTINGE: COSTS AND BENEFITS OF TRANSPARENCY

Benefits and Costs of Transparency:

PETER BRODIE: FOREIGN VIEW OF BENEFITS AND COSTS

SARAH HENRY: FOREIGN VIEW OF THE BENEFITS AND

1. Seek to achieve continuous improvement in statistical

1. The Empalement, and Blossom.

Erica, antheris muticis, inclusis; floribus solitariis, cernuis; calycibus

Caulis fruticosus, pedalis, erectus, filiformis; rami et ramuli filiformes,

1. The Empalement, and Blossom.

Erica, antheris muticis, sub-inclusis; corollis globosis, albidis, tomentosis,

Caulis erectus, laxus, filiformis, fuseus; rami filiformes, villosi; ramuli

Stem upright, weak, thread-shaped, and brown; branches thread-shaped,

1. Empalement, and Blossom.

Erica, antheris muticis, inclusis; corollis tubulato-ventricosis, læte

Caulis flexibilis, erectus, cinereus, sesquipedalis; rami pauci, erecti, raro

1. The Empalement, and Blossom.

Erica, antheris cristatis, inclusis; floribus umbellatis, cernuis, secundis, sub-

Caulis fruticosus, erectus, pedalis; rami sub-simplices, erecto-patenti.

1. The Empalement, and Blossom.

You might also like