3

ASSESSING LOCAL
GOVERNMENT
PERFORMANCE IN
DEVELOPING COUNTRIES
Matthew Andrews
Anwar Shah
Source: Shah, Anwar, editor (2003), Measuring Government Performance in
the Delivery of Public Services, Vol. 2 of Handbook on Public Sector
Performance Reviews. Washington D.C.: he World !an".
Introduction
Decentrali#ation is co$$on in $an% de&elo'ing countries. (ocal
go&ern$ents are )eco$ing $ore nu$erous in such countries and are
increasingl% re*uired to 'la% larger roles in 'ro&iding ser&ices,
alle&iating 'o&ert%, and facilitating de&elo'$ent (+itlin
2000). ,i&en the i$'ortant role local go&ern$ents are )eing called to
'la%, central go&ern$ents and de&elo'$ent organi#ations are starting to
as" how well the% are doing (and how the% can )e i$'ro&ed). he
'ro)le$ with as"ing such *uestions is that criteria for e&aluating local
go&ern$ents in de&elo'ing countries re$ain 'oorl% for$ed: What does a
-good. local go&ern$ent loo" li"e in the de&elo'ing world/ What
factors should one consider when e&aluating local go&ern$ents in such
settings/
his 'a'er is written to address such *uestions, with the
ai$ of identif%ing criteria for e&aluating local go&ern$ents in de&elo'ing
countries. his 'a'er ta"es a $i0ed 'ractice1 theor%
a''roach to identif%ing such e&aluation criteria. he
-'ractice. ele$ent )uilds on e0isting e&aluations 'ractice at the
local le&el in de&elo'ed countries li"e the 2nited States, which
t%'icall% focuses on issues of legal confor$ance and fiscal health. he
-theor%. di$ension introduces concerns a)out res'onsi&eness,
efficienc%, and accounta)ilit%, with the 'otential
of gains in theseareas underl%ing do$inant
argu$ents in fa&or of decentrali#ation and local 1le&el go&ernance.
he a''roach recogni#es that local go&ern$ents
in de&elo'ing countries face their own s'ecial issues, and that e&aluation
criteria identified for use in such settings $ust )e easil% accessi)le,
facilitating an o)ser&ation1)ased anal%sis (and re*uiring li$ited resources).
Evaluation Criteria
(ocal go&ern$ent e&aluation 'ractice in de&elo'ed
countries generall% focuses on issues of confor$ance and fiscal health.
he confor$ance focus $anifests in e&aluations of whether local entities
conduct their o'erations within the legal 'ara$eters set for the$, and
whether the% ado't 'rocesses re*uired (either )% law or )% so$e other
'rofessional standard setting entit%, li"e the ,AS!). he fiscal health
e$'hasis is reflected in e&aluations of resource use in local go&ern$ent,
with the usual focus )eing the le&el of fiscal disci'line e&ident in financial
$anage$ent 'rocesses.
he $ain e&aluation $echanis$s in these local
go&ern$ents are e$)edded in financial and accounting $anage$ent s%ste$s
and in the 'rocess of fiscal e&aluation underta"en )% outside rating agencies
in countries li"e the 2nited States. he influence of such $echanis$s and
the 're&alence of this "ind of e&aluation leads 3oltin to sa% that local
go&ern$ents in the 2nited States ha&e 4focused
'rinci'all% on financial re'orting and other financial
$atters . . . and legal co$'liance5 (3oltin 6777:83). 9&en recent e&al
uations in the 2nited States and )e%ond, designed to e$'hasi#e
'erfor$ance, tend to anal%#e confor$ance to s'ecific -)est 'ractice.
internal $anage$ent 'rocesses and fiscal outco$es (e0a$'les
includethe ,o&ern$ent :erfor$ance :ro;ect and the (ocal
,o&ern$ent and :u)lic Ser&ice <efor$ =nitiati&e wor" in Central and
9astern 9uro'e).
(egal and 'rocess confor$ance and fiscal health are
i$'ortant criteria for e&aluating local go&ern$ents in de&elo'ing
countries as well. ,i&en that local entities are created )% law and
use 'u)lic resources to fulfill their duties, it is i$'ortant that the%
o'erate within the 'ara$eters of legislation and ensure an ade*uate le&el
of fiscal health and disci'line. Where local 1le&el e&aluations
ha&e )een underta"en in the de&elo'ing world, these
considerations ha&e often )een the do$inant (and often onl%) criteria (as
reflected
in a 6773 e&aluation conducted )% the >ar&ard =nstitute for
=nternational De&elo'$ent in =ndonesia (>==D 6773)).
<el%ing on this do$inant 'ractice is 'ro)le$atic, gi&en that theoretical
argu$ents in fa&or of decentrali#ation and the for$ation of local
go&ern$ents identif% 'otential
decentrali#ation gains in other areas of interest, go&ern$ent res'onsi&eness,
efficienc%, andaccounta)ilit% to citi#ens
(ie)out 67?@, Aates 67B2, Shah 677C).
6
=ssues related to these
areas also re*uire e&aluation, es'eciall% in de&elo'ing countries where local
go&ern$ents are often created in res'onse to low res'onsi&eness,
efficienc%, and accounta)ilit% in central go&ern$ents. =n such situations
local go&ern$ents are re*uired not onl% to o'erate according to legislation
and $anage their finances well, )ut also to 'ro&ide the -right. ser&ices (in
res'onse to citi#en need), in the -right. (or $ost efficient) wa%, and with the
highest degree of accounta)ilit% to constituents.
Co$)ining the fi&e focal areas %ields the following )road
criteria for e&aluating local go&ern$ents in de&elo'ing
countries: confor$ance to legislati&e and 'rocess
re*uire$ents, fiscal health, res'onsi&eness, efficienc%, and accounta)ilit% to
citi#ens. All fi&e can )e further )ro"en down into $ore s'ecific criteria
rele&ant to the de&elo'ing countr% e0'erience in general. o ensure that the
a''roach ta"en in this 'a'er is 'rag$atic and facilitates o)ser&ation1)ased
anal%sis, such s'ecific criteria are introduced in the following with reference
to e0a$'les of effecti&e and ineffecti&e local go&ern$ents in de&elo'ing
countries.
Evaluatin Con!or"ance to
Lei#lation and Proce##
(ocal go&ern$ents in de&elo'ed and de&elo'ing countries co$e into
)eing )ecause of legislati&e action. (egislation
defines the 'owers and functions and res'onsi)ilities of local
6
,on#ale# (2000) e$'hasi#es these factors in her introduction to an e&aluation of
local go&ern$ent refor$s in the :hili''ines. She writes that, 4Decentrali#ation ena)les
the state (through local go&ern$ent units, or (,2s) to )e $ore res'onsi&e to the needs
and 'references of co$$unities. =t leads to $ore accounta)le decision1$a"ing and
greater inno&ation in how 'u)lic ser&ices are deli&ered.5
go&ern$ents. As such, acade$ic and 'ractitioner local
go&ern$ent e&aluations in de&elo'ed and de&elo'ing settings a''ro'riatel%
deri&e their foundational e&aluation criteria fro$ legal (and *uasi 1legal)
sources. hese criteria t%'icall% concentrate on local go&ern$ent
confor$it% with legislation (and other for$al re*uire$ents) regarding the
setting and u'holding of )%1laws, the generation
and collection of re&enues, the following of 'rocess
re*uire$ents in resource dis)urse$ent, and the 'ro&ision of ser&ices.
here are nu$erous studies of local le&el legislati&e acti&it% in countries
li"e the 2nited States, generall% as"ing if $unici'alities 'ass the "inds of
laws the% are $eant to (and usuall% focused on local1le&el 'olicing and the
regulation of 'ri&ate $ar"ets). Such studies ha&e also )een e0tended to the
de&elo'ing world, with researchers e&aluating whether
$unici'alities set and u'hold the laws the% are re*uired to, and if
the% use their law1setting 'owers to facilitate
de&elo'$ent in their ;urisdiction. A 'ositi&e e0a$'le of such anal%sis
relates to the setting of legislation in Ce)u Cit% in the :hili''ines (2D9SCA
2006). he cit% 'assed Ardinance 6388 to creati&el% facilitate the
de&elo'$ent of low1cost housing, allowing the use of a trust fund generated
fro$ the sale of cit%1 owned lots e0clusi&el% for shelter and its related
e0'enditures. he cit% al so used its law1setting a)ilit% to increase
the 'artici'ation of &endorsE associations in $ar"et $anage$ent (which is
recogni#ed )% the <e&ised +ar"et Code). 3inall%, the $unici'alit%
$ini$i#ed red ta'e ha$'ering ser&ice deli&er% )% instituting the S%ste$s
and :rocedures <ationali#ation in ,o&ern$ent (S:<=D,) regulations.
>alfani (677@: 6CC) 'ro&ides an e0a$'le of a $unici'alit% not
fulfilling its legal role effecti&el%. >e writes that 4a large s'here of ur)an
life5 in Dairo)i 4has o'erated outside the regulator% and directi&e authorit%
of the for$al s%ste$5 )ecause the $unici'alit% failed to set and u'hold
laws as re*uired. =n such situations the failure of the local go&ern$ent to
co$'l% with its de ure role can lead to social insta)ilit%
and disorgani#ation in co$$ercial acti&ities.
he legal $andate faced )% $unici'alities also relates to
their re&enue raising acti&ities. (aws t%'icall% li$it )oth the
"inds of re&enues a local go&ern$ent can raise and the wa%s in which the%
can raise different re&enue t%'es. =n the 2nited States e&aluators $ust
co$$onl% as" whether go&ern$ents
are raising re&enues in a legall% sustaina)le wa% (such that the $ethod of
re&enue raising does not clash with legislation). he :ro'osition 63 de)ate
in California, for e0a$'le, e0a$ined whether the 'rocess of collecting
'ro'ert% ta0es (and &aluing 'ro'ert%) was legal, or whether it contra&ened
e*ual treat$ent clauses in the national constitution. =n de&elo'ing countries
legislation tends to sha'e re&enue1raising a)ilities of local go&ern$ents.
(ocal go&ern$ents are re*uired to adhere to such
legislation. =n Forea, for e0a$'le, local go&ern$ents are gi&en access to
certain ta0es ($ostl% regulation and 'ro'ert%1 )ased) and are disallowed
fro$ using an% "ind of sales or inco$e ta0 to raise re&enues (Shin and >a
677C). =n South Africa local go&ern$ents ha&e a set of ta0 and user1charge
instru$ents identified in law, and face a li$it on lending (with Section 60
of the 677@ (ocal ,o&ern$ent ransition Act stating that, 4+unici'al
loans are not )ac"ed )% 'ro&incial or national go&ern$ents: An% $one%
)orrowed )% a $unici'alit% in accordance with this su)section and the
interest thereon, shall )e the financial o)ligation of the $unici'alit%
concerned and shall )e chargea)le to and 'a%a)le fro$ the re&enues and
assets of that $unici'alit%5). =t is i$'ortant to identif%
whether go&ern$ents in such situations raise re&enues in accordance with
such li$itations, )ecause if the% do not, then their re&enue1raising
'erfor$ance will not )e sustaina)le.
Si$ilarl%, local go&ern$ent )udgeting and 'rocure$ent
'rocesses are t%'icall% sha'ed )% national 1le&el legislati&e
re*uire$ents (or for$al regulations de&elo'ed )% *uasi1 legislati&e
agencies). =n the 2nited States for$al regulations e$anate
fro$ ,AS! and other entities, re*uiring certain "inds of )udgeting
'rocesses and re'orting standards. his is also
the case in de&elo'ing countries, with an e0a$'le co$ing fro$ the 6776
:hili''ines (ocal ,o&ern$ent Code. Cha'ter 3 of
the Code outlines re*uire$ents for the )udgeting and financial re'orting
'rocess in local go&ern$ents, with Section 368, for e0a$'le, re*uiring
(a$ong other things) that local go&ern$ent )udgets 'ro&ide su$$aries of
financial state$ents setting forth inco$e and e0'enditures during the
'receding %ear, the esti$ates of inco$e for the ensuing %ear (deter$ined
within legal 'ara$eters), esti$ated e0'enditures re*uired to carr% out all
local go&ern$ent functions in the %ear, all essential facts regarding long
ter$ o)ligations and inde)tedness of the local go&ern$ent unit, and other
financial state$ents and
data dee$ed necessar% to 4disclose in all 'ractica)le detail the financial
condition of the local go&ern$ent unit.5
=n this situation it is i$'ortant to as" whether
$unici'alities co$'lied with the re*uire$ents.
Si$ilar 'rocess re*uire$ents in de&elo'ing countries relate to 'lanning
and 'artici'ation. =n Ce)u Cit% in the :hili''ines, the cit% go&ern$ent
res'onded 'ositi&el% to such re*uire$ents and ado'ted
&arious sche$es to gi&e su)stance to
'artnershi's in accordance with the (ocal ,o&ern$ent Code. he $ost
co$$on is the s'ecial 'ro;ect contractual t%'e of arrange$ent. 3or instance,
the Ce)u :eo'leEs +ulti 1:ur'ose Coo'erati&e ser&ices the needs of $icro
enter'rises (9te$adi
2000, 2D9SCA 2006).
Contrasting e0'erience to this 'ositi&e e&idence of
legislati&e confor$it% co$es fro$ Dha"a and !angladesh (=sla$ and Fhan
677@: 73). he Dha"a +etro'olitan Area go&ern$ents
faced significant legislation re*uiring 'lanning and 'artici'ation, )ut the
$unici'alities in the area are recorded as not ha&ing an% 'lanning 'rogra$
and not )eing &er% 'artici'ator%. Si$ilarl%, in !angladesh a)out 620
4$unici'alities ha&e a $andate to 'ro'ose and i$'le$ent 'lans, )ut none of
the$ has an% 'lanning de'art$ent or e&en an official ur)an 'lanner.5
Another negati&e e0a$'le co$es fro$ !oli&ia, where
the (aw of :artici'aciGn :o'ular intended to change national and local
'ower structures )% re*uiring 'artici'ation of 'oor grou's in resource
allocation decision1 $a"ing 'rocesses. Andersson (6777) conducted a social
stud% in&ol&ing inter&iews of local officials and citi#ens, and found that
'artici'ation has not %et ha''ened in the $unici'alit% of Cai#a.
A'art fro$ the 'rocess of )udgeting and resource
allocation, local go&ern$ents also face legislated $andates
regarding the t%'e of ser&ice the% ha&e to allocate resources towards.
(egislation t%'icall% identifies such functional
res'onsi)ilities along with re*uire$ents related to ser&ice *ualit% and
standards. =n this light, the South African (ocal ,o&ern$ent ransition Act
of 6773 'ro&ides a )asic reference 'oint for e&aluating $unici'al
'erfor$ance )% identif%ing the following as 4duties of local go&ern$ents:5
4Water su''l%, sewerage 'urification, electricit% if so agreed )% all the
indi&idual local go&ern$ent )odies, refuse re$o&al, roads and stor$water
drainage, health ser&ices, e$ergenc% ser&ices,
financial ad$inistration, and an% other ser&ice agreed u'on5 (Section B (c)
(i)(aa) of the (,A, 6773). (ocal go&ern$ent studies in this setting
t%'icall% )egin )% as"ing: 4Are the local go&ern$ents 'ro&iding these
ser&ices/5 he general answer to such *uestion is %es, with $ost
go&ern$ents at least fulfilling the )asic ser&ices $andate ('ro&iding water
su''l%, sanitation and electricit% ser&ices). !ut legislation in South
Africa and other countries often sti'ulates the functional res'onsi)ilit%
further, s'ecif%ing who should recei&e ser&ices and what standard ser&ices
should )e $et. he South African
Constitution co$$its to 'ro&iding water, electricit%, and sanitation to all,
for e0a$'le, while $ost local go&ern$ents t%'icall% ser&ice less than two1
thirds of their constituents. +itlin (2000:3) suggests that $an% local
go&ern$ents in de&elo'ing countries would fail on a legislati&e re*uire$ent
for full ser&ice 'ro&ision: 4=t is clear that in $ost ur)an centers, local
go&ern$ents fail to $eet $an% of their res'onsi)ilities to large sections of
the 'o'ulation within their ;urisdiction.5
his line of *uestioning is seen in the 2nited States where theoretical
and 'ractical e&aluations of local go&ern$ent often as" whether
local entities (school )oards) 'ro&ide the a''ro'riate
standard of education to all citi#ens. =t is also 'ertinent in countries li"e
an#ania, where local go&ern$ents ha&e a legal role in 'ro&iding 'ri$ar%
schooling (her"ildsen
677C). (ocal go&ern$ents are re*uired to do $ore than
'ro&ide 'ri$ar% education, howe&er, the% are )ound )% a national $andate
to 'ro&ide the sa$e standard education for all citi#ens. -,ood 'erfor$ing.
local go&ern$ents 'ro&ide such standard education according to such
$andate, while -'oor 'erfor$ing. local go&ern$ents
'ro&ide &ar%ing *ualit% education.
(egal confor$ance in areas li"e ser&ice 'ro&ision, )udget
'rocess de&elo'$ent, re&enue raising acti&it%, and local regulator% acti&it%
thus 'ro&ides a foundation for e&aluating local go&ern$ents. his "ind of
e&aluation can also )e useful in 'ro&idingguidance as to
te$'oral li$itations, and co$'le0ities and e&en inconsistencies
with the institutional setting in which local go&ern$ent find the$sel&es (and
which will factor into an% e&aluation). his "nowledge can hel'
e&aluators in deter$ining wh% the local go&ern$ent is 'erfor$ing as it is,
and to 'ro&ide a''ro'riate ad&ice for re$ediation or i$'ro&e$ent.
=n South Africa, for e0a$'le,
local go&ern$ents failed to 'ro&ide ser&ices to all citi#ens in
677@ )ecause territories were e0'anded in that %ear (thus
creating a te$'oral li$itation on local go&ern$ents. a)ilities to a)ide )%
legislation). =n an#ania the 'ro&ision of education is considered
'ro)le$atic in that legislation si$ultaneousl% de&ol&es significant
res'onsi)ilit% to local go&ern$ents so that the% can sha'e education
'ro&ision to local needs, and then re*uires the standardi#ation of 'ro&ision
(her"ildsen 677C). =n Dairo)i, the 'oor regulator% 'erfor$ance of the cit% is
e0'lained as the result of a -disa)ling. legal conte0t that li$ited
resource access for the $unici'alit% and $ade it difficult for such to
a)ide )% legal re*uire$ents (>alfani
677@: 6CC).
Evaluatin Fi#cal $ealt%
3iscal health factors are closel% related to the legislati&e criteria for
e&aluating local go&ern$ents. =n $ost settings,
local go&ern$ents are re*uired to $anage their fiscal $atters carefull%,
ensuring that the% do not o&er1s'end and that their e0'enditure is in line with
their $andate (as it is generall% re'resented in the )udget or as it is
sti'ulated in national1 le&el 'olic% docu$ents or legislation).
=n countries li"e South Africa such re*uire$ents are
legislated, with $unici'alities and districts onl% allowed to s'end
$one% in accordance with agreed1u'on )udgets, and not allowed to run
a deficit on their o'erating accounts. +unici'alities that o&er1s'end are
considered 'oor 'erfor$ers )ecause the% create a fiscal )urden for their
constituents (reducing their a)ilit% to allocate resources to ser&ices in
future 'eriods )ecause of de)t ser&icing co$$it$ents).
+unici'alities that s'end differentl% to their $andate are also considered
'oor 'erfor$ers, as their )udget i$'le$entation does not $atch their
stated o);ecti&es (unless, of course, there are &alid reasons for the
difference )etween )udget 'lan and i$'le$entation).
he )asic criteria for identif%ing 'oor fiscal health on the
e0'enditure side are widel% "nown and easil% o)ser&ed fro$ standard
financial state$ents and )udgets: >igh (and
sustained) deficits and de)t, 'oor allocations (with significant resources
going to ad$inistration rather than to ca'ital $aintenance, for e0a$'le),
and a dis;oint )etween 'lanning
allocations and i$'le$entation. =n ter$s of these, the
literature and 'o'ular 'ress is flush with e0a$'les of 'oor 'erfor$ing
go&ern$ents. he 677C White :a'er in South Africa (D:(, 677C)
reflects on such: 4Co$)ined with ser&ice )ac"logs, colla'sedor
deteriorating infrastructure, and deteriorating
creditworthiness and )orrowing ca'acit%,
$unici'alities are e0'eriencing financial stress, and in so$e instances
crisis.5 :ress re'orts connected to this state$ent relate the fact that,
)% Se'te$)er 677B, local go&ern$ents in South Africa owed ten )illion
<and to &arious lending institutions and ser&ice 'ro&iders
such as 9s"o$ (the electricit% 'roducer) (Fihato
677C). his "ind of de)t led $unici'alities li"e Dels'ruit (the ca'ital of
+'u$alanga 'ro&ince) to sus'end ca'ital 'ro;ects and caused a ?C 'ercent
reduction in Hohannes)urgEs ca'ital )udget, resulting in 4the 'ro&ision and
$aintenance of infrastructure I)eingJ affected dra$aticall%5 (Fihato 677C).
Dot all go&ern$ents are 'oor 'erfor$ers in this categor%,
howe&er. wo large cities in South Africa, Dur)an and Ca'e own, were
until recentl% gi&en strong fa&ora)le -e&aluations. )% 'olic%$a"ers and in
the 'o'ular 'ress )ecause of their low deficits, high ca'ital and ser&ice
e0'enditures, and strong adherence to stated )udgetar% goals. !oth
$unici'alities ha&e recentl% )een e0'eriencing so$e financial difficulties,
howe&er, running deficits and dis'la%ing fiscal &ulnera)ilit%. his
shows the ti$e1s'ecific nature of the fiscal health e&aluation criteria, which
can 'ortra% a -health%. cit% in one %ear (with low deficits, for e0a$'le) and a
-&ulnera)le. cit% the ne0t (with high deficits, for e0a$'le).
he fiscal health criteria are also onl% as relia)le as the )udget and
financial re'orting 'rocess that deter$ines
financial figures. =n countries li"e China, local go&ern$ent off )udget
re&enues and e0'enditures were e0tre$el% large in the $id16770s,
introducing 4a degree of non 1trans'arenc% in the fiscal 'rocess5 and
%ielding deficits and allocations figures unrelia)le in ;udging true fiscal
health (Arora and Dorregaard
677B: 20). 9&aluators in such instances are re*uired to e0a$ine
not onl% the figures reflecting fiscal health on the e0'enditure side, )ut
also the 'rocesses )% which e0'enditures
are re'orted. Diesner (6777: 32) descri)es this e&aluation re*uire$ent
as such: 4Chec" co$'liance with accounting controlsKwhether rules
had )een followed,
'rocedures had )een esta)lished, 'ro'er accounting had )een $ade, and
controls were in 'lace and functioning.5
=n tande$ with the re*uire$ent that go&ern$ents $aintain disci'line on
the e0'enditure side, the literature also stresses that fiscall% health% local
go&ern$ents should ha&e their own relia)le re&enue sources (!ird 6773,
Aates 6773: 286). An
e&aluation of this as'ect of fiscal health re*uires anal%#ing the si#e of the
local re&enue )ase (the 'otential local resources a&aila)le to the
go&ern$ent) and the re&enue effort on dis'la% (the actual local resources
raised as a 'ercentage of the 'otential). As with other as'ects of the
fiscal health e&aluation criteria, there is a strong o&erla' with legislation in
this area. (egislation often deter$ines the si#e of the re&enue )ase )%
defining ta0 and user1charge a)ilities of local go&ern$ents. (ocal
initiati&es within such legislated 'ara$eters tend to deter$ine
'erfor$ance in ter$s of re&enue effort, howe&er. =t is not unco$$on to
find local go&ern$ents collecting re&enues inefficientl% in de&elo'ing
countries, using out1dated $odes of 'ro'ert% e&aluation or $easure$ent
of ser&ice access (li"e electricit% use), for e0a$'le.
,o&ern$ents that 'erfor$ well in ter$s of the re&enue criteria of
the fiscal health factor are identified as those that $a0i$i#e the si#e of their
re&enue )ase while at the sa$e ti$e ensuring that the% do not create social
inefficiencies when le&%ing ta0es or user fees (!ird 6773: 262). 90a$'les
of such go&ern$ents include those in South Africa that
ha&e de&elo'ed de)tor follow1u' $echanis$s and indigent
de)tor 'olicies to ensure that all constituents 'a% for ser&ices recei&ed
as the% area)le and Ce)uCit%.s use of
co$'uteri#ed assess$ent and a''raisal 'rocedures related to 'ro'ert% ta0es
(2% 677B:60?). 3urther e0a$'les include A)id;an.s
esta)lish$ent of neigh)orhood co$$ittees to engage the resources of
co$$unities in addressing econo$ic and infrastructure 'ro)le$s (D:(,
2000)
2
and the go&ern$ent 'artnershi' with co$$unit% 1)ased
organi#ations for ta0
2 =n this case co$$unit% co$$ittees wor" with the $unici'alit% to i$'ro&e street
cleaning, gar)age collection, securit% ser&ices, road $aintenance, street lighting, and to
i$'ro&e collection of ta0es and user fees associated with such ser&ice 'ro&ision.
he% ha&e collected
L620,000 in fees and ha&e $o)ili#ed L20,000 for s'ecific 'ro;ects, including
infrastructure de&elo'$ent.
collection in the Si"asso co$$une in +ali (Attahi 677B: 6C2).
3
=n all these e0a$'les the $unici'alities are seen to )e
addressing their need for a strong local re&enue source )% enhancing their
a)ilities to access the re&enue )ase (through creati&e ad$inistrati&e and
$anagerial $ethods).
A further wa% in which $unici'alities are seen to do such is
through 'assing legislation that i$'ro&es the functioning of their ta0 or fee
$echanis$s. =n Fen%a, local go&ern$ents
changed their s%ste$ of ha&ing $ulti'le )usiness licenses to ha&ing one
single )usiness 'er$it so as to si$'lif% their licensing 'rocess. he $o&e
resulted in an e0'ansion of the ta0 )ase (with all econo$ic acti&ities
now included in the instru$ent) and a strea$lined licensing 'rocess
(reducing their own collection costs as well as those of )usinesses). he ta0
has enhanced their fiscal status )% increasing re&enues (De&as and Fell%
2006).
hese e0a$'les show that fiscal health is a)out )oth the
recorded nu$)ers on financial state$ents and the 'rocesses
underl%ing such nu$)ers. !oth the nu$)ers and 'rocesses are generall%
accessi)le to e&aluators, and for$ a con&entional source of e&aluation
infor$ation. ogether with confor$ance to legislation, the fiscal health
criteria is co$$onl% used to e&aluate go&ern$ents in the
de&elo'ing world. Decentrali#ation literature holds,
howe&er, that a fiscall%
-health%. local go&ern$ent, that s'ends res'onsi)l% and has a
relia)le and si#ea)le local re&enue source, can onl% )e considered effecti&e if
it uses such resources in res'onsi&e, efficient and accounta)le wa%s (Aates
67B2, Aates 6773, Shah
677C).
Evaluatin Re#&on#ivene##
=n$an (67B7: 2BC12B7) argues that decentrali#ation in a de$ocratic
conte0t should lead to greater res'onsi&eness to constituent de$ands,
sa%ing that, 4Candidates $ust 'lease the &oters if the% ho'e to )e re
1elected.5 Si$ilarl%, Aates (6773: 280) argues that a 4case for
decentrali#ed finance is )ased on the 'resu$ed res'onsi&eness of local
go&ern$ents to the welfare of their res'ecti&e constituencies.5 hese
state$ents underlie a large 'ortion of the theoretical argu$ent in fa&or of
decentrali#ation: (ocal go&ern$ents are $ore li"el% to 'ro&ide the -right.
ser&ices than are higher1le&el go&ern$ents. he%
also offer a 'ractical a''eal in the de&elo'ing countr% conte0t, where
local go&ern$ents are often created in reaction to the low le&els of
res'onsi&eness )% central go&ern$ents. !ut literature and 'ractical
e0'erience show that not all local go&ern$ents are res'onsi&e to their
constituents, e&en if the% confor$ to legislation and $aintain high le&els of
fiscal health (Con%ers 6770, Aates 6773).
=n a $unici'al case fro$ South Africa, for e0a$'le, o)ser&ers found that
s'ending 4decisions are often directed )% a )ureaucratic agenda and
citi#en.s in'uts are li$ited,5 constraining the 'otential for local go&ern$ent
res'onsi&eness (3oundation for Conte$'orar% <esearch 6777: 8?). =n
the !oli&ian conte0t, Anderson (6777) o)ser&es that
ad$inistrators see 'o&ert% reduction as a hindrance to other econo$ic
de&elo'$ent o);ecti&es and ignore the de$ands of ca$'esinos ('easants)
(+Mrner 67CB, >arris 677?, :oole
677B). =n )oth cases the lac" of local go&ern$ent
res'onsi&eness (es'eciall% to the 'oor) results fro$ the structure of local
go&ern$ent 'rocesses and $anifests in su)1 o'ti$al s'ending acti&it% at the
local le&el. Such 'rocesses and outco$es constitute the criteria for
assessing
-res'onsi&eness 'erfor$ance. in de&elo'ing countr% local go&ern$ents.
,i&en e0'erience in the literature, two as'ects of the local go&ern$ent
ser&ice 'ro&ision 'rocess re*uire s'ecific attention in an% e&aluation of
res'onsi&eness: he le&el of local 'olitical influence on allocations
decisions and the le&el of ci&ic 'artici'ation in the decision1$a"ing
'rocess. he e0tent of
the 'o&ert% focus in de&elo'ing countr% local go&ern$ent allocations is
identified as a "e% outco$es indicator of res'onsi&eness.
he literature shows that local 1le&el res'onsi&eness is
eroded )% a high le&el of central go&ern$ent 'olitical influence on decisions
of local go&ern$ent ser&ice 'ro&ision. !en;a$in
(2000: 8?) finds that higher le&el go&ern$ent 'olitical and
ad$inistrati&e a''oint$ents created a hierarchical structure in !angalore
that $ade local go&ern$ents $ore res'onsi&e to central and regional
go&ern$ent de$ands than the% were to
the needs of local constituents. De&as and For)oe (2000) ha&e a si$ilar
finding in their stud% of Fu$asi in ,hana, where local 'olitical and
ad$inistrati&e re'resentati&es were strongl% influenced )% central
go&ern$ent, with the chief e0ecuti&e of the district and B0 'ercent of the
local asse$)l% a''ointed )% central entities. he% also found
that the Fu$asi local go&ern$ent was not &er% accounta)le or res'onsi&e
to its own citi#ens. =n this light, +itlin (2000:C) notes that the 4influence of
lin"s with higher le&els of go&ern$ent on local go&ern$ents5 seriousl%
i$'edes the res'onsi&eness of local entities.
8
=n ter$s of such
studies it is a''arent that local 1le&el
res'onsi&eness is enhanced where 'oliti cians and
ad$inistrators are a''ointed locall% through regular
de$ocratic 'rocesses, and where resources are sourced locall% and 'olitical
re'resentati&es are re*uired to in&ol&e citi#ens in allocation decisions and
i$'le$entation $onitoring acti&ities (!lair 2000).
According to such criteria, the count% of :Nre# OeledGn in
Costa <ica 'ro&ides an e0a$'le of a high e&aluation case (at least in
ter$s of ser&ice res'onsi&eness 'rocesses). A count% re'resentati&e in
:Nre# OeledGn cha$'ioned the de&elo'$ent of a 'rogra$ gi&ing local
citi#ens direct authorit% o&er discretionar% funds allocated to the$
fro$ higher le&els of go&ern$ent (!reslin and Ca$')ell 6777). hese
discretionar% funds were traditionall% used )% re'resentati&es to de&elo'
lines of 'atronage and clientelis$ in their co$$unities, )ut are now
channeled directl% to district councils for local allocation. he
district councils ha&e to show e&idence of direct consultation with their
citi#ens when using the funds. his re*uire$ent is a shift fro$ the old to'1
down a''roach of deciding s'ending allocations (!reslin and Ca$')ell
6777). (iterature suggests that the resource allocation 'rocess is $ore
res'onsi&e and co$$unities can see wor" )eing targeted and co$'leted
(A'ti# 677C, =nter A$erican De&elo'$ent !an"
2000). here is e&idence that ad$inistrators are recei&ing
8
Fharoufi (677B: B6) 'ro&ides a s'ecific e0a$'le of this effect in +orocco.
he author writes that the hierarchical structure in which local go&ern$ents are
located significantl% i$'edes res'onsi&eness: 4A $ulti'licit% of agencies and
co$$issions $a% )e a''ointed to initiate 'ro;ects which gi&e little i$'ortance to local
constraints or to the o&erall as'ects of the ur)an renewal 'ro)le$.5
$ore 'recise de$ands for actual results and 'u)lic entities in the count% are
$ore res'onsi&e and out'ut1oriented than in other counties, )orne out in
lower relati&e res'onse ti$e s to disasters.
he :Nre# OeledGn case s'ea"s to the i$'ortance of co$$unit%
'artici'ation in enhancing the res'onsi&eness of local go&ern$ents.
+itlin (2000) argues that 'artici'ator% local
go&ern$ents tend to )e res'onsi&e go&ern$ents as well. She cites a nu$)er
of cases of $odel 'artici'ator% 'rogra$s at the local le&el in which citi#en
&oice was elicited fro$ all 'arts of the local co$$unit% (not ;ust the
wealth%) and where local 'oliticians and ad$inistrators too" citi#en
co$$ents seriousl% ()% recording and res'onding to the$ all, for e0a$'le).
hese include the health% cities 'rogra$ in (eon, Dicaragua, the
en&iron$ent and de&elo'$ent 'rogra$s in =lo, :eru and Colu$)ia and the
'artici'ator% )udgeting 'rogra$s in !elo >ori#onte and :orto Alegre,
!ra#il.
he :orto Alegre case is an e0cellent e0a$'le of a
'artici'ator% 'rogra$ that enhances local go&ern$ent
res'onsi&eness. he :artici'ator% !udget in :orto Alegre (the ca'ital of the
state of <io ,rande do Sul) in&ol&es citi#en re'resentation on a !udget
Council and a !udget 3oru$. Co$$unit% re'resentati&es discuss the )udget
in these settings, with their contri)utions ta"en u' )% a s'eciall% de&elo'ed
ad$inistrati&e unit, the :lanning Ca)inet (,a'lan), which 'rocesses
co$$unit% de$ands and transfor$s the$ into go&ern$ent 'lans. A
Co$$unit% <elations Coordination Co$$ittee wor"s with ,a'lan to
$aintain acti&e records of 'artici'ation and ensurethat the
co$$unit% and its re'resentati&es recei&e feed)ac" on
the $anner in which their de$ands affect 'lans. he nu$)er of citi#en
interactions with go&ern$ent through this 'rocess increased fro$ 6,000
in
67C7 to 60,000 in 6773, with @8.? 'ercent of 'artici'ants classified as
'oor.
his case can )e contrasted with con&entional for$s of
'artici'ation in local go&ern$ent structures in de&elo'ing countries that
often fail to enhance res'onsi&eness (es'eciall% to the 'oor). =n South
Africa, for e0a$'le, local go&ern$ents are re*uired to in&ol&e citi#ens in
)udgeting 'rocesses, )ut seldo$ do so effecti&el% (:utu 2006, Andrews,
forthco$ing). =n the (ichten)urg $unici'alit%, where 'o&ert% le&els were
a)o&e ?0 'ercent in 677@, case re'orts state that although
'artici'ator% $echanis$s were in 'lace, ad$inistrators
selecti&el% ignored citi#en in'uts e$erging fro$ 'u)lic
$eetings: 4:ro)le$ 'erce'tions, which were not in line with facts and
figures or the 'rofessional ;udg$ent of officials and councilors, were
e0cluded5 (DCD1,O 6777: Dorth West :ro&ince Stud% 8). A si$ilar
stor% e$erges fro$ the ha)a Dchu local go&ern$ent, where the
'artici'ations a''roach was $ostl% seen to 4reflect a co$'liance a''roach5
with infor$ation collected fro$ disad&antaged grou's )eco$ing 4lost5 in
the 'rocess (DCD1,O 6777: 3ree State Stud% 6B, 6C). =n such cases
although 'oor co$$unitieshad the $echanis$s to
'artici'ate, their 'artici'ation was $uted and did not e$'ower the$ or
increase their $unici'alit%.s res'onsi&eness to their needs.
A'art fro$ those 'rocess, as'ects that distinguish
res'onsi&e local go&ern$ents fro$ non1res'onsi&e local
go&ern$ents in de&elo'ing countries, the literature also e$'hasi#es
e&aluating res'onsi&eness in ter$s of the 'ro1'oor 'ers'ecti&e e&ident in
)udget allocations. :o&ert% is a constant in the de&elo'ing world,
and res'onsi&eness to local need necessaril% i$'lies res'onsi&eness to
'o&ert%1related issues in such settings. =n this regard +itlin (2000: 3,?)
o)ser&es that all $unici'alities in the de&elo'ing world 'la% a $a;or role in
'ro&iding ser&ices 4that are critical5 for the 'oor, )ut that, 4he critical role
of )asic ser&ices in reducing 'o&ert% is often forgotten or dis$issed.5
Authors li"e +itlin and :orio (677B:3) identif% a nu$)er of indicators of the
'ro1'oor 'ers'ecti&e in local go&ern$ents, including the a$ount s'ent on
housing and the reha)ilitation or u'grading of slu$s and s*uatter areas,
$one% s'ent on de&elo'ing new infrastructure for)asic ser&ice
'ro&ision(li"e water, electricit%, and sanitation) and in
'ro&iding such ser&ices in 'oor areas, and the locall% financed acti &ities
ai$ed at sti$ulating s$all, infor$al co$$ercial enter'rises (li"e
the 'ro&ision of acco$$odation for haw"ers or the i$'ro&e$ent
of street trading facilities).
hese indicators are generall% o'en to e&aluation in a 'ractical sense,
with o)ser&ers )eing a)le to see the results of local go&ern$ent
inter&entions in 'oor areas. Where it is not
e&ident that local go&ern$ents are 'ro&iding ser&ices in 'oor areas (or
de&elo'ing infrastructure for such), or where local go&ern$ents are
re'ressing infor$al traders, it is o)&ious that
the go&ern$ent has not ado'ted a 'ro1'oor attitude. =n the de&elo'ing
world conte0t this is tanta$ount to sa%ing that the local go&ern$ent has
ado'ted a stance of li$ited res'onsi&eness
()eing res'onsi&e to elite de$ands rather than general constituent
de$ands). =n situations where go&ern$ents
detail their fiscal allocations 'rocesses, such o)ser&ation1)ased e&aluation
can )e su''le$ented with references to )udgets
and financial state$ents. An D,A called D=S>A (De&elo'ing
=nitiati&es for Social and >u$an Action) was a)le to access and anal%#e
'ro&incial )udgets in Western =ndia to identif% e0'enditure going to 'oor
grou's largel% )ecause the 'ro&incial )udget was &er% detailed and
legislation allowed ci&ic access to such detail (D=S>A 2000). =n their
-e&aluation. 'rocess, the% initiall% found that the
'ro&incial go&ern$ent was &er% unres'onsi&e to the needs of the 'oor.
?
Evaluatin E!!icienc'
As with res'onsi&eness, theor% 'resents 'otential efficienc% gains as a
$a;or reason wh% go&ern$ents should decentrali#e. Aates (6773: 280) states
that, 4he )asic econo$ic case for fiscal decentrali#ation is the
enhance$ent of econo$ic efficienc%.5 Si$ilarl%,
Arora and Dorregaard (677B: 8) state that, 4An i$'ortant rationale for
esta)lishing a decentrali#ed go&ern$ent . . . is to i$'ro&e econo$ic
efficienc%.5
@
Si$'l% 'ut, the argu$ent is that local go&ern$ents are not
onl% $ore li"el% to 'ro&ide the -right. ser&ices, the% are also $ore li"el% to
'ro&ide these ser&ices in the -right. wa% than are higher1le&el go&ern$ents.
!ut not all local go&ern$ents 'ro&ide ser&ices efficientl%. he cit% of
!a$a"o in +ali was "nown to ha&e
?
D=S>A.s :artici'ator% !udget Anal%sis 'rogra$ has de 1$%stified the 'rocess of
go&ernance through 'ro1acti&e )udgetar% anal%sis in&ol&ing 'ro1'oor D,As. he
)udget anal%sis has hel'ed to "ee' go&ern$ent accounta)le for general $one% flows
and 'olic% decisions. he 'ress and local acti&ist grou's, ha&e a de&ice to assist the$ in
understanding what go&ern$ent does, closing the 'ower ga' )etween infor$ed
)ureaucrats and uninfor$ed citi#ens (D=S>A 2000).
@
Another reference e$'hasi#ing the i$'ortance of )oth efficienc% is 'ro&ided )%
!ird (6773: 20B): 4+an% de&elo'ing countries are turning to &arious for$s of fiscal
decentrali#ation as one wa% of esca'ing fro$ the tra's of ineffecti&e and inefficient
go&ernance.5
'articularl% inefficient 'u)lic sanitation s%ste$s, for e0a$'le, with 70
'ercent of the e0'ensi&e indi&idual drainage s%ste$s )eing non1functional
and $ost residents ti''ing li*uid waste onto 'u)lic streets (D:(, 2000).
his "ind of waste could )e identified through acti&e e&aluation of
efficienc%.
(ocal go&ern$ent efficienc% has a nu$)er of co$'onents, including cost
efficienc% and co$'etiti&eness. he decentrali#ation literature
assu$es that local go&ern$ents will 'ro&ide ser&ices at lower cost than larger
go&ern$ents, and in a co$'etiti&e wa%, )ecause of the 'resence of alternati&e
ser&ice 'ro&iders (including other local go&ern$ents) (ie)out 67?@). his
assu$'tion is 'articularl% )ased on the argu$ent that citi#ens can &oice their
disa''ro&al with inefficient ser&ice 'ro&ision (through &oice $echanis$s) or can e0it
inefficient ;urisdictions (in fa&or of other ;urisdictions or of non1 go&ern$ental
'roducers in their own ;urisdiction). 9fficienc% can )e e&aluated
either )% e0a$ining the costs and co$'etiti&eness of
fiscal outco$es, or )% e0a$ining the &oice and e0it $echanis$s in 'lace in local
go&ern$ents (and the effect the% ha&e on 'roduction and 'ro&ision )eha&ior).
A stud% shows, for e0a$'le, that the efficienc% of !a$a"o.s cess'ool
o'erations i$'ro&ed significantl% in the 6770s, with records of decreased
ser&ice costs and i$'ro&ed ser&ice *ualit% (and co$'etiti&eness).
=n the 6770s the $unici'alit% decreased the cost of
drainage )% C0 'ercent and )uilt 6,C00 new cess'ools (D:(, 2000).
hese efficienc% and related 'roducti&it% gains, as reflected in
easil% identifia)le outco$es, $irror those re'orted on in the )est 2nited
States cit% go&ern$ent e0'eriences. =n the case of residential street
swee'ing in San Diego, for e0a$'le, ser&iceregularit%
increased in the late 6770s fro$ once a %ear to )etween fi&e and ten ti$es a
%ear, with costs decreasing at the sa$e ti$e (Andrews and +o%nihan
2002). he gain is e&en greater in ter$s of co$$ercial swee'ing
ser&ices (where)% roads in co$$ercial areas are cleaned). he efficienc%
gains in this setting are easil% identified )ecause the cit% directl% co$'ares
costs and ser&ice access with 'ri&ate alternati&es.
B
B
he infor$ation regarding San Diego.s 'erfor$ance in these areas co$es fro$
an inter&iew with 9d :lan" in August 677C as well as docu$entation 'ro&ided )% +r.
:lan".
(ocal go&ern$ent efficienc% can also )e e&aluated in ter$s of the
'rocesses used in ser&ice 'roduction. San Diego.s efficienc%
gains are largel% related to the fact that the cit% ado'ted a $anaged
co$'etition initiati&e to sti$ulate co$'etition
)etween cit% ser&ice de'art$ents and 'ri&ate fir$s offering si$ilar
ser&ices (Andrews and +o%nihan 2002). =n doing so, the local go&ern$ent
clearl% identified the -e0it. o'tions citi#ens en;o%ed (with citi#ens a)le to
e$'lo% 'ri&ate alternati&es instead of the cit% entities). Co$'etiti&e 'ressure
arising fro$ such identification led to the cit% ado'ting an a''roach to
)ench$ar"ing their 'erfor$ance against 'ri&ate alternati&es, which
facilitated internal ser&ice i$'ro&e$ents. Si$ilarl%, !a$a"o.s efficienc%
gains were the result of a creati&e 'roduction arrange$ent )etween the
cit% and an D,A, 4Higui5 (D:(, 2000). he D,A 'ro'osed that
residents )e res'onsi)le for 3B 'ercent of the costs of cess'ool construction,
and that @3 'ercent )e co&ered through a go&ern$ent s'ecial
de&elo'$ent fund. =t 'la%ed a coordinating role
with the co$$unit% and $anaged the cess'ool o'erations. =n
creating this colla)oration, the !a$a"o cit%
go&ern$ent created the ca'acit% for alternati&e ser&ice 'ro&ision ()% the
D,A) as well as a ci&il societ% source of co$$entar% a)out the efficienc% of
go&ern$ent ser&ice 'ro&ision. he new e0it and &oice o'tions sti$ulated a
$ore co$'etiti&e ser&ice 'ro&ision 'erfor$ance.
Ather e0a$'les of local go&ern$ents 'roducing ser&ices through
'rocesses that enhanced efficienc% include Ce)u Cit%, !uenos Aires and
Ah$eda)ad. Ce)u Cit% de&elo'ed a 'rocess of co$'aring ser&ice
'erfor$ance in cit% 'ro;ects (si$ilar to a )ench$ar"ing 'rocess) and
allocated e0tra resources to 'ro;ects that 'erfor$ed well, and to D,A.s with
a trac" record of deli&ering in areas related to 'o&ert% alle&iation
(9ta$adi 677B: 7C177). he <a$on A)oiti# 3oundation, =nc. was
su)contracted to $anage relief and reha)ilitation wor" in the after$ath of
%'hoon <u'in)g, for e0a$'le, while the D,A :agta$)a%a%ong was
)rought in to reha)ilitate ur)an 'oor districts. !uenos Aires 'ri&ati#ed
water and sanitation ser&ices to Aguas Argentinas, auto$aticall% creating
an alternati&e ser&ice 'ro&ider (>ardo% and Schuster$an 2000). Although
the 'ri&ati#ation had its 'ro)le$s (with the cit% not structuring the agree$ent
to include an incenti&e for the 'ri&ate fir$ to ser&e 'oor 'eo'le) it still
reflected an atte$'t to
i$'ro&e go&ern$ent efficienc%. he cit% of Ah$eda)ad ado'ted
inno&ati&e ser&ice 'ro&ision arrange$ents and
financing when it in&ol&ed itself in a nu$)er of strategic 'artnershi's in
ur)an de&elo'$ent. Ane of these is with the Slu$ Detwor"ing :ro;ect,
which wor"ed with slu$ dwellers as 'artners to re1de&elo' a street (Dutta
2000). =n this case the creati&e ser&ice 'ro&ision arrange$ent i$'ro&ed
efficienc% and facilitated greater res'onsi&eness to the
needs of 'oor constituents (the go&ern$ent was a)le in
this instance to 'roduce the right ser&ices in the right wa%).
Evaluatin Accounta(ilit'
Alongside i$'ro&ed res'onsi&eness and efficienc%,
enhanced accounta)ilit% is often seen as an e0'ected gain fro$
decentrali#ation. !ird (6773: 20C) identifies
accounta)ilit% gains as a central the$e in 4localist5
argu$ents. hese gains arelargel% e0'ected )ecause
decentrali#ation shifts go&ern$ent authorit% 4closer to the 'eo'le5 (Shin
and >a 677C). As with res'onsi&eness and efficienc%, howe&er,
accounta)ilit% gains are not alwa%s e&ident in local go&ern$ents. Such
gains re*uire e&aluation to esta)lish an effecti&e &iew of the *ualit% of local
go&ernance.
he word -accounta)ilit%. is $ost often used in relation to go&ernance 'rocesses as
it refers to 'olitical accounta)ilit% to citi#ens.
C
=n ter$s of decentrali#ation
theor% local 1le&el re'resentati&es are $eant to )e $ore accounta)le to their
constituents (=n$an 67B7). he theor% assu$es that local constituents ha&e
the a)ilit% to call such re'resentati&es to account for their actions and for their
'erfor$ance, and to disci'line or reward re'resentati&es accordingl%. his is
onl% 'ossi)le where 'olitical re'resentation is highl% locali#ed, howe&er, and citi#ens
ha&e infor$ation a)out go&ern$ent 'erfor$ance and &oice and e0it channels that
can effecti&el% )e used to ensure accounta)ilit% of their re'resentati&es. his is not
the case in $an% go&ern$ents, where citi#ens lac" infor$ation a)out 'erfor$ance nor
the $echanis$s necessar% for &oicing a res'onse.
C
Accounta)ilit% in this sense $eans 4holding indi&iduals and organi#ations res'onsi)le
for 'erfor$ance $easured as o);ecti&el% as 'ossi)le5 (:aul 677@: 3B).
,o&ern$ents co$$onl% internali#e infor$ation and ele&ate the
"nowledge of technical, $anagerial and 'olitical elites o&er the local -ti$e
1'lace. "nowledge of grassroots actors and constituents. Such
-trans'arenc% reducing. a''roaches (and the de&ices associated with such)
underscore an official and unofficial li$it on citi#en infor$ation accessK
infor$ation is not 'ro&ided to social grou's )ecause their in'ut is not &alued
or re*uired in the go&ernance 'rocessKand legiti$ates the e0clusion of
citi#ens (es'eciall% 'oor citi#ens) fro$ decision1 $a"ing. =t reinforces
an inward, $anagerial $odel of go&ernance, and 'rohi)its local1le&el
accounta)ilit%.
Cases where local go&ern$ent accounta)ilit% has i$'ro&ed in recent
%ears are re$ar"a)l% different to this, e0hi)iting high le&els of
infor$ation disse$ination to societ% at large.
=nfor$ation is disse$inated through D,As in so$e cases (as in
!angalore.s <e'ort Card5 (:aul 677@) and D=S>A.s )udget de$%stification
acti&ities (D=S>A 2000)). =n other cases local go&ern$ents the$sel&es
re'ort on 'erfor$ance infor$ation andPor 'rocess detail in highl% 'u)lic
&enues (with >ong Fong )eing a 'ro$inent e0a$'le ((a$ 677B)).
he 6776 (ocal ,o&ern$ent Code in the :hili''ines re*uires that
local go&ern$ents re'ort on 'erfor$ance, facilitating strong
accounta)ilit% lin"s )etween citi#ens and their re'resentati&es. Section 36@
(h) of the Code re*uires local go&ern$ents to, 4Conduct se$i 1annual re&iew
and general e0a$ination of cost and acco$'lish$ents against 'erfor$ance
standards a''lied in underta"ing de&elo'$ent 'ro;ects. A co'% of this
re'ort shall )e furnished to the local chief e0ecuti&e and the Sanggunian
concerned, and shall )e 'osted in cons'icuous and 'u)licl% accessi)le
'laces in the 'ro&inces, cities, $unici'alities and
!aranga%s.5 =n such cases, access to 'erfor$ance infor$ation
(es'eciall% when )ench$ar"ed against goals or co$'ara)le
alternati&es) facilitates a challenge )% citi#ens against the
4o)scurantist and re$ote culture of the )ureaucrac%, and reinforces
de$ocratic notions regarding the o)ligations of go&ern$ent officials and
elected re'resentati&es as 'u)lic ser&ants5 (Hen"ins and ,oet#
6777:@0?).
=n order for citi#ens to hold local go&ern$ents accounta)le, howe&er, the% need
channels through which the% can $ount such challenges (and res'ond to
infor$ation the% recei&e). Strong $onitoring and e&aluation
and &oice and e0it
$echanis$s facilitate such, and are nota)l% lac"ing in $an% local
go&ern$ents (:aul 6772, 677@). he $ost )asic of these $echan is$s is
the regular 'olitical choice 'rocess (where citi#ens ha&e an
o''ortunit% to &oice their res'onse to whate&er e&idence the% ha&e of
their 'olitical re'resentati&es. 'erfor$ance). (ocal go&ern$ents in
countries li"e Digeria lac" e&en this $echanis$, with re'resentati&es
a''ointed )% central go&ern$ent and citi#ens li$ited in their a)ilit% to
&oice a''ro&al or disa''ro&al for local go&ern$ent 'erfor$ance
(Ani)o"un 677B: 600). :olitical re'resentati&es in such
settings are unaccounta)le to local citi#ens largel% )ecause the $echanis$s
facilitating citi#en influence are $issing. =n
contrast to this, cities that would )e candidates for a high e&aluation
on the accounta)ilit% criteria ha&e acti&e electoral 'rocesses in 'lace
andPor other &oice $echanis$s citi#ens can use to call their re'resentati&es
to account for 'erfor$ance. Ce)u Cit%, for e0a$'le, facilitates citi#en
engage$ent through rallies, fora, audiences with the $a%or, 'artici'ation
in local 'lanning )odies, and the creation of citi#en $onitoring entities
(9ta$adi 2000). Citi#en &oice is acti&el% recorded in such
settings, and in $an% cases there is e&idence that local officials
res'ond directl% to those 'artici'ating through such (to indicate how
their in'ut affected decisions or out'uts in the go&ernance 'rocess).
A final area in which the literature increasingl% suggests that accounta)ilit%
re*uires e&aluation relates to local 1le&el inno&ation. =n $an% local
go&ern$ents, 'ri&ati#ation and other for$s of ser&ice 'roduction are )eing 'ursued
in rather unaccounta)le wa%s, leading to low le&els of ser&ice 'ro&ision for
constituents. !enin Cit% in Digeria 'ri&ati#ed a nu$)er of its ser&ices, including solid
waste, for e0a$'le. he 'ri&ati#ed s%ste$ left a lot to )e desired with regressi&e
'atterns of ser&ice access in e&idence (Agu 2000). he 'ro)le$ in !enin Cit% la% not
in the idea of 'ri&ati#ation, )ut in the fact that the go&ern$ent a''eared to &iew this
"ind of inno&ation as a wa% out of accounta)ilit% (washing its hands of
res'onsi)ilit% for 'ro&ision following the 'ri&ati#ation). he lesson is that
inno&ati&e solutions to 'u)lic sector 'ro)le$s need to )e de&elo'ed within the conte0t
of general accounta)ilit%, and should )e e&aluated in such conte0t as well.
his is 'articularl% i$'ortant in low1inco$e areas where $ar"et
solutions to 'u)lic 'ro)le$s need to )e e0tre$el% well de&elo'ed to a&oid
regressi&e effects
.
Conclu#ion
his 'a'er as"s the *uestions: What does a -good. local
go&ern$ent loo" li"e in the de&elo'ing world/ What factors
should one consider when e&aluating local go&ern$ents in such settings/ =n
res'onse to such *uestions fi&e factors were identified as constituting the
)road criteria e&aluators of local go&ern$ents should concentrate.
Co$)ining the discussion of each factor, a $odel local go&ern$ent would:
Confor$ to legislation in 'rocess and structure +aintain its
fiscal health (in outco$es and 'rocesses) Do the right things ()e
res'onsi&e)
=n the right wa% (with $a0i$u$ efficienc%) and
!e accounta)le to its constituents (in 'rocesses and for its out'uts
and outco$es).
his &iew of local go&ern$ent is founded on 'ractice and theor%,
and e0a$'les of -good. and -)ad. local go&ern$ents show so$e detail
of what is $eant in ter$s of each factor. 9ach of the fi&e factors
(legislati&e confor$ance, fiscal health, res'onsi&eness, efficienc%, and
accounta)ilit%) are $ulti1 di$ensional and re*uire e&aluation in ter$s of
)oth 'rocesses and outco$es. Such e&aluation is argued to 'ro&ide a
$ore co$'lete, a''lied and a''ro'riate &iew of local go&ern$ent
*ualit% in de&elo'ing countries than other alternati&es that t%'icall%
concentrate on legal confor$ance and fiscal health alone. he e&aluation
a''roach is also designed to facilitate o)ser&ation1)ased e&aluation, such
that interested 'arties will )e a)le to identif% -good. fro$ -)ad. local
go&ern$ents in the de&elo'ing world )% what the% see.
Re!erence#
Andersson, Vi)e"e. 6777. Po!ular Partici!ation in "olivia# Does the $aw
%Partici!aci&n Po!ular' Secure Partici!ation of the Rural "olivian
Po!ulation( Center for De&elo'$ent <esearch Wor"ing :a'er 77.@.
Co'enhagen: Center for De&elo'$ent <esearch.
Andrews, +atthew, and Don +o%nihan. 2002. 4Wh% <efor$s Don.t Alwa%s
>a&e to -Wor". to Succeed: A ale of wo +anaged Co$'etition
=nitiati&es.5 Public Performance and Management Review, +arch.
Arora, Vi&e", and Hohn Dorregaard. 677B. )ntergovernmental *iscal
Relations# the +hinese System in Pers!ective.
Washington, D.C.: =nternational +onetar% 3und, Asia and :acific De't.
and 3iscal Affairs De't.
Asian De&elo'$ent !an". 6777. 4(ocal ,o&ern$ent 2nits in the
:hili''ines to )e >el'ed to Attract :ri&ate Sector.5 ,sian Develo!ment "ank
-ews Release -umber .//0.111.
htt':PPwww.ad).orgPDocu$entsPDewsP6777Pnr6777633.as'.
Attahi, F. 677B. 4Decentralisation and :artici'ator% 2r)an ,o&ernance in
3ranco'hone Africa.5 =n +ar" Swilling (ed.), Governing ,frica's
+ities. Hohannes)urg: Witwatesrand
2ni&ersit% :ress 6@6Q260.
!en;a$in, Solo$on. 2000. 4,o&ernance, 9cono$ic Settings and
:o&ert% in !angalore .5 2nvironment and 3rbani4ation 62(6):
3?1?@.
!ird, <ichard. 6773. 4hreading the 3iscal (a)%rinth: So$e
=ssues in 3iscal Decentrali#ation .5 -ational 5a6 7ournal 8@:
20B12B Hune 6773.
De'art$ent of Constitutional De&elo'$ent and ,er$an
echnical Cor'oration (DCD1,O). 6777. )ntegrated
Develo!ment Planning Pilot Proects ,ssessment Study.
:retoria: South African De'art$ent of Constitutional
De&elo'$ent.
De&as, Dic", and Da&id For)oe. 2000. 4Cit% ,o&ernance and
:o&ert%: he Case of Fu$asi .5 2nvironment and 3rbani4ation
62 (6): 623Q63?.
D=S>A. 2000. :resentation )% +r. +istr%, :re$ Detwor", Washington, D.C.:
he World !an".
De'art$ent of :ro&incial and (ocal ,o&ern$ent (D:(,). 677C.
8hite Pa!er on $ocal Government. :retoria: South African
De'art$ent of :ro&incial and (ocal ,o&ern$ent.
D:(,. 2000. 26am!les of )nnovation in $ocal Government.
A&aila)le at: htt':PPwww.local.go&.#aPDCDPdcdinde0.ht$l.
Dutta, Sh%a$ S. 2000. 4:artnershi's in 2r)an De&elo'$ent: A <e&iew of
Ah$eda)adEs 90'erience .5 2nvironment and
3rbani4ation 62(6): 63Q2@.
9ta$adi, 3elisa 2. 2000. 4Ci&il Societ% :artici'ation in Cit%
,o&ernance in Ce)u Cit%.5 2nvironment and 3rbani4ation
62(6): ?B1B2.
3oltin, Craig. 6777. 4State and (ocal ,o&ern$ent :erfor$ance: =tEs i$e to
+easure 2'R 5 5he Government ,ccountants 7ournal. Arlington, S'ring
6777.
,on#ale#, 9leanor +. 2000. 4Decentrali#ation and :olitical
:artici'ation in the :hili''ines. 90'eriences and =ssues in Societal
ransfor$ation .5 =nstitute for :o'ular De$ocrac%. 3orthco$ing.
>alfani, +. 677B. 4he ,o&ernance of 2r)an De&elo'$ent in 9ast Africa.5 =n
+ar" Swilling (ed. ), Governing ,frica's +ities. Hohannes)urg: Witwatesrand
2ni&ersit% :ress 66?16@0.
>ardo%, Ana, and <icardo Schuster$an. 2000. 4Dew +odels for
the :ri&ati#ation of Water and Sanitation for the 2r)an :oor .5
2nvironment and 3rbani4ation 62(2): @3QB?.
>==D. 6773. 4=ncor'orating :ro&incial A'inion in the 9&aluation of (ocal
,o&ern$ent 3iscal Ca'acit% in =ndonesia.5 >ar&ard =nstitute for
=nternational De&elo'$ent. Discussion :a'er.
8B2:6Q62.
=n$an, <o)ert. 67B7.
=sla$, Da#rul, and +uha$$ed +oha))at Fhan. 677@. 42r)an ,o&ernance in
!angladesh and :a"istan.5 =n :atricia (.
+cCarne% (ed. ), +ities and Governance. Centre for 2r)an and
Co$$unit% Studies, 2ni&ersit% of oronto.
Hen"ins, <o), and Anne 1+arie ,oet#. 6777. 4Accounts and
Accounta)ilit%: heoretical =$'lications of the <ight1to1
=nfor$ation +o&e$ent in =ndia.5 5hird 8orld 9uarterly, 20(3):
@03Q22.
Fharoufi, +. 677B. 4,o&ernance of 2r)an Societ% in Dor th Africa.5 =n
+ar" Swilling (ed. ), Governing ,frica's +ities. Hohannes)urg: Witwatesrand
2ni&ersit% :ress 3BQC8.
Fihato, Caroline. 677C. 4Do De&elo'$ent Without +one%.5 8eekly
Mail and Guardian, 3e)ruar% @, 677C.
(a$, Dew$an +.F. 2000. 4,o&ern$ent =nter&ention in the 9ono$%: A
Co$'arati&e Anal%sis of Singa'ore and >ong Fong. 5 Public
,dministration and Develo!ment 20(?):37BQ826.
+itlin, Diana. 2000. 4owards +ore :ro1'oor (ocal ,o&ern$ents
in 2r)an Areas. 2nvironment and 3rbani4ation 62 (6): 3 166.
Diesner, >elen. 6777. 4(ocal ,o&ern$ent AuditingK=$'ro&ing the
:erfor$ance of ,o&ern$ent in the De0t Centur%.5 5he Government
,ccountants 7ournal 8C:3213C.
Aates, Wallace. 67B2. *iscal *ederalism. Dew Sor", >arcourt
!race Ho&ano&ich.
Aates, Wallace. 6773. 43iscal Decentrali#ation and 9cono$ic
De&elo'$ent.5 -ational 5a6 7ournal 8@:23B 183.
Agu, Vincent =. 2000. 4:ri&ate Sector :artici'ation and +unici'al Waste
+anage$ent in !enin Cit%, Digeria.5 2nvironment and 3rbani4ation
62(2):603166B.
Ani)o"un, :. 677B. 4,o&ernance and 2r)an :o&ert% in
Anglo'hone West Africa.5 =n +ar" Swilling (ed. ), Governing ,frica's
+ities. Hohannes)urg: Witwatesrand 2ni&ersit% :ress.
C?Q668.
:aul, Sa$uel. 6772. 4Accounta)ilit% in :u)lic Ser&ices: 90it, Voice and
Control.5 8orld Develo!ment 20:608BQ@0.
:aul, Sa$uel. 677@. 4Strengthening :u)lic Accounta)ilit%
through :artici'ation .5 =n Hennifer <iet)ergen1+cCrac"en (ed.),
Partici!ation in Practice, World !an" Discussion :a'er Do. 333.
Washington, D.C.: he World !an".
:orio, 9$$a. 677B. 42r)an ,o&ernance and :o&ert% Alle&iation
in Southeast Asia.5 =n 9$$a :orio (ed. ), 3rban Governance and
Poverty ,lleviation in Southeast ,sia. Tue#on Cit%: ,lo)al 2r)an <esearch
=nitiati&e 6Q80.
:utu, +'ho. 2006. 8ard +ommittees will 2ncourage +ommunity
Partici!ation. Fhan%isa, =ssue 0. Ca'e own: =DASA.
<e)ic", Hud%. 2006. 4:orto Alegre.5 OD9 Dail% Co$$entaries, +arch 6B,
2006.
Shah, Anwar. 677C "alance, ,ccountability, and Res!onsiveness#
$essons ,bout Decentrali4ation. :!erations 9&aluation
De'art$ent, Countr% and <egional 9&aluation Di&ision.
Washington, DC: World !an".
Shin, <o%, and Seon 1Seo) >a. 677C. 4=n Search of
Decentrali#ation and Deconcentration: (ocal Autono$% and 3iscal
<efor$ in Forea.5 7ournal of Public "udgeting,
,ccounting and *inancial Management 60:672Q26C.
a'ales, :roser'ina Do$ingo. 677@. 4he :hili''ines.5 =n :atricia (. +cCarne%
(ed. ), 5he +hanging -ature of $ocal Government in Develo!ing +ountries
. Centre for 2r)an and Co$$unit% Studies, 2ni&ersit% of oronto.
her"ildsen, Ale. 677C. $ocal Government and Households in
Primary 2ducation in 5an4ania# Some $essons for Reform. Center
for De&elo'$ent <esearch Wor"ing :a'er 7C.@.
Co'enhagen: Center for De&elo'$ent <esearch.
ie)out, Charles. 67?@. 4A :ure heor% of (ocal 90'enditures.5
5he 7ournal of Political 2conomy @8(?): 86@Q828.
2D9SCA. 2006. Partnershi!s for Poverty ,lleviation in +ebu +ity.
+AS Clearing >ouse for !est :ractices. A&aila)le at
htt':PP w ww.unesco.orgP$ostP)''o&er.ht $ . :hili''ines.
Vaa, +ari"en. 2000. 4>ousing :olic% After :olitical ransition:
he Case of !a$a"o.5 2nvironment and 3rbani4ation.
62(6):2B138.
)
RESULTS MATTER*
SUGGESTIONS FOR A
DEVELOPING COUNTR+,S
EARL+ OUTCOME
MEASUREMENT EFFORT
Harry P. Hatry
Source: Shah, Anwar, editor (2003), Measuring Government Performance in the
Delivery of Public Services, Vol. 2 of Handbook on Public Sector Performance
Reviews. Washington D.C.: he World !an".
he $anager of a soccer tea$ needs a running score of the ga$e. he
$anager needs this in order to hel' identif% whether a change in strateg% or
other action is needed, and su)se*uentl%, to find out whether those changes
resulted in the desired results (outco$es).
he $anager of a 'ri&ate )usiness needs regular feed)ac" on the
)usiness. 'rofita)ilit% so the $anager can deter$ine whether actions are
needed and, su)se*uentl%, whether that action led to the desired results.
Si$ilarl%, $anagers in an% go&ern$ent, and an% go&ern$ent agenc%,
whether at the national or local go&ern$ent le&el K need regular
feed)ac" on the *ualit% and outco$es of ser&ices to its citi#ens.
-%at I# Outco"e Mea#ure"ent.
Autco$e $easure$ent is the regular $easure$ent and re'orting of the
outco$es (results) of 'u)lic agenc% 'rogra$s. +easure$ent should )e at
least annual, )ut 'refera)l% $ore fre*uentl% such as *uarterl%. he
agencies $ight )e national go&ern$ent agencies or su)1national
agencies. Autco$e
$easure$ent has also )egun to )e used in so$e countries )%
;.<= Suggestions for a Develo!ing +ountry's :utcome Measurement 2ffort
'ri&ate, non1go&ern$ental organi#ations (D,As) to trac" the outco$es
of their own ser&ices.
Autco$e $easure$ent includes the $easure$ent of 'rogra$ results,
and the *ualit% of the wa% in which the ser&ice is deli&ered (such as how
long it ta"es custo$ers to get the ser&ice).
So$eti$es the ter$ outco$e $easure$ent is also used to include the
$easure$ent of the efficienc% with which the ser&ice is
'ro&ided. he ter$ 4efficienc%5 is defined as the ratio of the a$ount of
in'ut to the a$ount of 'roduct 'roduced. =n'uts can )e e0'ressed in
$onetar% units or a$ount of e$'lo%ee ti$e. he word 4'roduct5
traditionall% refers to the a$ount of !hysical out!ut of the agenc%.s
'rogra$s (such as nu$)er of $eters of roads re'aired). =n outco$e
$easure$ent, howe&er, it $eans relating the a$ount of in'ut to the a$ount
of outcome 'roduced. his 'ro&ides indicators such as
4e0'enditures 'er $eter of roads i$'ro&ed to a satisfactor%, or )etter,
condition.5
-%' Mea#ure Outco"e#.
Autco$e $easure$ent ser&es a nu$)er of )asic and &ital
go&ern$ental 'ur'oses:
=t hel's ser&ice $anagers a''l% whate&er resources the% ha&e to
'ro)le$ areas identified )% the outco$e infor$ation, in order to get
)est use fro$ li$ited resourcesU
=t 'ro&ides infor$ation to 'u)lic officials as to the e0tent to which the
'rogra$ is 4winning or losing,5 there)% hel'ing 'roduce i$'ro&ed
ser&ices to citi#ens K )% $oti&ating 'u)lic e$'lo%ees to
continuall% i$'ro&e the *ualit% and outco$es of the ser&ices the%
are deli&eringU
=t identifies the e0tent to which ser&ice *ualit% and outco$es
ha&e changed after ser&ice i$'ro&e$ent actions ha&e )een ta"enU
Harry P. Hatry = ;./
=t hel's in )udgeting, so that resources are allocated in wa%s $ost
li"el% to 'roduce the $a0i$u$ )enefit to citi#ensU
=t $a"es 'u)lic agencies $ore accounta)le for results to elected
officials and the 'u)licU and
=t can increase the 'u)lic.s trust in their go&ern$ent.
O(#tacle# to Outco"e Mea#ure"ent in
Develo&in Countrie#
De&elo'ing countries usuall% ha&e $an% o)stacles to i$'le$enting
outco$e $easure$ent. hese include the
following:
Ver% li$ited re&enues are a&aila)le for ser&ices and for outco$e $easure$ent.
>ighl% li$ited data 'rocessing technolog% is a&aila)le ()oth hardware
and software), re*uiring $anual
'rocedures, which can )e considera)l% $ore ti$e 1 consu$ing and
inaccurate.
Staffs t%'icall% (es'eciall% in o'erating agencies) ha&e little for$al
training and e0'ertise in the *uantitati&e
techni*ues needed for outco$e $easure$ent.
he highest le&els of the go&ern$ent $a% ha&e little understanding
and a''reciation for the i$'ortance of o)taining feed)ac" on ser&ice
*ualit% and outco$es.
,o&ern$ent 'ersonnel $a% ha&e *uite li$ited e0'erience in o)taining
in'ut fro$ their citi#ens (the custo$ers of their ser&ices). =n'ut is needed
to initiall% hel' identif% what ser&ice characteristics should )e $easured,
and su)se*uentl%, to o)tain custo$er feed)ac" on ser&ice *ualit% and
results of ser&ices recei&ed )% the custo$ers.
De&elo'ing countries t%'icall% ha&e $ulti'le donors who $a% ha&e
different interests and degrees of su''ort for outco$e $easure$ent.
;.;= Suggestions for a Develo!ing +ountry's :utcome Measurement 2ffort
Des'ite all these li$itations $ost de&elo'ing countries are li"el% to )e
a)le to, and should, i$'le$ent at least a rough &ersion of outco$e
$easure$ent. 9&en those countries whose go&ern$ents e0ercise tight
control o&er their 'o'ulations should underta"e outco$e $easure$ents, if
go&ern$ent officials are trul% interested in the welfare of their citi#ens and
not solel% concerned with retention of 'ower. (he latter
go&ern$ents $ight restrict the re'orting of the findings. >owe&er, as long
as the% see" to $a"e ser&ices as hel'ful to citi#ens as 'ossi)le, these
go&ern$ents should also i$'le$ent an outco$e $easure$ent 'rocess.)
<egardless of how 'oor a countr% is, as long as its go&ern$ent is
'ro&iding ser&ices to the 'u)lic, the 'u)lic agencies should atte$'t to do
that ;o) as well as 'ossi)le, that is, 'roduce as high ser&ice *ualit% and
outco$es as 'ossi)le. o do this, agencies need o);ecti&e, &alid, and
relia)le infor$ation as to how well it is doing in deli&ering *ualit%
ser&ices. Clearl%, outco$e
$easure$ent cannot re*uire so $an% resources that a ser&ice suffers rather
than i$'ro&es. he "e% issue is what can )e done at low cost to underta"e
such $easure$ent, and without co$'ro$ising the a)ilit% to o)tain at
least roughl% correct outco$e infor$ation. hat is the su);ect of the
re$ainder of this re'ort.
he "e% 'oint is that whate&er ser&ice a 'u)lic agenc% is 'ro&iding, it
al$ost alwa%s can i$'ro&e on the *ualit% and outco$es of the
ser&iceKwithout additional resources. Autco$e
infor$ation should ena)le agencies to $a"e
i$'ro&e$ents, and also, to )etter ;ustif% their re*uests for resources.
/a#ic Ste&# in I"&le"entin an
Outco"e Mea#ure"ent Proce##
o i$'le$ent outco$e $easure$ent, agenc% 'ersonnel should first
carefull% identif% the $ission and o);ecti&es of each
Harry P. Hatry = ;.>
of the agenc%.s ser&ices. 3ro$ these $ission state$ents and o);ecti&es, the
agenc% should identif% the outco$es and indicators to )e used to $easure
these outco$es. 90hi)its A, !, and C 'ro&ide e0a$'les fro$ 'rogra$s
ai$ed at education, welfare of children, and street cleanliness. 9ach
'ro&ides a state$ent of o);ecti&es, a list of outco$es that arise fro$ the
o);ecti&es, and s'ecific indicators for each outco$e. :otential sources of
data for each outco$e and t%'es of data collection 'rocedures are also
included on the e0hi)its. Data sources and collection 'rocedures are
discussed in the ne0t section.
/a#ic Outco"e Mea#ure"ent
Procedure# and T%eir Co#t#
90a$'les of 'rocedures that go&ern$ents in de&elo'ing countries can
use to o)tain )asic infor$ation on ser&ice *ualit% and outco$es are gi&en
)elow.
6. +ounts of maor incidents ?obtained =where !ossible= from agency
records @ . :ro)a)l% the first outco$e data that $ost go&ern$ents and their
agencies trac" on a regular )asis are counts of "e% incidents, ones that are
of $a;or i$'ortance to the go&ern$ent. hese include counts such as:
the incidence of &arious diseases, )% categor%U
the incidence of infant $ortalit%, )% causeU
nu$)ers of cri$es re'orted to law enforce$ent agencies, )%
categor%U
nu$)er of &ehicle traffic accidents, in;uries, and deaths, )%
causeU
educational achie&e$ent le&els, such as the nu$)er of children
co$'leting s'ecific nu$)ers of %ears of schoolingU and
'ollution content of sources of drin"ing water su''l%.
hese are e0a$'les of data that $ost go&ern$ents see" to
collect. De&elo'ing countries, howe&er, $a% not ha&e
s%ste$atic 'rocedures in 'lace for collecting and 'rocessing
;.A= Suggestions for a Develo!ing +ountry's :utcome Measurement 2ffort
such infor$ation, at least not fro$ all locations throughout the
go&ern$ent.s ;urisdiction. 9sta)lishing these data collection 'rocedures
can )e a 'ro)le$ for so$e de&elo'ing countries )ut should )e atte$'ted.
9&en if data sources are not full% relia)le, esta)lishing e&en rudi$entar%
'rocedures for data collection is desira)le and a''ro'riate. =ndicators such
as the a)o&e are )asic 4*ualit% of life5 indicators for the citi#ens of an%
countr% or cit%. 9&en rudi$entar%, $anual s%ste$s are 'refera)le to not
atte$'ting an% s%ste$atic data collection at all.
=nitiall%, the effort $ight )e concentrated in $a;or 'o'ulation areas
)ecause of difficulties in o)taining data fro$ outl%ing rural areas. A s%ste$
that 'rocesses data solel% fro$ a countr%.s largest cities is 'refera)le to
co$'letel% neglecting the infor$ation. Such data dri&e $an%
decisions a)out the allocation of scarce resources.
2. +ustomer feedback surveys . Sur&e%ing custo$ers can )e a &er%
useful tool in go&erning. =t is a $a;or wa% to o)tain credi)le,
reasona)l% accurate feed)ac" fro$ custo$ers of go&ern$ent ser&ices.
Sur&e%s need to )e underta"en in a
reasona)l% sound, 'rofessional $anner. Custo$er sur&e%s can 'ro&ide
&arious t%'es of infor$ation for outco$e $easure$ent, including:
<atings fro$ citi#ens of their o&erall satisfaction with indi&idual 'u)lic
ser&ices (thus, 'ro&iding data for such outco$e indicators as the
4'ercent of sur&e%ed custo$ers that rated a 'articular ser&ice as either
4e0cellent5 or 4good5 rather than 4fair5 or 4'oor5)U
<atings fro$ custo$ers of the s'ecific characteristics of those ser&ices
such as their ti$eliness and hel'fulness (thus, 'ro&iding data for
outco$e indicators such as the 4'ercent of sur&e%ed custo$ers
that rated a 'articular ser&ice characteristic,
such as ti$eliness, as either 4e0cellent5 or 4good5 rather than 4fair5
or 4'oor5)U
Harry P. Hatry = ;.B
3actual infor$ation a)out citi#ens. condition, attitudes, and )eha&ior,
such as citi#ens. health, earnings, use of 'u)lic ser&ices (such as
'u)lic transit), e0tent of cri$e &icti$i#ation, and
e0tent of e0tra 'a%$ents ()ri)es) needed to o)tain ser&ices. his
infor$ation 'ro&ides data for outco$e indicators such as 4'ercent of
citi#ens that ha&e )een a cri$e &icti$ at least once during the 'ast si0
$onths5)U
<easons wh% citi#ens had 'ro)le$s with s'ecific ser&ices (as"ed of
res'ondents who ga&e negati&e res'onses on *uestions a)out their
e0'eriences with 'articular ser&ices). hese res'onses can 'ro&ide
useful infor$ation as to 'ro)le$s that need attentionU
Suggestions for i$'ro&ing ser&ices. his infor$ation is a )onusU the
suggestions $a% 'ro&ide s'ecific guidance to 'u)lic $anagers for
i$'ro&ing ser&icesU
De$ogra'hic infor$ation on the 'o'ulation sur&e%ed. Such
infor$ation hel's identif% which 'o'ulation grou's ha&e had 'articular
'ro)le$s with ser&ices so that attention can )e directed towards the$.
90hi)it D 'ro&ides an e0a$'le of a custo$er sur&e% *uestionnaire used
in 2ganda. =t includes *uestions a)out the *ualit% of ser&ices recei&ed )%
citi#ens and a)out corru'tion e0'erienced )% the citi#ens. (he
*uestionnaire was s'onsored )% the World !an" and 2ganda. he results
are re'orted in 42ganda Dational =ntegrit% Sur&e% 67CC: 3inal <e'ort
August
677C, Ciet =nternational.5)
Citi#en sur&e%s, if 'ro'erl% done, 'ro&ide reasona)l% re'resentati&e
feed)ac" fro$ citi#ens. his is unli"e other
sources of citi#en feed)ac", such as o'en $eetings of citi#ens, grou'
discussions, and ta)ulations of co$'laints recei&ed fro$ citi#ens )% an
agenc% ()ecause onl% so$e 'eo'le attend such $eetings, and onl% so$e
'eo'le are willing or "now how to co$'lain).
;.C= Suggestions for a Develo!ing +ountry's :utcome Measurement 2ffort
3or so$e ser&ices an agenc% would sur&e% sa$'les of all households
(for ser&ices that $ost households can )e e0'ected to ha&e first1hand
e0'erience). 3or ser&ices that ser&e onl% s$all 'ro'ortions of the
'o'ulation, onl% those who ha&e )een custo$ers of the ser&ice need to )e
sur&e%ed (such as fa$ilies that ha&e recei&ed $aternal health care, or far$
fa$ilies that ha&e recei&ed technical assistance on far$ing 'ractices).
Sur&e%s can )e done of indi&iduals, households, or )usinesses. !usinesses
are li"el% to )e custo$ers for $an% 'u)lic ser&ices.
A 'rinci'al use of sur&e%s of custo$ers is to o)tain regular feed)ac"
fro$ custo$ers in order to trac" trends and to assess the e0tent of
i$'ro&e$ent after the go&ern$ent has changed its ser&ice deli&er%
a''roach.
Can de&elo'ing countries afford to underta"e such sur&e%s, es'eciall%
on a regular (e.g., annual) )asis/ Sur&e%s conducted on a regular )asis $a%
not )e feasi)le for $an% de&elo'ing countries without e0ternal su''ort.
(arge scale inde'endent sur&e%s using 'rofessional sur&e% fir$s usuall%
are not chea'.
Donor organi#ations such as the World !an" ha&e in the 'ast su''orted
such sur&e%s. 3or e0a$'le, the World !an" has
s'onsored 4Ser&ice Deli&er% Sur&e%s5 in a nu$)er of countries. hese
sur&e%s ha&e 'ro&ided feed)ac" fro$ re'resentati&e sa$'les of citi#ens on
'erce'tions of the *ualit% of indi&idual ser&ices and on the e0tent of
corru'tion these citi#ens ha&e encountered in atte$'ting to o)tain 'u)lic
ser&ices.
Another 'ossi)le source of hel' for custo$er sur&e%s are non1
go&ern$ental organi#ations (D,As) that ha&e an interest in the
$easure$ent of one or $ore ser&ices. D,As $ight 'ro&ide inter&iewers,
data entr%, andPor assistance in ta)ulation, data anal%sis, and re'ort
're'aration.
A less e0'ensi&e o'tion is for the go&ern$ent to underta"e the sur&e%s
itself. =n countries with a&aila)le and ine0'ensi&e $an'ower, a
go&ern$ent $a% find it feasi)le to use te$'orar% or 'er$anent 'ersonnel to
do the sur&e% inter&iews. he 'u)lic
Harry P. Hatry = ;.1
agenc%, howe&er, will still li"el% need so$e outside assistance for
de&elo'ing the *uestionnaire and for de&elo'$ent of a reasona)l%
re'resentati&e sa$'ling 'lan so that the% are un)iased.
After the *uestionnaires ha&e )een co$'leted )%
res'ondents, the 'u)lic agenc% will need to 'rocess that infor$ation
and ta)ulate it accuratel%. =n de&elo'ed countries, and so$e de&elo'ing
countries, such 'rocessing and ta)ulation can )e done with co$'uters,
using either $anual or auto$atic entr% of data fro$ the *uestionnaires. =f
such e*ui'$ent is not a&aila)le to the agenc%, ta)ulations will need to
)e $ade $anuall%. his is considera)l% $ore ti$e 1consu$ing and is
li"el% to lead to $ore $ista"es.
o contain sur&e% costs, 'u)lic agencies usuall% need to sur&e% onl%
s$all sa$'les of the custo$ers that the% ser&e. =t is not li"el% to )e feasi)le
for a large agenc% to tr% to sur&e% the full 'o'ulation of the nation or a cit%,
other than in &er% s'ecial, infre*uent, circu$stances, such as 601%ear
censuses.
=n de&elo'ed countries, agencies so$eti$es 'lace tight 'recision
re*uire$ents on sur&e% results. hese 'recision
re*uire$ents $ean large sa$'les are needed. Such 'recision is unli"el% to
)e needed )% de&elo'ing countries (and $a% e&en )e e0cessi&e in
de&elo'ed countries). 3or e0a$'le, re*uiring that sur&e%s 'ro&ide a 7?
'ercent confidence that the sur&e% data are within two or three 'ercentage
'oints of the true &alue for the 'o'ulation K will li"el% re*uire
considera)le e0tra cost that is e0cessi&e and unnecessar%. ,o&ern$ents
and their agencies can )uild u' their sur&e% ca'a)ilities o&er ti$e. he
earlier %ears $a% in&ol&e s$aller sa$'les and $ore a''ro0i$ate 'rocedures
than 'refera)le. )t is better to be roughly right than !recisely ignorant.
3. *ocus gro u!s . =f s$all re'resentati&e sa$'le sur&e%s are infeasi)le,
the go&ern$ent can fall )ac" on a 'rocedure that does not 'ro&ide
statistical credi)ilit% )ut can gi&e so$e useful
;..D= Suggestions for a Develo!ing +ountry's :utcome Measurement 2ffort
citi#en feed)ac". his $ethod is the use of 4focus grou's5 with custo$ers
of 'articular ser&ices. =n this 'rocedure, the agenc% in&ites a s$all nu$)er
of custo$ers, 'erha's ten to fifteen, for a two1hour session. At the session,
'artici'ants are as"ed a)out their e0'erience with the ser&ice, how the%
would rate the ser&ice on &arious characteristics, and wh% the% ga&e those
ratings. he% can also )e as"ed for suggestions for i$'ro&ing the ser&ice.
,o&ern$ent officials $a% )elie&e that these citi#ens will onl% as" for $ore
(costl%) ser&ice. >owe&er, 'artici'ants are also &er% li"el% to discuss
o'tions that do not re*uire additional resources.
he agenc% $ight hold a nu$)er of such $eetings in &arious locations
to o)tain a &ariet% of &iew'oints. he sessions should include, to the
e0tent 'ossi)le, 'ersons with &arious
de$ogra'hic characteristics, such as so$e sessions with 'artici'ants who
are &er% 'oor and other sessions with richer citi#ens. !oth rural and ur)an
areas should )e re'resented. =n $ost de&elo'ing countries, $an% ethnic
grou's recei&e the ser&ices. he $eetings should include each such $a;or
ethnic grou', 'ro)a)l% in different sessions.
he findings fro$ focus grou' $eetings are 'ri$aril% *ualitati&e, not
*uantitati&e. De&ertheless, the infor$ation
o)tained should 'ro&ide feed)ac" to the agenc% on how well it i s
deli&ering the ser&ice and where i$'ro&e$ents see$ necessar%.
2sers of focus grou's need to understand that these sessions do not
'ro&ide statisticall% relia)le data, such as that o)taina)le fro$ s%ste$atic
sur&e%s of re'resentati&e sa$'les of custo$ers.
8. 5rained observer assessment of key !hysical condition s 0 :u)lic
agencies should $onitor and trac" the condition of "e% facilities under
their res'onsi)ilit%. his a''lies to conditions such as the following:
road 4ridea)ilit%5,
condition of water an d sanitation facilities and e*ui'$ent, condition
of hos'itals and long1ter$ care facilities
Harry P. Hatry = ;...
condition of school )uildings and school roo$s,
cleanliness and sanitar% condition of streets and
neigh)orhoods,
'resence of rats and insects, and
e0terior condition of ho$es.
o $a"e these $easure$ents, the agenc% needs to de&elo' a well1
defined rating scale, one that identifies in s'ecific ter$s each rating
categor% for each condition that the agenc% wants to $onitor. A nu$)er of
agencies ha&e used rating sc ales using 'hotogra'hs to re'resent different
le&els of road conditions, street cleanliness, school roo$s, and housing. o
de&elo' a 'hotogra'hic rating scale, the agenc% would:
ta"e $an% 'hotogra'hs of a &ariet% of conditions of the facilities that it
wants to assessU
select 'hotos that re'resented each of 'erha's three, four, or fi&e rating
categories (using a scale that rates conditions, such as fro$ 4e0cellent5
to 4'oor5)U and
test the 'rocedures with 'ersonnel who are li"el% to do the ratings.
rained o)ser&er 'rocedures can usuall% use low1cost 'ersonnel to
$a"e the ratings. hese 'ersonnel $ight )e
'er$anent staff, te$'orar% e$'lo%ees, contract 'ersonnel, or &olunteers.
he agenc% needs to 'ro&ide ade*uate training in the use of the rating
scale to each 'erson that is to $a"e the o)ser&ations.
he data fro$ the ratings of all, or a sa$'le of, facilities are then
aggregated to 'ro&ide o&erall totals. his 'rocedure also 'ro&ides
reasona)l% relia)le outco$e $easure$ents on the condition of individual
facilities. hus, the 'rocedure can )e used to identif% where to )est
allocate li$ited resources (such as to facilities where the need see$s to )e
greatest and the greatest 'a%off in the use of scarce resources can )e
a''lied).
he feasi)ilit% of this 'rocedure in a de&elo'ing countr% de'ends on
the e0tent to which low1cost 'ersonnel are a&aila)le
;..<= Suggestions for a Develo!ing +ountry's :utcome Measurement 2ffort
to do the ratings. his is not a so'histicated 'rocedure. he ta)ulations of
the infor$ation can )e done $anuall%, )ut with less accurac% than if the
'u)lic a genc% is a)le to use auto$ated 'rogra$s. ,!!endi6 , 'ro&ides
$ore detail on the 'rocedures for the trained o)ser&er 'rocess.
Mea#ure"ent# o! E1tent o! Corru&tion
Corru'tion in the deli&er% of 'u)lic ser&ices is a $a;or concern toda%
for $an% countries and their donors. Where&er $one% is in&ol&ed, so$e
corru'tion will li"el% occur. ,o&ern$ents need to ta"e ste's to reduce it to
the e0tent 'ossi)le. o trac" the success of these efforts, go&ern$ents
(through their anti1corru'tion agencies) need to trac" t he e0tent to which it
is occurring. hus, indicators of the e0tent of
corru'tion at 'articular 'oints in ti$e should )e 'art of 'erfor$ance
$easure$ent s%ste$s in de&elo'ing countries.
90hi)it 9 is a sa$'le set of indicators that a''ears a''ro'riate for an anti
1corru'tion agenc% to trac" corru'tion on a regular, annual )asis. Such
$easure$ent is $ost li"el% to )e feasi)le for countries whose leaders are
acti&el% 'ressing for corru'tion reduction and which ha&e an
inde'endent auditPins'ector general office. he data for $ost of these
indicators would co$e fro$ anti1corru'tion agenc% records. 9ach
go&ern$ent will need to esta)lish its 'rocedures for recording the needed
data. 3ew of these indicators are li"el% to )e costl% to collect.
>owe&er, )ecause of their sensiti&it%, so$e of these data are li"el% to )e
contro&ersial.
A)taining data on the last grou' of indicators (nu$)ers
67122) re*uires sur&e%s of household and )usinesses. As
discussed earlier, citi#en sur&e%s re*uire s'ecial resources. =n that earlier
section, we noted wa%s to "ee' costs to a low le&el. Sur&e%s co&ering
corru'tion issues 'ro)a)l% ha&e a greater need to )e underta"en )% a
'rofessional organi#ation outside the go&ern$entKto 'ro&ide credi)ilit%.
he go&ern$ent can fund
Harry P. Hatry = ;../
the sur&e% as long as it does not i$'ose undue constraints on the sur&e%.
3unds $ight also co$e fro$ donor agencies. As noted earlier, the World
!an" has alread% )een s'onsoring a nu$)er of these in $an% countries.
Also, as noted earlier, an D,A $ight also )e willing to s'onsor the sur&e%
effort.
hese sur&e%s should )e of a re'resentati&e sa$'le of households. A
se'arate sur&e% can )e done of a re'resentati&e sa$'le of )usinesses.
!usinesses the$sel&es so$eti$es cause the corru'tion and, therefore, so$e
res'onses fro$ )usinesses would )e e0'ected to )e dishonest.
De&ertheless, go&ern$ents can 'ro)a)l% learn $uch fro$ the o&erall
res'onses. he
assu$'tion here is that $ost )usinesses would 'refer not to )e in&ol&ed
with corru'tion acti&ities. An i$'ortant ele$ent of sur&e%s that as"
res'ondents a)out their e0'eriences with corru'tion, is to assure
confidentialit% of res'onses. his $a% )e difficult in so$e countries if the
sur&e% s'onsors are not a)le to use a full% credi)le and 'rofessional sur&e%
organi#ation.
he *uestionnaire shown in 90hi)it D includes *uestions that as"
res'ondents a)out their 'ersonal e0'eriences with corru'tion, thus,
'ro&iding data for =ndicators 67122 of 90hi)it
9.
hese sur&e%s do not ca'ture all as'ects of corru'tion, onl% that of
which sa$'led citi#ens and )usinesses ha&e first1hand "nowledge and are
willing to include in their res'onses. !usinesses are not li"el% to identif%
hidden corru'tion atte$'ts, such as incidents of )ri)er% )etween
go&ern$ent 'ersonnel and 'ri&ate sector indi&iduals or organi#ations. he
corru'tion indicators V6 through V6C in 90hi)it 9 co&er onl% corru'tion
incidents that co$e to the attention of the go&ern$ent. he household and
)usiness sur&e%s reach out to citi#ens and )usinesses to o)tain infor$ati
on that is not re'orted.
;..;= Suggestions for a Develo!ing +ountry's :utcome Measurement 2ffort
Mea#ure"ent# o! Re#&on#e Ti"e
+an% go&ern$ents that ha&e esta)lished a 'erfor$ance
$easure$ent 'rocess started with indicators of res'onse ti$e. <es'onse
ti$e is the ti$e fro$ when a re*uest for ser&ice was $ade until the ti$e
the ser&ice organi#ation 'ro&ided a res'onse. <es'onse1ti$e data is
'articularl% i$'ortant for e$ergenc% ser&ices, such as citi#en re*uests for
'olice and fire agenc% assistance relating to cri$es, fires, and
e$ergenc% health care needs. Also, citi#ens usual l% ha&e a significant
concern a)out res'onse ti$es for $an% other ser&ices, such as the ti$e
re*uired to get registered, to recei&e assistance 'a%$ents, to o)tain
'er$its, and to recei&e a res'onse on an% a''lication to the go&ern$ent.
<es'onse ti$es on indi&idual ser&ice re*uests are
su)se*uentl% aggregated to 'ro&ide indicators such as 4a&erage res'onse
ti$e5 or 4'ercent of re*uests for which the res'onse ti$e e0ceeded -W.
hours.5 he *uantit% 4W5 would )e selected )% the agenc% as its
4standard.5
While res'onse ti$es are i$'ortant to custo$ers (and therefore, these
ti$es should )e trac"ed so that i$'ro&e$ents can )e $ade), such data do
not tell anything about the outcome of the service the customers received K
onl% how long it ta"es to get the ser&ice.
rac"ing res'onse ti$es $a% a''ear si$'le at first glance. >owe&er, it
is li"el% to )e $ore ti$e 1consu$ing than it a''ears. An o'erating agenc%
needs to record the ti$e that each re*uest for the ser&ice was initiall%
recei&ed, and then, again, record the ti$e when the ser&ice was 'ro&ided.
=n the western world, auto$atic recording de&ices are used for fire and
'olice res'onses to calls for ser&ices. =n de&elo'ing countries, such
auto$ated de&ices $a% )e a long ti$e off. =n the $eanti$e, howe&er, an
agenc% can esta)lish $anual 'rocedures. Staff would record these ti$es,
and su)se*uentl%, collect and ta)ulate those ti$es $anuall%. his
'rocedure is co$'licated
Harry P. Hatry = ;..>
)% the need to $atch u' the starting and ser&ice deli&er% ti$es for
indi&idual re*uests. =f a su)stantial ti$e (such as se&eral da%s, or e&en
se&eral hours) usuall% occurs )etween re*uests and ser&ice 'ro&ision, care
needs to )e ta"en to $atch the ti$e of the res'onse to the ti$e of the
'articular re*uest.
<es'onse ti$e $easure$ent is 'articularl% critical for so$e ser&ices
(such as an% e$ergenc% ser&ice). 3or $ost other
ser&ices, res'onse ti$e $easure$ent is li"el% to )e secondar% co$'ared to
the other $easure$ents descri)ed a)o&e.
;..A= Suggestions for a Develo!ing +ountry's :utcome Measurement 2ffort
Data 2ualit' Control
An i$'ortant issue for an% outco$e $easure$ent 'rocess is
$aintaining the integrit% of the data so that it is reasona)l% accurate and
credi)le to outsiders, such as to elected officials, citi#ens, and the $edia.
With $anual data collection or ta)ulation 'rocedures, the o''ortunit%
for inaccuracies and data $ani'ulation are 'articularl% high.
+ethods to reduce the 'otential for
inaccuracies and $ani'ulation are needed. Ane wa% is to 'ro&ide an
at$os'here in which data findings are not threatening to e$'lo%ees (if
the data do not loo" good). A
second need is for ade*uate training of those res'onsi)le for 'ro&iding and
ta)ulating the data in order to a&oid $ista"es and $isinter'retations.
3inall%, and 'ro)a)l% $ost credi)le to those outside the agenc%, the
go&ern$ent.s audit office can )e $ade res'onsi)le for re&iewing at least
sa$'les of the data and 'eriodicall% re&iewing data collection and
ta)ulation 'rocedures.
hese ste's will hel' encourage 'u)lic agencies to increase their care
in data collection and ta)ulation, and reduce the te$'tations for
$ani'ulation. he% will also hel' identif%
'oorl% i$'le$ented data collection, ta)ulation, and re'orting 'rocedures
so that i$'ro&e$ents can )e $ade.
U#in Outco"e Data
An earl% section of this re'ort identified wh% outco$e data should )e
collected and the data.s $a;or 'ur'oses. hese
include hel'ing to deter$ine )udget allocations and the use )% ser&ice
$anagers for i$'ro&ing their 'rogra$s.
Autco$e1)ased )udgeting (or 'erfor$ance1)ased )udgeting as it is
$ore 'o'ularl% called in de&elo'ed countries) is still &er% $uch in its
de&elo'$ental stage. 3or $ost outco$es, the
relationshi's )etween the a$ount of in'ut (funds andPor staff
Harry P. Hatry = ;..B
ti$e) and outco$es is not "nown. 3or e0a$'le, how $uch additional
funds would )e re*uired to increase custo$er satisfaction le&els for a
'articular ser&ice )% fi&e 'ercentage 'oints/ De&ertheless, if outco$e
data are a&aila)le to an agenc%, the data should )e useful in identif%ing
where i$'ro&e$ents are li"el% to )e $ost needed. 3or e0a$'le, if a
'ro)le$ shows u' for a 'articular ethnic grou' and not for others, this
would suggest that $ore attention should )e 'laced on the ser&ice 'ro&ided
to that ethnic grou'.
<egular outco$e data are li"el% to )e of considera)le use to 'u)lic
$anagers for deciding on actions throughout each %ear. his will )e so,
'articularl% if the data are collected 'eriodicall% throughout the %ear, such
as e&er% $onth or e&er% three $onths. 3or e0a$'le, Dew Sor" Cit% has
trac"ed street cleanliness throughout the cit% for $an% %ears. =ts
sanitation de'art$ent uses that infor$ation to allocate its clean 1u' crews.
his has led to su)stantial reductions in the nu$)er of areas with &er% dirt%
streets. Another e0a$'le, if dro'out rates ha&e )een
increasing in schools in so$e locations or with so$e custo$er grou's, the
$anagers $ight loo" to deter$ine the causes for this and tr% to alle&iate
the$ to the e0tent 'ossi)le.
A $a;or use is for $anagers to hold 4>ow Are We Doing/5 sessions
with their staffs shortl% after each 'erfor$ance re'ort has )een 're'ared and
disse$inated. Such $eetings can )e an e0cellent wa% to o)tain in'ut fro$
staff on suggestions as to what is causing 'ro)le$s identified in the latest
re'ort and to o)tain suggestions for correcti&e actions. =n later 4>ow Are
We Doing/5 sessions, staff would loo" to see the e0tent to which actions
ta"en ha&e led to the outco$e i$'ro&e$ents sought.
As indicated earlier, a $a;or use of outco$e data is to e0a$ine trends
o&er ti$e, whether )etween $onths or *uarters as well as across %ears.
Such infor$ation hel's $anagers and their agencies to deter$ine whether
the% are winning or losing the ga$e.
;..C= Suggestions for a Develo!ing +ountry's :utcome Measurement 2ffort
3inall%, 'erfor$ance re'ort for$ats can ha&e a su)stantial i$'act on
use, and 4usea)ilit%5 of 'erfor$ance infor$ation. <e'orts need
to )e clear, concise, attracti&e, and understanda)le.
An e0cellent wa% to dis'la% outco$e data is to ma! the results)%
geogra'hic location (e.g., region, district,
neigh)orhood) to indicate those locations in $ost need of i$'ro&e$ent.
+a's are eas% to understand and are &isuall% attracti&e. 3or e0a$'le, on a
$a' the agenc% $ight shade the areas of a cit% to dis'la% different le&els
of cleanliness of each 'ortion of a cit% Kthe dar"er the shading, the dirtier
t he streets. he cit% used the data to allocate its street crews. !% e0a$ining
the shading fro$ 're&ious re'orting 'eriods, the reader can readil% assess
the c hanges that ha&e occurred o&er ti$e. Data fro$ later %ears ha&e
shown a stead% decline in the n u$)er of dirt% areas. Data fro$ an% source
(trained o)ser&er ratings, custo$er sur&e%s, or agenc% records) can )e
dis'la%ed with $a'sKas long as the data are geogra'hicall% 4coded.5
Final Note
A $a;or issue that has caused 'ro)le$s in de&elo'ed countries, and
li"el% will in de&elo'ing countries, is the $isunderstanding that
outco$e data fro$ an outco$e $easure$ent s%ste$s auto$aticall%
indicate that the agenc%, and its $anagers, are 'ri$aril% res'onsi)le for the
o)ser&ed outco$es. =f the outco$es are not as good as e0'ected, higher
le&el officials (and the $edia) ;u$' to the conclusion that it is the fault of
the agenc%. his 'uts agenc% 'ersonnel &er% $uch on the defensi&e,
resulting in fear and te$'tation to $ani'ulate the data and it leads to
higher1le&el officials $is1using the data.
:u)lic officials at all le&els, and the $edia, need to )e infor$ed that, as
with the score of a soccer ga$e, and with the )otto$1line 'rofits of a 'ri&ate
)usiness, that outcome data only indicate results. 5hey do not indicate why
the results occurred.
Harry P. Hatry = ;..1
+an% factors outside the control of agenc% $anagers usuall% affect
outco$es. 3or e0a$'le, world econo$ic conditions, unusual
weather conditions, and actions )% custo$ers can affect $an% outco$esK
and )e )e%ond the control of the agenc%.
Autco$e $easure$ent infor$ation raises i$'ortant
*uestions that agenc% officials need to address. Agenc% officials then need
to identif% reasons for less1than1desira)le ser&ices and underta"e needed
i$'ro&e$ent action.
=n de&elo'i ng countries, scarce funds will al$ost ine&ita)l% )e a $a;or
reason for a significant 'art of outco$e 'ro)le$s. De&ertheless, 'u)lic
agencies in de&elo'ing countries are still res'onsi)le for s*uee#ing the
$ost out of their resources. he% need to use their scarce resources as
wisel% as 'ossi)le to 'roduce ser&ices whose *ualit% and outco$es are as
good as 'ossi)le. 9ffecti&e and efficient use of a&aila)le resources,
howe&er s$all an agenc%.s resources $a% )e, should )e the o);ecti&e
of 'u)lic agencies and their 'ersonnel. <egular, relia)le $easure$ent
of ser&ice *ualit% and outco$es should )e a $a;or tool of 'u)lic agenc%
$anagers in hel'ing 'roduce the $ost effecti&e and efficient use of scarce
resources.
Harry P. Hatry = ;.<.
E1%i(it A* Outco"e Indicator# !or Ele"entar' and
Secondar' Sc%ool S'#te"#
:bectiv e : :ro&ide children with education that leads to learning
that in turn 'roduces %oung adults who are a)le to wor" and li&e in a
$odern societ% .
Data Source#3Procedure#
6. AttendancePa)senteeis$ rates K
nu$)er and rate of student1da%s.
School records
2. AttritionPdro'out rates. School records
3. Du$)er of actual student1da%s. School records
8. :ercentage of students 'ro$oted to the ne0t
grade.
?. Du$)er of students graduated or
'ro$oted.
School records
School records
@. <esults of acade$ic test scores. ests
B. :ercentage of students achie&ing a
s'ecified a$ount of test score gain during
the school %ear.
C. Du$)er of disru'ti&e incidents of
&iolence on school grounds.
7. :ercentage of 'arents rating their children as
ha&ing i$'ro&ed in, or ha&ing good or e0cellent
wor" and stud% ha)its.
60. :ercentage of students who 4W5 %ears after
co$'letion of schooling (e.g., two %ears)
either are continuing their
education or are e$'lo%ed.
ests
School records
Sur&e% of
'arents
Sur&e% of
students
Source# his list is a)stracted fro$ Service 2fforts and ,ccom!lishments Re!orting# )ts
5ime Has +ome for 2lementary and Secondary 2ducation. 67C7. Dorwal",
Connecticut : ,o&ern$ental Accounting Standards !oard.
End Outcomes Data Sources
4serious5 in;uri es and ()) deaths.
2. Du$)er and rate of re'orted (a) a)use Agenc%
and ()) neglect )% first1ti$e 'arents. <ecordsPrained
A)ser&er <atings
a)use and ()) re 1neglect.
distur)ance.
$ental 'ro)le$s in their current 'lace
of residence).
safet% concerns (including cases in which the
decision was to lea&e in own
ho$e).
4serious5 health 'ro)le$s
Intermediate Outcomes
ha&e 4ade*uate5 i$$uni#ations.
)e%ond a s'ecified 4a''ro'riate5 length
E1%i(it /* Outco"e Indicator# !or +out% -el!are
:bective E. : Assure the 'h%sical and e$otional well1)eing
(safet%) of children
6. Du$)er and 'ercent of child (a)
Agenc% <ecords
3. Du$)er and rate of identified (a) re 1
Agenc% <ecords
8. Du$)er and 'ercent of children
indicating 4se&ere5 e$otional
rained A)ser&er
<atings
?. :ercent of children re'orting
4su)stantial5 fear )ecause of in1ho$e factors
(e.g., )ecause of 'h%sical or
Sur&e% of
children
@. Du$)er and 'ercent of children in the s%ste$
that su)se*uentl% had to )e
re$o&ed fro$ a residence )ecause of
Agenc%
<ecordsPrained
A)ser&er <atings
B. Du$)er and 'ercent of safet%1related
C. :ercent of children in the s%ste$ that
Agenc% <ecords
7. Du$)er of cases in which needed
4needed5 inter&ention was dela%ed
Agenc% <ecords
of ti$e K )ro"en out )% reason
categor%, such as 4no s'ace was
a&aila)le.5
E1%i(it / (continued4
C%ild Develo&"ent
:bective E< : 9ncouragePsu''ortPensure 'ro'er de&elo'$ent of children,
including health, education, and social s"ills.
End Outcomes
6. Du$)erP'ercent of children who de&elo'ed illnesses,
or other health conditions that deterred
de&elo'$ent or reduced a)ilit% to function
nor$all%. (3or school1age children, an indicator
such as 4nu$)erP'ercent who had o&er -W. da%s
a)sences for sic"ness5 $ight also )e used.)
Data Sources/
Procedures
Agenc%
<ecords
2. Du$)erP'ercent of children who $et nor$al
growth cur&es and heightPweight e0'ectations.
Agenc%
<ecords
3. Du$)erP'ercent of children who dis'la%ed
4age 1a''ro'riate5 social s"ills.
Agenc%
<ecords
8. Du$)erP'ercent of 're 1school children who
4achie&ed school readiness,5 such as
indicated )% tests, etc.
estsPrained
A)ser&er
<atings
?. Du$)erP'ercent of children who 'rogressed in
school according to 4nor$al5 de&elo'$ent, such as
)% )eing 'ro$oted
andPor 'assing all courses.
Agenc%
<ecords
@. Du$)erP'ercent of children with at least one
4se&ere5 school delin*uenc% 'ro)le$
in the 'ast school %ear.
Agenc%
<ecords
B. Du$)erP'ercent of chi ldren who dro''ed out of
school )efore co$'leting 4W5 %ears.
Agenc%
<ecords
End Outcomes
Data Sources/
Procedures
'erha's one or two %ears after the% co$'leted
school, to deter$ine if the% ha&e a regular ;o)
or are in 'ost1school
education.)
indicated )% &arious self1estee$ scales
de&elo'ed for children of &arious ages).
Intermediate Outcomes
)een 4full%5 i$$uni#ed. <ecords
had 4regular5 $edicalPdental e0a$s. <ecords
C. Du$)erP'ercent of children who o)tained estsPrained
4)asic life s"ills5 and 4self1sufficienc%,5 A)ser&er
such as indicated )% 'erfor$ance in the <atings
children.s last %ears of school on selected s"ills
tests, e.g., reading, writing, $ath,
&ocational s"ills, etc.
E1%i(it / (continued)
7. Du$)erP'ercent who )eca$e 4self1 sustaining,5
4inde'endent5 %oung adults after the% )eca$e
adults. (his indicator
re*uires follow1u' sur&e%s of %outh,
Sur&e% of
%oung adults
60. Du$)erP'ercent who indicate a
reasona)l e le&el of self1estee$ (such as
ests
66. Du$)erP'ercent of children who ha&e
Agenc%
62. Du$)erP'ercent of children who ha&e
Agenc%
E1%i(it C* Outco"e Indicator# !or Solid -a#te Collection
O(5ective# and Princi&al E!!ectivene## Mea#ure# !or Solid -a#te Collection
:verall :bective # o 'ro$ote the aesthetics of the co$$unit% and the health and safet% of the citi#ens )% 'ro&iding an en&iron$ent
free fro$ the ha#ards and un'leasantness of uncollected refuse with the least 'ossi)le citi#ens inconcei&a)le.
Quality
O!ecti"es
#haracteristics S$eci%ic Measure
&
Data #ollection Procedure
:leasing
aesthetics
Street, alle% and
neigh)orhood
cleanliness
6. :ercentage of (a) streets, ()) alle%s, the a''earance of which is
rated satisfactor% (or unsatisfactor%).
2. :ercentage of (a)households, ()) )usiness rating their
neigh)orhood cleanliness as satisfactor% (or
unsatisfactor%)
rained o)ser&er ratings
(a) >ousehold Sur&e%
()) !usiness Sur&e%
Affensi&e Adors 3. :ercentage of (a) households, ()))usinesses re'orting offensi&e
odors fro$ solid wastes.
(a) >ousehold Sur&e%
()) !usiness Sur&e%
A);ectiona)le
noise incidents
8. :ercentage of (a) households, ()))usinesses re'orting o);ectiona)le
noise fro$ solid waste collection o'erations.
(a) >ousehold sur&e%
()) !usiness Sur&e%
>ealth and
Safet%
>ealth ?. Du$)er and 'ercentage of )loc"s with one or $ore health
ha#ards.
3ire ha#ards @. Du$)er and 'ercentage of )loc"s with one or $ore fire
ha#ards.
rained o)ser&er ratings
rained o)ser&er ratings
3ires in&ol&ing
uncollected waste
B. Du$)er of fires in&ol&ing uncollected solid waste 3ire de'art$ent records
6
Afficials who wish to focus on the a$ount of dissatisfaction $a% su)stitute 4unsatisfactor%5 for the ter$
4satisfactor%5 in $an% of these $easures.
>ealth ha#ards
and unsightl%
a''earance
C. Du$)er of a)andoned auto$o)iles rained o)ser&er ratings
<odent ha#ard 7. :ercentage of (a) households, ()) )usinesses re'orting ha&ing seen
rats on their )loc"s in the 'ast three $onths.
(a) >ousehold sur&e%
()) !usiness Sur&e%
<odent )ites 60. Du$)er of rodent )ites re'orted 'er 6,000 'o'ulation. Cit% or count% health records.
E1%i(it C (continued)
Quality
O!ecti"es
#haracteristics S$eci%ic Measure
'
Data #ollection Procedure
+ini$u$
citi#en
incon&e1
nience
+issed or late
collections
S'illage of trash and
gar)age during
collections. Da$age
to 'ri&ate 'ro'ert% )%
collection crews.
66. Du$)er and 'ercentage of collection routes not co$'leted
on schedule.
62. :ercentage of (a) households, ()) )usinesses re'orting $issed
collections.
63. :ercentage of (a) households, ()) )usinesses re'orting s'illage )%
collection crews.
68. :ercentage of (a) households, ()) )usinesses re'orting 'ro'ert%
da$age caused )% collection crews.
Sanitation de'art$ent
records
(a) >ousehold sur&e% ())
!usiness sur&e% (a)
>ousehold sur&e% ())
!usiness sur&e%
(a) >ousehold Sur&e%
()) !usiness sur&e%
2
Afficials who wish to focus on the a$ount of dissatisfaction $a% su)stitute 4unsatisfactor%5 for the ter$
4satisfactor%5 in $an% of these $easures.
,eneral
Citi#en
satisfaction
Citi#en
co$'laints
:ercei&ed
satisfaction
6?. Du$)er of &erified citi#en co$'laints )% t%'e 'er
6,000 household ser&ed.
6@. :ercentage of (a) households, ()) )usinesses re'orting o&erall
satisfaction with the solid waste collection ser&ice the% recei&e.
Sanitation de'art$ent
records
(a) >ousehold sur&e%
()) !usiness sur&e%.
E1%i(it D* $ou#e%old 2ue#tionnaire
6. What is the se0 and age of the household head/
2. Dote se0 of res'ondent.
What is the relationshi' of the res'ondent to the household head/
3. What is the le&el of education of the household head/
8. What is the occu'ation of the fa$il% head/
?. >ow $an% $e$)ers of the household are there altogether/
What is the age and se0 of each one, starting with the %oungest/
X3or 'eo'le under 20, record whether in 'ri$ar% school
or notY
As" the following *uestions for each child in 'ri$ar% school
(u' to 8 children in 'ri$ar% school).
As" for children registered for 2:9 and in go&ern$ent1aided schools.
@. What class is the child in/
B. >ow $an% ti$es (if an%) has the child to re'eat a class/
C. What 'a%$ents ha&e %ou $ade for the child.s education during the
last %ear/
!oo"sP'ens etc.
2nifor$sPshoes ra&el
to school Afficial
school fees
90tra tuition 'a%$ent to the school
90tra 'a%$ents directl% to teachers
7. >ow satisfied are %ou with the teaching (a'art fro$ e0tra 'ri&ate
tuition) that the child is recei&ing at school/
Xsatisfied Deither satisfied not dissatisfiedY
60. >ow long ago did an% $e$)er of this household last ha&e a contact with:
he central 'olice/
he ;udiciar% ($agistrates courts and a)o&e)/ he
2<A/
;./D = Suggestions for a Develo!ing +ountry's :utcome Measurement 2ffort
he go&ern$ent health ser&ices/
he local ad$inistration/ (including (A 'olice, (C
courts,
local ta0)
Source# 2ganda Dational =ntegrit% Sur&e% 677C: 3inal <e'ort 677C. Ciet
=nternational.
E1%i(it D (continued)
XWrite answer in $onths (if less than one %ear)
or %ears since )eginning of last contactY
XAs" *uestions 60133 for the (AS contact for 9AC> AD9 of
the ser&ices contactedY
X=f none of the ser&ices e&er contacted, s"i' to T3 8Y
66. Which ser&ice was it/
62. What is the se0 and age of the 'erson ha&ing the contact with the
ser&ice/
63. Did %ou contact this ser&ice/ Ar wh% did the% contact %ou/
68. >ow did %ou first $a"ePha&e contact with the ser&ice to hel' %ou through
%our dealings with the$/
6?. >ow did %ou first $a"ePha&e contact with the ser&ice/ ('ersonal
contactPletterPtele'honeP
through friend or relati&e)
6@. Did %ou ha&e a contact in the ser&ice to hel' %ou through %our
dealings with the$/
6B. >ow long did it ta"e for %our 'ro)le$ to )e attended to/
XnPa if ser&ice $ade through the contactY
Xin hours ifX6 da%,
in da%s if 6 da% or $oreY
6C. >a&e %our dealings with the ser&ice )een co$'leted/
67. >ow long did it ta"e to co$'lete dealing with %our 'ro)le$/
Xif not %et co$'leted, 'ut ti$e ta"en u' until nowU
in da%s
if less than 6 $onth, otherwise in $onthsY
20. =f %ou &isited the ser&ice in %our dealings with the$, how $an% &isits
did %ou (or other $e$)ers of the household or so$eone acting for
%ou) $a"e/
26. >ow satisfied were %ou with the s'eed of ser&ice/
XSatisfiedY
Xneither satisfied or dissatisfiedY
XdissatisfiedY
22. >ow satisfied were %ou with the )eha&ior of the staff of the ser&ice
towards %ou/
XSatisfiedY
Xneither satisfied or dissatisfiedY
;./< = Suggestions for a Develo!ing +ountry's :utcome Measurement 2ffort
XdissatisfiedY
23. When %ou first contacted the ser&ice, what infor$ation were %ou gi&en )%
the ser&ice a)out how to use it/
(nonePs'o"enPwrittenPs'o"en and written)
Xif none, s"i' to T2?Y
28. >ow hel'ful was the infor$ation gi&en )% the ser&ice/
Ver% hel'ful So$ewhat hel'ful Dot at all hel'ful
E1%i(it D (continued)
2?. Did %ou "now how to $a"e a co$'laint a)out the ser&ice/
2@. Did %ou actuall% $a"e a co$'laint a)out the ser&ice/
2B. >ow $an% different staff did %ou $eet in %our deali ngs with the
ser&ice/
2C. Did %ou use the hel' of a )ro"er or agent in dealing with the ser&ice/
Xif no, s"i' to T30Y
27. >ow $uch did %ou ha&e to 'a% the )ro"er/
Xwrite $one% e*ui&alentY
30. Did %ou ha&e to 'a% an%thing e0tra, in $one% or goods, (a'art fro$ official
charges) to 'eo'le in the ser&ice to get %our 'ro)le$ dealt with/
Xif no, s"i' to T33Y
36. Did the 'erson(s) concerned as" %ou for $one% or goods or did %ou 'a% on
%our own initiati&e/
32. >ow $uch did %ou ha&e to 'a% e0tra to ser&ice wor"ers to get %our 'ro)le$
dealt with/
Xwrite total of e0tra 'a%$ent in $one% e*ui&alentY
33. =f %ou did not $a"e an% e0tra 'a%$ents to ser&ice wor"ers: Were %ou
as"ed for an% e0tra 'a%$ent, in $one% or
goods, )ut %ou refused/
Did %ou offer an% e0tra 'a%$ent )ut it was refused )% the ser&ice
wor"er/
6FOR ALL $OUSE$OLDS7
38. What do %ou thin" a)out the 'ractice of 'a%ing e0tra ()ri)es)
to ser&ice wor"ers in order to get a ser&ice in fa&or/
3?. What for$s of corru'tion do %ou thin" ha''en in this district/
3@. >ow $uch of a 'ro)le$ do %ou thin" there is with corru'tion in 'u)lic
ser&ices in this District/
XVer% $uchY XSo$ewhatY Xnot at allY
3B. >as this 'ro)le$ got )etter, sta%ed the sa$e or got worse in the last two
%ears/
X!etterY Xthe sa$eY XWorseY
;./; = Suggestions for a Develo!ing +ountry's :utcome Measurement 2ffort
3C. Which go&ern$ent de'art$ents or agencies in this district are the $ost
corru't, in %our o'inion/
Xallow u' to three answersY
37. Which de'art$ents or agencies are the least corru't/
Xallow u' to three answersY
80. What do %ou thin" the n ational go&ern$ent should do to sol&e the
'ro)le$ of corru'tion in 'u)lic ser&ices/
E1%i(it D (continued)
86. What do %ou thin" the local go&ern$ent in this district should do to
sol&e the 'ro)le$ of corru'tion in 'u)lic ser&ices/
82. What do %ou thin" co$$unities the$sel&es could do to hel' sol&e the
'ro)le$ of corru'tion in 'u)lic ser&ices/
83. >a&e %ou heard of the =ns'ectorate of ,o&ern$ent (the =,,)/
Xif -Do., s"i' T8818@Y
88. What does =,, do/
Xallow u' to three answersY
8?. >a&e %ou e&er $ade a co$'laint to the =,,/
8@. >ow satisfied were %ou with the wa% the =,, dealt with %our co$'laint/
XsatisfiedY
Xneither satisfied or dissatisfiedY
XdissatisfiedY
;./A = Suggestions for a Develo!ing +ountry's :utcome Measurement 2ffort
E1%i(it E* E1a"&le# o! Per!or"ance Indicator# !or
Trac8in Anti9Corru&tion E!!ort#
+an% of the following indicators should )e )ro"en out )% such characteristics
as: t%'e of co$'laintPincidentU se&erit% of the co$'laintPincident (such as
a$ount of funds in&ol&ed of the le&el of sus'ected official in&ol&ed)U
districtU source of the case U
organi#ational unit res'onsi)le for initiall% handling the caseU and the
difficult%Pco$'le0it% of the case (e.g., whether of considera)le, $ediu$, or low
difficult% co$'le0it%).
6. Du$)er of corru'tion co$'laints recei&ed and found after 'reli$inar%
in&e stigation to ha&e $erit. Dote: 4Corru'tion5 co$'laints should
e0clude non1corru'tion co$'laints, such as dela%s or $ista"es in $a"ing
'ension or other 'a%$ents that are not due to corru'tion 1related
ele$ents. hese are &er% i$'ortant )ut different 'ro)le$s. his nu$)er
will )e affected )% efforts to get citi#ens and )usinesses to re'ort corru't
acti&ities. his is li"el% to )e a 'ro)le$ for at least the first %ear or two
of a 'rogra$ to encourage $ore citi#ens and )usinesses to re'ort
incidents of corru'tion.
2. Du$)er and 'ercent of corru'tion co$'laints recei&ed that were not
in&estigated.
3. Du$)er of cases for which sufficient e&idence was o)tained to )e
'rosecuted and an arrest $ade.
8. Du$)er and 'ercent, of cases for which a reasona)le 'enalt% was assessed
and carried out, i.e., nu$)er, and 'ercent, of cases in&estigated that were
successfull% resol&ed.
?. Du$)er and 'ercent of cases not 'rosecuted )ecause of 'oor 'ractice in
in&estigation, such as e&idence gathering.
@. Du$)er and 'ercent of cases not successfull% 'rosecuted )ecause of
'oor 'ractice during the 'rosecution stage.
B. +onetar% &alue of $issing, $isused, su''lies and e*ui'$ent found during
audits.
C. :ercent of 'u)li c leaders who ha&e not 'u)licl% declared their assets.
7. A$ount of funds reco&ered through: (a) ad$inistrati&e or
()) legal action
60. Value of $one% reco&ered, 'ro'ert% sei#ed and confiscated for corru't
'ractices.
66. Du$)er and 'ercent of cases lost.
62. :ercent of co$'laint cases whose ti$e fro$ recei't of co$'laint until
the in&estigation was started e0ceeded 4W5 da%s.
;./C = Suggestions for a Develo!ing +ountry's :utcome Measurement 2ffort
E1%i(it E (continued)
63. :ercent of co$'laint cases whose ti$e fro$ recei't of co$'laint
until the in&estigation was co$'leted was 4S5 da%s.
68. :ercent of co$'laint cases whose ti$e fro$ recei't until case was finall%
resol&ed in the courts (not counting ti$e for a''eals).
6?. Du$)er of go&ern$ent e$'lo%ees re$o&ed for corru'tion acti&ities.
6@. Du$)er and 'ercent of go&ern$ent cases o&erturned on a''eals.
6B. :ercent of su)stantiati&e reco$$endations $ade to indi&idual
agenciesPde'art$ents (in s'ecial studies) that were i$'le$ented )%
the agenc% de'art$ent.
6C. Du$)er of cases that could not )e in&estigated )ecause of lac" of
'erson nelPfunds ('erha's )ro"en out )% district).
67. :ercent of citi#ensPhouseholdsP)usinesses in a re'resentati&e sur&e% who
re'orted that the% had $ade e0tra 'a%$ents for one or $ore ser&ices
()ro"en out )% ser&ice
and district).
20. :ercent of citi#ensPhouse holdsP)usinesses in a re'resentati&e sur&e%
who re'orted that the% )elie&ed that corru'tion in their district was less
than 62 $onths ago.
26. :ercent of citi#ensPhouseholdsP)usinesses in a re'resentati&e sur&e% who
re'orted that the% would $a"e a co$'laint (such as to the ins'ector
general or other 'u)lic official) if as"ed for a )ri)e.
22. Du$)er of citi#ensPhouseholdsP)usinesses that in a re'resentati&e
sur&e% re'orted that the% had: (a) seen or heard one of the radio or V
s'ots on corru'tion, had e0a$ined an anti 1corru'tion )rochures, or had
'artici'ated in a wor"sho' or se$inar on corru'tionU and ()) had ta"en
so$e action, called for )% the $aterial 'resented (such as re'orting
)ri)e re*uests fro$ 'u)lic officials).
A&&endi1 A: Trained O(#erver Procedure#
:
rained o)ser&ers can )e used to rate a &ariet% of i$'ortant outco$es that
can )e seen )% the e%es, or other 'h%sical senses, of an o)ser&er. =ts fa$iliar
for$ is in ins'ection functions, such as )uildings, health, and food safet%. A "e%
re*uire$ent for 'erfor$ance $easure$ent is that the rating scales and
'rocedures 'ro&ide reasona)l% accurate ratings.
A high degree of accurac% can )e $aintained if the 'rocedures 'ro&ide:
A well1defined rating s%ste$
Ade*uate training and su'er&ision of the o)s er&ers and the 'rocess,
A 'rocedure for 'eriodicall% chec"ing the *ualit% of the ratings.
rained o)ser&er ratings K if 'ro'erl% done K reduce the su);ecti&it%
in assessing 'h%sical conditions, such as street cleanliness. <atings scales, if
'ro'erl% de&e lo'ed, 'ro&ide trained o)ser&ers with s'ecific criteria against
which to assess conditions in a relia)le $anner. Without such co$$on criteria
to assess conditions, different o)ser&ers will li"el% co$e u' with different
ratings for a condition. Well1defined rating scales reduce the su);ecti&it% in
condition assess$ents.
he goal is for different o)ser&ers, at different ti$es, to gi&e a''ro0i$atel%
the sa$e ratings to si$ilar conditions.
A&&lication# o! Trained O(#erver Ratin#
o a''l% trained o)ser&er rati ngs to a 'articular 'rogra$ outco$e
should:
6
3or further details on trained o)ser&er 'rocedures, see Hohn ,reiner.
6778. 42se of <atings )% rained A)ser&ers,5 Cha'ter 60 in Hose'h S. Whole%, e t al.,
Handbook of Practical Program 2valuation. San 3rancisco, CA: Hosse%1!ass :u)lishers,
and How 2ffective ,re your +ommunity Services( (Second 9dition), Washington, D.C.:
he 2r)an =nstitute and =nternational Cit%PCount%
+anage$ent Association, 6772. Cha'ter 62 and A''endices 3 and @ (Cha'ters 2, 3, 8 and
C 'ro&ide e0a$'les).
;.;D = Suggestions for a Develo!ing +ountry's :utcome Measurement 2ffort
!e $easura)le )% 'h%sical o)ser&ation K through an% of the fi&e senses:
sight, hearing, s$ell, taste, or touch (though $ost a''lications in the
'u)lic sector, thus far, ha&e onl% used sight), and
!e one that can )e rated on a scale that identifies &ariations in
condition.
E1a"&le# o! Outco"e# !or -%ic% Trained O(#erver Ratin#
$ave /een U#ed to Provide Data Cleanliness
of roads and alle%sU Condition of
roadsU
Condition of 'ar"sU
Condition of facilities such as schools, nursing ho$es and hos'italsU
Condition of housingU
Tualit% of food 'ro&ided in 'u)licl% su''orted facilities (taste, s$ell, and
sight can )e used)U and
A)ilit% of handica''ed clients to 'erfor$ acti&ities of dail% li&ing, after
recei&ing 'u)lic ser&ice.
Advantae# o! Trained O(#erver Procedure#
he% can 'ro&ide relia)le, reasona)l% accurate ratings of conditions
that otherwise are difficult to $easure.
he data can )e used to assist 'rogra$s in allocating their resources
throughout the %ear, if the ratings are done 'eriodicall% (for
e0a$'le, the Dew Sor" Cit% Sanitation De'art$ent has for $an%
%ears regularl% used trained o)ser&er ratings of street cleanliness to
hel' allocate its street cleaning crews.
Di#advantae# o! Trained O(#erver &rocedure#
hese are 4la)or 1intensi&e5 'rocedures that re*uire 'ersonnel
ti$e to do the ratings.
<atings need to )e 'eriodicall% chec"ed to ensure that the o)ser&ers are
adhering to the 'rocedures.
:rogra$ 'ersonnel $a% not )e co$forta)le with using this 'rocedure, a
s it is not &er% co$$on.
T'&e o! trained o(#erver ratin #'#te"
Se&eral t%'es of rating s%ste$s can )e used )% trained o)ser&ers.
hese include ones that use:
Written descri'tions onl%U
:hotogra'hic rating s%ste$s that use 'hotogra'hs as the rating scalesU
;.;< = Suggestions for a Develo!ing +ountry's :utcome Measurement 2ffort
Ather &isual scales such as scales using drawings or &ideos rather than
'hotogra'hsU andPor
Co$)inations of the a)o&e.
9ach of these is )riefl% descri)ed )elow.
5rained observer systems using only written descri!tions . his is the si$'lest
t%'e of rating s%ste$. =t de'ends solel% on written descri'tions of each grade
used in the rating scale. hese rating descri'tions need to )e *uite s'ecific
a)out what constitutes each 'articular rating. his is necessar% to $aintain
the accurac% of the ratings.
An a))re&iated e0a$'le of a written scale for 4)uilding (or street)
cleanliness5 is the following:
<ating 6: +lean. !uilding (street) is co$'letel% or al$ost co$'letel%
cleanU a $a0i$u$ of three 'ieces of litter 'er floor
()loc") are 'resent.
<ating 2: Moderately +lean . !uilding (street) is largel%
cleanU a few 'ieces of isolated litterPdirt are
o)ser&a)le.
<ating 3: Moderately Dirty . So$e scattered litterPdirt is 'resent.
<ating 8: Dirty. >ea&% litterPdirt is 'resent in se&eral locations
throughout the )uilding (along the )loc").
Ad&antages of the written rating scales are: =t is the
4si$'lest5 for$ of rating scaleU
=t is the $ost fa$iliar 'rocedure, the one usuall% used )% 'u)lic agencies who
use trained o)ser&er ratings, such as for ins'ectionsU and
=t can also hel' agencies identif% resource allocation needs, )% s'ecific
locations where 'ro)le$s are 'resent.
90hi)it A16 illustrates an additional, &er% i$'ortant, use for such outco$e
infor$ation. he Cit% of oronto used these ratings of street conditions to he l'
deter$ine what re'airs were needed in each location K as noted )% the right1
hand colu$n.
Disad&antages of written rating s%ste$s are:
he written infor$ation for each rating grade needs to )e &er% s'ecific in
order to assure reasona)le accurac%, so that different
raters will gi&e a''ro0i$atel% the sa$e rating for a 'articular condition.
!ecause each grade on the rating scale needs to )e defined &er% clearl%, the
'rogra$ is li"el% to ha&e to s'end a considera)le a$ount of ti$e de&elo'ing
the ratingsU and
his 'rocedure can onl% )e used to $easure outco$es if the outco$e can
)e e0'ressed as a 'h%sicall% o)ser&a)le condition. 5rained observer systems using
!hotogra!hic scales . :hotogra'hic scales can )e &er% useful in 'ro&iding clear
definitions for each rating grade. :re 1selected 'hotos are used to re'resent the
&arious
grades on the rating scale. 9ach trained o)ser&er is gi&en (and trained in the use
of) a set of 'hotos, 'erha's four 'hotos for each grade on the scale.
5rained observer systems using other visual scales . Visual rating scales can
also use drawings or s"etches that re'resent each grade on a rating scale.
90hi)it A12 is an e0a$'le of a rating scale using s"etches to re'resent conditions
of school )uildings, in this case, the condition of school roo$ walls. his was
used )% the Dew Sor" Cit% school s%ste$ to trac" the 'h%sical condition of its
schools and to hel' $a"e decisions a)out re'airs.
;.;; = Suggestions for a Develo!ing +ountry's :utcome Measurement 2ffort
E1%i(it A9:* Toronto Road Condition Ratin Scale
Condition3Ratin
(atin) #ondition Descri$tion #omments
7 90cellent Do fault whatsoe&er
C ,ood Do da$age, nor$al
wear and s$all crac"s
B 3air Slight da$age, crac" fill
or $inor le&eling
re*uired.
@ <e'air 60Z of co$'lete
re'lace$ent cost
? <e'air 2?Z of co$'lete
re'lace$ent cost
8 <e'air ?0Z of co$'lete
re'lace$ent cost
3 <e'air B?Z of co$'lete
re'lace$ent cost
2 <e'air +ore than B?Z of
co$'lete
re'lace$ent cost
<ecentl% constructed
wor"
A&erage rating for Cit% of
oronto 'a&e$ents and
sidewal"s.
:a&e$ent re*uires
're&enti&e o&erla%. (e&el
of tolerance for Cit% of
oronto 'a&e$ents.
9ligi)le for
reconstruction 'rogra$.
Condition <ating 81
(e&el of olerance for
Cit% of oronto cur)s
and sidewal"s.
otal reconstruction
'ro)a)l% indicated
<e*uires co$'lete
reconstruction
6
=$'ossi)l
e to re'air
Scale Value ;
Scale Value 3
E1%i(it A9;* Trained O(#erver Ratin Scale* Condition o!
Sc%ool Cla##roo" -all#
School
Scorecard
<ating scale
Scale Value <
Scale Value :
Scale Value )
Scale Value =
Scale value >
Source : Dew Sor" Cit% De'art$ent 9ducation
.
E1%i(it A93* M a5or Ste&# in I"&le"entin a Trained O(#erver
Proce##
he following are ste's needed to i$'le$ent a trained o)ser&er 'rocess:
6. Decide what conditions should )e rated.
2. De&elo' a rating scale for each condition. 2se 'hotogra'hs, andPor
written guidelines as a''ro'riate.
3. Deter$ine which locationsPclients should )e rated, when, and how fre*uentl%.
<atings can )e a''lied to all or to selected locationsPclients. =f %ou onl% ha&e re
sources to rate so$e locations, choose the$ )% using 4rando$ sa$'ling5 so that
the locations chosen are re'resentati&e of all locationsPclients.
8. Select and train o)ser&ers. Sou can use 'rogra$ 'ersonnel or e&en school
students and &olunteers. +ore technical ratings, howe&er, will re*uire
'ersons with $ore 4'rofessional5 training, such as to o)tain ratings of safet%
ha#ards.
?. est the scale and o)ser&ers on a s$all nu$)er of locations to assure that
reasona)l% trained raters will gi&e a''ro0i$atel% the sa$e ratings.
@. 9sta)lish 'rocedures for su'er&ising the o)ser&ers, recording the data the%
collect, and transcri)ing and 'rocessing data.
B. Conduct the ratings at the desired fre*uenc%.
C. De&elo' and disse$inate re'orts on the findings. he re'orts will )e $ore
infor$ati&e if the% show the nu$)er and 'ercentage of locations that fell into
each rating categor%. A&oid re'orting onl% 4a&erage5 score, which can hide
i$'ortant distri)ution infor$ation. Also in re'orts, show changes in the ratings
fro$ 're&ious 'eriods.
7. 9sta)lish 'rocedures for s%ste$aticall% chec"ing the ratings of trained
o)ser&ers to assure *ualit% control and accurac% of the 'rocess. 3or e0a$'le, the
su'er&isor $ight 'eriodicall% chec" a s$all sa$'le of ratings done )% each
o)ser&er.
he following are additional ste's needed for a 'hotogra'hic rating
s%ste$
a"e a large nu$)er of 'hotogra'hs in locations re'resentati&e of the range
conditions to )e rated. hese 'hotos should show the actual t%'es of conditions
that the 'rogra$ wants to assess. Select a set of fa$iliar la)els, each
re'resenting a condition that the 'rogra$ e0'ects to find, such as: 4clean,5
4$oderatel% clean,5 4$oderatel% dirt%,5 and 4dirt%5 (for o)ser&ations of
cleanliness). Select a 'anel of 4;udges5 co$'rised of 'ersons with &aried
)ac"grounds who are not associated with the $easure$ent acti&ities. As"
the ;udges to sort the 'hotogra'hs into grou's that re'resent each condition.
3or each condition le&el, select the four or fi&e 'hotogra'hs that $ost ;udges
identified as re'resenting that le&el. hese 'hotogra'hs can then )e used as
the rating scale.
rain the o)ser&ers in the use of 'hotogra'hic scale. 3ield test the scale with
trained o)ser&ers to deter$ine if there is sufficient agree$ent a$ong the$
on the ratings. =f not, re&ise the 'rocedures.
De&elo' the final scale. :ac"age co'ies of the selected 'hotogra'hs in a
4"it5 for each trained o)ser&er.
?Auto"atin@ Trained O(#erver Ratin#
S$all hand1held co$'uters that can )e 'rogra$$ed so that trained o)ser&ers
can electronicall% record their ratings (as soon as the ratings are $ade) are
)eco$ing a&aila)le at low 'rices. his can considera)l% ease the a$ount of
later clerical wor" needed. 3or e0a$'le, the cit% of Dew Sor" has )een using
such 'rocedures to regularl% re 'ort on the condition of cit% )uildings
(including schools)Kto $eet legislati&e re*uire$ents. An D,A, the 3und for
the Cit% of Dew Sor" (a 'ri&ate 'u)lic interest organi#ation) has used hand1held
co$'uters to collect data on a nu$)er of 'h%sicall% o)ser&a)le conditions in
sa$'les of locations within Dew Sor" Cit%. he conditions o)ser&ed include,
a$ong others, defecti&e street signs, a)andoned cars, 'resence of rodentsP'ests,
and defecti&e street lights. his infor$ation is used to identif% s'ecific and
su$$ar% conditions in &arious cit% co$$unit% districts.