See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/2874104

Web Usability Today: Theories, Approach and Methods

Article · February 2004

Source: CiteSeer

CITATIONS READS

8 1,883

2 authors, including:

Luciano Gamberini
University of Padova
218 PUBLICATIONS   3,189 CITATIONS   

SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Cyber secure IoT domotic system for smart building View project

The Collective Experience of Empathic Data Systems View project

All content following this page was uploaded by Luciano Gamberini on 02 February 2013.

The user has requested enhancement of the downloaded file.

Giuseppe Riva & Carlo Galimberti (Eds.)
Towards CyberPsychology: Mind, Cognitions and Society in the Internet Age
Amsterdam, IOS Press, © 2001, 2002, 2003

7 Web Usability Today:

Theories, Approach and Methods
Luciano GAMBERINI, Elisabetta VALENTINI

Abstract. The aim of this work is to introduce the constant transformation and
evolution of the usability concept. An overview of methods, techniques and
theories concerning usability is supplied. The reported review starts from the
description of traditional ergonomic methods and models, coming to the
suggestion of innovative theoretical and methodological proposals. We claim
that usability should always take context into account when studying artifacts
such as hardware and software, as they are not to be considered as mere tools,
unrelated to the concrete situation in which they are used. Thus, usability has to
be implemented within a cultural framework, from which actions take their
meaning.

Contents

7.1 Usability: a concept in continuous evolution.................................. 110

7.2 Theories and approaches to WEB usability.................................. 111
7.3 Approaches, methods and techniques for Web usability............... 114
7.4 Conclusion................................................................................. 122
7.5 Acknowledgement...................................................................... 122

7.6 References ................................................................................. 122

109
7.1 Usability: a concept in continuous evolution

The concept of "usability" is formally defined by the International Standards Organization

(ISO) as: "… the effectiveness, efficiency and satisfaction with which a certain user may
achieve a specific objective in a particular environment" (ISO DIS 9241-11). According to
these standards, "effectiveness" represents the percentage of the use of the machine in
reference to its possibilities, "efficiency" concerns the quantity of "effort" required for the
purpose: the more effort that is required, the lower the efficiency of the object in question.
The concept of "satisfaction", finally, is related to the comfort that the user experiences
in using a certain product. Despite this rather precise definition, various authors agree on
the fact that the concept of usability is not an easy one to express. The limits of the
meaning of usability are often blurred or poorly defined [1], and with their constant
changes reflect the characteristics of the artifacts that we use on a daily basis. It concerns
the way in which we learn to interact with the world using rapidly changing technological
products that transform themselves and expand their range of characteristics and functions.
To have an idea of how vast this field of action is, we can simply observe the
heterogeneity of a number of situations related to new technology, of which we have
scientific demonstration. Observing the staff at Xerox when they use the web as a
communicative knowledge medium, replacing a failed expert system for the recognition of
faults [2], studying the suitability of a system of immersive virtual reality as preventative
training for emergency situations [3], providing systems of artificial intelligence for
adaptive web sites, that is, sites capable of modifying their structure and their interface
according the preferences of the users [4], designing asynchronous computer networks as
aids to university courses to improve the interaction between students and teachers [5],
means in any case discussing usability, while not being bound by definitions that are
difficult to propose.
We will run through the facts from the beginning. Traditionally, usability has been
considered important in the professional field for reasons that range from safety, to
annoyance, frustration, and factors of an economic nature that may involve productivity or
the sale of products. Even everyday objects such as VCRs, washing machines and
telephones may cause or help cause annoyance, frustration and stress, and in the worst
cases even accidents [6]. Donald Norman, in his historic "Psychology of everyday things"
[7] describes, starting from the everyday use of these objects, a model of user-world
interaction, characterized by a constant cycle that recurs between the user’s goal (intention,
action specification), execution and evaluation of new world state. Based on his strongly
cognitivistic theories, Norman argues four high-level design principles:

- Propose a good conceptual model with the aim of: allowing the correct planning of the
actions to be performed, intervening in anomalous situations or in the event of faults,
and reducing operating errors.
- Visibility: make the things visible. Allow the user to observe how a mechanism is
made, what parts it is made of, and how these interact, showing the consequences that
may arise from a certain action.
- Provide a good relationship between the parts of the product (natural mapping), above
all between the control devices and the parts that are operated. The mapping must
where possible be natural, that is, be based on physical or cultural analogies, so as to be
immediately understandable by the user.

110
- Envisage the use of feedback telling the user what action they have performed and what
consequences this has caused.

Another recent study on usability that has received considerable attention, regards a model
whereby the usability of objects is taken apart and described in its fundamental elements.
This is the model with five components proposed by Jordan and colleagues [8], [9]. The
final model [9] indeed proposes five reference concepts, which are, guessability,
learnability, experienced users performances (EUP), system potential and re-usability.
These are associated, respectively, with the first time use of a product for a particular task,
the number of task repetitions required until an acceptable level of "competence" is
reached, relatively stable level of performances, the practical best performance obtainable,
and finally the level of performance achieved when a user returns to a task after a long
period of non use [10].
The idea of contributing to the development of knowledge in terms of usability by
building descriptive-interpretative and heuristic frameworks to support developers and
designers, has accompanied the work of many researchers until the present day. It is within
this interpretative framework that web usability has begun to be discussed, that is, how to
make the most widespread system of Internet-based communication more usable.

7.2 Theories and approaches to WEB usability.

7.2.1 Interfacing with the Internet: Web usability and Human Factor Engineering

With the advent of the computer, studies on usability have found a wide basis for
application and today, following the mass introduction of the personal computer and
software into the home, concepts such as "user friendly" or "ergonomic design" have
become part of everyday language.
The most common perspective on usability and computers today originates from human
factor engineering and human machine system engineering literature. In these studies,
which include the research carried out by Nielsen [11,12] focused on the web, and the
above-mentioned works by Norman, the accent is placed on human performances as the
activity evaluated within contexts defined mainly or exclusively by the task and its
subcomponents. The measurements made as part of these studies are typical of cognitive
ergonomics, such as reaction time [13, 14] and those based on the detection of human error
[15-17]. The best performance is generally the fastest and most accurate; this is commonly
measured during the interaction between man and one or more products in strictly
experimental situations or in more ecological conditions, such as work situations.
Various authors have subsequently built practical models and laid down guidelines
based on these theories and aimed at the designers of World Wide Web sites. In this
approach, focused on the concept of the interface, the importance is placed on one hand on
the contribution of technology, which aims to provide hardware and software systems that
are increasingly fast and reliable, and on the other, on the knowledge of the cognitive
processes underlying human actions. The general focus is as mentioned the development of
user-centered design techniques that can be rapidly adopted by the teams that build web
sites [18], for example, provides one definition of usability by identifying five attributes of
a usable interface: (1) it’s easy to learn, (2) it’s efficient to use, (3) it’s easy to remember,
(4) it causes few errors, (5) it’s pleasant to use. The interdisciplinary character of Human
Factor suggests the consideration of the variables of the design environment, such as the
web design languages (HTML, XML, XHTML, Java) and the browser or bandwidth,
which are substantially technical and psychological variables implicated in the
111
interpretation of graphic options and colors [19], different typographic characters (fonts) in
the text [20], the links and the hypertexts [18, 21], and the hierarchical structures of the
sites, [22].
Studies on navigation in multimedia hypertexts represent a significant part of the studies
on Web Usability deriving from the human factor approach. These studies describe the
problem of usability through the examination of the cognitive skills involved during the
navigation of a web site [23]. Generally, the overall purpose of such research is to provide
a supplemental navigation system, also called remote navigation element, or more simply
navigational aid.
These are elements that help a user to locate information on a web site, and allow the
user to easily move from page to page [24]. Such aids may be separate from or included in
the structure of the site. They may be made up of elements that are commonly present in
the browsers (forward, back, home, bookmark) or specific elements on the web pages we
visit, such as the names of the pages, the URLs, a site search engine, or a map that
graphically illustrates a web site’s architecture. Some works also propose specific tools
that are able to simplify the users’ tasks, by providing them with new and specific
"instruments" [25], for example, analyze the relationship that is established between the
type of client (browser) and the user’s cognitive processes during navigation. What appears
clear it that the users make a large number of errors due to the models of hypertext
navigation facilities provided by the web client application. The authors present a system
(WebNet) that extends the navigational facilities of conventional WWW through the
creation of dynamic graphical overview diagrams (also see Zizi and Beaudouin-Lafon,
[26]), which can provide users effective indexes for moving between the links on the
Internet.
Other navigational aids have been proposed to support user browsing. Recently, Head,
Archer and Yuan [27] have presented MEMOS, an on-line history tool that allows users to
organize information retrieval sessions with better success compared to the traditional tools
offered by the browser. Campbell [28] proposes a sort of Road-signs on Web, with the aim
of letting the user know the connection speeds of the proposed links; these signs have the
result of improving the performance of the users, by improving link evaluation and
decision processes. Sørensen, Macklin and Beaumont [29] assume that "supporting Web
navigation implies the provision of tools and techniques enabling users to access Web
resources and maintain an index to these resources”. These authors present the study of six
experimental prototypes to support various aspects of bookmark maintenance and
information filtering.
It is perhaps starting from studies on a complex activity such as navigation that we can
understand how the task of making a web site useable cannot be separated from the aims of
the navigator, from the new ideas that emerge during netsurfing, from the context in which
information retrieval occurs or the economic transaction takes place during an e-commerce
session. Navigation in these electronic environments does not imply only cognitive
operations of a spatial nature: when navigating in a hypertext structure, we participate in
the birth of new concepts, the sharing of meanings, the formation of new knowledge, the
recontextualization of old ideas. If we try to set aside the idea of the web as a means of
information broadcasting and prepare to accept its cultural validity and social dimension,
the concept of usability acquires, as we will see in the next paragraph, new and interesting
dimensions.

7.2.2 The situated dimension of usability

The classic metaphor of the computer used in cognitive sciences to represent the human
mind clarifies the theoretical concept from which Human Factor Engineering research
112
derives: the human cognitive system is seen as a hierarchical structure made up of various
units, each of which handles a specific function. The organization of the functional
components or "architecture" is what allows the system to generate suitable interactions
with the world [30]. Perception, reasoning and action are thus seen as "separate" processes
that allow us to find and interpret the information from the outside world and allow the
creation and execution of pre-established plans present in the long-term memory. In this
theoretical framework, knowledge is represented as a symbolic system and conceived as
the transfer of information from one place to another, without considering the context in
which this knowledge is developed and put into action.
The context is, indeed, the focus of a more recent survey paradigm, that of situated
cognizance proposed by Clancey [31]. This theoretical framework encloses the
psychological aspects that are closely correlated to the physical and social world.
Perception, learning, reasoning and action are here proposed "in a new perspective that
does not define them as independent processes that are linked in a linear fashion, but rather
as coupled aspects of the mind, that is, linked by a relationship of co-determination" [30].
Knowledge too takes on a different connotation and becomes a process that develops
into action, which in turn is created in the contingent circumstances, that is, situated in the
context. Embracing a concept of this type means shifting the focus from processes linked
to the individual, to social dynamics. As stressed in the Theory of situated action by
Suchman [32], "… the dependence of the action on a complex world of objects, artifacts
and other actors, situated in time and space, is no longer treated as an extraneous problem
that the individual actor must face, but is rather conceived as the fundamental resource that
makes knowledge possible and gives action its meaning." (p. 179) [32]. In this theoretical
framework, individuals are recognized as social actors who carry their own baggage of
interests and purposes, which are modified and redefined as a result of their continuous
interactions with the environment. Hert [33], for example, in a study based on the retrieval
of information using a system of on-line publication cataloguing (OPAC), found that the
actions performed by users during the search for information were not completely defined
in advance, but rather depended on the elements present in the situation. The users
modified their initial objectives according to the elements that emerged during their
interaction with the system. Bardram [34] also shares the view that plans, considered as
forecasts of purposes, are central in human activity, and that purposes are revised and
reconstructed based on the elements and circumstances of the context. The author thus
proposes the redefinition of the role of plans and rules within the working activities of a
hospital.
The model proposed - "The Patient Scheduler" - is a system based on the
communication, planning and sharing of information relating to the care and treatment of
patients. Such information, organized chronologically, is made available to all doctors in
all hospital departments. Interacting with the computer network, the professionals from the
various departments can gain a global view of the treatment of the patients, and work
together with the other departments to organize and optimize the various activities. In this
way, plans can be continually modified and redefined based on the situations that emerge
during the care of the patients and the hospital’s activities. The plan thus becomes socially
co-constructed, and is shared between the actors, taking on a central role in the working
practices of an organization; the situations become the place where the expectations of the
actors and the opportunities of the environment meet and are constructed reciprocally [35].
In a wider perspective of situated action, which includes some assumptions of cultural
psychology, the concept of "artifact" is also included, which is essential for studying
people and the environment. Artifacts are tools that are "invented" for specific purposes,
and are used to reach the objectives that such contain. Artifacts in this sense take on an
essentially social role, and their meaning no longer exists separately "but emerges only
113
through their incorporation in social procedures" [36]. Artifacts, to be shared socially, must
be easy to interpret, and not create too many problems regarding their use, that is, they
should be "transparent". The importance of the sharing and the "cognitive transparency" of
artifacts is also stressed by Blackwell and Arnold, [37]. Their study was based on role-
playing methodology in which a number of planning psychology experts had to design a
software application. The task was situated in a setting conceived to be related to reality.
The simulation of the exercise was performed successfully, in that the participants had
understood the characteristics of the software from the very beginning. This made it
possible to shift the focus of the participants from the interaction with the artifact to the
negotiation between the demands of the social context and the purposes of the individual
actors, highlighting the influence of the "transparency" factor.
A subsequent assumption borrowed from cultural psychology involves that of not
considering individuals as isolated, but as belonging to communities. Inside these
communities the expectations, purposes and meanings of each actor are shared and
negotiated on common ground. In this sense, communities of practice are a privileged
space, in which the social meaning of the artifact is learned. Lave and Wenger [38] sustain
that learning is essentially situated and is the result of the increasing participation by an
individual inside a community of practice. Pennell, Durham, Ozog and Spark [39], sharing
this view, conducted a study to confirm its validity. This study was based on the
consideration that young people, during their education, do not have the possibility to learn
a professional writing style suitable for communication inside a work organization. With
the aim of ensuring such skills were acquired, a virtual organization was created based on
the use of the web. The situated learning of the students was allowed by the interaction of
each with their own tutor and a mentor from inside the organization, by carrying out
interviews and by the drafting of reports to be handed in at set intervals.
The self-assessment of the participating students, as well as the assessments made by
the mentors and the tutors, provided positive results, demonstrating the effectiveness of the
virtual environment created. In recent years, the computing artifacts that have pervaded
social communication, games, free time, companies and organizations, have forced the
individuals belonging to the various communities of practice to redefine tasks, identities,
roles and meanings. Computer mediated communication (CMC), is transformed from
being a purely cognitive artifact into a social artifact that is able to exceed geographical
barriers and allow actors from different parts of the world to relate to one another. It also
changes the ways of communicating, and increasingly takes on the forms of negotiation
and construction of shared meanings. "The meeting place becomes the Internet, the World
Wide Web (WWW), which is increasingly considered no longer as a purely physical
structure, but as a cultural space in which new forms of social relations and identity are
experimented" [35].

7.3 Approaches, methods and techniques for Web usability

7.3.1 Method and context.

The relationship between the theories described in the first part of this work and the
methodological approaches that will be presented in this part of the article, is not strongly
structured and has a rather dynamic character. It will not however be difficult for the reader
to recognize how some of the methods presented do not -at least in their original version-
pay particular attention to the context in which the actions of the participants are
determined. According to [40], the use of decontextualized techniques in reality poses
significant obstacles for the designer dealing with usability: the strongly abstract nature
114
that the guidelines resulting from this approach assume lead to a "wide variety of
interpretation in different contexts" (p. 228) [40] that may induce inappropriate choices.
Other methods that hold the context in better consideration follow rather faithfully the
theoretical suggestions described in cultural psychology [41] and cognitive-cultural
ergonomics [35]. These adopt the ideas behind distributed cognition and situated action
[42, 32] and show interest in the analysis of the "context" in terms of social and cultural
factors, including the presence of pre-existing plans or procedures, cognitive tools,
organizational structures, the context where the action occurs. The fact remains, however,
that even "non-contextualized" methods can be applied so as to consider "situated" aspects,
or vice-versa, even the best "contextual" method can be poorly applied without respecting
the theoretical framework that supports the function.
Another factor that complicates the work even further is the absence of individual
methods capable of independently measuring and assessing web usability. The trend we
find, above all in the aspect of works of an applicative nature, is the use of a series of
methods together. These can be applied at the same time, that is to say at the same moment
in the development process, or, as is more commonly the case, by spreading them across
the various phases of design and development of the sites. The use of a series of
methodological approaches allows more reliable assessments to be obtained by comparing
the results and the suggestions that derive from the various different investigations. In
reality, the apparent weakness that appears from this picture is not an indicator of
methodological insufficiency as such. It rather reflects the complexity that the Internet
assumes if examined not only as a reservoir of information, but also as "an environment
that allows us to reconsider our way of conceiving knowledge and communication" [43].

7.3.2 Log analysis

Server log files are records of web server activity. Log files contain mainly data on the
identity of the visitors, on the paths followed by the navigators through the site and the
time spent navigating on the pages. These records had the original function of helping site
administrators know if the bandwidth capacity of their server was consistent with the
activities of the web sites they hosted. In recent years, above all due to the needs deriving
from the increase in e-commerce sites, log files are used to track the activities and profiles
of the users who connect to the site. The knowledge that the analysis of the log leads to,
unlike all the other methods reported in this work, concerns very large numbers of people,
observed for relatively long periods of time, working in perfectly natural conditions. These
specific characteristics of web analysis allow “top down” evaluation of what the users as a
group spontaneously do inside the entire site during navigation. On the contrary, for
technical-methodological needs, most of the systems for evaluating the usability of web
sites are performed on limited samples of participants, using specially prepared situations
and over a normally brief time span that is restricted by the conditions of the experiment.
Using this technique, Cooper [44] has recently performed an analysis on a university’s
web-based library catalogue. He investigated usage patterns over 479 days, based on the
traces left by 2.5 million sessions carried out by the users.
Usability specialists adopting this method should however be careful that the log files
contain the complete package of data so as to be able to retrace all the actions that the users
performed during their visit. The ideal data that we should obtain are therefore: who visited
the site, how much time was spent in the various sections of the site, what path the visitors
took through the pages of the site, where visitors left the site and data about the success of
users’ actions, such as the download of files or economic transactions completed. In any
case, it should be stressed that the simple fact that a user has visited a page and has spent
some time there doesn’t allow us to claim with extreme certainty that they have read the
115
contents of that page. Not always then is the analysis of logs useful for the purposes
established. The use of log files for usability evaluation is besides an often imprecise
method, for reasons that depend on the environment of the server, the client systems, and
the use of tools that are "external" to the hypertext in order to reach a particular page. For
example, navigating using bookmarks, addresses sent by e-mail or typing the URL in the
corresponding field takes us directly to a page. This leads to a substantial failure in the
recording of the paths, which no longer describe a linear structure [45]. Currently, despite
the fact that the more commonly used measurements are still those involving accesses and
the path followed by the users, attention is also paid to the measurements of the transit
times on the pages in the site.
Fuller and De Graaff [46], for example, describe how these times can provide a path-
independent measurement of the behavioral trends in a distributed and diverse community.
The time spent on a page by users allows us to monitor the entire web site or some of its
parts, measuring which pages have aroused more interest among the visitors, and allowing
forecasts of future trends. In any case, even the measurement of the transit times on web
pages cannot be considered, in our opinion, a complete indicator of the usability of a site.
The method of log analysis for ergonomic purposes must in this way be integrated with
other methods of investigation. Kantner and Rusinski [47], with the aim of analyzing a
Beta-Version Web Site, used, together with automated data collection provided by the log
files, an on-line questionnaire and a follow-up interview able to provide qualitative
information on the opinions of the users. This data was added to the information on users
collected using a form that the users had to fill in when accessing the site. The authors
concluded their work by stressing how the combination of these methods provides results
that are more reliable than those provided by any one of the methods alone.

7.3.3 Heuristic Evaluation

This is an inspection method where usability experts study the software interface by
evaluating each individual element, such as the buttons or the links, and comparing these
with a list of widely approved and shared design principles that take the general name of
guideline checklists. Nielsen originally developed a checklist for the heuristic evaluation of
the web in collaboration with Rolf Molich [48], [49]. The checklists of optimal
characteristics that before these works consisted of long, all-inclusive lists, have today
been reduced to just a few elements, for practical reasons.
Nielsen [18] proposes 10 elements, for example. These are: visibility of system status,
match between system and the real world, user control and freedom, consistency and
standards, error prevention, recognition rather than recall, flexibility and efficiency of use,
aesthetic and minimalist design, help users recognize, diagnose, and recover from errors,
help and documentation.
A refined version of the checklist based on a factor analysis of 249 usability problems
[50] has been proposed in order to derive a set of heuristics with maximum explanatory
power and to face the problems of time that often dictate the rhythms of the analysis of
usability during the design of the web site [51]. Once the experts finish their work, they
provide feedback to the designer.
This feedback on the status of the system may be provided in the form of a structured
report, that is to say, a formal, independently written report. To make the work of the
experts more accurate, they can be accompanied by an assistant to transcribe the verbalized
findings.
This evaluation technique can be used at any time during the web site development
cycle, but it is best suited to the earlier stages [52].

116
7.3.4 Cognitive walkthrough

This too is a form of expert usability evaluation. Nonetheless, while in an expert appraisal
the investigator observes and studies above all the design of the site, providing comments
from time to time, in cognitive walkthrough the expert navigates by assuming the point of
view of a typical or inexperienced user. This method is often used to help orient designers
during the creation of the sites, providing feedback on the progress of the work, and on the
ease of learning of the interfaces that are proposed. De Villiers [53], for example,
conducted a study to evaluate the usability of two South African e-commerce web sites -
the Fortes King Hotels site, and the Cellular Shop site- using a set of techniques, including
cognitive walkthrough.
The participants, computer science students, used this technique to evaluate the two
sites, according to four parameters: the general characteristics of the sites, the task that the
users must perform, the actions required to complete the task, and the characteristics of the
users. The information collected allowed the identification not only of the sites’ merits, but
also of some aspects that could be improved, such as the excess of information that made
the consultation of some pages boring.
This method is suitable for testing the operation of the site even with completely
inexperienced users. Lisle, Dong and Isensee [54] used the cognitive walkthrough method
to identify potential problems for users visiting the IBM Web Ease of Use site for the first
time. The purpose was to study how the users handle the information presented and
interpret the feedback from the system.
This method of investigation contributed to the development of guidelines for
improving the usability of the site; some basic principles set down by the designers at IBM
were: maintain the simplicity of the interface, not allow the user to make errors, and
provide immediate feedback on their actions.
Pluralistic walkthroughs [55] involve a large group of end users, developers, product
designers, health/safety professionals and usability professionals, generally guided by a
session leader and facilitator. These navigate together, step-by-step, through a task
scenario, analyzing and evaluating each element of interaction. The advantage of this
pluralistic version of cognitive walkthrough is that it involves a greater number of points of
view, and thus also of a larger number of comments, which, originating from quite
different experiences and levels of knowledge, is assumed to be more significant in
identifying the problem areas of a web site. Cognitive walkthrough too, as in the case of
the heuristic evaluation above, is best used in the early stages of web site development.
Despite the fact that cognitive walkthrough is one of the most used methods in the study
of Web Usability, the general orientation provided to designers should however be
considered partial, in that it is now well-known that not all usability problems emerge from
the application of this technique.

7.3.5 Questionnaires

Questionnaires can be used in the evaluation of the usability of web sites, in that they allow
information to be obtained on the opinions, desires and expectations of the potential users
of the sites. These investigation tools are made up of a list of written questions that are
created and formulated according to what knowledge the team of designers considers to be
useful in order to develop the web site. In this sense, questionnaires are useful and
informative in all phases of the development and design of the site; indeed, they can be
used: before its creation for the purpose of knowing the expectations and desires of users in
advance, for the evaluation of a prototype or a site under construction in order to discover
117
the merits and aspects that may be improved, and after the final creation of the site, to
measure user satisfaction. Lisle, Dong and Isensee [54] developed an on-line questionnaire
to evaluate the usability of a site on the HCI they built. They included it in a series of IBM
sites with the aim of acquiring feedback from the users of such sites. They wanted to
obtain new comments on the site not from the usual fixed group of designers, but rather
through the participation of most of the users. The results were positive, in that they
received a large quantity of feedback through the compilation of the questionnaires by the
users.

7.3.6 Interviews and focus groups

These two techniques differ from questionnaires in that the experimenter interacts directly
with the users, eliciting opinions and comments on the product. The participants in this
type of investigation answer the questions on their experiences and preferences regarding
interaction with the site. While the interviews are often structured formally, the focus
group is less formal, and allows a large number of users to discuss the matters together,
with the aim of eliciting common problems and important issues for the evaluation of the
site in question. Vaughan and Schwartz [56] conducted a study in which a focus group was
used in the construction of a web site. The participants were asked to express their opinions
as potential users of the site, so as to obtain suggestions for adding new services or
improving those already present. In this case, the site was tested by eight participants, each
of whom was assigned tasks to be completed; in some cases, on the other hand, free
navigation can also be proposed. The results of Vaughan and Schwartz demonstrated the
appropriateness of the choices suggested by the group and used to orient the development
of the site.
These investigation techniques can be used, as in the case of questionnaires, in all
phases of the development and design of web sites and portals, in that the experimenter can
sort the questions according to the information that they wish to obtain from the users.

7.3.7 Think Aloud Protocol

This method involves a participant speaking about what they are doing or thinking when
they interact with an artifact. The method can be applied by assigning the subject a specific
task (for example, that of finding a particular subject on a site) or even allowing free
navigation, provided that a special hypertext system has been created for the participants to
work on; in this sense, the think aloud protocol is used especially for the evaluation of
prototypes or already existing sites.
The role of the experimenter during these work sessions is that of a group leader: they
must stimulate the participant to continue to think aloud, motivating them to describe what
is happening, the difficulties met, and the reasons for certain actions. This technique is
especially useful, as it is able to capture a wide range of cognitive activities of the users,
and is not limited to the identification of problems, but rather aims to provide information
on their origins and on which cognitive mechanisms they involve. Eveland and Dunwoody
[57], for example, used the think aloud protocol to analyze the activities of information
processing during the learning of information contained in the hypertext. Using
participants with different levels of experience and frequency of use of the web, the
authors performed a quantitative analysis of think aloud protocols obtained during
navigation. The authors found that users spend a substantial proportion of their cognitive
effort orienting to the content and structure of the web, and this effort weighs down
elaborative and evaluative processing, affecting the level of learning of the contents of the
pages on the site.
118
 One disadvantage of this method may arise in the case of interference between the
participants’ verbalizations and the task that they are performing [10]; in this case, people
are not performing one simple task, but rather two, that is, navigating in the hypertext and
verbalizing their actions, which distorts the results of the research. To overcome this
problem, two variations of the protocol have been created: Critical Response, in which the
users are asked to speak only during the execution of predetermined subtasks, and Periodic
Report, which involves, when the task is complex, verbalization only at predetermined
intervals of time. Another disadvantage of this method derives from the fact that the
reports may be adapted to the interlocutor, that is, the experimenter. This could involve a
"rationalization" of the report and a stiffening in the style of interaction with the web
pages. Therefore, to make the results more reliable, this technique is not normally used
alone, but rather integrated with other investigation protocols, such as for example the co-
discovery method.

7.3.8 Co-discovery method.

This method allows the usability of a site to be tested in all phases of its development:
during the design, the development of the prototype, and its final use. It involves the
interaction of two participants who must complete preset tasks while being observed by the
experimenter. They must help each other, as if they were working together to reach a
common objective using the web. As happens with think aloud protocol, they are
encouraged to perform the tasks and to explain aloud what they think of their actions and
the feedback received from the system. The advantage over the latter investigation protocol
consists in the fact that the verbalizations of the participants occurs more naturally, and the
interaction of two people working on the same task, comparing opinions, can lead to a
greater amount of important information than the thoughts of a single person. This has
been experimentally verified as part of a research study carried out by Lim, Ward and
Benbasat [58]. These authors compared the techniques of self-discovery and co-discovery
in evaluating the learning of a series of procedures during the execution of computer tasks.
The results show that the group with which the technique of co-discovery was used
obtained better performance, highlighting the greater effectiveness of the social technique
compared to the individual one.

7.3.9 Contextual Inquiry.

Contextual inquiry is a method of analysis and investigation, a process of discovery and

learning that synthesizes some aspects of ethnographic research studies and participatory
design methods. Proposed by Hugh and Holtzblatt [59], this method consists of
interviewing people in their own workplace while they perform their own real tasks. The
methodology involves the designer teams conducting the interviews at the same time, each
with one user, regarding the site in question. One of the assumptions underlying this
technique is that the environment where people work influences the way in which they use
artifacts. This technique thus has the purpose of providing the designers of sites with deep
and detailed knowledge of the work of the user, their scenarios and the terminology that
they use. These elements can then constitute the basis of the design. Traditionally,
designers have obtained information about the potential users of their sites using
techniques such as questionnaires or focus groups, which however do not consider the
context in which the users work. Understanding how the users work, on the other hand,
depends on the knowledge of the specific situations in which they act. To build a usable
product, then, it is necessary to clearly understand the context in which people will use it.

119
In this perspective, the user is seen as a partner who contributes to the design of the site, in
that they provide important data on the way in which the information is handled in their
specific workplace and on the limits that this sets. Given its characteristics, the contextual
inquiry method is used above all during the design or re-design of the system to correct the
poorer aspects of the site emerging from the application of this technique.
Lau and Staczek [60], for example, used the contextual inquiry method to evaluate the
important aspects in the maintenance of web sites. This evaluation was performed using a
system designed by researchers at AT&T, called Strudel. Following the Contextual Inquiry
methodology, the designers interviewed, in their workplace, six people involved in the
maintenance of web sites. As expected by the Strudel system, three basic issues emerged in
the maintenance of the sites: the content, the structure and the graphics. The results,
overall, provided suggestions on improving the usability of each of these specific aspects.
Another interesting application of this method was reported by Ritchie and Gosbee [61],
who conducted a research study to build the site of the Michigan Center for Rural Health.
The purpose of the site was to promote interaction between the experts at the various
rural health clinics and between the medical and administrative staff. As the users had
different degrees of expertise in the use of computers, the problem of the usability of the
site was fundamental.
The subjects, members of the medical and administrative staff at eight rural health
clinics, were interviewed in their workplace in relation to their problems and needs. By the
application of the contextual inquiry method, it was discovered that the features required to
make the site more easy to use were a simple home page, more visible distinction between
links relating to the clinical area and the administrative area, and on-line instructions
relating to the site search engine.

7.3.10 Object-Oriented and Scenario Based Techniques

Increasing attention to the contextual aspects of HCI has led to the development of a
project-related approach known as Object-Oriented Design (OOD) [62], which is
characterized by the emphasis being placed on the relationship between the tasks, the skills
of the designer and the elements that the situation-problem they have to work on is made
up of [36]. This model acts on two levels; at a lower level, it sets the objective of
identifying classes of users and purposes, as well as the relationship that links them
together and their implementation in the design of the system.
At a higher level, the objectives are the conceptualization of the most important requests
of the users, the analysis of a model of their behavior, and the design of the architecture of
the system to include all of these elements. As claimed by Carroll, Mack, Robertson and
Rosson [63], OOD represents an important step forward in the study of software
technology, because it offers new opportunities and possibilities for change. They also
stress that the key idea for promoting this new form of development is to link object-
oriented design to the use of scenarios.
The technique of including scenarios as a support to the design of computing artifacts
takes the name of scenario-based design. In its original meaning, scenarios are stories that
describe people and their activities [64]. These descriptions, which refer to concrete
situations, offer information on behavior in the use of computerized systems: what the
people who interact with the system try to do, which procedures they adopt or do not
adopt, which tasks they complete successfully or which errors they make and what
interpretations they give to what happens to them during their interaction with the artifact
[65]. This method has recently been applied by a number of academics in the evaluation of
web pages.

120
Erskine, Carter-Tod and Burton [66], for example, conducted a study to evaluate the
usability of the web site of the College of Education at Virginia Tech. The methodology
adopted was based on the use of scenarios built by the users. The participants in the study,
who didn’t have much experience with the web, were selected as potential users of the site.
Initially, they were provided with information on the functional characteristics that
define the experience of the user and the importance of user input in the design process.
The scenario built by the participants was divided into three sections: context (e.g. the
working environment of the user), purpose (e.g. the information required) and action (the
imagined path of navigation). The users were asked to build some scenarios without
referring to the elements (e.g. the links) that were effectively present on the site. In this
way, the scenarios produced were authentic indicators of the information required by the
users, how they planned to use it and how it should be structured.
The evaluation of the site was performed by comparing the actions imagined in the
scenarios, which reflect the desires and the expectations of the users, with those effectively
feasible on the site. This comparison led to the acquisition of useful information for the re-
design of the site and the creation of specific guidelines concerning the hierarchical
organization of the information, the basic characteristics of the home page and the
characteristics of some specific contents.
Osterbauer, Köhle and Grechening [67] conducted a study on the usability of web sites
based on the use of scenarios. They selected fifteen sites from the following categories:
banks, newspapers and insurance companies. The thirty-five participants, potential users
with different levels of web experience, were divided into five groups. Each group was
asked to evaluate three sites, one per category.
The scenarios involved the simulation by the participants of real situations using the
site. These simulations were videotaped. At a later time, the subjects were asked to design,
based entirely on their memory, a map of the hierarchical structure of the site. The
procedure used allowed the measurement of the capacity of the users to develop an
understandable and aware image of the structure of the information presented. This
capacity is a fundamental parameter in evaluating the usability of sites with a complex
hierarchical structure.
Scholtz, Laskowski and Downey [68], on the other hand, evaluated the usability of the
National Institute of Standards and Technology virtual library site, the resources of which
are mostly available to the public. They collected 28 scenarios provided by the employees
of the library, which included, following a scheme supplied by the researchers: a general
description of the scenario, the benefits of the proposed operations, the beneficiaries of
such operations, the frequency and the estimated importance of the scenario, and any
negative aspects.
The results, which led to the identification of the aspects to be revised in the
organization of the library, were particularly important, given the creation of an actual
virtual participatory design meeting. The methodology of collecting the scenarios, in fact,
involved these being made available to all the participants, the use of e-mails to advise the
participants of the arrival of new scenarios, comments on the scenarios proposed by the
others, and indications on the use of the information collected for the re-design of the site.
One particular approach of scenario-based design, finally, was used by Neale and Kies
[69]. They analyzed a series of brainstorming sessions in which the usability of the site of
the Human Factors Engineering Center at Virginia Tech was evaluated. The groups
considered were made up of between two and five participants. The participants were
representatives of potential users. The brainstorming task involved the generation of
scenarios using the elements of a number of lists relating, for example, to the needs and the
objectives of the users, or to the information to be included on the site. Scenarios useful for

121
the evaluation of the usability of the site were obtained regardless of the background and
the expertise of the participants.

7.4 Conclusion

Some more general considerations allow us to understand the importance of Web usability
today, and define the role it plays or may play in the development of the Internet.
According to Riva and Galimberti [70] “…the technological evolution of the media
leads us to believe that Internet could become, in the very near future, the predominant
medium, or rather, it is possible that it will become a general communication interface..”. It
is clear that most of the interest in Internet as a medium is now focused on the World Wide
Web.
The Web, which may be described as the hypertext [71] and hypermedia part of the
Internet, today embraces new tools for group communication and navigation systems that
are able to combine the potential of the hypertext with the advantages of virtual graphic
environments [72]. The Web favours the exploration logic of the user, the multiplication of
the means of access to information [73], the creation of new work environments [74], and
the possibility, in the widest sense, to connect communities of practices so that they can
share their expertise [43]. These are the systems that usability must guarantee, refine and
make available to a growing public. In this way, it seems clear that the role of usability is
today expanding and undergoing complete transformation.
Having overcome the metaphor of “the impact” of technology on society, and moved in
the direction of the co-construction of social situations, the idea of web usability has
acquired importance. Usability, for some time now successfully experimented within the
context of interface problems, is no longer required to respond in a decontextualized
manner to aspects linked to software or hardware as ends in themselves, separate from the
rest of the world and in particular the context in which they are used. Motivated by the
imminent mass diffusion of so-called “mobile technology”, the interest in web usability is
focused on the combination of “real” and “virtual”, on new communication and work
environments, on the hybrid nature of these [35], on the intermittence with which the social
life of the individual is mediated by these technologies, on the cultural framework that
surrounds actions and makes them understandable to all, and on the complexity of the
cultural, political and economic accessibility of the Web within the various different
contexts and range of situations.

7.5 Acknowledgment

This research was supported by the European Community funded “Telemedicine and
Portable Virtual Environment for Clinical Psychology” VEPSY UPDATED - project
(IST-2000-25323; web sites: http://www.psicologia.net and http://www.vepsy.com).

7.6 References

[1] M.G Morris and A. P. Dillon, The importance of usability in the establishment of organizational
software standards for end user computing, International Journal of Human - Computer Studies 45
(1996) 243-258.
[2] D.G. Bobrow, R. Cheslow and J. Whalen, Eureka: using the web as a community knowledge medium.
Stanford University, Palo Alto. CA, 1999.

122
[3] G. Mantovani, L. Gamberini, M. Martinelli and D. Varotto. Exploring the suitability of a virtual
Environments for safety training: Signals, Norms and Ambiguity in a simulated virtual escape.
Cognition, Technology and Work. Pp. 1-9.
[4] M. Perkowitz and O. Etzioni, Towards adaptive web sites: Conceptual farmework and case study.
Artificial Intelligence 118 (2000) 245-275.
[5] P. Cottone, N. Zingigrian and M. Maresca, Reti di apprendimento asincrono in sussidio ai corsi
universitari. Didamatica 2001. In Press, 2001.
[6] F. Ossola, Gli infortuni domestici. Padova: Muzzio, 1989.
[7] D. A. Norman, Psychology of everyday things. Basic Book:New York, 1998.
[8] P. Jordan, S. W. Draper, K.K. MacFarlane and S.A. McNulty, Guessability, learnability and
experienced users performances, in People and Computers VI. D. Diapere and N. Hammond, Eds.
University Press: Cambridge, 1991.
[9] P. W. Jordan, What is usability? in Contemporary Ergonomics, S. Robertson, Ed. London:Taylor and
Francis, 1994, pp 454-458.
[10] P. W. Jordan, An Introduction to Usability. Taylor and Francis. London,1998.
[11] J. Nielsen, Usability Engineering at a Discount. In Designing and Using Human Computer Interfaces
and Knowledge Based System, G. Salvendy a M. J. Smith, Eds. Elsevier: Amsterdam, 1989.
[12] J. Nielsen, Usability Engineering. San Diego, CA, Academic Press, 1993.
[13] M. S. Sanders and E. J. McCormick, Human Factors in Engineering and Design. Singapore: McGraw-
Hill, 1993.
[14] J.R. Wilson and E.N. Corlett (eds.) Evaluation of Human Work. Taylor & Francis, London, 1995.
[15] J. Rasmussen, What can be learned from human errors reports? In Changes in working life, K. Duncan
et al., Eds. London : Wiley, 1980.
[16] J.T. Reason, Human Error. Cambridge: Cambridge University Press, 1990.
[17] S. Roncato, L’errore umano e i processi di comprensione, in Ergonomia. Lavoro, sicurezza e nuove
tecnologie, G. Mantovani, Ed. Bologna: Il Mulino, 2000, pp 41-80.
[18] J. Nielsen, The art of Navigating Through Hypertext, Communications of the ACM 33, 3 (1990) 296-
310.
[19] H. Shubin, D. Falck and A. G. Johansen, Exploring colors in interface designs, Interaction 3 (4) (1996)
36-48.
[20] J. Sklar, Principles of Web Design, Course Technology. Thomson Learning, 2000.
[21] S. McDonald and R.J. Stevenson, Disorientation in hypertext: the effects of the three text structures
un navigation performances, Applied Ergonomics 27, 1 (1996) 61-68.
[22] J. Larson and M. Czerwinski, Web Page Design: Implication of Memory, Structures and Scent for
information Retrieval, In Proceedings of the ACM CHI ’97 Conference: Human Factor in Computer
System (1997) 111-117.
[23] L.A. Whitaker, Human Navigation, in Human Factor and Web development, C., Forsythe et al., Eds.
Mahwah NJ, US: Lawrence Erlbaum Associates, 1998, pp. 63-71.
[24] J.E. Alexander and M. A. Tate, Web Wisdom. How to evaluate and create information quality on the
Web. Lawrence Erlbaum Associates, Mahwah, New Jersey, 1999.
[25] A. Cockburn and S. Jones, Which way now? Analysing and easing inadequacies in WWW navigation,
International Journals of Human-Computer Studies 45 (1996) 105-129.
[26] M. Zizi and M. Beaudouin-Lafon, Hypermedia exploration with interactive dynamic maps,
International Journal of Human Computer Studies 43, 3 (1995) 441-464.
[27] M. Head, N. Archer and Y. Yuan, World Wide Web Navigation Aid, International Journal of
Computer Studies 53, 2 (2000) 301-330.
[28] C. Campbell, Facilitating Navigation in Information Space: Road Signs on the World Wide Web,
International Journal of Human Computer Studies 50,4 (1999) 309-327.
[29] C. Sørensen, D. Macklin and T. Beaumont, Navigating the World Wide Web: bookmark maintenance
architectures, Interacting with computers 13 (2001) 375-400.
[30] A. Carassa, Expertise. La conoscenza entra in azione, in Ergonomia. Lavoro, sicurezza e nuove
tecnologie, G. Mantovani, Ed. Il Mulino, Bologna, 2000, pp. 123-150.
[31] W.J. Clancey, Situated cognition. On human Knowledge and computer representation, Cambridge:
Cambridge University Press, 1997.
[32] L. Suchman, Plans and situated action, Cambridge: Cambridge University Press, 1987.
[33] C.A. Hert, Information Retrieval as Situated Action, Proceedings of the Annual Meeting of the
American Society for Information Science 32 (1995) 172-180.
[34] J.E. Bardram, Plans and Situated Action: An Activity Theory Approach to Workflow Systems, In
Proceedings of ECSCW’97 Conference, Lancaster UK, September 1997, pp.17-32.
[35] G. Mantovani, Network. Reti elettroniche e reti di significato, in Ergonomia. Lavoro, sicurezza e
nuove tecnologie, G. Mantovani, Ed. Il Mulino, Bologna 2000, pp 153-177.
[36] G. Mantovani, La qualità dell’interazione uomo -computer, Il Mulino, Bologna, 2000.

123
[37] A. F. Blackwell and H.L. Arnold, Simulating a Software Project: The pop Guns go to War, In
Proceedings of the 9th Annual Meeting of the Psychology of Programming Interest Group, R. Osborn
and B. Khazei, Eds, 1997, pp. 53-60.
[38] J. Lave and E. Wenger, Situated learning. Legitimate peripheral participation, Cambridge, Cambridge
University Press, 1991.
[39] R. Pennell, M. Durham, C. Ozog and A. Spark, Writing in Context: Situated Learning on the Web. In
Proceedings of ASCILITE97, Perth Australia, 1997, pp 463-469.
[40] S. Henninger, A methodology and tools for applying context -specific usability guidelines to interface
design, Interacting with Computers, 12 (2000) 225-243.
[41] M. Cole, Cultural psychology: A once and future discipline. Cambridge, MA: Belknap Press of
Harvard University Press,1996.
[42] E. Hutchins, Cognition in the Wild, Cambridge, Mass, The MIT Press, 1995.
[43] G. Mantovani, The Psychological Construction of the Internet: From Information Foraging to Social
Gathering to Cultural Mediation, Cyberpsychology and Behavior, 4,1 (2001) 47-56.
[44] M.D. Cooper, Usage Patterns of a Web based library catalog, Journal of the American Society for
Information Science and Technology, (2001) pp.243-267
[45] R. Stout, Web Site Stts: Traking Hits and Analyzing traffic. Berkeley, CA. Ösborne/ McGraw-Hill,
1997.
[46] R. Fuller and J.J. de Graaff, Measuring User Motivation from Server Log Files, 1998. Internet
address:http://www.microsoft.com/usability/webconf/fuller/fuller.html
[47] L. Kantner and L. Rusinsky, Analizing Usability of a Beta-version Web Site Through Server Logs,
user Profile data, and Online Questionnaire Responses, In Proceedings of UPA 98, Washington, DC,
1998.
[48] R. Molich, and J. Nielsen, Improving a human-computer dialogue, Communications of the ACM 33, 3
(1990) 338-348.
[49] J. Nielsen and R. Molich, Heuristic evaluation of user interfaces, In Proceedings of ACM CHI'90
Conference, Seattle, WA, 1-5 April, 1990, pp 249-256.
[50] J. Nielsen, Enhancing the explanatory power of usability heuristics, In Proceedings of ACM CHI'94
Conference, Boston, MA, April 24-28, 1994, pp152-158.
[51] J. Nielsen, Heuristic evaluation, in Usability Inspection Methods, J. Nielsen and R.L. Mack, Eds. John
Wiley & Sons, New York, NY. , 1994.
[52] K. Instone, Site Usability Heuristics for the Web, 1997. Internet address
http://www.webreview.com/1997/10_10/strategists/10_10_97_2.shtml
[53] C. De Villiers, using HCI tecniques to evaluate Electronic Commerce sites, Proceedings at CHI-SA
2000, South African Human-Computer Interaction Conference, University of Pretoria Conference
Centre, South Africa, 2000.
[54] L. Lisle, J. Dong and S. Isensee, Case study of Development of an Ease of Use web site, Proceedings
of 4th Conference on Human factors and the Web., Basking ridge, NJ, Usa, 1998.
[55] R. G. Bias, The Pluralistic Usability Walkthrough: Coordinated Emphathies, in Usability Inspection
Methods, J. Nielsen, Jakob, and R. Mack, Eds. John Wiley and Sons, New York, NY, 1994.
[56] M. W. Vaughan and N. Schwartz, Jumpstarting the Information Design for a Community Network,
Journal of the American Society for Information Science, 50, 7 (1999) 588-597.
[57] W.P.Jr. Eveland and S. Dunwoody, Examining information processing on the World Wide Web using
think aloud protocol, Media-Psychology, 2 (3) (2000) 219-244.
[58] K.H. Lim, L.M.Ward and I. Benbasat, An empirical study of computer system learning: comparison
co-discovery and self-descovery methods, Information system research 8(3) (1997) 254-272.
[59] Hugh B and Hlozblatt K. Apprenticing with the Customer. Comm. ACM, 38(5), 45-52
[60] T. Lau and J. Stazcek, A Contextual Inquiry-Based Critique of the Strudel Web Site Maintenance
System, (1999). Internert address: http// www.citeseer.nj.nec.com/lau99contextual.html
[61] E.M. Ritchie and J.W. Gosbee, Design and Evaluation of a World Wide Web Site for Rural Health
Practioners, The Journal of the Healthcare Information and Management Systems Society, 11 (3)
(1997) 97-112.
[62] I.Jacobson, M. Christersson, P. Jonsson and G. Overgaard, Object-oriented Software engineering: A
Use case Diven Approach. Addison-Wesley. Reading, MA, 1992.
[63] J.M. Carroll, R.L. Mack, S.P Robertson, and M.B Rosson, Binding objects to scenarios of use,
International Journal – Computer Studies, 41 (1994) 243-276.
[64] J.M. Carroll, Scenario-based design: Envisioning Work and Technology in System Development.
Wiley, New York,1995
[65] J. M. Carroll, Five reasons for scenario-based design, Interacting with Comp uters,13 (2000) 43-60.
[66] L.E. Erskine, D.R.N. Carter-Tod and J.K Burton, Dialogical techniques for the design of web sites.
Accademic Press, 1997

124
[67] C. Osterbauer, M. Köhle and T. Grechening, Web Usability Testing. In Proceedings of AusWeb2K,
the Sixth Australian World Wide Web Conference, 12-17 (2000).
[68] J. Scholtz, S. Laskowski and L. Downey, Developing Usability Tools and tecnique for Designing and
testing Web Sites. In Proceedings of 4th Conference on Human factor & Web. Basking Ridge, NJ.
USA. (1998)
[69] D.C. Neale and J.K. Kies, User-Generated Scenarios for Requirements Specification and Design,
Human-Computer Interaction Laboratory, On-line Report (Usability Engineering) (1997).
[70] G. Riva and C. Galimberti Editorial, The Mind in the web: Psychology in the Internet age,
Cyberpsychology & Behavior, Special Issue. 4 (1) (2001) 5 –12.
[71] J.H. Ellsworth and M.V. Ellsworth Marketing on the Internet: multimedia strategies for the WWW. J.
Wiley. New York, 1995.
[72] L. Gamberini and S. Bussolon, Human Navigation in Electronic Environments. Cyberpsychology &
Behavior. 4 (1) (2001) 57-65.
[73] E. Rullani, Tecnologie che generano valore: divisione del lavoro cognitivo e rivoluzione digitale,
Economia e Politica industriale. 93 (1997) 76-87.
[74] G. Riva, I nuovi ambienti di lavoro. in Ergonomia. Lavoro, sicurezza e nuove tecnologie, G.
Mantovani, Ed. Il Mulino, Bologna, 2000, pp. 203-226.

125
 126

View publication stats