You failed to involve enough users in your evaluation. Now what?

You can still derive some insights from the data you collected

What are H1, H2 and H3 in this picture of the TAM model?

Hypotheses of causality between factors

Say you want to evaluate your system usability. The first thing you do is...

Look for a pre-existing usability test

So, in the end, what was this "User-Centered Evaluation" pill/lecture?

A sort of user evaluation study

What is important to take into account in a crowdsourcing campaign?

Crowd's demographics (e.g. age, gender, language, ...), Crowd's potential cultural bias, Crowd's background knowledge about the evaluation topic

OK, you got it! So, in A/B testing you will test...

Two system variants with two similar but distinct groups of users

And how many users do you involve in a qualitative evaluation study?

Nah, 4 or 5 users in total are more than enough

What's best: qualitative or quantitative user evaluation?

Both qualitative and quantitative would be the best!

Ok, the number of required evaluators depends... on what?

The size of the target user population

What should NOT be changed in a standardized user evaluation questionnaire?

The formulation of the items/questions in the questionnaire, The minimum number of factors/dimensions of the evaluation

What are A and B in A/B testing?

Two variants of the same evaluated variable

What do you do in a "thinking out loud" method?

You ask the evaluators to describe what they do and think during evaluation

When can you run an evaluation campaign, say of a software tool?

A drawing on a napkip could be enough to evaluate

What is the evaluation protocol?

The set of evaluation questions you ask to participants, The instructions for evaluators to execute the required tasks

Now, say you need around 100 evaluators. Therefore you involve...

100 representative users from the target population

studbud

ISWS 2022 - User Centered Evaluation

20 termsStudBud Administration69 plays

quiz-based lesson for the 2022 edition of the International Semantic Web Research Summer School

Flashcards20 cardsFPractice quizPHost liveL

Questions 20

Answers

1
A famous user evaluation method is TAM. What does TAM stand for?
Theory Application Methodology
Terminology and Axioms Method
Technology Acceptance Model
Total Addressable Market
2
You failed to involve enough users in your evaluation. Now what?
You can still derive some insights from the data you collected
You have to keep going, even for months, until you reach the right number
You are doomed, better give up your PhD
You cannot include your evaluation in a paper
3
What are H1, H2 and H3 in this picture of the TAM model?
Order of Importance of factors
Hypotheses of correlation between factors
Hypotheses of causality between factors
Probabilities of co-occurrence between factors
4
Say you want to evaluate your system usability. The first thing you do is...
Re-define usability in your context
Use your system multiple times to correct bugs
Make sure your system is usable
Look for a pre-existing usability test
5
So, in the end, what was this "User-Centered Evaluation" pill/lecture?
A complete waste of my time
Just a gamified lecture
A sort of user evaluation study
A proper user evaluation study
6
What is important to take into account in a crowdsourcing campaign?
Crowd's demographics (e.g. age, gender, language, ...)
Crowd's potential cultural bias
Crowd's payment expectations
Crowd's background knowledge about the evaluation topic
7
OK, you got it! So, in A/B testing you will test...
Two system variants with two groups of users with different characteristics
Two system variants with two similar but distinct groups of users
Two system variants with the same group of users
Two groups of users with the same system variant
8
Warm-up question: in which Italian region are we now?
Sicily
Tuscany
Algarve
Emilia-Romagna
9
And how many users do you involve in a qualitative evaluation study?
At least 10% w.r.t. quantitative evaluation
3 evaluators per each "category" of users in the target population
Exactly the same as quantitative evaluation
Nah, 4 or 5 users in total are more than enough
10
What's best: qualitative or quantitative user evaluation?
Neither of them: I hate people
Always quantitative: numbers don't lie
Always qualitative: humanity is not explained by numbers
Both qualitative and quantitative would be the best!
11
Ok, the number of required evaluators depends... on what?
The number of people I know
The size of the target user population
The complexity of the evaluation task
The money I can invest to pay evaluators
12
What is crowdsourcing?
Making friends on the Web
Following celebrities on the web
Exploiting people on the Web
Cheating people on the web
13
What should NOT be changed in a standardized user evaluation questionnaire?
The formulation of the items/questions in the questionnaire
The maximum number of factors/dimensions of the evaluation
The order of the items/questions in the questionnaire
The minimum number of factors/dimensions of the evaluation
14
What are A and B in A/B testing?
Two runs of the same testing procedure
Two variants of the same evaluated variable
Two participants to the same study
Two possible outcomes of the same experiment
15
Who's your best friend for user evaluation analysis?
A philosopher
A rich sponsor
A statistician
An anthropologist
16
What do you do in a "thinking out loud" method?
You shout out to evaluators to hurry up
You let the evaluators scream to release tension before the evaluation
You clearly express your opinion on why the evaluated system is beautiful
You ask the evaluators to describe what they do and think during evaluation
17
When can you run an evaluation campaign, say of a software tool?
Only when the main features are implemented
A drawing on a napkip could be enough to evaluate
Only when all bugs are solved
Only when the user interface design is finished
18
What is the evaluation protocol?
The set of evaluation questions you ask to participants
The permission from the ethics committee to run the evaluation
The official paper on which you report the evaluation results
The instructions for evaluators to execute the required tasks
19
How many users do you need to involve in a user evaluation study?
It depends
30
300
3,000
20
Now, say you need around 100 evaluators. Therefore you involve...
100 friends from the target population
The first 100 people available
Ignore statistics, only the people I can get
100 representative users from the target population

ISWS 2022 - User Centered Evaluation Flashcards | StudBud