Comments on: Embracing the Un-Science of Qualitative Research Part Two - Ever-Evolving Prototypes are Ace http://www.disambiguity.com/embracing-the-un-science-of-qualitative-research-part-two-ever-evolving-prototypes-are-ace/ pretty design pending Thu, 20 Nov 2008 07:50:26 +0000 http://wordpress.org/?v=2.6.2 By: Kristen http://www.disambiguity.com/embracing-the-un-science-of-qualitative-research-part-two-ever-evolving-prototypes-are-ace/#comment-27375 Kristen Fri, 03 Aug 2007 19:02:46 +0000 http://www.disambiguity.com/embracing-the-un-science-of-qualitative-research-part-two-ever-evolving-prototypes-are-ace/#comment-27375 I agree with this 3 part series and look forward to the last part. But my question is, what about generalizations? Having rich data from a few participants is great, but what works for one doesn't always work for another. Can you then make generalizations? And if so, how can you be sure you're adequately representing the participant population? And, in a formal environment, how can you justify your generalizations? I agree with this 3 part series and look forward to the last part. But my question is, what about generalizations? Having rich data from a few participants is great, but what works for one doesn’t always work for another. Can you then make generalizations? And if so, how can you be sure you’re adequately representing the participant population? And, in a formal environment, how can you justify your generalizations?

]]> By: Embracing the Un-Science of Qualitative Research « ilab http://www.disambiguity.com/embracing-the-un-science-of-qualitative-research-part-two-ever-evolving-prototypes-are-ace/#comment-26595 Embracing the Un-Science of Qualitative Research « ilab Sun, 29 Jul 2007 21:25:17 +0000 http://www.disambiguity.com/embracing-the-un-science-of-qualitative-research-part-two-ever-evolving-prototypes-are-ace/#comment-26595 [...] Embracing the Un-Science of Qualitative Research Part Two - Ever-Evolving Prototypes are Ace One of the fundamentals of quantitative research is its systematic nature. It’s about measuring stuff. And, you don’t want that stuff to change as you’re measuring it for a number of reasons—not the least of which being that it makes it very difficult to plot on a graph [...] [...] Embracing the Un-Science of Qualitative Research Part Two - Ever-Evolving Prototypes are Ace One of the fundamentals of quantitative research is its systematic nature. It’s about measuring stuff. And, you don’t want that stuff to change as you’re measuring it for a number of reasons—not the least of which being that it makes it very difficult to plot on a graph [...]

]]> By: Qualitative Research » UIE Brain Sparks http://www.disambiguity.com/embracing-the-un-science-of-qualitative-research-part-two-ever-evolving-prototypes-are-ace/#comment-25939 Qualitative Research » UIE Brain Sparks Wed, 25 Jul 2007 17:05:22 +0000 http://www.disambiguity.com/embracing-the-un-science-of-qualitative-research-part-two-ever-evolving-prototypes-are-ace/#comment-25939 [...] You can read part 2 here: Embracing the Un-Science of Qualitative Research Part Two - Ever-Evolving Prototypes are Ace Part 3 is on the way. [...] [...] You can read part 2 here: Embracing the Un-Science of Qualitative Research Part Two - Ever-Evolving Prototypes are Ace Part 3 is on the way. [...]

]]> By: Michael Clarke http://www.disambiguity.com/embracing-the-un-science-of-qualitative-research-part-two-ever-evolving-prototypes-are-ace/#comment-24818 Michael Clarke Tue, 17 Jul 2007 15:45:24 +0000 http://www.disambiguity.com/embracing-the-un-science-of-qualitative-research-part-two-ever-evolving-prototypes-are-ace/#comment-24818 Ever-evolving prototype - I do like that. And if you're trying to hit a tough deadline, it's the only way to go. Ever-evolving prototype - I do like that. And if you’re trying to hit a tough deadline, it’s the only way to go.

]]> By: Lance http://www.disambiguity.com/embracing-the-un-science-of-qualitative-research-part-two-ever-evolving-prototypes-are-ace/#comment-24653 Lance Mon, 16 Jul 2007 15:34:46 +0000 http://www.disambiguity.com/embracing-the-un-science-of-qualitative-research-part-two-ever-evolving-prototypes-are-ace/#comment-24653 I like your general take on this but have a comment on "validation." I think you don't need statistical validity with observation-based testing because the validation is really the expertise of the expert. In other words, you observe something that is, like you said, a "big" or "obvious" issue. That assessment validates the finding. However, with A/B testing, there's a big caveat: are you measuring preference or performance? In other words, are you banking on what people say they like or say they'd do? Or are you measuring what you observed them doing, and are comfortable that that's what they'd do independent of any test effect? I've always found that hard to achieve, and have generally avoided using 121 testing to find preference for one design/solution over another. I like your general take on this but have a comment on “validation.”

I think you don’t need statistical validity with observation-based testing because the validation is really the expertise of the expert. In other words, you observe something that is, like you said, a “big” or “obvious” issue. That assessment validates the finding.

However, with A/B testing, there’s a big caveat: are you measuring preference or performance? In other words, are you banking on what people say they like or say they’d do? Or are you measuring what you observed them doing, and are comfortable that that’s what they’d do independent of any test effect?

I’ve always found that hard to achieve, and have generally avoided using 121 testing to find preference for one design/solution over another.

]]>