In 1904, a horse named Clever Hans caused a huge commotion in German society. Hans, a horse of the Orlov Trotter, could add, subtract, multiply, divide, work with fractions, tell time, use a calendar, and read, spell, and understand German. Not inconsiderable skills for a horse, even for a German horse.

These skills did not go unnoticed. In 1907, after much pressure from the media, the German Ministry of Education appointed a commission of thirteen people, with philosopher and psychologist Carl Stumpf at the helm. Its mission was to clarify the famous horse’s real abilities. They were known as the “Hans Commission”.

Hans’ owner was math teacher and horse trainer Wilhelm von Osten. Mr. Osten bragged about having taught his exceptional horse all these skills. What type of questions could Hans answer? This one, for example:

“If the eighth day of the month comes on a Tuesday, what is the date of the following Friday?”

(Clever Hans) Wikipedia

Patiently, Hans responded by tapping the correct answer on the floor. The questions were asked orally and in writing. Mr. Osten never accepted money for showing Hans’ qualities in public. You see, he didn’t want to become rich off this prodigy of nature.

The “Hans Commission´s” was that there was no hoax or trick in the horse’s performance. When Carl Stumpf read his team’s conclusions, he could not believe it. He reviewed again and again the methodology used to arrive at such a surprising conclusion:

1. The horse had been isolated from spectators, so there would be no cues.
2. The questions were asked by people other than Osten.
3. The questioners sometimes knew the answers beforehand, and sometimes they did not.
4. Using blinders, sometimes the horse could see the questioner, sometimes he could not.

Source: wikipedia.com – Clever Hans

Stumpf went over all the data, reports and notes. Furthermore, he carried out new and detailed tests on the horse, its owner and its team. . . Finally, Stumpf deciphered the huge mystery.

First, it was determined that Hans was able to know the correct answer, even when his owner was not the questioner. Thus, fraud was ruled out of the equation. Mr Osten was a good man who truly believed in his horse’s abilities. The solution is surprising even today. Stumpf demonstrated that the horse only gave the right answer if the questioner knew the answer and also the horse had direct visual contact with the questioner or with his owner (who, being a mathematician, calculated the correct answer).

Clever Hans was responding involuntarily to the body language of his owner or the questioner who knew the answer. Neither his owner nor the participants on the team were aware that they were giving any clues to the horse to solve the problem. A small muscle movement in their faces, a slight shift caused by the tension of knowing that the horse was close to the correct answer, informed Clever Hans that he should stop tapping on the ground.

This case gives its name to the effect known as “The Clever Hans effect” or “Experimenter´s bias”. It is a subjective bias towards a result that is expected or desired by the experimenter. The important thing is that it happens unconsciously on the part of the researcher.

A Personal Experience with Guerrilla Testing on Users

I’m not an expert on preparing tests on users, but for reasons that are beside the point now, I have carried out a few tests for a personal project. I thoroughly studied the book by Steve Krug: “Rocket Surgery Made Easy: The Do-It-Yourself Guide to Finding and Fixing Usability Problems””—a fantastic book and a fast read.

I carried out all the tests I had to. All the responses reaffirmed my assumptions and hypotheses. When I had finished the whole process, and after a weekend of getting away, upon listening to some of the interviews again, there was something that did not fit. Finally, I had the same revelation that professor Stumpf had: I had screwed up!!!

First mistake. I carried out the tests with friends and family. They knew what I was working on. In their responses, they didn’t tell me the truth. They wanted to please me and to encourage me in the new project. Moreover, in the initial discussion, I asked questions which contained statements that clearly influenced them. An example:

“Most people I have talked to previously would be willing to use this service. Would you also be interested in contracting it? My plan is to charge XX for this service, and the response has been positive in most cases. What do you think: is it expensive or cheap? Would you pay that amount?”

Rookie mistakes. They may be called guerrilla tests, but when you prepare them you must be as careful and methodical as with any other type of test. Also, leave friends and family who know you or know something about your project out of the study. It is more work at the beginning, but you will certainly spare yourself having to repeat tests—which is exactly what I have to do