When training requires that only two categories be identified, then the classifier (and therefore the organism) need only identify some difference that distinguishes them, and nothing more. The result is a tailor-made classifier: Tailored by the specifics of the binary training paradigm, and optimized solely for that dichotomous discrimination.
As a psychologist I use binary classification tasks quite often. My approach (and the approach that I recommend) is to let SVM algorithm do the classification. If it performs above chance then I look at the minima and maxima (i.e. the typical TRUE or FALSE stimulus) of the SVM classifier. If I find that the SVM exploits some stimulus ideosynchracies (what you call "specifics of the binary training paradigm") I improve the stimulus. I iterate this process until I obtain stimuli where SVM performs at chance. Then I conclude that there can't be any "tailor-made classifier" as SVM - a state of the art classifier - doesn't find any.
However, it is difficult to assess what proportion of correct responses are genuine classifications and what proportion are merely lucky guesses. Accuracy of 70% on a binary test could mean that the subject is guessing at random more than half the time.
The usual way to asses this is to compute the (say 95 %) confidence interval of accuracy with binomial distribution. Then you get statement like "accuracy is 70 [40-90] %" or "accuracy is 70 [69-71] %" (in "x [l,u]" x is the mean, l is the lower bound and u is the upper bound) . The latter case excludes the possibility of guessing.
Regarding your simulations, I assume that the airplane or motorbike images also had background. Then the category "background" overlaps with other categories and the poor performance of background category is not surprising. I assume behavioral experiments would not use overlapping categories. If we exclude the "background" category, I do not think that the results in figure 1 left support your conclusion.
You can also choose to receive updates via daily or weekly email digests. If you are following multiple preprints then we will send you no more than one email per day or week based on your preferences.
Note: You are now also subscribed to the subject areas of this preprint and will receive updates in the daily or weekly email digests if turned on. You can add specific subject areas through your profile settings.
Usage since published - updated daily