Isn't the question about the scatterplots somewhat misleading in itself? By using nonsense variables X and Y, don't you leave out the most important aspects of the question?

If X and Y are variables which are logically connected, but for which statistical testing shows no significant result. Might these variables not be likely to be connected to other variables which if included in the model m...

In a paper about correlations we show that many trained scientist failed to see problems in scatter plots but argue that what really matters is to establish the strength of a correlation (e interpret the r^2 and not the prob. under the null) -- I don't really understand how one could ever appreciate what could be the p-value r...

Since the paper came out, some in the academic community have pointed out that range for our plot axes were not fixed (e.g. Figure 1). This added difference in visual stimulus might have affected user’s perceptions, and made our significant v. nonsignificant plots not appropriately controlled and comparable.

The analysis uses lme4, but only includes subject intercepts, not plot intercepts. Standard recommendations for analysis of subject-item data like this would suggest anticonservatism can arise from not accounting for the stimulus sampling involved. This could also provide some insight about whether there were particular plots that fooled students.

From a quick glance at the code (great that it is available, but it could perhaps use some nice whitespace), it looks like the p-values are computed from a linear model and assuming normality and homoscedasticity (for them to be correct in small samples at least). Without each of these assumptions (normality, homoscedastic errors, and normal errors), how many of the "positive" cases remain so? For...

Answered below