Cochran’s Q assesses treatment differences in randomized block designs with binary data. We suggest using bootstrap p-values rather than p-values based on the chi-squared distribution for tests based on Q. These chi-squared p-values for Q are the only ones usually given in statistical software and can be inaccurate. The same approach allows improved p-values to be given for sparse two-way cross-classification data.