returns matrix resulting from matrix multiplication of transposed matrix mat with itself, aka Gram matrix.