Key Concepts and Formulas

1. Matrix Transpose Properties: For matrices $M, N$ (of compatible dimensions) and a scalar $k$:
   • $(M+N)^T = M^T + N^T$
   • $(kM)^T = kM^T$
   • $(MN)^T = N^T M^T$ (Note the order reversal!)
   • $(M^k)^T = (M^T)^k$ for any positive integer $k$.
2. Symmetric and Skew-Symmetric Matrices:
   • A square matrix $M$ is symmetric if its transpose is equal to itself: $M^T = M$.
   • A square matrix $M$ is skew-symmetric if its transpose is equal to its negative: $M^T = -M$.
3. Determinant of a Skew-Symmetric Matrix: For an $n \times n$ skew-symmetric matrix $P$, if $n$ is odd, then its determinant $\det(P)$ is always $0$.
4. Homogeneous System of Linear Equations: A system of the form $PX = O$ (where $P$ is a square matrix, $X$ is a column vector of variables, and $O$ is a null column vector) always has at least one solution, the trivial solution $X=O$.
   • If $\det(P) \neq 0$, the system has a unique solution (which is the trivial solution $X=O$).
   • If $\det(P) = 0$, the system has infinitely many solutions (including the trivial solution). A homogeneous system can never have "no solution".

Step-by-Step Solution

Step 1: Identify the given information and the goal. We are given two $3 \times 3$ real matrices, $A$ and $B$.

• $A$ is symmetric: $A^T = A \quad \ldots(1)$
• $B$ is skew-symmetric: $B^T = -B \quad \ldots(2)$ We need to determine the nature of solutions for the homogeneous system of linear equations $(A^2 B^2 - B^2 A^2) X = O$. Let the coefficient matrix be $P = A^2 B^2 - B^2 A^2$. The system is $PX = O$. To find the nature of solutions, we must evaluate $\det(P)$.

Step 2: Determine the symmetry properties of $A^2$ and $B^2$. We will use the transpose property $(M^k)^T = (M^T)^k$ to find the transposes of $A^2$ and $B^2$.

• For $A^2$: $(A^2)^T = (A^T)^2$ Since $A$ is symmetric, we substitute $A^T = A$ from equation (1): $(A^2)^T = A^2$ This implies that $A^2$ is a symmetric matrix.

• For $B^2$: $(B^2)^T = (B^T)^2$ Since $B$ is skew-symmetric, we substitute $B^T = -B$ from equation (2): $(B^2)^T = (-B)^2 = (-1)^2 B^2 = B^2$ This implies that $B^2$ is also a symmetric matrix.

Step 3: Determine the symmetry property of the coefficient matrix $P = A^2 B^2 - B^2 A^2$. Now, we find the transpose of the matrix $P$. $P^T = (A^2 B^2 - B^2 A^2)^T$ Using the transpose property for sums/differences, $(M-N)^T = M^T - N^T$: $P^T = (A^2 B^2)^T - (B^2 A^2)^T$ Next, we use the transpose property for products, $(MN)^T = N^T M^T$: $P^T = (B^2)^T (A^2)^T - (A^2)^T (B^2)^T$ From Step 2, we know that $A^2$ is symmetric ($(A^2)^T = A^2$) and $B^2$ is symmetric ($(B^2)^T = B^2$). Substituting these into the expression for $P^T$: $P^T = B^2 A^2 - A^2 B^2$ Now, compare $P^T$ with the original matrix $P = A^2 B^2 - B^2 A^2$. We can factor out $-1$: $P^T = -(A^2 B^2 - B^2 A^2) = -P$ Since $P^T = -P$, the matrix $P$ is a skew-symmetric matrix.

Step 4: Calculate the determinant of $P$. The matrix $P$ is a $3 \times 3$ skew-symmetric matrix. According to the properties of determinants of skew-symmetric matrices, for an $n \times n$ skew-symmetric matrix, if $n$ is odd, its determinant is always zero. Here, the dimension $n=3$, which is an odd number. Therefore, $\det(P) = 0$.

Step 5: Determine the nature of solutions for the system $PX=O$. We have a homogeneous system of linear equations $PX=O$. From Step 4, we found that the determinant of the coefficient matrix $P$ is $\det(P) = 0$. For a homogeneous system, if the determinant of the coefficient matrix is zero, the system has infinitely many solutions. This set of solutions includes the trivial solution $X=O$ and an infinite number of non-trivial solutions.

Common Mistakes & Tips

• Homogeneous Systems and "No Solution": A homogeneous system $PX=O$ always has at least the trivial solution $X=O$. Therefore, "no solution" is never a possible answer for a homogeneous system.
• Transpose Order: Be meticulous with the order of matrices when taking the transpose of a product: $(MN)^T = N^T M^T$. A common error is to write $(MN)^T = M^T N^T$.
• Skew-Symmetric Determinants: Remember that the property $\det(M)=0$ for an $n \times n$ skew-symmetric matrix $M$ is true only when $n$ is odd. For even $n$, the determinant is generally non-zero.

Summary

By utilizing the properties of matrix transposes, we first established that both $A^2$ and $B^2$ are symmetric matrices. Subsequently, we applied transpose properties to the coefficient matrix $P = A^2 B^2 - B^2 A^2$, proving that $P$ is a skew-symmetric matrix. Since $P$ is a $3 \times 3$ matrix (an odd dimension), its determinant $\det(P)$ must be zero. For a homogeneous system of linear equations $PX=O$, a zero determinant implies that the system has infinitely many solutions.

The final answer is $\boxed{\text{infinitely many solutions}}$, which corresponds to option (C).