Key Concepts and Properties of Determinants

This problem relies heavily on understanding and correctly applying fundamental properties of determinants for invertible matrices. Let $X$ and $Y$ be square matrices of the same order, and $c$ be a scalar. The key properties we will use are:

1. Determinant of a Product: $\det(XY) = \det(X) \det(Y)$. This property allows us to break down the determinant of a product of matrices into the product of their individual determinants.
2. Determinant of a Transpose: $\det(X^T) = \det(X)$. The determinant of a matrix remains unchanged upon taking its transpose.
3. Determinant of an Inverse: $\det(X^{-1}) = \frac{1}{\det(X)}$. Since A and B are invertible, their determinants are non-zero, making this property well-defined.

These properties are essential for simplifying complex determinant expressions and are frequently tested in JEE.

Step-by-Step Solution

Let's denote $\det(A)$ as $|A|$ and $\det(B)$ as $|B|$ for brevity.

Step 1: Simplify the first given condition We are given $\det(ABA^T) = 8$. Applying the determinant product property, $\det(ABA^T) = \det(A) \det(B) \det(A^T)$. Next, using the determinant of a transpose property, $\det(A^T) = \det(A)$. Substituting this, we get: $\det(A) \det(B) \det(A) = 8$ $(\det(A))^2 \det(B) = 8$ So, we have our first equation: $|A|^2 |B| = 8 \quad \text{(Equation 1)}$

Step 2: Simplify the second given condition We are given $\det(AB^{-1}) = 8$. Applying the determinant product property, $\det(AB^{-1}) = \det(A) \det(B^{-1})$. Next, using the determinant of an inverse property, $\det(B^{-1}) = \frac{1}{\det(B)}$. Substituting this, we get: $\det(A) \frac{1}{\det(B)} = 8$ So, we have our second equation: $\frac{|A|}{|B|} = 8 \quad \text{(Equation 2)}$

Step 3: Solve the system of equations for $|A|$ and $|B|$ We now have a system of two equations with two