1. Key Concepts and Formulas

• Cayley-Hamilton Theorem: Every square matrix satisfies its own characteristic equation. If $A$ is an $n \times n$ matrix and its characteristic equation is $P(\lambda) = c_n \lambda^n + c_{n-1} \lambda^{n-1} + \dots + c_1 \lambda + c_0 = 0$, then $A$ satisfies $c_n A^n + c_{n-1} A^{n-1} + \dots + c_1 A + c_0 I = O$, where $I$ is the identity matrix and $O$ is the zero matrix of order $n \times n$.
• Characteristic Equation: For a square matrix $A$, its characteristic equation is given by $\det(A - \lambda I) = 0$, where $\lambda$ is a scalar variable and $I$ is the identity matrix of the same order as $A$. For a $3 \times 3$ matrix, this equation is of the form $\lambda^3 - \text{Tr}(A)\lambda^2 + (M_{11}+M_{22}+M_{33})\lambda - \det(A) = 0$, where $\text{Tr}(A)$ is the trace of $A$ and $M_{ii}$ are the principal minors of $A$.
• Determinant of a $3 \times 3$ Matrix: For a matrix $\left[\begin{array}{lll}p & q & r \\ s & t & u \\ v & w & x\end{array}\right]$, its determinant is $p(tx-wu) - q(sx-vu) + r(sw-vt)$.

2. Step-by-Step Solution

Step 1: Determine the Characteristic Equation of Matrix A

• Why this step? The Cayley-Hamilton Theorem requires us to first find the characteristic equation of the given matrix $A$. The given polynomial equation in $A$ implies that it must be the characteristic polynomial (since it's monic and of degree 3).
• Given matrix $A = \left[\begin{array}{lll}2 & a & 0 \\ 1 & 3 & 1 \\ 0 & 5 & b\end{array}\right]$.
• First, we form $A - \lambda I$: $A - \lambda I = \left[\begin{array}{ccc}2-\lambda & a & 0 \\ 1 & 3-\lambda & 1 \\ 0 & 5 & b-\lambda\end{array}\right]$
• Next, we calculate the determinant $\det(A - \lambda I) = 0$. We expand along the first row for simplicity: $\det(A - \lambda I) = (2-\lambda) \left| \begin{array}{cc} 3-\lambda & 1 \\ 5 & b-\lambda \end{array} \right| - a \left| \begin{array}{cc} 1 & 1 \\ 0 & b-\lambda \end{array} \right| + 0 \left| \begin{array}{cc} 1 & 3-\lambda \\ 0 & 5 \end{array} \right| = 0$
• Evaluate the $2 \times 2$ determinants: $(2-\lambda)[(3-\lambda)(b-\lambda) - 5 \times 1] - a[1 \times (b-\lambda) - 0 \times 1] = 0$ $(2-\lambda)[\lambda^2 - (3+b)\lambda + 3b - 5] - a(b-\lambda) = 0$
• Expand and collect terms by powers of $\lambda$: $2(\lambda^2 - (3+b)\lambda + 3b - 5) - \lambda(\lambda^2 - (3+b)\lambda + 3b - 5) - ab + a\lambda = 0$ $(2\lambda^2 - 2(3+b)\lambda + 2(3b-5)) - (\lambda^3 - (3+b)\lambda^2 + (3b-5)\lambda) - ab + a\lambda = 0$ $2\lambda^2 - (6+2b)\lambda + (6b-10) - \lambda^3 + (3+b)\lambda^2 - (3b-5)\lambda - ab + a\lambda = 0$
• Combine coefficients for $\lambda^3$, $\lambda^2$, $\lambda$, and the constant term: $-\lambda^3 + (2+3+b)\lambda^2 + (-6-2b - (3b-5) + a)\lambda + (6b-10-ab) = 0$ $-\lambda^3 + (5+b)\lambda^2 + (a-5b-1)\lambda + (6b-10-ab) = 0$
• Multiply by $-1$ to make the leading coefficient positive (standard form of characteristic equation): $\lambda^3 - (5+b)\lambda^2 - (a-5b-1)\lambda - (6b-10-ab) = 0$ $\lambda^3 - (5+b)\lambda^2 + (1-a+5b)\lambda + (10-6b+ab) = 0$ This is the characteristic equation of matrix $A$.

Step 2: Apply the Cayley-Hamilton Theorem and Compare with the Given Equation

• Why this step? According to the Cayley-Hamilton Theorem, matrix $A$ must satisfy its own characteristic equation. We can then compare this matrix polynomial with the given matrix polynomial to find $a$ and $b$.
• From the characteristic equation derived in Step 1, applying the Cayley-Hamilton Theorem gives: $A^3 - (5+b)A^2 + (1-a+5b)A + (10-6b+ab)I = O \quad \text{(Equation 1)}$ (Note: $O$ is the zero matrix of order $3 \times 3$)
• The given equation is: $A^3 = 4 A^2 - A - 21 I$
• Rearrange the given equation to match the form of Equation 1 (equal to $O$): $A^3 - 4 A^2 + A + 21 I = O \quad \text{(Equation 2)}$
• Now, we compare the coefficients of $A^2$, $A$, and $I$ from Equation 1 and Equation 2:
  1. Coefficient of $A^2$: From Equation 1: $-(5+b)$ From Equation 2: $-4$ Equating them: $-(5+b) = -4 \Rightarrow 5+b = 4 \Rightarrow \mathbf{b = -1}$
  2. Coefficient of $A$: From Equation 1: $(1-a+5b)$ From Equation 2: $1$ Equating them: $1-a+5b = 1 \Rightarrow -a+5b = 0 \Rightarrow \mathbf{a = 5b}$
  3. Constant term (Coefficient of $I$): From Equation 1: $(10-6b+ab)$ From Equation 2: $21$ Equating them: $\mathbf{10-6b+ab = 21}$

Step 3: Solve for $a$ and $b$

• Why this step? We have a system of equations involving $a$ and $b$. Solving this system will give us the required values.
• From the comparison of $A^2$ coefficients, we found $b = -1$.
• Substitute $b = -1$ into the equation from $A$ coefficients ($a = 5b$): $a = 5(-1) \Rightarrow \mathbf{a = -5}$
• Finally, verify these values using the equation from the constant term ($10-6b+ab = 21$): $10 - 6(-1) + (-5)(-1) = 10 + 6 + 5 = 21$ Since $21 = 21$, our values of $a=-5$ and $b=-1$ are consistent and correct.

Step 4: Calculate the Required Expression

• Why this step? The problem asks for the value of $2a+3b$.
• Substitute the values of $a=-5$ and $b=-1$ into the expression: $2a + 3b = 2(-5) + 3(-1) = -10 - 3 = \mathbf{-13}$

3. Common Mistakes & Tips

• Determinant Calculation: Be meticulous with algebraic expansions and signs when calculating the determinant. A small error here propagates through the entire problem.
• Cayley-Hamilton Application: Remember to replace $\lambda^k$ with $A^k$ and the constant term $\lambda^0$ with $I$ (the identity matrix) when translating the characteristic polynomial into a matrix polynomial.
• Coefficient Comparison: Ensure that both matrix equations are arranged in the same form (e.g., $P(A) = O$) before comparing coefficients of corresponding powers of $A$ and the constant identity matrix $I$.

4. Summary

This problem is a direct application of the Cayley-Hamilton Theorem. The process involves two main stages: first, deriving the characteristic equation of the given matrix $A$ in terms of $\lambda, a,$ and $b$; and second, using the Cayley-Hamilton Theorem to convert this into a matrix polynomial equation. By comparing the coefficients of this derived equation with the coefficients of the given matrix equation, we form a system of linear equations for $a$ and $b$. Solving this system yields $a=-5$ and $b=-1$, which allows us to calculate the final expression $2a+3b$.

The final answer is $\boxed{-13}$, which corresponds to option (C).