Key Concepts and Formulas

To solve this problem, we will utilize the following fundamental concepts from 3D Geometry:

1. Direction Vector of the Line of Intersection of Two Planes: A line formed by the intersection of two planes is perpendicular to the normal vectors of both planes. If the normal vectors of the two planes are $\vec{n_1}$ and $\vec{n_2}$, respectively, then the direction vector $\vec{d}$ of their line of intersection can be found by taking their cross product: $\vec{d} = \vec{n_1} \times \vec{n_2}$
2. Angle between a Line and a Plane: If $\vec{d} = (l, m, n)$ is the direction vector of a line and $\vec{n} = (A, B, C)$ is the normal vector of a plane, and $\theta$ is the angle between the line and the plane, then the sine of this angle is given by: $\sin \theta = \frac{|\vec{d} \cdot \vec{n}|}{||\vec{d}|| \cdot ||\vec{n}||}$ Here, $||\vec{d}||$ and $||\vec{n}||$ represent the magnitudes of vectors $\vec{d}$ and $\vec{n}$, respectively. The absolute value ensures that the angle $\theta$ is taken as acute ($0^\circ \le \theta \le 90^\circ$).
3. Direction Cosines: For a line with direction vector $\vec{d} = (L, M, N)$, its direction cosines are given by $\left(\frac{L}{||\vec{d}||}, \frac{M}{||\vec{d}||}, \frac{N}{||\vec{d}||}\right)$. These values represent the cosines of the angles the line makes with the positive x, y, and z axes. Note that if $(l,m,n)$ are direction cosines, then $(-l,-m,-n)$ are also valid direction cosines for the same line.

---

Step-by-Step Solution

Step 1: Identify the Normal Vectors of the Given Planes

We are given three planes. To find their normal vectors, we write each plane equation in the standard form $Ax+By+Cz+D=0$, where the normal vector is $\vec{n}=(A, B, C)$.

• Plane 1 ($P_1$): $ax+by=3$. This can be written as $ax+by+0z=3$. The normal vector for Plane 1 is $\vec{n_1} = (a, b, 0)$.
• Plane 2 ($P_2$): $ax+by+cz=0$. The normal vector for Plane 2 is $\vec{n_2} = (a, b, c)$.
• Plane 3 ($P_3$): $y-z+2=0$. This can be written as $0x+1y-1z+2=0$. The normal vector for Plane 3 is $\vec{n_3} = (0, 1, -1)$.

Step 2: Determine the Direction Vector of the Line of Intersection

The line of intersection of Plane 1 and Plane 2 is perpendicular to both $\vec{n_1}$ and $\vec{n_2}$. Therefore, its direction vector, $\vec{d}$, is found by taking the cross product of these two normal vectors: $\vec{d} = \vec{n_1} \times \vec{n_2}$ $\vec{d} = (a, b, 0) \times (a, b, c)$ We calculate the cross product using the determinant form: $\vec{d} = \begin{vmatrix} \mathbf{i} & \mathbf{j} & \mathbf{k} \\ a & b & 0 \\ a & b & c \end{vmatrix}$ $\vec{d} = \mathbf{i}(bc - 0) - \mathbf{j}(ac - 0) + \mathbf{k}(ab - ab)$ $\vec{d} = bc \mathbf{i} - ac \mathbf{j} + 0 \mathbf{k}$ So, the direction vector of the line of intersection is $\vec{d} = (bc, -ac, 0)$.

Step 3: Apply the Angle Condition between the Line and Plane 3

We are given that the line of intersection makes an angle of $\theta = 30^\circ$ with Plane 3. We use the formula for the angle between a line and a plane: $\sin \theta = \frac{|\vec{d} \cdot \vec{n_3}|}{||\vec{d}|| \cdot ||\vec{n_3}||}$ We have $\vec{d} = (bc, -ac, 0)$ and $\vec{n_3} = (0, 1, -1)$.

First, calculate the dot product $\vec{d} \cdot \vec{n_3}$: $\vec{d} \cdot \vec{n_3} = (bc)(0) + (-ac)(1) + (0)(-1) = 0 - ac + 0 = -ac$

Next, calculate the magnitudes $||\vec{d}||$ and $||\vec{n_3}||$: $||\vec{d}|| = \sqrt{(bc)^2 + (-ac)^2 + 0^2} = \sqrt{b^2c^2 + a^2c^2} = \sqrt{c^2(a^2+b^2)} = |c|\sqrt{a^2+b^2}$ (We assume $c \ne 0$, otherwise $P_1$ and $P_2$ would be parallel or identical, not forming a unique line of intersection.) $||\vec{n_3}|| = \sqrt{0^2 + 1^2 + (-1)^2} = \sqrt{0 + 1 + 1} = \sqrt{2}$

Now, substitute these values into the angle formula with $\sin 30^\circ = \frac{1}{2}$: $\frac{1}{2} = \frac{|-ac|}{|c|\sqrt{a^2+b^2} \cdot \sqrt{2}}$ Since $|-ac| = |a||c|$ and we are given $a>0$, we have $|a|=a$. We can cancel $|c|$ from the numerator and denominator (as $c \ne 0$): $\frac{1}{2} = \frac{a}{\sqrt{a^2+b^2} \cdot \sqrt{2}}$ Rearrange the equation to solve for $a^2+b^2$: $\sqrt{a^2+b^2} = \frac{2a}{\sqrt{2}} = a\sqrt{2}$ Square both sides of the equation: $a^2+b^2 = (a\sqrt{2})^2$ $a^2+b^2 = 2a^2$ $b^2 = a^2$ This implies that $b=a$ or $b=-a$.

Step 4: Determine the Direction Cosines of the Line

The direction vector of the line is $\vec{d} = (bc, -ac, 0)$. Its magnitude is $||\vec{d}|| = |c|\sqrt{a^2+b^2}$. From Step 3, we found $a^2+b^2 = 2a^2$. Since $a>0$, $\sqrt{a^2+b^2} = \sqrt{2a^2} = a\sqrt{2}$. Therefore, the magnitude of the direction vector is $||\vec{d}|| = |c|a\sqrt{2}$.

The direction cosines are given by $\left(\frac{bc}{||\vec{d}||}, \frac{-ac}{||\vec{d}||}, \frac{0}{||\vec{d}||}\right)$: $\left(\frac{bc}{|c|a\sqrt{2}}, \frac{-ac}{|c|a\sqrt{2}}, \frac{0}{|c|a\sqrt{2}}\right)$ Let's simplify these expressions. We can write $\frac{c}{|c|}$ as $\text{sgn}(c)$, which is $1$ if $c>0$ and $-1$ if $c<0$. $\left(\frac{b}{a\sqrt{2}} \cdot \text{sgn}(c), -\frac{1}{\sqrt{2}} \cdot \text{sgn}(c), 0\right)$ Now we consider the two cases for $b$:

• Case 1: $b=a$ Substitute $b=a$ into the direction cosines: $\left(\frac{a}{a\sqrt{2}} \text{sgn}(c), -\frac{1}{\sqrt{2}} \text{sgn}(c), 0\right) = \left(\frac{1}{\sqrt{2}} \text{sgn}(c), -\frac{1}{\sqrt{2}} \text{sgn}(c), 0\right)$ If $c>0$, the direction cosines are $\left(\frac{1}{\sqrt{2}}, -\frac{1}{\sqrt{2}}, 0\right)$. If $c<0$, the direction cosines are $\left(-\frac{1}{\sqrt{2}}, \frac{1}{\sqrt{2}}, 0\right)$.

• Case 2: $b=-a$ Substitute $b=-a$ into the direction cosines: $\left(\frac{-a}{a\sqrt{2}} \text{sgn}(c), -\frac{1}{\sqrt{2}} \text{sgn}(c), 0\right) = \left(-\frac{1}{\sqrt{2}} \text{sgn}(c), -\frac{1}{\sqrt{2}} \text{sgn}(c), 0\right)$ If $c>0$, the direction cosines are $\left(-\frac{1}{\sqrt{2}}, -\frac{1}{\sqrt{2}}, 0\right)$. If $c<0$, the direction cosines are $\left(\frac{1}{\sqrt{2}}, \frac{1}{\sqrt{2}}, 0\right)$.

Comparing these possible sets of direction cosines with the given options: Option (A) is $\left(\frac{1}{\sqrt{2}}, \frac{1}{\sqrt{2}}, 0\right)$. This set of direction cosines matches the result from Case 2 when $b=-a$ and $c<0$. Since direction cosines $(l,m,n)$ and $(-l,-m,-n)$ both represent the same line, any of these possibilities is a valid answer.

---

Common Mistakes & Tips

• Missing Coefficients: When identifying normal vectors, remember to include zero coefficients for missing variables. For example, $ax+by=3$ means $ax+by+0z=3$.
• Absolute Value in Angle Formula: Always use the absolute value in the numerator $|\vec{d} \cdot \vec{n}|$ when calculating the angle between a line and a plane, as the angle is conventionally defined as acute.
• Sign Considerations: Be careful with signs when taking square roots (e.g., $\sqrt{X^2} = |X|$). The condition $a>0$ is important for $\sqrt{a^2}=a$.
• Direction Cosines vs. Ratios: Remember that direction cosines are normalized direction ratios. If $(L, M, N)$ are direction ratios, then $\left(\frac{L}{\sqrt{L^2+M^2+N^2}}, \frac{M}{\sqrt{L^2+M^2+N^2}}, \frac{N}{\sqrt{L^2+M^2+N^2}}\right)$ are the direction cosines. Also, $(l,m,n)$ and $(-l,-m,-n)$ represent the same line.

---

Summary

This problem required us to first determine the direction vector of the line of intersection of two planes by taking the cross product of their normal vectors. Then, we used the formula for the angle between a line and a plane, which involves the dot product of the line's direction vector and the plane's normal vector, along with their magnitudes. By setting the angle to $30^\circ$, we established a relationship between the coefficients $a$ and $b$. Finally, we used this relationship to find the direction cosines of the line, considering all possible signs for $b$ and $c$, and matched one of the resulting sets with the given options.

The final answer is $\boxed{\text{(A)}}$.