Well, I'm struggling with this, and thought that posting what I'm doing could help.

I have the transport equation:

$\displaystyle \hat \Omega \cdot \nabla I(\mathbf{r},\hat \Omega)+\mu_t I(\mathbf{r},\hat \Omega)-\mu_s\int d\hat \Omega' p(\hat \Omega \cdot \hat \Omega') I(\mathbf{r},\hat \Omega')=0$

with prescribed boundary conditions on the inner surface of the domain $\displaystyle V$ with boundary $\displaystyle \partial V$: $\displaystyle I(\mathbf{r},\hat \Omega)=f(\hat \Omega_r \cdot \hat n)I(\mathbf{r},\hat \Omega_r)$ for $\displaystyle (\mathbf{r},\hat \Omega) \in \Gamma_{-}$, $\displaystyle \Gamma_{\pm}=\{(\mathbf{r},\hat \Omega)| \mathbf{r}\in \partial V, \hat \Omega \cdot n \gtrless 0 \}$

Here $\displaystyle \hat n=\hat n(\mathbf{r})$ is the normal to the surface at the boundary, and $\displaystyle f(\hat \Omega' \cdot \hat n)$ is the Fresnel coefficient. This are Fresnel boundary conditions. Part of the radiation is reflected at the boundary according to this boundary condition, and $\displaystyle \hat \Omega_r=\hat R \hat \Omega$, being $\displaystyle \hat R=\hat R(\hat n) $ a reflection operator

I want to derive the adjoint equation for this problem, i.e., if I define the transport operator

$\displaystyle \mathcal{T}I=\hat \Omega \cdot \nabla I(\mathbf{r},\hat \Omega)+\mu_t I(\mathbf{r},\hat \Omega)-\mu_s\int d\hat \Omega' p(\hat \Omega \cdot \hat \Omega') I(\mathbf{r},\hat \Omega')$,

Then I should have for the adjoint operator that:

$\displaystyle \langle \psi, \mathcal{T}I \rangle=\langle I, \mathcal{T}^{\dagger} \psi \rangle$

This is what I've done:

$\displaystyle \langle \psi, \mathcal{T}I \rangle=\int d\mathbf{r}\int d \hat \Omega \psi(\mathbf{r},\hat \Omega) \left( \hat \Omega \cdot \nabla I(\mathbf{r},\hat \Omega)+\mu_t I(\mathbf{r},\hat \Omega)-\mu_s\int d\hat \Omega' p(\hat \Omega \cdot \hat \Omega') I(\mathbf{r},\hat \Omega') \right )$

After some manipulations it can be shown that:

$\displaystyle \langle \psi, \mathcal{T}I \rangle=\int d\mathbf{r}\int d \hat \Omega I(\mathbf{r},\hat \Omega) \left( - \hat \Omega \cdot \nabla \psi(\mathbf{r},\hat \Omega)+\mu_t \psi(\mathbf{r},\hat \Omega)-\mu_s\int d\hat \Omega' p(\hat \Omega \cdot \hat \Omega') \psi(\mathbf{r},\hat \Omega') + \oint_{\partial V} d\hat \Omega' \hat n \cdot \hat \Omega' \psi(\mathbf{r},\hat \Omega') I(\mathbf{r},\hat \Omega') \right)$

In order to obtain the adjoint equation, the last integral should vanish. I should impose boundary conditions in $\displaystyle \psi$ that make this surface integral vanish. However, I haven't been able to do it. My intuition says that I should have the boundary conditions $\displaystyle \psi(\mathbf{r},\hat \Omega)=f(\hat \Omega_r \cdot \hat n)\psi(\mathbf{r},\hat \Omega_r)$ for $\displaystyle (\mathbf{r},\hat \Omega) \in \Gamma_{+}$, this is based on some physical interpretation of the adjoint problem, I'm not sure this is correct neither, so I'm trying to prove it.

The boundary integral can be written:

$\displaystyle \oint_{\partial V} d\hat \Omega' \hat n \cdot \hat \Omega' \psi(\mathbf{r},\hat \Omega') I(\mathbf{r},\hat \Omega') = \int_{\Gamma_{+}} d\hat \Omega' \hat n \cdot \hat \Omega' \psi(\mathbf{r},\hat \Omega') I(\mathbf{r},\hat \Omega') +\int_{\Gamma_{-}} d\hat \Omega' \hat n \cdot \hat \Omega' \psi(\mathbf{r},\hat \Omega') I(\mathbf{r},\hat \Omega')$

Given the boundary condition for the transport problem, one also has that: $\displaystyle \int_{\Gamma_{-}} d\hat \Omega' \hat n \cdot \hat \Omega' \psi(\mathbf{r},\hat \Omega') I(\mathbf{r},\hat \Omega')=\int_{\Gamma_{-}} d\hat \Omega' \hat n \cdot \hat \Omega' \psi(\mathbf{r},\hat \Omega') f(\hat \Omega'_r \cdot \hat n ) I(\mathbf{r},\hat \Omega'_r)$

I thought of defining a reflection operator that acts on the scalars $\displaystyle \psi(\mathbf{r},\hat \Omega)$ and $\displaystyle I(\mathbf{r},\hat \Omega)$ such that:

$\displaystyle \hat R \psi(\mathbf{r},\hat \Omega)=\psi(\mathbf{r},\hat \Omega_r)$.

One would have for the surface integral that:

$\displaystyle \hat R \left( \oint_{\partial V} d\hat \Omega' \hat n \cdot \hat \Omega' \psi(\mathbf{r},\hat \Omega') I(\mathbf{r},\hat \Omega') \right)=\oint_{\partial V} d\hat \Omega' \hat n \cdot \hat \Omega' \psi(\mathbf{r},\hat \Omega') I(\mathbf{r},\hat \Omega')$

That is, the reflection operator would not change the value of the integral, because the integral is evaluated for every $\displaystyle \hat \Omega$, and reflecting it only would change the order on how the variables are swept. However, the definition I'm giving is vague, because in here I am applying to everything inside the parenthesis, before the integral operator. Now it would be crucial to know if I can commute the reflection operator with the integral operator, and act over the scalars $\displaystyle I,\psi$.

I've tried by applying this operator to the integral to obtain the boundary condition for the adjoint problem, and I've found a result, but I don't if what is did is right, because I haven't been rigorous with the application of the reflection operator, and on how it should act. I don't know if I am being consistent when I apply it.