Hi

I have an optimization problem as follows:

Determine C such that it minimizes Tr[C'ACB] subject to C'C= a.I

where all A, B and C matrices are N by N square matrices and N is an integer (for example 64).
{ ' } is the Hermitian operator. a is a scalar and I is the N by N Identity matrix.
Tr is the trace operator.
A and B are known and C should be determined by the optimization

Note that if there was no matrix B at the cost function, the problem was very easy to solve:
Min Tr[C'AC] subject to C'C= a.I

C=E (Unitary EVD matrix of A=EDE')



Thank you in advance for your help