Firstly note that a quadrilateral is determined by its (unordered) set of vertices.

There are three cases: the vertex colours could be RRBG, RBBG or RBGG (in some order). Take RRBG as an example. You need to count the number of ways of choosing 2 red out of 8, 1 blue out of 2 and 1 green out of 4. The other two cases follow similarly.