If all the three books together then it means that you can look at each combination as "sliding" along for each possibility.
You have 8 books in total which means if you start all books at the left-most side you get 6 possibilities where the books start at positions 1,2,3,4,5 and 6.
Now we need to find the number of possibilities of re-arranging the books and for this we use a standard combinatoric identity.
The number of ways arranging 8 books given 5 engineering books (or 3 physics books) is 8C5 = 8C3 or using R:
So the total number of possibilities is 6/56 = 3/28 as expected.