The danger of doing both is getting hurt in the tour. Hesjedal may be able to ride in the olympic road race but probably not at full strength. Samuel Sanchez probably won't defend his gold medal due to the broken wrist and shoulder he suffered in the latest crash. With only a week between the end of the tour and the olympic race, there isn't much recovery time. Time trialists have it a little better, a few more days in between.
Cavendish had to find another team when HTC folded. They were geared for him, their leadout train the best. They didn't have a real GC contender and were after stage wins. But they ran out of money. Most other upper echelon teams were either strictly GC or already had sprinters on board. He probably figured he could freelance with Sky, like Robbie McEwan did for his career. Also, he lost some weight this year for the olympics. The course has a small climb on it, not even a cat 4. But they do it 9 times. Sprinters are generally heavier/more muscled than climbers to generate the power for the sprint. The less weight you have to haul over a climb, the better. So Cavendish probably doesn't have the power that he is used to and that affects how he times his last thrust to the line. Being in the crashes hasn't helped.
HTC would control the race at the end because they had so many on their train. Other teams wouldn't allocate as many to their sprinters. With them gone, more teams have their trains going and that causes mass confusion at times. As Voekler said 'the road is only so wide and everyone is trying to be there.' A recipe for chaos. 20 withdrawals so far, 1 due to sickness, 19 for injury. The last time they had this many during the first week was when entire teams were tossed if one was caught doping. No dopings this week.