If it's acting odd, not showing any fear, etc, I would be worried.
However, right now the fox at the end of our driveway has kits, and she is out at all manner of the day hunting gophers. Saw her yesterday morning prowling the fence line, and a few days ago she was our in the late morning in the field below us.
In the early evening she and the babies will be outside their den, lying in the grass. So it's not completely out of character to see them in the daytime.