Suppose a group acts on a manifold and the action can be lifted to a principal bundle over it. We consider invariant connections and their gauge equivalence classes. When the base space is a single group orbit, the invariant connections were studied by H.C.Wang in the 1950s. We show that in general, the space of such connections decomposes according to elements in a group cohomology. We further study the geometry and analysis of the space of invariant connection.