We analyze the first order and second order differential structure of the space of probability measures with finite second moment on a Riemannian manifold endowed with the quadratic Wasserstein distance. After providing basics of Riemannian geometry, absolutely continuous curves and optimal transportation theory, we construct the so called weak Riemannian structure. Through the continuity equation we are able to define a suitable tangent space and to show the famous Benamou-Brenier formula. We then are able to associate to each absolutely continuous curve of measures a unique velocity vector field, lying in this tangent bundle. Afterwards we provide an alternative way of construction of a parallel transport in Euclidean space, by considering a submanifold of R^n and showing certain Lipschitz bounds for the angle between the tangent spaces of two points on this submanifold. By mimicking the results of the Euclidean case, we show the existence and uniqueness of a parallel transport in the Wasserstein setting. In the course of this construction, we define transportation maps, that allow us to map tangent vectors from one tangent space to another. They will also provide us with the analogous Lipschitz bounds as in the Euclidean case. Finally, we use the parallel transport to define the Wasserstein analogon of the Levi-Civita connection.