Bottom sediment in a river often acts as a sink and indicator of changes in water column and magnitude of anthropogenic influences through air and watersheds. Heavy metal concentration in sediments of Ganga River was studied along a 37-km stretch to assess whether there is a significant difference between sites situated upstream and downstream of Varanasi urban core. Metal concentration increased consistently along the study gradient, indicating the influence of urban sources. Concentration in the river sediment was found highest for Fe followed by Mn, Zn, Cr, Cu, Ni, Pb, and Cd. Mann–Kendall trend analysis showed marked seasonality in the concentration with values being highest in summer and lowest in rainy season. Enrichment factor revealed severe enrichment of Cd and Pb at downstream sites, and principal component analysis segregated sites into four distinct groups indicating source relationships. Concentrations of Cd, Pb, Ni, Cu, and Cr did exceed WHO standards. The study has relevance designing control measures and action plans for reducing sediment contamination in anthropogenic impacted rivers.