yes u should be worried ..u dint mention its size,consistency,colour,any discharge from the swelling,is it reducing on itself ..any history of injury is it painful/painless is it growing in size or the same .with out which diagnosis is not possible and most common causes for tesicle swellings are
1)hernia
2)hydrocele
3)lymph nodal either due to infection or malignancy(rare cause)
its better u see a general surgeon for this.