No, don't worry. The more you level up the stronger you'll get, and the way easier the game will get. Just make sure to use the strongest weapons and armor you find and keep putting points into whatever type of weapon you use. My latest character is now level 35 (because of Dead Money) and I can take on a group of deathclaws solo, mostly just because of how high of a level I am.
If you want to see if there is any better weapon you could be using, use this:
http://fallout.wikia.com/wiki/Fallout:_N…
By the way, use power armor when you can, side with the Brotherhood of Steel until they teach you power armor training, it's by far the best type of armor in the game.