Most big box stores sell some inexpensive rifles. As for handguns, there are some inexpensive ones, but not many. It just depends on what you're looking for. I would recommend going on Youtube and look up MD Prepper. He has a series on low budget firearms. Pretty good info. I like the Russian Mil-Surp stuff myself and got my C&R FFL. It was $30 and was pretty much the same paperwork you fill out when you buy a firearm.
The C&R is good for 3 years. This allows you to buy "C&R Eligible" firearms from distributors and private sellers; stuff like Mosins, and lots of others too. They can be shipped right to your door (rifle or pistol) eliminating the need to go through an FFL ($), driving to your FFL (Time/$) and if your state is like mine, having to pay sales tax on not only the firearm itself, but shipping cost's as well ($$$). My first C&R purchase was online and I saved about $120 in FFL Fees/Sales Tax. These were for a couple of Remington 30-06 Rifles. Basically, for the most part, if its manufacture date (the actual day) was 50 years ago, it's C&R Eligible.
Gunbroker is a great site, and if nothing else, can help you find what you're looking for and get some "real market" pricing. Some C&R sites are AIM Surplus, J&G Sales, Sarco Inc., Century International Arms, and many others. Pick up a copy of Shotgun News and you can find all kinds of distributors. And find gunshows. There are companies that do nothing but put on gunshows. Big, lots of stuff, gunshows. Many require you to join in order to purchase firearms, some don't. Have a C&R and you're in like flint if they do.
I like the Russian Mil-Surp weapons because they are plentiful, inexpensive (compared to like "modern weapons"), reliable, tough as nails and the ammo is CHEAP! May not be the prettiest weapons, but they function and can put a lot of lead downrange. And they just keep running, and running, and running.
Can't afford an AK ($550 or less)? Pick up an SKS. I picked one up a couple of weeks ago, mint, unissued, unfired, for $350. Can't afford a "standard" bolt action rifle, even an inexpensive one, get a Mosin 91/30. Most you should pay is $150. You can get 440 Rounds shipped to you for about $100. Plus, they should come with bayonet and tool kit. It's kind of cool. I mean, I haven't seen Remington, Weatherby, Mossberg or Savage build a rifle lately that came with a bayonet.
Go to Wikipedia and look up "List of Rifle Cartridges" and "List of Pistol Cartridges". Start comparing ballistics of 7.62x39, 7.62x54r, 5.45x39 to .308/30-06/5.56 etc. or the 7.62x25 (pistol) vs. 9mm and the 7.62 Nagant (pistol). You'll see the similarities ballistically, then go to Gunbroker, Midway USA, Cheaper Than Dirt and start comparing prices between Russian Surplus (or new manufacture) and the "American" ammo. You can get 1080 rounds of 5.45x39 shipped to you for about $170. Try that with 5.56; ain't gonna happen.
Trust me brother, I'm the father of 8 and my wife has a life long, debilitating illness that causes her to miss more work than she doesn't. And I'm a public school employee. Frugal is my middle name. And even when times were financially good, I rarely ventured into the expensive arena. Often getting rid of them if I did. I find what works at a price that doesn't make me want to throw up and to hell with the gun snobs. I don't need a thousand dollar custom fit rifle or a $1000 Kimber to shoot well with. I just use the fundamentals. I can kill something just as dead at long distance for a hundred or two, as I can with something for a grand.