Cheap, fast, reliable. Pick two.
I don't want to sound like a know-it-all, but if you want more power on the cheap, buy a six-cyl. The 328iS is a fine machine - trading your car + $2K would likely put you in the ballpark. Or budget for an M52B30 swap. $2K could get you really close to a complete swap if you cut a few corners. The alu block M5x from the E39 drops right in with a few mods. They aren't that much heavier than an M4x...and they make ~220HP right out of the box.
If an engine swap isn't your cup of tea, I'd consider cheap upgrades that add reliability first. Getting the motor in top condition is a priority. Make sure you have no stored codes. Test all sensors. Do a compression check to learn your motor's baseline. Upgrade to top-quality fully synthetic lubes in the motor, tranny & diff. Clean out the AFM/MAF and ICV. Replace your fuel filter. Make sure the PCV passage is clear, consider a catch can. Be sure the DISA is in working condition, if installed. Use only high-quality OEM air filters - the stock airbox is excellent as long as the filter is clean. Eliminate all vac leaks, replace every rubber part under the hood. Eliminate the engine-driven fan with an electrical radiator fan. If possible (do you have a serpentine belt setup?), eliminate the A/C and P/S systems entirely.
I'd also install a cooler thermostat, lower-temp aux coolant switch, and thin out the coolant - up to 75% distilled water mix (with water wetter) will save you a bit of power and improve temperature transfer. Keep the battery perfectly charged - maybe with a solar trickle charger, and test the system for shorts. A dying battery or worn-out regulator makes the alternator work far harder - and that's all lost HP.
I'd also consider a WBO2 setup to see what the motor is really doing. It gives a lot of insight, and they're only $200-ish.
Silicone hoses, upgraded injectors, a COP kit & good NGK plugs really help. Give the car a MarkD or Midnight chip, a shorter diff and some carefully-considered wheel choices. It sounds counter-intuitive but
smaller circumference tires will bump up your final drive ratio & give you free acceleration, though they might look a bit odd and will raise high-speed RPM substantially.
Weight reduction is cheap - but murders your resale value. BMW worked hard to keep weight down, but you might be able to eliminate 10% or so. Exhaust systems can help with both power and weight loss, but they're pricey for their gains.