Metal in the gearbox lube is very normal, every gearbox I've changed looked like metalflake paint with a lot of shavings on the drain plug too. Just the way it works, manuals are pretty sloppy all in all. If you were missing 3/4 you'd be seeing some really big chunks of metal in the oil and drain plug.
I'd guess the shifter linkage is wrong somehow. Maybe misaligned? Can you get under the car and watch the shifter while someone else rows through the gears?