What I've read about the Ninja is basically this:
They were mercenaries who lived in the mountainous Iga and Koga (and there was a third area) regions of Japan, whose services were purchased from the various constantly-warring lords. They had a life philosophy based on blunt practicality, character, and the "complete development" of a person that accounted for their legendary inventiveness in fighting, spying and assassination. However, their ways didn't mesh with the Samurai's hierarchy society, and when Japan was finally united under one Shogun, the Ninja's existence was seen as a threat to the unification, and the small areas from which the Ninja came were surrounded by a great army of Samurai. Iga and Koga were crushed into oblivion by the new Shogun (was it Oda Nobunaga, or one of his successors?) From there the Ninja ways faded into legend, but the fantastical tales of their exploits and weapon gadgetry eventually got swept into popular fame.
I read somewhere that the small Ninja swords might have been cut out of a large polearm blades.
There were straight swords in ancient Japan, the chokuto, but I think they were a lot different than a Ninja sword was supposed to be.