Thanks for the informative posts Chris. Let me ask a question to try and wrap my brain around it. If we put it into a modern military context (in VERY rough and broad ways). All soldiers learn the same basic skills. After that they learn certain specialized skills. So, your scouts would learn how to infiltrate gather information etc. Your combat medics would learn how to apply medicine in the field. Then, you might have a group of select soldiers that operate on their own outside of the regularly structured military. These guys would have to have training in alot of different areas instead of specializing in only one or two. A group like the Navy SEALS would have to learn different types of weapons, explosions, infiltration tactics, stealth and escape methods, combat medicine etc. and a host of other skills needed to complete missions.
This is how I have pictured the ninja. A soldier that recieved special training to carry out specific missions, so it seems that many of the things that are "ninja" related would be in other systems too because they are the same tools and tactics that the regular soldiers would be learning too in many cases. Just not rolled into one package.
I can see where you're coming from, but that's just not the reality in a few ways. The way these arts work is a little different to specialization after basic training... mainly as the intended recipients of such arts weren't really taught in the same way that modern armies are. In modern armies, typically, a new recruit goes through basic training, and then might get some additional training in one area or another, based on where they were being assigned. By contrast, most martial arts (speaking historically of Japanese systems here) were typically only trained by the higher classes, and the training would entail everything that the art contained, with the only real idea of specialization being based on the martial system itself. You really wouldn't train in one system for basic understanding, then another for a specialization.
It really should be remembered that the majority of what martial arts (formalized) there were were quite distinct from basic soldier training. This is mainly due to there being a dominant and separate warrior class, who ran the country, which made the make-up of the armies a little different. It changed depending on when exactly you're talking about, but as most of these Ryu-ha are from Sengoku Jidai or Edo Jidai (Warring States Period, mid 15th -17th Century, and Edo Period, basically peace-time from the 17th Century to the mid/late 19th Century), we'll take Sengoku Jidai as a template.
Armies could be broken, dominantly, into two distinct groupings (within it's structure). The common thought, or belief, is that they were made up exclusively of samurai, professional warriors trained from birth, but that simply wasn't the case. The samurai only really made up the higher ranks, the generals and commanders of the army, with the bulk of the forces (the infantry etc) being made up of conscripts, who might be farmers, peasants, or similar, under the jurisdiction of the particular Daimyo (feudal lord) of the area. As these conscripts weren't samurai, they wouldn't be given what we would class as martial art training... instead, they'd get a basic form of training with whatever weapon they were given (typically spears), and that would be about it. So while they would get the equivalent of modern "basic training" (well, a fair bit more "basic", to be honest!), there really wouldn't be much chance of any specialist training after that (excepting for those particularly skilled, who managed to catch the eye of their commander, or general, and get some special attention, sometimes to the point of being promoted to the status of "samurai", most famously in the case of Hideyoshi). Higher level training really was the providence of the samurai.
When it came to the training of the samurai ranks, what that would be would depend on a range of different variables, such as if there are any Otomo Ryu ("Official Ryu" of a particular Daimyo/domain), or what else was around for a particular samurai to train in. Some Daimyo would reward samurai seeking out licencing (training) in various disciplines, in order to ensure a wider skill set (what we would consider cross-training today), so if a samurai managed to get Menkyo in two arts, say swordsmanship and spear, he would receive a higher pay, if he got a third, say, suijutsu (military methods of swimming), or bajutsu (horsemanship), he might get a higher pay again, and perhaps a short sword, add more and he'd receive a long sword to match, and so on. Other Daimyo, on the other hand, would want to control what influence others had on their retainers, as to study a Ryu is to study a different way of thinking (in many ways), so would limit what a samurai of their domain could study.
Many Ryu at this point in time were more "sogo bujutsu", or composite schools. These Ryu-ha might have a major focus, such as Kenjutsu, or Sojutsu (spearmanship), but aimed to give a wider, overall education when it came to military matters. It really was more like Officer training than anything related to Basic Training, as the aim was to develop warriors with complete tactical understanding. A part of this training often included gunryaku heiho (or similar), essentially "battlefield strategy", which would include things like how to engage in or defend in siege warfare, how to set up an encampment, how to field an army, common or unique martial strategies for army movements, and the usage of various military assets, as it were, which would include how to engage and employ people as "ninja", whether specifically ninja themselves, or just regular samurai operating in a "ninja" engagement. In fact, in some Ryu-ha, the "ninjutsu" portion really is little more than "how to use a ninja properly".
It should be recognized, though, that even in this case, the idea of "use a ninja properly" doesn't necessarily even imply the warriors of Iga and Koga, but more about how to use people to get information, including how to turn local people into spies, and so forth.
In this case, you can see that the idea of "ninja" itself can even have different meanings, with some interpretations being the warriors of Iga and Koga, who, as part of their warriorship identity included a higher focus on espionage, even though not entirely exclusive to them, and another usage of the description being essentially the "job description" for anyone employed for the gaining of information, which could very easily be someone not of a warrior class. By the same token, there wouldn't be "military medics" who were soldiers, either, even though a number of martial traditions would include basically forms of first aid... the idea was that you'd look after yourself, rather than there being specific people for specific jobs, by and large.
Hmm, hope that made some sense... any questions, just ask. There's more to it, obviously, but I fear I'd just be confusing the issue if I put it all down now...
Here's my two cents ... everything in Japanese is that vague, since Japanese is a very context driven language. Here's an example for you ... Muso Jikiden Eishin ryu is one of the most widespread of Japanese sword arts, second only to kendo. It consists mainly of single person kata, wlthough there are two person kata when you get into the higher ranks. I know of at least two of their kata that are specifically regarding locating and killing someone in total darkness. Would that be considered ninjutsu oir anti-ninjutsu? MJER iaido just calls it kata and moves on. Also, once the country was unified in 1600, there was no real reason to continue training in mass warfare and espionage techniques. The vast majority of training slowly turned toward more of the dueling nature that it is today. Much was jettisoned and lost over the years in favor of concentrating on areas of the art that were still in use.
There are never any easy answers where the Japanese arts are concerned.
Ha, despite the popular imaginings, the idea of "killing in the dark" really isn't anything to do with ninjutsu... there's little historical evidence of any connection with the idea of "ninja" and the idea of assassination. Probably the most famous story that tries to link the two concepts is the death of Uesugi Kenshin in 1578. The story goes that Uesugi, who was engaged in battles with Oda Nobunaga, was rather distrustful of shinobi (ninja), especially those in the employ of Oda. At one time, one of Oda's ninja, one Ukifune Jinnai. Ukifune resolved to watch and strike when Uesugi's guard was lowered, however the Daimyo was constantly on guard. So Ukifune decided on a rather unique plan.... Ukifune was unique for a range of reasons, including his skills, but also in regards to his stature. He stood only about a meter tall (just over three feet), and he had taken advantage of this seeming limitation by training himself to remain in very small, enclosed areas, in order to gain access where others' couldn't... and saw an opportunity to use this unique skill in an attack on Uesugi.
Ukifune decided to wait in Uesugi's toilet.
He hid himself under the boards that formed the floor of the latrine, waiting for a long time for the inevitable to occur, and Uesugi's digestive process to reach it's conclusion. When Uesugi came in to relieve himself, Ukifune struck, thrusting directly upward with his sword as Uesugi sat above him. Uesugi screamed in pain, bringing his guards into the toilets, but by the time they arrived, Uesugi was already dead, and Ukifune had dropped into the sewerage below, escaping by breathing through the saya (scabbard) of his sword, making his escape to safety and freedom.
Catch is, of course, that this story is a legend. While it's true that Uesugi was in the latrines when he cried out in pain, and his soldiers came in to find him slumped, and carried him out, he was alive when they did so. In fact, he didn't die for another three days... and there was no sign of any injury from a weapon. The official historical records state that "at the ninth day of the third month, he (Uesugi Kenshin Daimyo) had a stomach ache in the bathroom. Unfortunately, the evil persisted until the thirteenth, when he died." There were tales of Uesugi being ill for some three years previously, and while it's uncertain exactly what he died of, it was more likely a form of cancer, or ulcer, even possibly a burst appendix that actually killed the warlord. As a result, the story of Ukifune can be discounted as a historical account of "ninja assassination", and, as it's one of the major supporting stories for the idea, the concept of ninja assassins doesn't have a lot of real support.
When it comes to the Edo-Jidai being a time of peace, and therefore a range of aspects of certain Ryu being jettisoned, as no longer needed, I would point out that ninjutsu, in particular, was one skill that retained relevance longer than a number of others. The way the peace was attained and maintained included a range of different measures for ensuring the loyalty of the various warlords, which included forcing Daimyo to maintain two separate households, one in their domain, and another in the new capital of Edo (present day Tokyo), keeping the Daimyo financially strained and unable to raise and maintain a military force able to rise up against the government. But to keep doubly sure, the Daimyo and their family would be in the different locations... while the Daimyo was in Edo, his family would be in the domain's house, and when the Daimyo went back to their home domain, their family were relocated to Edo. While that might seem like just a way to stress the families connection, that wasn't the point.
The Tokugawa government employed a range of "agents", essentially ninja, known as metsuke ("eyes"), under the direction of the Hattori leadership (Hattori Hanzo, possibly the most famous "ninja", had been instrumental in aiding Tokugawa Ieyasu's attaining of the position of Shogun). These metsuke were put in different positions in the Daimyo's households, without the Daimyo knowing who was one, and who wasn't. They might be the gardener (the classic one, really), a member of the village, cooks, really, anyone. Their role was to observe, and report anything on the movements of the Daimyo, or his family. So, if the Daimyo was found to be plotting anything while in his home, his family in Edo would be in real danger, and obviously if the Daimyo was plotting in Edo, he was under constant watch there, too. So the role of the ninja, maintaining and gathering information, was more important than learning spearmanship in a time when there were no pitched battles (well, very few ones).