So I had this idea and I figured I would discuss it will you all and see what you all thought and if you had any feedback on it.

First I like to think of Nightmare Magic as this confused magical art that often ponies claim to be forbidden because it deals with Nightmares, however like Necromancy in DnD 5e, Nightmare Magic isn't exactly designed to cause nightmares (though it certainly can) but is designed around the study of nightmares.

My thought is that characters can use Nightmare Magic to perform the following:

• Cause a sleeping pony to suffer a nightmare.
• Absorb a sleeping pony's nightmare (*-indicates a spell that can use the nightmare absorbed by this magic).
• Conjure up the pony's nightmare, either in a physical form or as an illusion.*

However, I'm sure there are other things that can be done with shadow magic so if you have any ideas, feel free to mention them.

6428535
I don't think it's Nightmare Magic, as much as it is Dream Magic. A way for somemone to tap the subconcious realm, like the case of Luna.

Now, this dream magic can be used in the physical realm, but one must be careful, for some forces from the dream realm can be dangerous, such as the force that turned Luna into Nightmare Moon, or what caused her to create the Tantabis.

But at the same time, I don't know how the dream realm works, so trying to fully explain dream magic is a science beyond my level of knowledge.

6428535
I am unsure of what context you're asking this about. Are you attempting to make sense of canon, develop an idea for a story, or something else? Which of these things you're asking heavily shapes the sorts of feedback that is useful.

6428702 Developing an idea for a story.

6428704
Ah. Thanks, that helps.

Well, you have a few ideas for what the magic is capable of doing. I'll start there.

Cause a sleeping pony to suffer a nightmare.

This probably is the source of why others would look askance at the category. Depending on how you decide nightmares are created, this could have a lot of different implications. My first reaction is to imagine this is a form of mind-influencing magic, where you're simply planting images into someone's unconscious imagination. The show has given us things like the Tantabus and the comics, as much as I dislike them, have an idea of nightmare entities that can parasitically infect a dreamer. If something along those lines is the source of nightmares, then your nightmare mage is summoning/creating/luring these things into a sleeping pony's dream, and that could be rather insidious and perhaps result in damage going past what the mage intended as other entities can be unpredictable and gain their own power.

Absorb a sleeping pony's nightmare (*-indicates a spell that can use the nightmare absorbed by this magic).

See above. As it would require there to be something to absorb, this somewhat implies the latter among my speculation above. On the one hand, this could be the "positive" side of the discipline, as a pony that went around absorbing nightmares would be seen as something of a healer. On the other hand, a villain could plant nightmares, allow them to grow and gain power, then use this spell as a sort of way to reap a crop to power other activities.

Conjure up the pony's nightmare, either in a physical form or as an illusion.

This lets the field of magic do something in the waking world. Probably important if you want to craft a villain and give him or her an ability to do something with a harvest of nightmare energy.

This brings me to some obvious questions to spur further development of the idea: who can use this magic and how does it work? Is it simply a category of spells that anypony could learn by reading the right books, the same way "time magic" is a category of otherwise independent spells that happen to affect time? Or is this more like "dark magic" which is "any magic that does anything, but uses the particular fuel of negative emotions" as often described in many fics?

If it is the former, then essentially each spell is just a regular one, probably created by one or more ponies, and they all work under normal unicorn magic style and limits. It also means that the second one, for example, could just be used to fuel any conceivable unicorn magic, allowing the pony to do more powerful magic than they otherwise would be capable of, or having more "stamina" to simply cast a lot more/harder. But, in theory, any unicorn could read the right book and learn one or more of these spells. Absorbing nightmares wouldn't seem like a restricted spell, and wouldn't necessarily lead a pony to do the other two spells under this style.

If it is the latter, then the "absorb nightmare" spell acts as a sort of requirement for everything else in the field. The field may replicate the effects of spells used in other disciplines, but it's sort of like writing code in different programming languages. Underneath the hood can be very different, as a nightmare mage needs to get their fuel in a particular way and route that energy in ways that make no sense to other kinds of casters, even if the end result is the same. One impact of this choice is that there are only a limited number of ways a nightmare mage could use their magic since only nightmare mages would develop spells, and the regular body of knowledge and books available to regular unicorn magic doesn't give the nightmare mage any new options to use their nightmare energy.

Going with the idea that nightmares are entities, it makes sense to go with the second style of magic described. When the nightmare mage uses that energy, it retains some element of personality from the pony it was absorbed from, and that could come out in whatever spell the nightmare mage casts with that energy.

A third option is that this magic is unique/personal to Luna/Nightmare Moon. We saw in A Royal Problem, that Luna lost the ability to do anything in the dream realm at all while Celestia held her magic. Celestia wasn't able to raise the moon until she had Luna's magic, and was surprised at the ease of doing so. More recently, we learned in Horse Play that Celestia's relationship with the Sun is that raising it is super difficult and required Starswirl the Bearded and five additional unicorns to combine their efforts to accomplish it daily while draining those assistant unicorns of their magic (hopefully nor permanently!). Celestia isn't unique in her ability to raise the sun, just that she's capable of doing it alone and without being drained by the process. So a pony's personal magic has limits to what it can and can't do, even if the things it can do are fairly strong and/or widely applicable.

With this option, you're mostly exploring what Luna/Nightmare Moon can do. However, a different pony might have their own personal magic which lets them do some stuff with nightmares. They wouldn't necessarily be able to do all the same things Luna can do (perhaps they can't influence the moon in the slightest, for example) but they can do some things Luna can't do (such as influence the waking world with nightmares).

Hopefully, that gives you things to think over and helps you in expressing your idea the way you want to in your fic.