NOTE: This post has panels and a minor spoiler from a recently released comic book (All-New, All-Different Avengers issue 6 – 2/24/16)
The X-Men (and all related characters) were created at Marvel Comics, meaning they exist in the same world as the Avengers and ally/fight with them in the pages of Marvel’s comic books. Even though they’re in the same universe on paper, there’s very little chance of some of these fictional people ever being on the silver screen together. You see, back in the day when Marvel was really hard up for dough they sold the film rights of their characters to various movie studios. Spidey and Daredevil went to Sony, the X-Men and Fantastic Four wound up at Fox, Hulk went to Universal (I think he’s still there? there’s some weird reason why they can’t make a solo Hulk movie, I’ve heard). When Marvel finally got their act together in the 00’s and formed Marvel Studios, they were left with a stable of less marketable characters (notably Iron Man, Captain America, and Thor) that they crafted into a multi-movie juggernaut, against all odds. Not THE Juggernaut, however, as he is an X-Men character and his movie rights are owned by Fox.

“I can’t appear in the same films as the Avengers or Spider-Man, BITCH!”
Over time a lot of these studios tried their hand at the superhero movie game, only to find that hand to be infected, maggot-ridden, and gangrenous. The franchise would flop and a certain amount of time would go by without a movie being in production, meaning the big-screen rights to the character defaulted back to Marvel. Take the Punisher, had two “eh” outings with….Lionsgate, maybe?…before a fallow period onscreen, at which point he came back into the Marvel fold and will soon be appearing in the Marvel Studios/Netflix Daredevil series (which is amazing, and my favorite part of the Marvel Cinematic Universe at the moment). For a long time DD himself was owned by Sony, who lost the rights after not following up on the Ben Afleck movie they made back in the day.

Did you know there was a whole subplot in the early 00’s Daredevil movie featuring Matt Murdock defending Coolio? Check out the Director’s Cut!
As of this writing most of Marvel’s wayward properties have returned home except for two notable exceptions; The X-Men and The Fantastic Four. Those are still firmly in the sweaty grip of Fox, who will continue to churn out pretty enjoyable X-Men films and milkfart FF movies to hold onto the rights until this bubble implodes and superhero films stop being profitable.
What this means is that as Marvel Studios continues to flesh out and expand their cinematic universe, they’re left with some pretty big (mutated) holes to fill. They’ve been turning to the inHumans, another race of superpowered, distrusted-by-John-Q-Public people, featuring them heavily in their comics and in the Agents of SHIELD TV show. Though they still put out X-Men comics (mainly because X-Men comics still bring in $$$), they seem to be marginalizing the X-Men and related characters as a whole. Scuttlebutt has it that X-writers have been ordered not to make up any new X-characters, since they would be basically creating them for Fox (this was denied by Marvel but it seems like the de facto state of the X-books).
I was reminded of this during a recent battle in issue 6 of All-New, All-Different Avengers (written by Mark Waid, art by Mahmud Asrar) during a battle with Kang the Conqueror:
DAAAAAMMN, that’s some shade. You can take it as good-natured trash talking (“our company softball team’s got way more team spirit than theirs!”) but given the state of the X-rights and the friction between Marvel Studios and Fox, you could read it as being kind of meta-catty.
Also catty; Marvel recently revealed that Magneto’s children, the mutant siblings Quicksilver and Scarlet Witch, are actually NOT Magneto’s kids and in fact not even mutants! Note that Pietro and Wanda are considered X-Men AND Avengers characters, giving them a weird “double dipping” shared rights status that lets them be used by both companies in their films (Quicksilver appeared in both X-Men: Days of Future Past and Avengers: Age of Ultron).
They also cancelled their Fantastic Four book in the lead up to the 2015 FF movie from Fox. And are quietly removing them from their merchandise.
There’s some trouble in fantasyland, folks.
On the other front, things between Marvel and Sony (who hold the rights to Spider-Man) seem pretty friendly, with Spider-Man slated to appear in Captain America: Civil War later this year. Guess they realized their own Spider-Man movies would be nothing but diminishing returns from here on out…it’s funny because a few years ago Sony execs said that Spider-Man would never, ever appear as a minor character in a movie.