ADR is a tricky thing to get right. While on set, an actor focuses purely on their performance, but once they get into the ADR booth, a lot of their focus has to go into the timing, because they have to match an already finished performance. It locks out a lot of the potential emotion. On set, an actor has emotional freedom (well, depends on the director, but for the most part he does). An actor can add a pause between words, switch up some timing etc... and is fully able to express emotion that way. But in the ADR booth, it's sort of like saying, "I want to you act as though you have complete emotional freedom, like you did the first time, except that I need you to do it exactly like this, and match your original emotion perfectly, even if you aren't feeling that emotion any more." It makes it very difficult to express real emotion when you have such a large constraint. Some actors are very good at it though (like Brando and Crowe), and can make you believe in it easily.
Then on the technical side of things, there can be a few factors. It is important to try to match the type of mic used for the line on set, match (roughly) the distance from the mic to the talent, etc... especially if you are only replacing a single line or a word. Reason being that it is very difficult to eq a sound to match another one perfectly. There will always be a slight difference, and you can minimize the difference by trying to match the original setup as much as possible. Maybe only 1% of your audience can identify the difference, but more than that might feel it. If you are replacing a whole scene, the technical side (matching the mic) doesn't matter as much.
Last thing I'll mention is the room tone. When you record on set, you are picking up a slight (hopefully) amount of room tone in the mic. When you record ADR, you are in a "dead" booth, with no (again, hopefully) room tone. In order for the ADR to fit, you have to lay room tone (often called a "glue track") under the ADR to make it sound like the rest of the dialogue. This is assuming the sound mixer was even afforded the opportunity to record room tone (which often doesn't happen anymore, especially in faster moving productions). Even with the "glue track," it doesn't ever quite sound 100% right. It's something that you have to hide, or "burry" in the mix. This is something that almost nobody can pick up on unless you are trained to hear it, but subconsciously, your brain will tell you something isn't quite right.
Hope this helps a bit?