I'm sure there are plenty of magicians who started as adults. Starting younger just gets you that much farther ahead. But starting younger doesn't help a whole lot becuase your hands are too small and you're not as smart(hehe). You definately shouldn't quit, if it's something you love, keep at it.