The sets are all indeed "odd men out" sets, each with a single word which doesn't match the others (usually in a thematic way). The common theme for each set also uniquely matches a President of the United States (although repeated names like Harrison have only one for both names). When the sets are put in order by president, with the same set standing in for identically named presidents, the odd ones out spell the phrase INAUGURAL, BLACK, MOTH, HUGO, LOW, CANNON, CRYSTAL, OR EYE. This is a clue for the word BALL.
Explanation of the sets, put back in order:
elegant, thankless, harmful, midshipman, like Washington, are all words with hidden body parts (leg, angle, arm, hip, shin). The odd one out is ISRAELI, which contains "ear" backward.
photo, liar, society, hunting, like Adams, are all last words of Robin Williams movie titles. The odd one out is NINE, the first word of a Williams movie title.
damnable, blacklist, gnocchi, demijohn, like Jefferson, are all words with two pairs of alphabetically consecutive letters. The odd one out is AUTOBAHN, which has two such pairs backwards (UT, BA).
Amelia Earhart, Cyrano de Bergerac, Madame du Pompadour, Helen of Troy, like James Madison, are all XXXenophile cards. The odd one out is UNCLE SAM (just because).
avenue, hi-fi, kowtow, alfalfa, like Monroe, all phonetically end in Greek letters (nu, phi, tau, alpha, rho). The odd one out is GOATEE, which ends in the English letter T.
Davis, Green Street, Science Park, Revere Beach, like Quincy Adams, are all the next-to-last stations on T lines. The odd one out is UMASS, which is in the middle of a line.
Lynch, Altman, Scott, Howard, like Jackson, are all 2001 Best Director nominees. The odd one out is ROBBINS, who was nominated much earlier.
Duke of Clarence, George III, Pierre Curie, David Livingstone, like Martin Van Buren, are all historical characters played by Nigel Hawthorne. The odd one out is ALEXIS DE TOCQUEVILLE, who was not.
moldier, cattle, Bajor, wavy, like Harrison, are all military first letter changes (soldier, battle, major, navy, garrison). The odd one out is LEASE, a first-sound change for "peace."
petals, placed, twang, prelim, like Tyler, all form new words when the first letter is dropped and the rest is reversed. The odd one out is BIRCH, which forms a new word when the last letter is dropped and the rest is reversed.
limb, tang, Walt, bum, like Polk, are all dance curtailments (limbo, tango, waltz, bump, polka). The odd one out is LAMBDA, a deletion of "lambada."
pale, tufty, pursue, blown, like Taylor, are all letter changes on colleges (Yale, Tufts, Purdue, Brown). The odd one out is ANTILOCK, a phonetic variation on "Antioch."
spanking, readjust, departmental, chartreuse, like Fillmore, all split evenly down the middle to form two words. The odd one out is CANVASES, which splits unevenly (like "Washington").
vetos, rotates, parson, soapstone, like Pierce, are all cooking anagrams (stove, toaster, aprons, teaspoons, recipe). The odd one out is KEVIN, a near-anagram of "knives."
sneerer, crisis, slyly, Nefertiti, like Buchanan, all end in a repeated bigram. The odd one out is MUCHACHA, which ends in a repeated trigram.
Redford, crevice, pretape, earlier, like Lincoln, are all cryptograms of each other. The odd one out is OUTPOST, which has a slightly different pattern.
Simon, neck, through, half, like Johnson, are all parts of "X and X" phrases. The odd one out is TIME, which only words as "time after time" or "time and time again."
sired, ogee, chimers, dairy, like Grant, are all letter changes on mythic creatures (siren, ogre, chimera, fairy, giant). The odd one out is HABIT, a sound change on "hobbit."
son, reign, hale, due, like Hayes, are all weather homonyms. The odd one out is HIGH, a weather word with a homonym.
Faron, Artemis, Duchess, Figaro, like Garfield, are all cartoon cats. The odd one out is UBU ("Sit, Ubu, sit! Good dog.").
White, Getty, Gould, Dana, like Arthur, are all Golden Girls cast members. The odd one out is GOLD, which is part of the title, not the cast.
utilitarian, bestseller, folklore, apocope, like Cleveland, all contain five-letter palindromes. The odd one out is OPPOSITE, which contains a four-letter palindrome instead.
Repeat of Harrison (LEASE).
Repeat of Cleveland (OPPOSITE).
Carstensz, Elbrus, Aconcagua, Vinson, like McKinley, are the highest elevations on one of the seven continents. The odd one out is WHITNEY, the highest elevation in the 48 states, but not in North America.
Prairie, Guy, Sherlock, Kingston, like Roosevelt, are all first names of Sesame Street muppets (Dawn, Smiley, Hemlock, Livingston III, Franklin). The odd one out is CAMILLA, the chicken from The Muppet Show.
sport, eastern, whelm, frigging, like Taft, all form nautical words when beheaded. The odd one out is ABUSES, since buses are land vehicles.
prince, head, brine, ping, like Wilson, are all sporting good brands. The odd one out is NET, which is sporting equipment.
burlesque, revisited, morning, tracks, like Harding, are all last words of Dylan album titles. The odd one out is NASHVILLE, a first word.
brisket, mildew, hotel, warmonger, like Coolidge, all start with weather descriptions. The odd one out is OFFSPRING, which sort of ends with one.
thunder, coupon, drafter, Shinto, like Hoover, are all two letters plus a preposition. The odd one out is NEVER, one letter plus an adverb.
Repeat of Roosevelt (CAMILLA).
Ace, Fletcher, Chip, Hank, like Truman, are all Jim Carrey charaters. The odd one out is ROBIN, another character from a Jim Carrey movie.
Powhatan, meadow hen, New Hyde Park, Erewhon, like Eisenhower, all contains question words (what, when, why, who, how). The odd one out is YELLOWHAMMER, which has a lot of the same letters, at least.
Altima, patrons, callous, herbivore, like Kennedy, are all two male names plus a letter. The odd one out is SAMSARA, a male name plus a female name.
Repeat of Johnson (TIME)
Zion, onions, shoo-in, Soho, like Nixon, are all formed from letters with rotational symmetry. The odd one out is AXION, which has the A.
Plath, spook, bows, disk, like Ford, are all last letter changes on place settings. The odd one out is LASS, a beheadment of one.
Ulster, Bening, cotillion, Brandeis, like Carter, all become new words by switching two adjacent letters (luster, benign, octillion, brandies, crater). The odd one out is OBSERVE, which forms "obverse" by switching non-adjacent letters.
mitten, tinsel, solemn, favor, like Reagan, all start with a musical note (do-re-mi-fa-sol-la-ti). The odd one out is RONDO, which ends with a note.
carp, planet, tanka, tramp, like Bush, all curtail to transportation. The odd one out is ECARTE, which requires both the first and last letters to delete.
Alanis Morisette, Kurt Cobain, Liz Phair, Elvis Presley, like Bill Clinton, are all Wesley Willis songs. Sorry. The odd one out is YOKO ONO, who has a song by the Beatles about her, and really, the Beatles are pretty much the anti-Wesley Willis, as far as I can tell.
Repeat of Bush (ECARTE).