Fazal Majid's low-intensity blog

Sporadic pontification

Scanning your iTunes library for DRM-infested books

Tor, the leading publisher for Science Fiction and Fantasy books, announced they would be doing away with DRM in their eBooks. The product pages for their books on iBooks now mention “At the publisher’s request, this title is being sold without Digital Rights Management software (DRM) applied”. I figured it would be a good idea to uncripple the many Tor eBooks I have in my collection.

I wrote a quick little Python script to scan my growing iBooks library for books that could be updated. The procedure is to delete the book from both iTunes and iPads, then download it anew (restarting iTunes is also needed after deleting). Apple keeps track of your purchases and will not charge you again.

#!/usr/bin/env python
import sys, os.path, glob, zipfile, platform, xml.etree.ElementTree

# publishers who have forsaken DRM
good = ['Tom Doherty']

if platform.mac_ver()[0] > '10.8':
  bookdir = os.path.expanduser('~/Library/Containers/com.apple.BKAgentService/Data/Documents/iBooks')
else:
  bookdir = os.path.expanduser('~/Music/iTunes/iTunes Music/Books')

os.chdir(bookdir)

ok =  '\033[1;32mDRM-free    \033[0m'
bad = '\033[1;31mDRM-infested\033[0m'

count = 0
salvageable = 0

def extract(meta):
  creator = ''
  status = ''
  pub = ''
  et = xml.etree.ElementTree.fromstring(meta)
  try:
    creator = et.findall('*{http://purl.org/dc/elements/1.1/}creator')
    creator = creator[0].text
    title = et.findall('*{http://purl.org/dc/elements/1.1/}title')
    title = title[0].text
  except:
    assert '!DOCTYPE plist' in meta
    next_tag = None
    for e in et[0].iter():
      if e.tag == 'key' and e.text in ('artistName', 'itemName'):
        next_tag = e.text
        continue
      if next_tag == 'artistName':
        creator = e.text
        next_tag = None
        continue
      elif next_tag == 'itemName':
        title = e.text
        next_tag = None
        continue
  pub = [x for x in good if x in meta]
  return creator, title, pub

def find_meta(file_list, opener):
  for m in file_list:
    if m.endswith('.opf') or m == 'iTunesMetadata.plist':
      meta = opener(m).read()
      return extract(meta)
  
for fn in glob.glob('*/*.epub'):
  status = ok
  suffix = ''
  if os.path.isdir(fn):
    suffix = '(directory)'
    if os.path.exists(fn + '/META-INF/encryption.xml'):
      status = bad
      count += 1
    meta = find_meta(os.listdir(fn), lambda x: open(fn + '/' + x))
  else:
    z = zipfile.ZipFile(fn)
    try:
      i = z.getinfo('META-INF/encryption.xml')
      status = bad
    except KeyError:
      pass
    meta = find_meta(z.namelist(), z.open)
    z.close()
  creator, title, pub = meta
  print status, fn, suffix
  print '\t', creator
  print '\t', title
  if status == bad and pub:
    print '\t\033[1;32mThis is published by', pub[0],
    print 'and could be re-downloaded DRM-free\033[0m'
    salvageable += 1

print count, 'books are DRM-infested'
print salvageable, 'could be cured'

Unfortunately, it seems like the DRM-stripping is still work in progress. Out of the Wheel of Time series, for instance, only the first one is now DRM-free on the iBooks store.

undr ~>drmbooks.py
DRM-free     Books/0083D0AEC37E08453347DD12B1C6F980.epub
    Greg Bear
    Blood Music
DRM-free     Books/09178837756A4DFF8347EC377345A37B.epub
    Heinz Wittenbrink
    RSS and Atom
DRM-free     Books/0AD752E995042C7E12F11917AB58C6B8.epub
    Wes McKinney
    Python for Data Analysis
DRM-free     Books/14BDC66A99E878EC232FFAFA73B341EF.epub
    Fritz Leiber
    Swords and Deviltry-Fafhrd and the Gray Mouser-Book1
DRM-free     Books/15A1D7FE9B7D815C6FBE1A9A77D7143E.epub
    Glen Cook
    A Fortress in Shadow
DRM-free     Books/1793F9DE1319B96FDE7E36EB8A1BC961.epub
    Scalzi, John
    Old Man’s War
DRM-free     Books/1D08BE221E8BC8F2A371EFEDE55029AC.epub
    Ben Fry
    Visualizing Data
DRM-free     Books/24D6EC36CDEA0C1E8612CC61A89EA098.epub
    None
    Node Cookbook
DRM-free     Books/29DA285F0051C431BD8BA3D1AEC5EAA6.epub
    Fritz Leiber
    The Swords of Lankhmar: Fafhrd and the Gray Mouser-Book 5
DRM-free     Books/2E88CD68DFD8408CD0E7C0ACB1E78714.epub
    Glen Cook
    A Cruel Wind: A Chronicle of the Dread Empire
DRM-free     Books/32996A9995040064818BAE4DFB66E92F.epub
    Kelly Link
    Magic for Beginners
DRM-free     Books/34D3CD13D47E5FEBC6DCF7EF011113BD.epub
    David Drake
    Lord of the Isles
DRM-infested Books/357298432.epub
    Iain M. Banks
    The Player of Games
DRM-infested Books/357311036.epub
    Iain M. Banks
    Use of Weapons
DRM-infested Books/357377857.epub
    Iain M. Banks
    Against a Dark Background
DRM-infested Books/357396585.epub
    Brent Weeks
    Night Angel: The Complete Trilogy
DRM-infested Books/357657026.epub
    Iain M. Banks
    Transition
DRM-infested Books/357658374.epub
    Po Bronson
    NurtureShock: New Thinking About Children
DRM-infested Books/357662058.epub
    Iain M. Banks
    Consider Phlebas
DRM-infested Books/357669769.epub
    Iain M. Banks
    Matter
DRM-infested Books/357914731.epub
    Herbert, Frank
    Dune Messiah
DRM-infested Books/357918110.epub
    Dalrymple, William
    City of Djinns
DRM-infested Books/357923567.epub
    Patrick Rothfuss
    The Name of the Wind
DRM-infested Books/357929995.epub
    Herbert, Frank
    Dune (40th Anniversary Edition)
DRM-infested Books/357969577.epub
    Herbert, Frank
    God Emperor of Dune
DRM-infested Books/357987322.epub
    Herbert, Frank
    Children of Dune
DRM-infested Books/357994537.epub
    Herbert, Frank
    Heretics of Dune
DRM-infested Books/357994652.epub
    William Dalrymple
    White Mughals: Love and Betrayal in Eighteenth-Century India
DRM-infested Books/357996119.epub
    Stross, Charles
    Wireless
DRM-infested Books/360601506.epub
    Ursula K. Le Guin
    The Dispossessed
DRM-infested Books/360609519.epub
    Greg Egan
    Schild’s Ladder
DRM-infested Books/360627712.epub
    Raymond E. Feist
    Rides a Dread Legion
DRM-infested Books/360627930.epub
    Neal Stephenson
    Anathem
DRM-infested Books/360628773.epub
    Raymond E. Feist
    At the Gates of Darkness
DRM-infested Books/360641088.epub
    Mihaly Csikszentmihalyi
    Flow
DRM-free     Books/361491495.epub
    Basil Hall Chamberlain
    Aino Folk-Tales
DRM-free     Books/361494664.epub
    Poul William Anderson
    Industrial Revolution
DRM-free     Books/361523763.epub
    Lafcadio Hearn
    The Romance of the Milky Way / And Other Studies & Stories
DRM-free     Books/361527545.epub
    Saki
    When William Came
DRM-free     Books/361539032.epub
    Saki
    The Chronicles of Clovis
DRM-free     Books/361557387.epub
    Saki
    Reginald in Russia and other sketches
DRM-free     Books/361557834.epub
    Sir Arthur Conan Doyle
    The Adventure of the Dying Detective
DRM-free     Books/361559391.epub
    Sir Arthur Conan Doyle
    The Valley of Fear
DRM-free     Books/361560694.epub
    Lafcadio Hearn
    Chita: a Memory of Last Island
DRM-free     Books/361561399.epub
    Confucius
    The Analects of Confucius (from the Chinese Classics)
DRM-free     Books/361562678.epub
    Saki
    The Toys of Peace, and other papers
DRM-free     Books/361562764.epub
    Sir Arthur Conan Doyle
    The Memoirs of Sherlock Holmes
DRM-free     Books/361564075.epub
    Poul William Anderson
    The Burning Bridge
DRM-free     Books/361564898.epub
    Henry David Thoreau
    Walden
DRM-free     Books/361565201.epub
    Saki
    Beasts and Super-Beasts
DRM-free     Books/361565806.epub
    Isaac Asimov
    Youth
DRM-free     Books/361572327.epub
    Lafcadio Hearn
    Kokoro / Japanese Inner Life Hints
DRM-free     Books/361573126.epub
    Sir Arthur Conan Doyle
    Through the Magic Door
DRM-free     Books/361575882.epub
    Sir Arthur Conan Doyle
    Tales of Terror and Mystery
DRM-free     Books/361578744.epub
    Lafcadio Hearn
    In Ghostly Japan
DRM-free     Books/361588265.epub
    Lafcadio Hearn
    Books and Habits from the Lectures of Lafcadio Hearn
DRM-free     Books/361673695.epub
    E. C. Babbitt
    More Jataka Tales
DRM-free     Books/361686559.epub
    Poul William Anderson
    Security
DRM-free     Books/361713863.epub
    Sir Arthur Conan Doyle
    The Return of Sherlock Holmes
DRM-free     Books/361721797.epub
    Sir Arthur Conan Doyle
    The Adventure of the Cardboard Box
DRM-free     Books/361725352.epub
    Saki
    The Unbearable Bassington
DRM-free     Books/361725959.epub
    Sir Arthur Conan Doyle
    The Adventure of Wisteria Lodge
DRM-free     Books/361726975.epub
    Saki
    Reginald
DRM-free     Books/361727237.epub
    Sir Arthur Conan Doyle
    The Adventure of the Red Circle
DRM-free     Books/361732286.epub
    Poul William Anderson
    The Valor of Cappen Varra
DRM-free     Books/361736043.epub
    Lafcadio Hearn
    Kwaidan: Stories and Studies of Strange Things
DRM-free     Books/361741563.epub
    Lafcadio Hearn
    Japan: an Attempt at Interpretation
DRM-free     Books/361743007.epub
    Ambrose Bierce
    The Devil’s Dictionary
DRM-free     Books/361743953.epub
    Sir Arthur Conan Doyle
    The Adventure of the Devil’s Foot
DRM-free     Books/361744178.epub
    Sir Arthur Conan Doyle
    His Last Bow
DRM-free     Books/361745602.epub
    Poul William Anderson
    The Sensitive Man
DRM-infested Books/362435686.epub
    Ansary, Tamim
    Destiny Disrupted
DRM-infested Books/366773380.epub
    Esslemont, Ian C. C.
    Return of the Crimson Guard
    This is published by Tom Doherty and could be re-downloaded DRM-free
DRM-infested Books/373338999.epub
    Jordan, Robert
    The Path of Daggers
    This is published by Tom Doherty and could be re-downloaded DRM-free
DRM-free     Books/375554215.epub
    Orson Scott Card
    The Lost Gate
DRM-infested Books/376217648.epub
    Steven Erikson
    Reaper’s Gale
    This is published by Tom Doherty and could be re-downloaded DRM-free
DRM-infested Books/376227359.epub
    Jordan, Robert
    Winter’s Heart
    This is published by Tom Doherty and could be re-downloaded DRM-free
DRM-infested Books/376227401.epub
    Jordan, Robert
    Crossroads of Twilight
    This is published by Tom Doherty and could be re-downloaded DRM-free
DRM-infested Books/376227406.epub
    Jordan, Robert
    Knife of Dreams
    This is published by Tom Doherty and could be re-downloaded DRM-free
DRM-infested Books/376227409.epub
    Sanderson, Brandon
    The Gathering Storm
    This is published by Tom Doherty and could be re-downloaded DRM-free
DRM-infested Books/376227423.epub
    Jordan, Robert
    New Spring
    This is published by Tom Doherty and could be re-downloaded DRM-free
DRM-free     Books/376231110.epub
    Cook, Glen
    Surrender to the Will of the Night
DRM-infested Books/376231528.epub
    Robert Jordan and Brandon Sanderson
    Towers of Midnight
    This is published by Tom Doherty and could be re-downloaded DRM-free
DRM-infested Books/378317076.epub
    Jordan, Robert
    A Crown of Swords
    This is published by Tom Doherty and could be re-downloaded DRM-free
DRM-infested Books/378317808.epub
    Robert Jordan
    Lord of Chaos
    This is published by Tom Doherty and could be re-downloaded DRM-free
DRM-free     Books/379451459.epub
    Le Guin, Ursula K.
    Word for World is Forest, The
DRM-free     Books/37EC6E895E6BD70BEB48D9F1553D608E.epub
    Eben Hewitt
    Cassandra: The Definitive Guide
DRM-free     Books/380490608.epub
    Jordan, Robert
    The Eye of the World
DRM-free     Books/380494444.epub
    Asimov, Isaac
    The End of Eternity
DRM-infested Books/381497257.epub
    Harold McGee
    On Food and Cooking, The Science and Lore of the Kitchen
DRM-infested Books/381622032.epub
    IAIN M. BANKS
    Look to Windward
DRM-infested Books/381683084.epub
    Ursula K. Le Guin
    Tehanu
DRM-infested Books/381935940.epub
    Richard Adams
    Watership Down
DRM-infested Books/382674388.epub
    Steven Erikson
    Dust of Dreams
    This is published by Tom Doherty and could be re-downloaded DRM-free
DRM-infested Books/383912791.epub
    Steven Erikson
    Bauchelain and Korbal Broach
    This is published by Tom Doherty and could be re-downloaded DRM-free
DRM-infested Books/385975662.epub
    Jordan, Robert
    The Dragon Reborn
    This is published by Tom Doherty and could be re-downloaded DRM-free
DRM-infested Books/385981104.epub
    Steven Erikson
    The Bonehunters
    This is published by Tom Doherty and could be re-downloaded DRM-free
DRM-infested Books/385981116.epub
    Steven Erikson
    Midnight Tides
DRM-free     Books/385982966.epub
    Brust, Steven
    To Reign in Hell
DRM-infested Books/385987858.epub
    Steven Erikson
    Gardens of the Moon
    This is published by Tom Doherty and could be re-downloaded DRM-free
DRM-infested Books/385989170.epub
    Steven Erikson
    House of Chains
DRM-infested Books/385992628.epub
    Jordan, Robert
    The Fires of Heaven
    This is published by Tom Doherty and could be re-downloaded DRM-free
DRM-infested Books/385992927.epub
    Steven Erikson
    Toll the Hounds
    This is published by Tom Doherty and could be re-downloaded DRM-free
DRM-infested Books/385992930.epub
    Jordan, Robert
    The Great Hunt
    This is published by Tom Doherty and could be re-downloaded DRM-free
DRM-infested Books/385998417.epub
    Jordan, Robert
    The Shadow Rising
    This is published by Tom Doherty and could be re-downloaded DRM-free
DRM-infested Books/386016540.epub
    Esslemont, Ian C. C.
    Night of Knives
    This is published by Tom Doherty and could be re-downloaded DRM-free
DRM-infested Books/388403394.epub
    Steven Erikson
    The Crippled God
DRM-infested Books/389191300.epub
    Iain M. Banks
    Surface Detail
DRM-infested Books/390877859.epub
    Loewen, James W.
    Lies My Teacher Told Me
DRM-infested Books/393310992.epub
    Erikson, Steven
    Memories of Ice
DRM-infested Books/394745271.epub
    Steven Erikson
    Deadhouse Gates
    This is published by Tom Doherty and could be re-downloaded DRM-free
DRM-free     Books/394745833.epub
    Walton, Jo
    Among Others
DRM-free     Books/395536306.epub
    Sir Arthur Conan Doyle
    The Adventures of Sherlock Holmes
DRM-free     Books/395537209.epub
    Sir Arthur Conan Doyle
    The Sign of the Four
DRM-free     Books/395539542.epub
    Sir Arthur Conan Doyle
    A Study in Scarlet
DRM-free     Books/395540660.epub
    Sir Arthur Conan Doyle
    The Hound of the Baskervilles
DRM-free     Books/395686685.epub
    Dante Alighieri
    Divine Comedy, Longfellow’s Translation, Complete
DRM-free     Books/395688318.epub
    Edgar Rice Burroughs
    A Princess of Mars
DRM-free     Books/395688375.epub
    Lafcadio Hearn
    Glimpses of an Unfamiliar Japan / First Series
DRM-infested Books/395926792.epub
    Fukuyama, Francis
    Origins of Political Order
DRM-infested Books/396269736.epub
    Herbert, Frank
    Chapterhouse: Dune
DRM-infested Books/398283114.epub
    Rothfuss, Patrick
    The Wise Man’s Fear
DRM-free     Books/3A5FBC58E821CFDF15C8C4E85657481E.epub
    Jon Hicks
    The Icon Handbook
DRM-free     Books/410943153.epub
    Edwin A. Abbott (A Square)
    Flatland: A Romance of Many Dimensions
DRM-free     Books/413463878.epub
    Brust, Steven
    Tiassa
DRM-free     Books/418293515.epub
    Heinlein, Robert A.
    Glory Road
DRM-infested Books/419950945.epub
    Isaac Asimov
    Foundation
DRM-infested Books/419950970.epub
    Isaac Asimov
    Foundation and Empire
DRM-infested Books/419950976.epub
    Isaac Asimov
    Second Foundation
DRM-infested Books/419968238.epub
    Scott Lynch
    The lies of Locke Lamora
DRM-infested Books/419968784.epub
    Scott Lynch
    Red Seas Under Red Skies
DRM-infested Books/420037362.epub
    Kim Stanley Robinson
    The Years of Rice and Salt
DRM-infested Books/420281728.epub
    Richard Wiseman
    59 Seconds: Think a Little, Change a Lot
DRM-infested Books/420445771.epub
    Isaac Asimov
    Foundation’s Edge
DRM-infested Books/420446058.epub
    Isaac Asimov
    Foundation and Earth
DRM-infested Books/420725428.epub
    Mike Resnick
    Kirinyaga: A Fable of Utopia
DRM-infested Books/421025353.epub
    William Dalrymple
    The Last Mughal
DRM-free     Books/421124117.epub
    Brust, Steven
    The Desecrator
DRM-infested Books/422530144.epub
    Max Barry
    Machine Man
DRM-free     Books/422718511.epub
    Apple Inc.
    Mac Integration Basics
DRM-free     Books/426914658.epub
    Brust, Steven
    Five Hundred Years After
DRM-free     Books/428235697.epub
    Vinge, Vernor
    A Fire Upon The Deep
DRM-infested Books/429173089.epub
    Ursula K. Le Guin
    The Other Wind
DRM-infested Books/429173713.epub
    Ursula K. Le Guin
    Tales from Earthsea
DRM-infested Books/429699133.epub
    Rajaniemi, Hannu
    The Quantum Thief
    This is published by Tom Doherty and could be re-downloaded DRM-free
DRM-infested Books/431617578.epub
    Walter Isaacson
    Steve Jobs
DRM-infested Books/432519291.epub
    Susan Weinschenk
    100 Things: Every Designer Needs to Know About People
DRM-infested Books/434509014.epub
    Stross, Charles
    Rule 34
DRM-free     Books/434522188.epub
    Larry Niven, Jerry Pournelle
    The Mote In God’s Eye
DRM-free     Books/434811509.epub
    Asher, Neal
    Cowl
DRM-infested Books/436646026.epub
    Neal Stephenson
    Reamde
DRM-infested Books/436691174.epub
    Julia Child
    Mastering the Art of French Cooking
DRM-infested Books/443149884.epub
    Daniel Kahneman
    Thinking, Fast and Slow
DRM-infested Books/446155927.epub
    Esslemont, Ian C. C.
    Stonewielder
    This is published by Tom Doherty and could be re-downloaded DRM-free
DRM-free     Books/447591195.epub
    Asher, Neal
    The Skinner
DRM-infested Books/454252718.epub
    William B. Norton
    The Internet Peering Playbook: Connecting to the Core of the Internet
DRM-infested Books/455525627.epub
    Amar Chitra Katha
    Birbal The Genius
DRM-infested Books/458461362.epub
    Pamela Druckerman
    Bringing Up Bebe
DRM-free     Books/45B90418E467D479DCDDF23B932C648C.epub
    Douglas Crockford
    JavaScript: The Good Parts
DRM-infested Books/460822066.epub (directory)
    Scott Lynch
    The Republic of Thieves
DRM-free     Books/465679A557523FDB836005CF4BB9380E.epub
    Scott Berkun
    Mindfire
DRM-infested Books/479594044.epub
    Saladin Ahmed
    Throne of the Crescent Moon
DRM-infested Books/479717436.epub
    David Crist
    The Twilight War: The Secret History of America’s Thirty-Year Conflict with Iran
DRM-infested Books/479771801.epub
    William Dalrymple
    In Xanadu
DRM-infested Books/489957500.epub
    Bruce Schneier
    Liars and Outliers
DRM-infested Books/491186678.epub
    James Blish
    Cities in Flight
DRM-infested Books/491668459.epub
    Neal Asher
    Shadow of the Scorpion
DRM-infested Books/491669284.epub
    Glen Cook
    A Matter of Time
DRM-infested Books/491669288.epub
    Glen Cook
    Darkwar
DRM-free     Books/492199230.epub
    Frederik Pohl
    The Tunnel Under the World
DRM-free     Books/492199569.epub
    Frederik Pohl
    The Knights of Arthur
DRM-infested Books/494939678.epub
    Esslemont, Ian C. C.
    Orb Sceptre Throne
    This is published by Tom Doherty and could be re-downloaded DRM-free
DRM-infested Books/498634992.epub
    Charles Stross
    The Apocalypse Codex
DRM-infested Books/499392787.epub
    Daniel Suarez
    Kill Decision
DRM-free     Books/4AE7DDA54BEEFCD65157927546B18063.epub
    Roberto Ierusalimschy
    Programming in Lua 2ed
DRM-free     Books/4DCFF682728B765A1CE221F3D7C21536.epub
    Glen Cook
    Starfishers Volume 3: Stars’ End
DRM-free     Books/501278407.epub
    Colette
    Chéri
DRM-free     Books/501758197.epub
    David Brin
    Existence
DRM-infested Books/501758516.epub
    Scalzi, John
    Redshirts
    This is published by Tom Doherty and could be re-downloaded DRM-free
DRM-infested Books/503019669.epub
    J.R.R. Tolkien
    The Lord of the Rings
DRM-infested Books/503153300.epub
    J.R.R. Tolkien
    Tales from the Perilous Realm
DRM-infested Books/503154129.epub
    J. R. R. Tolkien and Christopher Tolkien
    The Book of Lost Tales, Part One
DRM-infested Books/503154991.epub
    J. R. R. Tolkien and Christopher Tolkien
    The Book of Lost Tales, Part Two
DRM-infested Books/503155327.epub
    J.R.R. Tolkien
    The Children of Húrin
DRM-infested Books/503163148.epub
    J.R.R. Tolkien
    The Hobbit Deluxe
DRM-infested Books/503164678.epub
    J.R.R. Tolkien
    The Silmarillion
DRM-infested Books/503167303.epub
    J.R.R. Tolkien
    Unfinished Tales of Númenor and Middle-earth
DRM-infested Books/504209078.epub
    Isaac Asimov
    Prelude to Foundation
DRM-infested Books/504371982.epub
    Iain M. Banks
    The Hydrogen Sonata
DRM-free     Books/511060740.epub
    Frederik Pohl
    The Hated
DRM-free     Books/511143617.epub
    Frederik Pohl
    The Day of the Boomer Dukes
DRM-free     Books/511252896.epub
    Frederik Pohl
    Pythias
DRM-free     Books/513357605.epub
    Hannu Rajaniemi
    The Fractal Prince
DRM-free     Books/513868CF0AA46D293EE27F74BC399760.epub
    Jerry Pournelle
    West of Honor
DRM-infested Books/520233773.epub
    Nate Silver
    The Signal and the Noise
DRM-free     Books/520897403.epub
    Hugh Howey
    Wool Omnibus
DRM-free     Books/525170910.epub
    Cory Doctorow and Charles Stross
    The Rapture of the Nerds
DRM-infested Books/526136048.epub (directory)
    Hetty van de Rijt & Frans Plooij
    The Wonder Weeks
DRM-infested Books/529020127.epub
    Neal Asher
    The Departure
DRM-infested Books/529424632.epub
    Guy Gavriel Kay
    The Lions of Al-Rassan
DRM-infested Books/536312376.epub
    Glen Cook
    Garrett for Hire
DRM-free     Books/537023027.epub
    Erikson, Steven
    Forge of Darkness
DRM-infested Books/541673159.epub
    Ursula K. Le Guin
    The Tombs of Atuan
DRM-infested Books/541673162.epub
    Ursula K. Le Guin
    The Farthest Shore
DRM-infested Books/546126326.epub
    Jan Morris
    HAV
DRM-infested Books/551241606.epub
    Ursula K. Le Guin
    A Wizard of Earthsea
DRM-infested Books/551567785.epub
    Murray R. Spiegel, PhD
    Schaum’s Outline Mathematical Handbook of Formulas and Tables, Fourth Edition
DRM-infested Books/551747038.epub
    Chris Hedges and Joe Sacco
    Days of Destruction Days of Revolt v2b
DRM-infested Books/552144691.epub
    Tamim Ansary
    Games without Rules
DRM-free     Books/553878102.epub
    Charles Stross
    A Tall Tail
DRM-infested Books/563408849.epub
    Iain Banks
    Stonemouth
DRM-infested Books/568731449.epub
    William Dalrymple
    Return of a King: The Battle for Afghanistan, 1839-42
DRM-infested Books/569232538.epub
    Neil Gaiman
    The Ocean at the End of the Lane
DRM-free     Books/571678152.epub
    Glen Cook
    The Return of the Black Company
DRM-free     Books/571678945.epub
    Glen Cook
    The Many Deaths of the Black Company
DRM-free     Books/573656304.epub
    Glen Cook
    Chronicles of the Black Company
DRM-free     Books/573656441.epub
    Glen Cook
    The Books of the South
DRM-free     Books/576233114.epub
    Jack Vance
    Demon Princes
DRM-infested Books/578851675.epub
    Zilpha Keatley Snyder
    Below The Root
DRM-infested Books/580642602.epub
    Max Barry
    Lexicon
DRM-infested Books/588794444.epub
    Charles Stross
    Neptune’s Brood
DRM-free     Books/5CCB889F91585202637A3C5FBFD8409F.epub
    Glen Cook
    Starfishers-The Starfishers Trilogy Volume II
DRM-infested Books/600938002.epub
    Gardner Dozois
    The Year’s Best Science Fiction: Thirtieth Annual Collection
DRM-free     Books/606232503.epub
    Ian C. Esslemont
    Blood and Bone
DRM-free     Books/610920977.epub
    Robert Jordan and Brandon Sanderson
    A Memory of Light
DRM-free     Books/6122FF30560866BD75257E6CCC264371.epub
    Fritz Leiber
    Swords and Ice Magic-Fafhrd and the Gray Mouser-Book 6
DRM-infested Books/619483561.epub
    Raymond E. Feist
    Magician’s End
DRM-free     Books/622837311.epub
    Le Comte De  Lautréamont
    Les chants de Maldoror
DRM-free     Books/62497334A4189B1D00E9BDFD95724E2E.epub
    Fritz Leiber
    The Knight and Knave of Swords-Fafhrd and the Gray Mouser-Book 7
DRM-infested Books/645571245.epub
    Iain Banks
    The Quarry
DRM-free     Books/647688922.epub
    Steven Brust and Skyler White
    The Incrementalists
DRM-infested Books/651331715.epub
    Susan Crawford
    Captive Audience
DRM-infested Books/654456347.epub
    Iain Banks
    The Wasp Factory
DRM-infested Books/662310218.epub (directory)
    Various Authors
    Star Wars: Empire Volume 3 – The Imperial Perspective
DRM-infested Books/662310219.epub
    Paul Gulacy
    Star Wars: Crimson Empire
DRM-infested Books/664297397.epub (directory)
    Various Authors
    Star Wars: Empire, Vol. 4: The Heart of the Rebellion
DRM-infested Books/664343525.epub (directory)
    Paul Chadwick, Doug Wheatley & Tomás Giorello
    Star Wars: Empire, Vol. 2: Darklighter
DRM-infested Books/664894567.epub
    John Ostrander
    Star Wars: Dawn of the Jedi Volume 1—Force Storm
DRM-infested Books/664910993.epub (directory)
    Scott Allie, Ryan Benjamin & Brian Horton
    Star Wars: Empire Vol. 1
DRM-free     Books/666772E4C21E2017E71C10F1990840BC.epub
    Pieter Hintjens
    ZeroMQ - Connecting your Code
DRM-infested Books/674224604.epub (directory)
    Various Authors
    Star Wars: Empire, Vol. 5: Allies and Adversaries
DRM-infested Books/674226603.epub (directory)
    Thomas Andrews, Scott Allie & Various Authors
    Star Wars: Empire, Vol. 6: In the Shadows of Their Fathers
DRM-free     Books/697938901.epub
    Charles Stross
    Equoid: A Laundry Novella
DRM-free     Books/6BBD61A64A46FFFF4AE7C5FCEB9CFCEE.epub
    David Drake
    Balefires
DRM-free     Books/71C79A253586282ABE73C2975237EB08.epub
    Glen Cook
    Sung in Blood
DRM-free     Books/73BC5DD0E51611BDC359CBAB48CC203F.epub (directory)
    Crane, Stephen
    The Red Badge of Courage
DRM-free     Books/7A0D1AEE343638A8A3769DABB90CD4CB.epub
    Clay A. Johnson
    The Information Diet
DRM-free     Books/7B46914B55481714DED3D711288978FA.epub
    Glen Cook
    Shadowline-The Starfishers Trilogy I
DRM-free     Books/809DCC75FE60E749458CB27636EE6777.epub
    Mercedes Lackey
    The Secret World Chronicle
DRM-free     Books/8350F5DFECCD6AA88714400FDF4F6831.epub
    François de La Rochefoucauld
    Réflexions ou Sentences et Maximes Morales
DRM-free     Books/853BD1C65277D276BA09E04EBFEB73EF.epub
    None
    PostgreSQL 9 Administration Cookbook
DRM-free     Books/85907063B2351E91C6E7F5052C090BFD.epub
    None
    PostgreSQL 9.0 High Performance
DRM-free     Books/8663E7BF735C06318FF450532A67F1C2.epub
    Glen Cook
    Passage at Arms
DRM-free     Books/8690D69995483C5D2DF6AD38BE53C1D7.epub
    Paolo Bacigalupi
    The Windup Girl - Second Electronic Edition
DRM-free     Books/8993920EF1C0E8FE8CB47A20BD955F53.epub
    Fritz Leiber
    Swords Against Death-Fafhrd and Gray Mouser-Book 2
DRM-free     Books/8DC6ED0A99161EBB516E370A60ED1121.epub
    Jonathan Zdziarski
    Hacking and Securing iOS Applications
DRM-free     Books/8DD0D22CD05B0B8D52DF9DB93FC8616B.epub
    Fritz Leiber
    Swords in the Mist-Fafhrd And the Gray Mouser-Book 3
DRM-free     Books/912CB8B110736684549EAC4FC36665AB.epub
    Neil Gaiman and Dave McKean
    Signal to Noise
DRM-free     Books/93061FDFD6EEC01FD8CE4295A049C97C.epub
    Cory Doctorow
    Homeland
DRM-free     Books/9592D5001632B94D1FEFC98B3A40E049.epub
    Kelly Link
    Stranger Things Happen
DRM-free     Books/990CA5799084407151488A7C563DF269.epub
    Tom Hughes-Croucher
    Node: Up and Running
DRM-free     Books/9CCCA9E217602948B84BE9A7A21C2753.epub
    Mike Resnick
    Birthright: The Book of Man
DRM-free     Books/9DA2C8D7E941C37FA83192E5849AA850.epub
    Q. Ethan McCallum
    Parallel R
DRM-free     Books/A7DB557515983DB67F74E2BE351FC319.epub
    Glen Cook
    Reap the East Wind
DRM-free     Books/B18AB85267B815E540C7E370F8D97726.epub
    Lars George
    HBase: The Definitive Guide
DRM-free     Books/B1A586D24AE5F2450010F8664F8E059D.epub
    Cory Doctorow
    Pirate Cinema
DRM-free     Books/B562B5D74F8677C946B0A6F81EB34F1B.epub
    Gotthold Ephraim Lessing
    Nathan the Wise; a dramatic poem in five acts
DRM-free     Books/BE42D426242836AA171539B7415732E6.epub
    Glen Cook
    The Swordbearer
DRM-free     Books/BE80AD93781746E45CF95607A7BEE687.epub
    Charlie Stross
    Bit Rot
DRM-free     Books/C39FBC59673713A57D404D62BE85C4DC.epub
    Lauren Beukes
    Zoo City
DRM-free     Books/C81CBBD470EFDBC62359BDAD12FDF551.epub
    Thomas Hobbes
    Leviathan
DRM-free     Books/D17DC1A83BD90AFB24055A01831BFAB4.epub
    Glen Cook
    The Dragon Never Sleeps
DRM-free     Books/DA8FBAB386EC503A0EF21E481D214521.epub
    David Flanagan
    JavaScript: The Definitive Guide
DRM-free     Books/DBAD0DE7A4305B90F6AC33C673FE68E8.epub
    Glen Cook
    A Path to Coldness of Heart
DRM-free     Books/DBCCB54B140D1158250B24B7E3E81B63.epub
    Scott Chacon
    Pro Git
DRM-free     Books/DF6C56F9A8719294B13BA91BED5E1667.epub
    Mike Resnick
    Ivory
DRM-free     Books/E8D76FDD351E68D92AE4A3F3AEA7EC6A.epub
    Paolo Bacigalupi
    Pump Six and Other Stories
DRM-free     Books/EEE65A68378C2510E18DF24CD767AC9A.epub
    Fritz Leiber
    Swords Against Wizardry-Fafhrd and the Gray Mouser-Book 4
DRM-free     Books/F3BC14466A16C96CCD8FFE00DCCF8147.epub
    Glen Cook
    An Empire Unacquainted With Defeat
DRM-free     Books/F534835595041374B814D151A633E69D.epub
    Peter Watts
    Blindsight
DRM-free     Books/F8BF760284ADC800B290DCA6D8EA7EF2.epub
    Ben Klemens
    21st Century C
DRM-free     Books/FE8C57863E40B98CB732FEE4BFDB60BB.epub
    Glen Cook
    An Ill Fate Marshalling
8 books are DRM-infested
26 could be cured

Update (2013-11-06):

OS X 10.9 Mavericks and the new iBooks app changed the location of the iBooks directory, I changed my script accordingly (and made it adjust depending on which OS version you have). Also, the file names have changed and no longer embed author and title, so I am extracting them from the XML metadata files.

The Gresham’s law of Amazon Web Services

In the bad (good?) old days when currency’s worth was established by the amount of gold or silver in coinage, kings would cut corners by debasing currency with lead, which is almost as dense as gold or silver. In the New World, counterfeiters debased gold coins with platinum, which was first smelted by pre-columbian civilizations. Needless to say, the fakes are now worth more than the originals.

The public was not fooled, however, and found ways to test coins for purity, including folkloric ones like biting a coin to see if it is made of malleable gold, rather than harder metals. People would then hoard pure gold coins, and try to rid themselves of debased coins at the earliest opportunity. This led to Gresham’s Law: bad money drives out good money in circulation.

After a year of using Amazon Web Services’ EC2 service at scale for my company (we moved to our own servers at the end of 2011), I conjecture there is a Gresham’s Law of Amazon EC2 instances – bad instances drive out good ones. Let me elaborate:

Amazon EC2 is a good way to launch a service for a startup, without incurring heavy capital expenditures when getting started and prior to securing funding. Unfortunately, EC2 is not a quality service. Instances are unreliable (we used over 80 instances at Amazon, and there was at least one instance failure a week, and sometimes up to 4). Amazon instances have poor disk I/O performance that makes them particularly unsuitable to hosting non-trivial databases (EBS is even worse, and notoriously unreliable).

Performance is also inconsistent—I routinely observed “runt” m1.large instances that performed half as well as the others. We experienced all sorts of failure modes, including disk corruptions, disks that would block forever without timing out, sporadic losses of network connectivity, and many more. Even more puzzling, I would get 50% to 70% failure rate on new instances that would not come up cleanly after being launched.

Some of this is probably due to the fact we use an uncommon OS, OpenSolaris, that is barely supported on EC2, but I suspect a big part of this is that Amazon uses low-end commodity parts, and does not proactively retire failed or flaky hardware from service. Instances that have the bad luck of being assigned to flaky hardware are more likely to fail or perform poorly, and thus more likely to be be destroyed, released and a new one reassigned in the same slot. The inevitable consequence of this is that new instances have a higher likelihood of being runts or otherwise defective than long-running ones.

One work-around is to spin up a large number of instances, test them, and destroy the poor-performing ones. AWS runts are usually correlated with slower CPU clock speeds, as older machines would be running older versions of the Xen hypervisor Amazon uses under the hood, have less cache, slower drives and so on. Iterating through virtual machines as if you are picking melons at a supermarket is a slow and painful job, however, and even their newer machines have their share of runts. We were trying to keep only machines with 2.6 or 2.66GHz processors, but more than 70% of the instances we were getting assigned were 2.2GHz runts, and it would usually take creating 5 or 6 instances on average to get a non-runt.

In the end, we migrated to our own facility in colo, because Amazon’s costs, reliability and performance were just not acceptable, and we had long passed the threshold beyond which it is cheaper to own than rent (I estimate it at $5,000 to $10,000 per month Amazon spend, depending on your workload). It is not as if other cloud providers are any better—before Amazon we had started on Joyent, which supports OpenSolaris natively, and their MTBF was in the order of 2 weeks, apparently because they replaced their original Sun hardware with substandard Dell servers and had issues with power management C-states in the Dell server BIOS.

The dirty secret of cloud services is that there is no reliable source of information on actual performance and reliability of cloud services. This brings out another economic concept, George Akerlof’s famous paper on the market for lemons. In a market where information asymmetry exists, the market will eventually collapse in the absence of guarantees. Until Amazon and others offer SLAs with teeth, you should remain skeptical about their ability to deliver on their promises.

Withings smart baby monitor review

One of the joys challenges of being a first-time parent is being exposed to a bewildering array of gadgets and equipment required to care for the baby, from baby car seats, strollers and diaper pails to 2-axis rocking robots (thanks Rohit!). There is an entire cottage industry of books like Baby Bargains that help you navigate through the confusing and sometimes questionable or outright unnecessary choices.

I have a Withings body weight scale that I really like and I was excited to learn they were going to release a networked video baby monitor. It took a while to get to market in the US, however, so in the interim I purchased a Philips Advent DECT digital baby monitor, which ended up unusable in practice, because its microphone sensitivity is so poor that you can barely hear anything. When the Withings baby monitor finally became available in the US, I immediately ordered it.

Withings is clearly taking design cues from Apple, from the lavishly designed packaging to the glossy white plastic RoundedRect aesthetic and the use of a magnetic clip to attach the baby monitor to the crib. The clip is serviceable, but the magnets are not quite strong enough to hold the unit firmly onto the crib. I would not trust it to keep the monitor from toppling when the baby grows and kicks at the crib. Fortunately they also include a flip-out tab on the base of the unit that can be inserted into a slit on the clip to prevent sliding, although it is not obvious and it took me a while before I discovered this key feature.

The wall wart is a generic black model with swappable AC prongs for international markets, and detracts from the overall package, but since the monitor has a micro-USB input, you can always use another standard AC to USB type A adapter like the iPhone’s, with a USB type A to micro-USB cable. A rechargeable battery is included, with 2 hours’ claimed life, I did not verify that spec.

The initial out of the box experience is good: you connect to the device from your iPhone or iPad using Bluetooth (no messing around with a USB cable as with the Withings scale), enter the WiFi settings in the Withbaby app, and then use WiFi to access the device afterwards. It is as streamlined an experience as you can expect without a keyboard on the unit. There is also an Ethernet jack (it is unclear whether it supports power over Ethernet), but my house was built in 1928 and is not wired upstairs where the baby lives.

Once you enter your credentials into the app, it connects to the monitor and shows you the video and sound. If you put it in the background, you have the option of monitoring audio. Withings will also send you alerts via push notifications if the temperature or humidity is excessive, or if it detects noise or motion. The default settings are way too twitchy, however, and you will find yourself disabling audio notifications as the deluge of alerts is just too much.

The device includes a night light with selectable color, a lullaby player, and the ability to speak to your baby, all controlled through the app. At the front you also have touch controls to turn some of these features on. This is actually a bad idea, as on two occasions I started the lullaby by accident as I was fumbling with it in a dark room, and woke up my baby as a result. Another design flaw is the pulsing blue night light when the unit is rebooting, the Airport Express like amber/green status LED in the back is quite sufficient. Frankly the only one of these features that is useful is the speaker, and the ability to stream from your music collection, such as Dr. Harvey Karp’s white noise selections would be preferable to the canned lullabies.

The video camera is advertised as having a 3 megapixel sensor. It has a wide-angle lens and you can “pan” using the usual iPhone or iPad gestures. The lens is a fixed-focus plastic one, and optical clarity is so-so at best, optimal focus seems to be at 50cm or so. One great feature is the monitor has a normal and night vision mode, similar to the one on some Sony HAD camcorders, with an IR illuminator that provides light for the night vision mode. This means you can watch your baby toss and turn in an otherwise pitch-black room.

You can use the baby monitor from outside your network, and it works fine, even over a 3G connection. Withings allows you up to 15 minutes per day, anything beyond that requires paying them $6 for each 100 minutes. Coming on top of an already expensive device, this seems like a naked money grab from anxious parents. (Updated 2012-09-29: remote monitoring is now free and unlimited).

When the unit works, it is absolutely great: good sound sensitivity and the video feature mostly works as advertised. Unfortunately it frequently does not function, and I find myself performing a hard reboot by removing the battery far more often than I would like. Among the pathologies:

  • Once it falsely reported the unit was closed and thus video inaccessible
  • Once the camera was in a frozen state, it took a power cycling to get the video moving again.
  • Yesterday I could not connect at all, no matter how many times I rebooted my Airport Extreme, the monitor and my wife’s or my iPads. Some detective work using a packet sniffer showed the app was trying to connect to babyws.withings.net using HTTP, which is aliased to s11.withings.net, and that server was down. Some of the documentation suggests you can use the Bluetooth connection to access the monitor, but I was not able to figure out how to do this.

This brings me to a crucial point. The baby monitor is a safety device, and it is utterly unacceptable for its functioning to be dependent on a cloud service, which can and will be a single point of failure. It should use Bonjour or similar discovery methods to work on the LAN, and rely on Withings’ servers only when accessing it from outside the home LAN’s perimeter. I wonder if Withings’ eagerness to nickel-and-dime users by charging for outside monitoring led to this critical design flaw.

The bottom line is the Withings smart baby monitor is a very frustrating device, with its obvious potential marred by failures of execution. If it worked consistently, it would be a top-notch product worthy of its Apple inspiration and lofty price tag, but the general lack of reliability means I cannot recommend it until the bugs are ironed out. Consider it an alpha release at best.

Update (2012-09-17):

Here’s how to make the Withings not-so-smart baby monitor more usable:

  • Remove the battery from the unit and hook up the micro-USB power adapter to a Belkin WeMo remote-controlled power switch. This allows you to power-cycle the baby monitor remotely from the same iPhone or iPad you are using the monitor software on.
  • Hide the blue led with gaffer’s tape. This prevents the blue light on reboot from waking the baby. Unlike duct tape, gaffer’s tape can be removed without leaving glue residue, although the aesthetics of dark gray gaffer’s tape on the gleaming white unit are questionable at best.
  • I haven’t tried covering up the touch controls with gaffer’s tape, which would eliminate the risk of triggering a jingle and waking the baby. The WeMo eliminates the need to enter the room and tinker with the baby monitor.

It’s quite sad to have to pay an extra $50 to work around buggy hardware and software, but it makes a big difference.

Update (2013-05-20):

The micro-USB connector failed and the baby monitor is now essentially a doorstop. Not surprising given how flimsy micro-USB is, compared to mini-USB, for insignificant space savings. Micro-USB was a Nokia design rammed through the USB-IF. In theory it has better insert-remove cycle life than mini-USB, but in practice I’ve never had mini-USB fail, whereas it is a frequent occurrence with micro-USB.

Update (2015-08-17):

USB-C is an improvement over micro-USB, hopefully some future version of the baby monitor will use it. Still nowhere near as robust as Lightning or tip-ring-sleeve, though.

Welcome, baby Afsheen!

My first child, Afsheen Zahra Majid, formerly code-named “Lolo”, was born Tuesday 2012-01-03 at 10:47 AM Pacific Time in San Francisco. She weighed 4,148 grams (9.2lbs) and was 56.5cm long (20.25″) at birth. She has a full head of hair, strong lungs, and bright shiny eyes that were wide open when she came out.

Both mother and child are exhausted and recovering at California Pacific Medical Center, and learning to know one another. I can only watch with rapt attention at the miracle of life unfolding, bursting as I am with joy.

Prints not-so charming

Ansel Adams wrote a celebrated series of photo instructional books. It is organized as a trilogy: The Camera, The Negative and The Print.

Of these, the camera business is still going strong, buoyed by sales of digital cameras and upgraders to DSLRs (although market saturation looms). Negatives are an endangered species as digital photography largely supplanted film. Prints are still going strong, whether using traditional silver halide, inkjet or offset printing, but that probably won’t stave off Kodak’s impending demise.

Last year, I wrote to X-Rite to complain that their software for the Colormunki color calibrator wouldn’t let me profile the full brightness range of my monitor, and limited it to an artificial ceiling designed around the limitations of print. Their response:

The ColorMunki software will not allow a range over 140cd/m luminance value.  This would basically defeat the purpose of using a device to measure luminance when you set the values this high.  While your monitor can be extremely bright however using a value of 300+ would allow your monitor to show you a brightness value that would negatively affect your prints and they would end up coming out so dark that you would be unable to see them.  The industry standard for photographic works is to use the luminance values setup at anywhere from 80 – 120cd/m.  That said, if your ambient light conditions are ranging very high then you should probably be using a value of 120cd/m which is the industry standard and what we recommend.

In the ColorMunki software there is an option to set the luminance according to your ambient light conditions.  This is a pretty high threshold in the software where as you have to read somewhere around 350-400 lux to get a target luminance above the 80 threshold.  If it does go above this value then you certainly will want to stay within the 120 range.  Anything below that will give you a target value of 80.

Implicit in this response is the assumption the only reason why you would want to calibrate your monitor is so you can make more accurate prints. Prints only have a 100:1 or so contrast ratio, compared to LCD monitors for which the static contrast ratio is closer to 1000:1, and plasma TVs go higher yet. This is why images on screen look more vibrant and punchy than prints, as did slides back in the day.

If Apple does indeed come up with the oft-rumored iPad 3 with double the resolution of current models, or 2048×1536 at 260ppi, you would have an easily portable display device with near gallery-grade resolution and massive storage capacity. At that point, prints would lose the last advantage they still hold over digital display technologies.

Image display on screen is the new normal, and X-Rite needs to get with the program. I want to see the full dynamic range and contrast of my images on my screen, not a Hobson’s choice between inaccurate color and crippling them with limitations from legacy print technology.