Posted on

The “True” Corpus of American English

In an analysis of the Corpus of Historical American English (COHA) and Google Books, Professor Mark Davies of Brigham Young University studies the effectiveness of both engines and their ability to properly read the English language. Both are corpuses of the American English language, but Google Books has 500 billion words compared to COHA’s 400 million. Davies argues that although COHA has a significantly smaller database, the trending patterns still mirror those of Google Books. Because of this, Davies argues that COHA is actually the more effective corpus—a smaller database means far fewer data to sift through and that means quicker search results and faster information.

COHA’s “toys” are what make it a more useful database in Davies’s eyes. While Google provides the same basic function (showing the frequency of word usage throughout the decades), COHA is able to track concepts, related words, and changes in meaning. Whereas Google tends to have a one-track mind, just like it’s general search engine, COHA manages to “think” about relations for the words being placed into the search. Looking for things such as relation, form, root words, or even cultural shifts, the searches are much more comprehensive.

Design is a huge issue for some researchers, and with this in mind, Google definitely has the upper hand. True, as Davies puts it, COHA is able to effectively portray the same statistics, but I had a genuinely hard time navigating the site. Bar graphs are nice for portraying how many dark-haired, blue-eyed people are in a class, but for a corpus of American English, I found them rather ineffective. Google Books had a much more pleasing site, nicer to the eye, and easier to follow the pattern of the language. Yes, COHA’s tables are nice for their alternative searches, but as I said, traversing the site is actually rather difficult. Google provides a much more streamlined site, and provides actions that are easy to follow

Unfortunately for Google, I think the amount of words in the database have made it impossible to create proper analyses for the grammar, word meanings, and word foundations that COHA is successfully able to analyze. If Google Books created an efficient way to sift through all of that information quickly enough, then it would immediately become the preferred site. However, because of its inability to process the large amounts of data (largely its own fault) has rendered it ineffective to be a “true” corpus of the American English language.

 

Posted on

Millions of Digitized Books, Hundreds of Fascinating Conclusions

Jean-Baptiste Michel et al.’s short and sweet article Quantitative Analysis of Culture Using Millions of Digitized Books raises a number of bold points that show just how valuable Google’s bold (and originally considered foolhardy) Google Books project has been to historians.  The project uses nearly 5.2 million books (over 4% of those written, a very significant standard) containing over 5 billion words and searches them.  Let’s pause for a minute and think about what that means.  25 years ago, or even 10 years ago, if you said you wanted a search through a sizable sample of every book ever written for certain words, you would have had your head examined.  The paper points out that it would take a human 80 years just to read all the books written in one year, 2000.  Here’s this device that can go through the entire corpus in literally quicker than a blink of the eye.  Roughly 2/3rds of the 500 billion words are in English and there’s only a significant sample size for the books 1800 and on (though there are a fair amount from 1600-1800), but even with these limitations, the work allowed the researchers to come to some bold conclusions.
“What conclusions?” you ask?  Try this one on for size: they estimate that most dictionaries might only contain as little as 52% of the living lexicon at any given moment.  They estimate the total lexicon of 1-grams (single words, excluding symbols, numbers, typoes, etc) at 544k in 1900, 597k in 1950, and 1,022,000 in 2000 (counting n-grams that are used over 1/1,000,000,000 of all English words).  Some of these are not in dictionaries due to dictionaries’ traditional dislike of compound words, but others are inexcusable (they point to “deletable” as a particularly ironic example).  This lexical “dark matter,” in their charming expression, are words that are fresh for research.  No OED biography has ever examined every facet of these words, and no amount of looking up will find them.  The n-gram has saved these potentially valuable expressions from the invisibility of  their hidden nature.

 

Another bold feature the n-gram allows is to trace the rise and fall of terms over time.  Much has been made of the example of the engram for “World War I” vs “Great War,” where Great War holds strong until 1939, then falls off, while World War I rises to pick up the slack, but it’s hardly the only example.  You can do the n-gram test yourself and see the decline of a good many words and phrases, and the introduction of others.  Ever been curious to see if anyone said “Yadda-yadda-yadda” before Jerry Seinfeld?  Want to map “Reality Television” vs. “Situational Comedy” and see if you can identify the year Survivor was released?  Want to compare Claude Lamarck with Charles Darwin or Karl Marx with Sigmund Freud?  The world is your oyster.

 

The n-gram can also detect the death of older, archaic forms of words.  “Spilled” is becoming the past tense of “to spill,” but there is no use in crying over spilt milk about it, spilt had a long run.  Contemporary spouters of aphorisms think that all that glitters is not gold, but their fathers sagely opined that all that [i]glisters[/i] is not gold.  Indeed, past tense verbs that end in “t” are fighting a slow, steady, losing battle against “ed.”  Can they survive?  I feel I’ve spoilt the ending of this struggle, but I’ve been burnt on these predictions before.

 

The final section of the article struck (or will it become “striked?”) a more somber note: repression.  Examining the use of the word “Trotsky” in Russian language sources through the 1920s tells a harrowing tale, but everyone expected as much.  (I wanted to run a similar test on “New Economic Policy” vs “Five Year Plan,” but, alas, I speak no Russian, and the English results are pretty meaningless).  What is more interesting is the revelation of people never before suspected of repression.  The Nazi regime’s list of degenerate artists was apparently far more extensive than generally known, as people never included in the traditional narrative saw their mentions in German press fall off the face of the earth in the late 1930s.  Again, this was just a cursory exercise: this n-gram search opens up the possibility of a new way of looking both at the more blatant Nazi/Soviet repression, and the more subtle blacklisting preferred in the West.  There are millions of possibilities that n-grams open up for these millions of books.

Posted on

The Persistence of the Wasteland

I thought I’d give an update on my project.  You will recall that I’ve been using the Fallout series as a benchmark for examining changes in American nuclear culture from 1945-2011.  It is striking how prevalent the image of the high desert is in current American concepts of a post-nuclear world.  Below are just a few of the photographs from games and movies.  It is simply impossible to display a post-nuclear world without reference to what Jeffrey Womack calls “The Landscape of Death.”  Yet, significantly there is little to no use of this landscape trope prior to the early 1970s.  My argument is that this is the result of the environmental movement, a reinvigorated anti-nuclear/disarmament movement, and most importantly the release and wide dissemination of films and images from the New Mexico and Nevada above ground nuclear tests, which permanently associated the high desert and the mushroom cloud together in American minds.  There is, in fact an almost total switch of thematic focus.  Most of the books of the mid to late 50s feature a decimated or extinct humanity in a pristine world, a world wiped clean by bombs.  The latter movies, books and games feature a resilient, surviving and tenacious humanity in a world utterly devoid of nature.  This changing focus speaks to larger fears about the affect of technology on our environment which simply was not a part of the zeitgeist prior to 1970.

 

Also, notice the theme of the barren road, and the loan traveler.  I’m not sure how to interpret why that image is so striking, and used so repeatedly.  Anyone have any ideas?

Posted on

Project Started!

Hey guys!

Here is the link to my video game blog : http://pixellatedculture.blogspot.com/

The first post is about Fallout 3 for those interested in the Fallout-verse, but knowledge of the game is not entirely necessary.

I chose Blogger as it is part of Google and links up to a bunch of other services. Blogger also gave me a bunch of tools to put on my blog, as in links to Twitter and Facebook, putting my own Twitter on the site, a page view tracker, an RSS button for subscribers and more! There’s even an option for Ad Sense, which I opted out of. Blogger also has a variety of templates and great customization. I wasn’t feeling to creative, so I just tweaked one of the basic ones. It was super easy to set up and get started. Blogger also has a button at the top for “Next Blog” so if you’re feeling random it takes you to another blog, thus potentially increasing my traffic. The only trouble I has was hyperlinks. Oh, they work and it’s fairly intuitive to get them in there, it’s just the field won’t let you copy the link in so you have to type it yourself. There is a “test the link” feature so you know if you typed it in right, but it’s still mildly annoying.

It’s a little early to tell how successful this will be, but I have hope. I didn’t realize Blogger has so many features when I signed up so that was a pleasant surprise. Anyway, check it out and leave comments telling me what you think, either here or at the blog itself.

Edit:

So now I have two posts up. The second is on Bioshock. So far, I haven’t gotten any comments, but I have gotten plenty of views. For the rest of the process for the project time limit, I do have the list of games I want to speak about. I do plan to keep the alternating schedule of history week vs lit week.

However, I do desire comments as I want this blog to be an interactive experience. I want people to leave suggestions and give me ideas. I’m hoping that I can encourage people to leave a comment and this is just new blog jitters or something like that. I also recently redesigned by template to look a little more classy and the background isn’t so distracting.

I’m also having trouble deciding format. I’m thinking my blog looks a little bland because the it doesn’t have pictures. However, I’m afraid that putting too many pictures would disrupt the flow. Right now I have hyperlinks for aspects of the game which are important. What’s the best way to go about the picture issue?

Posted on

Database as a Genre of New Media

Lev Manovich is an accomplished thinker in the field of new media.  In his short piece, “Database as a Genre of New Media,” he makes the case that databases represent a fundamental paradigm shift in the way that people think about the organization and presentation of information.  Databases as a non-narrative, not necessarily linear way of organizing data did not originate with the digital age – they were found previously in, say, encyclopedias or photo archives – but they have experienced a renaissance in that time.  Video games, your hard drive, and the Internet are all databases, and they all represent a way to present data free of the constraints of logic and coherence imposed by the narrative form.

As Manovich puts it, “As a cultural form, database represents the world as a list of items and it refuses to order this list. In contrast, a narrative creates a cause-and-effect trajectory of seemingly unordered items (events). Therefore, database and narrative are natural enemies.”  He argues that the very term “narrative” is abused in the interactive databases of the Internet and video games, where users may respond to preprogrammed variables, whether they are hyperlinks or Koopa Troopas.  A narrative is something carefully constructed by its author constituting “a series of connected events caused or experienced by actors.”  It is careless to assume that a user will automatically derive this experience from a database without considered input from its author – narrative is  “used as all-inclusive term, to cover up the fact that we have not yet developed a language to describe these new strange objects.”

Manovich argues that since databases are free of the “cause-and-effect trajectory” of the narrative form, they can, through ever-increasingly complex organizational forms come to represent a more complete simulacrum of reality.  The implication of his vision seems to be that databases will mimic real-life systems in incredible detail – a city, a historical figure, or even a whole historical society – and users will be able to interact with these simulacrums in apparently natural, non-narrative ways.

Imagine – if, instead of writing an exhaustive three volume biography of Theodore Roosevelt, Edmund Morris had programmed the entirety of his research into an algorithm which imitated Teddy himself.  Students of history wouldn’t need to read about Teddy – they could go bear hunting with a database that simulates his appearance, his behavior, his patterns of speech in virtual reality.  In this way, they could experience the man as he was – Teddy 2.0 would not shoot that simulated bear cub either.  Am I getting this right?

Each method – narrative and database – has its own merits to recommend it, but as the genre of database evolves into ever more sophisticated forms, narrative as a construct is likely to fall more and more by the wayside in favor of organizational techniques better suited to their unique matter.

A little help – am I overstating his argument?  Missing it completely?