First off, your Captain Preservation TV show idea made me burst out laughing.
I appreciate how you harped on the detrimental role that preoccupation with software can have on an institution trying to build up their digital preservation infrastructure. I admit, however, while reading Chudnov’s article, particularly when he talked about programmer’s lack of understanding of the users, I found myself thinking several times, “but if we have no programmers to help us work on software, how much time will we, intermediate to beginner level coders (at best) waste on the necessary grit of programming?”
Granted, this argument only becomes a real issue for larger collections that need a lot of customization in their software, and I’m sure Chudnov was highlighting an extreme of a very wide spectrum, but bear with me as I play devil’s advocate and pose a line of questioning:
To what extent should we as archivists know how to code ourselves or rely on ready-made software packages? And when (if ever) should we enlist the help of programmers? Is it more beneficial if the programmers do not understand how to connect with our users, but can code well, or if we the archivists know what our users want but spent ten times the amount of man-hours to code something? Or are the two options equally cost effective?
Chudnov himself said that archivists are, as a group, rubbish at programming. His solution seems to be “don’t worry about it, learn on the job”, which in most situations may be the best advice to follow. However, I wonder if following Chudnov’s advice to the letter would prove disastrous for some institutions that really should invest more time and funds (or what little they may have of it) into getting expert help in programming their digital environment properly.
Any (good) analog archivist knows that there is a certain point where they need to hand over their materials to a conservation team, who knows far more about chemistry (whether they appreciate the intellectual value of the object or not) and will ensure the piece can stick around for future generations to enjoy. Should not digital archivists know when it is time to hand off their digital infrastructures to the experts? Or is it, as Chudnov seems to suggest, simply not worth it?
You raise a lot of good points! I think the answers here are really variable, depending on resources: time, people, money. Something to keep in mind, though, is that you get better at programming/scripting as you do it. Perhaps the first job might take you ten times the hours, but then the next project should take you less, and so on. This would particularly be the case if you had a lot of collections going into digital repositories with the same types of materials and designated communities. Also, Chudnov suggests that archivists go for simple solutions first, thus reducing resource usage until it’s definitely needed. Of course, in situations that are clearly more complex, and in those where IT resources are already ready available, it might not be best.
This is an interesting discussion, and I particularly like the question Margot posed: “To what extent should we as archivists know how to code ourselves or rely on ready-made software packages?” Most archivists, unless they have a programming background, will come to the task unprepared to code a working program. Many, as Chudnov notes, don’t count software implementation as a strength. But that’s okay, because his main point, with which I agree, is that there is a learning curve. I don’t think archivists should entirely rely on ready-made software packages because I think it’s important for them to be familiar with the services they provide to the users. Like it or not, archivists are now inextricably intertwined with the world of computer programming. We can moan all we like, but unless we ourselves gain exposure to the back end of these programs, we won’t be able to be of service to the people who use the programs.
While I certainly would not want anyone to trust me with building a digital repository (at least not at the moment), I don’t think it would hurt anyone if I were to familiarize myself with the process. Maybe this time I might let an expert take the reins, but who’s to say I couldn’t be a bigger help next time?
I really appreciate you highlighting the idea of having content creators prepare their items for digital preservation. In an archival environment where “More Product, Less Process” processing (focused less on endless refoldering and more on providing access) is becoming more and more a necessity, it seems almost organic to apply similar attitudes to digital preservation. Just like it is a good idea to have a donor of paper files provide an inventory of their donation, I believe that requiring a similar level of participation from the donor/creator of digital files would similarly help with preservation of and access to digital materials.
It’s not a perfect solution–I imagine there are just as many content creators as there are archivists who might be unaware of concepts like fixity–but it is still a good policy.
Great point about MPLP! It’s interesting to consider how some aspects of digital and non-digital spaces allow us to use broad concepts across archives, even if some of the details might be a bit more variable.
Thank you for your (very entertaining) post! I think you are completely correct about organizations getting to Level 1, especially concerning file fixity – up until about 6 months ago, I had no idea what fixity or hashes or checksums even were, and I still wouldn’t if I had not taken very specific graduate-level courses on digital preservation. The field of digital preservation is growing exponentially, it is true, and the number of educated professionals who know about fixity (this is us!) is increasing (which is great! ). However, I hesitate to say that many people who are currently working in repositories, especially the small-scale organizations that you mentioned, and who may have been there a long time, (before they even had digital holdings), will not be aware of the importance of fixity and many other digital preservation practices as outlined by NDSA.
What is the solution here? You suggest making sure content creators “provide digital objects ready for preservation when they give materials to an archive,” a very valid point! This would eliminate the need for the archivists ingesting the materials to check fixity… at least at first. It at least would make everything much easier for the receiving archive/ist, and allow organizations to at least get to Level 1… or part of the way there. However, further preservation methods are critical once the material has been ingested, as outlined by OAIS and NDSA, and these are things that will be challenging, especially to those small-scale institutions, regardless of the state that the materials are in when donated.
Level 1 also includes assuring two copies of data that are not collocated. Small institutions may not have the resources to do this, whether it be man-power or simply space. The second Level 1 standard if fixity, as you discussed. An information security policy (the next Level 1 benchmark) should be doable… although could be easily overlooked. The same goes for an inventory of the content. The last criteria concerns file formats, and is another area of knowledge that staff at some institutions may not possess, and not have the time or resources to learn.
I like how Chudnov speaks about not over-thinking things, from the complexity of content to repository software choices. Perhaps this mindset can be extended to the NDSA Level 1 criteria for smaller institutions with limited resources? I’ve proposed this without actually knowing how it could be implemented… but I think Mr. Chudnov is on to something. What do you think?
To start, I think to a certain extent, people who are educated in digital preservation have to spread the word around, and hopefully over time, more organizations will have access to people who have experience with or knowledge of digital preservation.
I like your line of thinking, with going more simple– so for example, start with the quickest things first. Get your files backed up to an external hard drive and store it somewhere at least one coffee-spill away from the computer you have the files on. Check and make sure your files are the same format and size as the ones your donor sent you. TIFFs for life. I think the question then becomes, what do you prioritize first?
I had to think about some of these same topics at work this week. We refer to NDSA Levels pretty often, and have been discussing our file fixity levels for a while (we are at level 1, more or less, though with pieces of levels 2 and 3 in place). For creating fixity info, we use Bagit, which I am accustomed to running via command line and batch scripts.
I decided this week to install Bagit on a new machine that we use for processing archival images, as part of an effort to generate checksums as early as possible in our workflow. When I looked up the Bagit software, I discovered that the newest Java version, which I use on other machines, no longer allows command-line functionality. Grr. They have a Python version, which I am now playing with, but I have to rethink a lot of little automated functions we use (or else find old versions of the Java app).
In a way, this is fine. The Python version is probably way better. I’ll figure out how to adapt our scripts. But I think this little scenario highlights a big problem for the digital preservation community. As everyone is pointing out, most people have never even heard of fixity. If it is so important, the community needs to be able to deliver tools to organizations that will never have staff who will look at github repositories and rewrite code. Bagit is really simple in theory! But these tools need to work for the average volunteer at a cultural institution, not just people who are learning programming. They need to be stable and backward-compatible, and at least as easy to use as, say, adding a row to an Excel spreadsheet.
I don’t think people should have to learn coding to do digital preservation, but I do think the programmers need to do a much better job of establishing tools for ordinary people. The NDSA philosophy of being less intimidating needs to apply to a lot more aspects of the community beyond the scope of their standards.
Hello Alice,
First off, your Captain Preservation TV show idea made me burst out laughing.
I appreciate how you harped on the detrimental role that preoccupation with software can have on an institution trying to build up their digital preservation infrastructure. I admit, however, while reading Chudnov’s article, particularly when he talked about programmer’s lack of understanding of the users, I found myself thinking several times, “but if we have no programmers to help us work on software, how much time will we, intermediate to beginner level coders (at best) waste on the necessary grit of programming?”
Granted, this argument only becomes a real issue for larger collections that need a lot of customization in their software, and I’m sure Chudnov was highlighting an extreme of a very wide spectrum, but bear with me as I play devil’s advocate and pose a line of questioning:
To what extent should we as archivists know how to code ourselves or rely on ready-made software packages? And when (if ever) should we enlist the help of programmers? Is it more beneficial if the programmers do not understand how to connect with our users, but can code well, or if we the archivists know what our users want but spent ten times the amount of man-hours to code something? Or are the two options equally cost effective?
Chudnov himself said that archivists are, as a group, rubbish at programming. His solution seems to be “don’t worry about it, learn on the job”, which in most situations may be the best advice to follow. However, I wonder if following Chudnov’s advice to the letter would prove disastrous for some institutions that really should invest more time and funds (or what little they may have of it) into getting expert help in programming their digital environment properly.
Any (good) analog archivist knows that there is a certain point where they need to hand over their materials to a conservation team, who knows far more about chemistry (whether they appreciate the intellectual value of the object or not) and will ensure the piece can stick around for future generations to enjoy. Should not digital archivists know when it is time to hand off their digital infrastructures to the experts? Or is it, as Chudnov seems to suggest, simply not worth it?
You raise a lot of good points! I think the answers here are really variable, depending on resources: time, people, money. Something to keep in mind, though, is that you get better at programming/scripting as you do it. Perhaps the first job might take you ten times the hours, but then the next project should take you less, and so on. This would particularly be the case if you had a lot of collections going into digital repositories with the same types of materials and designated communities. Also, Chudnov suggests that archivists go for simple solutions first, thus reducing resource usage until it’s definitely needed. Of course, in situations that are clearly more complex, and in those where IT resources are already ready available, it might not be best.
The comparison you make with conservation is an interesting one. Even here, I think resources come into play– the Library of Congress even states, “Conservation work to address damage is time consuming and costly to do correctly” and goes on to suggest leaving items as-is in certain circumstances.
Hey Alice and Margot,
This is an interesting discussion, and I particularly like the question Margot posed: “To what extent should we as archivists know how to code ourselves or rely on ready-made software packages?” Most archivists, unless they have a programming background, will come to the task unprepared to code a working program. Many, as Chudnov notes, don’t count software implementation as a strength. But that’s okay, because his main point, with which I agree, is that there is a learning curve. I don’t think archivists should entirely rely on ready-made software packages because I think it’s important for them to be familiar with the services they provide to the users. Like it or not, archivists are now inextricably intertwined with the world of computer programming. We can moan all we like, but unless we ourselves gain exposure to the back end of these programs, we won’t be able to be of service to the people who use the programs.
While I certainly would not want anyone to trust me with building a digital repository (at least not at the moment), I don’t think it would hurt anyone if I were to familiarize myself with the process. Maybe this time I might let an expert take the reins, but who’s to say I couldn’t be a bigger help next time?
Alice,
I really appreciate you highlighting the idea of having content creators prepare their items for digital preservation. In an archival environment where “More Product, Less Process” processing (focused less on endless refoldering and more on providing access) is becoming more and more a necessity, it seems almost organic to apply similar attitudes to digital preservation. Just like it is a good idea to have a donor of paper files provide an inventory of their donation, I believe that requiring a similar level of participation from the donor/creator of digital files would similarly help with preservation of and access to digital materials.
It’s not a perfect solution–I imagine there are just as many content creators as there are archivists who might be unaware of concepts like fixity–but it is still a good policy.
Great point about MPLP! It’s interesting to consider how some aspects of digital and non-digital spaces allow us to use broad concepts across archives, even if some of the details might be a bit more variable.
Hi Alice!
Thank you for your (very entertaining) post! I think you are completely correct about organizations getting to Level 1, especially concerning file fixity – up until about 6 months ago, I had no idea what fixity or hashes or checksums even were, and I still wouldn’t if I had not taken very specific graduate-level courses on digital preservation. The field of digital preservation is growing exponentially, it is true, and the number of educated professionals who know about fixity (this is us!) is increasing (which is great! ). However, I hesitate to say that many people who are currently working in repositories, especially the small-scale organizations that you mentioned, and who may have been there a long time, (before they even had digital holdings), will not be aware of the importance of fixity and many other digital preservation practices as outlined by NDSA.
What is the solution here? You suggest making sure content creators “provide digital objects ready for preservation when they give materials to an archive,” a very valid point! This would eliminate the need for the archivists ingesting the materials to check fixity… at least at first. It at least would make everything much easier for the receiving archive/ist, and allow organizations to at least get to Level 1… or part of the way there. However, further preservation methods are critical once the material has been ingested, as outlined by OAIS and NDSA, and these are things that will be challenging, especially to those small-scale institutions, regardless of the state that the materials are in when donated.
Level 1 also includes assuring two copies of data that are not collocated. Small institutions may not have the resources to do this, whether it be man-power or simply space. The second Level 1 standard if fixity, as you discussed. An information security policy (the next Level 1 benchmark) should be doable… although could be easily overlooked. The same goes for an inventory of the content. The last criteria concerns file formats, and is another area of knowledge that staff at some institutions may not possess, and not have the time or resources to learn.
I like how Chudnov speaks about not over-thinking things, from the complexity of content to repository software choices. Perhaps this mindset can be extended to the NDSA Level 1 criteria for smaller institutions with limited resources? I’ve proposed this without actually knowing how it could be implemented… but I think Mr. Chudnov is on to something. What do you think?
To start, I think to a certain extent, people who are educated in digital preservation have to spread the word around, and hopefully over time, more organizations will have access to people who have experience with or knowledge of digital preservation.
I like your line of thinking, with going more simple– so for example, start with the quickest things first. Get your files backed up to an external hard drive and store it somewhere at least one coffee-spill away from the computer you have the files on. Check and make sure your files are the same format and size as the ones your donor sent you. TIFFs for life. I think the question then becomes, what do you prioritize first?
Alice and everyone,
Great discussion!
I had to think about some of these same topics at work this week. We refer to NDSA Levels pretty often, and have been discussing our file fixity levels for a while (we are at level 1, more or less, though with pieces of levels 2 and 3 in place). For creating fixity info, we use Bagit, which I am accustomed to running via command line and batch scripts.
I decided this week to install Bagit on a new machine that we use for processing archival images, as part of an effort to generate checksums as early as possible in our workflow. When I looked up the Bagit software, I discovered that the newest Java version, which I use on other machines, no longer allows command-line functionality. Grr. They have a Python version, which I am now playing with, but I have to rethink a lot of little automated functions we use (or else find old versions of the Java app).
In a way, this is fine. The Python version is probably way better. I’ll figure out how to adapt our scripts. But I think this little scenario highlights a big problem for the digital preservation community. As everyone is pointing out, most people have never even heard of fixity. If it is so important, the community needs to be able to deliver tools to organizations that will never have staff who will look at github repositories and rewrite code. Bagit is really simple in theory! But these tools need to work for the average volunteer at a cultural institution, not just people who are learning programming. They need to be stable and backward-compatible, and at least as easy to use as, say, adding a row to an Excel spreadsheet.
I don’t think people should have to learn coding to do digital preservation, but I do think the programmers need to do a much better job of establishing tools for ordinary people. The NDSA philosophy of being less intimidating needs to apply to a lot more aspects of the community beyond the scope of their standards.