Digitising an Archive: The Factory Approach
The FP6 PrestoSpace Project [1] aims to develop systems that will permit quick, efficient and economically accessible preservation of analogue media [2].
Stream UK has built on the knowledge gained from three years of working on this project along with expertise from over seven years encoding within the industry to develop a complete encoding factory solution, based on the PrestoSpace project where the focus is on developing a semi-automated 'preservation factory' approach to preservation of audio-visual collections aimed at driving down the cost of digitising the archive below the 1€ per minute level.
The solution is housed in a stand-alone rack unit and includes barcode tracking of tapes through the system to allow archive owners to take an existing collection of off-line material and convert it into a high-quality archive and a Web-ready preview format including all existing metadata and maintaining an overview of the process via the barcode tracking system which produces a searchable interface that can be accessed via a LAN or Internet connection.
Each unit includes:
- A stand-alone encoding system built into a portable rack unit
- Barcode tracking of tapes
- The facility to import metadata
- Live video capture in multiple formats
- Deck control via a frame-accurate RS232 interface
- Batch processing of multiple encoded clips simultaneously
- Encoding to all formats MPEG1 & 2 (4:2:2 & 4:2:0 at up to 80Mbps), Windows Media, Quicktime, Real, MPEG4, WAV, AVI, Flash, MP3
- Import from and export to an image sequence
- Email notification and FTP distribution
- Noise reduction, colour correction, horizontal and vertical filtering.
Figure 1: The Stream UK Encoding Factory
This article looks at:
- Tracking the process of digitisation from tape to fully indexed archive;
- The encoding technology behind the system;
- Storage of the archive;
- Encoding formats that are available; and
- A conclusion on the whole process.
Complete Tracking of the Process
During the encoding of an archive, maintaining control of the assets and an overview of the process is essential. Stream UK has developed an interface which allows:
- Existing metadata to be entered;
- Individual tapes to be tracked via barcodes; and
- An overview of the status to be accessed via a normal Web browser.
Any tape that is introduced to the system goes through a five-step process:
- Any metadata that exists in electronic format is imported into the database. If this includes editing points then these will be recognised by the system.
- A barcode is added to the tape and scanned into the system.
- The tape is inserted into the player and the appropriate encoding profiles selected.
- When the tape has finished encoding (real-time) it is removed from the system.
- The status of the job is recorded on the board.
The Encoding Technology
The engine that powers the Stream UK Encoding Factory is based on the Digital Rapids [3] range of hardware encoders. These encoding cards are considered by experts to be the best solution for obtaining a digital format from a non-digital source and include the facility to ingest via SDI-capture [4].
Figure 2: The Encoding Administration Area
The key video features of the hardware video processors include:
- Key Video Processing Features
- Motion adaptive de-interlacing
- 3:2 and 2:2 film process de-interlacing
- 3D (temporal) noise reduction
- 2D (spatial) noise reduction
- Independent horizontal and vertical filtering
- Aspect ratio conversion
- Proc-amp controls
Different cards are available to support hardware MPEG-2 or MJPEG 2K encoding for the archive, with ingest being either via an in-built player or via inputs attached to the breakout panel.
All encoding is done in real-time, with deck-control allowing hands-free operation. The Factory will plug in directly to any robotics-based playout system.
Figure 3: Details of Individual Tapes
Storage of the Archive
Storage of a multimedia archive needs to be appropriate to the use that media is put to. The increasing affordability of disk storage makes RAID5 an ideal option for both the high-quality archive and the Web-ready fomats.
According to the PrestoSpace Storage Media Quality Assurance report [5], RAID5 offers the ideal combination of access speed, price and reliability.
The Stream UK Encoding Factory comes with a standard 2 terabytes of storage included, enough for around 100 hours of archive footage encoding in both high-quality archive (MPEG-2 25Mpbps, for example) and preview formats.
Additional storage is available within the same chassis up to 1 petabyte.
Encoding Formats
Stream UK will work with the archive owner to develop a complete set of appropriate profiles for the encoding formats.
Any archive owner seeking to digitise will need to select formats that are appropriate to the end-user. Normally this means:
- a high-quality archive that is suitable for:
- future transcoding; and
- as a direct replacement for the original off-line copies.
- A preview-quality archive that is suitable for users to see what is available in the full archive.
The Stream UK Encoding Factory supports all of the following formats, including support for Windows Media DRM. The table below provides a summary of the comparisons between the various formats:
Table 1 - Comparison of Encoding Formats
Format | Comments | Use |
| MPEG-1 | Part of the Motion Pictures Experts Group [6] set of formats, MPEG-1 has now been superceded by some of the later profiles. It does still have advantages in terms of playability across a wide range of software clients, but it has been left behind in terms of quality - any preview material needs to be encoded, stored and delivered at a comparitively high bit-rate. | MPEG-1 is good for preview encoding at bit-rates from 500Kbps to 2Mbps. |
| MPEG-2 | Still the benchmark for hig-quality archive encoding. Although MPEG-2 is a lossy format (meaning that some of the original information is thrown away by the encoding process), data-rates over about 20Mbps result in excellent quality. Parameters such as GOP need to be chosen with care. MPEG-2 is the standard for a wide range of applications across the entire A/V industry including all the transcoding software, playout over broadcast channels, editing suites etc. Archive-quality footage can also be easily viewed on a standard PC. | MPEG-2 is good for archive encoding at bit-rates from 8Mbps to 80Mbps. |
| MPEG-4 | This codec belongs to the next generation of encoding profiles, along with the common proprietory formats like Windows Media. The quality is comparable and MPEG-4 is supported by a wide range of platforms. Digital rights management and copyright issues remains outstanding. | MPEG-4 is good for preview encoding at bit-rates from 200Kbps to 2Mbps. |
| MJPEG 2K | This is a relatively new codec and it is only in 2006 that some of the major encoding card manufacturers began to implement the scheme within hardware encoding cards. Encoding in MJPEG 2K is lossless, which is attractive to many archives. Playback is not yet possible on normal PCs but transcoding gives excellent results. | MJPEG 2K results in bit-rates of around 80Mbps. |
| Windows Media 9 | Windows Media 9 is the proprietory codec with the highest market penetration, it can be viewed by around 73% of all Internet-connected computers. This codec is also very useful for preview-quality encoding since the DRM capabilities are the most advanced and allow content owners to maintain control of assets even through open systems. Later versions of editing software can edit this format natively. | Windows Media is good for preview encoding at bit-rates from 200Kbps to 2Mbps. |
| Quicktime | There is no single QuickTime format, but the .mov file extension includes a range of codecs of which the Sorenson 3 video codec and the Qdesign2 audio codec are leading the way. Quality is comparable with Windows Media and Mpeg-4 although a shade lower for most types of footage. QuickTime is easily viewable by those with Apple OS, something that is often attractive to the creative or educational sectors. Editing support is good from all major providers, as is transcoding support. | Suitable for both preview quality and archive quality at a range of bit-rates from 200Kbps to 30Mbps. The QuickTime formats are well supported by the professional edit suites available. Quality of transcoding is not as high as for MPEG-2 and MJPEG 2K. |
Conclusion
The conversion of an archive into digital format is one of the most important steps that the owner will ever take. Essential to this process is the correct choice of process and equipment. The Stream UK Encoding Factory allows archive owners to take control over the choices presented to them and to use an expert resource before and during the process. The solution as presented is most suitable for archives that have between 50 and 500 hours of footage to digitise, although multiple units can be integrated for larger archives.
References
- PrestoSpace Project http://www.prestospace.org
- Daniel Teruggi, "Can We Save Our Audio-visual Heritage?", April 2004, Ariadne Issue 39.
http://www.ariadne.ac.uk/issue39/teruggi/ - Digital Rapids PCI cards http://www.digital-rapids.com/Products_PCICards.html
- Editor's note: SDI = Serial Digital Interface
- PrestoSpace Storage Media Quality Assurance report http://www.prestospace.org/project/deliverables/D12-6.pdf
- Motion Pictures Experts Group: http://www.mpeg.org