Below are some of the most important questions and concepts you need to keep in mind when designing or choosing a video Transcoding system. They are described in general terms and more detail is needed to really optimize a Transcoding system for your particular workflow. Keep in mind that there are systems that require some level of programming or scripting and that might have a big impact on your budget.
In deciding how to automate your video Transcoding workflow, you should ask your self the following questions:
1) What is the volume of videos you need to process?
2) How fast do you need the output videos?
3) Where do the input videos reside? (Single or multiple locations.)
4) What is the input format(s)? (Device, file format.)
5) What is the output file format(s)?
6) Where do you need to deliver the output videos?
The answer to the first couple of questions will help you determine how large and complex your system is going to be. If your volume is very low, from 1 to a couple of dozen videos a week, you could set up a simple software-based, batch encoding system in a single computer. As your video demands begin to grow, you could batch process the videos in more than one computer, and manage the queue manually.
There is a point where manually managing a queue not only becomes a full time job, but also introduces too many human errors. You should consider setting up a system with automated queue management. The queue system should provide you with different error handling choices. How you handle encoding errors will ensure that your Transcoding system continues to operate on subsequent files. It is critical to prevent your queue from jamming up due to an expected error. They queue system should allow you to set and change priority levels for your Transcoding jobs and ultimately, it should let you dictate how you allocate both processing and storage resources. It should let you bring online and take off line additional hardware without shutting down the system.
As your volume and throughput demand increases, you need to begin to consider what kind of hardware you need to use with your software solution or whether you need a hardware accelerated platform. If you want to stay with a software based solution you need to look at multiple CPU computer systems, what kind of file storage (disk I/O is usually the slowest part of the system) is available, and Network connectivity. Keep in mind that the Transcoding process takes a good portion of the time in processing your videos, but it is network latencies and disk I/O bottlenecks what often brings down productivity. A good system makes sure that more than enough bandwidth is allocated for file I/O.
The location of your input videos is an important consideration on how you design your system. Do you have a library of content that resides in a single location? You should have then a Transcoding system that is located closest to the content and make sure that the data throughput from your sources has enough bandwidth to avoid I/O bottlenecks.
Does the content reside at multiple locations? You might increase your throughput by having a encoding system at a few of the source locations. Your queue system and asset management system should be designed in a way that it allows you to distribute the load across locations if necessary.
Is the content being sent to you from many locations? Here, you have different factors at play and a few options. First, you must make sure that you have enough bandwidth to receive the input videos. Here you can leverage a CDN to increase your bandwidth and take advantage of the geographical proximity to the submitter. Second, you must make sure that your file storage (for input files) is adequate and scalable. Third, you might be better off setting up a distributed encoding solution where videos are converted at a point closest to the submitter, or even at the submitter end. Fourth, content management must be able to track all the incoming videos and move them to the output location(s) with flexibility.
Input formats or devices that need to be supported are important constrains in designing the Transcoding system. Do you need tape support? Do you need analog or digital video inputs? (SDI?) If so, you need a capture card with a serial machine control interface and make sure that it supports the right kind of video inputs. You also need to make sure that the input file format is supported by your hardware of software Transcoding system.
The output file format(s) is also an important consideration. Do you need to output videos in one single format? What format? Some systems charge more for certain output formats or doesn’t support them at all. Don’t assume, read the specifications or ask the vendor.
Where and how you are going to deliver your videos is the last piece of the Transcoding system. Again, you need to make sure that you have enough bandwidth to support your desired output rate. Consider a CDN when you need to deliver to multiple geographic locations.
Next post will cover user generated video transcoding workflows.
3 comments:
Thanks for writing about encoding, workflows,... As you mentioned if volume increases you need a workflow to manage encodes between machines,.. have you seen some application - workflow engines to handel this functionality. I worked with watchdirectory not open source but a small "cheap" and really helpful application.
The professional systems I described in a previous post offer this kind of functionality right out of the box. Rohzet, DigitalRapids, Inlet Designs products, etc.
Managing transcoding across machines requires some form of a message queuing system. Typically, you would have a Server distribute the work across the different machines available. If you work in Windows, you can use the MSMQ or WCF, but it requires some development. I think that even of some facilities have created their own encoding farms using FFMPEG.
Thanks for the quick comment. Yes, we used Rhozet and Flipfactory for professional encoding. I was thinking more about some UGC as you mentioned some facilities have created some encoding farms using ffmpeg, but could not find to much information about that. Well is probably the big secret of that companies ;)
Post a Comment