about

The Project

Videojuicer managed at the time many video assets for the US-based classical record label New World Symphony (NWS). They wanted to be able to form an educational searchable library of all these assets, which users could browse and watch video’s relevant to them. It was a similar idea to Ted-Ed but aimed just at classical music.

Videojuicer had already built a proof of concept (POC) for this application in Ruby on Rails. The POC was dubbed phase 1 and this new application would be phase 2.

These videos have a lot of metadata around them that was required to give the videos context and educational value. This meant that any application that was developed would need to be able to manage this metadata, quickly and in a meaningful way.

The project was to be managed using an Agile methodology, which allowed the application to grow and develop with the user requirements over time. Like all agile projects, the aim was to develop and deliver working code at the end of every two-week sprint. We choose to use an agile method to give the client who was the product owner a sense of the progress we were making and what direction that the app was taking. This would allow us to adapt to any changes in the requirements, and allow the product owner to feedback any alteration to the specification they felt was required.

The Challenges

The performance was a key issue for the client, the previous version of the app was slow when searching. We identified that this had been caused by a large amount of data that has been stored and thus had to be searched through. The POC did not make use of any searching libraries such as Sphinx or Solr

The metadata had to be flexible but also remain meaningful. New metadatum keys had to be added easily and without the need for a developer. This meant that we would need to build some form of the metadatum key manager. This adds complexity to the search also as we will need to be able to search through an “unknown” list of metadata.

The customer had a set budget for this project that had been allocated to them. This meant that we had to consider what we could get done for that budget, and feed this back to the customer.

The last requirement was the system had to be easy to use, in affect this was going to use a crowdsourcing methodology to gather all the data around the videos. For the idea to work, the interface had to be easy enough to use that the end users would want to add additional data. The basic design and UX would be carried out by the client's design team, with advice from us.

Skills used

During the initial pitch and throughout the project, I was very much involved with the Agile project management. I started with taking the project requirements and specification and breaking it out into a set of user stories. This would then be used to get feedback from the client. Once all the relevant stories had been created, I would then take these stories and along with the other developer on this project allocate them to a developer and a sprint. To keep track and allow the business, and the product owner to easily see the progress we used the online-based project management tool Pivotal Tracker.

As the phase 1 application, we elected to develop phase 2 in Ruby on Rails. This was because both the other developer and myself where Ruby developers. The Rails application framework also allowed us to rapidly build and implement features; this made it ideal for the agile project philosophy that we were aiming for. Lastly, with the previous POC having been written in rails, this meant that we could re-use parts of the code.

Javascript and CoffeeScript were used to create the interactive front end. There weren’t many restrictions on this other than to keep it modular, maintainable and easy to use. We elected to use CoffeeScript because both developers were comfortable in it, it was also by then been in included in rails as default which meant no other steps would be needed to integrate it.

Due to the for the mentioned requirement that it had to have a flexible metadata system. It was decided that we would use a hybrid of a traditional SQL database and a NoSQL database, in this case, MongoDB. Mongo was chosen because I already had experience with it having used it for other projects, it was also the NoSQL solution used within Videojuicer, this meant that the other members of the development and server team were familiar with it also.

Work

Most of the initial work was planing and pitching. This meant generating project documentation, and distilling the customers requirements into a single set of Agile User Stories. The user stories would be the denotive last of what works needed to be carried out. It would also be used to measure progress throughout the project. I drafted , and refined the user stories , working with the product owner and other interested parties. I then took these user stories and entered them in to the Pivotal Tracker agile project management software.

Along with the other developer and product owner , we took these stories from the icebox and ordered them , adding them to the backlog. Pivotal tracker then generated a basic sprint plan based on the current project velocity. This meant that even before a line of code was written the product owner and other interested parties could see when features where likely to be delivered.

It was decided that sprint one , would not deliver any working code. This was because there where a fair few tasks that had to be carried out before work could start on actual features. This included configuring a new rails application with additional Gems that we required. For the traditional SQL solution we decided to make use of Datamapper. The original phase1 application had use Datamapper , having been partly written by Dan Kubb one of Datamapper’s maintainers. This meant that we were able to reuse models created for the original application , with the new application. For the NOSQL solution , we where using Mongo, originally I decided to use Mongo Mapper as that was the Mongo ORM that I was familiar with. Unfortunately Mongo mapper didn’t give us the control we wanted at the time , it also didn’t have a very tight integration with rails. We finally decided to go with Mongoid which has a tighter rails integration and better way of managing and accessing embedded documents.

The next part of sprint one was to pull apart the original POC application and throw away anything we didn’t want or need. We then took what was useful to use and integrated it into the new Rails application. This was mainly the models surrounding videos , the Videojuicer API and their specs.

We also moved from using Ruby 1.8.7 to 1.9.3 and Rails 2 to Rails 3. This required some of the code that we pulled from the POC to be refactored to work with the new Ruby and Rails version. The upgrade was required to not only take advantage of new Ruby and Rails performance and features , but also to make sure that we had all the latest security updates.

From there I built out a meta datum managing system based around Mongo. With documents to represent keys , meta data and how they linked to a specific video. I made a fundamental design floor here. I didn’t play on the strength of the NOSQL database design , and instead created a Frankenstein monster mash up of traditional SQL and NOSQL database design. I realised later that this was a mistake because it meant that we where not getting the best use out of mongo. Unfortunately nether the other developer or myself realised this mistake at the time. If I was to do this project again, I would likely build it so that a document represented a video, and contained all the meta data regarding the video. I would then a have a collection of documents that was the definitive list of meta datum , for validation purposes. This design would allow the data to be searched and access a lot quicker. It would also play more to Mongo DB strengths.

The rest of the sprints where very traditionally agile. Lasting approximately 2 weeks each with a 1 week gap in between as a contingency. Features where delivered to the product owner at the end of the sprint, they then evaluated the features and produced feedback.