In the end of the spring we started a new website project for one of our customers where one of the criterias was to integrate an external media library. The media library had a set of webservice-methods which provided us with the structure and files of the media library and the data.
Our goal was to implement the media library into Sitecores own media library without accualy storing it there, and to use Sitecores media-cache and resize engine.
This posed a series of questions
- How would we integrate the media library?
- How would we store the used media-items without having to store everything from the external media library inside sitecore and thus publishing it to the web-database?
- How would we be able to use Sitecore media-cache and resize engine?
- And finally, what impact to the systems preformance would all this do?
This part will cover the first questions namely how to integrate the media library.
When integrating the external media library, we constructed a DataProvider that would call the external media librarys webservices and add them as items in the media library tree. For this we used and modified the dataprovider from the Share resource youtube module.
What we did first was creating templates for each of the object-types we were going to collect.
The basic idea here was to use alot of sitecores functionality, so these templates interited from each of the sitecore media templates. And since all of the would have some information in common, we created a base templates which containd the external media item id and the path to the item.
When this was done we created what would become the root of the external media library inside sitecore media library by simply create an item based on the Media Library Folder.
Now we were set to continue with the dataprovider, so now lets go on to the coding part:)
Since we had our set of templates and one of the templates would act as a folder, we only wanted the dataprovider to trigger when the folder template was expanded. To achive this we would have to override the method GetChildIDs.
public override IDList GetChildIDs(ItemDefinition itemDefinition, CallContext context)
{
if (itemDefinition.TemplateID.Equals(new ID(_folderTemplateID)))
{
ID parentId = itemDefinition.ID;
IDictionary<string, string> ids = LoadDataID(parentId);
IDList idList = new IDList();
foreach (var itemId in ids)
{
IDTableEntry idEntry = IDTable.GetID(prefix, itemId.Key);
ID newId;
if (idEntry == null)
{
newId = ID.NewID;
IDTable.Add(prefix, itemId.Key, newId, parentId, itemId.Value);
}
else
{
newId = idEntry.ID;
}
idList.Add(newId);
}
return idList;
}
return null;
}
We started of with this. Here we have the folder template id stored in a private variable _folderTemplateID and checks if the current item is based on that template. If the current item is a our folder item we call the LoadDataID which in turn calls the webservices and provides a dictionary with name and id. These are then, based on wether they exist or not in the Sitecore IDTable, either added or fetched from the IDTable.
This worked fine until we started to modify inside the external media library. When we started to move items inside the media library we started to get some NullReferenceException. After some troubleshooting we finally understood what was going wrong. The Sitecore IDTable acctually caches the ID added to it for some time. So if an external media item where to change the its parent, we would stil get the old parent id reference from the Sitecore IDTabel which in turn would spell dissaster for us.
What we did was going through all keys in the IDTable stored for our prefix that had the current parentId and did not exist in the Dictinary of IDs provided by the webservice.
public override IDList GetChildIDs(ItemDefinition itemDefinition, CallContext context)
{
if (itemDefinition.TemplateID.Equals(new ID(_folderTemplateID)))
{
ID parentId = itemDefinition.ID;
IDictionary<string, string> ids = LoadDataID(parentId);
IDList idList = new IDList();
foreach (var itemId in ids)
{
IDTableEntry idEntry = IDTable.GetID(prefix, itemId.Key);
ID newId;
if (idEntry == null)
{
newId = ID.NewID;
IDTable.Add(prefix, itemId.Key, newId, parentId, itemId.Value);
}
else
{
newId = idEntry.ID;
}
if (!_handledItemIds.Contains(newId))
_handledItemIds.Add(newId);
idList.Add(newId);
}
foreach (var entry in IDTable.GetKeys(prefix).Where(k => k.ParentID.Equals(parentId) && !ids.Keys.Contains(k.Key)))
{
IDTable.RemoveID(prefix, entry.ID);
if (_handledItemIds.Contains(entry.ID))
_handledItemIds.Remove(entry.ID);
}
context.DataManager.Database.Caches.DataCache.Clear();
return idList;
}
return null;
}
The _handledItemIds is a list that is used to check the current item is something we can process in the GetItemDefinition function which is the next step in our code.
The GetItemDefinition is a method in the DataProvider class that creates a ItemDefinition that tells the system what kind of item it is (Template), what ID it has and its name. Here we only want to handle the items we got from the webservice and to avoid having to look in the IDTable if our ID exist there, which takes a loooong time. So we stored all handled item ids in the _handledItemIds and basically created a method that checks if a certain id exists.
So what we did was override the GetItemDefinition which is called everytime Sitecore displays an item and checked it the current request was for a external media item.
public override ItemDefinition GetItemDefinition(ID itemId, CallContext context)
{
ItemDefinition itemDef = null;
string itemName = string.Empty;
if (CanProcessMediaItem(itemId, context))
{
string originalID = GetOriginalRecordID(itemId);
ExtMediaData itemInfo = GetMediaDataInfo(itemId);
if (itemInfo == null)
{
var n = GetCachedExternalNode(originalID);
itemName = ItemTools.GetValidNodeName(n.Name);
var resourceTid = GetResourceID(n);
itemDef = new ItemDefinition(itemId, itemName, resourceTid, ID.Null);
_items.Add(itemId, new ExtMediaData(itemId, resourceTid, itemName, n));
}
else
{
itemDef = new ItemDefinition(itemId, itemInfo.Name, itemInfo.TemplateID, ID.Null);
}
try
{
ItemCache itemCache = CacheManager.GetItemCache(context.DataManager.Database);
if ((itemCache != null) && (itemDef != null))
{
ReflectionUtil.CallMethod(itemCache, "RemoveItem", true, false, false, new object[] { itemDef.ID });
}
}
catch (Exception exception)
{
Log.Error("Can't clear cache for the External Media item", exception, this);
}
// Do not cache items for this data provider
if (itemDef != null)
((ICacheable)itemDef).Cacheable = false;
}
return itemDef;
}
This method can be a little confusing, but what we do here is first checking if the current request is for a external media item by running “if (CanProcessMediaItem(itemId, context))”, this method checks in the _handledItemIds if the id exist and returns true or false. What we check next is if we have done this procedure before and cached the information. We do this by calling ExtMediaData itemInfo = GetMediaDataInfo(itemId); which returns an ExtMediaData from a Hashtable of it exist. If it exist we create an ItemDefinition based on the info, however if it doesn’t exist we will have to create it request all information. To do this we need the id of the external media item which this request does “string originalID = GetOriginalRecordID(itemId);”. Since we store the original id in the sitecore IDTable this method just checks in the IDTable and returns the referenced id.
string GetOriginalRecordID(ID id)
{
IDTableEntry[] idEntries = IDTable.GetKeys(prefix, id);
if (idEntries != null && idEntries.Length > 0)
{
return idEntries[0].Key;
}
return null;
}
Now we have the id, and early in the project we did another webservice request to get the media from the external media library. However we soon found out that this took to long. What we did was to modify LoadDataID called in GetChildIDs to store all collected media items in a collection that we lated use to fetch the item again from within GetItemDefinition by calling “var n = GetCachedExternalNode(originalID);”. Now we only need the Template id for the item. From the webservices provided from the external media library we only got the file extension, so we created a function that based on an extension return the correct template id. Nothing fancy. Basically a switch.
We now have all the information we need to create an ItemDefinition. This is then cached in the same HashTable that is queried earlier in GetItemDefinition so that we don’t have to request the webservice again for this item during this session.
The last thing we do is override the GetItemFields where we, based on “hard coded” fields, return the values from the external media item.
public override FieldList GetItemFields(ItemDefinition itemDefinition, VersionUri versionUri, CallContext context)
{
FieldList fields = new FieldList();
if (CanProcessMediaItem(itemDefinition.ID, context))
{
string originalID = GetOriginalRecordID(itemDefinition.ID);
TemplateItem templateItem = ContentDB.Templates[itemDefinition.TemplateID];
if (templateItem != null)
{
ExtMediaData ytItemInfo = GetMediaDataInfo(itemDefinition.ID);
if (ytItemInfo != null)
{
foreach (var field in templateItem.Fields)
{
var value = GetFieldValue(field, ytItemInfo);
if (!string.IsNullOrEmpty(value))
fields.Add(field.ID, value);
}
}
}
}
return fields;
}
Ofcourse we check if this is an item that we want to handle by calling CanProcessMediaItem. Then we get the cached media item, and the corresponding sitecore template. What we want to do here is to go through all fields, and collect the data from the media item into the corresponding field. We had some trouble here because we allways added the result into the field, even if it was empty, which was a misstake, since we wanted to use the templates standard values aswell. But by checking if the returned value was empty or not, we solved that problem.
The GetFieldValue isn’t realy that interesting, but I’ll show it anyway:)
private string GetFieldValue(TemplateFieldItem field, ExtMediaData itemInfo)
{
string val = string.Empty;
switch (field.Name)
{
case "External Path":
val = itemInfo.Data.Adress;
break;
case "Width":
val = itemInfo.Data.Width.ToString();
break;
case "Height":
val = itemInfo.Data.Height.ToString();
break;
case "Title":
val = itemInfo.Data.Name;
break;
case "Extension":
val = itemInfo.Data.Extension.Replace(".", "");
break;
case "Mime Type":
val = itemInfo.Data.MimeType;
break;
case "ExtMediaId":
val = itemInfo.Data.Id.ToString();
break;
case "Size":
val = itemInfo.Data.Size.ToString();
break;
case "Format":
break;
case "Dimensions":
break;
}
if (val == null) { val = string.Empty; }
return val;
}
As you can see it only checks if the current field is one of those i want to add information to, and returns the value.
With this done, we had to configure the dataprovider to work in the master database. We did this by creating an include file that looked like this:
<configuration xmlns:patch="http://www.sitecore.net/xmlconfig/" xmlns:x="http://www.sitecore.net/xmlconfig/">
<sitecore>
<dataProviders>
<externalmediadp type="Pyramid.DataProviders.ExternalMediaDataProvider, Pyramid.DataProviders">
<param desc="folderTemplateID">{416F14A1-0D15-422D-991E-C8039B074F05}</param>
<param desc="resourceTemplateFolder">{369D90E5-2394-4E6C-8BF1-2BCDE4EDDA28}</param>
<param desc="externalMediaField">ExtMediaId</param>
<param desc="contentDatabase">master</param>
</externalmediadp>
</dataProviders>
<databases>
<database id="master" singleInstance="true" type="Sitecore.Data.Database, Sitecore.Kernel">
<dataProviders hint="list:AddDataProvider">
<dataProvider ref="dataProviders/externalmediadp">
<disableGroup>delete</disableGroup>
</dataProvider>
</dataProviders>
</database>
</databases>
</sitecore>
</configuration>
Here we define a dataprovider and provide the constructor with some attributes, “folderTemplateID”, “resourceTemplateFolder”, “externalMediaField”, “contentDatabase”. To get this attributes, the will have to be added to the dataproviders contructor and then store them in private variables. Then we add the dataprovider to the master database.
Now the dataprovider will call the webservice, get all items and folders from the external media library to the sitecore media library. Or will it?
Theoretically yes, however our provider is built in such a way that it needs an external parent id to get the external children… yes?… Well what about the root folder? yeah we will have to configure it with an external id aswell. And as you saw in the GetFieldValue we acctually have a field called “ExtMediaId” where we store all the collected items. So after configuring our root folder, that we manually created earlier, with the media librarys root item id it now works.
To summarize this part, well something we have learned is to never request an item from sitecore using the the database. By doing this you might end up in an infinit loop. For example if we would request a external folder item using for exampel Sitecore.Context.Database.Item[theId], this would trigger all configured dataproviders for that database to call GetItemDefinition for that item, among them your dataprovider will be called. And depending on where in the code you call did the call, you might end up doing the same request again.
Our solution to this was to create an instance of the ItemProvider and do our request using it, like this:
var provider = new ItemProvider(); Item item = provider.GetItem(theIdTotheItemYouWant, ContentDB.Languages[0], Sitecore.Data.Version.Latest, ContentDB, SecurityCheck.Enable);
This specifies that its an item from sitecore you want that exist in the database and nothing more.
Thats all for now… stay tuned for the next part where we will look in to how we did to store the used media without storing the entire media library in sitecore. And we will also look at how we did to use sitecore resize and caching for media.
Best regards